Nursing Center - Journal Issue

Keywords: Evaluative tasks, Judgment process, Nursing, World Wide Web

Abstract

The World Wide Web is increasingly becoming an important source of information for healthcare professionals. However, finding reliable information from unauthoritative Web sites to inform healthcare can pose a challenge to nurses. A study, using grounded theory, was undertaken in two phases to understand how qualified nurses judge the quality of Web nursing information. Data were collected using semistructured interviews and focus groups. An explanatory framework that emerged from the data showed that the judgment process involved the application of forms of knowing and modes of cognition to a range of evaluative tasks and depended on the nurses' critical skills, the time available, and the level of Web information cues. This article mainly focuses on the six evaluative tasks relating to assessing user-friendliness, outlook and authority of Web pages, and relationship to nursing practice; appraising the nature of evidence; and applying cross-checking strategies. The implications of these findings to nurse practitioners and publishers of nursing information are significant.

The Internet has been described as the "incubation of knowledge"1^(p38) and is becoming an increasingly important source of health information. The Internet and its subset, the World Wide Web, henceforth to be referred to as the Web, have deregulated information publishing empowering Web users, including health consumers, practitioners, and researchers. The number of people searching the Internet for health information is increasing as the medium is convenient and cheap.2 Its cost-effectiveness and easy accessibility3 are making it an indispensable tool for healthcare professionals. However, the exponential growth of Web information has not been reciprocated with an expansion in validated information. In particular, the quality of information from less authoritative Web sites remains of a variable nature.4-8 Because anyone can publish on the Web, assessing the credibility of its content becomes of the essence and at the same time difficult.9,10 The focus of this article relates to the findings of a study on how nurses judge nursing information on the Web.

Literature Review

Healthcare professionals are utilizing health information on the Web for clinical purposes and for updating professional knowledge, but they are concerned about its quality.11 Nurses have identified the Web as an information resource for knowledge acquisition on nursing care and nursing interventions.12 As a digital resource, the Web can match timeliness, context, and relevance, yet trustworthiness cannot be ensured because finding reliable information from unauthoritative Web sites to inform healthcare can pose a challenge to nurses,8 making the judging of online information for evidence-based care imperative. Access to reliable health information has been identified as an issue in a worldwide survey by Health on the Net Foundation (HON).13 Respondents stated that improvement in credibility, trustworthiness, and accuracy of online health information would help. A survey14 found that 42% of advanced practitioner nurses used evidence from the Web to support their practice, but only 48% considered themselves competent at evaluating Web information. Judging the accuracy and reliability of Internet health information can be a challenge for both health professionals and patients because good and inaccurate information is published side by side.2

There is evidence to suggest that many patients visiting health professionals with health information garnered from the Internet had misunderstood their medical conditions because of misinterpretation of online health information.15 Patients' vulnerability is accentuated with the evidence that many patients appeared to believe that health information on the Internet is reliable16 and trustworthy.17 In a recent study, Kalichman et al18 found that 32% of respondents rated information on an unreliable page as accurate, and 29% rated that particular Web page as trustworthy and credible. This finding clearly indicates that patients become vulnerable to misinformation because they are not able to critically validate Web information from unauthoritative sites. Therefore, it is important that nurses not only develop skills to judge online health information but are also able to teach patients how to apply these skills19 because of the potential exposure to misinformation and unfounded claims when accessing health information on the Internet.17,18

The lack of a professional gateway to monitor the content of the Web and the lack of a proper editorial process 2,20 have placed the responsibility to validate information with Internet users at the point of use.21 Because no agreed standards exist to evaluate health information on the Web,9,22,23 a range of strategies are in place to help online health information consumers: databases,24 rating scale,25 guidelines,26 evaluative tools,8 and frameworks.27 In an attempt to further assist users in their judgment process, Breast Cancer Knowledge Online Portal28 provides a quality report of Web sites with the information on their database. However, many of these tools are of a generic nature.20 To overcome this issue, specific evaluation tools had been designed for multiple sclerosis,6 Alzheimer's disease,29 and breast cancer.22 Sellitto and Burgess27 designed a framework assigning weightings to qualitative evaluation features (authority and currency, accuracy, objectivity, and privacy). However, there is evidence to suggest that Web users only occasionally utilized these evaluative tools,3 and in addition, some tools had too many criteria to encourage their use.30

A well-utilized and trusted set of guidelines is the one designed by HON, an organization bestowing a trust mark to Web sites publishing heath information. Aimed at the general public, health professionals, and Web publishers, the HON Code, as it is commonly known, consists of eight principles to encourage the dissemination of quality health information on the Web.26 Web site owners voluntarily seek certification from the HON Code review committee, and sites abiding by the eight principles are allowed to display the HON Code logo. The foundation regularly monitors Web sites to ensure that certification is active. However, a study by Morel et al31 found that having the HON logo on a Web site was a not a predictor of quality.

Another initiative is The Information Standard scheme developed by the Department of Health in the United Kingdom to help members of the public identify trustworthy health and social care information. To gain certification to display the Information Standard logo for 3 years, information producers must meet the stringent quality criteria of The Information Standard. There is an annual surveillance evaluation by the accreditation body to ensure that information producers comply with the set quality standard.32

A tool developed for the print medium and applied within the Internet environment is the DISCERN instrument. To judge the reliability of a publication, it has eight questions relating to the relevance of information, information sources utilized, their currency, level of bias, and whether additional sources of support were offered. The instrument also has seven questions relating to treatment choices that focus on information on how the treatment works, risks involved, treatment choices, and impact on the overall quality of life. The final question asks the user to rate the quality of the publication based on the answers to the previous 14 questions. The strength of this instrument lies in the rigorous process involved in its original development. Although validated to evaluate printed information on patient education, the instrument has not been fully validated in the Internet medium. In a study, Morel et al 31 concluded that the DISCERN score is a not a predictor of content quality. Khazaal et al33 found a brief version of DISCERN to be a reliable tool for patients to assess Web information, but emerging evidence suggests that the instrument is more effective when utilized by healthcare professionals than by patients.34

In an attempt to exploit the digital and networked facilities of the Internet medium, Eysenbach and Thompson 35 developed a second-generation tool called FA4CT (or FACCCCT), which stands for Find Answers and Compare from different sources, Check Credibility and Check Trustworthiness. It provides a three- step approach for Web users to seek and verify medical information on the Web.22 Step 1 asks users to translate a medical query into search terms for a Google search to find three Web sites containing answers to the query. Only if there is no consensus in the answers provided by the three Web sites do users proceed to step 2, which advises users to validate the three Web sites, and step 3, which guides users toward a reputation score.

METHODS

If "quality is in the eye and ear of the beholder"36^(pp558) in relation to health information, then the voice of the users must be heard. Only a few studies have sought the views of Web users on how they evaluate the reliability of Web-based health information.37 Metzger3 advised that future research should focus on what users do to assess credibility of online information (Table 1), and Eysenbach22 had argued for a process-oriented approach to judge online information. Therefore, to understand how qualified nurses, as Web users, judge the quality of Web nursing information, this study has used a grounded theory approach.38 It was important that the sample of nurses selected could share their thoughts and experiences to help the researcher's understanding of the process of judging the quality of Web-based nursing information. Professional nurses, with experience of using the Web, were thought to have the knowledge essential to inform this study. During phase I of the study, 20 participants were recruited from a population of postqualifying nurses undertaking part-time undergraduate courses at a university in the United Kingdom. In phase II, the recruitment of 13 nurse practitioners from four critical-care units within an NHS Trust in the United Kingdom was undertaken before data saturation was achieved. Semistructured face-to-face interviews and focus groups were utilized to elicit information from the nurses in phase I of the study. Interview questions related to the regularity of participants' access to the Internet for general and clinical information, their views on the quality and value of Web information not originating from published journals, and the criteria they use to judge these sources. The researcher acted as moderator for the group discussions focusing on the initial findings of the study, and the participants explored their judgment in relation to clinical aspects, presentation and sources of information, evidence, and recognized sources. Interviews and group discussions took place in a purpose-built communication suite within a university at a time convenient to the participants. In phase II, to give context to the study, the interviewing of participants was undertaken after viewing Web pages. Evidence suggests that contextualizing information must be given due consideration in the process by which users judge quality on the Web.39 The participants were initially asked about their regularity of accessing the Internet. Then, the questions focused on how they judge information relating to relevance to practice, sources utilized, presentation, authors' credibility, and reliability. The participants were interviewed at a time convenient to them and took place in a seminar room, which has a computer with Internet access within a hospital setting. All interviews and group discussions were audiotaped with the participants' consent. Consistent with the grounded theory approach, data analysis was carried out concurrently with data collection using open, axial, and selective coding and constant comparative method.

Table 1 - Click to enlarge in new window

Table 1 Dual Processing Model

Ethical Considerations

Ethical approval was obtained from the University Ethical Committee and the local Healthcare Trust Ethical Committee. Informed consent was obtained from participants, and anonymity was ensured. Participants were reminded that they can withdraw from the study at any time without giving any reasons and without consequences.

RESULTS

All participants stated that they utilized the Internet to access clinical information at least twice weekly, but 80% (n = 23) reported accessing online information for clinical, educational, and other purposes on a daily basis.

An explanatory framework 40 emerged from the data showing how nurses judge nursing information on the Web. The framework illustrates an interplay between modes of cognition, forms of knowledge, and some specific evaluative tasks. The judgment process is dependent on the level of Web information cues, the nurses' critical skills, and the time they can afford. Because details of the framework have been published elsewhere,40 this article focuses on the six evaluative tasks (Table 2) relating to user-friendliness, outlook and authority of Web pages, nature of evidence, relationship to nursing practice, and cross-checking strategies. These evaluative tasks are now elaborated upon.

Table 2 - Click to enlarge in new window

Table 2 Evaluative Tasks and Subtasks

Assessing User-friendliness of Web Pages

User-friendliness of a Web page has been an important issue when nurses were engaging with Web information. Participants have identified navigation, user control, and functional features as part of this category. Whereas some Web designers have acknowledged the user's need for easy navigation, others have overlooked this need. This inconsistency may be due to the lack of regulation within the Internet environment. This is reflected in the following extract:

For some Web pages[horizontal ellipsis] you know[horizontal ellipsis] it's very easy to find your way around the site, and you can go and find what you're looking for[horizontal ellipsis] I've come across some, where I find it quite a struggle to know where everything is. It's quite frustrating[horizontal ellipsis]

Freeman and Spyridakis 5 found that information presented in a user-friendly format assisted users in their judgment process. This argument is supported by McKemmish et al,23 who stated that presenting online information in a user-friendly manner would empower users to proceed to the next stage in their information-seeking process by evaluating the quality of the information.

Assessing the Outlook of Web Pages

The physical impact of information presented on the Web was also discussed by the participants. Elements the participants have associated with this evaluative task are the physical impact of a Web page, the way it is structured, and its level of readability. One of them said, "[horizontal ellipsis]but for me as a consumer using it[horizontal ellipsis] I found the visual effects and the way it's laid out[horizontal ellipsis] extremely helpful[horizontal ellipsis] it's hard to explain[horizontal ellipsis]."

Readability is explained by this participant, "You can be frightened off by technical terms[horizontal ellipsis] if you start reading it, and it's easy to read[horizontal ellipsis] then you can put it into context better.."

This aspect of a Web page has also been included in Childs'41 guidelines as "appearance of a site." Metzger3 also identified the presentation of information as a primary factor in the process of assessing the quality of information by Web users.

Assessing Authority of Web Pages

Many facets of the Internet are identified in the data. The Internet is perceived as an important medium to access information related to the participants' work, particularly government papers and policies, which are available much earlier than are the traditional print medium. This category also relates to the low level of reliability of Web information from nonofficial sources. This issue seems to be associated with personal and commercial motives of publishing information on the Internet. Reliability is clearly reflected in these quotes:

I don't feel it's completely reliable. You can't always believe what they are telling you. Some of the information is from companies trying to sell products.

[horizontal ellipsis] You know on the Internet[horizontal ellipsis] and you're in a hurry and need information[horizontal ellipsis] you need to be very, very selective errmmm[horizontal ellipsis] because of the sort of the ease with which you can find rubbish.

Web sites participants classified as authoritative were those from government organizations, academic institutions, professional organizations, and global organizations. Examples of authoritative sources are included in this extract: "I've only visited authoritative sites like the Department of Health, [horizontal ellipsis] the RCN [Royal College of Nursing], the King's Fund[horizontal ellipsis]."

A study assessing online information on back pain also identified medical institutions and government Web sites as being trustworthy.9

Assessing Relationship to Nursing Practice

Participants appear to validate Web information on its relationship to practice, as illustrated by the following:

I think, when it actually relates to practice[horizontal ellipsis] I would value quite high because, at the end of the day, you're looking at developing your own practice, really [horizontal ellipsis] and to improve on your practice.

Assessing the practice relevance of Web information also includes a safety aspect, as stated by one participant, "Information can be unsafe to use[horizontal ellipsis] on the Internet."

Participants were also preoccupied with the cultural origin of Web information. The cultural issue has appeared in Childs' guidelines 41 under the heading "foreign sites," highlighting different cultural practices, different terminologies, different treatments, and different availability of treatments and medications. A participant's concern is reflected in the following extract: "You know[horizontal ellipsis] a lot of American data[horizontal ellipsis] it's really interesting but doesn't relate to nursing practices in England."

Another participant sees the other side of the coin with the following extract: "I may be wrong, but I do think we have a lot to learn from America, and I think there is some relevant sort of stuff out there[horizontal ellipsis]."

Appraising the Nature of Evidence

This task relates to research evidence, references, and the diversity of information published on the Web. In this quotation, evidence-based approach to care is clearly preferred to a ritualistic approach to care:

You have the two sides of the coin. You've got the evidence-based stuff, and you have the ritualistic-based stuff. They can go hand in hand, but I think with evidence-based stuff, [horizontal ellipsis] it is the way forward, and it is the best practice to go for rather than the ritualistic stuff.

The number of references was thought to be important by the participants in this study. However, this has to be measured against the quality of the references. The importance of quality has been highlighted in the following extract: "The more references they give, the wider it looks[horizontal ellipsis] the better. The quality of references is more valuable than the number [of references]."

The data also confirm that a wide perspective of views is published on the Internet, depending on the motivation of the author or publisher of information. The author's motivation in publishing is highlighted in the following quotation:

I'd look at who has compiled the Web site, whether it's a relative of someone who has died, whether it's a parent of a child with a particular illness, [horizontal ellipsis] whether there is an emotional aspect to the Web site.

Applying Cross-Checking Strategies

Under this category, participants identified cross-checking with online information, with printed materials, and with peers and colleagues. The following extracts highlight examples for online checking:

[horizontal ellipsis]some of them do have an e-mail address, and if I was particularly interested, I wouldn't hesitate in getting in touch with the compiler of the Web site. But having done all of that and looked at the Web site, I would compare it with other Web sites.

The breadth of links to other places [horizontal ellipsis]I think[horizontal ellipsis] I find very useful[horizontal ellipsis] so a site that isn't in itself authoritative[horizontal ellipsis] I can always get to check it out[horizontal ellipsis]

[horizontal ellipsis]the hyperlinks[horizontal ellipsis] I can follow these up to check where the information is coming from[horizontal ellipsis]

Interestingly, the FA4CT tool 35 includes cross-checking with online sources in the validation process. Freeman and Spyridakis5 also found that Web users compare online health information with their existing knowledge and seek additional online information to assist them in their judgment process. It is not surprising that cross-checking information with peers forms part of this category because, in relation to clinical decision making, nurses prefer colleagues as sources of information.42 Checking with peers is illustrated in the following quotation:

Sometimes I discuss stuff with my colleagues at work[horizontal ellipsis] if it's a new idea[horizontal ellipsis] it helps to find out whether you can use it in practice.

A participant has expressed her preferred cross-checking strategy; thus,

I've gathered shelves and shelves of papers[horizontal ellipsis] and if I do come across something, it's simply a case of turning from the Web to the printed medium behind me and checking it out.

The verification of Web information with other reliable sources has been advocated by Meola 43 and was also an outcome from Kim's44 study.

Additional Factors Underpinning the Judgment Process

To carry out these judgment tasks, the nurses were dependent on the level of cues on the Web, the time afforded for these tasks, and their critical skills. Two examples on Web cues are author's credentials and referencing sources, and these are illustrated in the following:

The qualification of the person writing it[horizontal ellipsis]

I would want to know who wrote it [horizontal ellipsis]and whether that person has experience in intensive care.

I've got a habit of looking at the references [horizontal ellipsis] first[horizontal ellipsis] to see if they are up-to-date[horizontal ellipsis]

The fact that it doesn't give any references makes me think that it is probably more hearsay....

Affording time to carry out these tasks is another factor identified by the nurses, as indicated by the following statement:

We have not got the time necessarily [horizontal ellipsis] to run through all the minor organizations that might be out there, and that might actually be useful [horizontal ellipsis] and from my point of view, what we need are useful inks.

In a recent study, time was a factor perceived by nurses as a barrier to using Web-based resources.45

The nurses were also using critical skills in their judgment process as illustrated by the following two examples:

[horizontal ellipsis]To an extent I suppose it's up to me to gauge where the information comes from[horizontal ellipsis] and what the value is.

[horizontal ellipsis]For example, foreign materials in the eyes can be removed by irrigation[horizontal ellipsis] but they are not substantiating why they would do that[horizontal ellipsis] using research. So it's again something that you may use to get some ideas but would need to look into it further to substantiate why you would use it[horizontal ellipsis].

Critical appraisal skills were also identified as being part of the judgment process in a study by Kim.44

DISCUSSION

The self-regulation approach to ensure that Web information is of a good quality relies on Web publishers to abide by codes for best practice and guidelines.46 Because the data from this study revealed that the level of Web information cues is important in the nurses' judgment process, the emerged evaluative tasks and subtasks can be invaluable to Web publishers of nursing information. Whereas some publishers take their responsibility seriously,47 it appears that others are not adhering to established codes and guidelines as the quality of health information published on the Web remains a concern.7,9 Important aspects of information relevant to nursing have been integrated in many tasks, and these relate to evaluative metadata. The list of evaluative tasks and their associated subtasks can inform publishers on the inclusion of relevant data that may help nurse users to judge Web information with more confidence. For example, knowing that the cultural origin of Web information is an important factor in the judgment process of nurse users, publishers could include the data pertaining to this aspect of nursing information. Author's expertise is another example that the participants have identified as one of the criteria for judging the author's credentials. Evidence suggests that metadata can play a crucial role in empowering users to validate the quality of Web information.48

Because Web users have an important role to play in the information-validating process,46 many guidelines and checklists have been developed for that purpose. However, using a checklist approach has been criticized as inappropriate because it focuses on information that is internal to a Web site.43 To establish the credibility of a Web page, a contextual approach that focuses on comparing information to other Web sites and/or other offline sources has been suggested.43 In order to move away from a checklist model and to encourage Web users to critically evaluate online information, an evaluative strategy is essential. The findings from this study are appropriate as they go beyond the limitation of a list of evaluative criteria. This study has confirmed the relevance of a process approach because nurses have identified critical skills, Web information cues, and time as factors essential to undertake these evaluative tasks. It is noteworthy that the tasks identified in this study closely relate to Metzger's3 process model, which consists of three phases: exposure (to online information), evaluation, and judgment. The evaluative tasks are also supported by Chandrasheker and Hockeman's10 evaluative process, which consists of predictive, evaluative, and confirmatory phases.

Patients are increasingly using the Internet to access health related with the risk of misinterpretation.15 To reduce this risk, there is growing evidence for nurses to assist patients in accessing reliable health information15,16 and recommending Web sites to patients.7 In fact, in a recent survey by HON,13 72% of participating healthcare professionals felt that it was part of their role to direct patients to trustworthy Web sites. The evaluative tasks that have emerged from the data provided by qualified nurses can assist the nurse to use her professional knowledge with much more confidence to undertake this role effectively. In addition, it has been suggested that healthcare professionals could educate their patients on the existence and use of evaluative tools when accessing health information on the Internet.4 The teaching of critical skills to patients by nurses has also been advocated.19 Recent evidence confirms that many nurses were already assisting patients with the process of evaluating online information.49 Therefore, while the nurse empowers the clients, the empowerment of the nurse is realized with the aid of the findings revealed from this investigation. Teaching patients how to identify credible information is not the only way professionals can address the judging of credibility on the Web. It had been argued that nurses should also be a in a position to generate information for the Web to benefit their patients.50 Recent evidence suggests that healthcare professionals should play an active role in generating health information of high quality to assist patients with accessing accurate online information.2 Improving the ability of Web users to either access credible Web sites or filter untrustworthy ones can minimize the risk of them coming across unreliable ones.

One limitation of this study lies in its self-report research method. This approach to data collection might have caused potential response bias because nurses, as professionals, knew they should critically analyze the information they obtain online. To minimize bias, in phase II, interviews were conducted immediately following the appraisal of three Web pages, which allowed the participants to focus on a recent specific episode rather than generalizing their opinions.44 The sampling strategy adopted excluded qualified nurses inexperienced in judging Web information. These nurses might have provided different perspectives on how nursing information on the Web should be evaluated. Also, caution must be exercised when interpreting the findings of this study as they are not generalizable to other universities and NHS Trust settings, although they could be of interest to others in similar situations. Further research is required to address this issue.

CONCLUSION

The complexity and multidimensional nature of the Web suggest that a single evaluative strategy is unlikely to find application across all existing domains or Web sites. The findings from this study give credence to a process approach, adding to the emerging number of second-generation evaluative tools to judge the quality of information on the Web. Fundamentally, the voice of nurses, as professionals, is being added to this ongoing debate on the quality of Web information. It is also important to acknowledge that the evaluation process will depend on the intent and purpose of the users when contextualizing Web information. Therefore, different users may well use different processes at different times to evaluate online information. The evaluative process emerged from this study will serve as one invaluable contribution to the repertoire of many global strategies being advocated to manage the quality of information on the Web.