‘— — _‘_-"' \_ ffJ’J'JJ J ‘ JJJ JJ‘ JJ Jui J J l | J J J J J M lJ J J J J l J J J J J JJ l J J J J l J l J JJ J JJ [J J l J JJ '1 l J J JJ J DEVELOPMENT OF A WEBSITE TO PROVIDE INFORMAIION m HEALTH-CARE CONSUMERS ON THE BASICS 0E EVALUATI’NGINFORMATION ON THE WORLD ' - WIDE WEB Schefiariy Preiect for the Degree‘o'fm SEN} ‘ MICHIGAN STATE'UNIVERSITY I - ‘ ; GAYLELJ-woLEE ' _ 1998 .' LIBRARY Michigan State University PLACE IN RETURN BOX to remove this checkout from your record. TO AVOID FINES return on or before date due. MAY BE RECALLED with earlier due date if requested. DAT . DATE DUE DATE DUE \ 2m _ a. JV 7 6/01 c:/ClRCiDateDue.p65-p. 15 WWW Information Evaluation 1 Running head: WORLD WIDE WEB EVALUATION WEBSITE DEVELOPMENT OF A WEBSITE TO PROVIDE INFORMATION TO HEALTH- CARE CONSUMERS ON THE BASICS OF EVALUATING INFORMATION ON THE WORLD WIDE WEB By Gayle L. Wolfe A SCHOLARLY PROJECT Submitted to Michigan State University in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE College of Nursing 1998 WWW Information Evaluation Website 2 ABSTRACT DEVELOPMENT OF A WEB SITE To PROVIDE INFORMATION TO HEALTH- CARE CONSUMERS ON THE BASICS OF EVALUATING INFORMATION ON THE ‘ WORLD WIDE WEB By Gayle L. Wolfe There is a vast amount of information on health-care available on the Internet and World Wide Web. This information is being accessed by rapidly increasing numbers of health-care consumers each year for purposes Of self-care. This would not be a problem if all the information were accurate and reliable. Sadly, this is Often not the case. There is online information that is inaccurate, misleading, slanted, or unreliable in other ways. Therefore, health-care consumers need a way to evaluate online information sources. A Web site can be used to provide information to consumers; and those who need to evaluate online information are already online. Resources that delineate criteria for the evaluation of information are available on the Web, but none are Specifically geared towards helping consumers make sense of Web-based health-care information. A Web site (at http://pilotmsu.edu/user/wolfegay) was developed, utilizing a self-care framework, to provide information to health-care consumers on the basics of evaluating Web-based health information. Links are provided to support the information on the Web site. WWW Information Evaluation Website 3 ACKNOWLEDGMENTS Many thanks are in order as I complete my Master's degree experience. I thank my scholarly project committee, Dr. George Allen (chair), Dr. Gail Casper, and Dr. Joan Predko for their invaluable support and assistance. There were those of us who thought it could not be done, but these rare individuals saw that it could and would be completed! My loving family deserves a major thank you. Their patience and their support of a mother who was always in the cold, dank basement clunking away on her keyboard were laudatory! Thanks for all the hugs, snacks, cups Of coffee, and refills of my water bottle! You kids kept me alive! SO, this project is dedicated to my children: Michael Edward Wolfe Leilani Eileen Wolfe Caitlin Colleen Wolfe Jennifer Lynne Wolfe You 're the greatest! WWW Information Evaluation Website 4 TABLE OF CONTENTS INTRODUCTION 5 THE INTERNET AND WORLD WIDE WEB 7 BACKGROUND OF THE PROBLEM 9 STATEMENT OF THE PROBLEM ll PURPOSE OF THE PROJECT 11 CONCEPTUAL FRAMEWORK: SELF-CARE 12 WHAT IS SELF-CARE? ................................................................................................................................ 12 CONCEPTUAL DEFTN l'TlON ......................................................................................................................... 14 WHAT IS THE APN'S ROLE IN SELF-CARE? ................................................................................................ 15 LITERATURE REVIEW 15 EVALUATION CRTTERIA ............................................................................................................................. 16 Authority ............................................................................................................................................. l6 Objectivity ........................................................................................................................................... 18 Coverage ............................................................................................................................................. 19 Relevance ............................................................................................................................................ 19 Accuracy ............................................................................................................................................. 20 Currency ............................................................................................................................................. 20 Usability .............................................................................................................................................. 21 METHODOLOGY 22 HARDWARE .............................................................................................................................................. 25 SOEDNARE ................................................................................................................................................ 25 CONNECUON ............................................................................................................................................ 25 HTML ...................................................................................................................................................... 25 POSSIBLE METHODS FOR EVALUATING THE WEB SITE 26 CONSUMER HEALTH INFORMATICS 27 JADVANTAOES ........................................................................................................................................... 27 INSADVANTAGES ...................................................................................................................................... 28 RELATIONSHIP OF CHI TO CURRENT PROJECT ........................................................................................... 29 IMPLICATIONS FOR ADVANCED PRACTICE NURSES AND PRIMARY CARE ....................... 29 SUMMARY 30 REFERENCES 31 APPENDIX 41 FOOTNOTES 42 WWW Information Evaluation Website 5 Introduction There is a new paradigm in health-care—that of the empowered consumer. Consumers are being educated in health matters; and, according to Grueninger (1995), that education has been instrumental in bringing about improvements in mortality, morbidity, life expectancy, and quality of life. The patient, or health active consumer, is becoming a partner of the practitioner, and the clinician is taking on more of a role of counselor and facilitator (Ferguson, 1995; Kahn, 1993; Grueninger, 1995). Health active consumers take responsibility for their health through self-directed involvement; the concerned individual is becoming more involved in the process of making health-care decisions (Wallace, 1996). This "new" paradigm is not all that new to nursing. Nurses have long recognized the right of health-care consumers to be informed, active participants in their care. Nurses support the empowerment of the health-care consumer and have a major responsibility for enhancing clients' capacity for self-care (Pender, 1996). Self-care can be defined as "the decisions and actions taken by someone who is facing a health problem in order to cope with it and improve his or her health" (Health Canada, 1998, online). Information gathering is the first Step in the self-care process. Consumers of health-care need information to inform their decision-making (Glode, 1996; Cooling, Kidd, & Sloggett, 1997; Ferguson, 1995; Wallace, 1996) and to choose and evaluate services and providers (Delaney, 1996). Well informed individuals, those with access to and an understanding of health information, play a more active part in their treatment and tend to have better health outcomes (Krishna, Balas, Spencer, Griffin, & Boren, 1997; Cooling, Kidd, & Sloggett, 1997; Deering, 1996; Kahn, 1993). Active involvement in WWW Information Evaluation Website 6 self-care has been acknowledged as an important factor in achieving national health goals (Pender, 1996). Unfortunately, most people do not have ready access to the information they need to be well-informed consumers of health—care (Council on Competitiveness], 1996). Individuals are increasingly accessing the Internet and World Wide Web (WWW, or the "Web") for health—care related information (Coiera, 1995, 1996; Ferguson, 1997; Health On the Net Foundation, 1997). Estimates of Internet use are difficult to make, but according to Lowes (1997), roughly 26 million people were using the Internet as Of October 1995, and use doubles each year. Additionally, there were at least 10,000 sites devoted to health-care, mostly on the Web. Internet access to health information is very important to the health—care paradigm shift—"With it, each of us is free to search the entire world for health information. Knowledge becomes the key that unlocks our power of choice, and lets us take control of our health and our lives" (Robertson, 1997, p. 17). Health information is abundantly available on the Internet, in all manner of guises, from multitudinous sources, and of variable quality and accuracy (Council on Competitiveness, 1996; Goldwein & Benjamin, 1995; Glowniak, 1995; Coiera, 1996; United States General Accounting Office (US. GAO), 1996). This information can be very useful when it is accurate, understandable, and appropriate for the consumer’s needs. It can be dangerous when inaccurate, biased or unsubstantiated (Council on Competitiveness, 1996). In order to utilize online information safely and effectively, consumers need to be able to evaluate information sources. An advanced practice nursing intervention to enhance consumer appreciation Of information sources would be helpful. It would help WWW Information Evaluation Website 7 consumers answer questions such as: What makes an information source credible? What would lead an informed individual to be cautious regarding presented information? What are signs that would indicate a source is a good one? Assessment of information sources requires the use of critical thinking skills (Ferrell, 1997a; Hinchliffe, 1997; Jadad, 1997; Kirk, 1997a; Mather, 1996; Silberg, Lundberg, & Musacchio, 1997). The focus of these skills for the purpose of Web information evaluation can be demonstrated utilizing a Web site. The Web enables self- directed and life-long learning (Cunningham, 1996). Therefore, a Web site would be an appropriate means to reach individuals who are searching the Web for health information. The Internet and World Wide Web The Internet is a worldwide network of computer networks—that is, many computers linked together so that information can be shared among them (Pallen, 1995a). Transfer Communication Protocol/Internet Protocol (TCP/IP) allows computers to communicate with each other through the telephone system or by other means such as high-speed data links, satellites, and infrared waves (Studach, 1997). The Internet can be accessed through direct or indirect connections. A direct connection is usually available through universities, the government, or large corporations; users can access a gateway to the Internet either through a remote terminal or telephone connection. Commercial service providers, freenet, and indirect service providers (e.g., CompuServe, America Online, and Prodigy) also have gateways to the Internet; Internet access is provided through use of a personal computer with a modem (Edwards, 1997; Marra, Lynd, McKerrow, & Carleton, 1996; Ferguson, 1996; Shellenbarger & Thomas, 1996; Pallen, WWW Information Evaluation Website 8 1995b). Using the Internet, one can access information on computers in remote locations (Edwards, 1997; Glowniak, 1995). The World Wide Web is a subset of the Internet. The resources within it are in hypertext format, offering "a simple, consistent, and intuitive interface to the vast resources of the Internet" (Edwards, 1997, p.42). The W can be considered a software layer which provides users a simple way to access information (Coiera, 1995). Text and images can be easily viewed by use of this software layer, consisting of browsing programs such as Microsoft Internet Explorer and Netscape Navigator. Using these browsers and additional player programs, audio and video media are available online as well. The Internet is also: the fastest growing communications medium in history. In 1994, it doubled in size, as it has done yearly since 1988, and now encompasses nearly five million "host computers" on over 28,000 networks. It is now estimated to reach between 15 million and 30 million people, encompassing at least 30,000 databases, with one million new users logging on each month. These new users, along with daily advances in technology, are generating a spectacular increase in information traffic of 15 percent per month (Council on Competitiveness, 1996, online). There is abundant information on the Web on virtually any topic, including health and medical topics. According to the Council on Competitiveness (1996), among current methods Of disseminating health information to consumers (e. g., health newsletters, magazines, television, Videos, information kiosks, CD-ROMS, and online information) the Web is the most promising opportunity for delivering health information, and WWW Information Evaluation Website 9 consumers are interested in using this new technology to obtain health information. They are more interested in obtaining health information using the Web than in activities such as online shopping (79% of consumers polled reported being "somewhat to very interested" in Obtaining health information online vs. 32 % "somewhat to very interested" in "shopping online"). Background of the Problem There is an abundance of health information available to "surfers" of the Web. There are obviously sources with merit, but many sources are not to be trusted. This is due to the nature of the Web. Virtually anyone with a computer, modem and Internet connection can now publish on the Web, bypassing time-tested editorial and peer-review processes that help to ensure information quality in print media (Kassirer & Angel], 1996; Roberts & Spooner, 1997; Smith, 1997b). The Web has the potential to become the world’s largest vanity press; anyone can serve as author, editor, and publisher and can fill any of these roles anonymously if desired (Silberg, Lundberg, & Musacchio, 1997). SO, much Of the information on the Web can be fraudulent, unsubstantiated, and, having undergone no peer review process, possibly seriously flawed or inconsistent (U.S. GAO, 1996; Impicciatore, Pandolfini, Casella, & Bonati, 1997; Marra, Carleton, et al., 1996; Wyatt, 1997). The information may be misleading, out-of—date, dangerous or wrong (Pallen, 1996). There may be no obvious cues which let the reader know the information is invalid (Reynolds, 1995). It may be presented in a biased fashion to steer consumers toward a certain kind of therapy (U.S. GAO, 1996), or may be information presented for marketing purposes disguised as editorial text (Bahan, 1996). Additionally, there is so much information that it becomes difficult to determine what is meaningful and what is WWW Information Evaluation Website 10 not (Council on Competitiveness, 1996; Brown, 1997). Information overload may ensue — being on the Web is akin to "filling a water glass with a fire hose" (U.S. GAO, 1996, p. 18) — the consumer may be overwhelmed with too much technical information. "The Sheer volume of health information could be viewed as a treasure trove for the health active consumer, but more Often it is so overwhelming that apathy is the natural reaction" (Wallace, 1996, online). Problems with information reliability become increasingly important when one looks to the Web for information concerning health matters. Many of those who surf the Web for health information have "little way of knowing for sure whether what they find is science or snake oil. Not only is this confusing, it can also be dangerous" (Sikorski & Peters, 1997, p. 1431). Additionally, much Of the medical information on the Web is intended for health-care professionals, not for direct consumption by the public. Lay consumers of such information may have problems with interpreting even valid, accurate information and placing it in context so that decisions can be made (Kassirer & Angel], 1996). As summarized by Jadad and Gagliardi: With the Internet and the rapid development of other information technology, we are facing a sea change in health care. The Internet is creating new Opportunities to improve decisions and communication in health care, but it can also generate many unprecedented problems. In the midst of an unparalleled information revolution, good communication, scholarly discussions, and rigorous evaluations are more crucial than ever. If we fail to meet the current challenges, we may miss an extraordinary opportunity to make health care more efficient and equitable, moving instead into a health care environment ruled by confusion, battles of WWW Information Evaluation Website 11 Opinion, anxiety, and unnecessary conflicts (1998, online). Statement of the Problem The problem is that health-care consumers have access to a wide range of information, which they may not be equipped to interpret. There is a potential for too much information, which makes the information difficult to understand or assimilate. Additionally, information may be biased, inappropriate, or just plain wrong. What is the solution? There are several possibilities. Practitioners could create sites that serve patients well—well organized, current, user-friendly pages which utilize the best resources (Bahan, 1996); that is, sites known to be trustworthy (Jadad, 1997). Consumers could be provided easy access to guidelines developed by recognized authorities (Coiera, 1995). Alternatively, health-care providers could act as interpreters of the information patients access on the Web (Cooling, Kidd, & Sloggett, 1997; Glode, 1996). Or, consumers could develop the ability to evaluate the quality of online information (Bahan, 1996; Brown, 1997; Silberg, Lundberg, & Musacchio, 1997; Jadad, 1997; Mather, 1996; Hinchliffe, 1997; Kirk, 1997a). Purpose of the Project A site was developed on the World Wide Web to provide information to adult health-care consumers on the basics of critical evaluation of Web information sources. A Web site provides a forum for the exchange of such information (Shellenbarger & Thomas, 1996). The purpose of the Site is to enhance consumers' self-care abilities, increasing their awareness of the need for careful appraisal of Web-based information resources, by fostering an understanding that not all information sources are created WWW Information Evaluation Website 12 equal. The site is presented as the effort of an advanced practice nurse, and thereby increases the visibility of advanced practice nurses on the Web. A Web site is an ideal route for providing such information, as individuals browsing the Web for health information are already online. Links to the sources used to develop the site are presented. The evaluation criteria are geared to the analysis Of health- related information. Conceptual Framework: Self-Care Efforts directed at activating the public for self-care Should be led by nurses in collaboration with other health professionals and communities. Individuals play a critical role in the determination of their own health status since self-care represents the dominant mode of health-care in our society (Pender, 1996). One of the strategies suggested by the World Health Organization (WHO) to achieve health for all by the year 2000 included empowering people by providing information and decision-making Opportunities (WHO, 1988). The use of computer technologies such as the World Wide Web can enhance health-care consumer access to information, thereby facilitating such empowerment. Health-care consumer empowerment is consistent with APN philosophy. Support of such empowerment would include support of efforts to facilitate appropriate consumer use of the Web. What is self-care? Self-care is a universal requirement for sustaining and enhancing life and health. Self-care is an activity that individuals personally initiate and perform in order to maintain health (Sadgrove, 1997). Self-care directed toward health protection and health promotion can be defined as "activities initiated or performed by an individual, family or WWW Information Evaluation Website 13 community to achieve, maintain, or promote maximal health" (Pender, 1996, pp. 97-98). It includes actions directed toward minimizing threats to personal health, self-nurturance, self-improvement, and personal growth. Active involvement in self-care is widely acknowledged as an important strategy for achieving national health goals (ibid.). The World Health Organization (WHO, 1983) defines self-care as referring to "activities individuals, families, and communities undertake with the intention of enhancing health, preventing disease, limiting illness, and restoring health. These activities are derived from knowledge and skills from the pool Of both professional and lay experience. They are undertaken by lay people on their own behalf, either separately or in participative collaboration with professionals." Rather than being viewed as a failure Of the individual to use medical services, self-care is seen as a continuum of self- initiated behaviors that may enhance the health and independent functioning of individuals (National Institute on Aging, 1997). Orem (1985) defines self-care as "the practice of activities that individuals initiate and perform on their own behalf in maintaining life, health, and well-being" (p. 84). The goal Of self-care is the attainment of self-care requisites: universal self-care requisites, developmental self-care requisites, and health-deviation self-care requisites. Nurses help the client attain self-care requisites in three ways: wholly compensatory interventions (the nurse does all the care for the client, who is unable to engage in self-care), partly compensatory interventions (both nurse and client engage in meeting self-care needs), and supportive educative interventions (client system needs assistance in decision making, behavior control, and acquisition of knowledge and skills) (Orem, 1985). WWW Information Evaluation Website 14 Self-care is action associated with the use of oneself as a resource. It means taking care of one’s own health and acting autonomously, making decisions for oneself and by oneself (though it may involve consulting others, both laypersons and professionals) (Health Canada, 1998). A definition of self-care compiled by Dean (1986) summarizes most aspects of the foregoing discussion: Self-care involves the range Of activities individuals undertake to enhance health, prevent disease, evaluate symptoms and restore health. These activities are undertaken by lay peOple on their own behalf, either separately or in participation with professionals. Self-care includes decisions to do nothing, self-determined actions to promote health or treat illness, and decisions to seek advice in lay, professional and alternative care networks, as well as evaluation of and decisions regarding action based on that advice (p. 62). Conceptual Definition For the purposes of this project, self-care is defined as activities undertaken by individuals to enhance or restore health and to prevent or treat illness. These activities can be undertaken separately or through consultation with or in collaboration with health- care professionals. Self-care activities include health-care information retrieval and health-care decision-making. The intervention described in this project falls under Orem's supportive-educative system—the Web site is designed to enhance health-care consumer acquisition of knowledge (health-care knowledge on the Web) and skills (evaluation of Web-based information) to promote self-care. WWW Information Evaluation Website 15 What is the APN'S role in self-care? It is the APN'S role to enhance the client's self-care agency. As a client advocate, the APN works to promote a transfer of responsibility to the client by creating a climate of mutuality in Which the nurse assists the client in exercising his/her rights and in improving self-care abilities (this is empowerment of the client). As a consultant, the APN uses the problem-solving process and provides information regarding health-care to health-care consumers (Given & Peek, 1998b). The APN delivering primary care places an emphasis on wellness or on promoting client and family ability to cope with illness. This includes supporting and enhancing client strengths and assets. The APN provides support to assist and guide clients in managing their own care to cope with crisis. The APN is also concerned with educational services—how information is transmitted, and how the client intemalizes information to promote optimal health maintenance and enhanced capacity for self-care (Given & Peek, 1998a) Literature Review This literature review focuses on material that was used to define the content Of the Web site — critical evaluation of information on the Web. Multiple sources describe information evaluation techniques. Most have been developed by library science faculty at various universities in order to aid students in evaluating online information for research purposes. They generally elaborate on traditional criteria for the evaluation of printed information. Online sources were emphasized in this review, as hyperlinks to cited sources will be provided in the project’s product, the Web site. WWW Information Evaluation Website 16 The criteria for evaluation of online information discovered in the review process included authority, objectivity, coverage, relevance, accuracy, currency, and usability. Descriptions of the criteria follow. Evaluation Criteria Authority Authorship was noted as perhaps the most important criterion used in the evaluation of information (Kirk, 1997a). It is one Of the components of authority. Authority can be evaluated by looking at the author’s qualifications: education, experience, occupation, position/title, expertise in the topical area, and organizational or institutional affiliation (Alexander & Tate, 1996, 1997; Laverty, 1997; Fenton, 1997; Grassian, 1997b; Harris, 1997; Jacobson & Cohen, 1996; Hinchliffe, 1997; Kirk, 1997a, 1997b; Brandt, 1997; Ormondroyd, Engle, & Cosgrave, 1996; Patterson, 1997; Scholz, 1996; Smith, 1997a, 1997b; Tillman, 1997; Wilkinson, Bennett, & Oliver, 1997). Additionally, it is important to be able to contact the author, by a means other than just an email address (as, generally speaking, email addresses are more "anonymous" and temporary than "land-based" addresses and telephone numbers), for comments, questions, or requests for further information (Alexander & Tate, 1997; Harris, 1997; Kirk, 1997a, Wilkinson, Bennett, & Oliver, 1997). Authority includes the identification of the sponsoring organization or publishing body to determine its credibility as well (Alexander & Tate, 1997; Laverty, 1997; Fenton, 1997; Ferrell, 1997b; Grassian, 1997a, 1997b; Harris, 1997; Jacobson & Cohen, 1996; Kirk, 1997a, 1997b; Patterson, 1997; Scholz, 1996; Smith, 1997a, 1997b; Wilkinson, Bennett, & Oliver, 1997). Credentials of author WWW Information Evaluation Website 17 or sponsor may be validated by traditional print or electronic means (Alexander & Tate, 1997; Fenton, 1997). Authority can be supported by outside sources. Independent evaluation of the Web resource may be helpful in evaluation of its content (Harris, 1997; Ormondroyd, Engle & Cosgrave, 1996; Patterson, 1997; Jadad & Gagliardi, 1998). Metainformation is information about information, and can be in summary or evaluative form. Summary metainformation includes abstracts, content summaries, or tables of contents. Evaluative metainformation includes recommendations, reviews, ratings and commentaries (Harris, 1997). Metainformation tools include review tools such as Excite, Lycos Top 5%, Magellan Internet Guide, and Webcrawler Best of the Net, general guides such as Argus Clearinghouse and Gale’s Cyberhound Guide, directories such as Yahoo and the WWW Virtual Library, and library resources which are fee-based (Tillman, 1997). Each evaluative tool has its own criteria for judging Web sites; the tools must be evaluated to see if they have criteria that are appropriate for the intended use. More recently, tools have been developed to evaluate Sites that contain health information. Jadad and Gagliardi (1998) found 47 different rating instruments used to evaluate Web sites with health information on the Internet. They found the evaluation tools to be incomplete, and questioned the value of the tools, as it was unclear "whether they should exist in the first place, whether they measure what they claim to measure, or whether they lead to more good than harm" (ibid., online). Another method of evaluating the site is through referring links. If a quality source refers to or links to a document, there is increased confidence in the quality of the second document (Patterson, 1997). For example, Sites created by credible agencies (e.g., WWW Information Evaluation Website 18 the Federal government, universities, medical centers, or hospitals) may contain links to health-related documents. Such linked sites are lent credibility by their association with the referring site. However, material in the linked Sites may change. Since referring sites Often do not monitor the Sites they link to, caution is in order even with a credible primary site. Objectivity Objectivity is another criterion for the evaluation of information. Looking at the organization responsible for the site may be helpful in the establishment of Objectivity. If the organization has a commercial, ideological, political or other vested interest in the topic, bias may be present (Harris, 1997; Fenton, 1997; Ferrell, 1997b; Grassian, 1997a; Kirk, 1997a; Brandt, 1997; Wilkinson, Bennett, & Oliver, 1997). One of the major hindrances to Objectivity is conflict of interest (Harris, 1997). The purpose of a page — public service, education, sales, persuasion or presentation of an opinion on a controversial issue — may be an indicator Of its Objectivity (or lack thereof) (Fenton, 1997; Patterson, 1997, Smith, 1997a, 1997b). Emotional or persuasive language may herald a lack of Objectivity (Harris, 1997; Ormondroyd, Engle, & Cosgrave, 1996). Objectivity means that all sides of issues are portrayed and entails a lack of propaganda, disinformation, obvious errors, or misleading omissions (Fenton, 1997; Wilkinson, Bennett, & Oliver, 1997). Another marker of objectivity is a lack of advertising or, minimally, a clear differentiation of advertising content from the text of the document (Alexander & Tate, 1997). WWW Information Evaluation Website 19 Coverage Coverage refers to the completeness Of the information presented. It includes both the depth and the breadth Of the information presented on the subject (Alexander & Tate, 1996), as well as the comprehensiveness of the site or information (Laverty, 1997; Grassian, 1997b; Harris, 1997; Jacobson & Cohen, 1996). Breadth refers to coverage of all aspects of the subject, depth to the level of detail presented (Smith, 1997a, 1997b). Coverage can be validated by comparison with print resources on the same topic to determine if the information presented is equivalent in scope (Ferrell, 1997b; Grassian, 1997b; Patterson, 1997). The coverage of the topic is greatly influenced by the audience for whom the information was intended (Hinchliffe, 1997). Relevance Relevance is defined as "relation to the matter at hand: practical and esp. [3] social applicability: pertinence" (Webster’s New Collegiate Dictionary, 1977, p. 976). This criterion is particularly important in the evaluation of health-related information. Relevance was not addressed directly in many of the documents evaluated. Relevance can be related to the purpose one has in looking for the information (Smith, 1997b). To what use will the information be put? IS the information at an appropriate use level (Fenton, 1997)? Laverty (1997) mentions relevance only tangentially, under the heading "Compare With Other Resources," i.e., perhaps books or journal articles would provide more suitable or reliable information than Web sources. Wilkinson, Bennett, and Oliver (1997) address relevance and scope of content, using evaluation questions such as: Is the content related to the user’s needs? IS the information sufficiently current and the coverage Of the topic sufficiently broad to meet the user’s needs? The intrinsic value (and WWW Information Evaluation Website 20 hence the potential relevance to M use) of the information on the Web site and the relative value of the Web page in comparison to the range of information available are addressed by Grassian (1997b). Accuracy Accuracy is the assurance that the information presented is actually correct — i.e., the details are verifiable (Harris, 1997; Alexander & Tate, 1997; Fenton, 1997; Harris, 1997; Jacobson & Cohen, 1996). Although it is difficult to judge the accuracy of information without a thorough understanding of the topic presented, it can be assessed partially by identification of the methods used to obtain the information, citation of sources used, presence or absence of a peer review mechanism, and whether or not statistics and other methodological issues are presented to support the information presented (Wilkinson, Bennett, & Oliver, 1997; Kirk, 1997a; Fenton, 1997; Grassian, 1997a). It can be assessed by comparison with other works, e.g., print resources (Patterson, 1997; Ferrell, 1997b; Smith, 1997a). Obvious errors or misleading omissions are telling indicators of inaccuracy. Other indicators of inaccuracy would be such features as spelling and grammatical or typographical errors: if care was not taken to detect these problems, content errors may have been committed as well (Wilkinson, Bennett, & Oliver, 1997). Currency Currency refers to the temporal aspect of the document. When was the document created, placed on the Web, copyrighted, or last revised or updated? What edition of the work is presented? When was the information in the document gathered—how up-to- date is the study (Alexander & Tate, 1997; Fenton, 1997; Ferrell, 1997b; Grassian, 1997a; WWW Information Evaluation Website 21 Harris, 1997; Jacobson & Cohen, 1996; Kirk, 1997a, 1997b; Ormondroyd, Engle, & Cosgrave, 1996; Patterson, 1997; Scholz, 1996; Smith, 1997a; Tillman, 1997; Wilkinson, Bennett, & Oliver, 1997)? Currency is particularly important in fields that change rapidly, e.g., science and medicine (Harris, 1997; Fenton, 1997). Information regarding currency is needed to ensure that one is getting up-to-date information. Usability Usability issues relate to the actual structure of the Web document. These criteria include ease of use of the site ("user-friendliness"); for example, navigating around the site Should not be a problem (clear site map, menus, table of contents, or hypermedia index available) (Fenton, 1997; Wilkinson, Bennett, & Oliver, 1997; Jacobson & Cohen, 1996). The site should be logically arranged, with the use of good graphic design (Grassian, 1997b; Jacobson & Cohen, 1996; Wilkinson, Bennett, & Oliver, 1997; Smith, 1997a). Multimedia should be incorporated appropriately, adding to rather than distracting from the content (Fenton, 1997; Grassian, 1997b; Laverty, 1997; Wilkinson, Bennett, & Oliver, 1997; Smith, 1997a). Information should be concise to eliminate as much as possible the need for lengthy scrolling through the document (Wilkinson, Bennett, & Oliver, 1997). Hyperlinks should be intact and functional (Jacobson & Cohen, 1996; Smith, 1997a; Wilkinson, Bennett, & Oliver, 1997). Consideration must be taken also of the varying levels of technology that may be used to access the site. For example, the better sites are accessible to a wide variety of users; whenever possible, text- only, non-frames views should be available (Alexander & Tate, 1996; Grassian, 1997a, 1997b; Smith, 1997a). Finally, enhancements should be added (within reason) to aid WWW Information Evaluation Website 22 those with access problems (e.g., text versions of image and sound files for individuals with visual or hearing deficits) (Grassian, 1997a; Wilkinson, Bennett, & Oliver, 1997). Methodology First, a literature review was conducted to document that health-care consumers had a need for assistance with the evaluation of Web-based health information. Once the need was established, a search of the health-care literature and the Web provided the criteria for information evaluation. These criteria were developed into an online document to provide health-care consumers with information on Web information evaluation. A question arose regarding the reading level of Intemet/Web users. The language of the information on the Web sites needs to be geared toward the "average" adult Web user. The problem lies in determining the characteristics of the average Web user. The average Web user is a "moving target," whose demographic characteristics can change rapidly. Studies also disagree on the demographics of Internet users based on questions surrounding methodological issues as well as when data were gathered (Raney, 1998). Schwartz (1996) reported the findings of the American Internet User Survey (released in January 1996). The survey found that 9.5 million Americans, or 3.6 percent of the population, use the Internet, including 1.1 million children under the age of 18; that the average Internet user was 36 years old; and that use of the Internet closely followed college education. The average household income of an Internet user was $62,000. Overall, 31 percent of Internet users were under age 30, 27 percent were in their thirties, 26 percent were in their forties, and 13 percent were age 50 or older. A poll conducted for Business Week in early 1997 found that 27% of Internet users had a high school WWW Information Evaluation Website 23 education or less, 32% had some college, 23% were college graduates, and 18% had post- graduate education (Education levels of US. surfers, 1998). According to the 7th Graphics, Visualization & Usability Center (an academic research center affiliated with Georgia Tech's College of Computing) survey, conducted in April and May of 1997, 54% of respondents had completed a college or advanced degree, 10.5% high school or less, and 29.23% some college (Graphics, Visualization, & Usability Center, 1997). However, the profile is changing. The Baruch College-Harris telephone survey of 3,008 adults was conducted in winter 1997/1998 (Birdsell, Muzzio, Krane, & Cottreau, 1998). According to the survey results, the percentage of Web users with a high school education or less doubled between September 1995 and winter 1997-98—from 16% to 32%—while the proportion of college graduates had gone from 56% to 38%. Those with a high school education or less are still significantly under-represented on the Web, since they make up 52% of the adult American population compared to the 22% with at least a college degree. Nineteen percent of all Americans with a high school education or less currently log onto the Web compared to 53% of those with at least a college degree. Higher education thus translates into an increased likelihood to use the Web (Novak & Hoffman, 1998). It seems safe to assume a literacy level greater than that of the average newspaper reader (8lh to 9’h grade level) (personal communication, K. Greenwood, CCAS Technology manager, MSU, 3/31/98). In light of the survey findings reported above, an assumption of at least a 10‘h grade reading level is supported. The online document was therefore written to keep this reading level as a maximum. Overall, the reading level of the intervention document was 8.6 (Flesch-Kincaid reading level), with a score of 54.3 on WWW Information Evaluation Website 24 Flesch reading case, based on the grammar-checking program available in Microsoft Word 972. Individual paragraphs may have been more or less complex than the overall score, which represents an average. Individuals developing educational materials to be placed on the Web should be cognizant of the guidelines for the evaluation of information, so they can maintain credible sites for consumer health information. The following working guidelines for the development of online materials were postulated by the American Medical Informatics Association (AMIA): 0 Content should include a clear and concise statement of objectives and purpose. 0 Content validity—content should be clearly documented as to source, i.e., peer-reviewed publication, personal opinion, etc. 0 Content ethics—ensure that the content is not biased towards any one point of view. o Who Should participate in production of content—all parties from content expert—computer/graphic design, librarian and most NB client or patient in order to compare needs perceived by content experts vs. patient/client. They are often disparate. 0 Develop explicit criteria for evaluation of content and presentation during all phases of the project. 0 Develop strategy for delivery 0 Develop a concise plan for maintenance and updates, i.e. personal responsibilities, funding for work, etc. (AMIA, 1996, online). WWW Information Evaluation Website 25 These strategies provided guidance during the development of this project. Hardware The Web Site was developed on a personal computer, an IBM Aptiva PC with 166 MHz Pentium/MMX processor, 32 megabytes of RAM, 56K flex modem, 2.5 GB hard drive, 12x CD-ROM drive, and 3.5" floppy drive. Software The Web site was developed and published using FrontPage 98, a Web publishing program produced by Microsoft. This software requires a processor chip which is a 486 or later. Files for the Web site were managed by FrontPage 98 and using Ipswitch WS_FTP (file transfer protocol program). The browser program used was Internet Explorer 4.01. Additional HTML add-ins (counter, etc.), which are server specific, were added as HTML programming. Word processing was done using Microsoft Word 97. The operating system used was Windows 95. Connection The Internet was accessed through Michigan State University’s Michnet gateway connection, with a connection speed of 28.8K bits per second (BPS). The Web site is located in MSU’S directory. The address (URL—uniform resource locator) for the home page is http://pilot.msu.edu/user/wolfegav. M Hypertext markup language (HTML) is the coding language used to create hypertext documents for use on the World Wide Web (Marra, Lynd, et al, 1996). HTML is the lingua franca of the Web. It is the language in which Web pages are written and Web information is distributed (Powell & Wickre, 1995). A working knowledge of WWW Information Evaluation Website 26 HTML was very helpful in troubleshooting problems that developed with the Web pages. Possible Methods for Evaluating the Web Site The purpose of this Web site is to enhance health-care consumer awareness of the variable quality of Web-based information resources. How can we determine if this goal was met? In addition, does the site meet its own criteria as to quality? The users of the Web site themselves may answer these questions. Requests for feedback and an e-mail application to increase the ease of providing feedback have been added to each Web page. Thus, the Site may be evaluated by the readers as they provide comments to the author. It is also possible to create an online survey for users of the Web site, gathering information on factors such as overall impression of the site, usability, esthetics, readability, knowledge gained, and quality of the site as defined by information provided on the site. The URL for the Web site may be submitted to Web reviewers—cg, Argus Clearinghouse, Magellan, Excite, Webcrawler—for independent review and evaluation (Tillman, 1997). See the Tillman site (http://www.tiac.net/users/hope/findqual.html) for further information on the rating systems. In addition, since the content of this site relates to the evaluation of information, the site may be reviewed by information science professionals for content quality and accuracy. Evaluation of outcomes is problematic with most health education — subjects can be tested for increased knowledge, but it is often not possible to determine if the increased knowledge leads to an improved health outcome (Consoli, et al, 1995). In summary: For those investing resources in a web site, a key question is its likely impact on WWW Information Evaluation Website 27 clinical processes and patient outcomes and its cost effectiveness compared with other methods for delivering the same information. Tentative answers to this question can be obtained by studying the impact of the site on the knowledge of sample users in laboratory settings, but its real impact on clinical practice can be studied only in the field (Wyatt, 1997). Consumer Health Informatics This Web site is an example of a consumer health informatics intervention. Consumer health informatics (CHI), a subset of medical informatics, has been defined as "the use of modern computers and telecommunications to support consumers in obtaining information, analyzing their unique health-care needs, and helping them make decisions about their own health" (US. GAO, 1996, p. 1); and the "study of consumer interfaces in health-care systems—a subspecialty of medical informatics" (Ferguson, 1997, p. 251). Studach ( 1997) identified three components of CHI: (1) computers, technology, communications, or information science; (2) human factors; and (3) some outcome related to health. CHI also can be organized into three general categories: (1) applications which provide health information to the user (one-way communication); (2) those which tailor specific information to the user’s unique situation (customized communication); and (3) those which allow the user to communicate and interact with health-care providers or other users (two-way communication) (U.S. GAO, 1996). Advantages Informatics offers a new avenue through which health-care consumer needs for information may be met, thereby empowering their self-care. The consumer can readily review material at his or her own pace and at the needed level of detail. CHI systems can WWW Information Evaluation Website 28 respond to the consumer's information needs quickly and efficiently. Some studies have Shown that CHI may reduce the need for some unnecessary medical services, thereby lowering health—care costs. An example of this would be consumers obtaining information on the risks and benefits of an elective procedure and consequently deciding not to undergo the procedure. In addition, approximately 80 percent of all health-care involves problems treated at home. Effective management of these problems by the health-care consumer can prevent the illness or injury from progressing to the point of needing professional intervention (U.S. GAO, 1996). There are other advantages, as well (Wallace, 1996): o Anonymity—increased ability to remain unknown while accessing or reporting personal or sensitive information, which allows a more accurate representation of health data. 0 Outreach—improved access by individuals in rural and underserved areas. 0 Convenience—the ability to use the system any time, day or night. 0 Scope—increased ability to reach large numbers of people. 0 Support—ease of establishing online relationships with others. Disadvantages As discussed above, information quality is a very significant issue. There is the potential for information to be incomplete, inaccurate, or outdated. Some experts have said that networks might carry information that has been changed or taken out of context by the online service or by consumers themselves. Others have identified the potential for biased information that may have been developed by a person or organization with a vested interest (U.S. GAO, 1996). WWW Information Evaluation Website 29 Relationship of CHI to current project In View of the advantages of CHI, it makes sense to promote the use of these information gathering and consumer education opportunities. However, the health information must be available at an appropriate and useful comprehension level (Wallace, 1996). Additionally, since consumers often have trouble obtaining unbiased, quality information, it would be helpful to consumers to be able to evaluate CHI resources— specifically, as in this project, information sources on the World Wide Web. Implications for Advanced Practice Nurses and Primary Care The advanced practice nurse (APN) in primary care has, among many roles, those of patient educator and patient advocate. Interventions that increase patient knowledge increase both participation in the health-care process and client empowerment, thereby potentially improving outcomes. The Web is a resource that many patients are using to increase their health knowledge. Evaluation of information sources may be problematic for these individuals. The APN can refer clients to Web sites which detail evaluation of Web information sources (such as the Site developed for this project) to assist patients with such problems. This will save time for the APN (who will not need to go into detail regarding information evaluation during a client visit), and will increase patient benefits. Once satisfied that clients have information on the differentiation of appropriate from inappropriate sources, the APN can feel comfortable in directing the client to the Web for health information. The APN must, however, reinforce the need to discuss information retrieved from the Web with the primary care provider prior to putting it into practice. Clients often bring materials retrieved in an online search to their primary care practitioner for interpretation and evaluation. The primary care practitioner is the ideal WWW Information Evaluation Website 30 conduit between the client and Web information resources and can assist the client both with clinical interpretation of the information and with placing the information in context (Cooling, Kidd, & Sloggett, 1997). Keeping current with online resources will help the APN in primary care maintain awareness of current trends in outcomes—based health-care. The APN can evaluate the online information sources and then refer patients and families to reliable sources (Roberts & Spooner, 1997; Krishna, Balas, Spencer, Griffin, & Boren, 1997; Silberg, Lundberg, & Musacchio, 1997). Information gathering is the first step of the decision-making process. A cautionary note: information found on the Web should be a part of—not a substitute for— the relationship between client and provider that is central to the health-care decision- making process (Silberg, Lundberg, & Musacchio, 1997). It should enhance, not replace, the provider-client interaction (Cooling, Kidd, & Sloggett, 1997). Summary A Web site detailing the need for evaluation and the criteria for evaluation of Web-based health-care information resources is beneficial to health-care clients and practitioners alike. It can enhance clients' self—care abilities, promoting empowerment of these clients. Such a Web site was developed by the author, in the hopes of enhancing consumer appreciation of the evaluation of information resources on the World Wide Web. WWW Information Evaluation Website 31 References Alexander, J ., & Tate, M. (1997). Checklist for an informational Web page [Online]. Available: http://www.science.widener.edu/~withers/inform.htm [1998, February 21]. Alexander, J ., & Tate, M. (1996). The Web as a research tool: Evaluation techniques [Online]. Available: http://www.science.widener.edu/~withers/evalout.htm [1998, February 4]. American Medical Informatics Association. (1996). What are the ground rules for developing intemet-based CHI tools—summary from meeting of November 1996 [Online]. Available: http://www.rchusherbca/amia/summl.html [1998, February 4]. Bahan, I. (1996). Patient information on the Internet. Tennessee Medicine, 89(10), 365-366. Birdsell, D., Muzzio, D., Krane, D., & Cottreau, A. (1998). Web users are looking more like America. The Public Perspective 9(3), 33-35. Brandt, D. S. (1996). Evaluating information on the Internet [Online]. Available: http://thorplusJib.purdue.edu/~techman/evaluate.htm [1998, February 3]. Brandt, D. S. (1997). Why we need to evaluate what we find on the Internet [Online]. Available: http://thorplus.libpurdueeduhtechman/eval.html [1998, February 20]. Brown, S. (1997). What kinds of impacts may the growth of Intemet-based CHI have on the organization and practice of health care in the next decade? [Online]. Available: http://www.rcq.usherb.ca/amia/questSrep.html. [1998, February 4]. WWW Information Evaluation Website 32 Coiera, E. (1995). Medical informatics. British Medical Journal, 310(6991), 1381- 1387. Available: http://www.bmi.com/archive/699led1.htm [1998, February 3]. Coiera, E. (1996). The Intemet’s challenge to health care provision. Bgli_sh Medical Journal, 312(7022), 3-4. Consoli, S. M., Said, M. B., Jean, J ., Menard, J., Plouin, P., & Chatellier, G. (1995). Benefits of a computer-assisted education program for hypertensive patients compared with standard education tools. Patient Education and Counseling, 26(1-3), 343- 347. Cooling, N ., Kidd, M., & Sloggett, S. (1997). Use of computers by general practitioners for patient education. Australian Family Physician, 26(1), 31-36. Council on Competitiveness. (1996). Highway to health: Transforming US. health care in the Information Age [Online]. Available: http://nii.nistgov/pubs/coc hghwv to hlth/title pagehtml [1998, February 3]. Cunningham, C. A. (1997). The World Wide Web and the coming educational revolution [Online]. Available: http://www.neiu.edu/~ccunning[daric.htm [1998, February 21]. Dean, K. (1986). Self-care behaviour: Implications for aging. In K. Dean, T. Hickey & B.E. Holstein (Eds), Self-care and health in old age: Health behaviour implications for policy and practice (pp. 58-93). London: Croom Helm. Deering, M. J. (1996). Consumer health informatics status report [Online]. Available: mpzllodphposophs.dhhs.gov/confmce/partnr96/testimon.htm [1998, February 4]. WWW Information Evaluation Website 33 Delaney, C. (1996). Use of nursing informatics in advanced nursing practice roles for building healthier populations. Advanced Practice Nursing Quarterly, 1(4), 48-53. Education levels of US. surfers [Online]. (1998). Available: http://www.cyberatlas.com/market/demographics/education levels.html [1998, May 29]. Edwards, M. J. A. (1997). The Internet for nurses and allied health professionals, (2nd ed.). New York: Springer. Fenton, S. (1997). Information quality: is the truth out there? [Online]. Available: http://ils.unc.edu/~fentS/310/ [1998, February 3]. Ferguson, T. (1995). Consumer health informatics. Health care Forum Journal 3(1), 28-33. Available: http://www.thfnet.org/th950105.HTM [1998, February 6]. Ferguson, T. (1996). Health online. Reading, MA: Addison—Wesley. Ferguson, T. (1997). Health online and the empowered medical consumer. J_oir_g Commission Journal on Quality Improvement, 23, 251-257. Ferrell, K. (1997a). Truth, lies, and the Internet. CINET. Features. Digital Life. 10/9/97. http://www.cnet.com/Content/Features/Dlife/Truth/ [1998, February 3]. Ferrell, K. (1997b). Truth-seeking on the Net [Online]. Available: http://www.cnet.com/Content/Features/Dlife/Truth/ss07.html [1998, February 20]. Given, B., & Peek, P. (1998a). Appendix A—The advanced practice role in primary care [Online]. Available: http://www.msu.edu/unit/nurse/classes/SOl/appa.html [1998, June 2]. Given, B., & Peek, P. (1998b). Appendix B—Role Characteristics of APN [Online]. Available: http://www.msu.edu/unit/nurse/classeS/SOl/classl-roles.html [1998, June 2]. WWW Information Evaluation Website 34 Glode, L. M. (1996). Challenges and opportunities of the Internet for medical oncology. Journal of Clinical Oncology, 14, 2181-2186. Glowniak, J. V. (1995). Medical resources on the Internet. Annals of Internal Medicine, 123, 123-131. Goldwein, J. W., & Benjamin, 1. (1995). Internet-based medical information: Time to take charge. Annals of Internal Medicine, 123(2), 152-153. Graphics, Visualization, & Usability Center. (1997). GVU's 7th WWW user survey [Online]. Available: http://www.cc.gatech.edu/gvu/user surveys/survev-l997-O4/ [1998, May 29]. Grassian, E. (1997a). Thinking critically about discipline-based World Wide Web resources [Online]. Available: http://www.library.ucla.edu/libraries/college/instruct/discp.htm [1998, February 3]. Grassian, E. (1997b). Thinking critically about World Wide Web resources [Online]. Available: mad/WWW.library.ucla.edu/librarieS/college/instruct/critical.htm [1998, February 3]. Grueninger, U. J. (1995). Arterial hypertension: Lessons from patient education. Patient Education and Counseling, 26(1-3), 37-55. Harris, R. (1997). Evaluating Internet research sources [Online]. Available: http://www.sccu.edu/faculty/R Harris/evalu8it.htm [1998, February 20]. Health Canada. (1998). Supporting self-care: The contribution of nurses and physicians [Online]. Available: http://www.hcc.gc.ca/hppb/healthcare/pubs/selfcare/index.html [1998, June 4]. WWW Information Evaluation Website 35 Health On the Net Foundation. (1997). Health on the Net survey results [Online]. Available: http://www.hon.ch/cgi-bin/quest/quest intemet , http://www.hon.ch/cgi- bin/quest/quest netSep [1998, March 6]. Hinchliffe, L. J. (1997). Evaluation of information [Online]. Available: http://alexia.lis.uiuc.edu/~ianicke/Eval.html [1998, February 4]. Impicciatore, P., Pandolfini, C., Casella, N., & Bonati, M. (1997). Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home [Online]. Available: http://www.bmi.com/archive/7098ipl .htm [1998, February 20]. Jacobson, T., & Cohen, L. (1996). Evaluating Internet resources [Online]. Available: httpz/lwwwalbany.edu/library/internet/evaluate.html [1998, February 3]. Jadad, A. R. (1997). How should consumers use health care information on the Internet? [Online]. Available: http://www.rcq.usherb.ca/amiafitestZreILhtml. [1998, February 3]. Jadad, A. R. & Gagliardi, A. (1998). Rating health information on the Internet: Navigating to knowledge or to Babel? [Online]. Available: http://www.ama-assn.org/sci- pubs/ioumals/archive/jama/vol 279/no 8/irv71042.htm [1998, March 4]. Kahn, G. (1993). Computer-based patient education: A progress report. M_._I_)_, Computing, 10(2), 93-99. Kassirer, J. P., & Angell, M. (1996). The Internet and the journal [Online]. Available: http://www.neim.org/scripts/search/page.pl?location=public&volume=332&page=1709 [1998, February 20]. WWW Information Evaluation Website 36 Kirk, E. E. (1997a). Evaluating information found on the Internet [Online]. Available: http://milton.mse.ihu.eduz800llresearch/education/net.html [1998, February 3]. Kirk, E. E. (1997b). Practical steps in evaluating intemet resources [Online]. Available: httpzllmilton.mse.ihu.edu2800l/researCh/education/practical.html [1998, February 24]. Krishna, S., Balas, E. A., Spencer, D. C., Griffin, J. 2., Boren, S. A. (1997). Clinical trials of interactive computerized patient education: Implications for family practice. Journal of Family Practice, 45(1), 25-33. Laverty, C. (1997). Evaluating World Wide Web information [Online]. Available: http://stauffer.queensu.ca/inforef/tutorials/qcat/evalinthtm [1998, March 6]. Lowes, R. L. (1997, January 27). Here come patients who’ve "studied" medicine online. Medical Economics, 175-184. Marra, C. A., Lynd, L. D., McKerrow, R., & Carleton, B. C. (1996). Drug and poison information on the Internet, part 1: An introduction. Pharmacotherapy, 16(4), 537- 546. Marra, C. A., Carleton, B. C., Lynd, L. D., Marra, F., McDougal, A. R., Chow, D., & McKerrow, R. (1996). Drug and poison information on the Internet, part 1: An introduction. Pharmacotherapy, 16(5), 806-818. Mather, P. (1996). Critical literacy: the WWW's great potential [Online]. Available: http://ei.cs.vt.edu/%7Ewwwbtb/book/chap6/critical.html [1998, February 20]. WWW Information Evaluation Website 37 National Institute on Aging. (1997). Self-care behaviors and aging. NIH Guide 26(9), PA Number: PA-97-048 [Online]. Available: http://www.nih.gov/grants/guide/pa- files/PA-97-048.html [1998, June2]. Novak, T. P., & Hoffman, D. L. (1998). Bridging the digital divide: The impact of race on computer access and Internet use [Online]. Available: http://www2000.ogsm.vanderbilt.edu/paperS/race/science.html [1998, May 13]. Orem, D. E. (1985). Nursing: Concepts of practice (3rd ed.). New York: McGraw- Hill. Ormondroyd, J ., Engle, M., & Cosgrave, T. (1996). How to critically analyze information sources [Online]. Available: http://www.library.comell.edu/okuref/research/skill26.htm [1998, February 3]. Pallen, M. J. (1995a). A Guide to the Internet for Medical Practitioners: 1. Introducing the Internet [Online]. Available: http://www.bmi.com/archive/70l7ed2.htm [1998, March 5]. Pallen, M. J. (1995b). A Guide to the Internet for Medical Practitioners: 3. The World Wide Web [Online]. Available: hflfl/www.bmlcom/archive/70l9ed.htm [1998, March 5]. Pallen, M. J. (1996). Medicine and the Internet: Dreams, nightmares and reality. British Journal of Hospital Medicine, 56(10), 506-509. Patterson, S. (1997). Evaluating and citing Internet resources [Online]. Available: http://www.udmercy.edu/htmls/Academics/libraw/webpage [1998, February 3]. Pender, N. J. (1996). Health promotion in nursing practice (3rd ed.). Stamford, CT: Appleton & Lange. WWW Information Evaluation Website 38 Powell, B., & Wickre, K. (1995). Atlas to the World Wide Web [Online]. Available: httpzl/www.rhythm.Com/~bpowell/Atlas/Ch3.htm#Ch3 [1998, February 21]. Raney, R. F. (1998, May 10). Studies reach contradictory conclusions about the Internet population. The New York Times [Online]. Available: httpzllsearch.nytimcs.com/search/daily/bin/fastweb?getdoc+cyber-lib+cyber- lib+2 l240+0+WAAA+Studies%7EReach%7EContradictory%7EConclusions [1998, May 13]. Reynolds, T. M. (1995). Popular medical information on the Internet. Lancet fl(8969), 250. Roberts, J. R., & Spooner, S. A. (1997). Pediatric Internet resources: Creation and growth of the PEDINFO Index. Archives of Pediatric and Adolescent Medicine, 151, 592—597. Robertson, D. (1997). Self-managed care: Internet is key to paradigm shift. Mb Care on the Internet, 1(2), 11-17. Sadgrove, S. (1997). Models of the patient care process: A clinical perspective [Online]. Available: http://www.hsd.uVic.ca/HIS/programs/s97cours/h270/ssadgrov/as1.htm [1998, June 4]. Scholz, A. (1996). Evaluating World Wide Web information [Online]. Available: http://thorplus.lib.purdue.edu/research/classeS/gs 1 75/3 gs l75/evaluation.html [1998, February 4]. Schwartz, K. (1996, January 13). Growing net access chips away at other media, new study finds. The New York Times [Online]. Available: http://www.nytimes.com/96/0l/l3/cyber/01 l3netuse.html [1998, May 13]. WWW Information Evaluation Website 39 Shellenbarger, T., & Thomas, S. (1996). Creating a nursing home page on the World Wide Web. Computers in Nursing, 14(4), 239-245. Sikorski, R., & Peters, R. (1997). Oncology ASAP: Where to find reliable cancer information on the Internet. Journal of the American Medical Association, 277, 1431- 1432. Silberg, W. M., Lundberg, G. D., & Musacchio, R. A. (1997). Assessing, controlling, and assuring the quality of medical information on the Internet. Journal of the American Medical Association, 277, 1244-1245. Smith, A. G. (1997a). Criteria for evaluation of Internet information resources [Online]. Available: http://www.vuw.ac.nz/~agsmith/eva1n/index.htm [1998, February 4]. Smith, A. G. (1997b). Testing the surf: Criteria for evaluating Internet information resources. The Public-Access Computer Systems Review [Online], 8(3). Available: http://info.lib.uh.edu/pr/v8/n3/smit8n3.htm]. [1998, February 4]. Studach, J. (1997). Health promotion informatics: A literature review [Online]. Available: http://www.healthvamerican.edu/iohnsmethodslitreviewfinal.html [1998, February 20]. Tillman, H. (1997). Evaluating quality on the Net [Online]. Available: http://www.tiac.net/users/hope/findqual.html [1998, February 3]. United States General Accounting Office. (1996). Consumer health informatics: Emerging issues (GAO/AIMD-96-86). Washington, DC: Author. Wallace, P. (1996). On the edge of great change [Online]. Available: httpzl/www.largnet.uwo.ca/tvdhc/change edge/whole.html [1998, February 20]. WWW Information Evaluation Website 40 Webster’s New Collegiate Dictionary. (1977). Springfield, MA: G. & C. Mem'am Company. Who's on the Net in the US. [Online]? (1998). Available: http://www.cyberatlas.com/market/demographics/index.html [1998, May 29]. Wilkinson, G. L., Bennett, L. T., & Oliver, K. M. (1997). Consolidated listing of evaluation criteria and quality indicators [Online]. Available: http://itechl.coe.uga.edu/Faculty/GWilkinson/criteria.html [1998, February 20]. World Health Organization. ( 1988). From Alma-Ata to the year 2000: Reflections at midpoint. Geneva, Switzerland: WHO. Wyatt, J. C. (1997). Commentary: Measuring quality and impact of the world wide web. Available: httpzl/www.bmjcom/archive/7098ip2.htm [1998, February 20]. WWW Information Evaluation Website 41 Appendix Attached find copies of the Web Site documents as they appear on the World Wide Web at httpz/lpilot.msu.edu/user/wolfegay. Resources for Evaluating Internet Information Sources Page 1 of 2 Introduction Users of the Internet and World Wide Web come across a great deal of inform Is all the information to be trusted? I think not. Almost anyone with a computer, modem, and a link to the Web can publish a page. Individuals have their own reasons for publishing Web pages - they may simply want to share information they've found useful, or they may wish to pres an opinion or point of view. The Web is now also a highly commercial medium - may have something to sell. As an advanced practice nurse (fl), 1 am concerned that health care consum may be getting inaccurate or harmful health information on the Web. If "bad" information or advice is followed, the individual's well-being may suffer. Obviou there are good Web sites providing information that can be trusted. But there a also sites which publish information that is not trustworthy. How do we tell the 9 sites from the not-so-good? You have to have an eagle eye to Spot _. the differences between good and bad information! - - I will receive my Master of Science in Nursing degree from Michigan State University. During my studies toward that degree, I developed an interest in the of the Internet and World Wide Web, and found that the Web is being accesse more and more by people looking for answers to health and medical questions. However, I was unable to find a resource to help health care consumers evalua information on the Web. So I did some searching of my own, and developed or for the evaluation of information on the Web. I'd like to share what I've found w you. Please let me know if this information is helpful to you. I would like to point out that when you are looking for health information, it is prudent to discuss what you've discovered with your primary physician or nurse practitioner. He or she can help you evaluate the information in greater depth, hflnn'hailnt ma: MtEli::a-r/wnlfiaoav/evnloflt’70intrn htm 0/70/02 Resources for Evaluating Internet Information Sources Page 2 of 2 can also assess the situation from a professional standpoint. Many primary ca providers are open to discussion of alternative therapies, and are willing to expl all options with you. If you are interested in trying therapies that are not in the "mainstream" of medical practice, run it by your provider. They may have information on issues such as side effects, interactions with medications, etc., o which you may not have been aware. Evaluation of Web-based information sources If you are interested in reading my scholarly project, click here. You are visitor number Questions or comments? Click here! To ae Return to top of home page This page was last updated on 08/25/98. Copyright 0 1998 Gayle L. Wolfe How Do We Evaluate Web Resources? Page 1 of 10 How Do We Evaluate Web Resources: There are a number of criteria which can help determine if information is "good" information. These are the basics: J”? Authority Objectivity . Accuracy Coverage Relevance Usabilig .i- Sources For the purpose of determining whether health information on the WWW can be trusted, some of these criteria are more important than others. Some are easier to get information about than others as well. Authority Probably the most important single criterion is authority. If you don't know who is writing the information, you can't really trust it. Who wrote the information? 5" Is the author a physician, nurse practitioner, or other health professional? E What are their credentials, affiliations, and professional experience? E What are their qualifications for writing on the topic? I Are they acknowledged experts in the field you're reading about? I Someone can be a respected expert in, for example, pediatrics. That doesn't qualify them as an expert in women's health. Health and medical knowledge is so specialized now, it's impossible to be an all-around expert. Q Can you be relatively certain the author is who slhe claims to be? There should be a way to contact the author for questions or comments. t E-mail address E "Snail-mail" address or telephone number Where is the Web page located? hfin'llnilnt men nthIIIIcor/tunlf’pamrlhnm‘MOnfnoAODAvnhmm htm 0/70/02 How Do We Evaluate Web Resources? Page 2 of 10 5 Look at the URL (uniform resource locator - address). I Is the page part of the Website of an organization whose name you can trust? 5 university (.edu) I government (.gov) I institutional (.org) t commercial (.com) it, Does the address show that it's a personal home page? I A personal home page is one published by an individual who or may not be affiliated with a larger institution. a Personal home pages can be put up by anyone - they need to be carefully scrutinized before the information can be accepted as trustworthy. K It's not always easy to tell if you're reading a personal home page or not. Some are very professional looking. E The URL address of the page may have a variety of endings (e.g., .com, .edu, etc.), but a tilde (~) is frequently somewhere in the URL. If you feel you can trust the sponsoring organization, you can probably trust the Web documents within their site. However, regardless of the source, you need to evaluate any information critically before you put it into practice. Discuss information retrieved from any source with your health care provider - he or she will be glad to help you with interpretation of what you've found. Your nurse practitioner or physician has the experience and know-how to steer you in the right direction. Remember - your health, and your family's health, are at stake here! Objectivity Objectivity, or perhaps more important, the lack of objectivity, is fairly easy to detect if you know what you're looking for. Objectivity means all sides of issues are portrayed in a fair light. There is no propaganda or misinformation. The information is free from obvious errors or misleading omissions. However, this may not be easy to detect, if you don't know the subject matter well already. To see if information is presented objectively, look for the following: {4, Purpose of the page R Is it to sell a product of some kind? I Usually, if there is an area for ordering a product on the page, the page probably contains bias! m Is it to persuade you of the correctness of a certain opinion on a controversial issue? i Are all sides of the issue presented fairly? How Do We Evaluate Web Resources? Page 3 of 10 a Is it to present current information, as a public service? Again, look at whose site it is. 5 What kind of organization is responsible for the information? I If the organization has a commercial, ideological, political or other vested interest in the topic, bias may be present. R One of the major hindrances to objectivity is conflict of interest. What type of language is used? What tone does the page have? R Emotional or persuasive language may show a lack of objectivity. "5 ‘9 What about advertising? Is there any on the Web page? E If it is present, it should be clearly separated from informational text, so you know when one ends and the other begins. (By the way, I've written a book on this topic. . . click hei to order.) Accuracy Accuracy is important in judging information sources. Accuracy means the information presented is correct and exact. The accuracy of a source is more difficult to evaluate than authority and objectivity - if you don't already have a good understanding of the topic, it's hard to tell if the information presented is accurate. However, there are ways to get an idea if information is accurate or not. Look for the following: explanation of the methods used to obtain the information 3"“ listing of reference sources used evidence that content was reviewed by other authorities for accuracy _ information on how studies were conducted and analyzed _ " lack of obvious errors or omissions ‘3' lack of spelling, grammatical, and typographical errors 1 if care was not taken to detect these problems, content errors may have been missed also Accuracy can also be assessed by comparing the information with other sources. Does it go along with, or contradict, information you've seen in other sources (information provided by your health care provider, newspaper and magazine articles, etc.)? it Is the information "too good to be true?" Then it probably isn't truel Coverage Coverage means the completeness of the information presented. It is also difficult to determine without a thorough understanding of the topic. The coverage of the topic is greatly influenced by the audience for whom the information was written. For example, information intended for use by health How Do We Evaluate Web Resources? Page 4 of 10 care professionals would probably have greater coverage than information intended for use by health care consumers. Coverage includes: '=‘ The depth and breadth of the information (comprehensiveness) I Breadth - coverage of all aspects of the subject u Depth - level of detail presented Ways to assess coverage: 9:“ Look for obvious gaps or omissions in the coverage of the topic. Does the information presented leave you with unanswered questions? 5? Compare the information presented with print resources on the same topic. Is the information presented equivalent in breadth and depth? Relevance Relevance is defined as "relation to the matter at hand: practical and especially social applicability: pertinence." This criterion is particularly important in the evaluation of health-related information. Is the information suited to your needs? Is it pertinent? It can be related to: ”e“ The purpose of the Web page ,, 9'” The purpose you have in looking for the information i" The utility or usability of the information E Why are you looking for the information? 3 Is the content related to your needs? at; Is the information current and the coverage broad enough to meet your needs? Again, run any information you've found by your health care provider to see if it's applicable to your situation. It's difficult to be objective about the issues when confronted with illness in yourself or a loved one. Your physician or NP can help you sort things out. Time Aspects The time aspects of a document are particularly important in fields which change rapidly, for example, science and medicine. It's important to get up- to-date health information. Even a few months can be crucial in a field in which drugs and treatments are evolving so rapidly. Time aspects of the document are shown below: How Do We Evaluate Web Resources? Page 5 of 10 7,33 When was the document: I Created? 8 Placed on the Web? & Copyrighted? a Last revised or updated? I Look in the footer (the bottom of the document) for dates. This is the most common place for them. 3“: What edition of the work is presented? 3 When was the information in the document gathered? Usability How easy to use is the Web site you're evaluating? Is it "user-friendly?" fl, Navigating around the site should be easy. & clear site map or table of contents e menus 3E The site should be logically arranged, with the use of good graphic design. 3 Multimedia should be used appropriately. It should add to, rather than distract from, the content. elk Information should be concise, to reduce lengthy scrolling through the document. Hyperlinks should be intact and operable. '33- Consideration must be taken of the varying levels of technology which may be used to access the site. R The site should be accessible to most users; text-only, non-frames views should be available. E When possible, enhancements should be added to aid those with access problems (e.g., text versions of image and sound files for individuals with visual or hearing problems). Sources Here are some sites to explore which support the information provided above. Many were used to develop this Web page. They are listed in APA format. Author and year of publication are listed first, followed by title of the source, and URL of the source. The date I first accessed the site is in brackets. Check out: Alexander, J., & Tate, M. (1997). Checklist for an informational Web page [Online]. Available: http:/lwww.science.widener.edu/~witherslinform.htm [1998, February 21]. at; Checklist of questions to ask to evaluate the quality of informational Web How Do We Evaluate Web Resources? Page 6 of 10 pages. Alexander, J., & Tate, M. (1996). Evaluating Web pages: Links to examples of various concepts [Online]. Available: httpzllwww.science.widener.edu/~withers/examples.htm [1998, April 28]. a; Links to sites/pages which illustrate the types of Web pages and evaluation concepts. Alexander, J., & Tate, M. (1996). The Web as a research tool: Evaluation techniques [Online]. Available: httpzllwww.sciencewidener.edul~witherslevalouthtm [1998, February 4]. Outline of need for evaluation and evaluation techniques. Auer, N. (1998). Bibliography on evaluating Internet resources [Online]. Available: httpzl/refserver.lib.vt.edullibinstlcritTHINK.HTM. [1998, February 3]. 3* Comprehensive bibliography, including print resources. Beck, 8. (1997). The good, the bad, and the ugly: Or why it's a good idea to evaluate Web sources [Online]. Available: httpzl/lib.nmsu.edu/staff/susabeck/eval.html [1998, February 3]. Evaluation criteria, with links to examples to evaluate. Brandt, D. S. (1996). Evaluating information on the Internet [Online]. Available: httpzllthorplus.lib._purdue.edu/~tech_manlevaluate.htm [1998, February 3]. .3 Good discussion of Internet information evaluation. Brandt, D. S. (1997). Why we need to evaluate what we find on the Internet [Online]. Available: httpzllthorplus.lib.purdue.edu/~techmanleval.html [1998, February 20]. % Checklist of questions based on the above. Fenton, S. (1997). Information quality: is the truth out there? [Online]. Available: httpzllils.unc.edul~fents/310/ [1998, February 3]. Information quality checklist in part of the document - also information access issues, etc. Ferrell, K (1997). Truth, lies, and the Internet. C|NET. Features. Digital Life. How Do We Evaluate Web Resources? Page 7 of 10 1019/97. httpzllwww.cnet.com/Content/Features/DlifelTruthl [1998, February 3]. 52%. C-Net article - entertaining and informative. Ferrell, K (1997b). Truth-seeking on the Net [Online]. Available: http:l/www.cnet.com/Content/Features/Dlife/'1' ruth/ssO7.html [1998, February 20]. Grassian, E., & Zwemer, D. (1997). Hoax? Scholarly research? Personal opinion? You decide! [Online]. Available: httpzllwww.library.ucla.edu/Iibraries/college/instruct/hoaxlevlinfo.htm [1998, May 29]. 3 An exercise in evaluation, with questions to consider about sample sites. Grassian, E. (1997).Thinking critically about World Wide Web resources [Online]. Available: httpzllwww.library.ucla.edullibraries/college/instruct/critical.htm [1998, February 3]. 3 Checklist of questions for WW resource evaluation. Grassian, E. (1997).Thinking critically about discipline-based World Wide Web resources [Online]. Available: http2/Iwww.librarygucla.edu/libgrieslcollege/instructldiscp.htm [1998, February 3]. Additional points to consider regarding Web sites for subject disciplines. Harris, R. (1997). Evaluating lntemet research sources [Online]. Available: http2/lwww.sccu.edulfaculty/RiHarris/evalu8it.htm [1998, February 20]. 3 Comprehensive overview of rationale and technique for information evaluation. Hinchliffe, L. J. (1997). Evaluation of information [Online]. Available: httpzllalexia.lis.uiuc.edu/janicke/Evalhtml [1998, February 4]. 3 Brief overview of evaluation of information as relates to Internet resources. Holy Names College. (1996). Evaluating World Wide Web resources [Online]. Available: httpzllwww.hnc.edu/academiclife/libragl/evalweb.html [1998, May 24]. How Do We Evaluate Web Resources? Page 8 of 10 Jacobson, T., & Cohen, L. (1997). Evaluating Internet resources [Online]. Available: http:/Iwww.alba_ny.edu/library/internet/evaluate.html [1998, February 3]. ii Brief checklist of criteria to look for in evaluation of lntemet resources. Kirk, E. E. (1997). Evaluating information found on the Internet [Online]. Available: httpzl/milton.mse.jhu.edu:8001Iresearch/education/net.html [1998, February 3]. 3 Overview of information evaluation, with links to other helpful documents. Kirk, E. E. (1997). Practical steps in evaluating Internet resources [Online]. Available: httpzllmilton.msejhu.edu:8001lresearch/education/practicgl.html [1998, February 24]. Steps to take to find information regarding authorship, publishing body, and currency of the Internet document using your browser. Laverty, C. (1998). Evaluating World Wide Web information [Online]. Available: httpzllstauffer.queensuglinforef/tutorials/qcat/evalint.htm [1998, March 6]. B& Brief overview of Web information evaluation, with URL interpretation. Ormondroyd, J., Engle, M., & Cosgrave, T. (1996). How to critically analyze information sources [Online]. Available: mpzllwww.Iibragcornell.edu/okuj'ef/research/skill26.htm [1998, February 3]. 3 Overview of traditional criteria for information evaluation, including initial appraisal and content analysis. Patterson, S. (1997). Evaluating and citing Internet resources [Online]. Available: http:/Iwww.udmercy.edu/htmls/Academics/library/webpagg [1998, February 3]. 35;, Good overview of the topic. Patterson, S., & Schroeder, R. (1997). Evaluating and citing Internet resources: An annotated Webography [Online]. Available: httpzl/www.ud_mercy.edu/htmlslAcademics/librafl/webog [1998, February 4]. eiWebographylbibliography including search engines, URLs, citation of sources, evaluation of Webpages. How Do We Evaluate Web Resources? Page 9 of 10 Patterson, S., Wendt, A., & Schroeder, R. (1997). Evaluating information on the Internet [Online]. Available: httpzl/www.udmercy.edu/htmls/Academics/libra_ryleva|uati [1998, February 3]. Organizing page for the U-D Mercy pages on Internet information evaluation. Richmond, B. (1996). Ten C's for evaluating Internet resources [Online]. Available: http://www.uwec.edulAdmin/Libraryl10cs.htmI [1998, February 24]. £3 Questions to ask re: qualities of Internet resources. Scholz, A. (1996). Evaluating World Wide Web information [Online]. Available: httpzllthorplus.Iib.purdue.edu/research/classeslgs175l3gs175levaluation.html [1998, February 4]. 5;, Overview of Web document elements - where to find information in the Web document. Smith, A. (1997). Criteria for evaluation of Internet information resources [Online]. Available: http:/lwww.vuw.ac.nz/~agsmith/evalnlindex.htm [1998, February 4]. "Toolbox” of criteria for evaluation of Internet information resources. Smith, A. (1997). Evaluation of information sources [Online]. Available: http:/lwww.vuw.ac.nzl~agsmithlevalnlevalnhtm [1998, February 4]. Smith, A. G. (1997). Testing the surf: Criteria for evaluating Internet information resources. The Public-Access Computer Systems Review [Online], 8(3). Available: http:l/info.lib.uh.edu/prlv8/n3/smit8n3.html. [1998, February 4]. A Comprehensive document detailing information evaluation, Web review sites; complete with references and Webography. Stepno, B., & Henshaw, B. (1995). Quality of information...and disinformation online [Online]. Available: httpzllblakeoit.unc.edu/~rbstepnoldisinfo.html [1998, February 4]. Moved to: httpzllwww.unc.edul~rbstepnoldisinfo.htm [1998, May 4]. 3 Information and exercises in the evaluation of information quality. Terrass, R. (1997). Evaluating Internet resources [Online]. Available: How Do We Evaluate Web Resources? Page 10 of 10 httpzllwebwn.net/~usrlricter/weblvalid.html [1998, February 4]. 3 A Webography designed for radiology science professionals. Tillman, H. (1997). Evaluating quality on the Net [Online]. Available: http:/Iwww.tiac.net/userslhope/findqual.html [1998, February 3]. Information on Internet review tools, as well as information evaluation. Wilkinson, G. (1997). Evaluating the quality of Internet information sources [Online]. Available: http://itech1.coe.uga.edu/Faculty/GWiIkinson/webeval.html [1998, February 4]. Describes the project at University of Georgia - to develop criteria and procedures to help users evaluate Internet information quality. Wilkinson, G. L., Bennett, L. T., & Oliver, K. M. (1997). Consolidated listing of evaluation criteria and quality indicators [Online]. Available: httpzllitech1.coe.uga.edulFaculty/GWiIkinsonlcriteria.html [1998, February 21]. flVery exhaustive listing of criteria (based on the University of Georgia project described above). Wittman, S. M. (1998). Evaluating Web sites [Online]. Available: http:llacs.oa_kton.edul~wittmanlfindleval.htm [1998, April 28]. Wu, H. (1997). Evaluating & citing information found on the Net [Online]. Available: http://www.jsr.cc.va.usllrclevaele.htm [1998, February 4]. . 3‘ Brief Webography. You are visitor number et rntoto of ome a e This page was last updated on 08/28/98. Copyright 0 1998 Gayle L. Wolfe Just kidding Page 1 of 1 Order Form You are visitor number ooooooooooooooooo bib-[meievaluauea :313:W§il