EXPLORING ONLINE NEWS CREDIBILITY THROUGH AESTHETICS AND USER-CONTRIBUTED CONTENT By Michael S. Reed A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of MASTER OF ARTS Telecommunication, Information Studies, and Media 2011 ABSTRACT EXPLORING ONLINE NEWS CREDIBILITY THROUGH AESTHETICS AND USER-CONTRIBUTED CONTENT By Michael S. Reed This study explores how interactive features on a news website affect a user's perception of credibility and quality. A between-subjects experiment (n=122) was conducted online using four websites with varying levels of user-contributed content to determine if there was an effect. No significant difference in user judgments was found between the four websites. Further analysis showed sub-dimensions within the credibility measure and specifically found significant differences between groups when making judgments regarding accuracy. The study concludes by offering design recommendations to encourage user-contributed content on news websites and suggestions for future work in the field of design-related credibility research and online news websites in general. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! This thesis is dedicated to my wife, Katy, for her love and support during my journey for my Masters degree and to our five-month-old daughter, Elena, whose smiles kept me going through the final stretch. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! """! ACKNOWLEDGMENTS I owe my sincere gratitude to my committee chair, Cliff Lampe, and committee members, Robert LaRose and Carrie Heeter, for their guidance, support, and patience through this thesis. It was an honor to have the opportunity to learn from them over the course of my graduate studies. ! ! ! ! ! ! ! ! ! ! "#! TABLE OF CONTENTS ! LIST OF TABLES......................................................................................................................... vi LIST OF FIGURES ..................................................................................................................... viii 1. INTRODUCTION .......................................................................................................................1 2. LITERATURE REVIEW ............................................................................................................3 Online News Credibility ......................................................................................................3 Aesthetic Credibility ............................................................................................................5 User-contributed Content.....................................................................................................6 Research Questions..............................................................................................................7 3. METHODS ..................................................................................................................................8 Sampling Procedure & Description .....................................................................................8 Measures ..............................................................................................................................8 Design & Procedures .........................................................................................................10 4. RESULTS ..................................................................................................................................13 5. DISCUSSION ............................................................................................................................15 Limitations & Shortcomings..............................................................................................20 Design Recommendations .................................................................................................22 Future Work .......................................................................................................................25 APPENDIX....................................................................................................................................28 LIST OF REFERENCES...............................................................................................................42 ! ! ! ! "! LIST OF TABLES 1. Dimensions of Credibility............................................................................................................9 2. Summary of Credibility Measure, One-way ANOVA ..............................................................14 3. Summary of Credibility Measure, Means..................................................................................14 4. Pairwise Comparison of Means, Tukey HSD ............................................................................14 5. Summary of Quality Measure, One-way ANOVA ...................................................................14 6. Summary of Quality Measure Means ........................................................................................15 7. Pairwise Comparison of Means, Tukey HSD ............................................................................15 8. Participant attitudes towards news article comments ................................................................17 9. Summary of Comment Enjoyment/Usefulness Measure, One-way ANOVA...........................18 10. Summary of Comment Enjoyment/Usefulness Measure, Means ............................................18 11. Summary of Comment Environmental Involvement, One-way ANOVA...............................19 12. Summary of Comment Environmental Involvement, Means ..................................................19 13. Summary of Accuracy Measure, One-way ANOVA ..............................................................20 14. Summary of Accuracy Measure, Means..................................................................................21 15. Credibility (TOTAL) One-way ANOVA ................................................................................33 16. Credibility: Summary of Means and F-values .........................................................................34 17. Quality Measure (TOTAL), One-way ANOVA......................................................................35 18. Quality: Summary of Means and F-values ..............................................................................36 ! "#! 19. Other Factors: Summary of Means and F-values ....................................................................37 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! "##! LIST OF FIGURES 1. Respondents primary source for news .........................................................................................8 2. Stimulus Material. Screenshots with manipulated variables. ....................................................11 3. Side by side comparison of screenshots. ...................................................................................37 4. Screenshot of website A ............................................................................................................38 5. Screenshot of website B.............................................................................................................39 6. Screenshot of website C.............................................................................................................40 7. Screenshot of website D ............................................................................................................41 ! ! ! ! "###! 1. INTRODUCTION News delivery and consumption changed with the introduction of the commercial Internet in the 1990's. The Internet made publishing easier, cheaper, and faster. As the Internet's popularity grew, the sources of information grew as well. Major media outlets created websites to supplement their print versions and independent bloggers began publishing their own articles and opinions around the web. By the end of 2005, 50 million Americans used the Internet to find news each day (Pew Internet, 2006). As of 2010, the Internet is the third most popular news platform - ahead of radio, local and national newspapers, behind only local and national television news (Pew Internet, 2010). People no longer look to one or two sources for news, but browse, share, and interact through social networks, blogs, email, RSS, message boards, and text messaging. 37% of Internet users are participating in news through comments on stories, sharing stories on social networks, or tagging content (Pew Internet, 2010). Social network usage has grown exponentially (Pew Internet, 2009) in recent years and has profoundly changed how news is spread. Social networks offer users new ways to quickly share news online and participate in real-time as news happens (Pew Internet, 2010). One aspect in the changing ecosystem of news has been the introduction of independent niche news websites that focus on specific topics (i.e. technology, entertainment, politics) or concentrate coverage geographically by region, state, city, or neighborhoods (e.g. "hyperlocal"). These new websites range from professional blogs run by one or two people to full-scale newsrooms with reporters covering a wide variety of topics. One of the challenges facing these ! "! new forms of online journalism, whether it be blogs, regional news websites, or niche-topic websites, is building credibility in today's competitive media environment. The challenges facing these new online ventures affect writers, publishers, journalists, web designers, and advertisers alike. News websites have been slower to adopt new forms of interactivity offered by the Internet. Chung (2008) found a majority of news website producers had concerns about moderating user interaction on their websites. Interactive features require time and resources, which puts a greater workload on publishers and reporters. In these challenging times for print newspapers, every decision related to a website's design, functionality, or usefulness has to be carefully considered. The goal of this study was to explore how interactive features and website design affect a reader's view of credibility and quality. It aims to build on research related to "surface credibility" - or how much a user believes something based on simple inspection (Fogg, 1999) and other studies that have found effect between aesthetic treatments and user perception of credibility and quality (Robins & Holmes, 2006). ! #! 2. LITERATURE REVIEW People assess credibility online differently and have multiple concepts of credibility depending on the information they're evaluating online (Hilligoss and Rieh, 2007). Therefore the dimensions of credibility vary through different contexts and topics. For example, the factors used to judge the credibility of a website focused on health information are different than the factors someone would use while seeking news online. One proposed framework for credibility assessment (Hilligoss & Rieh, 2007), suggests people tend to apply certain aesthetic-based heuristics like the "quality of layout" and "professional design" to judge credibility. Other studies have explored the effect of reader motivations (Johnson, et al. 2007), the influence of message content (Austin & Dong, 1994), the effect of interactive features (Chung, 2008), and how aesthetic variables influence perception (Fogg, et al. 2001; Lindgaard, et al. 2006; Robins, & Holmes, 2006; Cousaris & Watrall, 2008; Tractinksy, et al. 2005). The following is a review of the previous research related to news credibility, aesthetic factors, and user-contributed content. Online News Credibility The Internet not only changed the way news is delivered, but changed how news is received. Prior to the widespread use of the Internet, news was primarily a "one-to-many" form of communication and news credibility research focused on credibility of the source (Sundar, 1998). People discussed the news in the public sphere, but not at the speed and scale provided by Internet. With the introduction of the Internet, the delivery of news was reshaped into a "manyto-many" flow of information where readers could comment, share, and participate with the news in new ways. This change also gave scholars new opportunities to explore credibility as it related to the Internet as a medium for news delivery. ! $! Previous studies regarding news credibility have focused on message content and source. Austin and Dong (1994) tested the effect of message content versus the reputation of the source by having participants read a fictitious news articles and answer questions related to credibility and reality of the stories. They found that participants relied more on the content of the article than the reputation of the source when making believability judgments. Sundar (1998) had participants read several types of news articles (local news, national news, international news, sports, entertainment, business) and found that people rate stories with quotes significantly higher that stories without quotes. Finally, Kiousis and Dimitrova (2006) had participants rate news articles versus press releases and found that there was no significant difference in credibility judgments between the two. Recent research has found that the Internet is at a disadvantage in terms of credibility when compared to other media. Hilligoss and Rieh (2007) conducted interviews with undergraduate students and found that the Internet was rated lower in terms of credibility when compared to books and scholarly journals. Similarly, Metzger, et al. (2000) found that college students perceive information found online to be less credible than newspapers. Further complicating the matter was a follow up study by Flanagin and Metzger (2003) that found that college students rate information online as more credible when compared to the general population, but are less likely to verify information they find online. Cassidy (2007) found that online journalists rated news information online as significantly more credible than print newspaper journalists. These studies illustrate the differences between populations when judging credibility online. ! %! Aesthetic Credibility Research exploring the effects of aesthetics and design on credibility has developed through the years. The Elaboration Likelihood Model (Petty & Cacioppo, 1986) discusses how peripheral cues affect people's attitudes. ELM suggests procedures to manipulate these cues regardless of the arguments being made (similar to later research on the effect of aesthetic factors on credibility). While Petty and Cacioppo admit these procedures may not show why a variable was effective or rule out the possibility of other factors, but the suggested procedures can indicate whether or not a variable has the potential to be a peripheral cue. Aesthetic factors have shown to have to an influence on user's credibility judgments. In a survey with over 2,500 participants, Stanford's Persuasive Technology Lab found that "design look" was most frequently mentioned for assessing a website's credibility (Fogg, et al. 2003). Website design was mentioned more frequently than information focus, accuracy, name recognition, reputation, tone of the writing, readability, and many more factors. Robins and Holmes (2006) presented the same content using different levels of aesthetic treatment (high vs. low) and found that the content with a higher aesthetic treatment was judged as having higher credibility. The researchers called this the amelioration effect and suggested that more research was needed to explore exactly what features, elements, or configurations impact credibility. Further research has explored specific design elements and how they relate to a user's perception of a website. Coursaris, et al. (2008) manipulated the "color temperature" (warm vs. cool) of a ! &! website and found that using a cool primary color reflected positively on the user's perception of the website. Similarly, the existence of greater simplicity and options for reading information easily were shown to positively influence a person's loyalty to online news website (Flavian, et al. 2005). Rain and Karmikel (2009) found that structural elements (name, navigation menu, links to external sites, images, etc) all were positively associated with perceptions of credibility on health websites. Another form of aesthetic research looks at what Fogg describes as "surface credibility", or how much a user believes something based on simple inspection. In a preliminary study using online experiments, Fogg, et al. (2001), found that banner ads reduced the perceived credibility of a website, while including a formal photo of an author increased the credibility of the website. Related to this concept is research showing that users judge the quality of a website in under a second of viewing it (Lindgaard, et al., 2006). Based on this finding, a study by Alsundani and Casey (2009) asked participants to view 13 pairs of websites side-by-side and judge their credibility after viewing it for 3.42 seconds. "Unity" (balance, contrast, harmony, dominance) in design was found to be an effective factor for making immediate judgments on credibility. User-contributed Content News websites have implemented interactive features to allow readers to rate, comment, discuss, and share stories through various channels (email, social networks, mobile devices). Research has explored how these interactive features affect a reader's perception and experience. These findings support the idea that interactive features affect a user's perception of online news. ! '! Chung (2008) surveyed over 500 respondents and found that interactive features are generally used infrequently, but perceived credibility of online news was a positive predictor for use of interactive features like submitting stories, emailing reporters, writing letters to the editor, using message boards, and chats. Flavian, et al. (2005) found that the perceived usability of a website had a significant influenced on a reader's loyalty. Sundar (2000) had participants read three news articles with varying levels of multimedia - images, photos, and video. While multimedia features did affect reader's ability to recall stories, those features had no significant affect on news credibility. Further research by Sundar, et al. (2003) had groups of participants view a political candidate's website with varying levels of interactivity and found that the level of interactivity influenced participants’ perception of the candidate. Research Questions To further explore exactly what features, elements, or configurations impact credibility, this study specifically focuses on the following questions: RQ1: Does the ability to comment on an online news story affect a user's perception of credibility? RQ2: Does the existence of comments on an online news story affect a user's perception of credibility? RQ3: Does the existence of social media features on an online news story affect a user's perception of credibility? ! (! 3. METHODS Sampling Procedure & Description Undergraduate students from introductory telecommunications courses at a major Midwestern university in the United States were recruited by email and offered extra credit to complete the online survey. One hundred and twenty-two (n=122) of those who responded were usable for the sample. Participants' age ranged from 18 to 28 years old. 61% were male and 39% were female. 59% ranked the Internet as their primary source for news. 49% had been using the Internet for 610 years and 45% had been using the Internet for more than 10 years. 84% of the participants graduated from a high school within the state covered by the environmental website used as stimulus material. Figure 1. Respondents primary source for news ;:<46! "$0! 7-859:9-.5! "%0! *+,-.+-,! &/0! 1-2-34546+! "%0! ! )! Measures Credibility is a difficult concept to measure because individual definitions of credibility can differ. The factors of what influences credibility also vary depending on the user motivations. In order to deal with this fact, several scales and measures have been developed by scholars. Table 1. Dimensions of Credibility Sundar (1998) Accuracy, believability, bias, fairness, objectiveness, sensationalism Flanagan and Accuracy, believability, bias, trustworthiness, completeness of Metzger (2000) information Cassidy (2007) Accuracy, believability, fairness, comprehensiveness Credibility An overall credibility score was calculated for each participant by adding scores of 11 survey items. The measure was based on four 5-point Likert scale questions (strongly agree/strongly disagree) using four dimensions of online news credibility - believable, fair, accurate, comprehensive (Flanagin & Metzger, 2000; Cassidy, 2007) and a 7-item, 5-point semantic differential scale with further measures of credibility - accurate, comprehensive, believable, fair, biased, objective, and factual (Sundar, 2000). The scale was reliable (! = Alpha reliability = 0.8493, Standardized alpha = 0.8589). Quality Next, a measure of quality was constructed using scores from an 8-item, 5-point semantic differential scale. Participants were asked to rate the website using the following items: well written, organized, clear, coherent, user-friendly, interactive, and enjoyable. An overall quality score was calculated by adding these scales. The scale was reliable (! = Alpha reliability = 0.7952, Standardized alpha = 0.8133). ! /! Design & Procedures A between-subjects experiment (randomized, post-test only) was conducted. Undergraduate students in introductory telecommunications courses were recruited through email and offered extra credit to complete the online survey. A link to the survey was included in the recruitment email. Students who did not choose to take the survey were offered an alternate option for extra credit. Participant's answers were anonymous. Stimulus Material A regional environmental news website was used as stimulus. The website was chosen for two reasons. First, the author of this study had previously served as lead web designer on the site and had familiarity with the design, content, and features. Second, the website serves as a good representation of a niche news website focusing on a specific geographic region. Similar websites covering different topics (i.e. technology, entertainment, politics), regions, and cities are attempting to fill the void that traditional news organizations have left as the economics of news has changed. An article from the website was selected based on the following criteria: 1) non-controversial topic, 2) related to national environmental news, and 3) had more than one comment. The selected article was titled "Researchers study how climate change chases fish from streams". A screenshot of the article was then taken and manipulated using Photoshop to create four versions of the same article. Using a screenshot controlled for variables such as layout, fonts, colors, and photos that can render inconsistently on different browsers and computer platforms. The manipulated variables for each screenshot were as follows: ! "=! Figure 2. Stimulus Material. Screenshots with manipulated variables. For interpretation of the references to color in this and all other figures, the reader is referred to the electronic version of this thesis. Website A Website B Website C Website D story story + the ability to story + the ability to story + the ability to comment (comment comment (comment comment (comment box) box) + comments on box) + comments on story story + social media features (social media icons, RSS feed icons) ! ""! Procedures A link to access the online survey was included in the recruitment email. After agreeing to a consent form and answering basic questions about their Internet use and primary news sources, participants were presented the following statement: On the next page, you will view an image of a website. Please take a few minutes to review it and get an overall sense of the page (you are not required to read everything on the page). Please answer the questions beneath the image to the best of your knowledge. Participants were then randomly assigned to one of the four screenshots. After viewing the screenshot, participants were first asked if they had previously visited the actual website being used as stimulus material (participants who had visited the stimulus website before were not included in the sample). Participants were then asked to rate the website using the previously detailed scales related to credibility and quality. A manipulation check was included in the scale to verify if participants recognized the level of user-contributed content of the screenshot. After rating the website, participants were asked about their attitudes towards news, comments, and Internet self-efficacy. Finally, participants were asked several demographic questions. Participants who had previously visited the stimulus website were excluded from the final data set. Incomplete surveys were also excluded from final analysis. This accounted for some of the variance between group sizes. Participants were randomly assigned to a group by the survey tool, so there was little control over the final group sizes. Prior to data cleansing, group sizes ranged from 30 to 50 participants. ! "#! 4. RESULTS Data was analyzed using Microsoft Excel and R. A one-way ANOVA was used to test for credibility differences among four websites. Perception of credibility did not differ significantly across the four groups, F (3, 118) = 0.995, p = 0.3978. Table 2. Summary of Credibility Measure, One-way ANOVA Df Sum of Squares Mean Square Group 3 87.4 29.147 Residuals 118 3456.7 29.294 Table 3. Summary of Credibility Measure, Means Mean Website A (story) 24.28947 Website B (story + comment box) 25.25000 Website C (story + comment box + 26.76190 comments) Website D (story + comment box + 25.64516 comments + social media icons) F 0.995 Pr(>F) 0.3978 sd 5.177435 4.865945 5.448897 n 38 32 21 6.156560 31 Table 4. Pairwise Comparison of Means, Tukey HSD. Credibility Measure (95% family-wise confidence level) Estimate lower upper Group B - Group A 0.9605 -2.4195 4.3405 Group C - Group A 2.4724 -1.3581 6.3029 Group D - Group A 1.3557 -2.0538 4.7651 Group C - Group B 1.5119 -2.4444 5.4682 Group D - Group B 0.3952 -3.1550 3.9453 Group D - Group C -1.1167 -5.0982 2.8647 A one-way ANOVA was used to test for quality differences among four websites. Perception of quality did not differ significantly across the four groups, F (3, 118) = 0.4483, p = 0.719. Thus, these findings support the conclusion that interactive features have no significant effect on a user's perception of credibility or quality of a news website. Table 5. Summary of Quality Measure, One-way ANOVA Df Sum of Squares Mean Square Group 3 24.74 8.2454 Residuals 118 2170.08 18.3905 ! "$! F 0.4483 Sig. 0.719 Table 6. Summary of Quality Measure Means Website A (story) Website B (story + comment box) Website C (story + comment box + comments) Website D (story + comment box + comments + social media icons) Mean 18.84211 19.71875 18.42857 19.22581 sd 3.709202 4.531142 5.343888 3.887670 n 38 32 21 31 Table 7. Pairwise Comparison of Means, Tukey HSD. Quality Measure (95% family-wise confidence level) Estimate lower upper Group B - Group A 0.8766 -1.8029 3.5562 Group C - Group A -0.4135 -3.4502 2.6232 Group D - Group A 0.3837 -2.3192 3.0866 Group C - Group B -1.2902 -4.4266 1.8462 Group D - Group B -0.4929 -3.3074 2.3215 Group D - Group C 0.7972 -2.3591 3.9536 Based on the results of both one-way ANOVAs in this study, the answer to all three research questions is that there is no significant difference between credibility or quality judgments based on the level of user-contributed content offered by a news website. ! "%! 5. DISCUSSION While the results of the experiment are somewhat disappointing, there is still knowledge to be drawn from the data. It is most likely true, as the study suggests, there is no significant relationship between the selected variables and perception of credibility or quality. Even as previous studies have shown significant relationships between credibility and aesthetic quality, design features as minor as a comment box or social networking icons are not significant credibility signals to users of news websites. These features most likely complement other factors such as overall aesthetic quality, brand awareness, author name, author photo, as well as message characteristics like grammar, spelling, and tone. It may also be that college students are more likely to accept what they read online as true and are less likely than the general population to consider credibility when reading news online. Flanagin and Metzger's (2003) research indicated that college students verify the information they find online significantly less than the general population. 98% of the participants in this study said they were confident in their ability to use the Internet (82% strongly agree, 16% somewhat agree). It is possible their high level of Internet self-efficacy combined with their decreased likelihood to verify information online impedes their ability to make well-informed judgments regarding credibility. This survey, by design, used an article with a topic that was not controversial or meant to trigger an extreme response, as it could have overly impacted participant’s judgments regarding aesthetics. But this could have also weakened the results if this topic was of insufficient interest to prompt participants to make thorough judgments regarding credibility. In future work, it would be beneficial to explore how article topics affect user perceptions of credibility. ! "&! The data shows mixed results regarding attitudes towards interactive features related to commenting or recommending. While Chung (2008) found credibility as a strong predictor for using interactive features, this study suggests that interactive features don't predict credibility. The results of this study suggest that college students don't rely heavily on comments or interactive features when judging the credibility of news websites. The data supports this as 75% of respondents said they think, in general, comments make news articles more interesting, but only 57% said comments were useful to them. This supports the findings of this study. If participants don't find comments on news articles useful to them, then it is likely that they aren't using them to judge the credibility or quality of a website. Sundar found that interactivity had a positive effect on a user's perception of credibility of a political candidate's website, but this does not seem to be the case for news. Overall there is a mostly positive view of reading comments and sharing news online, yet posting comments directly on news websites is much less popular. 62% of participants said they were "very likely" or "somewhat likely" to read comments on a news website, but only 20% said they were "very likely" or "somewhat likely" to post a comment on a news website. 42% of participants said they were either "very likely" or "somewhat likely" to read comments on the stimulus website, whereas only 10% said they were "very likely" or "somewhat likely" to post a comment on the stimulus website. Table 8. Participant attitudes towards news article comments. In general “Very likely" or "somewhat likely" to read comments “Very likely" or "somewhat likely" to post a comment Enjoy reading comments on a news website (strongly agree or somewhat agree) ! Stimulus website 62% 20% 10% 71% "'! 42% N/A Table 8. (cont’d) Comments make news websites more interesting (strongly agree or somewhat agree) Comments on news websites were useful to them (strongly agree or somewhat agree) 75% N/A 57% N/A Data was also analyzed to learn how level of enjoyment for reading comments varied by treatment. Six survey questions related to involvement were totaled for each participant. A oneway ANOVA was used to test for the level of enjoyment of reading comments among four websites. Level of enjoyment did not differ significantly across the four groups, F (3, 118) = 0.8764, p = 0.4555. Table 9. Summary of Comment Enjoyment/Usefulness Measure, One-way ANOVA Df Sum of Squares Mean Square F Pr(>F) Group 3 76.5 25.515 0.8764 0.4555 Residuals 118 3435.6 29.115 Table 10. Summary of Comment Enjoyment/Usefulness Measure, Means Mean sd Website A (story) 14.50000 4.421997 Website B (story + comment box) 16.59375 6.262327 Website C (story + comment box + 15.52381 6.177532 comments) Website D (story + comment box + 15.35484 4.943335 comments + social media icons) n 38 32 21 31 Although level of enjoyment did not vary by treatment condition, further analysis was done to check for correlations between level of involvement and credibility judgments. A Pearson correlation coefficient was calculated for the relationship between participant’s enjoyment of reading comments and their total credibility score. A weak correlation that was not significant was found (r (120) = .087, p > .05). Enjoyment of reading comments is not related to credibility. ! "(! Participant’s level of interest in environmental issues as it relates to credibility was also explored. Three survey questions related to participant’s interest and involvement in environmental issues were totaled into one score. A one-way ANOVA was used to test for the level of involvement in environmental issues among four groups. Level of involvement did not differ significantly across the four groups, F (3, 118) = 0.909, p = 0.439. Table 11. Summary of Environmental Involvement Measure, One-way ANOVA Df Sum of Squares Mean Square F Pr(>F) Group 3 115.2 38.39 0.909 0.439 Residuals 118 4983.7 42.235 Table 12. Summary of Environmental Involvement Measure, Means Mean sd Website A (story) 23.94737 6.120181 Website B (story + comment box) 25.90625 6.869096 Website C (story + comment box + 26.19048 7.263739 comments) Website D (story + comment box + 26.06452 5.999642 comments + social media icons) n 38 32 21 31 A Pearson correlation coefficient was also calculated for the relationship between participant’s interest in environmental issues and their total credibility score. A weak correlation that was not significant was found (r (120) = .072, p > .05). Enjoyment of reading comments is not related to credibility. As noted in the literature review section, the Elaboration Likelihood Model describes how peripheral cues affect people's attitudes. In this experiment, the manipulated variables appear to not have been effective cues for participants to make credibility judgments. Nor did the groups vary based on their level of enjoyment of comments or their involvement in environmental issues. ! ")! Type II Error These results demonstrate Type II error. Accepting the null hypothesis based on each participant’s total credibility score resulted in a claim of no difference between groups when in fact, there were significant differences regarding sub-dimensions of credibility. To further explore the Type II error, power analysis was done using R. A power of 0.4405 was calculated using 21 as the common number of participants in each group, at a significance level of .05 and a “medium” effect size of 0.25. This low value of power shows that the sample size was not large enough. Further analysis showed that 45 participant’s per group would yield a power of 0.803, a minimum desirable level of power when using .05 level of significance. Sub-dimensions of Credibility Some data suggest that there was in fact a difference between the four groups. Upon further inspection, two factors were extracted from the overall credibility scale for insights into the subdimensions of credibility, specifically the concept of accuracy. The two survey questions regarding accuracy were then totaled to create a score for each participant. A one-way ANOVA was used to test for the accuracy sub-dimension among four groups. Accuracy score differed significantly across the four groups, F (3, 118) = 2.7564, p = 0.04548*. Tukey’s HSD revealed that users who saw the website with less user-contributed content (m = 3.94, sd = 1.16) gave the website a higher accuracy rating than users who saw a website with more user-contributed content (m = 4.64, sd = 1.40). Table 13. Summary of Accuracy Measure, One-way ANOVA Df Sum of Squares Mean Square Group 3 12.355 4.1182 Residuals 118 176.301 1.4941 ! "/! F 2.7564 Pr(>F) 0.04548 * Table 14. Summary of Accuracy Measure, Means Mean Website A (story) 3.947368 Website B (story + comment box) 4.375000 Website C (story + comment box + 4.761905 comments) Website D (story + comment box + 4.645161 comments + social media icons) sd 1.1612521 0.9755065 1.3749459 n 38 32 21 1.4035286 31 These results suggest that user-contributed content influenced participant’s judgment of accuracy in a negative way. While it is difficult just what factor caused participants to judge the website as less accurate, it could have been simply a result of having to process more peripheral information on the page, or possibly something specifically mentioned in the user-submitted comments on the article. These results also suggest that accuracy itself is regarded differently than other dimensions of credibility like fairness, bias, and trustworthiness. When making credibility judgments, readers may judge a source to be accurate, but neither fair nor unbiased. Or the inverse, readers may believe a news sources as being fair and unbiased, but not necessarily accurate. Limitations & Shortcomings It could be argued that by only surveying college students, the results of this survey are not applicable to the general population. This study is still relevant to the general population because college students are the first generation to have grown up using the Internet and have relied on it as a source of news. At a time when the news and media landscape is going through a period of major upheaval, understanding college student’s attitudes and behaviors towards news is important because they represent the market for news consumption in the future. ! #=! A possible threat to internal validity was the timing of the survey. The survey was administered during the 2010 mid-term elections, resulting in an increase of campaign advertisements challenging the credibility of political opponents. It is possible that some participants had a heightened sense of credibility and were more likely to question the believability, accuracy, or fairness of news in general. Another threat to internal validity may have been the national and local oil spills that occurred during the months prior to the survey. Both of which could have raised participant's awareness to environmental issues and affected their response to an environmental news story. Lack of user motivation most likely also played a role in this study. Giving participants stronger motivations to investigate the website thoroughly through a scenario such as researching a paper for school or looking for health news may have resulted in stronger scores. In this study, participants were simply asked to review the website. It would have been beneficial to present them with a scenario that provided more motivation to trust the website they were rating. A possible scenario statement could have read, "Imagine you're working on a paper for a class assignment and need to find a recent environmental story. You search the Internet for "environmental news" and the third result is the following website." While previous research regarding "surface credibility" (Fogg, 1999) has established that users make judgments about a website's credibility by simply looking at the page, it is difficult to isolate specific design elements that significantly influence their decisions. It should also be noted that the placement of most of the aesthetic variables in this study were “below the fold”, ! #"! meaning that participants likely did not see the variables in the first few seconds of viewing the page. This was confirmed by the manipulation check question in the survey. Participants failed to notice the manipulated variables when reviewing the screenshots. This design flaw could have resulted in participants using other non-aesthetic factors to make credibility judgments about the website. Aesthetics establish an initial reaction or emotion, but credibility is most likely established over longer periods of time - days, weeks, or years. The concepts related to credibility such as believability, fairness, accuracy, and comprehensiveness require an established relationship between reader and publisher. Factors such as who is recommending the website or if they have visited the website before also play a role in user's perceptions of credibility. In design research there are limitations to an online survey. Design research cannot rely solely on numbers. Based on this study, interactive design features are not significant factors in a user's perception of a news website's credibility or quality. The features used in this study offered negligible differences between the four websites, and therefore were unable to significantly influence participant's scores. These results are consistent with research by Sundar (2000) and Collins (2006) that showed no significant difference in credibility scores using independent variables such as multimedia features, website creator, or brand. Design Recommendations Exploring credibility through design features has challenges. People's tastes, perceptions, attitudes, and preferences towards design differ and change over time. Technology tends to stay ! ##! one-step-ahead of design by introducing new platforms, formats, and devices. Developers and designers have learned to quickly adapt and create usable, meaningful, and aesthetically pleasing interfaces to help users enjoy technology. The results of the study are meaningful for news website designers in several ways. First, it provides quantitative data suggesting that certain features on a news website have no significant effect on a user's perception credibility or quality. This lack of influence on user perceptions suggests that news website designers need to explore new ways to encourage interaction with the hope of engaging users and building credibility. Second, the results shows that users are much less likely to post comments on news websites than they are to simply read them. Again, this shows how designers have failed to create an experience that takes advantage of the usercontributed content offered by the Internet. Finally, the findings show that users find comments on articles interesting, but not useful. Designers should find ways to improve the experience of commenting and comments on news websites. Most news websites today offer the ability to interact with news articles by commenting, sharing, or recommending, and while it is unlikely these features will be left out of news website in the future, the findings do provoke questions about their effect. Today's news websites follow established layout and design principles. The focus is on the content of the article, usually justified on the left side of the page. A comment box and comments are found at the bottom of the article and treated as supplementary content. Features related to sharing and recommending are then embedded in various places on the page - before the first paragraph of the article, in ! #$! dedicated boxes for user tools, or at the bottom of the article. The right margin of the article is usually reserved for recent comments, links to other articles, and advertisements. This study suggest that interactive features for commenting and sharing should be given more prominence to more effectively influence user perceptions of the website. The challenge for designers is to make better use of these features by integrating them more into online news articles without interfering with overall experience. Offering users a more interactive experience will keep readers engaged and could increase return visits, indirectly improving the credibility of a news website. Interactive features should not be an afterthought to the overall design of a news website. News website designers should explore new ways to embed these features into the user experience. The following are several recommendations to news website designers. 1. Explore more effective ways to support conversation on news websites. Offering users an experience that mimics the real-life ways that people discuss news rather than a list of the most recent comments. For example, integrating comments or the ability to comment within an article. 2. Provide better tools for sharing and recommending articles on a news website. While encouraging users to share on other networks (i.e. email, Facebook, Twitter) is important to build an audience, designers and developers could create features on their own websites provide context around articles. For example, designers could incorporate a ! #%! "shared score" to aggregate how many times a story has been shared on various networks and prominently place it on the article. 3. Experiment with alternative page layouts to encourage user-contributed content. For example, remove the primary menu on an article page or move the comment box to a more prominent position next to the article to reduce clutter and promote interaction. Future Work Going forward, it would it would be beneficial to supplement this study with qualitative data to further explore credibility and quality. Future work should focus on qualitative research such as one-on-one interviews or field studies to gain a deeper understanding of how users judge the credibility of news websites within a given set of circumstances. One-on-one interviews using the same stimulus material would offer an opportunity to ask follow up questions related to credibility and gain a better understanding of how these features affect perception. Participants could be asked to rank the most important features on the page, and then a follow-up study would be designed using those variables to explore significant differences. Future work should also include a follow up study based on the design recommendations mentioned above. Comparing a screenshot from this study with a substantially different version using features that encourage interaction could be done to explore if the differences affect user perceptions of credibility and quality. As noted previously, aesthetic features should be placed “above the fold”, so participants are more likely to immediately see the variables with their initial viewing of the page. ! #&! Finally, future work should also explore different news topics (i.e. business, sports, entertainment) with regards to aesthetics to see if there are differences in perception of credibility. For example, based on prior research, it's possible that users would have a heightened sense of awareness if the news website was related to health issues. This could affect their overall scores related to credibility. This type of research would be useful to news website designers when considering design features for news websites that primarily cover one topic. Conclusion The objective of this study, at its basic level, was to explore the relationship between website design and a user's judgment of information. By exploring how users interact with websites, this research improves understanding for web designers, journalists, and independent news organizations, so they can make informed design decisions. The findings of this study reject the conclusion that reader comments, the ability to comment, or interactive icons have a significant effect on a user's perception of credibility or quality of news websites. Even though these factors did not reveal significant differences, the results are still relevant because they suggest that design features that publishers have shown reluctance to implement, in fact, do not have as much an effect on their credibility as they imagined. The findings also provides insights for news website designers and suggest that design improvements could be made to encourage interaction on news websites. Credibility is something that is built over time. While some interactive features may not have an immediate effect on credibility judgments, they do promote reader engagement and build ! #'! communities around news websites, which entices readers to return to the site. These return visits to a website will help drive participation and establish credibility in the long run. ! #(! APPENDIX ! #)! Survey Instrument Please indicate how much you agree with the following statements. This website is a believable source for environmental news. _____ Strongly agree _____ Somewhat agree _____ Neutral _____ Somewhat disagree _____ Strongly disagree This website is an accurate source for environmental news. _____ Strongly agree _____ Somewhat agree _____ Neutral _____ Somewhat disagree _____ Strongly disagree This website is a fair source for environmental news. _____ Strongly agree _____ Somewhat agree _____ Neutral _____ Somewhat disagree _____ Strongly disagree This website is a comprehensive source for environmental news. _____ Strongly agree _____ Somewhat agree _____ Neutral _____ Somewhat disagree _____ Strongly disagree Please rate the website using the following scales. Accurate / Inaccurate Believable / Unbelievable Biased / Unbiased Clear / Unclear Comprehensive / Incomprehensive Factual / Not factual Fair / Unfair Informative / Uninformative Important / Unimportant Objective / Subjective Persuasive / Not persuasive Sensationalistic / Not sensationalistic ! #/! Well-written / Poorly written Organized / Unorganized Interactive / Not interactive Useful / Not useful Coherent / Incoherent Confusing / Clear Enjoyable / Not enjoyable Sophisticated / Unsophisticated User-friendly / Not user-friendly How likely are you to read the comments on an article on this website? _____ Very likely _____ Somewhat likely _____ Neutral _____ Somewhat unlikely _____ Not likely at all How likely are you to post a comment on an article on this website? _____ Very likely _____ Somewhat likely _____ Neutral _____ Somewhat unlikely _____ Not likely at all I would recommend this website to my friends. _____ Strongly agree _____ Somewhat agree _____ Neutral _____ Somewhat disagree _____ Strongly disagree I would recommend this website to my family. _____ Strongly agree _____ Somewhat agree _____ Neutral _____ Somewhat disagree _____ Strongly disagree I would share a story I found interesting on this website with my friends on a social network (i.e. Facebook, Twitter, Delicious)? _____ Strongly agree _____ Somewhat agree _____ Neutral _____ Somewhat disagree _____ Strongly disagree ! $=! Please indicate how much you agree with the following statements. "I am confident in my ability to use the Internet." "I enjoy reading comments on news articles online." "Comments on articles on news websites are useful to me." "Comments on articles make news websites better." "Comments on articles make news websites more interesting." "I am interested in environmental news." "I am well informed about environmental issues." "Environmental issues are important to me." How likely are you to post a comment on an article on a news website? _____ Very likely _____ Somewhat likely _____ Neutral _____ Somewhat unlikely _____ Not likely at all How likely are you to read the comments on an article on news websites? _____ Very likely _____ Somewhat likely _____ Neutral _____ Somewhat unlikely _____ Not likely at all ! $"! Credibility Measure Scale Reliability Alpha reliability = 0.8493, Standardized alpha = 0.8589 Reliability deleting each item in turn: Table 15. Credibility (TOTAL) One-way ANOVA Alpha accurate/inaccurate 0.8295 believable.unbelievable 0.8401 unbiased/biased 0.8623 comprehensive.incomprehensive 0.8358 factual.notfactual 0.8320 fair.unfair 0.8323 objective.subjective 0.8563 This.website.is.a.believable.source. 0.8329 for.environmental.news. This.website.is.a.comprehensive.so 0.8272 urce.for.environmental.news This.website.is.a.fair.source.for.en 0.8262 vironmental.news. This.website.is.an.accurate.source. 0.8623 for.environmental.news. Group Residuals Df 3 118 Sum of Squares 87.4 3456.7 Std.Alpha 0.8384 0.8492 0.8666 0.8477 0.8423 0.8436 0.8666 0.8416 r(item, total) 0.6455 0.5007 0.2833 0.5442 0.5925 0.5998 0.2812 0.6061 0.8405 0.6423 0.8393 0.6525 0.8666 0.2833 Mean Square 29.147 29.294 F 0.995 Mean Website A (story) Website B (story + comment box) Website C (story + comment box + comments) Group D (story + comment box + comments + social media icons) ! sd n 24.28947 25.25000 5.177435 4.865945 38 32 26.76190 5.448897 21 25.64516 6.156560 31 $#! Pr(>F) 0.3978 Table 16. Credibility: Summary of Means and F-values Website A Website B Website C (story) (story + (story + comment comment box) box + comments) Credibility (TOTAL) Accurate Believable Comprehensive Fair 24.28947 25.25000 26.76190 Website D (story + comment box + comments + social sharing icons) 25.64516 2.078947 1.921053 2.105263 2.157895 2.250000 1.875000 2.281250 2.406250 2.380952 2.095238 2.619048 2.619048 2.290323 1.870968 2.387097 2.516129 Unbiased Objective Factual This website is a believable source for environmental news. This website is a fair source for environmental news. This website is an accurate source for environmental news. This website is a comprehensive source for environmental news 3.263158 2.684211 2.026316 1.868421 3.250000 2.500000 2.187500 1.781250 3.095238 2.619048 2.333333 2.047619 2.774194 2.548387 2.258065 1.838710 0.3477 0.5315 0.1002 0.04981 * 1.7844 0.1539 0.351 0.7885 0.8832 0.452 0.7915 0.501 2.078947 2.187500 2.190476 2.419355 0.8658 0.461 1.868421 2.125000 2.380952 2.354839 3.5812 0.01599 * 2.236842 2.406250 2.380952 2.387097 0.2336 0.8728 ! $$! F Pr 0.995 0.3978 1.1106 0.7378 2.1295 2.6845 Quality Measure Scale Reliability Alpha reliability = 0.7952, Standardized alpha = 0.8133 Reliability deleting each item in turn: Table 17. Quality Measure (TOTAL), One-way ANOVA clear.confusing clear.unclear coherent.incoherent organized.unorganized well.written.poorly.written user-friendly/not user-friendly interactive/not interactive enjoyable/not enjoyable Group Residuals Df 3 118 Sum of Squares 24.74 2170.08 Alpha 0.7763 0.7810 0.7578 0.7590 0.7621 0.7518 0.8144 0.7783 Std.Alpha 0.7953 0.8012 0.7786 0.7808 0.7834 0.7732 0.8193 0.8014 Mean Square 8.2454 18.3905 r(item, total) 0.4838 0.4460 0.6210 0.6099 0.5758 0.6533 0.3287 0.4663 F 0.4483 Mean 18.84211 19.71875 sd 3.709202 4.531142 n 38 32 Website C (story + comment box + comments) 18.42857 5.343888 21 Group D (story + comment box + comments + social media icons) 19.22581 3.887670 31 Website A (story) Website B (story + comment box) ! $%! Sig. 0.719 Table 18. Quality: Summary of Means and F-values Website A Website B Website C (story) (story + (story + comment comment box) box + comments) Quality (TOTAL) 18.84211 19.71875 18.42857 Website D (story + comment box + comments + social sharing icons) 19.22581 Clear/confusing Well written Organized Coherent User friendly Interactive 2.421053 2.157895 1.921053 2.131579 2.105263 2.921053 2.343750 2.187500 2.093750 2.187500 2.343750 2.937500 2.333333 2.380952 1.952381 2.190476 2.047619 2.142857 2.387097 2.451613 2.161290 2.354839 2.322581 2.354839 Enjoyable Clear/unclear 3.105263 2.078947 3.500000 2.125000 3.095238 2.285714 3.096774 2.096774 ! $&! F 0.0596 1.0966 0.8051 0.5991 1.1794 3.8501 Pr 0.4483 0.9808 0.3535 0.4934 0.6168 0.3206 0.01138 * 1.8105 0.149 0.3761 0.7704 Table 19. Other Factors: Summary of Means and F-values Website A Website B Website C (story) (story + (story + comment comment box box) + comments) How likely are you to read the comments on an article on this website? How likely are you to post a comment on an article on this website? I would recommend this website to my friends. I would recommend this website to my family. I would share a story I found interesting on this website with my friends on a social network (i.e. Facebook, Twitter, Delicious). ! 3.315789 3.187500 3.095238 Website D (story + comment box + comments + social sharing icons) 2.741935 4.157895 4.093750 4.285714 4.096774 0.1685 0.9174 2.736842 2.937500 2.714286 2.838710 0.3437 0.7938 2.736842 2.937500 2.714286 2.838710 0.3437 0.7938 2.763158 2.562500 2.142857 2.580645 1.5735 0.1995 $'! F Pr 1.2091 0.3096 Treatment Screenshots Figure 3. Side by side comparison of screenshots Website A Website B Website C story story + the ability to story + the ability to comment (comment comment (comment box) box) + comments on story ! $(! Website D story + the ability to comment (comment box) + comments on story + social media features (social media icons, RSS feed icons, recent comments) Figure 4. Screenshot of website A ! $)! Figure 5. Screenshot of website B ! $/! Figure 6. Screenshot of website C ! %=! Figure 7. Screenshot of website D ! %"! REFERENCES ! %#! REFERENCES Alsudani, F., & Casey, M. (2009). The Effect of Aesthetics on Web Credibility. Austin, E.W., & Dong, Q. (1994). Source v. content effects on judgments of news believability. Journalism Quarterly, 71, 973–983. Cassidy, W.P. (2007). Online News Credibility: An Examination of the Perceptions of Newspaper Journalists. Journal of Computer-Mediated Communication, 12(2), article 7. http://jcmc.indiana.edu/vol12/issue2/cassidy.html Chung, D. (2008). Interactive Features of Online Newspapers: Identifying Patterns and Predicting Use of Engaged Readers. Journal of Computer-Mediated Communication. Collins, J., (2006). An Investigation of Web-Page Credibility. Consortium for Computing Sciences in Colleges. Coursaris, C., Swierenga, S., & Watrall, E. (2008). An Imperical Investigation of Color Temperature and Gender Effects on Web Aesthetics. Dimitrova, D., & Kiousis, S. (2006). Differential impact of Web site content: Exploring the influence of source (public relations versus news), modality, and participation on college students’ perceptions. Flanagin, A.J., Metzger, M.J. (2000). Perceptions of Internet information credibility. Journalism and Mass Communication Quarterly, 77 (3), 515–540. Flavian, C., Guinali, M., Gurrea, R., (2005), The influence of familiarity and usability on loyalty to online journalistic services: The role of user experience. Fogg, B.J., & Tseng, H. (1999). The Elements of Computer Credibility. Fogg, B.J., Marshall, J., Kemeda, T., Solomon, J., Rangnekar, A., Boyd, J., & Brown B. (2001). Web Credibility Research: A Method for Online Experiments and Early Study Results. Fogg, B.J., Soohoo, C., Danielson, D., Marable, L., Stanford, J., & Tauber, E. (2003). How Do Users Evaluate the Credibility of Web Sites? A Study with Over 2,500 Participants. Haig, W.L. (2006). How and Why Credibility-Baseed Company Logos are Effective in Marketing Communiation in Persuading Customers to Take Action: A Multiple Case Study Toward a Better Understand of Creativity in Branding. Ph.D. dissertation, Southern Cross University, Lismore, Australia. Hilligoss, B., & Rieh, S.Y. (2007). Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. ! %$! Johnson, T. J., Kaye, B. K., Bichard, S. L., & Wong, W. J. (2007). Every blog has its day: Politically-interested Internet users' perceptions of blog credibility. Journal of ComputerMediated Communication, 13(1), article 6. http://jcmc.indiana.edu/vol13/issue1/johnson.html Lindgaard, G., Fernandes, G., Dudek, C., & Browñ, J. (2006). Attention web designers: You have 50 milliseconds to make a good first impression! Behaviour and Information Technology, Volume 25, Number 2, Number 2/March-April 2006, pp. 115-126 (12). Metzger, M., Flanagin, A., Zwarun, L. (2003). College student Web use, perceptions of information credibility, and verification behavior. Petty R., & Cacioppo J. (1986). The Elaboration Likelihood Model of Persuasion. Pew Internet (2006). Online News: For many home broadband users, the Internet is a primary news source, http://www.pewinternet.org/Press-Releases/2006/Online-news-hits-a-newhighwater-mark-in-popularityFor-many-home-broadband-users-online-n.aspx Pew Internet (2005). The State of Blogging. http://www.pewinternet.org/Reports/2005/TheState-of-Blogging.aspx Pew Internet (2009). Adults on Social Network Sites, 2005-2009. http://pewinternet.org/Infographics/Growth-in-Adult-SNS-Use-20052009.aspx Pew Internet (2010). Understanding the Participatory News Consumer. http://pewinternet.org/Reports/2010/Online-News.aspx Pew Internet (2010). How Media Consumption Has Changed Since 2000 http://pewinternet.org/Presentations/2010/Jun/How-Media-Consumption-Has-Changed-Since2000.aspx Rain, S., & Karmikel, C. (2009). Health information-seeking and perceptiosn of website credibility: Examinging Web-use orientation, message characteristics, and structural features of websites. Robins, D, & Holmes, J. (2006). Aesthetics and credibility in web site design. Shaikh, A.D., Fox, D., & Chaparro, B. (2007). The Effect of Typeface on the Perception of Email. Sundar, S. (1998). Effect of Source Attribution on Perception of Online News Stories. Journalism & Media Communications Quarterly. Sundar, S., (2000). Multimedia Effects on Processing and Perception of Online News: A Study of Picture, Audio, and Video Downloads. ! %%! Sundar, S., Kalyanaraman, S., Brown, J., (2003). Explicating Web Site Interactivity Impression Formation Effects in Political Campaign Sites. Tractinksy, N., Cokhavi, A., Kirschenbaum, M., & Sharfi, T. (2005). Evaluating the consistency of immediate aesthetic perceptions of web pages. ! %&!