CONSTRUCTION AND EVALUATION OF DATA-DRIVEN LEARNING MODULES FOR EFL WRITERS’ HEDGING IN ACADEMIC ENGLISH By Adam Pfau A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Second Language Studies – Doctor of Philosophy 2024 ABSTRACT This experimental study, and partial replication of Sun and Hu (2020), examined how and to what extent direct (i.e., hands-on corpus use) or indirect (i.e., hands-off corpus use) online data-driven learning (DDL) modules were able to assist Chinese writers from an English as a foreign language (EFL) context in using more appropriate hedging (i.e., in patterns and frequencies closer to native English-speaking writers). Participants' perceptions regarding the challenges and benefits of the online corpus tools, as well as the online instructional modules, were assessed using Likert-scale surveys, open-ended survey responses, and interviews with select participants aided by a stimulated recall procedure. Students participated in four weeks of online DDL instruction, and a pre- and post- instruction writing task. Participants' hedging frequencies and patterns were statistically examined pre- and post- instruction, with log likelihood and mixed-effects modeling. Participants' Likert-scale survey responses were also examined with their open-ended survey responses and interview data. Results showed that, when both groups (direct and indirect DDL groups) were considered together, there was a significant difference across time (pre- and post- instruction) on the frequency of hedge use, indicating that DDL instruction, in general, helped to significantly increase hedge usage frequencies over time. When each group was examined separately, the direct DDL group showed a significant change over time, and the indirect DDL group showed non-significant changes over time. Therefore, while DDL instruction as a whole seemed beneficial, a more detailed analysis revealed that the direct instruction was statistically more effective. Results from the surveys and interviews identified larger patterns, and then more specific sub-themes, that participants found challenging and beneficial regarding the interface, functions, and texts, of the corpora. Challenges included, among others, the lack of easily accessible help information on the interfaces, error messages and account issues, a lack of understanding the role of certain functions, and issues with the compilation of the corpora. Benefits included, among others, the authenticity of the texts, the ability to do comparisons (i.e., across text types and learner English populations), and the ability to see keywords in larger contexts. ACKNOWLEDGEMENTS Before anything, I cannot start this dissertation without thanking my advisor, Dr. Charlene Polio, who has been, hands down, the absolute best mentor I could have imagined throughout my entire graduate school experience, and more broadly throughout this chapter of my life. Her guidance, support, friendliness, and mentorship has allowed me to develop and grow in ways that I never, never, would have thought possible six or seven years ago, and has contributed so much to making this part of my life as enjoyable as it has been. I’d also love to thank Dr. Patti Spinner for overseeing my first real teaching (TA’ing) experience and helping make the second half of my PhD program so rewarding and fun. Also, Dr. Dustin De Felice for his unending kindness, and for being a constant source of laughter and company at Wells Hall. I also need to thank all of the remaining faculty in the Second Language Studies and MA TESOL programs. What terrific programs, filled with terrific people, and I will always value and look back fondly on all the encounters I shared with all of them. I can honestly say I thoroughly enjoyed every second of my time in both programs. I also want to thank all of the friends I made during my graduate school experience, who helped make it so much fun for me. There are way too many to name, but just several include: Mary Li, Quy Pham, Soon-Young Oh, Muta Mutashobya, Eunmi Kim, Alyssa Wolfe, Phil Montgomery, Ngoc Huy, Anthony Lizan, Xinyi Guan Valerie, Steven Gagnon, and, again, way too many others to list. These friendships are one of the best things I got out of graduate school, and I know they will carry on for a long, long, time. Also, Darwin, Tammi, and Trent, for helping make Wells Hall a place I enjoyed going into. I also want to thank my brother, Aaron, and my parents, Alan and Barbara, for supporting me during the last six or seven years, and throughout my entire Michigan State University iv experience. Also, I really need to thank Tawny, my former Dollar Tree supervisor and longtime friend, who helped me grow and develop before I even started my graduate school journey. I feel so lucky to have always been surrounded by such wonderful people in each chapter of my life. Lastly, I want to thank the wonderful people I am now working with, all of the staff at the Michigan Judicial Institute, for welcoming me into my post-graduate school life and for being so supportive during that transition. If leaving graduate school means ending one chapter of your life and beginning another, they all helped show me that the chapter I am now starting can be just as enjoyable and rewarding. v TABLE OF CONTENTS CHAPTER 1: INTRODUCTION ...................................................................................................1 CHAPTER 2: LITERATURE REVIEW .........................................................................................4 CHAPTER 3: METHODS .............................................................................................................25 CHAPTER 4: RESULTS ...............................................................................................................62 CHAPTER 5: DISCUSSION .......................................................................................................106 CHAPTER 6: CONCLUSION ....................................................................................................126 REFERENCES ............................................................................................................................139 APPENDIX A: HYLAND AND MILTON’S (1997) LEXICAL ITEMS ................................... 147 APPENDIX B: RECRUITMENT SURVEYS ............................................................................. 148 APPENDIX C: PRE- AND POST-INSTRUCTION WRITING PROMPTS .............................. 149 APPENDIX D: ACTIVITY #1 .................................................................................................... 150 APPENDIX E: ACTIVITY #2 FOR THE DIRECT DDL GROUP ............................................ 152 APPENDIX F: ACTIVITY #2 FOR THE INDIRECT DDL GROUP ........................................ 153 APPENDIX G: ACTIVITY #3 FOR THE DIRECT DDL GROUP ............................................ 154 APPENDIX H: ACTIVITY #3 FOR THE INDIRECT DDL GROUP ........................................ 155 APPENDIX I: ACTIVITY #4 FOR THE DIRECT DDL GROUP ............................................. 157 APPENDIX J: ACTIVITY #4 FOR THE INDIRECT DDL GROUP ......................................... 158 APPENDIX K: LEARNER PERCEPTION QUESTIONNAIRES ............................................. 159 APPENDIX L: STANDARD INTERVIEW QUESTIONS ......................................................... 161 vi CHAPTER 1: INTRODUCTION In this chapter, I introduce the current experimental study, in which I examined how and to what extent direct (i.e., hands-on corpus use) or indirect (i.e., hands-off corpus materials) online data-driven learning (DDL) modules were able to assist Chinese writers from an English as a foreign language (EFL) context in using more appropriate hedging (i.e., in patterns and frequencies closer to native English speaking (NES) writers) in their academic English. Additionally, I also assessed the participants’ perspectives concerning any potential difficulties, challenges, or benefits they perceived in engaging with the online language corpora and online DDL modules provided to them. The current study’s design is a partial replication of Sun and Hu’s (2020) recent study regarding direct and indirect DDL instruction for hedging. In their study, they similarly examined the effectiveness of a direct and an indirect approach to DDL in improving Chinese L2 learners' use of hedging by offering students several weeks of instructional activities. Students’ pre- and post- instruction essays were examined for the frequencies and varieties of hedges used. As detailed further in the Methods chapter (Chapter 3), the current study adopted several of their methods, though with several substantial alterations which will be just briefly summarized here. First, the traditional face-to-face instruction of Sun and Hu’s study was adapted to a computer-assisted language learning (CALL) context in which all instruction and activities were hosted on online modules created with an online learning management system (i.e., Canvas). This allowed for a comparability between the two studies to see whether Sun and Hu’s findings could be replicated when the face-to-face classroom context was replaced with CALL modules for DDL. This also contributes to what Crosthwaite and Steeples (2022) note as the more limited research which examines DDL in modalities outside of the traditional classroom, such as online 1 or blended contexts. Secondly, select participants in the current study were interviewed, with the aid of stimulated recalls, post-instruction. All participants were also given Likert-scale and open- ended survey questions after the instructional activities in which they were able to reflect on their thoughts regarding the online modules and corpus tools. This allowed the current study, when contrasted with Sun and Hu’s, to gain a larger quantity of student perceptions and insight on the challenges, difficulties, and benefits they encountered when engaging with the online language corpora and online modules. Thirdly, as explained further in the Methods chapter, the instructional activities developed for the current study were informed by previous corpus research on Chinese L2 learners’ hedging patterns. This allowed the material and activities of my study, unlike that of Sun and Hu’s, to be based on corpus research which points to where Chinese L2 learners are shown to struggle with hedging regarding specific lexical items or hedging grammatical categories. However, though the current study was informed by corpus research regarding a very specific learner English population (i.e., Chinese), it remains generalizable in how it can possibly demonstrate the process of developing online DDL modules that are informed by corpus research to researchers and practitioners. Additionally, students’ perceptions regarding the challenges and benefits they saw in interacting with the online DDL modules and corpora, as well as their suggestions, may help highlight various factors (e.g., technology concerns, training issues) that practitioners should consider when implementing online DDL in an instructional context or when developing online instructional modules. This dissertation is structured as follows: In Chapter 2, I review the relevant literature regarding some key concepts including, for instance, patterns of hedging by Chinese EFL writers and different approaches to DDL; in Chapter 3, I walk through the procedure, materials, and data analysis methods utilized within the current study; in Chapter 4, I provide the results of the 2 quantitative and qualitative analyses; in Chapter 5, I discuss the relevance of the results and draw implications from them; in Chapter 6, I conclude with some limitations of the current study, further implications from the findings, and recommendations for future similar research. 3 CHAPTER 2: LITERATURE REVIEW In this chapter, I offer an overview of previous literature on several topics, including: the functions and importance of hedging in academic English; patterns of hedging that have been found to be characteristic of Chinese EFL writers; the effectiveness of direct and indirect approaches to DDL; and literature that has examined students’ responses to engaging with DDL instruction. Also, I provide a theoretical rationale for research which, as in the current study, uses online DDL practices and evaluates learners’ user experiences as they engage with that online instruction. The chapter ends with the research questions which motivated this study. 2.1. What is the relevance of hedges for written academic English? Lakoff (1972), who made significant contributions to the understanding and analysis of hedges, described the linguistic devices as “words or phrases whose job is to make things more or less fuzzy” (p. 471). Since then, researchers have contributed their own definitions to the concept, which has included defining hedges as devices used to weaken the strength of a declaration (Zuck & Zuck, 1986) or as devices used to say less than what you mean to say (Markkanen & Schröder, 1997). In the simplest terms, hedges are used to express varying degrees of doubt or certainty (Hyland & Milton, 1997; Hyland, 2005), and a linguistic means to demonstrate a lack of complete commitment to a declaration (Hyland, 1998). Hedging is prevalent both in academic discourse and everyday casual encounters and involves the use of a variety of lexical items (e.g., might, I think) to hedge a claim or statement. In casual conversation, for instance, hedging represents an interpersonal and communicative resource for speakers (Hyland, 1998), one which can allow them to use hedged statements to express tentativeness or intentional vagueness in a way that makes statements more acceptable to a listener (Salager-Meyer, 1994). The purposes that hedges serve for academic writing are similar 4 in that hedges can allow writers to “reduce the potential threat” that their claims may cause others, while also allowing students to engage in the “diplomatic negotiation” needed to “achieve acceptance from [an] audience” in an academic community (Khamesian, 2015, p. 183). Hedging thus allows students to demonstrate their awareness of academic discourse conventions when presenting an opinion or claim (Afshar et al., 2014). Researchers have additionally noted that the ability of hedges to express varying levels of doubt or certainty is especially important to argumentative essay writing (Hyland & Milton, 1997; Sun & Hu, 2020), the academic writing genre examined in this study. This is because it is a genre in which students are asked to give their opinion on a prompt, offer claims to support it, and evaluate evidence (Hinkel, 2005; Sun & Hu, 2020). Further, frequent hedging use has been found to be an important component of effective persuasive writing (Connor & Lauer, 1988), the writing style that students need to use as they try to convince their readers of the validity behind their position or argument. While doing so, writers are required to evaluate the certainty of their statements and suggest to readers whether the information they are providing is tentative (i.e., opinion) or definite (i.e., fact) (Hyland & Milton, 1997), something they can achieve with the assistance of hedges. Hedging can, as Karunarathna (2000) noted, give student writers the resources to discuss both what they know and what they think with suitable levels of certainty, which is an essential balance in argumentative essays where both objective and subjective claims are often made side by side. Using hedges is thus crucial for students to manage their own positions in such essays as they argue for a point of view or an opinion with a tone appropriate for the level of existing knowledge concerning that viewpoint (Salager-Meyer, 1994). They also allow writers to provide proper framing when introducing supporting evidence, depending on how definitive the evidence is (e.g., this evidence shows something vs. this evidence may show 5 something). Rezanejad et al. (2015) similarly noted that a writer’s use of hedges allows their readers to make their own judgements on the degree of certainty or doubt behind a declaration, resulting in a negotiation between reader and writer that is crucial to successful argumentative essay writing. Failing to make use of appropriate hedging can have a negative impact on the perceived quality of L2 learners’ argumentative essay writing. Previous findings have, for instance, demonstrated that frequent hedging is associated with higher perceived essay quality (Crossley et al., 2014; Hinkel, 2005) and that “the inclusion of more hedging devices has been found to be a trait of stronger writing competence and higher scoring essays” (Sun & Hu, 2020, p. 5) in testing situations. This demonstrates the importance for L2 writers to use appropriate hedging in their argumentative essays, as it is a genre of academic writing that English learners encounter often in their academic or professional lives and the kind of writing task that many English learners must engage in when taking standardized English tests to advance their education or careers (e.g., the IELTS academic test). It is important to note, though, that students’ use of hedging in an argumentative essay (e.g., an IELTS essay) is likely to differ greatly from their use of hedging in other academic writing contexts, such as an academic research paper. This can be for a variety of reasons, including that research writing involves students’ navigating their role in an academic community populated by fellow researchers (Khamesian, 2015), and because argumentative essay writing often involves a greater reliance on subjectivity and opinion due to a lack of access to outside sources. Therefore, all texts examined in the current study, as well as in the previous corpus work that this study is informed by, are student essays of a comparable nature and composition, so that students’ hedges in one context (i.e., argumentative essay writing) are not 6 being compared to their hedges in an entirely different context (i.e., research paper writing). Further, argumentative essays are used here because they require writers to take a stance on an issue and control the amount of strength behind their claims, something often achieved with the use of hedges (Min et al., 2019), and because it is a genre that students engage with frequently in their professional pursuits (e.g., standardized English exams) or university-level writing. 2.2. What patterns of hedging are characteristic of Chinese L2 learners? Using hedges has been largely shown to be a difficult task for Asian L2 writers from specific L1 backgrounds, such as Chinese writers (Connor & Lauer, 1988), who can often produce work that seems to be overstated or contain many exaggerated claims due to a lack of hedging (Hinkel, 2005). Unqualified, assertive, writing has been found to be more characteristic of certain Asian L2 writing as opposed to NES writing, broadly speaking (Bloor & Bloor, 1991), though especially in work by Chinese L2 writers (Brown & Brown, 1982; Hu et al., 1982; Allison, 1995; Hinkel, 2005). In fact, Chinese L2 writers have often been shown to significantly under-use hedges in academic English compared to both NES writers as well as Asian L2 writers from several other L1s (e.g., Japanese, Korean), with their work having been found to often feature a surplus of direct and assertive statements due a lack of appropriate hedging (Hinkel, 2005). Sun and Hu (2020), for instance, in their summary of the literature on Chinese L2 writers’ hedging, note that they often are found to use a narrow set of lexico-grammatical choices to convey a lack of certainty, or a probability, of a statement. Further, Chen (2010) used a corpus-based approach to examine the hedges in student essays written by NES and Chinese L2 writers. The findings showed that the Chinese learners “employ[ed] significantly fewer “downtoners” (i.e., might, would, possible) ...compared with the native speakers in academic writing” (p. 42). Hu and Cao (2011), meanwhile, compared hedges 7 in abstracts of applied linguistics articles based on a corpus of 649 published abstracts from eight journals, and similarly found that the abstracts written for the English-medium journals featured significantly more hedges than those written for the Chinese-medium journals. Wu (2022) examined the use of hedges in economic papers between two corpora: those written by NES writers and those by Chinese writers. Their study found that considerable variations in the overall number and frequency of hedges, with the Chinese writers employing significantly fewer hedges, including a notable decrease with tentative plausibility shields (e.g., I think, I believe). The authors suggest that Chinese authors’ lack of familiarity with hedges and linguistic transfer from their L1 likely played a role in explaining these results. Winardi (2009), in another examination of applied linguistics articles written by American and Chinese L2 authors, found that, while the overall frequencies of hedges were relatively similar, there were differences in the types of hedges that each employed; namely, the Chinese writers utilized significantly less plausibility shields, those used to convey the speaker’s uncertainty or doubt about their statements. In Yang (2013), it is similarly noted that "the Chinese authors used more convictions than their counterparts and seemed to be more assertive in their statements" (p. 5). Yang (2013) examined the use of hedges across three academic writing corpora: those written by NES, those written by Chinese L1 writers in an English L2, and those written by Chinese L1 writers in Chinese. They adopted Hyland's (1998) classification of hedges to analyze the frequencies of their various types. They found considerable variations in the overall number and frequency of hedges, as well as the types of hedges across the three corpora. Namely, significantly higher frequencies of hedges used in the NES corpus (n=1474 raw hedges, 19.7 per 1000 words) compared to the Chinese L2 corpus (n=353, 9.0 per 100 words) and the Chinese L1 corpus (n=486, 11.0 per 100 words), especially concerning more tentative 8 plausibility-style hedges. Yang (2003) similarly found that the Chinese L2 writers included in the study used a lower frequency of plausibility shield hedges in favor of stronger hedges and more assertive writing. Wang (2008) also examined hedging shields, focused on medical journal articles written by Chinese L2 and NES writers, and found that NES authors used a greater variety of plausibility shield hedges within their articles compared to their Chinese L2 counterparts, which again are those hedges (e.g., I think, I believe) used to indicate uncertainty from a speaker towards the content of their statement. There are several possible reasons that Chinese L2 writers do not hedge their statements in patterns similar to NES writers in their academic writing. Allison (1995) noted that this could potentially be due to inadequate lexico-grammatical resources, a sentiment echoed by Sun and Hu (2020). Hyland and Milton (1997) additionally note that it could be the result of sociopragmatic violations in which Chinese L2 writers "fail to distinguish between the conventions of different text types...[and] actually carry an inappropriate degree of directness for an academic register" (p. 194). Negative linguistic transfer from the Chinese L2 writers’ L1 has also been suggested as a potential cause. Milton and Hyland (1999), in a comparative study of the hedges used by L2 and NES writers, found Chinese L2 writers to use significantly more inappropriately direct and unqualified assertions compared to NES writers. They note that this can possibly be attributed to hedging patterns in Chinese learners’ L1, which may then lead to them being “observed to be especially inappropriately authoritative and [make] unjustifiably strong assertions” (p. 56) in their L2. Hinkel (2005) also noted that, in Chinese L1 rhetoric, exaggerated statements can be viewed as a way to increase the persuasiveness of an argument and is often considered an effective rhetorical technique, which may partially explain why there is a prevalence of it in their L2. 9 The differences in L2 hedging use between Chinese EFL writers and writers from other Asian L1 backgrounds demonstrates that it is important to note that Asian languages and cultures are not monolithic. Indeed, EFL learners from specific L1s, “with their own [linguistic] characteristics and culture-bound” traits, are likely to use hedges “differently from other [Asian EFL population] writers” (Tran & Tang, 2022, p. 120). Research into the hedges used by Asian EFL writers from other L1 backgrounds, for instance, has produced varied results. Japanese EFL writers, for instance, have been shown to oftentimes over-use hedging or softening devices, even compared to NES writers (Kamimura & Oi, 1998; Lida, 2008; Hinkel, 2002; Bernaisch, 2020). While it was mentioned that assertive and direct statements are often characteristic of Chinese L1 rhetoric (Hinkel, 2005), Hinkel (2002) notes that hedges and softening devices are a very common feature of Japanese rhetoric as a way to minimize potential differences of opinions or for face-saving. Kamumura and Oi (1998) also note that Japanese rhetoric shows high frequencies of “softening devices in diction…[which] come from Japanese “high-context” culture” (p. 318). Results of studies which have compared the hedges used by NES writers with Korean L2 writers (Choi & Ko, 2005), Vietnamese L2 writers (Thuy, 2018), and Asian writers from other L1s, have all produced varied results. However, there are also studies which compare hedging frequencies between NES writers and NNES writers from various languages being lumped together (e.g., NES writers compared to Asian L1 writers more broadly). These studies present a methodological issue, which is also highlighted by the differences in all of these findings concerning specific Asian L1s. The issue is that frequencies or patterns of hedging can be context specific to different EFL populations (Tran & Tang, 2022) and that, “due to the nature of interlanguage development of L2 learners, it is evident that the learners’ native language and native culture would have an influence” on their 10 hedging in their L2 (Chen, 2010, p. 30). That is, Asian L1s and cultures are not monolithic, and comparisons of a linguistic feature, such as hedges, should be conducted between specific languages as opposed to grouping multiple languages together, given that each language will have its own unique characteristics and surrounding culture. Thus, the current study examined hedges used exclusively by Chinese L2 writers instead of blending multiple Asian EFL writers together, so as to be able to create DDL instruction which targeted hedges that Chinese L2 writers in specific have been shown to struggle with using in patterns similar to NES writers. In the Methods chapter (Chapter 3) of this dissertation, I detail the previous corpus research which informed the current study, and which illustrates the specific lexical items or grammatical categories of hedges that Chinese EFL writers were shown to use in patterns significantly different from NES writers, as well as Asian L2 writers from other L1 backgrounds. Another limitation of many of these previous studies is that their analysis often relies on overall frequencies, or frequencies per 100 or 1000 words, for each corpus. These numbers allow for a simple examination of the differences in the overall frequencies or distribution of hedges across the corpora, and there are statistical tests that can be used to determine significance based on frequencies of occurrence and corpus size (e.g., log likelihood tests). However, this information alone does not allow for a more thorough examination which considers an analysis of the data with multiple variables that can affect the outcome, in the case where some variables might be fixed (i.e., consistent across observations, preset by the researcher) and others may be random (i.e., vary randomly across observations, not controlled by the researcher). Mixed-effects modeling is one such analysis method that can take these random factors into account, while also allowing for an examination of variation between observations of the individual (Brown, 2021), such as the frequency and distribution of hedging for each individual at each time point. The 11 current study therefore utilizes this method of analysis, described in detail in the Results chapter, to offer a more robust examination of the participants' hedging use, which will allow the results here to be more thorough than that of many previous hedging corpus studies. 2.3. DDL: Theoretical rationale Because these corpus findings were used to inform the construction of online DDL instructional activities, I offer here a brief theoretical rationale for a DDL approach, under which learners explore corpora, or engage with corpus-informed material, to detect linguistic patterns.. Specifically, a DDL approach is supported by both the constructivist theory for learning (Boulton, 2010) as well as the usage-based learning perspective. O'Keeffe (2021) notes that constructivism offers support for the use of corpus materials in the classroom, and that there are numerous “constructivist ideals in DDL” (p. 6). The constructivist theory of learning argues that learning involves active construction, by which learners actively build their own knowledge and take a central role in their own learning processes. Under this theory, the mental structures that learners create are built, to some degree, by the learner as they interact with language and the context in which it's used (Flowerdew, 2015). The constructivist theory of learning therefore emphasizes the active role of learners in constructing their own understanding, through experiencing things and reflecting on those experiences. Learning is thus not a passive process, but an active and dynamic process where knowledge construction is a key element. It is also therefore a theory of learning, given that it offers a theoretical explanation for how people learn and suggests strategies that can make learning more effective, based on its principles (e.g., experiencing large amounts of data and actively constructing knowledge from it). 12 Quite similarly, one of the central tenets to DDL is its potential to motivate learners to construct their own knowledge (i.e., learner autonomy) by independently exploring corpora or linguistic data and noticing phenomena within it (O’Keeffe, 2021; Cobb & Boulton, 2015). O’Keeffe notes that this link is why “constructivism is seen as a pedagogical hallmark for DDL” (p. 7). O'Sullivan (2007) further notes that the ability of DDL to improve learners' cognitive skills (e.g., prediction, observation, noticing) is based on constructivist learning tenets where learners often rely on their own cognitive skills in free-range exploration. Corino and Onesti (2019) additionally note that DDL's focus on learner autonomy, the use of authentic language materials, and the development of metalinguistic knowledge, all align DDL with constructivist and learner-centered theories of language learning. Therefore, effective DDL instruction aligns itself closely with several of the strategies and principles highlighted by the constructivist theory of learning; namely, knowledge construction (i.e., as learners sift through large amounts of data), learning as an active process (i.e., learning is dynamic and actively changes as learners notice frequency patterns in the data), and based on reflection (i.e., learners reflect on patterns or phenomena that notice within the data). Similarly, the usage-based approach to language supports many of these same DDL tenets. The usage-based approach emphasizes the role of language use in developing language structures and highlights the role of usage patterns in learning language (Ellis & Wulff, 2014). That is, its focus is on exposure to language and experience with language constructions in context rather than, for instance, innate knowledge or isolated explicit instruction of linguistic rules. DDL, and corpus-based instruction, often involves learners independently searching for, and being exposed to, specific constructions or lexical items in specific authentic contexts, rather than being explicitly instructed on the “rules” of those constructions (O’Sullivan, 2007). Geluso 13 and Yamahuchi (2014) note that, in usage-based approaches, frequency is a key concept, as learners will develop language structures based on their frequent and repeated exposure to constructions within a certain context. Vyatkina (2016) additionally notes that "DDL is fully compatible with usage-based second language acquisition (SLA) theories that conceive of language as an open-ended dynamic system" (p. 6) because students’ learning emerges as a result of the construction of meaning, social interaction, and cognitive processes, that occur as students are repeatedly exposed to language use data (i.e., DDL). 2.4. User experience research: Theoretical rationale I also offer here a brief theoretical rationale for user experience (UX) research which, as in the current study, examines users’ (e.g., learners’) experiences as they interact with some sort of system (e.g., online modules, online corpora). Such research is often supported by the Usability Theory. Peterson (2007) notes that the underlying concept of the Usability Theory is to make technology compatible with users, or "that the things people use to complete a task should not hinder the completion of that task by the nature of its design" (p. 337). In other words, Usability Theory examines the extent to which a design (e.g., of online modules, online corpora) can be used by certain users (e.g., student learners) to achieve specific goals in an intended way (e.g., activities using those modules or corpora). Usability Theory allows researchers, for instance, to examine if there are difficulties or challenges with some platform, such as an online interface, which would make it difficult for a user to successfully navigate or engage with that platform to complete a task. Lohr (2000) additionally notes that, as access to educational technology tools has increased, and as CALL instruction in classrooms has become more frequent, the relationship between the digital interface and the student, or learner, has become more crucial for educational researchers to address. 14 While the concept of the Usability Theory, on its surface and as briefly described here, may seem simple, researchers have defined it as a theoretical framework in various ways. Peterson (2007, p. 339) notes that researchers frequently cite the International Organization for Standardization, an organization that develops international standards for various industries and sectors, when defining the concept of the Usability Theory. They define it as a theoretical concept that can drive research which examines “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use." Nielson (1993), meanwhile, offers a more expansive definition, in which he argues that the Usability Theory includes five components: (i) learnability, or how easy a program is to learn; (ii) efficiency, or how easily users are able to use a program once it has been learned; (iii) memorability, or how well the user is able to remember how to do certain things with a program; (iv) errors, or how good the program is at reducing navigation errors made by users; and (v) satisfaction, or how pleasant the program is to use for users. The Usability Theory differs from another central tenet of UX, User-Centered Design Theory (UCT), in that UCT refers to involving users in the process of developing something (i.e., the design is centered around the user). In UCT, users, and their feedback, plays a central role in the active process of developing a product, prototyping it, revising it, and so on down the course of development (Endsley et al., 2003). UCT is more focused on improving a specific product along its development, using user experiences as a source of feedback. Because the current research offered a completed product to users (i.e., finished online modules), and was primarily interested in assessing the components that Nielson (1993) highlights, as mentioned above (e.g., how well learners were able to use the modules and corpora to accomplish activities, 15 how pleasant the tools were for learners), the Usability Theory was the more applicable UX theory to motivate the current research. Specifically, the current study incorporated UX and Usability Theory principles in how it sought to evaluate learners’ experiences of engaging with the online modules developed for this study, and the online corpora that learners had to engage with to complete their tasks. The ability of learners to successfully complete the activities using those modules and corpora with ease, efficiency, and satisfaction, was a crucial focus of this research. This is because the students’ thoughts and experiences highlighted in the current study can possibly help point to factors or issues that are important for future practitioners or researchers to consider when developing online DDL instruction. 2.5. Online asynchronous learning: Theoretical rationale Increasingly, and especially since the COVID-19 pandemic, there have been significantly expanded opportunities for language learning beyond the traditional face-to-face classroom setting. This shift includes online distance learning, which can be delivered either synchronously or asynchronously. Each delivery method has distinct characteristics and considerations. A brief overview of key principles and theoretical underpinnings of asynchronous online learning, which is the mode of delivery employed in the present study, is offered here. Asynchronous online learning, as Riwayatiningsih and Sulistyani (2020) note, has been defined as a form of education and instruction which does not occur in the same place or at the same time. Blake (2009) further elaborates that "much learning still occurs when individuals are alone, working by themselves in isolation at odd times and places" (p. 829). This approach allows students to access educational content at their convenience, providing flexibility in when they engage with course materials, which can be done remotely from their devices. Lamy (2013) 16 additionally notes the "specific concerns related to distance, openness, flexibility and support for learners" (p. 155) that makes asynchronous online distance learning different than hybrid learning or synchronous computer-assisted learning. Researchers have sought to identify the factors contributing to successful asynchronous online learning. Swan (2002) identifies three key elements: a consistent and clear course structure, an engaged and attentive instructor or facilitator, and tasks that promote critical thinking and authenticity. Beck (2004) emphasizes additional crucial characteristics, including access and student motivation, online socialization, knowledge construction, information exchange, and an instructor who designs an online learning environment that effectively facilitates student learning. Researchers have identified several additional benefits and drawbacks of online asynchronous learning modules compared to synchronous online learning and traditional face-to- face classrooms, primarily related to flexibility and accessibility (e.g., Crocco & Culasso, 2021; Lei & Lei, 2019; Gamorot et al., 2022). Asynchronous online learning allows learners to access course materials at any time, enabling them to balance their studies with other personal or professional commitments. However, this flexibility may also lead to a lack of immediacy in real-time interactions, such as spontaneous clarifications or immediate feedback, potentially causing students to feel isolated from their instructors and peers (Croft et al., 2010). Online asynchronous learning modules also can promote self-directed learning, encouraging students to develop critical skills like time management and to engage deeply with the material at their own pace. This approach supports differentiated learning, allowing students to tailor their study habits or engagement with the content to their individual needs (White, 2014). However, Hampel and Stickler (2012) emphasize that explicit training must be provided to enable both students and teachers to effectively use asynchronous online learning platforms (p. 135). 17 Regarding a theoretical rationale for asynchronous learning and its defining characteristics, constructivist theory, as discussed in sub-header 2.3, provides a foundation for the use of asynchronous online learning modules, such as those utilized in the current study. According to constructivist principles, learning is an active and contextualized process of constructing knowledge rather than passively receiving information (O’Keeffe, 2021). Asynchronous online modules align with this approach by allowing learners to engage with course content in a self-paced manner that suits their individual learning goals. This flexibility enables students to explore content deeply at their own pace, reflect on their experiences as they interact with the material, and connect new information to their existing knowledge base (O'Sullivan, 2007). Asynchronous learning allows students to navigate course content freely, either by delving into it extensively or by sequencing it in a way that improves their understanding, which might be less feasible in a more structured and guided face-to-face classroom environment. These autonomous interactions with the material help students negotiate meanings, test their ideas or preconceptions, build on their previous life experiences, and gain multiple perspectives, all of which enrich their learning (Can, 2009; Emmerson, 2019). Furthermore, the flexibility of asynchronous modules allows learners to revisit discussions and reflect on activities or feedback over time, promoting continuous learning. These aspects are integral to constructivist theory, which emphasizes learner-centered and reflective learning. 2.6. Direct and indirect approaches to DDL In this section, I briefly summarize some relevant DDL literature, including literature which, as in the current study, distinguishes between different DDL approaches. I start, however, by more broadly synthesizing research regarding the purported benefits of DDL for language learners, as well research that has examined learners’ perceptions of engaging with DDL. 18 Crosthwaite and Steeples (2022) note that, in addition to the increasing use of language corpora within language teaching and learning contexts (that is, DDL), there has been increasing recent research which has pointed to how incorporating corpus materials can aid L2 learners with their academic L2 writing. Such incorporation not only allows instructors more flexibility with designing courses as well as classroom content (Vyatkina, 2016), but also raises learners' awareness about language use in authentic contexts (Sun & Hu, 2020). O’Sullivan (2007) notes, further, that DDL can help students to become more independent learners, and can improve many cognitive abilities of students, including, among other things, their ability to predict, observe, notice, reason, think, interpret, reflect, and guess. These benefits are often considered to result from "guiding learners through a series of concordance-based activities...familiarizing them with various types of investigations...and stimulating...appropriate learning strategies through practice” (Kennedy & Miceli, 2001, p. 8). In other words, stimulating self-directed exploration and learning as learners construct knowledge rather than consume it (i.e., constructivist approach to learning, described in Section 2.3). The peer- or instructor- interactions that can occur during these corpus-based and inquiry-based tasks also can contribute to the construction of knowledge and the benefits of the DDL (Crosthwaite & Steeples, 2022). This is why, as research has noted, most instructors often take a guided approach to DDL in which the corpus work with students is often first approached as an apprenticeship by which the students learn by example and by their experience of working with the instructor (Kennedy & Miceli, 2001; Tribble & Jones, 1997; Stevens, 1991). Instructors can, for instance, lead activities to help aid the students in navigating the corpus and can also highlight specific examples from the corpus or walk students through the large quantities of online texts. Eventually, over time, students are encouraged to use the corpus on their own with reduced 19 intervention from the instructor, with students finally able to browse the corpus freely to complete a pre-determined learning outcome. Additionally, in studies in which learners’ perceptions were collected regarding the incorporation of corpora into their learning contexts, many participants have been found to react positively. This includes, for instance, Geluso and Yamaguchi's (2014) implementation of DDL into an EFL curriculum with the aim of improving fluency in L2 speaking. In their study, they also assessed students' perceptions and attitudes towards the DDL instruction, finding that a majority of students found the instruction to be useful and beneficial, despite technical issues (e.g., cut-off concordance lines) and pedagogical issues (e.g., unfamiliar vocabulary). The benefits of corpus informed DDL that has been found in previous research (e.g., increased vocabulary gains, exposure to authentic texts), as well as the positive reactions from students towards the DDL instruction (e.g., Bernardini, 2004; Crosthwaite, 2017; Sun & Hu, 2020), has also been found when that DDL instruction came via an online or blended modality. This includes research such as Crosthwaite's (2020) work on designing and evaluating a short private online course on DDL for tertiary L2 writing, as well as Mishan's (2013) examination of DDL in the context of blended learning (i.e., the mix of online and classroom-based activities). However, despite the benefits and favorable attitudes from students, such studies which have examined students’ perceptions have also highlighted some challenges or difficulties that students can face with DDL. This is especially true in online or blended DDL, in which there is often a greater “perceived complexity of its implementation” (Sun & Hu, 2020, p. 2), and in which students must navigate the corpus data itself (e.g., the concordance lines, the texts) as well as online interfaces and online learning contexts (e.g., learning management systems, online modules). Researchers have noted that technological problems are often the major source of 20 issues concerning online DDL instruction (Yoon & Hirvela, 2004; Boulton, 2010), with Boulton noting that “effective hands-on DDL requires considerable training” when in an online modality (p. 3). Kennedy and Miceli's (2001) study, for instance, which created tasks informed by the Contemporary Written Italian Corpus for intermediate-level students, found that, while the students felt overall positively towards the corpus incorporation, many faced difficulties in interacting with the online corpus. Kennedy and Miceli noted that inadequate training was likely one of the primary reasons for the difficulty, and remark that, in effective corpus based DDL, instructors need to first have "adequately equipped them [the students] as "corpus researchers" (Kennedy & Miceli, 2001, p. 81). This remark points to the need of students to be able to sufficiently navigate the online corpus so that they are able to seek the information they need to complete a task or solve a problem, and this is a concern that goes back as far as Johns (1988), who noted that students needed to be able to develop data-extraction and navigating strategies so that, eventually, “we simply provide the evidence needed to answer the learner's questions, and rely on the learner's intelligence to find answers” in the data (Johns, 1988, p. 2). This is especially true in online contexts in which, as Boulton (2010) notes, students will face more success from the DDL if they are already familiar with the existing technology and searching techniques involved in interacting with the interface and data. That is, the difficulty students can have in navigating language data or concordance lines can be compounded in online (or blended) contexts, in which students also need to learn how to engage with the interface or functions of online corpus tools (as well as any online platforms that may be hosting them or delivering them to students). 21 Crosthwaite and Steeples’ (2022) examination of a half-year DDL experiment on the development of the passive voice in young learners also illustrates this point in their findings, which showed that several students had difficulty in understanding the steps needed to use the online corpora, how to navigate to certain components or tools, what the functions of certain tools were, and also an overall confusion regarding the online interface about what they were supposed to be doing. The current study, which itself provided corpus informed DDL to students using online modules, as illustrated in the Methods chapter (Chapter 3), utilized many of the same methods (i.e., surveys or post-instruction questionnaires) as such studies to collect students’ thoughts and perceptions regarding the difficulties or challenges they encountered when engaging with the online corpora or online modules. In addition, the current study also was able to collect larger quantities of feedback regarding the online modules and corpora by interviewing select participants. The current study also contributes to a gap in the existing DDL literature, specifically regarding the differentiation between two different approaches to DDL: direct and indirect. Sun and Hu (2020) noted that “most existing studies have compared some form of DDL with traditional teaching but have not examined or compared different approaches to DDL” (p. 2). That is, there are studies which have examined students’ perceptions, and the impact on their learning, of having students directly engage with corpus data and conduct their own searches on the corpora (i.e., direct DDL) or research which has examined much of the same factors when students instead simply engage with corpus-informed, teacher-led, materials with no direct corpus use (i.e., indirect DDL). There is much more limited research, however, which has examined multiple DDL approaches together, resulting in what Chen and Flowerdew (2018) remark as “a need for more descriptions of different [DDL] approaches” (p. 356). A synthesis 22 article by Boulton (2008), which examined 39 DDL research articles, similarly found that only eight of them identified an indirect DDL approach as a possible alternative to direct DDL. Boulton (2010) notes that these findings may lead some to assume that only direct DDL, with its "considerable training....and technological considerations" (p. 4), is useful for learners, which may cause trepidation by instructors, especially those with limited classroom access to computers or laptops. This concern was echoed by Sun and Hu (2020), who also noted that the lack of research into different DDL approaches may give the false impression that direct DDL is the only appropriate method of DDL, which can result in teachers fearing DDL as too intimidating, complex, or time consuming to implement in their classrooms. This gap also results in limited research regarding the effectiveness of the two approaches when used in comparable contexts (i.e., similar methods and students). Sun and Hu’s study, for instance, was a rare study that used both a direct and indirect DDL approach on Chinese EFL learners’ hedges and found that the indirect DDL treatment was reliably more effective than the direct DDL treatment at increasing the frequency and variety of hedges used. The current study, as mentioned, adopted several of the methods used within Sun and Hu’s study, and similarly examined the effectiveness of both approaches when used in a comparable context. Direct DDL is thus defined here as DDL in which learners directly access corpora to carry out their own searches to complete a task or activity, with indirect DDL defined as that in which learners simply engage with corpus-informed activities prepared by an instructor in advance (Boulton, 2017). 2.7. Research questions The research questions for the current study are the following: 23 RQ #1: Is online DDL instruction effective in improving Chinese L2 learners’ use of hedges? RQ #2: Do a direct and an indirect approach to online DDL instruction differ in their effectiveness? RQ #3: How do learners perceive the incorporation of online DDL instruction, and the use of either online corpora or corpus-informed materials? RQ #4: What are students’ user experiences as they engage with the online DDL modules and online corpora? As far as how effectiveness is being operationalized, a higher frequency of hedging alone does not guarantee more effective hedging—as it is possible to use more hedges in a manner inconsistent with NES norms. However, previous research, as well as the corpus analysis that this study is informed by, have consistently shown that Chinese EFL writers typically use significantly fewer hedges than NES writers and writers from other Asian L1 backgrounds. Therefore, an observed increase in hedging among the participants likely reflects a movement toward the hedging patterns characteristic of NES writers, suggesting a potential improvement in hedging effectiveness. 24 CHAPTER 3: METHODS This chapter describes the materials, procedure, and participants, utilized in the current study. Additionally, I describe the previous corpus-based research which informed the instructional activities constructed in the current research. The overall methods, though all discussed in greater detail below, are briefly summarized here. Online DDL modules for hedging in academic writing were constructed. Student participants, (n=46), all Chinese EFL students, were placed into one of two groups: the direct DDL instruction group (n=23), or the indirect DDL instruction group (n=23). Participants in both groups then completed a series of writing tasks and instructional activities over a four-week period, all hosted on the Canvas learning management system. Additionally, participants in the direct treatment group recorded their computer screens (i.e., screen capturing). The activities for the direct DDL group involved hands-on corpus activities, while the activities for the indirect DDL group involved corpus-informed activities in which participants did not directly interact with online corpora. After all of the instructional activities, participants completed a learner perception survey, and select participants were interviewed along with the aid of a stimulated recall procedure using their screen recordings. 3.1. Substantial changes made to Sun and Hu’s (2020) design As mentioned in the Introduction, this study adopted several of Sun and Hu’s (2020) methods, though with three substantial alterations. While those changes were highlighted earlier, they are again provided here in Table 1. In specific, Table 1 offers some design choices from within Sun and Hu’s original study, some adaptation made for the current study, and the rationale for those changes. 25 Table 1 Changes made to Sun and Hu’s (2020) design Sun and Hu (2020) Traditional face-to-face classroom instruction. The Current Study CALL via online modules, hosted on a learning management system (i.e., Canvas). Centered the DDL activities on a list of hedges that, according to grammar reference material, “were likely to be the easiest and the most useful to acquire…thus offering the greatest benefit for the least learning effort” (Sun & Hu, 2020, p. 12). Assessed learners’ perceptions of the corpus activities using a post- instruction survey. Centered the DDL activities around hedges, and hedging patterns, that Chinese L2 learners have been shown to struggle with, as found in previous corpus research. Assessed learners’ perceptions of the activities, online corpora, and online modules, using Likert-scale surveys, open-ended survey questions, and select interviews aided by a stimulated recall procedure (which included screen capturing). Rationale To assess whether Sun and Hu’s findings were able to be replicated in a CALL context, as well as to assess learners’ perceptions regarding the online modules in addition to the language corpora. To examine whether targeted, corpus informed, DDL instruction was able to help Chinese L2 writers’ hedging better reflect that of NES academic writing. To collect, and analyze, more substantive, and real-time, qualitative thoughts and perceptions from the participants. 3.2. Previous corpus-based research on Chinese EFL writers’ hedges As mentioned, the instructional activities in the current study were informed by corpus research I conducted concerning Chinese EFL learners’ use of hedges. This section briefly summarizes that research, both to demonstrate how it was used to influence the construction of the activities in the current study and also to further support the literature discussed earlier (Chapter 2) regarding Chinese EFL writers’ hedging patterns. Specifically, my previous corpus research examined the frequencies of hedges in student essays from six learner English populations, using a corpus-based approach, including writers 26 from the following EFL regions: China (CHN), Indonesia (INDO), Japan (JPN), Korea (KOR), Taiwan (TWN), and Thailand (THA). A comparable NES corpus was also included to compare the learner English corpora with a NES reference. 100 argumentative essays from each group, including the NES writers, were compiled, for a total corpus of 319,347 words. The student essays were all retrieved from the International Corpus Network of Asian Learners of English (ICNALE), an international learner corpus that includes more than 10,000 topic-controlled oral speeches and written essays produced by university students within ten countries in Asia, as well as a NES corpus that is comparable to the learner English essays in terms of topic, genre, and length. Additionally, the L2 participants featured in the six learner corpora were categorized into proficiency groups so that their hedging use across different proficiency levels was able to be investigated. The specific hedging lexical items included in this study, and the procedure of categorizing them, was based on the taxonomy (Appendix A) and methods proposed by Hyland and Milton (1997), which included a list of “75 of the most frequently occurring epistemic lexical items in native speaker academic writing [for hedging]” (p. 187). Findings included analysis of normalized frequencies (per 100 words) of hedges (and hedging patterns), with log likelihood statistical tests used to assess where there was significance. Normalized frequencies demonstrated that, as shown in Figure 1, among the seven corpora (the six learner corpora as well as the NES reference corpus), the CHN corpus featured the second least number of overall hedges (second to the INDO corpus). The CHN corpus featured 2.90 hedges per 100 words, which log likelihood statistical tests found to be significantly lower than the 3.79 in the NES corpus, as well as the hedges from the JPN, TWN, and THA corpora. 27 Figure 1 Normalized frequencies (per 100 words) of overall hedges across all seven corpora Additional comparative analyses examined the distribution of hedges across specific frequent lexical items. The overall patterns of the top ten frequent lexical items per corpus were relatively similar in terms of which lexical items were represented, and there was little variance within the top five lexical items for each. However, a closer analysis for several lexical items, namely will, would, and think, revealed several striking differences in frequency. Specifically, regarding the CHN corpus, Chinese L2 writers were found to significantly over-use the certainty marker will when compared to the NES corpus (.714 compared to .476 per 100 words), and to significantly under-use the more tentative would (.082 to .554) and think (.356 to .628). The finding regarding Asian EFL writers’, and in specific Chinese L2 writers’, over-use of the strong will certainty marker has been shared by previous work such as Brown and Brown (1982) and Hyland and Milton (1997). Figure 2 highlights these findings, which aligns with previous literature which has noted that Chinese L2 writers tend to use an over-abundance of certainty and 28 assertion in their writing due to a lack of appropriate hedging and caution (Sun & Hu, 2020; Hinkel, 2005). Figure 2 Normalized frequencies (per 100 words) of will, would, and think across all seven corpora Further statistical findings from this study also illustrated that Chinese L2 writers significantly under-used modal verb and lexical verb hedges compared to NES writers, as visually illustrated in Figure 3. 29 Figure 3 Normalized frequencies (per 100 words) of hedging grammatical categories across all seven corpora Across the proficiency groups examined in this study, the CHN corpus showed the most significant increased use of hedges by their advanced proficiency writers when compared with the lower proficiency group, as shown in Figure 4. Of all the proficiency groups, the basic level group of Chinese writers utilized the least number of overall hedges per 100 words. This demonstrates that lower-proficiency Chinese writers may be in special need of targeted hedging instruction, but also that there seems to be significant learning growth as they move to advanced proficiency. However, even the advanced writers in the CHN corpus had significantly fewer hedges than the NES writers, possibly pointing to the need for continued hedging instruction even as writers move to intermediate or advanced proficiencies. 30 Figure 4 Normalized frequencies (per 100 words) of overall hedges across proficiency groups Many of these findings were integrated into the current study, as the DDL materials were designed to target hedges that Chinese L2 learners were demonstrated to struggle with (explained more below), including, for instance, a higher variety of tentative modal hedges and lexical verb hedges. However, it needs to be noted that, regarding the statistical findings represented above, statistical significance was determined based only on log likelihood tests. Log likelihood tests have frequently been used in corpus comparison studies (Rayson & Garside, 2000) and are a useful way to compare the frequency of occurrences in two corpora, relative to the sizes of the corpora. However, there are some limitations to this kind of testing when compared to, for instance, mixed-effects modeling. Mixed-effects modeling allows for the ability to account for the variability in the data, to estimate the random factors' variance (i.e., variance across all participants at each time point), and overall is a more robust statistical test. Therefore, while the Chinese L2 learners’ hedging patterns identified above do align with previous literature and are 31 supported with log likelihood tests, more robust testing would be needed to confirm the significance findings. These results, thus, are simply being used in the current study to point to hedging frequencies that Chinese L2 learners may potentially be shown to struggle with, again supported also by similar previous research (see Chapter 2, sub-header 2.2.). 3.3. Materials, procedure, and data collection 3.3.1. Participants The participants in this study consisted of 46 Chinese EFL students, who currently were living in China. All participants were recruited via snowball recruiting, in which acquaintances with connections in China, or in many cases were teaching in China, were asked to spread around flyers encouraging participation (i.e., via e-mail, social media, etc.). Potential participants filled out a survey indicating interest and were contacted if they met the eligibility criteria. All participants who were invited to join the online courses were compensated $40 after they completed all of the instructional tasks, with an additional $10 compensation for those who were invited to be interviewed afterwards. As in Sun and Hu’s (2020) study, all participants were undergraduate students with upper-intermediate to advanced English proficiency skills. This was partially done for comparability between the current study and Sun and Hu’s, and also because students of such proficiencies are more likely to be in the process of studying in English or pursuing graduate school in the near future and thus have “an immediate need for focused instruction in English academic writing and [are] therefore more likely to respond toward and benefit from DDL” (Sun & Hu, 2020, p. 7). Also, Sun and Hu note that most “previous DDL studies [are] focused on postgraduate students" (p. 7), with a synthesis by Chen and Flowerdew (2018) finding that only one third of the DDL studies that they identified included undergraduate 32 students. Including undergraduate students in the current study thus helps contribute to such research. Assessing whether participants met an upper-intermediate or advanced proficiency threshold involved having participants contribute their test scores to as many standardized English tests as possible, depending on which they have taken. They also included the dates of those test scores. Participants were only invited to participate if their test scores from the last five years indicated that they met, at least, an upper-intermediate level. Though notoriously difficult to map scores from one standardized test score to another, scores from the provided tests were mapped onto the Common European Framework of Reference (CEFR) scoring system, using tools from the websites of the various standardized tests. The IELTS, TOEFL, ETS, and various other tests, for instance, all have sections of their official webpages in which they map various scores on their tests to the CEFR scoring system. All participants' scores were mapped using those tools, ensuring that they all met, at least, the B2 Upper Intermediate score on the CEFR. For the IELTS test, for example, this would map onto a score of 5.5-6.0. For the TOEFL iBT test, as another example, this would map onto a score of 72-94. This would also include participants who had passed the Test for English Majors Band 4 (TEM-4), “a compulsory and standardized proficiency test for English majors in China that is equivalent to IELTS 5.5” (Sun & Hu, 2020, p. 8). Surveys were used to collect this, and other, demographic information about the participants. Some of this demographic information is illustrated in Table 2. The participants’ majors ranged from everything including Law, Applied Linguistics, Computer Science, to Psychology, though the majority of participants were either English or English Education majors (n=25). 33 Table 2 Participants’ demographic information Demographic Information Gender Male Female Age 18-19 20-21 22-23 24-26 Year of Study First Second Third Fourth Fifth of more Experience of staying in a predominately English-speaking country No experience at all Less than one year of experience One-to-two years’ experience Two-to-three years’ experience n= 21 25 21 14 7 4 17 5 14 8 2 28 13 2 3 Additionally, two questions addressed participants’ previous experience (or exposure) to language corpus tools, as well as their general computer habits: Which of the following best reflects your experience with language corpora prior to this session?; and Which of the following best reflects your habits of computer use? The results for the first question indicated that, overall, the participants had very little, or no, meaningful prior experiences or knowledge about language corpora. Most participants agreed they were “completely unfamiliar” with language corpora (n=32, 69.5%), while some said they “have only heard of the concept before” (n=8, 17.4%), and a few saying they were “somewhat familiar” (n=6, 13.0%). No participants indicated that they were “very familiar”. The results from the second question indicated that most participants used their computers at the level of “daily use for a variety of reasons (professional and personal)” (n=31, 67.3%), while some used it at the level of “daily use for one reason (professional or 34 personal)” (n=12, 26.0%), with only a few citing their use as “frequent” (n=2, 4.3%) or “infrequent” (n=1, 2.2%). After collecting this demographic data and inviting the selected participants to join the online modules as students, all participants were then randomly assigned to one of the two treatment groups (i.e., direct, n=23, or indirect, n=23). 3.3.2. Canvas After the participants had been randomly distributed into the two treatment groups, they were invited to join one of two online courses (i.e., direct or indirect), which were created using online modules hosted on the Canvas learning management system. When they accepted the invitation, they were allowed to access the main course page, which included, as shown in Figure 5, a schedule of all the activities they were expected to complete and a timeframe (e.g., Week 1, Week 2) for when they should complete each task. All of this information was also communicated to all participants in e-mails throughout their participation. 35 Figure 5 A portion of the main course page for the direct DDL online course, feature the course schedule Because participants were added as they signed up and were deemed eligible given the current study’s criteria for inclusion, there was not a single uniform time frame for all participants. Rather, participants completed the tasks over a four-week period from the time they were added to the course, which was monitored by the researcher. Reminder e-mails were sent to participants frequently to keep them on track during the four-week period. All of the activities in the four-week period, as described in greater detail below, were able to be completed by the participants asynchronously, and therefore there were no required meeting times, which gave the participants a great deal of flexibility in completing the activities at any time of the week which worked best for them. Because Canvas tracks the amount of time each participant spends on the platform, it would have been easy to determine if a participant was simply clicking through an activity without giving it any thought and remove that data from the overall data collection. Fortunately, there were no instances of this happening, and all participants appropriately 36 completed the tasks, as evidenced by either their screen recordings (i.e., for the direct group) or the duration of time that each spent on the Canvas system. Similar to other learning management systems (e.g., D2L), Canvas allows students to complete assignments, engage with material, and track their progress, all on the same platform. It allows instructors to receive activity submissions (in a variety of formats), assess student work, and create sequences of tasks or units. 3.3.3. Instructional activities and procedure Table 3, below, lists the order of activities that all participants engaged in during the duration of this study. Below that, I walk through each activity in more detail to highlight what participants had to do during each task. Each of the task protocols, as they appeared on the Canvas platform, can additionally be seen in the appendices at the end of this dissertation. For each of the activities (i.e., Activity 1 through 4), participants in the direct DDL group were instructed to record their computer screens as they completed the tasks. The participants then uploaded these screen recordings to the Canvas system. There were several instances in which participants had difficulties uploading their video files to Canvas, in which case I collaborated with those participants to work out other solutions (e.g., DropBox, Google Drive, alternative sites they were aware of). For the direct treatment group, the instructional activities for the online course had them interact with two different online language corpora, the Corpus of Contemporary American English (COCA) and the International Corpus Network of Asian Learners of English (ICNALE), both of which could be accessed on a web browser with no software downloads. The specifics of how each corpus was integrated into the activities is described in detail below, with the activities themselves located in the appendices, though the basic functions of both corpora allow writers to search for specific keywords, view those words in the context of authentic language use, among other functions. ICNALE, for instance, allows 37 learners to view the frequency for a keyword in writing by one learner English population (e.g., Chinese) as compared to another learner population or to NES writers. COCA, meanwhile, allows learners to view strong collocations between a keyword and additional words, among a variety of other features. Table 3 Procedure of tasks for the current study, adapted from Sun and Hu (2020) Task/Week Direct Group Indirect Group Purpose Introduction to the course (Week 1) Anticipated Duration ~ 10 minutes Watch a pre- recorded instructional video Watch a pre- recorded instructional video Pre-instruction writing task (Week 1) ~ 40-60 minutes Write a ~500- word essay on the prompt Write a ~500- word essay on the prompt Activity 1 (Week 1) ~ 20 minutes Paired hedges Introduction to the corpora (Week 2) (with and without the hedging lexical items) ~ 20 minutes Pre-recorded videos introducing the students to the functions of the two corpora Paired hedges (with and without the hedging lexical items) N/A Activity 2 (Week 2) ~ 30-40 minutes COCA and ICNALE information- gathering task based on a list of hedges Dictionary information- gathering task based on a list of hedges Briefly introduce students to the concept of hedging and its rhetorical functions To examine pre- instruction hedging frequencies and variety of hedges Encourage learners’ awareness and noticing of hedging Briefly introduce students on how to use the online language corpus tools to gain information about language use (direct group only) Provide the students with lexico- grammatical resources for realizing hedging 38 Table 3 (cont’d) Activity 3 (Week 3) ~ 30-40 minutes Activity 4 (Week 4) ~ 30-40 minutes ICNALE comparison task: NES v. Chinese learners Exemplar excerpt sharing task ICNALE essay task: differences between their usage and that of a NES Pre-treatment essay revision task Post-instruction writing task (Week 4) ~ 40-60 minutes Write a ~500- word essay on the prompt Write a ~500- word essay on the prompt Learner Perception Questionnaire (After completing all tasks) ~ 10-15 minutes Interviews of select participants (After completing all tasks) Respond to a questionnaire about corpus and computer use, and their thoughts on the modules, activities, and corpora Respond to a questionnaire about corpus and computer use, and their thoughts on the modules and activities Help participants notice the gap between NES and Chinese learners’ usage Encourage reflection on the effectiveness of hedged and unhedged statements To examine post- instruction hedging frequencies and variety of hedges Get further perceptions from learners about corpus use Get further perceptions from select learners in the direct DDL group about their perceptions on using the online corpora The pre-instruction writing task (Appendix C) required students to write an argumentative opinion essay on one of two prompts: Do you think that information in scientific research, business and the academic world should be made publicly accessible?; or, Do you think the construction of new railway lines or improvement of existing public transport should be prioritized? These topics were chosen to align with the same prompts used in Sun and Hu’s 39 (2020) study, which they chose from the academic writing tasks of official IELTS examination papers. Sun and Hu note that choosing these topics, as used from IELTS, was intended to "piggyback on the validity and reliability of its writing tasks because hedging has been found to be positively correlated with performance in IELTS writing tasks, with more proficient (and higher-scoring) test-takers tending to use more hedges" (p. 667). Half of each group (i.e., half of the direct group, half of the indirect group) wrote about the first prompt, while the other half wrote about the second. This was done since the participants, within the post-instruction writing task, then wrote about the opposite prompt to help address the learning effect if only one prompt was used for both the pre- and post- instruction essays. Simply put, no participant wrote about the same prompt for their post-instruction essay as they did for their pre-instruction essay. Participants were instructed to take a maximum time of one hour to complete their essay and were told to complete a roughly 500-word essay on the topic. Because the writing tasks were not counting for a grade, or for any certification, it was seen as unlikely that participants would cheat on their essays (e.g., with the aid of ChatGPT or outside sources). However, as discussed in the Conclusion chapter regarding the limitations of the current study (Chapter 6, sub-header 6.2), there was no monitoring of the participants’ essay writing. They were able to either write their essays directly in a text box on the Canvas platform or write them on a separate text program (e.g., Word, Notepad) and upload their submissions to the Canvas platform as file uploads, as can be seen in Figure 6. All participant essays were then downloaded and collected for data analysis as simple text (.txt) files. 40 Figure 6 Multiple modes of submission for the pre-writing task Activity 1 (Appendix D) was the same for both groups, the direct and indirect. Learners were presented with eight hedged segments pulled from the ICNALE NES student essays. The hedges were then removed and both versions (those with the hedges and those without) were presented as pairs to the learners, as shown in Figure 7. The learners were instructed to compare the two and identify the lexical items that made them different. Because Chinese EFL writers’ have been shown, within the previous literature as well as my previous corpus-based research, to struggle with tentative modals (e.g., would) and lexical verbs (e.g., think, believe), essay segments with such lexical items were used so as to hopefully implicitly direct their attention to more tentative expressions. 41 Figure 7 A portion of the Activity 1 page (student-view) The instruction period consisted of myself, via pre-recorded videos, offering participants short instruction on the concept of hedging and its rhetorical functions, as well as (for the direct group) a short pre-recorded workshop in which I briefly introduced how to use the main functions of the two online corpora that they had to consult during their tasks: the COCA and the ICNALE online language corpora. This instruction was delivered via pre-recorded videos that were uploaded onto the Canvas system, as illustrated in Figure 8. 42 Figure 8 Example of a portion of one of the instructional pre-recorded video pages Activity 2 was a different task depending on which group the participant was in. All participants were presented with a list of frequent hedging lexical items (Appendix A, from Hyland & Milton, 1997) across different grammatical categories (e.g., modal verbs, lexical verbs). The direct group (Appendix E) was instructed to identify, from the list provided, five modal verb hedging devices that they believed to be most frequently used in English and check their usage in COCA. They also were instructed to pick five lexical items from the lexical verb category from the list that they were not as familiar with and use COCA to learn more about how they are used. The indirect group (Appendix F) was asked to identify five modal verb hedging devices, as well as five from the lexical verb category they were not as familiar with, from the given list, and study how those items were used by accessing a dictionary of their choosing. The rationale for this was to let the Chinese EFL learners experience a greater variety of lexical verbs 43 and modals verbs, two categories in which they were found, as mentioned, to use significantly less than NES and several other L2 learner English populations. Experiencing a variety of modal verbs would also allow them to explore modals outside of the certainty will modal that Chinese EFL writers tend to significantly over-use compared to NES writers. Figures 9 and 10 illustrate what this activity looked like for both groups, as on the Canvas system. Figure 9 A portion of the Activity 2 page, student-view (direct group) 44 Figure 10 A portion of the Activity 2 page, student-view (indirect group) Activity 3 was a different task depending on which group the participants were in. For the direct DDL group (Appendix G), they needed to pick three modal verb hedging devices (i.e., will, would, might) and three lexical verb hedges (i.e., think, believe, appear) and compare the differences in their usage by NES writers and Chinese learners by accessing ICNALE. After, they needed to pick two essays from the ICNALE corpus, one from a Chinese L2 writer and one from a NES writer, and compare the frequency and variety of hedges used in either. The indirect treatment participants (Appendix H) completed a gap-filling task including a 494-word essay from ICNALE with ten different hedging devices gapped, including many modal verb and lexical verb examples. The students then filled in the hedges that they felt should go in the blank spaces, and afterwards were provided with the full essay and instructed to compare their answers with the hedges used by the original NES writer and examine the communicative and rhetorical differences that exist. The rationale for these were for the two groups to compare Chinese L2 writers’ usage patterns to that of NES writers, especially concerning specific hedging devices that Chinese L2 learners have been shown to either significantly over- or under- use compared to 45 NES writers. The goal was for them to possibly realize that there are discrepancies in how they would hedge something, versus that of NES writers. Figures 11 and 12 illustrate how these activities appeared on the Canvas system. Figure 11 A portion of the Activity 3 page, student-view (direct group) Figure 12 A portion of the Activity 3 page, student-view (indirect group) 46 Activity 4 was a different task depending on which group the participants were in. For the direct DDL group (Appendix I), they were instructed to browse the COCA texts and select a short passage from an essay which they thought exemplified an effective and appropriate use of hedging. They then shared their chosen excerpt and wrote why they believed it to be a good example of hedging and reflected on why it was chosen. The indirect group (Appendix J) was instructed to review their pre-treatment essays, consider where they could now hedge their statements within it, and revise their essays accordingly. The rationale for these tasks was to encourage reflection on the effectiveness of hedged and unhedged statements. Figures 13 and 14 illustrate how these activities appeared to the learners on the Canvas system. Figure 13 A portion of the Activity 4 page, student-view (direct group) 47 Figure 14 A portion of the Activity 4 page, student-view (indirect group) The post-instruction writing task took place at the end of the four-week period. It, like the pre-instruction writing task, required students to write a roughly 500-word essay within an hour. As mentioned, all participants wrote about the essay prompt which they did not write about for their pre-instruction writing task. The learner perception questionnaire, which participants completed asynchronously after the four weeks of instruction, was influenced from Sun and Hu’s (2020) study, though with several additions. The full protocol for these surveys is presented in Appendix K. Each survey started with several questions asking about the learners’ previous experience with language corpora and their computer habits. The following questions for each survey differed for each group. These questions were presented to participants via the Qualtrics survey platform, where students answered them using a five-point Likert scale (agree, slightly agree, neither agree nor disagree, slightly disagree, disagree). After those Likert-scale questions, there were two more open-ended questions that participants responded to so that I was able to collect thoughts or 48 perspectives about their perceived benefits or challenges in interacting with the online activities or online language corpora. Lastly, select participants (n=9) from the direct DDL group were invited to be interviewed after all of the tasks were completed. Only participants from the direct DDL group were included in the interviews, as they were the ones who directly interacted with the corpus tools in addition to the Canvas modules, and therefore were able to offer their perspectives on both. The nine participants were chosen in part because the video recordings they provided were the longest which, in combination with the content of the videos, suggested that they either had the most difficulty in completing the tasks or in engaging with the modules or corpora. However, it is also possible that the participants with the longer videos were simply more engaged in the tasks. Therefore, the videos were further screened to identify whether or not challenges or difficulty possibly contributed to the length of their videos. Longer pauses while completing the activities, having to backtrack and try a search again, or seemingly clicking around spontaneously on the interface to try to navigate to a certain function, among other things, were all considered as preliminary signs that a user may have been having difficulty while completing the tasks. Figure 15, below, shows an example of what the participants’ screen recording videos looked like while they were interacting with the online modules. Figures 16 and 17, meanwhile, show examples where participants seemingly ran into difficulties with the corpus tools. In Figure 16, for instance, the participant was trying to search for the lexical item couldn’t after doing a preliminary search for could. In COCA, words with an apostrophe (e.g., don’t) or are a contraction of two separate words (e.g., gonna) are often entered in as separate words by the corpus tagger. Therefore, users need to search them as separate words (i.e., instead of entering couldn’t, users should enter could n’t with a space in between). In Figure 17, meanwhile, a user 49 was seemingly confused at how to compare frequencies between two different populations (e.g., NES writers versus Chinese L2 writers). The snapshot shared here is from a larger portion of video in which they clicked around at different boxes and functions on the screen, seemingly at random, running into errors. They did, however, eventually figure out how to correctly format the search and complete the activity. These are just two instances, among many, from the video screen recordings which motivated my decision to invite select participants to be interviewed afterwards. Figure 15 Example of a participant’s screen recording video 50 Figure 16 Example of a participant running into difficulty with one of the corpus tools (i.e., and error message) Figure 17 Example of a participant running into difficulty with one of the corpus tools (i.e., confusion of how to complete a search) The nine participants chosen for interviews were also selected in part because, among those who had longer video recordings and preliminary signs of difficulty, they were the ones willing to be interviewed. From those nine, two of the participants were interviewed without the aid of a stimulated recall, as they requested to only be interviewed online through text. Interview questions were sent to them to fill out. For the remaining seven participants who were willing to 51 be interviewed virtually, a more thorough and systematic analysis, and coding, of their video recordings was conducted prior to the interviews, as described in the qualitative analyses section of this chapter (sub-header 3.4.2.2). Their screen recordings were used to conduct a stimulated recall with them during their interviews, as also detailed below. 3.4. Data analysis methods The following sections discuss the analyses that were conducted on the quantitative and qualitative data collected. To briefly summarize here, however, the quantitative data collection included the participants’ pre- and post- instruction essays as well as the Likert-scale questions on the learner perception surveys; the qualitative data analysis included the open-ended questions on the learner perception surveys, the participants’ recorded screen capture videos, and interviews with select participants (n=9) which were aided by the stimulated recall procedure (n=7). 3.4.1. Quantitative analyses 3.4.1.1. Participants’ pre- and post- instruction essay Regarding the participants’ pre- and post- instruction essays, all of the essays were saved as simple text (.txt) files, and all extraneous text (e.g., essay titles, participants’ names) were removed so that only the essay text itself remained. Next, four different corpora were created: the direct DDL group’s pre-instruction essays, the direct DDL group’s post-instruction essays, the indirect DDL group’s pre-instruction essays, and the indirect DDL group’s post-instruction essays. Table 4, below, highlights the size of each of those four corpora. AntConc (v. 3.5.9.; Anthony, 2011), a freeware corpus analysis toolkit for concordancing and text analysis, was used to collect corpus size information. The overall corpus, with all four corpora added together, consisted of 37,918 words, with a mean length of 412.15 words per essay. 52 Number of essays Corpus size (words) Mean essay length 23 445.26 Table 4 The corpora consisting of both groups’ pre- and post- instruction essays Corpus Direct DDL group (pre-instruction) Direct DDL group (post-instruction) Indirect DDL group (pre-instruction) Indirect DDL group (post-instruction) 10,241 9,344 9,182 9,151 23 23 23 399.22 406.26 397.87 After the corpora were created, AntConc was again used to search each corpus for Hyland and Milton’s (1997) list of the 75 most frequently occurring hedges as used by native speakers in their academic writing. As mentioned, the current study adopts Hyland and Milton’s (1997) taxonomy of hedges, in part because it offers an expansive list of hedges that relate to expressing doubt and certainty in academic writing, and in part because that study serves as an exemplar of how to search for hedges within multiple corpora of many writing samples. The list of lexical items that Hyland and Milton (1997) selected to examine hedges was created by identifying items common to native speaker usage, which they based on a variety of sources (listed below Appendix A). AntConc was used to examine the frequency of the different hedges identified by Hyland and Milton. The surrounding context was also recorded, which allowed for each individual instance of a hedge to be manually examined in order to make sure that the instance was, in fact, a hedge used to express certainty or doubt and not the same word(s) being used within a different context (e.g., for the adjective clear, such instances as “it was a clear day outside”). All cases where it was found that a hedging device was not being used were removed from the data so that only manually confirmed cases of hedging were included in the final frequency counts. The wildcards offered by the AntConc software, which allows users to search for different morphological variants of words or only stems of words, was used to account for the 53 different variants of the words as well as potential misspelling of words within the corpora (e.g., argu*/argue*, to account for argues, argued, or possible misspellings of the verb argue). This extraction and analysis procedure was in line with previous studies that used a combination of AntConc analysis and manual annotation when searching corpora for a list of discrete lexical items (Takahashi, 2014; Tran & Tang, 2022). After the hedging lexical items were extracted from the corpora, and the frequencies counted, several analyses were then conducted on the data. These are detailed below in the Results chapter with more specificity. The first analysis conducted involved examining the normalized frequency of hedges per 100 words of each of the corpora. With that information, log likelihood statistical tests were used to determine where there were significant differences. Further, a mixed-effects modeling analysis was then conducted. Mixed-effects modeling is a statistical method used to analyze data that has both fixed and random factors. Unlike, for instance, an ANOVA, this allows an analysis of the data with multiple variables that can affect the outcome, but in the case where some variables might be fixed (i.e., consistent across observations, preset by the researcher) and others may be random (i.e., vary randomly across observations, not controlled by the researcher). ANOVAs, while useful with data sets of fixed factors, cannot consider the many random factors that can affect an outcome in a real-world situation when you are dealing with human participants. Mixed-effects modeling can take these random factors into account, while also allowing for an examination of variation between observations of the individual (Brown, 2021). In the current study, mixed-effects modeling was used to analyze the effects of the instruction on the students' hedges. This included the fixed effects (e.g., which group a participant was assigned to [direct- or indirect- DDL], time [pre- and post- instruction]). It also 54 accounted for random effects for each individual student or essay to account for individual differences. This process is described in the Results, but, as a simple overview, involved specifying a random effects structure in the model to account for the correlation between observations from the same group or individual (Bates, 2010). By using mixed-effects modeling, I then had the ability to account for the variability in the data in a way I could not just by using ANOVAs. Being able to estimate the random factors' variance, I was able to control more for variability and improve the accuracy of the findings, which, in addition to the per 100 words and log likelihood statistics, help better identify how the fixed factors impacted students' hedges. 3.4.1.1. Learner perception surveys (Likert-scale questions) Regarding the Likert-scale questions on the learner perception surveys, all of the participants’ surveys were downloaded and their responses to the prompts (i.e., agree, slightly agree, neither agree nor disagree, slightly disagree, disagree) were tallied on an Excel spreadsheet. Stacked bar graphs were then used to illustrate the percentages of participants who felt a certain way towards specific prompts. As mentioned above, the learner perception surveys can be seen in Appendix K. However, Table 5, below, summarizes the Likert-scale portion of the surveys for both the direct and indirect DDL groups. Table 5 The Likert-scale portions of the learner perception surveys Direct DDL group Indirect DDL group Likert-Scale Questions: I understand the purpose of using corpora in these sessions. I found the searching technique easy to learn. The classroom activities (e.g., dictionary The exercises were helpful for learning hedges. I had some difficulty in performing the searching technique. work) were useful for learning hedges. My perception of the importance of hedging changed after taking the sessions. 55 The sessions increased my confidence in using hedges. My English writing improved after taking the sessions. Table 5 (cont’d) The instructions provided by the teacher were helpful for learning the searching technique. The authentic texts in the corpora are difficult to understand. The corpora helped me learn the usage of hedges. The corpora helped improve my English writing. I found the corpora more helpful than a dictionary. I want to see greater use of corpora in the language classroom. When I have problems in English language learning, I will use corpora as a reference tool. Learning about corpora has increased my confidence about writing in English. Overall, corpora are a very useful resource for English writing. 3.4.2. Qualitative analyses 3.4.2.1. Learner perceptions surveys (open-ended questions) In addition to the Likert-scale questions on the learner perception surveys, there were also several open-ended questions, summarized below in Table 6. Table 6 The open-ended portions of the learner perception surveys Direct DDL group Indirect DDL group Open-Ended Questions: What are your thoughts about the benefits of the hands-on corpus use for English language instruction? What are your thoughts about the problems of the hands-on corpus use for English language instruction? What are your thoughts regarding the benefits of the online learning tasks? What are your thoughts regarding the problems of the online learning tasks? The open-ended survey data were copied to a Microsoft Excel sheet, where each data source was examined separately. Kostere and Kostere's (2021, pp. 57-59) theoretical coding guide, a step-to-step guide of interpretive coding, influenced the process of examining the survey 56 data and the development of themes from it. In specific, their steps include, among others, the following: reviewing each participants' data and developing early-identified categories (i.e., patterns) with the research questions in mind, while remaining open to new potential patterns or themes; highlighting sentences or phrases that seem meaningful and coding or describing that data; reviewing highlighted text for relevancy; clustering related data and generating themes; and explaining each pattern with relevant exemplar data quotes. Because the current study was interested in the participants’ user experiences while interacting with the modules and corpus tools, the tenets of Usability Theory, mentioned above in the literature review, were considered when examining the data. As noted, Usability Theory broadly concerns assessing how technology is compatible with users, or "that the things people use to complete a task should not hinder the completion of that task by the nature of its design" (Peterson, 2007, p. 337). In the current study, for instance, Usability Theory speaks to the extent to which the design of the online modules or online corpora helped or prevented the participants in achieving specific goals in an intended way (i.e., completing the activities). By considering this theory while examining the qualitative data, both from the surveys and the interviews, I examined, for instance, whether there were difficulties or challenges that the participants experienced when successfully navigating or engaging with the modules or corpora. This is because, as noted earlier, Usability Theory includes a focus on how easily a tool (i.e., the modules or corpora) can be learned by users, how well they are able to use it once instructed on how to do so, how well users are able to complete tasks using it, and the users' perceptions or reactions towards using it (Nielson, 1993). Therefore, considering this theory when evaluating my data focused part of my analysis on my participants' experiences of using the modules or 57 corpora (e.g., the interfaces, the functions featured on them), as well as their perceptions of the tools and their ability to use them to complete their activities. After the analysis, the patterns from the data were visually represented using a web-like structure, as can be seen in the Results chapter. This is very similar to methods used by Crosthwaite and Steeples (2022) in their examination of corpus-assisted development of the passive voice for science writing with female secondary school students, and also similar to how they graphically represented their qualitative data. Their study similarly included students’ open- ended perceptions of using the corpus tools and presented those findings in a similar web structure after coding it for patterns and themes. 3.4.2.2. Interviews (aided with the stimulated recall procedure) The interviews with the select participants (n=9) included two steps: (i) examining the participants’ recorded screen captures and coding them for relevant patterns or themes (e.g., highlighting moments of difficulty in them), and (ii) developing the interview protocol and process. This sub-section briefly describes both of those steps. Regarding the analysis and coding of the screen capture videos of the select participants who were willing to be virtually interviewed on camera (n=7), the first step involved downloading their video recordings of all four activities and organizing them for efficiency. After, several videos from the participants were screened to determine how to code them or identify themes or patterns from within them. Specifically, the goal was to highlight moments in which the participants were facing difficulty or challenges in using the corpus tools to complete the activities. Therefore, similar to as done with the open-ended survey results, several videos (n=4) were viewed and early-identified themes were developed regarding the difficulties that participants seemed to face (e.g., having to backtrack to a page to try again, long pauses). The 58 four videos were screened again and, remaining open to new potential patterns or themes, those initial themes were then clustered with related themes and patterns were generated. Those eventual patterns, which were used as indicators that a participant may have been experiencing difficulty in their search, included the following: long pauses, backtracking to a previous page, seemingly random clicking around, encountering error messages or pages, leaving the corpus tool tab and returning to the Canvas platform or instructional videos (i.e., ostensibly for further guidance), or, in rare cases, giving up (i.e., completing the instructional activity without thoroughly conducting the corpus searches). After these patterns were constructed from the viewing of four of the videos, the four recordings provided by all seven of the interviewed participants were reviewed, and moments from the videos (i.e., time-stamps were recorded) were highlighted as meaningful or relevant to those patterns. Those moments were used in the interviews with the participants to stimulate (i.e., stimulated recall) their memory of what they were thinking or feeling during those moments of difficulty, alongside additional standard interview questions (e.g., What difficulties stood out to you about navigating them [the two corpus tools]? Did you have any trouble using them or navigating the interface?; What was the most frustrating part about the tools, if anything?). The full list of those standard interview questions, which were meant to supplement the natural conversation during the stimulated recall moments, can be seen in Appendix L. Specifically regarding the stimulated recall, prior to the interviews, participants were briefed on the purpose of the interview and told that they would be shown video clips from their work on the activities. The videos were then presented to the participants, and they were encouraged, with the use of open-ended questions, to reflect on their thoughts, feelings, and actions at the moment. This would include, for instance, presenting a participant with a video in which they were conducting 59 a search and encountering an error, or a problem, and asking open-ended questions such as: Can you describe what you were thinking when you decided to take those steps?; Looking back at this moment, why do you think you chose to approach the search in that way?; or How did you feel when this happened, and why? It should also be noted that an ideal stimulated recall would be conducted right after participants completed the tasks, so that their memories of the activities would be at their best and that they would be more likely to remember specific thoughts or emotions they felt while completing the tasks. However, due to the logistics of participants finishing tasks at different times, finding those who agreed to be interviewed, and arranging times to meet across different time zones, the interviews were often conducted a week or so after their completion of the tasks, which is less ideal. The interviews themselves were recorded, and saved as video files, as exemplified with Figure 18, and then later as text transcripts. Otter.ai was used to create transcriptions from the original files. The transcripts were edited for misrecognizing words or speech (i.e., due to accented speech or sub-par audio), failing to properly punctuate sentences, to clean them up (i.e., due to pauses or interruptions), and overall, to more accurately reflect the spoken words during the interview. After all the interviews were completed, a similar process was conducted as detailed in sub-header 3.4.2.1 regarding the open-ended survey results. Namely, the interviews were coded for initial themes and then re-reviewed for additional themes or to condense themes. Eventually, a list of patterns was identified from those themes, and representative excerpts from the interviews were selected to highlight those patterns and offer examples to flesh those patterns out in a discussion. This process is explained further in the Results chapter, alongside the findings from that process. 60 Figure 18 Example of a video recorded interview with a participant, identifiable information removed For the remaining two participants who only agreed to be interviewed over text, a common set of interview questions were sent to them, via e-mail, that they were to respond to. The questions were developed based on similarity to the topics discussed in the other synchronous interviews. Even though these two asynchronous interviews did not provide me the chance to probe into the participants’ answers, and while these two interviews were not aided by the stimulated recall, they still provided valuable insight into those participants’ experiences and perceptions of engaging with the modules and corpus tools. As with the open-ended survey question data, mentioned above, the patterns from this interview data were visually represented using a web-like structure, as can be seen in the Results chapter, similarly to what has been done in previous DDL research that collected and analyzed qualitative data (e.g., Crosthwaite & Steeples, 2022). 61 CHAPTER 4: RESULTS The following chapter illustrates the results of the analyses detailed above. The chapter is structured in the following way: the first sub-header offers results of the various statistical analyses conducted on the participants’ pre- and post- instructional essays; the second sub-header offers results concerning the Likert-scale survey questions; the third sub-header offers results from the qualitative analysis of the open-ended survey questions, and the fourth sub-header offers results from the qualitative analysis of the interview data. The Discussion chapter which follows this one, Chapter 5, discusses these results and draws implications from them. 4.1. Participants’ pre- and post- instruction hedging The first step in the quantitative analysis was to use the frequency counts from the hedges extracted from the corpora to compare the normalized frequencies (per 100 words) of the total number of hedges from each corpus. That information is visually depicted in Figure 19, below, and descriptive statistics are provided in Table 7. Figure 19 Normalized frequencies (per 100 words) of the total hedges from each corpus Hedges (normalized per 100 words) from pre- and post- instruction essays 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 Direct group (pre-) Direct group (post-) Indirect group (pre-) Indirect group (post-) 62 Table 7 Normalized frequencies (per 100 words) of the total hedges from each corpus Group Direct group (pre-) Direct group (post-) Indirect group (pre-) Indirect group (post-) Hedges per 100 words Raw hedge frequency/corpus word count 351/10241 389/9182 342/9344 353/9151 3.4274 4.2365 3.6601 3.8575 Range of frequencies (per 100 words) 1.7274-6.5934 3.2941-6.0837 2.2448-5.8496 2.6246-6.0172 As can be seen in Figure 19, both groups, the direct and indirect DDL groups, used more frequent hedging in their post-instruction essays. For the direct DDL group, the pre-instruction essay corpus had a frequency of 3.4274 hedges per 100 words and the post-instruction corpus had a frequency of 4.2365 per 100 words. For the indirect DDL group, the pre-instruction corpus had 3.6601 hedges per 100 words and the post-instruction corpus contained 3.8575 per 100 words. The first step to determine where significant differences existed was to conduct log likelihood statistical tests on the above results, which is based on the likelihood ratio, which expresses how many times more likely the data (i.e., hedges) are under one model than the other (i.e., direct or indirect DDL). Log likelihood tests therefore compare the frequency of occurrences in two corpora, relative to the sizes of the corpora, and are frequently used for corpus comparisons (Rayson & Garside, 2000). In comparing the frequency of occurrences between the pre-instruction and post-instruction corpora from the direct DDL group, the log- likelihood statistic (G2=8.30) indicates a statistically significant difference (p < 0.01), suggesting a distinctive pattern of hedging use between the two corpora and that the difference in hedging use is likely not merely random variation. For the two corpora from the indirect DDL group, the log-likelihood statistic (G2=0.48) does not exceed the critical threshold for significance at the 63 conventional levels (i.e., p<0.05 or p<0.01), indicating that the variance in frequency between the two corpora is not statistically significant. This suggests no substantial difference in the usage pattern of the hedges between thew pre- and post- instruction essays for the indirect DDL group. To further examine these findings with a more robust test, mixed-effects modeling was conducted. The first step involved preparing the data for the statistical testing. An Excel sheet was created in which each row represented an observation (in this case, a participant at each time point), and columns were created for "Participant ID", "Group" (i.e., direct or indirect), "Time" (pre- or post- instruction), and "Hedge Frequency". Because the essay length varied, out of concern it might affect the count of hedges, hedge frequency relative to essay length (i.e., hedges per 100 words) was used as the outcome variable rather than raw frequency. This setup allowed for a detailed analysis of the changes in hedging use from pre- to post-instruction, accounting for within-participant variations by including each participant at both time points, in a way that simply using log likelihood statistical tests do not. Figure 20 illustrates what the set-up of the data looked like on the Excel sheet once the hedge frequencies for each essay were converted to per 100 words using the word count of each individual essay. 64 Figure 20 The data preparation for the mixed-effects modeling tests, which included hedge frequencies per 100 words for two time periods of all 46 participants Once all of the data were prepared, and the hedging frequency per 100 words substituted the “Hedge Frequency” column, the analysis was conducted using the statistical software program JASP. The 'Group' variable, which represented the type of treatment (i.e., direct or indirect), showed an F-value of 0.425 with a p-value of 0.518, indicating no statistically significant difference in hedge usage frequencies between the two treatment groups. The 'Time' variable, representing the difference between pre-instruction and post-instruction, showed an F- value of 5.271 with a p-value of 0.027, indicating a statistically significant difference in hedge usage frequencies from before to after the instruction, and suggesting that instruction over time had some effect on the use of hedges. The non-significant group effect (p = 0.518) and a non- significant interaction between group and time (p = 0.224) in the mixed-effects model suggest that the change in hedge usage from pre- to post-instruction did not differ significantly between the direct and indirect treatment groups when considering the entire sample together. It is possible, however, given the log likelihood statistical results, that the overall time effect was 65 driven primarily by the direct treatment group and that the direct treatment group's significant change could be diluted when combined with the indirect group's non-significant change. To examine this in greater detail, further testing was done, using the mixed-effects modeling to compare the effect of time on the direct and indirect DDL groups individually. For the direct DDL group, the model again tested for the effect of time on the frequency of hedges and found an F-value of 4.811 with an associated p-value of 0.039, indicating that the effect of time is statistically significant at an alpha level of 0.05, since the p-value is less than 0.05. This suggests that there was a significant change in the frequency of hedges used in the essays from the pre-instruction to the post-instruction phase within the direct treatment group. The same analysis was conducted on the indirect DDL group individually, where the results showed an F- value of 0.801, with an associated p-value of 0.380. With a p-value greater than the level of 0.05, the effect of time on hedge usage was identified, similarly to the log likelihood tests, as not statistically significant for the indirect treatment group. This implies that there was no significant change detected in the use of hedges from the pre-instruction to the post-instruction phase for participants in this group, unlike the direct DDL group. Table 8, below, lists all of the results for the mixed-effects modeling tests, including when conducted on both groups at the same time (i.e., variables of Group, Time, and the Group * Time interaction) and individually (i.e., variable of Time). As a summary thus far: When both groups (direct and indirect) are considered together, there is a significant effect of time on the frequency of hedge usage, indicating that instruction, in general, helped to significantly alter hedge usage frequencies over time. However, when you look at each group separately: The direct DDL group shows a significant change over time, and the indirect DDL group does not show a significant change over time. Therefore, while 66 instruction as a whole seems beneficial, the more detailed analysis shows that it was actually the direct instruction that was statistically effective, and the indirect instruction did not produce a statistically significant change in hedge frequency. Table 8 Mixed-effects modeling statistical test results Effect Model Time Direct and Group indirect DDL Time * Group groups Time Direct DDL group Indirect DDL group df 1, 44.00 1, 44.00 1, 44.00 1, 22.00 1, 22.00 Time F 5.271 0.425 1.518 4.811 0.801 p 0.027 0.518 0.224 0.039 0.380 Regarding the variety of the hedges used (i.e., the number of unique hedging lexical items used), there was a difference between the two groups. The direct DDL group showed an increase in the number of different hedges used between the pre-instruction corpus (with a rate/density of unique hedge types relative to the size of the corpus at 0.332 per 100 words) and the post-instruction corpus (with a rate/density of unique hedge types relative to the size of the corpus at 0.501 per 100 words). The indirect DDL group, meanwhile, showed a decrease in the variety from the pre-instruction corpus (with a rate/density of unique hedge types relative to the size of the corpus at 0.321 per 100 words) and the post-instruction (with a rate/density of unique hedge types relative to the size of the corpus at 0.273 per 100 words). These results (Figure 21) indicate that, while both treatment groups showed an increased frequency of hedges in their post- instruction corpora compared to their pre-instruction corpora, the direct DDL group utilized more unique hedges in doing so, whereas the indirect DDL group used fewer unique hedges even while showing frequency increases across time. However, statistical tests to determine if the differences in the proportions of unique hedges between the corpora are statistically significant were conducted (i.e., both a two-proportion z-test, specifically designed to compare the 67 proportions between two groups, and a chi-square test for independence, which can be used to examine the distribution of words being "unique hedges" versus "not unique hedges" associated with the corpora). The results from both demonstrate that, for the direct group, there was no significance (p>0.05). For the two-proportion z-test, the z-statistic was found to be z=-1.84, indicating the lower number of unique hedges in the pre-instruction when compared to the post- instruction corpus, with p=0.06 (slightly higher than the p<0.05 significance threshold). The chi- square test for independence also showed no significance, with χ2=2.971, p=0.08. For the indirect group, there was similarly found to be non-significant results for the variety of hedges (p>0.05), between the two-proportion z-test results (z=0.598, p=0.550) and the chi-square test for independence (χ2=0.214, p=.644). This suggests that, while there may be observed differences in hedge varieties between the two corpora for each group, it cannot be said that such differences are statistically significant increases or decreases. Figure 21 Proportion of unique hedges per 100 words 0.6 0.5 0.4 0.3 0.2 0.1 0 Pre-Instruction Post-Instruction Direct DDL Group Indirect DDL group 68 Figure 22, meanwhile, shows the frequencies of the lexical verb hedges and modal verb hedges between the two corpora for each group. Lexical verb and modal verb hedges, as illustrated and discussed in the Methods chapter (sub-header 3.2), were the two grammatical categories of hedges that Chinese EFL learners were shown to significantly under-use when compared to NES writers. Therefore, those two categories were identified as specific areas in which Chinese EFL learners may need, and benefit from, targeted and focused instruction. The activities in the current study, as explained in the Methods chapter, therefore included a focus on those two categories. Figure 22 Frequency of modal verb and lexical verb hedges (per 100 words) 2.5 2 1.5 1 0.5 0 Direct DDL Group (pre-) Direct DDL Group (post-) Indirect DDL Group (pre-) Indirect DDL Group (post-) Modal Verbs Lexical Verbs As can be seen in Figure 22, both groups showed an increased use of lexical and modal verb hedges across their pre-instruction and post-instruction essays, indicating that they used both categories with higher frequencies after participating in the four-week online course. For the direct DDL group, there was a frequency of 0.6835 lexical verb hedges per 100 words in the pre- instruction corpus, compared to 0.8712 in the post-instruction corpus. There were 1.7966 modal 69 verb hedges per 100 words in the pre-instruction corpus compared to 2.2435 per 100 words in the post-instruction corpus. For the indirect DDL group, 1.016 lexical verb hedges were identified per 100 words in the pre-instruction corpus, with 1.136 per 100 words identified in the post-instruction corpus. For modal verb hedges, there were 1.5732 per 100 words in the pre-instruction corpus and 1.7702 per 100 words in the post-instruction one. Similar statistical tests, as noted above for the total frequency of hedges, were again used to identify significance. Regarding the modal verbs, a comparison between the pre-instruction and post-instruction corpora for the direct DDL group showed results that exceeded the threshold for significance at the 0.05 level (i.e., p=0.02, p<0.05) but not at the p<0.01 level, suggesting that the frequency variation may cautiously be considered potentially significant or indicative of a pattern. For the comparison between modal verb hedges in the pre- and post- instruction corpora for the indirect DDL group, the results showed no significance (p=0.29), as it did not surpass the thresholds for statistical significance (i.e., p<0.05). For the lexical verb hedges, a comparison between the pre- and post- instruction corpora for the direct DDL group showed results that did not meet the requirements for significance (i.e., p=0.13, p>0,05), as did the results for the comparison between the corpora for the indirect DDL group (i.e., p=0.43, p>0,05). These findings, along with additional findings regarding more specific lexical items, will be elaborated on and discussed further in the Discussion chapter. As a summary, however, there were significant increases (at p<0.05 for the log likelihood test, and p<0.01 for the mixed-effects modeling test) across time (pre- to post- instruction essays) for the direct group, but no significance found for the indirect group’s slight increase over time. Also, when the two groups were examined simultaneously, there was found to be a significant (at p<0.05) difference 70 between the pre-instruction corpora and the post-instruction corpora, indicating a significant hedging increase when looking at both groups overall. Regarding variety, the direct group showed an increase in variety, with the indirect group showing a decrease; however, neither result was found to be significant. For the modal and lexical verb categories, both groups increased their frequencies across time, though the only significance (at p<0.05) was found for the direct group’s increase in modal verbs. It should be noted that, regarding the analysis above, only the overall frequencies, frequencies by grammatical category, and variety of hedges were examined. As mentioned in the Methods chapter, the hedges were each manually examined from the corpora to ensure that the lexical items were indeed being used for the functions of hedging. However, there was no analysis concerning if the specific lexical items for hedging being used by the writers made the most sense, or were the most appropriate, in the larger context of the essay or essay segment. Again, as far as how effective hedging was operationalized in this study, hedging frequency was primarily evaluated. A higher frequency of hedging alone does not guarantee more effective hedging, as it is possible to use more hedges in a manner inconsistent with NES norms. However, previous research, as well as the corpus analysis that this study is informed by, have consistently shown that Chinese EFL writers typically use significantly fewer hedges than NES writers and writers from other Asian L1 backgrounds. Therefore, an observed increase in hedging among the participants likely reflects a movement toward the hedging patterns characteristic of NES writers, suggesting a potential improvement in hedging effectiveness. Additionally, when reviewing the increased use of hedges in participants’ post-instruction essays, the hedges appeared to be used logically and correctly, with no examples of egregiously inappropriate uses 71 of hedges. To illustrate with just one example, representative segments from one participant’s pre- and post- instruction essays are offered here. Excerpt 1: A segment from a participant’s pre-instruction essay: First and foremost, the public includes a variety of people such as children, old women, etc. It is impossible for everyone to fully understand the information in scientific research, business and the academic world. There is no denying that all people are not equipped with the expert to comprehend what is the exact meaning of the information. Therefore, it is inevitable that facts will be distorted in the process of transmission, which is very dangerous especially for children. What’s worse, there will also be social panic. It is as well a wise choice that information including expertise are not provided to the public in terms of scientific research, business and the academic world. Next, I’m not saying people except the exact fields don’t have a right to get to know the information. If the information is conducive to some people outside the field or can make their life more comfortable, then it is the researchers, businessmen and academicians that will translate the information into the language or version which other people can interpret. For instance, much of the research on the virus during the COVID-19 period is beyond the comprehension of ordinary people. Therefore, it is difficult for ordinary people to understand and apple useful information into their daily lives. At this time, experts become the bridge between specialized knowledge and knowledge that can be accepted by the public. Last but not least, if information in scientist research, business and the academic world should be made publicly accessible, it is unavoidable that there is incorrect information among it, which will cause misunderstanding or even worse consequences. Moreover, it can violate the privacy of the researchers and academicians. Therefore, keeping the information private is a kind of protection for them. The society has the responsibility to create a good and quiet research environment for them. (Participant # 8, Direct DDL group) 72 Excerpt 2: A segment from a participant’s post-instruction essay: Firstly, improving existing public transport is less costly than new railway. Buses and subways system is already built and operating. Small improvements like more bus or better scheduling may make a big difference for many people. On the other hand, building new railway is very expensive and can take many years to finish. Money that use for one new railway line might be used to improve many other types of public transport. Secondly, existing transport is already in places where people live and work. If you make it better, more people can easily benefit. For example, more frequent buses can help people get to work or school easier. But new railway often connect city to city, and not everyone needs to travel this much distance often. In my experience, I take bus and subway every day, but I only take train for special occasions like holiday or to visit family in another city. Also, it is faster to make improvements on the current system. Adding more buses or fixing broken subway car can probably happen in shorter time. New railway can take many years. During that time, people still need to use the old system, so why not make it better first? Some people argue that new railway can bring economic growth, but improving existing transport can likely also do this by making it easier for people to go to work and for business to move goods. (Participant # 8, Direct DDL group) As can be seen in the post-instruction essay, for instance, the use of various hedges (e.g., may, might, probably) are coherent and largely appropriate in the context in which they appear. This is reflective of essays from the other participants, as well. 4.2. Likert-scale survey questions The first step in the quantitative analysis of the survey data involved taking the number of participants who responded to each point on the Likert-scale survey (i.e., agree, slightly agree, neither agree nor disagree, slightly disagree, disagree) and tallying them. Those numbers were then used to determine the percentages of participants who responded each way. Figures 23 and 24, below, visually depict the data for, respectively, the surveys from the direct and indirect groups. 73 Figure 23 Likert-scale survey responses for the direct DDL group Agree Slightly Agree Neither Agree Nor Disagree Slightly Disagree Disagree Overall, corpora are a very useful resource for English writing. Learning about corpora has increased my confidence about writing in English. When I have problems in English language learning, I will use corpora as a reference tool. I want to see greater use of corpora in the language classroom. I found the corpora more helpful than a dictionary. The corpora helped improve my English writing. The corpora helped me learn the usage of hedges. The authentic texts in the corpora are difficult to understand. The instructions provided by the teacher were helpful for learning the searching technique. I had some difficulty in performing the searching technique. I found the searching technique easy to learn. I understand the purpose of using corpora in these sessions. 0 5 10 15 20 25 74 Figure 24 Likert-scale survey responses for the indirect DDL group Agree Slightly Agree Neither Agree Nor Disagree Slightly Disagree Disagree My English writing improved after taking the sessions. The sessions increased my confidence in using hedges. My perception of the importance of hedging changed after taking the sessions. The classroom activities (e.g., dictionary work) were useful for learning hedges. The exercises were helpful for learning hedges. 0 5 10 15 20 25 As can be seen from the survey of the direct DDL group (n=23), Figure 23, the results indicate that a slight majority of the participants at least saw value in using the corpora to complete the activities. A majority of participants, for instance, when combining the “slightly agree” and “agree” categories on the scale, indicated that learning about corpora increased their confidence about writing in English (n=13, 56.5%), that it helped them improve their writing (n=12, 52.2%), helped them learn hedges (n=17, 73.9%), and that they understood the role of the corpora in the activities they completed (n=15, 65.2%). Additionally, a majority of participants indicated that they thought that the corpus tools may hold potential value for their English learning, or that they would like to see an increased use of corpora in language learning contexts. For instance, a majority of participants indicated that they found corpora to be a useful resource for English writing (n=20, 86.9%), that they plan to utilize corpus tools in the future (n=17, 75 73.9%), and that they want to see greater use of corpora in language classrooms (n=22, 95.6%). Also, a slight majority indicated that they found the corpora more helpful than a standard dictionary (n=12, 52.2%). However, despite most participants indicating that the corpora were helpful for them in completing the tasks, and potentially beneficial to language learning, they also indicated that there were challenges or difficulties in interacting with the corpora. For instance, when asked whether they found the searching technique easy to learn, 60.8% of participants (n=14) indicated that they slightly disagreed or disagreed with that claim (with an additional n=5 indicating they did not agree nor disagree). Also, 69.6% (n=16) indicated that they had at least some difficulty in performing the searching technique to complete the tasks. 82.6% of participants (n=19) did indicate, however, that the instructions provided by the teacher were helpful for learning the searching process. Lastly, 43.4% of the participants (n=10) indicated that the authentic texts in the corpora were difficult to understand, which points to another potential challenge when integrating corpora into instruction, as corpora are often compiled of authentic sources of text. These results will be discussed further in the Discussion chapter (Chapter 5) of this dissertation. From the surveys from the indirect DDL group (n=23), Figure 24, the results show that the majority of participants indicated that the corpus-informed activities were helpful for their writing. 56.5% (n=13), for instance, indicated that the activities helped improve their writing, when looking at those who either “slightly agreed” or “agreed”. 65.2% (n=15) indicated that the sessions increased their confidence in using hedges, 91.3% (n=21) stated that the activities changed their perceptions of the importance of hedges, 100% (n=23) broadly found the instructional modules useful, and 100% (n=23) found the specific instructional activities useful for learning hedges. These results will also be discussed further in the next chapter. 76 4.3. Open-ended survey questions The participants’ responses to the open-ended survey questions were extracted and collected on an Excel sheet. For the direct DDL group, this included responses to the following questions: What are your thoughts about the benefits of the hands-on corpus use for English language instruction?; What are your thoughts about the problems of the hands-on corpus use for English language instruction? For the indirect DDL group, this included responses to the following: What are your thoughts regarding the benefits of the online learning tasks?; What are your thoughts regarding the problems of the online learning tasks? After, I went through all of the responses and coded them for themes and patterns, as influenced by Kostere and Kostere's (2021, pp. 57-59) theoretical coding guide, a step-to-step guide of interpretive coding, as mentioned in the Methods chapter. Their guide includes, for instance, reviewing each participants' data and developing early-identified categories (i.e., themes) with the research questions in mind, while remaining open to new potential patterns or themes. Additionally, it involves highlighting sentences or phrases that seem meaningful and coding or describing that data, reviewing highlighted text for relevancy, clustering related data and generating patterns, and explaining each pattern with relevant exemplar data quotes. While reviewing the participants’ responses, I specifically kept RQ#3 and RQ#4 in mind. RQ#3 (i.e., How do learners perceive the incorporation of online DDL instruction, and the use of either online corpora or corpus-informed materials?) meant that I reviewed their responses while considering anything that spoke to learners’ perceptions of the online instruction, whether positive or negative. I initially coded anything that spoke to a learner perception and then, after reviewing all survey responses, began categorizing them into themes or patterns depending on similarity. Regarding RQ #4 (i.e., What are students’ user experiences as they 77 engage with the online DDL modules and online corpora?), I kept in mind Nielson's (1993) components of Usability Theory, concerning user experience theory (as discussed in Chapter 2, sub-header 2.4), namely: learnability, or how easy a program is to learn; efficiency, or how easily users are able to use to complete a task; memorability, or how well the user is able to remember how to do certain things with a program; errors, or how good the program is at reducing navigation errors made by users; and satisfaction, or how much learners enjoy using a program. With these in mind, I reviewed all of the survey responses, and highlighted any instance in which a participant noted something (positive or negative) that spoke to any of the user experience issues from that list of components. After, I reviewed all of the highlights and developed categories of themes or patterns from within the data. After the development of themes and patterns, I re-analyzed the data to synthesize the patterns (e.g., grouping similar themes together, creating a hierarchy of themes) and gather data excerpts to represent each of the various themes. At the broadest level, the largest themes, like the top of an umbrella, were the following: for the direct DDL group, (i) challenges of using the corpora, and (ii) benefits of using them; for the indirect group, the (i) usefulness of the online activities versus the (ii) shortcomings of the online activities. Figures 25, 26, 27, and 28, illustrate the various sub-themes that were identified under those larger themes, along with the frequency of their occurrence within the participants’ data. Along with each, data excerpts are used to highlight each of the larger themes, and sub-themes, identified within those figures. As seen in Figures 25 and 26, the major sub-themes identified from the direct DDL groups’ open-ended survey responses, which asked them about their experiences with the online corpora, were largely centered around their perceptions (good or bad) regarding the corpora’s interface (i.e., the point of human-computer interaction, namely websites that the corpora were 78 hosted on), their functions (i.e., the range of options or tools that users can engage with while using the corpora), and their texts (i.e., the actual texts that the corpora were consisted of). From there, more specific themes were highlighted, along with the frequency of participants in the direct group (n=23) who had left comments illustrating those themes. Figure 25 Open-ended survey responses regarding challenges of using the online corpora (direct DDL group) Online corpus tools Challenges Lack of instructions (n=13) Lack of visual cues (n=8) Confusing error messages (n=6) Navigation difficulty (n=11) Hard to identify important functions (n=9) Finding examples (n=12) Difficult to comprehend (n=8) Compilation of corpora (n=7) Interface Functions Texts In Figure 25, regarding the difficulties or challenges that the participants' faced with the interface of the corpora, lack of instructions, lack of visual cues, confusing error messages, and navigation difficulty were the most frequent patterns identified from the survey results. The lack of instructions theme was highlighted in different ways by the participants. For instance, one participant noted that users "need good guidance or it's easy to get lost. Also, tech issues maybe? Not everyone is good with computers", which highlights their perception that instructional 79 guidance or training may be required before students access the corpora to complete tasks. Another participant similarly noted, "It was very hard to learn how to use at first since there weren’t really many instructions or "how-to" on how to access the tools. Also, without proper guidance, students might pick up incorrect ideas about how to use something or how to write/talk", illustrating how the lack of instructions on how to use the corpora may lead students to make incorrect or wrong language assumptions from their searches. This is supported by another participant who noted that the corpora were "very difficult to learn. [The online modules'] Instructions video was very helpful." As illustrated in the Likert-scale survey results, many of the participants found the instructional videos on the online modules helpful, which may point to their frustration that little instruction, in their perceptions, were included on the corpora interfaces. Another participant noted that they "Also, need more guidance on how to use it [the corpora] properly for learning English", which similarly speaks to how participants felt as though further instructions were needed to ensure that the tools were being used to learn English properly, accurately, and effectively. These representative excerpts from the survey data were also highlighted by other participants who echoed similar sentiments regarding the need for more instructions on the webpages themselves. The lack of visual cues theme similarly often spoke to the need for instruction, as participants mostly referred to it in regards to visual help cues. One participant, for instance, noted that they were frustrated at the lack of “instructions when you hover over buttons or items”, something that could provide instructions to users in an intuitive and visual way. Another participant noted that "didn't know what to click on to see the full text", which led them to believe that "Sometimes examples are not relevant to what I'm learning." This was interpreted as speaking to visual cues, as perhaps more obvious visual cues indicating how users could enlarge the full text or access it would have led students to more easily identify that 80 function to assess whether or not texts were relevant to their searches. Regarding confusing error messages, participants noted that sometimes the corpora would give them error messages when conducting searches they had already done successfully, or when following the instructions. This was almost exclusive to the COCA corpus, which does have a tendency to provide error messages even to correct searches, forcing the user to backtrack to the previous search page, reset their search parameters, and attempt again. Navigation difficulty largely spoke to participants being unsure of where to go to complete more specific searches, or where to access certain information (i.e., such as full texts, or specific comparisons). One participant noted, for instance, said that that they did "not know what to look at/for and [the corpora] maybe overwhelming to some students without a teacher." Another noted that it was "very hard to figure out what to click or where to go in to complete the assignments." As for the difficulties with the functions, it being hard to identify important functions and finding examples were highlighted as two major patterns in the participants' responses. Participants, for instance, noted that there were too many options for different types of searches or functions on the corpora, which led to them having “too many options, but not knowing what is important.” This comment speaks to how it was sometimes difficult for participants to know which functions were important and relevant, both overall and to the specific tasks they were completing. Another participant noted that it was difficult to “find specific things I want to learn...features not clear. This makes it difficult to use as a learning tool”, again speaking to their challenges in identifying which functions would lead them to the learning outcomes that they wanted. When it comes to finding examples, participants often noted that it was challenging for them to use the functions of the corpus tools to find relevant examples of keywords. One noted, for example, that “I think the main problem is how to select appropriate cases of language use 81 from the corpus in English language instruction”, which speaks to difficulty in setting the search parameters so that they are only able to find relevant (i.e., hedges) uses of specific keywords. Another noted that “It's confusing to find right examples”, also likely speaking to the difficulty in using the functions to highlight only specific uses of lexical items. When it concerns the difficulty or challenges they saw in the texts that the corpora were compiled of, the major themes included text that was difficult to comprehend and the overall compilation of the corpora. Several participants noted that the text in the corpora, whether because of the authenticity of the text or because of the vocabulary being used, could be difficult to understand. One noted, for instance, that “Sometimes, examples are too complicated and I get lost”, with another indicating that “some words [are] too difficult to understand.” One noted, very interestingly, that “the learner's familiarity and mastery of the vocabulary will affect the effectiveness of the corpus in language teaching. In my learning process, as I did not master the word "speculate", I could not understand how to use this word when going through the language use examples in the corpus, and even now I still do not know how to use it.” Another noted that “For me, the corpus is a bit difficult because it has too many unfamiliar words. It's hard to understand context and meaning sometimes”, which also may speak to the participants’ frustration with the compilation of the corpora. Several participants noted that the texts in the corpora, and the way the corpora were compiled, made it difficult for them, whether based on the quantity of texts, the type of texts, or the proficiency of the writers. One participant noted, for instance, that ‘I don't know if every sentence is grammatically correct, especially the ones from blogs”, indicating that they saw a need to be careful to pay attention to what type of text one is looking at when using to learn language use patterns. Another noted that the “examples can be too academic or formal” in their searches, with another noting that “I think corpus use can be 82 confusing sometimes. For someone like me, who is not very advanced in English, the examples can have slang or idioms that are hard to understand”, also referring to the difficulty of having corpus tools, such as COCA, which are compiled of different text types and registers. Figure 26 Open-ended survey responses regarding benefits of using the online corpora (direct DDL group) Interface Online corpus tools Benefits Functions Texts Graphic/visual representation (n=6) Finding words (n=15) NES comparisons (n=9) Accessing full texts (n=9) Authenticity (n=11) Quantity (n=16) In Figure 26, regarding what participants saw as the benefits of the corpus tools on their learning, the patterns were similarly broken down into the categories: interface, functions, and text. For the interface, the only major pattern highlighted was the graphic/visual representation of certain components. Namely, participants highlighted the effectiveness of the graphs used to show frequency differences in lexical items between different learner English populations (or between a learner English population and NES). One noted that it was very helpful "to see the comparison bars with similar words" as used by different learners of English. Another visual that participants complemented was the highlighting of the keyword in the context of the larger text. One participant, for instance, noted that “Sometimes, the targeted sentence is not long enough for 83 readers to get its meaning”, seemingly missing the function which lets users open the full texts provided in the concordance lines. However, in this complaint, they did note that having the target words highlighted and in the middle of the concordance lines made it easier to find relevant examples of the keywords in the larger texts. As for the the functions, the participants highlighted the ability to do NES comparisons, the ability to access full texts, and the ability to find word examples, as benefits. One participant noted that “Also, it's fun to see the difference between English speakers and non-English speakers by searching for the frequency”, speaking to the ICNALE comparison task. Another noted that that task was a benefit, indicating that “Seeing differences in how Chinese and English speakers use different words…I think it can help Chinese students use English words.” Another participant, very articulately, noted that “Secondly, the collection of native speakers and foreign language learners in the corpus allows foreign language learners to understand the difference between Chinglish and authentic English expressions more intuitively and clearly in English teaching, which cannot be gained from dictionaries.” The ability to access full texts was also listed as a benefit, with participants noting that it was nice to be able "to see real essays and not just sentences from them", that "It provides large amount of legit and professionally used sentences", and that there are "So many different types of texts and essays from real students." They also spoke to the ability to find example words as a benefit, with one participant noting that they "really liked that it wasn't just definitions like in a dictionary but more examples and a lot of different kinds", which "Can help understand common phrases and how words are used." Others noted, additionally, that "it's good for learning about word meanings in different contexts", and that "Seeing many examples of writing from student essays and not just definitions of words" was what they considered a benefit. 84 Regarding the texts themselves, participants noted that benefits included the authenticity of the texts, as well as the quantity of text compiled. Speaking to the text authenticity, participants noted that the corpora “Provides real-world examples, so easier to understand how language is used", "I think it is very helpful for understanding how real English is used in different situations. By looking at real examples, I can learn more about common phrases and language patterns that are hard to see", and that an "online, updatable corpus also helps to collect more comprehensive and richer examples of language use, which can provide more teaching resources for English language instruction." Another put it quite succinctly, “It is nice that there are lots of different types of texts from actual English students and that it is real essays.” Overall, the participants certainly seemed to enjoy that they were able to browse actual, authentic essays from real students, rather than artificial sentences. Speaking to the quantity of texts, participants noted that "It shows me many examples of how English is really used. This way, my English gets better, especially in reading and knowing more words", and that the "corpus has numerous example sentences which, I think, are essential to learning a new word apart from its Chinese meaning." Another mentioned that the quantity of the text was useful in letting them see how words can be used differently across different contexts or situations: "It helps me see many examples of English usage. This way, I learn how to use words in different situations." As can be seen in Figures 27 and 28, the major sub-themes identified from the open-ended survey responses from the indirect DDL group, which asked about their experiences of interacting with the online activities, were largely centered around their perceptions (again, good or bad) of the activities’ interfaces (i.e., as hosted on the Canvas LMS), ability to improve their understanding of hedges (i.e., learning a variety of hedges, and realizing their importance), and their ability to improve their academic writing (i.e., making them more proficient in 85 academic English). As with the results for the direct DDL group, more specific themes were highlighted, along with the frequency of participants in the indirect group (n=23) who had left comments illustrating those themes. Additionally, representative excerpts are provided from participants, which illustrate examples of those themes. Figure 27 Open-ended survey responses regarding usefulness of the online activities (indirect DDL group) Interface Online activities Usefullness Ability to improve their understanding of hedges Ability to improve their academic writing Assignment submission (n=9) Course organization (n=12) Variety of hedges (n=12) Importance of hedges (n=17) Writing "more professional" or "academic" (n=9) Downtowning (or, softening) (n=10) In Figure 27, which illustrates that participants highlighted as things they found useful in the online activities, the major categories of themes, again, were the following: the interface, the ability of the online modules and activities to improve their understanding of hedges, and the ability of them to improve their overall academic writing. For participants’ perspectives concerning the interface, they found the assignment submission tools, and the course organization useful. For the assignment submission, they especially appreciated that there were multiple options available to submit assignments for each 86 activity. One participant noted that, "Canvas was really cool because it let me do everything for the assignments on the website, and submit it as a Word file if I wanted to", whereas another noted that "The online learning on Canvas was a big help...I liked being able to draw on the documents provided by the instructor", which referred to the ability to submit assignments by annotating directly on the documents and submitting them that way. For course organization, participants noted things such as, “I find the online module on Canvas easy to use and understand", "The tasks on Canvas are good", "I like the way the online tasks teach, and it was easy to go through the whole course", and "I enjoyed how the online tasks on Canvas were organized." As for the ability of the modules and activities to improve their understanding of hedges, participants noted that they were able to help them with the variety of hedges they know as well as seeing the importance of hedges. Namely, for the variety of hedges, participants noted that, “It helped me learn new hedges", which contributes to learners' hedging variety with the addition of new lexical items, as well as a participant noting that they learned "different ways to express ideas so they are not too strong." Several participants highlighted specific activities which helped improve their variety of hedges. One noted, "It makes [a] big difference, and I learned some words I didn't know how to use before to revise my essay in Activity 4." Another said, "I really liked the dictionary searching task since it let me look up different words online and discover their true meaning." Participants also commented on how the activities illustrated to them the importance of hedges, noting such things as "These activities have not only introduced me to a variety of hedging techniques but have also demonstrated their practical application in academic discourse." More comments include "Before, I didn't get why it's so important. Now, I see how using hedges makes my writing sound better, but not too hard", "This is really cool for when I 87 write essays. I learned to sound less sure, which is actually good in English", and "I used to write very directly, but now I know how to mix it up. Using hedges makes my writing sound less bossy." Regarding the ability of the modules or activities to help participants’ overall academic writing, the major patterns included the ability of them to help participants write more “professional” or “academic” (as they saw it) or the ability to help them downtown or soften their writing. For the former, participants noted that "Now, I can write in a way that’s not too pushy. It's like making my English sound smarter but still easy to read", noting that they can "sound less bossy, which is great for school", how it helped "enabling me to craft my arguments in a more nuanced and sophisticated manner", that "This has improved my academic English significantly, making me sound more careful and less certain", and that the activities improved their "understanding how to soften my language in English. This is very important for writing in a university." Another spoke of professionalism by noting that, “this make[s] my academic English sound more professional.” Regarding the second pattern, participants noted that "I learned many new ways to make my writing less direct", that they encountered "new words to not sound too strong in English", that "I feel better about using hedges in my essays so I do not sound so strong", and that they feel more confident writing in ways that "sound more careful and polite." 88 Figure 28 Open-ended survey responses regarding shortcomings of the online activities (indirect DDL group) Online activities Shortcomings Interface Ability to improve their understanding of hedges Notifications (n=6) Activity sequencing (n=7) Limited understanding of various hedges (n=13) Basic or simple activities (n=11) Ability to improve their academic writing Not enough extensive writing (n=7) Limited engagement or feedback (n=8) In Figure 28, the categories again included the interface, the ability of the online modules and activities to improve their understanding of hedges, and the ability of them to improve their overall academic writing. In reference to the interface, the major patterns concerning what the participants saw as shortcomings of the online modules included a lack of notifications and the activity sequencing. The lack of notifications included participants noting that "Using Canvas wasn't very easy at the start since I wasn't told when to complete the tasks", although the researcher did e-mail them a schedule. Additionally, one was provided on the main course page on Canvas. Likely, the participant here was frustrated that they did not receive notifications automatically from Canvas letting them know when to progress to certain tasks. To be fair, this could have been enabled on Canvas but, since the participants were joining and finishing the course at different points in 89 time, was disabled. Another participant noted that “Canvas was okay, but when I submitted my assignments, I wasn't sure if they were really sent”, possibly also referring to it being more beneficial if they received multiple notifications letting them know they successfully completed a task. As for the activity sequencing, participants noted that "The activities on Canvas were fine, but I wanted to see more kinds of hedges over time. Using only my own examples was a bit boring", that "I had to ask for help a few times because I didn't know what to do next or what order things were in", and that "Sometimes I felt lost and didn't know which activity to do next." One noted that “The layout of Canvas was kind of confusing. I wasn't sure what to do in what order, and I sometimes missed tasks because they were not easy to find”, and another that “Using Canvas for the first time was a challenge. I often didn't know what to click on, and the instructions were not always clear. This made learning a bit harder.” While a sequencing of the activities was provided on the course main page, it is likely that participants could not easily locate the main page once they started, or their complaints speak more broadly to not seeing any cohesion or logic behind the sequencing. Regarding the ability of the modules to improve their understanding of hedges, they found the shortcomings to include a limited understanding of various hedges after completing the activities. This included comments such as “They [the activities] were quite basic and didn't push my understanding of the subject matter as much as I had hoped”, that they “provided basic information, but I didn’t get to see many examples of how hedges are used in real academic contexts, which would have been much more helpful for understanding and applying the concept”, and that the modules “didn't really dive deep into showing how to use hedges effectively...I didn't know why English speakers and Chinese speakers would use hedges differently or if one is better than the other.” 90 The ability of the modules to improve their overall academic writing was also addressed, with participants noting that they found the shortcomings to include activities they found basic or too simple, not enough chances for extensive writing, and limited feedback and engagement. For the first, participants often remarked that the activities were too shallow or basic for them and did not sufficiently challenge them. Remarks include "but when it came to the learning tasks, they were just okay...just focused on a few and see a few examples", "I used only my examples, and it was not so helpful", and "After getting the hang of the platform, I realized that the tasks didn't challenge me enough." Regarding the lack of extensive writing, participants noted that they only really wrote extensively at the start and end of the module and that the rest of the instructional period was largely relegated to “instead, just focused on a few [hedges] and see a few examples", and that "I really wished they had included more examples of hedges and chances to practice using them in our writing." Regarding the limited engagement and feedback, participants largely centered these complaints around the limited opportunities for peer- or instructor- interaction. Indeed, none of the tasks asked the participants to interact with each other at all, and the participants only interacted with the instructor when reminders to complete tasks were sent or through the pre-recorded instructional videos. Comments concerning this included "I missed talking to other students and the instructor. It's hard to learn when you can't easily ask questions or talk about the tasks" and "I felt a bit alone and couldn't get help or share ideas." Also, one participant noted that "Luckily the instructor was very nice in helping me figure out what I was supposed to do next when I was having trouble!", pointing to a positive reaction of the rare interaction that occurred across the four weeks of instruction. These results will be discussed further in the Discussion chapter of this dissertation. 91 4.4. Interview (and stimulated recall) data As mentioned, seven participants were interviewed with the aid of video clips from their screen recordings which showed moments in which they seemingly came across difficulty or challenges, and two additional participants were interviewed with text responses to a series of questions (for a total, n=9). As with the open-ended survey results, the commentary by the participants was collected and coded for themes and patterns. Since all of the participants who were interviewed were from the direct DDL group, as they were the ones who interacted with the corpora, their interviews were analyzed using the same major patterns which emerged from the direct DDL group’s open-ended surveys, namely the challenges or benefits that they saw with the corpus tools’ interface, functions, and the texts which they were compiled of. As with the open- ended survey results, key themes from the interviews are presented here, in Figures 29 and 37, in a cloud-like structure, showing themes which occurred related to those three patterns. The results here are intended to supplement the findings from the open-ended survey results and offer more insight into participants’ perceptions when they are questioned in a real-time, longer-form, format with the aid of their video recordings. Figure 29 shows the themes which emerged from the interviewee’s thoughts regarding the challenges of using the online corpora. 92 Figure 29 Interview responses regarding challenges of using the online corpora (direct DDL group, n=9) Error messages/account issues (n=5) Interface Limited "help" information (n=8) Online corpus tools Challenges Functions Language barriers (i.e., on the sites themselves) (n=4) Lack of information about text sources (n=3) Lack the meanings of the words/phrases (n=4) Need for training or teacher guidance (n=5) Texts Corpus compilation (n=6) As can be seen in Figure 29, the challenges that the interviewed participants noted concerning the interface of the corpora largely had to do with error messages/account issues, limited help information, and language barriers (i.e., needing to use translation services on the corpus tool sites themselves). As for error messages and account issues, participants noted that using the corpora, especially COCA, was sometimes made difficult by issues such as seemingly random error messages and needing to create certain kind of account for full and unlimited access. One participant, for instance, noted that "The reason [they may not continue using it] might be the failure of creating an account...which was really frustrating...You need to use it patiently and slowly. Otherwise, it won’t respond and you have to go back and try again… And it’s quite disturbing and time-consuming.” This comment came up during moments of stimulated recall, in which the participant was asked what their thought process was when, in the video, as 93 exemplified in Figure 30, they kept encountering error messages related to their account and searches. Figure 30 A moment from a participant’s screen captures, used to aid in the interview (i.e., encountering an error message) Another participant, when similarly prompted with a video clip from their own account issues (Figure 31), similarly noted that “If COCA could be open to guests, just like what ICNALE do, it would be nicer”, in reference to how ICNALE does not require users to create an account to have unlimited access and searches. Regarding issues with ICNALE, however, one participant noted that “it did take me several minutes to enter the searching webpage for ICNALE, because I first went into the information site for ICNALE and I couldn’t find the entrance of the searching site there." This comment also speaks to earlier complaints mentioned in the surveys that information help can be difficult to find, especially concerning the ICNALE corpus. 94 Figure 31 A moment from a participant’s screen captures, used to aid in the interview (i.e., encountering an error message) Participants also noted that they frequently encountered error messages (e.g., Figures 32 and 33) using the corpora, especially COCA, and that the frequent error messages were confusing and demoralizing to them because they often did not understand if they did the search incorrectly or whether it was a problem on the website or a limitation of their account access. Several participants noted that this uncertainty resulted in them adopting “going back to the main page” as a default solution for anytime they encountered trouble. This can be problematic, as it meant that random, or account access, errors trained the participants to always click back to the main page, which they may have then done when encountering errors which were legitimately a result of their searching technique. Perhaps, in such cases, participants did not try to examine how to correct their searches, but rather just defaulted to assuming it was an interface or account problem. 95 Figure 32 A moment from a participant’s screen captures, used to aid in the interview (i.e., encountering an error message) Figure 33 A moment from a participant’s screen captures, used to aid in the interview (i.e., encountering an error message) On a similar note, regarding the limited amount of help information on the corpus tools, participants noted comments such as “I don't know and am not familiar with many of the functions, so in the process of using them, I imitate the operation according to the teacher's 96 instructional video, and there are times when I may accidentally click on the wrong thing or there is an unknown error due to some other reasons, and I don't know how to deal with it, and I can only go back to the previous step and try again, so this reduces my efficiency to a certain extent.” This comment also indicates that participants were often unaware of why they were receiving error messages, and additionally that they were unaware of how to respond to them other than to go back to the previous page and try again. Another participant similarly noted that they “didn’t know if I was doing something wrong or if my computer or Internet was messing up…or what the corpus wanted me to do.” Regarding the language barrier issues, concerning the interface, multiple participants noted that, because the interfaces almost exclusively operated in English, it could be challenging for newcomers to the tools to use them effectively. One participant, for example, noted that "as both these two websites are all English, there may be a language reading barrier when browsing the websites [for non-native English speakers]...although translation software can be used." They further noted that the use of such software "would waste a little bit of time", reduce the efficiency of the learning, "and sometimes there are inaccurate translations." They were prompted to recall this thought when presented with a video clip (Figure 34) in which they accessed a side-bar to search up information for a word they were unfamiliar with (i.e., humanities) while conducting a corpus search. 97 Figure 34 A moment from a participant’s screen captures, used to aid in the interview (i.e., searching up the definition of a word) Another participant noted that “for non-native English speakers, maybe the language barrier will be a certain resistance”, speaking to similar concerns. Another participant, speaking about their potential future use of the corpora, said “I think the translation software will enable me to make better use of these two corpora.” For issues related to the functions of the corpora, participants largely centered their comments around a lack of information about the sources of where text came from, lack of information about the words (e.g., meanings or definitions), and a need for training or teacher guidance to effectively use the tools. For the first theme, participants noted things such as “when searching for the context of a word in ICNALE, the page doesn't show the information and source of the corpus, which also makes me feel a bit disappointed.” ICNALE does show some surrounding context of the texts (e.g., what kind of writing sample it is, the status of the writer), but not as thorough as the COCA corpus. As for the second theme, participants negatively compared the corpus tools to dictionaries and e-dictionaries because they do not focus so much 98 on the definition or meanings of words. One participant, for example, noted that “they [the corpora] lack the meanings of the words, which is also important in English learning...If you are talking about [a] paper dictionary, it’s too heavy and it takes a while before I navigate the word...as for e-dictionaries, I think it's the best." Another noted that "I think that if we are trying to learn a new word that we are not familiar with, we should still use a dictionary to look it up and learn it first. This is because dictionaries have...definitions and meanings." Another participant, while complimenting the COCA corpus tool, alluded to how their favorite tool on it was the Word function, since it operates similar to that of an electronic dictionary: "My favorite part is the section called Word inside COCA. This function shows me the information of a word in detail, including the common topics...[and] definitions. This is my favorite module because it allows me to get the most out of the word's usage in a short period." This sentiment was echoed in one interview, when a participant was prompted to comment on why they left the corpus tool pages and consulted an online dictionary (Figure 35), something that was not a part of the activity. They responded that they were not able to locate the simple definition of the word in the corpus tools and, since they did not know the word, they were unsure how to comprehend it in the context of the examples that the corpora offered them. 99 Figure 35 A moment from a participant’s screen captures, used to aid in the interview (i.e., leaving the corpus tools to use an online dictionary) Lastly, regarding participants’ feeling that the presence of an instructor, or training beforehand, are required to effectively use the corpus tools, comments were made such as they were only able to effectively use the tool at first "since I had already read the introductory videos provided by the instructor in advance as required in the course when I first used these corpus sites", and that "I think it was relatively easy to learn because of the teacher's videos." Another noted that "I don't seem to see this information [instructions] on the ICNALE site, or maybe I just overlooked it and didn't pay attention...I think I would need help to do it like the activity wanted." Another noted that while “I think that if you simply look up information about how often words are used then these two sites are pretty easy”, but that it is more challenging if users try to seek information, or conduct searches, beyond that. The challenges related to the texts of the corpora largely had to do with what they saw as issues concerning the compilation of the corpora. For instance, a participant noted that "I found that in the COCA corpus, most of this corpus information is from articles or blogs from 2019 and 100 before, which makes me feel that these corpora seem a bit outdated, and the corpora would make me feel less professional and authoritative. It would be nice if these corpora could be updated in a timely manner." Another noted that the ICNALE corpus was very skewed in the disciplines it features. They noted that it included student writing, but largely writing from scientific disciplines or from certain fields, and more limited writing samples from fields such as English education or more general argumentative essays such as they would encounter on many standardized English tests. They thought that a more balanced compilation would be helpful, since "a lot of the essays were talking about things I wasn't very familiar with", which made it difficult for them to comprehend the texts. This was echoed by several other participants. Though largely seen as an issue with ICNALE, one participant was prompted to mention it during a stimulated recall when they saw video of themselves examining the breakdown of text types by discipline in COCA (Figure 36). Figure 36 A moment from a participant’s screen captures, used to aid in the interview (i.e., examining frequencies across text types) For the perceived benefits of using the online corpora, the results are similarly broken up 101 into the categories: interface, functions, and text. Figure 37 shows the visualization of the main identified themes, which are discussed further below, alongside representative excerpts from the interviews. Figure 37 Interview responses regarding benefits of using the online corpora (direct DDL group, n=9) Interface Online corpus tools Benefits Functions Texts Homepage information (n=3) Seeing words in context (n=8) NES comparisons (n=6) Frequency in registers/text types (n=4) Authenticity (n=9) Quantity of texts (n=6) As can be seen in Figure 37, the main benefit the participants saw in the interface was in regard to helpful information located on the homepages of the tools, though it is worth noting that only three participants noted any benefits regarding the interface at all. This included, for instance, one participant who noted that "The COCA website has some helpful information on the homepage about the use of the website, which helps to help newcomers learn more about the website." For the benefits they saw in the functions of the corpus tools, participants noted the ability to see words in context, the ability to do comparisons with NES, and the ability to see frequencies across different registers or text types. One participant noted that they particularly 102 liked the ability to see different keywords in the context of larger language use: "I found the Context modules in both tools to be very useful...The Context module allows me to visualize how to use the word." This comment speaks to how providing the larger context of a keyword, and multiple examples of context, can allow learners to better picture what actually using a new word looks like in practice. Another participant similarly noted that “both of them [the corpora] provide me with many real sentences which is important for me when being confused about the use of some words.” Another function that the participants noted as helpful was the ability to use the ICNALE corpus to compare usage patterns between different learner English populations, or between a learner population and NES writers. One participant, for instance, was shown a video clip, Figure 38, in which they were conducting the activity in which they had to compare the frequencies of several words between Chinese EFL writers and NES. In this part of the video, they were encountering difficulties which, instead of showing them a comparison between Chinese EFL writers and NES writers, they were instead just seeing a comparison across multiple proficiencies of Chinese EFL writers (i.e., because they forgot to select NES writers as a comparison group). They eventually figured out how to do the correct search and were prompted in the interview what their thought process was during those moments. The stimulated recall video prompted them to explain that, despite the initial challenges largely due to a confusing interface layout, one of their favorite things about the corpus was "the Frequency one...gives me a better idea of whether a word is better for academic situations or everyday, as well as the differences between native and non-native speakers." This comment spoke to both the population comparisons that ICNALE allows, as well as the ability to see frequencies across different registers or text types in the COCA corpus, and possibly points to the helpfulness of how both corpora visualize their comparisons in bar graphs for users. Later on, speaking on whether they 103 will continue to use the corpora after this period of instruction, the same participant noted that “I will also use that section of Frequency to learn and try to make my expressions closer to native English speakers.” Figure 38 A moment from a participant’s screen captures, used to aid in the interview (i.e., incorrectly attempting a Chinese and NES comparison chart) For the benefits concerning the texts of the corpora, participants noted the authenticity of the texts as well as the quantity of the texts. Regarding the authenticity, one participant highlighted it as a benefit, with the caveat that it might be more useful for learners who have already learned a word. When discussing their thoughts on the differences between corpora and dictionaries, that participant noted that while they thought a dictionary may still be most useful for learners first encountering a word, “if it is for a learner who already knows the word, maybe they should use a corpus to learn the word more...a corpus can provide more real examples of the words being used." Another noted that both corpora “provide[d] me with many real sentences which is important for me” when they encounter confusion about the usage patterns of words. 104 One of the two participants who were interviewed through text provided these similar comments when prompted about their thoughts comparing corpora to dictionaries: “Both of the tools [the corpora] respond quite fast and their authenticity is essential for English learners.” These comments also speak to how participants mentioned the quantity of texts, and the ability to sift through large amounts of data, as another benefit of the tools. Along with the results from the learner perception surveys, these results will be discussed further in the Discussion chapter. 105 CHAPTER 5: DISCUSSION The following chapter discusses the results from the previous chapter and draws implications from them based on this and previous research. This chapter is organized by individually discussing each of the four research questions that motivated the current study. As mentioned, those four research questions consisted of the following: RQ #1: Is online DDL instruction effective in improving Chinese L2 learners’ use of hedges?; RQ #2: Do a direct and an indirect approach to online DDL instruction differ in their effectiveness?; RQ #3: How do learners perceive the incorporation of online DDL instruction, and the use of either online corpora or corpus-informed materials?; RQ #4: What are students’ user experiences as they engage with the online DDL modules and online corpora? 5.1. RQ #1: Is online DDL instruction effective in improving Chinese L2 learners’ use of hedges? Regarding the first research question, the quantitative results seem to indicate that, largely, it is, at least in regard to hedging frequencies. Both treatment groups displayed an increased frequency of hedging across time. While a higher frequency of hedging alone does not guarantee more effective hedging—as it is possible to use more hedges in a manner inconsistent with NES norms—the overall trend suggests progress. Notably, previous research, as well as the corpus analysis that this study is informed by, have consistently shown that Chinese EFL writers typically use significantly fewer hedges than NES writers and writers from other Asian L1 backgrounds. Therefore, the observed increase in hedging among the participants likely reflects a movement toward the hedging patterns characteristic of NES writers, suggesting an improvement in hedging effectiveness in these groups. The mixed-effects modeling, when conducted on both the direct and indirect DDL groups simultaneously, showed an overall significant effect of the Time variable on the participants’ use of hedges. This means that there was an overall significant 106 increase in the use of hedges in the post-instruction corpora when compared to the pre- instruction corpora. That is, when both groups (direct and indirect) are considered together, there is a significant effect of Time on the frequency of hedge usage, indicating that instruction, in general, helped to significantly alter hedge usage frequencies over time. Additionally, a log likelihood statistical test was conducted where both groups’ pre-instruction essays were combined and compared to the combined collection of both groups’ post-instruction essays. The results (G2= 6.78, p = 0.0099) similarly indicate a significant increase in the use of hedges between the two corpora, indicating that, taking both groups as a whole, there was a significant improvement in hedging frequency across time. Overall, then, it seems that DDL instruction did help the Chinese EFL writers use a higher frequency of hedging in their writing between their pre- and post- instruction writing tasks. Before diving deeper into the differences between the two different DDL approaches, this broader finding supports previous research regarding the effectiveness of DDL, overall. While the research on DDL for improving the use or frequency of hedges is very limited, this finding overall aligns with Sun and Hu (2020), who similarly found an overall increase in the use of hedges between the pre- and post- instruction writing tasks (though, as mentioned in the next sub-header, with differing areas of significance when examining the two treatment groups in isolation). Additionally, while not specific to hedges, Mizumoto and Chujo (2015), a meta- analysis that targeted studies which used DDL approaches for learning lexico-grammatical items in EFL contexts, found that "the DDL approach worked particularly well for learning vocabulary items", and that "the results of the current meta-analysis would provide further support for the use of a DDL approach in the classroom, which could be an alternative methodology for facilitating the learning of lexico-grammatical items" (p. 1) rather than more traditional teaching 107 approaches. Other DDL studies which have examined the ability of DDL to promote vocabulary learning has found similar results, with researchers noting that “DDL has received much attention as an effective method to improve L2 vocabulary, in that it…provides learners with large amounts of authentic language input for linguistic inquiries” (Lee et al., 2020, p. 11). Lee et al.'s study, for instance, found that, among 35 South Korean undergraduate students who underwent DDL instruction, also utilizing the COCA corpus, there were positive correlations between higher vocabulary proficiency and students who were able to "utilize DDL-focused strategies" (p. 6). Yao (2019), meanwhile, in their examination of two treatment groups of Chinese students, found that those who encountered the DDL treatment were more effectively able to learn vocabulary items as opposed to those who experienced the more traditional teaching methods, and that there was an overall favorable attitude from the students regarding the DDL implementation. Boulton (2009) similarly found, in an examination of 132 first-year students, that DDL, utilizing corpus data, seemed to prove more effective for helping improve vocabulary recall as opposed to the more traditional teaching methods or grammar resources also used in the study. Yilmaz and Soruc (2015), another DDL study which utilized the COCA corpus and compared a DDL treatment group to a group with more traditional teaching methods, also found that "pre/post-test results showed that the class receiving vocabulary through concordance performed much more than the control class which received traditional vocabulary instruction" (p. 2626). These studies are just several examples, among others (e.g., Lee & Lin, 2018; Shaw, 2011), which have spoken to the effectiveness of DDL in learning lexical items. Again, many of these examples are focused on learning vocabulary more broadly and not focused specifically on hedging lexical items. However, they still speak to how previous literature has largely supported the ability of DDL approaches to help students learn, retain, and 108 utilize higher frequencies of new lexical items, thereby increasing their lexical variety. The current study centered its focus on lexical items utilized for hedging, and, overall, found similarly positive results. Namely, again, that, when examining the effects of instruction on hedge usage frequency across both direct and indirect DDL groups collectively, the mixed- effects modeling revealed a significant difference over time, suggesting that the DDL instructional intervention was effective in modifying hedge frequency. This overarching result indicates a positive impact of the instruction as a whole on hedge usage frequencies from pre- instruction to post-instruction. Though the findings of the present research are more nuanced when comparing the results of the different treatment groups, the overall finding listed in this sub-header seems to support such previous studies. 5.2. RQ #2: Do a direct and an indirect approach to online DDL instruction differ in their effectiveness? Again, examining the two treatment groups in isolation provides more nuance. In the individual mixed-effects modeling tests, and the log likelihood tests, the direct DDL group displayed significant increases in hedging between the pre- and post- instruction essays. The indirect DDL group, meanwhile, while showing increased hedging use between the pre- and post- instruction essays, had non-significant findings. That is, when you look at each group separately: The direct group shows a significant change over time, implying that, in this group, the instruction led to a significant increase in the frequency of hedge usage from before to after the instruction; the indirect group does not show a significant change over time, implying that the instruction did not lead to a significant change in hedge usage frequency. Therefore, while DDL instruction as a whole seems beneficial, a more detailed analysis shows that it was actually the direct instruction that was statistically significantly effective, whereas the indirect instruction did not produce a statistically significant change in hedge usage frequency. This indicates that the 109 aggregate improvement in hedge usage can likely be attributed primarily to the success of the direct instructional approach. Additionally, while both groups had an increase in the use of hedges pre- and post- instruction, the direct DDL group showed an increase in the variety of unique hedges used over time, whereas the indirect DDL group showed a decrease. This means that the participants in the direct DDL group significantly increased their use of hedges (i.e., frequency) over time while also increasing the unique number of hedges used (i.e., variety). Participants in the indirect group non-significantly increased their frequency but did so with a more limited number of unique hedges. However, as noted in the Results, neither of the changes in hedging variety were found to be statistically significant. There are several implications from these findings, and several interesting contrasts with the findings of Sun and Hu (2020), whose study the current research is a partial replication of. In their study, they found no significant differences between the frequencies of hedges for the direct DDL group, though noted that “these results suggested that the direct DDL treatment had a sizeable effect on the frequencies of hedge use that was not reliably detected in terms of statistical significance” (p. 675). That is, while they found no significance across time, the direct DDL group showed “differences of a medium effect size [that were] practically meaningful” (p. 675). As for the variety for the direct DDL group, they found a slight decrease from pre- and post- writing task, “indicating that the direct DDL treatment was not effective in improving the variety of hedging devices employed by the students” (p. 676). For the indirect DDL group, they found statistically significant increases across time for the frequency of hedges, and slight non- significant increases in variety. 110 The results are almost the opposite in the current study, which showed significant frequency increases for the direct DDL group as well as increases in variety, and non-significant frequency increases for the indirect group along with a decrease in variety. There are several possible reasons for these discrepancies. Of course, limitations of the current study, or differences in the participants (e.g., they used undergraduate students all of one major, whereas the current study included participants from all kinds of majors) are all possible explanations. Also, as the current study was only a partial replication of Sun and Hu’s study, it was not expected for the results to perfectly mirror theirs, especially as several substantial alterations were made to their design. In fact, those substantial alterations may possibly help explain the contrast in the results. Namely, Sun and Hu’s study was conducted in a traditional classroom environment with the presence of an instructor, whereas the current study took place all online, asynchronously, in a CALL context. Therefore, it is worth asking whether the change in modality could possibly have influenced the results. One potential hypothesis is that, given that the current study was conducted in an environment with increased learner autonomy (i.e., asynchronous online learning versus traditional teacher-led learning), this encouraged students to engage in more free exploration of the corpora. That is, students in the direct DDL group may have been given more autonomy to learn on their own, which encouraged more incidental learning of hedges as they explored the corpus tools. It should be noted that, while the estimated time to complete most activities was listed as between 30 and 40 minutes (which was a generous amount of time to do so), many participants returned screen recordings that, for an individual activity, could exceed 90 minutes. One possibility is that learners simply struggled enough with the searching techniques that it took 111 longer than estimated for them to complete the activities. However, reviewing the videos showed that, in many cases, students would conduct searches or interact with the tools in ways that were not required by the activities (i.e., searching for hedges from the list that were not part of the categories they were supposed to be searching, continuing to search for hedges after they had completed the assignment). This is also supported by students’ qualitative responses towards the benefits of the corpus tools, which included many comments about how they appreciated how much quantity of data there was to go through. This includes comments such as: "It provides large amount of legit and professionally used sentences", that there are "So many different types of texts and essays from real students", and that the ability to scan large amounts of essays "helps me see many examples of English usage. This way, I learn how to use words in different situations". Indeed, much of the benefit that the participants saw with using the tools, as mentioned in the Results chapter, had to do with the ability to freely access large amounts of writing. If the participants in the direct DDL group did indeed explore the corpus tools more, they likely encountered a wider use of a variety of hedges and were exposed to more authentic examples of hedging use. Seeing these usage patterns may have had the potential to show them how the hedges could actually be used, which may have increased their confidence in using more frequent hedging in their post-instruction essays. This could possibly help explain why the direct group not only was more effective and increasing hedging frequency, but also at improving variety. The indirect DDL group, meanwhile, was not exposed to many examples of hedging use in context, and rather consulted dictionaries of their choosing or came up with their own example sentences to complete the instructor-prepared activities. It is possible, though more thorough questioning of the participants would need to be done to confirm the hypothesis, that the lack of a teacher presence for the direct group (i.e., whether monitoring their progress or in preparing the 112 material for them) allowed those participants more freedom in “playing” with the corpora or conducting their own searches out of curiosity, which led to far greater exposure to hedging usage patterns and variety of hedges than their counterparts in the indirect group. A second possible explanation for why the results between the two studies were somewhat different, again, apart from the differences made to their design and possible differences in the participants, concerns the broader differences between in-person and online learning contexts. The traditional classroom environment of Sun and Hu's study may have afforded greater real-time social interaction, which has been shown to be able to enhance motivation, clarify issues immediately through peer or instructor support, and help promote collaborative learning experiences (Phillip et al., 2013). Additionally, the presence of an instructor in an in-person setting has been shown to be able to lead to more spontaneous and dynamic adjustments to the teaching practices based on immediate evaluations of student understanding and engagement (Ambrose et al., 2010). Also, it is important to note that the physical learning environment itself may play a factor. Physical classroom spaces may offer fewer distractions compared to online learning (Cowit & Barker, 2022), though this largely was controlled by having participants need to screen capture themselves while completing the activities. Lastly, students' ability to adapt to online learning environments varies, based on a variety of factors (e.g., experience with technology, familiarity with online learning), which may have had a role (Li et al., 2023). While all of the participants in the current study self-reported frequent and regular Internet use, the subtle differences in those experienced with online learning platforms and those who were not may have influenced their participation in the activities, or their ability to effectively engage with them. In an online setting, differences in, for instance, digital literacy (Prior et al., 2016) or the ability to pick up on and to "utilize DDL-focused 113 strategies" (Lee et al., 2020, p. 6) has been shown to be able to impact and influence learning outcomes. It is therefore possible that, given that Sun and Hu’s participants worked in a traditional classroom environment, with their peers and instructor, these environmental factors supported the more traditional indirect teacher-led DDL methods as opposed to the hands-on, more autonomous, direct DDL methods. Sun and Hu even note that, regarding the indirect group, “the learning tasks in the indirect approach were similar in format to those familiar to the students and therefore required little training and allowed the class time and the students’ attention to be focused on the learning target” (p. 681). In contrast, the flexible and independent online asynchronous learning context in the current study may have better supported the hands- on corpus learning of the direct group in contrast to the traditional instructor-prepared activities of the indirect group. The online context of the current study, for instance, may have afforded more flexibility, with the participants being able to engage with course materials at times that best fit their schedules, something which has been highlighted as a benefit of online learning (Daymont et al., 2011), especially for those juggling other commitments. Also, as mentioned above when discussing the greater ability to independently free-roam the corpus in an online asynchronous environment, there may have been expanded access to learning materials given that the participants were able to deviate from their specified tasks and use as much time as they want, whereas in a traditional classroom the time use is often more structured and monitored by the instructor. Also, the online context in the current study may have provided more of a chance for, similar to the point above, self-paced learning, which research has shown can benefit students by accommodating different learning speeds and styles (Skylar, 2019). This self-paced approach may help ensure that learners can understand the material or tools they are using (e.g., a corpus 114 tool) before moving on, which may be why the direct group participants often spent more time than expected when completing several of the tasks. The participants in Sun and Hu's traditional classroom study, meanwhile, were under more structured time constraints, given the logistics of an in-person classroom. Therefore, though more evidence would need to bear this hypothesis out, it may be possible that the indirect DDL group in Sun and Hu’s study outperformed the direct group because of environmental factors of the traditional classroom (e.g., peer and instructor interaction), or tenets of in-person teacher-led instruction (e.g., more monitoring of time and task), lending themselves well to indirect instructor-prepared activities as opposed to more independent and autonomous tasks. Meanwhile, the direct DDL group in the current study may have outperformed the indirect group because of factors concerning asynchronous online learning contexts (e.g., less time and task monitoring, more flexibility), or tenets of more autonomous online learning (e.g., the ability for more free-range exploration), lending themselves well to hands-on independent corpus activities. 5.3. RQ #3: How do learners perceive the incorporation of online DDL instruction, and the use of either online corpora or corpus-informed materials? 5.3.1. The direct DDL group Many of the findings in the Likert-scale survey responses among the direct DDL group are quite similar to those in Sun and Hu (2020). For instance, many of the participants in Sun and Hu’s study also highlighted their perceived usefulness of corpus tools, their desire to see them used in more classrooms, and their overall positive impression with using the corpora. As in Sun and Hu's study, participants in the direct DDL group largely seemed to respond quite well to the DDL implementation and engaging with the corpora. Themes that were identified from their open-survey results, and interviews, for instance, included that they: 115 enjoyed being able to sift through large amounts of authentic texts; being able to see frequencies of words across different text types, registers, or learner English populations; being able to see keywords surrounded by context; and the ability to access full texts. Many of these themes relate to the ability of the corpora, unlike that of a dictionary, to give students access to many examples of keywords surrounded by the larger context of authentic language use. As noted in the Results chapter, students also liked the additional functions of the corpora, such as the Frequency functions on the COCA corpus, which allows users to see frequency comparisons of keywords across registers or text types, and the ICNALE corpus, which allows for frequency comparisons across learner English populations or between learners of English and NES. Major drawbacks highlighted by the direct DDL group included: it being hard to identify the important functions, or the role of certain functions; it being difficult to find examples of certain usages of words, the authentic texts in the corpora being difficult to understand; the lack of definitions of words in an easily searchable spot; and the compilation of the corpora being skewed. Therefore, participants’ perceptions of the incorporation of online DDL instruction, and specifically the use of the online corpora, was mixed, which may point to considerations instructors or researchers should take into account when implementing direct DDL. For instance, participants noted that the compilation of the corpora (i.e., whether a mix of academic and blog writing in COCA, or the ICNALE corpus being skewed by discipline) made it difficult for them, either due to registers changing across text types or encountering words that they were unfamiliar with. Therefore, instructors may consider preparing students to encounter these diverse authentic texts or instruct students how to set parameters to the searches so that they only encounter texts from certain kinds of sources. Another issue that came up concerned not knowing which 116 functions were relevant to specific searches, or how to find specific kinds of information using different searches (e.g., how to search for a keyword only when it is performing as a specific part of speech). Again, this can be something which instructors spend more time preparing their students for; additionally, instructors can also possibly use it as a teaching moment and draw students’ attention to how every occurrence of a keyword is not performing the same grammatical role. Participants were also disappointed that the corpora did not provide them with as clear and concise a definition (or other basic information) about a word, as a dictionary would (with the exception of the Word function on COCA, which was praised for being similar to a dictionary). As mentioned earlier, this challenge could possibly be alleviated by preparing students for direct hands-on DDL or training them to "utilize DDL-focused strategies" (Lee et al., 2020, p. 6). That is, preparing them for learning through large amounts of language usage patterns as opposed to dictionary definitions and artificial example sentences. 5.3.2. The indirect DDL group Some of the benefits identified by the indirect DDL group can be summarized as including: the activities helping them learn a variety of hedges; helping them see the importance of hedging as a rhetorical strategy; helping them write, as they saw it, more "professional" or "academic"; and that they helped them downtown or soften their writing. Drawbacks can be summarized as including: that they only provided them a limited, surface-level, review of hedges; that the activities were too simple and basic and not challenging enough; that there were not enough opportunities provided for more extensive writing; and that there was limited peer- or instructor- engagement or interaction. An implication from these results possibly touches on the differences between traditional classroom learning contexts and asynchronous online learning contexts. Several of the 117 participants in the indirect group found that the activities were not challenging enough, did not go far enough in showing them hedging usage, and did not show them enough examples of hedges across multiple contexts (i.e., several complained that they were mostly relegated to using their own examples). This complaint was not mentioned at all by any participant in the direct group. A possible explanation of this complaint may concern how, in an independent online learning context, the hands-on corpus use of the direct group enabled participants to either tailor the activities to their preferred level or do conduct their own searches beyond the scope of the activities (i.e. free roaming) and therefore give themselves greater exposure to a wider variety of hedges (as well as an increased frequency of usage data). Meanwhile, the instructor-prepared, hands-off, activities for the indirect group did not afford them the possibility to explore the corpus tools at all and relegated them largely to completing the rather straightforward activities. These activities may work well in a traditional classroom environment, such as in Sun and Hu’s (2020) study, where a teacher can spontaneously alter the activities or where students can engage in peer- or instructor- interaction (the lack of interaction was another complaint mentioned by several participants). In an asynchronous online context, however, it is possible that more proficient or advanced students were simply able to rush through the activities and not have the ability to engage with much more learning beyond the scope of the activities. If they had been in the direct group, meanwhile, they may have had more opportunities to freely explore the corpus tools and adapt their learning experience to a more challenging level. More probing into the participants would be needed to know whether or not this is what they were thinking; for instance, participants could have been asked what could have made their learning experiences richer or more challenging for them. Because that insight was not collected, it cannot be said whether access to the corpora would have made them feel as though the level of their learning 118 was more in their control. However, given that multiple participants noted the easiness and surface-level learning of the activities in the indirect group, and no such comments came up in the direct group, it is a possible hypothesis. It should be noted again, however, that the participants overwhelmingly found the indirect activities helpful to their knowledge of hedges and helpful to their overall academic writing. Part of that may be due to the fact that they genuinely learned about hedges from the activities and that they felt the activities helped them improve, and part of it may be because, as Sun and Hu (2020) note, these sort of indirect instructor-prepared activities are what students are used to and more familiar with and therefore they may respond well to them. Whatever the case may be, the results of the current study suggest indirect DDL as a viable and effective alternative to direct DDL, and one that instructors may be especially willing to try if they do not feel adequately prepared or trained to implement the more time- and resource- intensive direct DDL. The indirect DDL here was shown to improve participants’ frequency of hedges, though without statistical significance, and, as noted here, participants largely responded quite well to it and found it useful and effective. These benefits were identified even in the asynchronous online context of the current study. In Sun and Hu’s traditional classroom study, the benefits of indirect DDL were even more pronounced. Therefore, there is some evidence that indirect DDL can be an effective alternative to direct DDL, which can often be more challenging to implement, in both online and classroom contexts. 5.4. RQ #4: What are students’ user experiences as they engage with the online DDL modules and online corpora? Researchers have noted that technological problems are often the major source of issues concerning online DDL instruction (Yoon & Hirvela, 2004; Boulton, 2010), with Boulton noting that “effective hands-on [direct] DDL requires considerable training” when in an online modality 119 (p. 3) because of the need to access not only the corpus (i.e., the concordance lines, the texts) but also the interface or website associated with it. That is, learners need to be able to navigate online functions to complete hands-on searches on a corpus, which is something they do not need to do in indirect, hands-off, DDL with material prepared in advance by an instructor. Kennedy and Miceli's (2001) study, for instance, which created tasks informed by the Contemporary Written Italian Corpus for intermediate-level students, found that, while the students felt overall positively towards the corpus incorporation, many faced difficulties in interacting with the online corpus due to such technological issues. Kennedy and Miceli note that inadequate training was likely one of the primary reasons for these difficulties, and remark how, in effective online corpus based DDL, instructors need to first have "adequately equipped them [the students] as "corpus researchers" (Kennedy & Miceli, 2001, p. 81), and that adequate training could likely have helped avoid many of the technological errors (e.g., not knowing what certain functions did, navigation difficulties) that the participants faced. Similarly, Boulton (2010) notes that students will face more success from online DDL if they are already familiar with the existing technology and searching techniques involved in interacting with the interface and data, given that issues with the technology are often highlighted as a barrier to effective corpus use. That is, if students are familiar with the corpus tools or websites, and have some familiarity with how to navigate the interfaces, they are able to devote more of their working memory and attention to noticing linguistic patterns as opposed to struggling to figure out the tools themselves. Many of those sentiments were identified in the current study. Despite positive and encouraging remarks about the effectiveness of the corpora as learning tools, and about how the online activities helped them to see the importance in the functional uses of hedging, many participants noted a variety of technology issues when using the corpora to complete the 120 activities. Indeed, Sun and Hu (2020) remark that, while the indirect DDL group in their study improved their hedging use more significantly than those in the direct DDL group, “the learning tasks in the indirect approach were similar in format to those familiar to the students and therefore required little training and allowed the class time and the students’ attention to be focused on the learning target” (p. 681). That is, there is considerable training and technological requirements associated with direct DDL which may make it more challenging to implement, especially with learners who may have limited technology experience. The participants in the current study, after all, all reported very heavy (daily-level) computer usage, and likely had an easier time picking up on the corpus tools compared to learners with more limited computer use. Even so, they still reported problems with the technology that made their learning less effective. In their open-ended survey responses, and interviews, the following themes were identified by participants in the direct DDL group as patterns of challenges that they faced while interacting with the corpora interfaces: encountering error messages and account issues; the limited help information available on the corpora, language barriers on the websites (i.e., needing to use translation software); navigation difficulty; and a lack of visual cues. In contrast, the only benefits that participants really highlighted concerning the interfaces of the corpora included some helpful information on the homepage, mostly for the COCA corpus, and the graphic representation of some information, namely the frequency bar graphs in both corpora. As mentioned above, many of the Likert-scale survey responses between the current study and Sun and Hu’s were quite similar. Perhaps the starkest differences between the two, regarding the direct DDL group, are in regard to the following prompts: I had some difficulty in performing the searching technique; and I found the searching technique easy to learn. In Sun and Hu's study, only 54% of the direct DDL group participants indicated that they had difficulty 121 in performing the searching technique. In the current study, 69.6% (n=16) of participants reported difficulty, a 15.6 percent point increase. Additionally, in Sun and Hu's study, 100% of the participants in the direct group said that they found the searching technique easy to learn. In the current research, meanwhile, 60.8% percent of participants either slightly disagreed or disagreed with that claim, meaning that the majority of direct group participants found the searching techniques difficult to learn. The differences in these results suggest that the participants (in the direct group) in the current study, when compared to those of Sun and Hu’s, faced greater difficulty in learning how to do the corpus searches (i.e., the searching technique) and ran into a higher frequency of challenges when trying to do so. A possible rationale for these differences, and one supported by previous research, is that, as Boulton (2010) notes, “effective hands-on DDL requires considerable training” when in an online modality (p. 3), and that students will face more success from the online DDL if they are already familiar with the existing technology and searching techniques involved in interacting with the interface and data. Sun and Hu (2020) similarly note that there is a greater “perceived complexity of its implementation” (p. 2) regarding online DDL as opposed to classroom-based DDL, as students must navigate the corpus data itself (e.g., the concordance lines, the texts) in addition to the technology (e.g., online interfaces, online platforms the corpora or corpus-based activities are hosted on). Therefore, the differences in these survey results may be a result of the participants in the current study, unlike the traditional classroom setting of Sun and Hu’s study, having all of their instruction and instructional activities delivered to them via CALL online modules that they were able to complete asynchronously (i.e., a purely online learning context). The participants in the current study therefore had to navigate several things which the participants in Sun and Hu’s study did not, namely: independently accessing the online modules 122 (as hosted on the Canvas LMS), having all instruction delivered to them asynchronously in which they did not interact with the instructor, independently accessing all instructional activities (i.e., greater learner autonomy), and independently accessing the two language corpora online with only minimal asynchronous guidance and instruction from the instructor. The participants, thus, were largely left to autonomously navigate the online modules and online corpora in order to complete the activities, with only short pre-recorded instructional videos introducing them to both. In Sun and Hu’s study, meanwhile, the traditional in-person classroom format meant that the instructor had greater flexibility in, for instance, responding to student questions or challenges and interacting with the students. Therefore, while the asynchronous online learning context in the current study may have afforded participants in the direct group some benefits (e.g., more freedom to independently explore the corpora, more independent learning experiences), it may additionally have had the drawback of not giving the learners as much corpus tool assistance as a traditional in-person class may have been able to. In essence, the findings from the surveys in the current study are aligned with previous research which has focused on DDL in online contexts. Crosthwaite and Steeples’ (2022) examination of a half-year DDL experiment regarding the development of the passive voice in young learners, for instance, showed that several students faced difficulty in navigating the corpus tasks due to issues with the technology, including the following: understanding the steps needed to use the online corpora; how to navigate to certain components or online tools; what the functions of certain tools were; and also an overall confusion regarding the online interface. As shown in the Results chapter, many of those same concerns were echoed by participants in the current study. Similarly, Kennedy and Miceli (2001) note that inadequate training with the online corpus was likely one of the primary reasons for the difficulty that students faced in their study. 123 This includes, they noted, the "need to...master specific [searching] techniques, which are not necessarily intuitive" (p. 87). Indeed, as noted in the Results chapter and briefly here, 69.6% of the direct group participants in the current study reported difficulty in the searching technique, and 60.8% percent disagreed that the searching techniques were easy to learn. Therefore, a hypothesis as to the discrepancy between several Likert-scale items in the survey from the present research and Sun and Hu’s may likely be that, as the current study took place asynchronously and online, the participants lacked the instructor guidance of a traditional classroom setting and needed more preparation and training than the short, pre-recorded videos were able to give them. For the indirect DDL group, challenges related to the interface (i.e., the online modules) that came up in the open-ended survey responses were largely centered around the, at times, confusing sequencing of the activities, and the lack of notifications that they experienced. The benefits they saw in the interface included the ability to submit assignments in multiple formats and the overall course organization. None of the results from the indirect group's Likert-scale survey responses, all of which were quite favorable to the online activities, differed much from that of Sun and Hu's indirect DDL group. However, considering the issues and benefits that the participants noted, it would have been quite easy to alleviate many of those concerns. Canvas, for instance, allows for instructors to set more notifications (i.e., the current study did not, given that participants were completing tasks at different times), and it certainly would have been possible to make the sequencing of activities clearer. One of the benefits they noted, meanwhile, namely the ability to submit assignments in a variety of formats, certainly is a useful component of the LMS which gives participants the flexibility to complete assignments in a variety of ways. Though outside the scope of this paper, functions such as this help make the LMS modules more 124 accessible, both for students who may face technological barriers (e.g., accessing certain tools in restricted countries like China) and those who may face a variety of disabilities (e.g., requiring the use of a screen reader). All of their comments, highlighted in the Results chapter, therefore speak to a need for instructors to consider the most effective way to integrate course content onto LMS platforms, and which features or functions of those platforms may help students navigate the course more easily. 125 CHAPTER 6: CONCLUSION This final chapter concludes with a brief summary of the discussion from above, as well as some broader implications from the current study on the existing research. Additionally, it also includes a focus on the limitations of the current study and recommendations for where future related research should be directed. 6.1. Concluding thoughts An important implication of this study concerns the multiple approaches to DDL identified here (i.e., direct and indirect). As discussed, concerning the results, there were found to be significant frequency gains made among participants in the direct DDL group across time, and non-significant gains for the indirect group. The direct DDL group, however, increased the variety of unique hedges they used (albeit non-significantly), whereas there was a decrease in the indirect group (again, non-significantly). Additionally, while both groups increased the frequencies with which they used modal verb hedges and lexical verb hedges, only the modal verbs for the direct group showed any significance. When both groups were examined, using mixed-effects modeling, together, there was also a significant difference across the Time variable, indicating an overall significant increase in the use of hedges after DDL more broadly. Implications from these results suggest that DDL, largely, was effective at improving the participants’ frequencies of hedges, with the direct DDL proving statistically more effective. Not only did it provide significant increases, but it also, though without significance, increased the participants’ variety of hedges. The Discussion chapter already discussed possible reasons for the disparity between the two groups, though a broader implication is that, as both approaches to DDL were moderately successful in increasing learners’ hedging frequency (significantly so, when both groups were examined together), indirect DDL may still be a viable and effective 126 alternative to direct DDL. Not only because of the results highlighted in the current study, but also because, in Sun and Hu’s (2020) study, it was found to significantly increase their participants’ hedging use. It was noted earlier that the lack of research which examines multiple approaches to DDL can lead some to assume that only direct DDL, with its "considerable training....and technological considerations" (Boulton, 2008, p. 4), is useful for learners, which may cause trepidation by instructors, especially those with limited classroom access to computers or laptops. Also, that the lack of research into different DDL approaches may give the false impression that direct DDL is the only effective method of DDL, which can result in teachers fearing DDL as too intimidating, complex, or time consuming to implement in their classrooms. Indirect DDL does not require, however, “equipping them [the students] as "corpus researchers"” (Kennedy & Miceli, 2001, p. 81), providing students access to technology and possibly corpus accounts, or having them independently conduct their own searches. Rather, instructors prepare material that is corpus-informed or prepared using texts extracted from a corpus. Therefore, it, in many logistical ways, can be far easier to implement in a classroom context and for an instructor to monitor (Sun & Hu, 2020). Therefore, if indirect DDL is presented more frequently in research as an alternative to direct DDL, and if it is found to be moderately successful when compared to direct DDL, as some of the results here and in Sun and Hu’s study suggest, it may be embraced as a viable alternative for instructors who are not adequately ready, or equipped, to utilize direct DDL in their classes. Given that DDL here, taken as a whole, was shown to be effective, and that much previous literature has found DDL broadly speaking to be effective when compared to more traditional teaching approaches (e.g., Lee & Lin, 2018; Shaw, 2011; Yilmaz & Soruc, 2015), this could encourage more classrooms to 127 implement DDL in some capacity, rather than simply the classrooms prepared to embrace hands- on corpus use. A second important implication of this study concerns drawing an awareness of what participants, or learners, saw as the benefits and shortcomings of those different approaches. Specifically, as the results show, major themes highlighted by the participants in the direct DDL group included, for several examples, the following: being able to sift through large amounts of authentic texts; being able to see frequencies of words across different text types, registers, or learner English populations; being able to see keywords surrounded by context; and the ability to access full texts. Major drawbacks highlighted by the direct DDL group included the following: it being hard to identify the important functions, or the role of certain functions; it being difficult to find examples of certain usages of words, the authentic texts in the corpora being difficult to understand; the lack of definitions of words in an easily searchable spot; and the compilation of the corpora being skewed. Benefits identified by the indirect DDL group included, for instance, regarding the module: it helped them learn a variety of hedges; it helped them see the importance of hedging as a rhetorical strategy; it helped them write, as they saw it, more "professional" or "academic"; and that it helped them downtown or soften their writing. Drawbacks included: that it only provided them a limited, surface-level, review of hedges; that the activities were too simple and basic and not challenging enough; that there were not enough opportunities provided for more extensive writing; and that there was limited peer- or instructor- engagement or interaction. The implications from these findings, as mentioned in the Discussion chapter, are that indirect DDL, which is hands-off and instructor led, may possibly sometimes result in learning experiences that are not rich, challenging, or extensive enough when compared to direct DDL, at 128 least in an online context. Certainly, this is not true of all indirect DDL, but it is perhaps possible that if the indirect group participants who voiced such complaints were in the direct DDL group, they would have had more opportunities for free range exploration of the corpora and could have taken more charge in autonomous and independent learning. In doing so, perhaps they could have adapted the activities to suit the level of challenge that they wanted, or at least explore additional usage patterns and examples. As they were in the indirect DDL group, however, they merely had to complete the tasks as dictated and prepared by the instructor, with no access to the corpora. The lack of learner autonomy and free-range exploration of the corpora is possibly one hypothesis for why participants in the indirect DDL group found the activities to often be simple, boring, or limited, which are not complaints that came up by participants in the direct DDL group. However, it is worth noting that the majority of participants in the indirect DDL group still maintained that the activities helped them improve their use of hedges, taught them about the importance of hedges, and, to some degree, helped improve their academic writing. Implications from the direct DDL group would concern the need for students to be adequately trained to use any potential corpus tools prior to having them complete activities using them. This training should likely include technological training, as that certainly was an issue which was raised by participants in the current study, but perhaps should also include training students to understand what language corpora are. Many of the drawbacks highlighted above (e.g., the corpora not providing simple definitions of words, the authentic texts being difficult to comprehend, the compilation of the corpora including sources that the participants were unfamiliar with) all speak to a misunderstanding of how one is supposed to learn with a corpus as opposed to, say, a dictionary. With corpora, learners are expected to notice patterns in frequencies, be exposed to a wide variety of authentic texts, and engage in more free-range 129 exploration to help them notice usage patterns. With dictionaries, often, learners just seek specific information about words, and the words are often presented in isolation with little, or artificial, usage examples. Therefore, practitioners who utilize corpus tools in their DDL instruction should be careful to explain to students what corpora are, and what learning with them looks like. If students have certain correct expectations about corpora going into DDL instruction (i.e., that it would look different from a dictionary), or if they have been trained on strategies for navigating large amounts of language usage, then perhaps they would be able to focus more on picking up effective DDL strategies (e.g., independent searches, comparisons across contexts, examining concordance lines for usage differences) for optimizing the benefits they get from the tools. Lee et al. (2020), for instance, in their study of 35 South Korean undergraduate students who underwent DDL instruction, also utilizing the COCA corpus, found that there were positive correlations between higher vocabulary proficiency and students who were able to "utilize DDL-focused strategies" (p. 6) while engaging with the DDL. Therefore, in addition to the technological training, mentioned below, practitioners should also be sure to make sure that their students are equipped, or prepared, to learn those effective DDL-focused strategies in order to maximize the benefits they get from the DDL instruction. Lastly, the emphasis on the technological issues faced by the participants in this study, when interacting with the corpora and also the modules, can possibly help inform practitioners or researchers who, as in the current study, hope to construct online DDL for their learners. As mentioned before, many of the participants noted that a lack of instructions (beyond those provided by the instructor), a lack of intuitive help cues (e.g., getting information by hovering over buttons), and confusion over the functions and the error messages, all hindered their ability to utilize the corpus tools with maximum efficiency. Highlighting these concerns can possibly 130 help practitioners identify what to focus on in their pre-DDL trainings. Perhaps, for instance, practitioners can create their own “help” or “instructions” sheets for corpora which, like the ones featured here, either do not have them or do not present them in a way that is satisfactory and accessible to students. If not creating their own, practitioners can at least possibly spend some time in training to make sure every student knows how to access any available help or help documentation (which may be housed on different areas of the websites then the searching page) prior to letting students engage in their corpus searches. As noted by several participants in the current study, frustration with the technology, and the searching techniques, led them to oftentimes not know whether they were doing something wrong, if their Internet was messing up, or if the corpora were experiencing issues. One participant even noted that “without proper guidance, students might pick up incorrect ideas about how to use something or how to write/talk”, illustrating an important point, which is that if students are prevented from doing correct and accurate searches, they may get the wrong lessons or takeaways from the corpus activities. Therefore, practitioners should be careful that adequate training or help is provided if students are new to using such corpus tools. They should also make sure that they have reviewed the technological requirements for each tool prior to attempting to use it, as some tools, like COCA for instance, have limitations of usage depending on whether students have accounts or not. COCA also frequently presents error messages, even with correct searches, that require users to backtrack to the previous page, reset their search conditions, and try again. Knowing these limitations beforehand can help instructors explain errors or limitations to students as they come up during instruction and help clarify things, so students don’t take away wrong impressions from the data or turn frustrated and demoralized with using the corpus tools. 131 6.2. Limitations of the current study There are several limitations of the current study that need to be addressed, which will lead to a discussion of recommended directions for future similar research. First, while the number of participants featured in the current study (n=46) consisted of more than those featured in Sun and Hu’s (2020) study (which had n=24, n=13 in one treatment group and n=11 in the other), partially due to the affordances of having participants engage with the instruction online, there still needs to be caution when generalizing the results. This is true due to the limited number of participants, but also because, as mentioned earlier, Asian languages and cultures are not monolithic and EFL learners from specific L1s, “with their own [linguistic] characteristics and culture-bound” traits, are likely to use hedges “differently from other [Asian EFL population] writers” (Tran & Tang, 2022, p. 120). The current study examined only Chinese EFL learners, which means that generalizations more broadly to Asian EFL writers or NNES writers should be done thoughtfully to make sure that such comparisons note the linguistic, social, and cultural, variations that exist between different learner English populations. Future research which adopts a more comparative design, such as possibly examining the same DDL instruction on comparable speakers (e.g., similar ages, similar educational backgrounds) from different Asian L1s, could potentially help examine whether similar DDL instruction is similarly effective across different learner English populations. However, for the current study, the results should be viewed as representative only of Chinese EFL learners, especially given that, as noted in the Methods chapter, the activities were tailored around hedging patterns and lexical items that Chinese EFL writers in specific were shown to struggle with compared to NES writers. Second, and a very timely limitation, is that the increasing prevalence of AI tools (e.g., ChatGPT) means that students now have new, widely accessible, tools that can assist them in 132 their writing. In fact, with these newer AI tools, students can have them largely write entire essays for them, or even help complete instructional tasks for them. The current study required participants in the direct DDL group to screen capture themselves while completing the activities, and so it can safely be said that no AI use was detected during those activities. However, participants did not have to screen record themselves during the writing tasks. It, therefore, is possible that select participants utilized AI tools, or additional help (e.g., e- dictionaries) while completing their writing tasks. I will note, however, that none of the submitted essays were found to be artificially perfect or sophisticated. Most of the essays had several errors, and none of them seemed to reflect what an essay would look like if, for instance, a user asked ChatGPT to write them an essay about that prompt. Also, none of the time records from the participants (i.e., the amount of time they spent on the Canvas system, automatically collected and reported by Canvas) seemed to reflect rushing through any tasks and were all fairly consistent. Additionally, even if participants used some additional help, such as spell checkers or e-dictionaries, the kind of help they were likely to receive from them (e.g., spelling words correctly, grammar correction, looking up vocabulary items) is unlikely to have significantly impacted their use of hedging throughout. However, in hindsight, it is a limitation of the current study that participants did not record their screens during the writing tasks as well, to better control for the conditions in which they completed those tasks. That being said, learner autonomy and independence are often key components of online learning, especially with asynchronous online learning, and this limitation speaks to a larger discussion concerning safeguards against AI in such online learning contexts that is outside the scope of this paper. Third, a limitation which will be discussed further in the next sub-header concerning recommendations for future research, is that the current study took place only during a four-week 133 instructional period. Only participants’ pre- and post- instruction essays were examined. Therefore, it is impossible to know if the gains made in hedging frequency will stick beyond their participation in this four-week course, and also whether they will continue to consult corpus tools for their language learning purposes. As noted in the next sub-header, research which examines participants over a longer period of time, and examines more writing samples spread across that time, would be helpful in identifying whether such gains have any true staying power. 6.3. Recommendations for future related research Regarding recommendations for future related research, there still exists a need for DDL research which, as in the current study or Sun and Hu’s (2020) work, examines multiple approaches to DDL. As noted earlier, there remains a gap in the existing DDL literature, specifically regarding the differentiation between two different approaches to DDL: direct and indirect. Chambers (2010) refers to these approaches as indirect access, where learners learn about language use by studying concordance lines that are prepared ahead of time by an instructor, and direct access, in which learners themselves are using online corpus tools or software to actually conduct their own analyses and searches. Sun and Hu (2020) note that “most existing studies have compared some form of DDL with traditional teaching but have not examined or compared different approaches to DDL” (p. 2). Further, Chen and Flowerdew (2018) remark there to be a “a need for more descriptions of different [DDL] approaches” (p. 356), with a synthesis article by Boulton (2008), which examined 39 DDL research articles, similarly finding that only eight of them identified an indirect DDL approach as a possible alternative to direct DDL. As discussed, the problem with studies which do not make their readers (i.e., practitioners or other researchers) aware of differing approaches is that an exclusive focus on direct DDL may lead some to believe that it is the only approach useful for learners. 134 Boulton (2010) notes that direct DDL, by its nature, involves "considerable training....and technological considerations" (p. 4), which may lead instructors to be cautious to try to implement it in their classrooms, especially those in schools with limited classroom access to computers or laptops. Sun and Hu (2020) also note that the lack of research into different DDL approaches may give the false impression that direct DDL is the only appropriate method of DDL. Presenting multiple DDL approaches, and examining their effectiveness, may help introduce practitioners to approaches that are more suitable, or less intimidating and time consuming, for their classrooms, such as indirect approaches. Another helpful future direction would be research which takes a more longitudinal approach to examining DDL. In the current study, participants’ pre- and post- instructional writing samples were analyzed. Even while there were encouraging signs of increased hedging frequency, however, there is no way of knowing whether these gains would be sustained over a longer period of time. Future research which continues to examine participants’ writing samples across, possibly, multiple delayed post-instruction essays could help evaluate whether such gains are short term and temporary or if they have some staying power. A rare example of longitudinal research which examined the effect of instruction on students' hedges, among other linguistic features, is Crosthwaite and Jiang (2017), which explored the longitudinal development of L2 academic ‘stance’ features (i.e., including hedges, among other features such as boosters and attitude markers) from an English for Academic Purposes course in Hong Kong. Over a semester's worth of instruction was conducted, with pre-, mid- and post-instruction writing tasks. Results showed significant longitudinal differences in the frequencies of the devices, with a rise in the use of hedging and a decrease in boosting and self-mention devices, leaving students, at the end, with a more cautious and careful writing style. Crosthwaite and Jiang's study would 135 seem to suggest that there is the potential for longer-term gains in hedging use following targeted instruction, which is an encouraging finding. Similar research which occurs over the course of an entire semester, or longer, would be encouraged and welcome. Similarly, and just as important, longitudinal research such as that could also see whether participants who say that they plan to continue to use the corpora as learning tools actually do so. In the current study, for instance, 73.9% (n=17) of the participants in the direct DDL group indicated on their Likert-scale survey responses that they would continue to reference corpus tools in the future as they continue their language learning. It would be incredibly beneficial to have longitudinal research which can help illustrate how often participants who express such desires actually continue to consult corpus tools over an extended period of time, and why (or why not). Such findings could potentially help practitioners and researchers consider strategies to more effectively ensure long-term corpus use among language learners, possibly by developing tasks or activities which highlight the long-term benefits to them. This could also potentially help highlight the advantages of corpora over dictionaries to participants. In the current study, also in the Likert-scale survey responses, only a slim majority of participants in the direct DDL group, 52.2% (n=12), said that they found the corpus tools to be more useful than a dictionary. This may be, at least partially, because participants view dictionaries as resources that are consistently helpful over time, rather than just tools to use during certain activities or tasks. Being able to help convince learners that corpus tools can be similarly beneficial as consistent language references that they return to over time may help learners pick up skills to more effectively use large sources of authentic texts to help with their vocabulary or overall language learning. A final recommendation for future research concerns research which continues, as in the current study, to contribute to what Crosthwaite and Steeples (2022) note as the more limited 136 research which examines DDL in modalities outside of the traditional classroom, such as online or blended contexts. Much of previous DDL research, as noted, is centered around in-person classroom contexts (i.e., in which there is often a strong instructor presence and more limited learner autonomy or free-range exploration). There can be benefits to such learning contexts. The peer- or instructor- interactions that can occur during these classroom-based corpus tasks can contribute to the construction of knowledge and the benefits of the DDL (Crosthwaite & Steeples, 2022). This is why, as research has noted, most instructors often take a guided approach to DDL in which the corpus work with students is often first approached as an apprenticeship by which the students learn by example and by their experience of working with the instructor (Kennedy & Miceli, 2001; Tribble & Jones, 1997; Stevens, 1991). In blended and online contexts, there is often a greater “perceived complexity of its implementation” (Sun & Hu, 2020, p. 2), and students must navigate the corpus data itself (e.g., the concordance lines, the texts) as well as online interfaces and online learning contexts (e.g., learning management systems, online modules). Additionally, there is often, especially in asynchronous online contexts, less interaction with peers or an instructor. However, the benefits include giving students a great deal of learner autonomy in completing the tasks rather independently, which is why Boulton notes that “effective hands-on DDL requires considerable training” when in an online modality (p. 3). Chambers (2010) additionally notes that online DDL implies a level of active participation, as the learner interacts with the online corpus and associated interfaces, throughout their learning process, in a way that dictionaries, grammar reference texts, and teacher-led activities, cannot often provide. Ludwig and Tassinari (2021) further highlight the benefits of such online instruction, such as increased learner autonomy, the ability for more independent learning and exploration, and the flexibility that online learning can offer. However, they also note, alongside 137 O’Keefe (2021), that these benefits can come at the cost of the lack of instructor control over learning outcomes, the lack of peer- or instructor- interaction, and a lack of instructor control over how students are interacting with the corpus tools one on their own. Therefore, online DDL comes with both a variety of advantages and potential drawbacks, as also illustrated in the survey and interview results from the current study. Future research which continues to examine these benefits and drawbacks can help inform practitioners about best practices with integrating DDL into blended or online learning, and can help draw their attention to possible challenges, difficulties, or student concerns. 138 REFERENCES Afshar, H.S., Moradi, M.R., & Hamzavi, R. (2014). Frequency and Type of Hedging Devices Used in the Research Articles of Humanities, Basic Sciences and Agriculture. Procedia - Social and Behavioral Sciences, 136, 70-74. Allison, Desmond. (1995). Assertions and Alternatives: Helping ESL Undergraduates Extend Their Choices in Academic Writing. Journal of Second Language Writing, 4(1), 1-15. Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. John Wiley & Sons. Beck, H. (2004). Teaching creative writing online. New Writing, 1(1), 23–36. https://doi.org/10.1080/147907204 08668189 Bernaisch, T. (2021). Hedges and Gender in the Inner and Expanding Circle. In T. Bernaisch (Ed.), Gender in World Englishes (Studies in English Language, pp. 94-120). Cambridge: Cambridge University Press. doi:10.1017/9781108696739.005 Bernardini, S. (2004). Corpora in the classroom: An overview and some reflections on future developments. In J. Sinclair (Ed.), How to use corpora in language teaching (pp. 15–36). Amsterdam, NL: John Benjamins. Blake, R. J. (2009). The use of technology for second language distance learning. The Modern Language Journal 93.1, 822–835. Bloor, M., & Bloor, T. (1991). Cultural expectations and socio-pragmatic failure in academic writing. In P. Adams, B. Heaton, & P. Howarth, Socio-cultural issues in English for academic purposes (pp. 1-12). London: MacMillan Publishers Ltd. Boulton, A. (2008). But where’s the proof? The need for empirical evidence for data-driven learning. In M. Edwardes (Ed.), Proceedings of the BAAL Annual Conference 2007 (pp. 13-16). London: Scitsiugnil Press. Boulton, A. (2009). Testing the limits of data-driven learning: Language proficiency and training. ReCALL, 21(1), 37-54. Boulton, A. (2010). Data-driven learning: Taking the computer out of the equation. Language Learning, 60, 534–572. Boulton, A. (2010). Language awareness and medium-term benefits of corpus consultation. In Gimeno Sanz, A. (Ed.), New trends in computer-assisted language learning: Working together (pp. 39–46). Madrid, Spain: Macmillan ELT. Boulton, A. (2017). Corpora in language teaching and learning. Language Teaching, 50, 483– 506. 139 Brown. D., & Brown. L. (1982). Some linguistic differences in the written English of Chinese and Australian students. Language Learning and Communication, 1,39- 49. Can, T. (2009). Learning and teaching languages online: A constructivist approach. Novitas- ROYAL (Research on Youth and Language), 3(1). Connor, U., & Lauer, J.M. (1988). Cross-cultural variation in persuasive student writing. In A. C. Purves (Ed.), Contrastive rhetoric (pp. 138-159). Newbury Park, CA: Sage Publications. Chambers, A. (2010). What is data-driven learning? The Routledge Handbook of Corpus Linguistics. Chen, H.-I. (2010). Contrastive learner corpus analysis of epistemic modality and interlanguage pragmatic competence in L2 writing. Arizona Working Papers in SLA & Teaching, 17, 27–51. Chen, M., & Flowerdew, J. (2018). A critical review of research and practice in data-driven learning (DDL) in the academic writing classroom. International Journal of Corpus Linguistics, 23, 335–369. Choi, Y.H., & Ko, M. (2005). Hedging in EFL Academic Writing of Korean Postgraduates. English teaching, 60, 3-27. Cobb, T., & Boulton, A. (2015). Classroom applications of corpus analysis. In D. Biber & R. Reppen (Eds.), The Cambridge handbook of English corpus linguistics (pp. 478–497). Cambridge: Cambridge University Press. Corino, E., & Onesti, C. (2019). Data-Driven Learning: A Scaffolding Methodology for CLIL and LSP Teaching and Learning. Frontiers in Education. Cowit, N. Q., & Barker, L. J. (2022). Student Perspectives on Distraction and Engagement in the Synchronous Remote Classroom. In Digital Distractions in the College Classroom (pp. 243-266). IGI Global. Crocco, E., & Culasso, F. (2021). Synchronous and Asynchronous Approaches in Accounting Education: A qualitative overview of their benefits, drawbacks and challenges. The Impact of COVID-19 on Accounting, Business Practice and Education, 143-153. Croft, N., Dalton, A., & Grant, M. (2010). Overcoming isolation in distance learning: Building a learning community through time and space. Journal for education in the built environment, 5(1), 27-64. Crossley, S.A., Roscoe, R., & McNamara, D.S. (2014). What is successful writing? An investigation into the multiple ways writers can write successful essays. Written Communication, 3, 184–215. 140 Crosthwaite, P. (2017). Retesting the limits of data-driven learning: Feedback and error correction. Computer Assisted Language Learning, 30, 447–473. Crosthwaite, P. (2020). Taking DDL online: Designing, implementing and evaluating a SP OC on data-driven learning for tertiary L2 writing. Australian Review of Applied Linguistics, 43(2), 169–195. Crosthwaite, P., & Jiang, K. (2017). Does EAP affect written L2 academic stance? A longitudinal learner corpus study. System, 69, 92-107. Crosthwaite, P.R., & Steeples, B. (2022). Data-driven learning with younger learners: exploring corpus-assisted development of the passive voice for science writing with female secondary school students. Computer Assisted Language Learning. Daymont, T., Blau, G., & Campbell, D. (2011). Deciding between traditional and online formats: Exploring the role of learning advantages, flexibility, and compensatory adaptation. Journal of Behavioral and Applied Management, 12(2), 156. Ellis, N. C. & Wulff, S. (2014). Usage-based approaches to SLA. In B. VanPatten & J. Williams (Eds.)., Theories in second language acquisition: An introduction (2nd Edn), chapter 6. NY: Routledge. Emmerson, D. (2019). The use of synchronous and asynchronous technological tools for socio- constructivist language learning. The Journal of Language Learning and Teaching, 9(2), 1-6. Endsley, M.R., Bolte, B., & Jones, D.G. (2003). Designing for Situation Awareness : An Approach to User-Centered Design. Flowerdew, L. (2015). Data-driven learning and language learning theories: Whither the twain shall meet. In Leńko-Szymańska, A. & Boulton, A. (Eds.), Multiple affordances of language corpora for data-driven learning (pp. 15–36). Amsterdam, Netherlands: John Benjamins. Gamorot, J. J., Balighot, R., Circulado, S. J., & Loquias, A. (2022). Asynchronous Online Classes through the Lenses of Students’ Experiences: Opportunities, Drawbacks and Proposed Measures. Asynchronous Online Classes through the Lenses of Students Experiences: Opportunities, Drawbacks and Proposed Measures, 112(1), 15-15. Geluso, J., & Yamaguchi, A. (2014). Discovering formulaic language through data-driven learning: Student attitudes and efficacy. ReCALL, 26, 225–242. Hinkel, E. (2002). Second language writers’ text. New York, NY: Routledge. Hinkel, E. (2005). Hedging, inflating, and persuading in L2 academic writing. Applied Language 141 Learning, 15, 29–53. Hu, Z., Brown, D., & Brown, L. (1982). Some linguistic differences in the written English of Chinese and Australian students. Language Learning and Communication, 1, 39-49. Hu, G., & Cao, F. (2011). Hedging and boosting in abstracts of applied linguistics articles: A comparative study of English- and Chinese-medium journals. Journal of Pragmatics, 43, 2795–2809 Hyland, K. (1998) Hedging in Scientific Research Articles. Amsterdam: John Benjamins. Hyland, K. (2005). Stance and engagement: A model of interaction in academic discourse. Discourse studies, 7(2), 173-192. Hyland, K., & Milton, J. (1997). Qualification and certainty in L1 and L2 students’ writing. Journal of Second Language Writing, 6, 183–205. Johns, T. (1988). Whence and whither classroom concordancing. In T. Bongaerts, P. De Haan, S. Lobbe, & H. Wekker (Eds.), Computer applications in language learning (pp. 9-27). Dordrecht, The Netherlands: Foris. Kamimura, T. & Oi, K. (1998). Argumentative strategies in American and Japanese English. World Englishes, 17(3),307-323. Karunarathna, Buddhima. (2020). Improving the Use of Language Hedges in Academic Writing through Reading Journal Articles. Advances in Language and Literary Studies. II. 17-23. 10.7575/aiac.alls.v.11n.3p.17. Kennedy, C., & Miceli, T. (2001). An evaluation of intermediate students’ approaches to corpus investigation. Language Learning & Technology, 5, 77–90. Khamesian, M. (2015). Rhetorical impact through hedging devices in the ‘Results and discussion’ part of a civil engineering research article. Advances in Language and Literary Studies, 6(6), 183-186. Kostere, S., & Kostere, K. (2021). The Generic Qualitative Approach to a Dissertation in the Social Sciences: A Step by Step Guide. Routledge. Lamy, M.-N. (2013). Distance CALL Online. In M. Thomas, H. Reinders & M. Warschauer (eds.), Contemporary Computer-Assisted Language Learning. London: Continuum, 141– 158. Lakoff, G (1972, April 14-16). Hedges: a study in meaning criteria and the logic offuzzy concepts. Papers from the eighth regional meeting Chicago linguistic society (pp. 183- 228). 142 Lee, P., & Lin, H. (2019). The effect of the inductive and deductive data-driven learning (DDL) on vocabulary acquisition and retention. System, 81, 14-25. Lee, H., Warschauer, M., & Lee, J. H. (2020). Toward the establishment of a data‐driven learning model: Role of learner factors in corpus‐based second language vocabulary learning. The Modern Language Journal, 104(2), 345-362. Lei, S. A., & Lei, S. Y. (2019). Evaluating benefits and drawbacks of hybrid courses: Perspectives of college instructors. Education, 140(1), 1-8. Li, Z., Lou, X., Chen, M., Li, S., Lv, C., Song, S., & Li, L. (2023). Students’ online learning adaptability and their continuous usage intention across different disciplines. Humanities and Social Sciences Communications, 10(1), 1-10. Lida, E. (2008). Hedges in Japanese English and American English medical research articles (Order No. AAIMR32526). Available from Linguistics and Language Behavior Abstracts (LLBA). (85714455; 201002642). Lohr, L (2000). Designing the instructional interface, Computers in Human Behavior 16 161- 182. Ludwig, C., & Tassinari, M. G. (2023). Foreign language learner autonomy in online learning environments: the teachers’ perspectives. Innovation in Language Learning and Teaching, 17(2), 217-234. Markkanen, R., & Schröder, H. (1997). Hedging: A Challenge for Pragmatics and Discourse Analysis. Milton, J.. & Hyland, K. (1999). Assertions in students’ academic essays: A comparison of L1 and L2 writers. In R. Berry, B. Asker, K. Hyland, & M. Lam (Eds.), Language Analysis, Description and Pedagogy.(pp. 147-161). Hong Kong: HKUST. Min, S., Paek, J.K., & Kang, Y. (2019). Exploring the Use of Hedges and Stance Devices in Relation to Korean EFL Learners’ Argumentative Writing Qualities. ENGLISH TEACHING. Mishan, F. (2013). Demystifying blended learning. In B. Tomlinson (Ed.), Developing materials for language teaching (pp.207–223). Bloomsbury. Mizumoto, A., & Chujo, K. (2015). A meta-analysis of data-driven learning approach in the Japanese EFL classroom. English Corpus Studies, 22, 1-18. Nielsen, J. (1993). Usability engineering. The Computer Science and Engineering Handbook. O'Keeffe, A. (2021). Data-driven learning – a call for a broader research gaze. Language Teaching, 54(2), 259-272. doi:10.1017/S0261444820000245 143 O’Keeffe, A. (2021) ‘Data-driven learning, theories of learning and second language acquisition, in search of intersections’, in P. Pérez-Paredes and G. Mark (Eds) Beyond Concordance Lines: Corpora in Language Education, Amsterdam: John Benjamins, 35-55. O'Sullivan, Í. (2007). Enhancing a process-oriented approach to literacy and language learning: The role of corpus consultation literacy. ReCALL, 19(3), 269–286. Petersen, D. (2007). Usability theory, practice and evaluation for learning objects. Learning objects: Applications, implications, & future directions, 337-370. Philp, J., Adams, R., & Iwashita, N. (2013). Peer interaction and second language learning. Routledge. Prior, D. D., Mazanov, J., Meacheam, D., Heaslip, G., & Hanson, J. (2016). Attitude, digital literacy and self efficacy: Flow-on effects for online learning behavior. The Internet and Higher Education, 29, 91-97. Rayson, P. and Garside, R. (2000). Comparing corpora using frequency profiling. In proceedings of the workshop on Comparing Corpora, held in conjunction with the 38th annual meeting of the Association for Computational Linguistics (ACL 2000). 1-8 October 2000, Hong Kong, pp. 1 - 6. Rezanejad, A., Lari, Z., & Mosalli, Z. (2015). A cross-cultural analysis of the use of hedging devices in scientific research articles. Journal of Language Teaching and Research, 6(6), 1384-1392. doi: http://dx.doi.org/10.17507/jltr.0606.29 Riwayatiningsih, R., & Sulistyani, S. (2020). The implementation of synchronous and asynchronous e-language learning in EFL setting: A case study. Jurnal BASIS (Jurnal Bahasa dan Sastra Inggris), 7(2), 309-318. Salager-Meyer, F. (1994). Hedges and textual communicative function in medical English written discourse. English for Specific Purposes, 13(2), 149-170. Shaw, E. M. (2011). Teaching vocabulary through data-driven learning. Brigham Young University. Skylar, A. A. (2009). A comparison of asynchronous online text-based lectures and synchronous interactive web conferencing lectures. Issues in Teacher education, 18(2), 69-84. Stevens, V. (1991). Classroom concordancing: Vocabulary materials derived from relevant, authentic text. English for Special Purposes Journal, 10, 35-46. Sun, X., & Hu, G. (2020). Direct and indirect data-driven learning: An experimental study of hedging in an EFL writing class. Language Teaching Research. 144 Swan, K. (2002). Building Learning Communities in Online Courses: the importance of interaction. Communication & Information, 2(1). https://doi.org/10.1080/146363102 200000501 Thuy, T. (2018). A Corpus-Based Study on Cross-Cultural Divergence in the Use of Hedges in Academic Research Articles Written by Vietnamese and Native English-Speaking Authors. Soc. Sci., 7, 70. https://doi.org/10.3390/socsci7040070 Tran, T. Q., & Tang, T. B. (2022). Hedging in the results and discussion section of English applied linguistics research articles by Vietnamese and foreign writers. Journal of Language Teaching and Research, 13(1), 119-124. doi: https://doi.org/10.17507/jltr.1301.14 Tribble, C., & Jones, G. (1997). Concordances in the classroom: Using corpora in language education. Houston, TX: Athelstan Vyatkina, N. (2016). Data-driven learning of collocations: Learner performance, proficiency, and perceptions. Language Learning & Technology, 20, 159–179. Vyatkina, N. (2016). Data-driven learning for beginners: The case of German verb-preposition collocations. ReCALL, 28(2), 207-226. doi:10.1017/S0958344015000269 Wang, Zhou. 2008. A corpus-based contrastive study on hedging in Chinese and English research article abstracts. Journal of Huazhong University of Science and Technology (Social Sciences Edition) 22(6). 59–63. Winardi, A. (2009). The use of hedging devices by American and Chinese writers in the field of applied linguistics. Sastra Inggris Journal, 8(3), 228-237. White, C. (2006). Distance learning of foreign languages. Language Teaching, 39, 247 - 264. Yang, Y. (2003). A contrastive study of hedges in English and Chinese academic discourse. Unpublished MA thesis. China: Changchun, Jilin University. Yang, Y. (2013). Exploring linguistic and cultural variations in the use of hedges in English and Chinese scientific discourse. Journal of Pragmatics, 50, 23–36. Yang, H. (2006). A comparative study of scientific hedging by Chinese writers and English writers. Language Education papers, 3(3), 58-62. Yao, G. (2019). Vocabulary learning through data-driven learning in the context of Spanish as a foreign language. Research in Corpus Linguistics, 7, 18-46. Yılmaz, E., & Soruç, A. (2015). The use of concordance for teaching vocabulary: A data-driven learning approach. Procedia-Social and Behavioral Sciences, 191, 2626-2630. 145 Yoon, H., & Hirvela, A. (2004). ESL student attitudes toward corpus use in L2 writing. Journal of Second Language Writing, 13, 257-283. Zuck, J. G. and L. V. Zuck. (1986). Hedging in news Writing. In A. M. Cornu, Vanparijs, J. Delahaye, M. & L. Baten. (eds.) Beads or Bracelets? How do we Approach LSP?. Oxford: Oxford University Press, 172-180. 146 APPENDIX A: HYLAND AND MILTON’S (1997) LEXICAL ITEMS Table 9 The list of hedging lexical items from Hyland and Milton (1997) used for my QRP, and analysis of my pilot study and this dissertation work Part of speech Modal verbs Adjectives Nouns Lexical verbs Adverbs Included lexical items could, couldn’t, may, might, should, shouldn’t, would, wouldn’t, will, won’t always, (not) always, apparent, certain, a certain extent, clear, evident, possible, probable claim, doubt, estimate, evidence, possibility appear, argue, assume, believe, claim, doubt, estimate, expect, indicate, know, predict, presume, propose, seem, speculate, suggest, suppose, tend, think about, actually, almost, apparently, approximately, around, certainly, clearly, definitely, doubtless, essentially, evidently, frequently, generally, in fact, indeed, largely, likely, never, normally, obviously, of course, often, perhaps, possibly, presumably, probably, quite, rarely, relatively, sometimes, surely, undoubtedly, usually * Compiled by Hyland and Milton (1997) from Holmes’ (1983 & 1988) and Kennedy’s (1987) analyses of the Brown and LOB corpora; Hyland’s (1996a) research article corpus; and literature on grammars and modality including Coates (1983, 1987); Leech & Svartvik (1994); Lyons (1977); Perkins (1983); Skelton (1988); Quirk et al. (1972). 147 APPENDIX B: RECRUITMENT SURVEYS Table 10 Demographic/Recruitment Surveys (protocol, will be hosted on Qualtrics) Questions Notes The survey will start with the consent form completed for the IRB. What is a good e-mail address to reach you at regarding your participation in this research? What e-mail address would you like the compensation to be sent to after completing all tasks for this research? How would you like to receive your compensation upon completing this study? [PayPal, Venmo, Amazon E-gift Card, Cashapp] What is your age? What is your student status? What year are you into your studies? What is your major? What is your gender? How much experience do you have living in a country (e.g., America) where English is the dominant language spoken. Are you able to provide any English standardized testing scores, such as the College English Test (CET), IELTS, or Test for English Majors (TEM) scores? If so, what are they and when are they from? Thanks for your responses to the questions above. The researcher of the study will be in contact soon. Open-ended Open-ended Multiple Choice Open-ended Undergraduate; Graduate (M.A.); Graduate (PhD) First year; Second year; Third year; Fourth year; Fifth year or more Open-ended Male; Female; Non-binary/Third gender; Prefer not to say None; Less than one year; One to two years; Three years or more Open-ended Prompt they will see after completion. 148 APPENDIX C: PRE- AND POST-INSTRUCTION WRITING PROMPTS The following are the prompts that students will see during their pre- and post-instruction writing tasks. As mentioned, two prompts will be used and will be rotated in the following way: for both groups, half of the students will have Prompt #1 and half will have Prompt #2 for the pre- instruction writing task, and then those will switch for the post-instruction writing task. The writing tasks will be completed on an open text box provided on the Canvas modules. Prompt #1: Please read the essay prompt below. After, you will be given an hour to write a short argumentative essay (of around 500 words) in which you will write about your opinion about the topic. Please write your essay in the text box provided once you start this assignment. Essay Prompt: Do you think that information in scientific research, business and the academic world should be made publicly accessible? Prompt #2: Please read the essay prompt below. After, you will be given an hour to write a short argumentative essay (of around 500 words) in which you will write about your opinion about the topic. Please write your essay in the text box provided once you start this assignment. Essay Prompt: Do you think the construction of new railway lines or improvement of existing public transport should be prioritized? 149 APPENDIX D: ACTIVITY #1 Table 11 The following activity will occur on the Canvas LMS, which has a function in which students can use an online highlighter to “draw” a highlight over certain parts of an uploaded document. Instructions: You are presented below with eight different pairs of sentences taken from student essays. Please compare them and identify the language items that make them different. Use the online highlighter function to highlight the language items that make each pair different. Please record your computer screen as you complete the following task(s). You may record your screen with the following free website (https://www.apowersoft.com/free-online-screen- recorder) or may use a different program to do so of your choosing, such as Zoom. Original Each interaction she has with other human beings falls almost immediately, I think, to needing money. Hedges Removed Each interaction she has with other human beings falls immediately to needing money. This is where repairs and corrections might enter the picture, providing speakers with the ability to change what he or she just said, to a certain extent. While I do believe that it is a good strategy to immerse Carlos in print, requiring him to complete a book report each night will likely have a negative effect and may further frustrate him. This is where repairs and corrections enter the picture, providing speakers with the ability to change what he or she just said. While I do believe that it is a good strategy to immerse Carlos in print, requiring him to complete a book report each night will have a negative effect and further frustrate him. Carwin’s education, which is evident to Clara and the rest of the family when they meet him, means that he was probably aware of social norms, but might have chosen willfully not to follow them. Carwin’s education, which is evident to Clara and the rest of the family when they meet him, means that he was aware of social norms, but has chosen willfully not to follow them. Since starting work, I think I can say that I am more confident, and probably manage my time a lot better than any of my classmates. I also seem to have gained a certain level of respect from people that I wasn't aware of before. Since starting work, I can say that I am more confident, and manage my time a lot better than any of my classmates. I also have gained respect from people that I wasn't aware of before. 150 Table 11 (cont’d) Having a part-time job may be one of the best things college students can do to help prepare themselves for their futures, and I believe can help them with their future careers. Having a part-time job is one of the best things college students can do to help prepare themselves for their futures, and can help them with their future careers. It is extremely interesting to talk to people who are not from the same country as you because they might have a very different culture than you, and I would argue a very different way of thinking. College students these days probably have more personal and tuition expenses than those in the generation of our parents, and so we might need to help our parents out by getting a part-time job to help invest in our education as well. It is extremely interesting to talk to people who are not from the same country as you because they have a very different culture than you, and a very different way of thinking. College students these days have more personal and tuition expenses than those in the generation of our parents, and so we need to help our parents out by getting a part-time job to help invest in our education as well. 151 APPENDIX E: ACTIVITY #2 FOR THE DIRECT DDL GROUP Please record your computer screen as you complete the following task(s). You may record your screen with the following free website (https://www.apowersoft.com/free-online-screen- recorder) or may use a different program to do so of your choosing, such as Zoom. You are presented below with a table featuring 75 common "hedges" used for expressing different degrees of certainty or doubt. First, browse over that list and familiarize yourself with it. Using the table below, complete the following two tasks: Task #1: Choose any five hedges from the list of modal verbs that you think are among the most frequently used in English. Check their usage in the COCA and the ICNALE Online Corpus, browse through examples of how they are used in argumentative essays, and report. Task #2: Choose five hedges from the lexical verb category that you are not as familiar with and use the concordance lines of COCA and the ICNALE Online Corpus to learn about their usage, and report. Assignment Instructions: Please use the text entry box in this assignment to list the five modal verb hedges you selected (Task #1) and provide an example sentence from the corpora for each hedge. For Task #2, list the five lexical verb hedges you were not as familiar with, and give an example of each from the corpora (copy and paste) that you think is a good example of how each of the five are used. After, write your own example sentences using all ten of the hedges you selected for Task #1 and Task #2. If it is easier, you may also complete this activity on a Word document and upload it using the File Upload option. [Hyland and Milton’s (1997) list of hedges from Appendix A] 152 APPENDIX F: ACTIVITY #2 FOR THE INDIRECT DDL GROUP Please record your computer screen as you complete the following task(s). You may record your screen with the following free website (https://www.apowersoft.com/free-online-screen- recorder) or may use a different program to do so of your choosing, such as Zoom. You are presented below with a table featuring 75 common "hedges" used for expressing different degrees of certainty or doubt. Using the table below, complete the following task: Task #1: Choose five hedges from the lexical verbs category, and five from the modal verb category, on the table and study the usage of the hedges by consulting dictionaries of your choice (e.g., online dictionaries). Assignment Instructions: Please use the text entry box in this assignment to (i) list the five lexical verb hedges and five modal verb hedges you selected, (ii) what online dictionary of your choosing you consulted, and what it said about them, and (iii) use the definitions, or examples, you saw for the hedges to create example sentences of your own using each hedge (one sentence for each hedge you selected). You can also, if it is easier, upload your response as a word document using the File Upload option. [Hyland and Milton’s (1997) list of hedges from Appendix A] 153 APPENDIX G: ACTIVITY #3 FOR THE DIRECT DDL GROUP Please record your computer screen as you complete the following task(s). You may record your screen with the following free website (https://www.apowersoft.com/free-online-screen- recorder) or may use a different program to do so of your choosing, such as Zoom. You are presented with the same table of hedges below as was used in Activity #2. Using the same table again, complete the two tasks below. Task #1: Look at three modal verb hedges from the list (will, would, might) and three lexical verb hedges from the list (think, believe, appear), and compare how frequently they are used by Chinese learners of English and native-English learners, using the ICNALE Online Corpus. Task #2: Pick two essays from the ICNALE Online Corpus (one from a Chinese learner of English and one from a native-English writer). Read the essays and compare the varieties (how many different hedges) and frequencies (how many total hedges) were used by each writer. Assignment Instructions: Use the text entry option below, or upload a text document (e.g., Word, Wordpad, Notebook) to the File Upload button below, to include (i) the ICNALE frequency numbers for the six lexical items (will, would, might, think, believe, appear) for both Chinese learners and native-English writers (Task #1), and (ii) the full texts (copied and pasted) of both essays you selected with each hedge bolded, as well as the total frequency of hedges and the total variety of hedges listed after each essay (Task #2). [Hyland and Milton’s (1997) list of hedges from Appendix A] 154 APPENDIX H: ACTIVITY #3 FOR THE INDIRECT DDL GROUP Please record your computer screen as you complete the following task(s). You may record your screen with the following free website (https://www.apowersoft.com/free-online-screen- recorder) or may use a different program to do so of your choosing, such as Zoom. You are presented with the same table of hedges below as was used in Activity #2. Using the same table again, complete the task below: Task #1: There is a student essay written by a native speaker of English included from the ICNALE corpus, with each of the hedges in it removed. Using the table of hedges below, try to fill in the hedges with what you think is the most appropriate hedge to use in each blank spot. Afterwards, scroll down to the next page to view the full original essay. Notice any differences between the hedges you put in the spots, and the ones that were there in the original essay and try to examine if the different hedges changed the meaning of the sentences at all. Assignment Instructions: Use the text functions (the large "T" button) on the file below to type in the hedges in the essay that you think should go there. Do not scroll down or look at the original essay before filling in each blank. After you are done, then scroll down and compare your answers. Highlight any sentence which you think has had its meaning dramatically changed when you consider the original hedges instead of the hedges you supplied in the blanks and leave a comment next to those sentences briefly explaining why you feel so. If the text function below will not work for you, you may also download the file to complete the task and then re-upload the document using the File Upload option below. [Hyland and Milton’s (1997) list of hedges from Appendix A] ICNALE essay with hedges removed: I __________ that having a part-time job is a good option for a student. I've had a part-time job since my first semester of college, and I have learned a lot from working there. My friends __________ to complain that they do not want to find any part time job because they are already very busy with their schoolwork and their social lives. It __________ true that having a part-time job __________ makes it difficult to find time to do everything that you want or need to do but putting yourself in situations in which you're uncomfortable is __________ one of the best ways to grow. You also __________ become stronger as a person, student, and an employee, by working while studying. What's more, by having a part-time job, students __________ make enough money to fund all of their personal expenses such as food, rent, and entertainment. They __________ even __________ have money left over. The fact of the matter is that students who do not have money __________ be able to have as much fun as I have or have as much freedom as I have. It __________ seem sad, but money really is power. If you have a part-time job, you can make money, you can gain skills, and you can make professional contacts. ICNALE essay with hedges: I think that having a part-time job is a good option for a student. I've had a part-time job since my first semester of college, and I have learned a lot from working there. My friends tend to complain that they do not want to find any part time job because they are already very busy with their schoolwork and their social lives. It might be true that having a 155 part-time job sometimes makes it difficult to find time to do everything that you want or need to do but putting yourself in situations in which you're uncomfortable is often one of the best ways to grow. You also will become stronger as a person, student, and an employee, by working while studying. What's more, by having a part-time job, students could make enough money to fund all of their personal expenses such as food, rent, and entertainment. They might even likely have money left over. The fact of the matter is that students who do not have money won’t be able to have as much fun as I have or have as much freedom as I have. It may seem sad, but money really is power. If you have a part-time job, you can make money, you can gain skills, and you can make professional contacts. 156 APPENDIX I: ACTIVITY #4 FOR THE DIRECT DDL GROUP Please record your computer screen as you complete the following task(s). You may record your screen with the following free website (https://www.apowersoft.com/free-online-screen- recorder) or may use a different program to do so of your choosing, such as Zoom. For this final activity, please complete the task below using the COCA online corpus. Task #1: Spend some time browsing through different texts in the COCA corpus, either by searching for specific hedges or just by browsing different types of texts. Find and copy one paragraph (from a larger text) that you think includes good examples of hedging use and includes multiple hedges. Assignment Instructions: Paste the paragraph into the text entry box below, or in a text document (e.g., Word, Wordpad, Notebook), and bold any hedges in it. Then, write a short reflection (1-3 paragraphs) about why you think the paragraph is a good example of hedging use, and why you selected it. Upload the text file to the File Upload feature below if you used a separate text document. 157 APPENDIX J: ACTIVITY #4 FOR THE INDIRECT DDL GROUP Please record your computer screen as you complete the following task(s). You may record your screen with the following free website (https://www.apowersoft.com/free-online-screen- recorder) or may use a different program to do so of your choosing, such as Zoom. For this final activity, recall the essay you wrote at the start of this module, which you can access by going back to the first writing assignment of this module you completed (you were either Group A or Group B). To access it, click on the "Submission Details" link, which will take you to your essay. Task #1: Revisit the essay you wrote at the start of this module and consider where you could add, or improve, your use of hedges. Revise your essay to include new hedges, or to change existing hedges to more appropriate ones. Activity Instructions: Using the Text Entry Box, or the File Upload feature below to submit a text file (e.g., Word, Wordpad, Notebook), include your original essay, as well as your new, revised, essay. In your revised essay, please highlight any hedges that you have changed from your original essay. 158 APPENDIX K: LEARNER PERCEPTION QUESTIONNAIRES Table 12 Learner perception questionnaires Direct DDL group Indirect DDL group Background Questions: Which of the following best reflects your experience with language corpora prior to this session? (Likert Scale) How would you characterize your experience with language corpora prior to this session? (Open-ended) Which of the following best reflects your habits of computer use? (Likert Scale) How would you characterize your habits of computer use? (Open-ended) Which of the following best reflects your experience with language corpora prior to this session? (Likert Scale) How would you characterize your experience with language corpora prior to this session? (Open-ended) Which of the following best reflects your habits of computer use? (Likert Scale) How would you characterize your habits of computer use? (Open-ended) Likert-Scale Questions: I understand the purpose of using corpora in these sessions. I found the searching technique easy to learn. The classroom activities (e.g., dictionary The exercises were helpful for learning hedges. work) were useful for learning hedges. My perception of the importance of hedging changed after taking the sessions. The sessions increased my confidence in using hedges. My English writing improved after taking the sessions. I had some difficulty in performing the searching technique. The instructions provided by the teacher were helpful for learning the searching technique. The authentic texts in the corpora are difficult to understand. The corpora helped me learn the usage of hedges. The corpora helped improve my English writing. I found the corpora more helpful than a dictionary. I want to see greater use of corpora in the language classroom. When I have problems in English language learning, I will use corpora as a reference tool. Learning about corpora has increased my confidence about writing in English. Overall, corpora are a very useful resource for English writing. 159 Table 12 (cont’d) Open-Ended Questions: What are your thoughts about the benefits of the hands-on corpus use for English language instruction? What are your thoughts about the problems of the hands-on corpus use for English language instruction? What are your thoughts regarding the benefits of the online learning tasks? What are your thoughts regarding the problems of the online learning tasks? 160 APPENDIX L: STANDARD INTERVIEW QUESTIONS List of standard interview questions, used to supplement the natural stimulated recall interactions • Regarding the language corpora that you used during the activities (COCA and ICNALE): o What difficulties stood out to you about navigating them? Did you have any trouble using them or navigating the interface? o What was the most frustrating part about the tools, if anything? o Did you find the tools easy to learn and use at first? Did the websites provide enough information to use them? Were the interfaces (or, the webpages) easy to use for newcomers to the tools? o Is there anything that prevented you from using the tools fully? Was there anything that would have made it easier for you to use the tools? o What did you find helpful or useful about the tools? What were your favorite things about them? o How do you think using the tools compares to a dictionary when it comes to language learning or improving your English? o Do you think you will continue to use language corpora tools, such as these (COCA and ICNALE) as you continue to advance your English? 161