TEACHING AND LEARNING IN A CONTENT-BASED CLASSROOM: UNDERSTANDING PEDAGOGY AND THE DEVELOPMENT OF L2 WRITERS’ METACOGNITIVE GENRE AWARENESS By Matthew Jeffrey Kessler A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Second Languages Studies – Doctor of Philosophy 2020 ABSTRACT TEACHING AND LEARNING IN A CONTENT-BASED CLASSROOM: UNDERSTANDING PEDAGOGY AND THE DEVELOPMENT OF L2 WRITERS’ METACOGNITIVE GENRE AWARENESS By Matthew Jeffrey Kessler The origins of second language (L2) writing as a field can be traced back to the early 1990s, when it first began to gain momentum as an area of interdisciplinary inquiry (Matsuda & De Pew, 2002). Since then, the field has evolved to become rich and diverse in scope, as scholars have examined numerous phenomena by drawing on an array of theoretical and methodological approaches. In particular, two areas that have received significant attention are investigations involving (1) genre-based teaching and learning, and, (2) the linguistic development of L2 writers, respectively. Though much has been learned, research involving genre-based pedagogies has often been insular and limited to applied linguistics-related contexts, and little is known about the pedagogical practices of L2 writing instructors in contexts such as those involved in Content-Based Instruction (CBI). Relatedly, as Polio (2017) has noted, few studies have addressed how L2 learners’ genre knowledge actually develops over time in specific contexts, as development has tended to be assessed using timed-writing tasks. Thus, the current dissertation examines the nexus of these two areas, exploring phenomena related CBI pedagogical practices and L2 learners’ subsequent genre learning and performance. This study takes place in a Master of Laws (LLM) program in a Midwestern law school. Adopting a classroom-based ethnographic approach (e.g., Bloome, 2012), I explored the beliefs and pedagogical practices of one writing instructor, as he taught a professional legal genre called the office memorandum during a semester-long CBI legal research and writing course. Data consisted of: in-class observations, field notes, semi-structured interviews, artifacts, and a visualization reflection. In addition to observing the instructor, I tracked the developing genre awareness among six (N = 6) L2 English learners/LLMs throughout the semester using a case study design. Students’ development was examined using metacognition theory (Flavell, 1979; Schraw & Dennison, 1994), and data included: a survey, semi-structured interviews, and modified stimulated recalls with students’ office memoranda. Finally, the relationship between students’ genre awareness and their perceived writing performance was explored. This was addressed through eliciting holistic rankings and evaluative comments from the course instructor and by comparing them with/against students’ self-reported metacognitive genre awareness. The findings highlight the instructor’s rich, multifaceted philosophy of teaching L2 writing and the office memorandum, which in turn, subsequently influenced both his curriculum design and in-classroom teaching practices. The impact of this pedagogy was also apparent in its resulting influence on LLM students’ development of metacognitive genre awareness, as students’ self-reports often reflected those themes addressed by the instructor in the areas such as understanding the genre’s audience. Apart from the influence of pedagogy, the impact of students’ prior experiences (academic and professional) with related genres can also be seen in influencing their developing genre awareness. In terms of performance, it appears that an increased ability to self-report one’s metacognitive genre awareness (both a general awareness plus how one uses that awareness while writing) is connected to perceived writing quality. In closing, I discuss both the strengths and limitations of the current study, including characteristics related to the study’s design. Finally, I expound on the implications of this dissertation’s findings, and I address their significance and potential to influence future teaching and research in the field of L2 writing, and, in second language acquisition more broadly. iv ACKNOWLEDGEMENTS First and foremost, I feel obliged to acknowledge that this dissertation is due only in small part to my own efforts. Rather, this study is the culmination of the hard work and dedication of many, including: the faculty at Michigan State University, who took great pains to provide me with training and guidance over the past three years; my peers in the Second Language Studies program, who provided their fellowship and advice (specifically Caitlin Cornell, Dustin Crowther, Jungmin Lim, and Susie Kim); the participants of this study, who spent many long hours engaging with my inquiries; my family, for their continued support over the years; and the journal Language Learning, whose financial support made this study a reality—the list of acknowledgements can go on and is far longer than what can adequately be described here. In particular though, I am indebted to J. Elliott Casal, who back in 2016 encouraged me to quit my job and pursue a PhD. Were it not for Elliott’s encouragement and enthusiasm, I would not be writing these words today. Likewise, a special thank you to Dr. Charlene Polio for taking me on as a PhD student at the outset of my doctoral studies. I owe much of my success to her guidance and feedback. I am also incredibly appreciative of the efforts of Dr. Peter De Costa and Dr. Shawn Loewen, who both have served as mentors and taught me a great deal during the past few years. An extra special thank you also goes to Dr. Jeremy Francis, who volunteered to serve on my dissertation committee, and, whose guidance has shaped many facets of the current study. Finally, thank you to my wife, Mim. Her unwavering support, adventurous spirit, and steady hand have continued to keep me balanced while propelling me forward. It is an understatement to say that without her, none of this would be possible. Thank you. iv vii viii ix 1 5 5 14 21 30 32 32 40 41 48 52 52 79 107 117 117 125 131 134 RQ1: Law Instructor’s Pedagogical Practices in a Content-Based LLM Writing Course RQ2: The Development of L2 Learners’ Metacognitive Genre Awareness RQ3: The Relationship Between Students’ Metacognitive Genre Awareness and Perceived Text Quality CHAPTER 4: DISCUSSION TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES KEY TO ABBREVIATIONS INTRODUCTION CHAPTER 1: LITERATURE REVIEW Genre-Based Teaching and Learning Content-Based Instruction (CBI) and L2 Legal Education Programs Metacognition Theory The Current Study CHAPTER 2: METHODOLOGY Context and Participants Researcher Positioning Data and Data Collection Timeline Data Analysis CHAPTER 3: FINDINGS RQ1: Law Instructor’s Pedagogical Practices in a Content-Based LLM Writing Course RQ2: The Development of L2 Learners’ Metacognitive Genre Awareness RQ3: The Relationship Between Students’ Metacognitive Genre Awareness and Perceived Text Quality Limitations CHAPTER 5: CONCLUSION Implications and Future Research Directions APPENDICIES APPENDIX A Background Questionnaire (LLM Students) v 139 139 148 149 APPENDIX B Semi-Structured Interview Questions (Professor Meyer) APPENDIX C Semi-Structured Interview and Stimulated Recall Questions (Students) APPENDIX D Coder Training Sheet REFERENCES 151 154 156 158 vi LIST OF TABLES Table 1 Case study participant bios Table 2 Research questions (RQs) and corresponding sources of data Table 3 Timeline for data collection Table 4 Overview of schedule, topics, and activities for legal research and writing course Table 5 Aashna’s development of metacognitive genre awareness of the office memo Table 6 Jing’s development of metacognitive genre awareness of the office memo Table 7 Luisa’s development of metacognitive genre awareness of the office memo Table 8 Paul’s development of metacognitive genre awareness of the office memo Table 9 Yuki’s development of metacognitive genre awareness of the office memo Table 10 Yurui’s development of metacognitive genre awareness of the office memo Table 11 Top 15 most frequently recurring metacognition comment codes Table 12 Overview of LLMs’ development of metacognitive genre awareness Table 13 Professor Meyer’s evaluative comments on the first office memo assignment 37 42 46 53 81 84 88 92 96 100 103 105 110 Table 14 Professor Meyer’s evaluative comments on the second office memo assignment 112 Table 15 Professor Meyer’s memo rankings and students’ total metacognition comments 115 vii LIST OF FIGURES Figure 1. Metacognition model Figure 2. Professor Meyer’s visualization reflection Figure 3. Screenshot of graphic organizer for comparing cases Figure 4. Screenshot of sample memo outline Figure 5. Screenshot of Meyer’s office memo manual Figure 6. Screenshot of Meyer’s annotated full-length memo Figure 7. Recurring themes of audience, organization, and language in weekly schedule Figure 8. Screenshot of research plan and log 22 57 62 67 69 70 71 78 viii KEY TO ABBREVIATIONS CBI Content-Based Instruction EFL English as a Foreign Language ESL English as a Second Language ESP English for Specific Purposes ID Individual Differences IIED Intentional Infliction of Emotional Distress JD L1 L2 Juris Doctor First Language Second Language LLM Master of Laws RQ Research Question SFL Systemic Functional Linguistics SJD Doctor of Juridical Science SLA Second Language Acquisition ix INTRODUCTION In the field of second language acquisition (SLA), the concept of genre has played a critical role in the exploration of second language (L2) learners’ literacy, writing, and linguistic development since the term first appeared in the literature in the early 1980s (Paltridge, 2014; Tardy, 2011a). Defined by Hyland (2007) as “abstract, socially recognized ways of using language” (p. 149), genres—typically those academic or professional in nature—have been both taught and examined using a variety of theoretical approaches over the years (e.g., Halliday, 1996; Martin, 1984; Miller, 1984; Swales, 1990). Irrespective of the approach, few would argue that one of the central aims of much genre-based research has been to shed light on the distinctive and occluded features of different genres. Whether exploring the linguistic, rhetorical, and/or social dimensions, Tardy (2016) highlights that an overarching goal has been to demystify or to reveal underlying textual frames and discourse community practices, and thus, to provide learners with a point-of-entry for engaging with unfamiliar genres. In the literature to date, there have been numerous edited volumes and studies devoted to examining the written discourse of different genres through text analyses and corpus-based methods (e.g., Biber, Connor, & Upton, 2007; Bhatia, 1993; Hyland, 2012; Kuteeva & McGrath, 2013; Tardy, 2011b; Samraj & Monk, 2008; Swales, 2004). Likewise, beyond the texts themselves, there has been considerable effort paid to exploring and describing the implementation of genre-based pedagogies from the vantage points of teachers and/or students (e.g., de Oliveira & Lan, 2014; Flowerdew, 2016; Mahboob, Dreyfus, Humphrey, & Martin, 2010; Racelis & Matsuda, 2013; Swales & Feak, 2012; Tribble & Wingate, 2013). Despite the insights gleaned into the construction of various genres and suggestions regarding how teachers might integrate them into the classroom, Polio (2017) has noted that there is a dearth of research 1 examining how L2 learners’ genre knowledge develops over time, stating that “much research on writing development…relies on timed writing [i.e., argumentative essay writing] which may bear little resemblance to writing done for academic courses” (p. 267), or, to those types of writing found beyond academia in professional contexts. In the past decade, there have been several efforts to trace learners’ developing genre knowledge using longitudinal designs (e.g., Negretti, 2012; Negretti & Kuteeva, 2011; Yasuda, 2011; Yeh, 2015). The work of Negretti has been particularly innovative, as she and her colleagues have adopted the educational psychology theory of metacognition (Flavell, 1979; Schraw & Dennison, 1994)—which is, the ability to control and reflect upon one’s own thinking and learning—to examine the development of various dimensions of learners’ genre knowledge. The concept of metacognition itself has played an important role in both L1- and L2-based research, as numerous studies have shown that individuals who display greater metacognition surrounding a given task tend to outperform their peers in areas such as reading (e.g., Pintrich, 2004; Wang & Chen, 2013), writing (e.g., Qin & Zhang, 2019), receiving/retaining corrective feedback (e.g., Sato & Loewen, 2018), and more. However, though metacognition has played an integral role in SLA research, it has only recently been adopted for examining longitudinal development, especially involving L2 writing and genre. Importantly, due to the interplay of cognitive, social, and textual elements involved in the teaching and learning of written genres, Negretti (2017) has suggested that using metacognition for exploring L2 learners’ genre development may be one way to address scholars’ calls (e.g., Hacker, Keener, & Kircher, 2009; Ortega, 2012) for providing a more holistic view of the dynamic interplay among such elements. While metacognition appears to be suitable for investigating the development of L2 learners’ metacognitive awareness in a given genre(s), the few prior studies that have adopted the 2 model for examining longitudinal changes (i.e., Negretti, 2012; Negretti & Kuteeva, 2011; Yeh, 2015) have been conducted in a fairly unified context with three similarities: (1) The instructors are all highly linguistically trained, typically with advanced degrees in applied linguistics or teacher training in English as a Second/Foreign Language (ESL/EFL); (2) the students themselves are taught academic research genres (i.e., academic research papers); and (3) typically, the students are explicitly taught using an ESP genre-based Swalesian move analysis (Swales, 1990, 2004), in which students often read the works of John Swales and then practice applying rhetorical move analyses to research articles before writing on their own. Since, thus far, genre learning has been explored only by using academic genres and following highly controlled ESP pedagogical approaches, it is important to investigate additional genres and instructional contexts as well, as not all genre-based teaching and learning unfolds under such narrow, applied linguistics-specific conditions. In the area of legal education in the United States, there are hundreds of Master of Laws programs (known as LLM programs) that are designated primarily for international students (Hartig & Lu, 2014). Over the years, LLM programs have seen increasing enrollment with approximately 19,800 students in 2019 compared to roughly 11,100 enrolled in similar programs only six years earlier (American Bar Association, 2019). Typically lasting one-year in length, LLM programs provide international students with an overview of the U.S. common law system, instruction in specialized law topics (e.g., international tax law, intellectual property), and frequently, instruction in written genres such as the office memorandum (i.e., office memo), which is a commonly utilized genre throughout much of the legal profession (Chew & Pryal, 2016). While genres such as the office memo are taught to thousands of L2 (and L1) English students each year, the context in which this instruction occurs typically does not resemble those 3 conditions under which previous L2 metacognitive genre awareness studies have taken place in that: (1) the instructors are typically lawyers who are legally trained, yet often not trained linguistically to work with ESL students; (2) the students themselves are taught non-academic genres (i.e., they are taught professional legal genres); and, (3) the students are not explicitly taught using ESP/Swalesian move analyses. Clearly, it is integral that SLA researchers understand not only how L2 learners’ development unfolds within the context of applied linguistics, but also, as to how learners’ genre knowledge develops across a variety of fields and genres. Relatedly, it is also important to explore the pedagogical practices of instructors across different academic and/or professional contexts, as understanding content educators’ varying approaches to raising genre awareness have the potential to inform our own future research and pedagogy within applied linguistics, and in particular, within SLA. Thus, in the current dissertation, I attempt to address these aforementioned issues, as I explore various phenomena related to both teaching and learning in a content-based legal writing course. Over the span of one academic semester, I adopted a classroom-based ethnographic approach to explore the pedagogical practices of a law instructor as he taught the office memo genre (and raised genre awareness) to a group of L2 English learners enrolled in an LLM legal research and writing course. In addition to exploring the pedagogical practices of the instructor, through the adoption of metacognition theory, I used a case study design to document a group of LLM students’ (N = 6) developing genre awareness as they learned to write the office memo over the 15-week semester. Finally, related to the development of students’ metacognitive genre awareness, I also explored the extent to which students’ abilities to verbalize their metacognitive genre knowledge was reflective of their perceived writing performance from the vantage point of the course instructor. 4 CHAPTER 1: LITERATURE REVIEW In this chapter, I provide a broad overview of genre, including a discussion of how the concept has been both understood and adopted as a lens for research and teaching in the field of SLA. The overview then moves to the need for additional research to explore L2 genre-based teaching and learning across a variety of settings, such as those involving Content-Based Instruction. In particular, this includes a discussion of the context of L2 legal education and LLM programs in the United States, which typically involve the teaching and learning of professional legal genres. The focus of the chapter then shifts to the theory of metacognition, its origins and importance to L1- and L2-based research, and crucially, the emerging potential of using the theory as a lens for understanding different facets related to genre—and, specifically—for understanding L2 writing development across different academic and professional genres. Finally, the chapter concludes with a brief overview of the current study and the three research questions that guide the dissertation. Genre-Based Teaching and Learning Origins and definitions of genre. Though the term genre has long-played a critical role in fields such as Literature and Rhetoric and Composition studies (see Bawarshi & Reiff, 2010 and Halasek, 1999 for thorough reviews), the term did not officially make its debut into the field of SLA until the early 1980s (Paltridge, 2014; Tardy, 2011a). However, since its emergence nearly four decades ago, the concept has featured prominently, serving as a driving force behind much research and pedagogy. And, nowhere has the concept been more influential than in the domain of L2 writing. Over time, L2 writing scholars have defined the concept of genre both in terms of features related to the texts themselves, and also, in terms of the social communities associated 5 with those texts. For instance, Hyland (2004) has described genres as “labels we use to categorize texts and the situations in which they occur” (p. 4), while Swales (1990) has defined genres as “a class of communicative events…[that] are recognized by expert members of the parent discourse community” (p. 58). Underscoring both Hyland’s and Swales’ definitions is the central idea that when it comes to genres, the texts that we consume (as readers) and produce (as writers) are socially constructed. This is to say: different communities use different genres for specific communicative purposes, and each respective discourse community has certain socially constructed expectations regarding what different genres should consist of in terms of features that involve language, content, organization, style, and much more. Importantly as well, is that genres are an essential part of our daily lives. Regardless of whether one works as a teller at a bank, a custodian at an elementary school, or a researcher at a university, we constantly encounter genres and we use them both personally and professionally for the purposes of communicating and deciphering information. Even in the act of starting one’s day, from the time we wake up to the time we arrive at work, we are likely to encounter many different genres. For example, in opening one’s cupboard for breakfast, we may find various cereal or oatmeal boxes; in checking one’s cellphone, we may be exposed to text messages, professional/business emails or news articles; and in driving to work, we are likely to encounter a variety of advertisements displayed on billboards, city buses, and benches. The point here is: Each of these different genres is easily recognizable to us due to what are often specific identifying or distinctive features that have come to be associated with those genres over time. In addition, our knowledge of and ability to recognize or produce these genres can be facilitated by a number of factors. Differing approaches to genre-based teaching and learning. Because of the ubiquitous 6 nature of genres, the ability to understand, and oftentimes, to produce different genres can be highly critical, especially when it pertains to success in academic and/or professional contexts. Due to their importance, over the years, a number of theoretical approaches have been developed for the purposes of analyzing and teaching genres to L2 learners in the classroom. In the domains of applied linguistics and within SLA in particular, the three approaches that have received the most attention to date are: English for Specific Purposes (ESP), Systemic Functional Linguistics (SFL; sometimes referred to as the ‘The Sydney School’), and the New Rhetoric approach. It was John Swales who first introduced the concept of genre to the field of applied linguistics in 1981 with his seminal work examining the rhetoric of academic research article introductions (see Swales, 1981). Since that time, Swales and a handful of others (e.g., Ken Hyland and Ann Johns) have been instrumental in developing and promoting what are now known as ESP approaches to genre pedagogy. As Paltridge (2014) notes, in ESP approaches, central is the concept of genre analysis, which is the act or process of describing (typically academic) genres, as individuals analyze both the lexico-grammatical and rhetorical features of a text in relation to a given rhetorical context. For instance, a researcher/teacher might wish to examine the aforementioned academic genre of research article introductions, dissecting the most frequently recurring rhetoric or rhetorical moves (Swales, 1990) that authors utilize in their writing, with the ultimate goal being to make those moves more transparent to their novice L2 learners so that in turn, the learners can attempt to engage in writing the genre. Numerous researchers have advocated for using the ESP approach in teaching L2 learners how to understand and analyze conventions related to academic and professional writing (e.g., Bhatia, 1993; Johns, 2002). As a result, several instructional books have been developed, especially for 7 the audience of university-level graduate students (e.g., Swales & Feak, 2000, 2012). Apart from ESP, another genre-based pedagogy that has been adopted by many educators is that of SFL, which is sometimes referred to as “The Sydney School” due to the approach’s early development and prevalent use throughout Australia. SFL as an approach emerged primarily out of the works of Halliday (1978) and Martin (1984), with more fully developed book-length treatments coming later on by Halliday (1994), Halliday and Matthiessen (2004), and Martin and Rose (2008). In such works, unlike ESP approaches, the authors describe SFL in terms of the importance of understanding specific genres in relation to other integral factors such as those of context, language, and register. In essence, SFL views genres less so as fixed sets of rhetoric, but rather, as resources for meaning making among a range of linguistic choices. Additionally, unlike ESP approaches, which have traditionally been adopted in the teaching of more advanced L2 learners, SFL is oftentimes intended for novice students who may be in primary or secondary school settings, or, perhaps adult immigrants as well (see Johns, 2008 or Macken-Horarik, 2002 for reviews). In an SFL approach to teaching genre, instruction is highly scaffolded. Oftentimes, (though not a requirement), genres are taught using a “teaching-learning cycle” (e.g., Rothery & Stenglin, 1994). This teaching-learning cycle involves multiple stages, including: the teacher modeling and deconstructing the text, followed by joint construction of the text by the teacher and students, and finally, independent text construction by the learners themselves (e.g., Caplan & Farling, 2016; Nagao, 2019; Yasuda, 2011). The final approach that has received some attention—though admittedly, much less so than ESP and SFL—is the New Rhetoric approach, which was first developed by scholars such as Miller (1984) and Bazerman (1988, 1997). When it comes to the writing and analysis of different genres, New Rhetoricians view both the cognitive and social elements of genres as 8 highly interconnected and non-separable. Therefore, in this view, New Rhetoricians are often concerned with the social actions related to the production of written genres. This has led scholars such as Russell (1997), who have worked to develop the theory, to promote the adoption of sociocognitive theories such as Activity Theory (e.g., Engeström, 1987, 1999) as a lens for understanding how genres are created within a given social context, and, the social purposes that those genres serve. For example, through adopting Activity Theory, teachers can design activities to help students understand the complex factors that go into producing a specific written genre, such as considering the discourse community’s norms (e.g., specific terms or acronyms used by the members), the rules of that community (e.g., the social acceptance of using first person pronouns when writing), and much more. However, given that the New Rhetoric approach itself is regarded as highly theoretical and more focused on the social nature of genres, there have been relatively few attempts to translate the theory into classroom pedagogy (see Devitt, Reiff & Bawarshi, 2004). Johns (2008) and others have noted this as well, stating that unlike ESP and SFL genre-based pedagogies, the New Rhetoric approach is typically not as familiar to L2 instructors, as “many of the New Rhetoric theorists claim that it would be difficult, if not impossible, to apply their theories to the classroom” (p. 242), especially given its reliance on adopting lenses such as Activity Theory for the purposes of creating activities. Overview of genre-related research. Despite the obvious differences among the aforementioned approaches, there are numerous similarities. Importantly, ESP, SFL, and the New Rhetoric approaches all view genres as strongly interconnected to community and social practices. Additionally, as Tardy (2016) notes, one of the central aims of these genre-based pedagogies is to assist learners in comprehending the most commonly employed textual frames, and by extension, to provide novices with a point-of-entry for engaging with unfamiliar genres. 9 Thus, irrespective of the approach, over the years there has been a swell in genre-based studies with these goals in mind, and these prior studies can generally be classified into two broad themes, including: (1) those studies that are devoted to examining the written discourse of different genres through text analyses and corpus-based methods, and (2) those studies that have attempted to explore the implementation of genre-based pedagogies (i.e., ESP, SFL, and New Rhetoric approaches) from the vantage points of teachers and/or students, with a focus on how teachers raise awareness of particular genres. Since Swales’s (1981) work examining the rhetoric of academic research article introductions, a number of researchers have been inspired to explore the construction of various other academic and professional genres (e.g., Biber, Connor, & Upton, 2007; Bhatia, 1993; Hyland, 2012; Kuteeva & McGrath, 2013; Lu, Yoon, & Kisselev, 2018; Tardy, 2011b; Samraj & Monk, 2008; Swales, 1990, 2004). As discussed, in such studies, the primary goal has been to dissect individual discourse-community practices and to help L2 learners understand various features of the written genres so that they, too, may attempt to engage in them. In conducting these studies, researchers have adopted a variety of methods for examining the structural composition of texts, including: studies that are solely corpus-based in examining linguistic features such as lexical bundles (e.g., Biber et al., 2007; Hyland, 2012); studies that are qualitative in nature, examining features such as rhetorical moves (e.g., Bhatia, 1993; Swales, 1990, 2004); or, more recently, studies that have adopted a mix of corpus and qualitative methods, examining not only recurring strings of formulaic language such as phrase frames, but also integrating more fine-grained analyses involving practitioners’ perspectives as well (e.g., Lu, Yoon, & Kisselev, 2018). Though these text-based studies have been most prevalent in the literature to date, 10 another area of interest has been for researchers to examine the pedagogical practices of teachers in L2 writing classrooms. Specifically, because much scholarship has focused primarily on theory building and examining text-related discourse, researchers such as Leki, Cumming, and Silva (2008) and others (e.g., Yigitoglu & Reichelt, 2014) have called for more investigations into actual L2 instructors’ classroom practices. In particular, Leki et al. (2008) have noted that this has been “a perplexingly overlooked and underreported aspect of research” (p. 81), and that such research is essential not only for increasing our understanding of the current state of the field, but also, to better connect theory, research, and pedagogy in the future. Thus, multiple researchers have responded to these calls and attempted to further understand how instructors in different contexts approach the teaching of genres and raising genre awareness among their learners (e.g., de Oliveira & Lan, 2014; Flowerdew, 2016; Harman, 2013; Mahboob, Dreyfus, Humphrey, & Martin, 2010; Myskow & Gordon, 2010; Racelis & Matsuda, 2013; Swales & Feak, 2012; Tribble & Wingate, 2013; Yigitoglu & Reichelt, 2014). For instance, some studies have focused on describing teachers’ selections of course materials and activities (e.g., Myskow & Gordon, 2010; Yigitoglu & Reichelt, 2014), while other studies have attempted to show how teachers’ pedagogical practices have impacted their students’ development of genre awareness and their subsequent writing (e.g., Harman, 2013; Hultin & Westman, 2018; Tardy, 2015). In one such study by Harman (2013), the researcher explored an L2 writing instructor’s pedagogical practices in teaching response-based writing to a 5th grade classroom of ESL students in a literature course. Harman adopted a classroom-based ethnographic approach (e.g., Bloome, 2012), and over the span of five months, she collected data that included: audio and video recordings of the classroom, field notes, student texts and assignments, and interviews with the course instructor and students. In her findings, Harman highlighted how the instructor’s 11 practices imbued her ESL learners with a metalinguistic awareness surrounding the types of linguistic choices that were available to them when writing. Additionally, Harman stated that the instructor’s approach to teaching the genre encouraged her L2 learners to both borrow (from their readings/literature) and play with different lexical patterns when writing their texts. In another classroom-based ethnographic study, Tardy (2015) investigated an undergraduate research methods class in environmental science in which the instructor was preparing students for their senior-year research project. Over a period of 10-weeks, Tardy collected field notes, course materials, and conducted interviews with the classroom professor and students. In presenting her findings, Tardy explained how through a combination of explicit instruction and modeling, the professor of the course walked students through the research process and attempted to build their literacy and reading skills. Also, since how and/or whether students absorbed this knowledge was of interest, Tardy noted how many of the students tried to imitate their professor by repeating the professor’s words and phrasing both in their writings and orally during their interviews with Tardy herself. Notably, the studies described above show the complex nature of the interplay between teachers’ pedagogical practices, students’ development of awareness, and students’ actual writing. Relatedly, additional studies by Tardy (2005, 2009) and Gao (2012) have revealed even further intricacies, namely, that the acquisition of genre knowledge is multifaceted and a skill that is gradually built up over time. Importantly, too, is that such studies point to the conclusions that genre knowledge and performance may be susceptible to individual factors such as a learner’s content familiarity, linguistic differences, and even L1-based knowledge such as a student’s rhetorical expectations. Finally, it is worth noting that what all of the aforementioned studies have in common is 12 that the researchers adopted a classroom-based ethnographic approach as part of their research designs. In understanding the connections between teachers’ practices and students’ subsequent genre awareness and writing, scholars have argued that classroom-based ethnographies can play an important role by helping researchers more fully understand the context within which the writing and genres are situated (e.g., Johns, Bawarshi, Coe, Hyland, Paltridge, Reiff, & Tardy, 2006; Paltridge et al., 2016). Relatedly, Russell (2001), Yigitoglu and Reichelt (2014), and others have long-stated a need for even more ethnographic or qualitative studies that “explore in rich detail the discipline-specific practices of learning and teaching to write” (Russell, 2001, p. 59). As Paltridge et al. (2016) further explain: Ethnographically oriented research has been especially valuable not only for demonstrating why the process of developing academic literacies is so unpredictable but also why knowledge taught and learned in classrooms may not be obviously transferred to new learning and literacy environments. (p. 164) Thus, though a number of studies have explored genre teaching and learning in numerous academic contexts including those in K-12 (e.g., Harman, 2013; Myskow & Gordon, 2010) and university settings (e.g., Gao, 2012; Tardy, 2005, 2015; Yigitoglu & Reichelt, 2014), more studies are needed in a greater variety of instructional contexts in order to help SLA researchers more fully understand the scope of genre-based teaching and learning. Beyond the need for researchers to explore the nature of L2 teaching and learning across different instructional contexts, one additional (and related) area of need is for more for research to be conducted that examines how learners’ genre knowledge develops over time both in and across these contexts. As Polio (2017) has noted, to date, most researchers who have explored learners’ writing development have tended to do so by relying on timed writing tasks—that is, by 13 assessing linguistic development through the use of 30- or 60-minute timed argumentative prompts (e.g., Bikowski & Vithanage, 2016; Bulté & Housen, 2014; Crossley, Kyle, & McNamara, 2016; Storch, 2009; Yoon & Polio, 2017). Notably, as well, is that these types of timed tasks often do not resemble the specific genres or types of writing that L2 learners actually use in their daily lives in academic or professional contexts. In regards to this particular issue, I return to address it later on in this chapter. Content-Based Instruction (CBI) and L2 Legal Education Programs Research and teaching involving CBI. As discussed in the previous section, there have been a number of qualitative and/or ethnographic-inspired studies that have explored the teaching and learning of genres in a variety of academic contexts. However, one area that has been largely overlooked is that of Content-Based Instruction (CBI). Unlike in traditional language-focused courses such as those in ESL/EFL or FL teaching contexts, CBI courses typically center on the specific subject matter being delivered, while developing learners’ language-related skills is oftentimes of secondary focus. As an instructional approach, CBI claims numerous SLA theories and related concepts as key motivations, such as comprehensible input (e.g., Krashen, 1985), interaction (e.g., Long, 1996), peer scaffolding (Vygotsky, 1978), cognitive learning theory (e.g., Anderson, 1983), and more. As Stoller and Grabe (1997) have noted, CBI approaches typically “view language as a medium for learning, and content as a resource for learning language” thereby supporting “purposeful and meaningful language use” (p. 78). This is also why in many CBI courses, the curriculum attempts to deliver course content in conjunction with assisting students in developing a specific skill (or, a set of skills related to reading, writing, speaking, and/or listening) that are necessary for success in a particular domain. As such, Kasper (2000) has highlighted that even though a CBI instructor’s focus is typically on 14 the course content itself, teachers do oftentimes attempt to engage in a variety of instructional practices that will help facilitate the L2 acquisition process for their learners, including: editing course materials so that they are more accessible to L2 learners’ proficiency levels; presenting information in a variety of aural and visual formats, such as utilizing visual aids like videos and graphic organizers; and much more. Over the years, researchers exploring CBI have investigated a vast number of phenomena, including those related to: education policy (e.g., Dalton-Puffer, Nikula, & Smit, 2010); the development of syllabi and the implementation of different types of CBI programs (e.g., sheltered classes versus language immersion) by administrators and FL teachers (e.g., Cammarata, 2009; Fitzsimmons-Doolan & Stoller, 2018); CBI’s effectiveness as an approach in isolation and/or when compared to more traditional non-immersion programs (e.g., Hanzawa, 2018; Thomas, Collier, & Collier, 2011), and more (see Lyster, 2017 and Tedick & Wesley, 2015 for reviews). However, it is important to note that in the body of CBI literature to date, most studies have tended to occur within the context of examining bilinguals in K-12 settings rather than adults in university-level settings. Likewise, though a number of studies have investigated CBI in the contexts of STEM-related disciplines, additional contexts in the social sciences and those such as law/legal education remain relatively unexplored. Master of Laws (LLM) programs. As mentioned, one particular CBI context that has been underexplored—especially in relation to L2 genre teaching and learning—is Master of Laws programs (Baffy, 2017). In the area of legal education in the United States, there are hundreds of Master of Laws programs (colloquially known as LLM programs). As Hupper (2015) has noted, LLM programs were originally created as an alternative for legal doctorate programs for J.D.-holders who wished to further specialize in a particular area of law, yet who 15 did not meet the academic standards to pursue the doctorate. Originally, very few foreign- trained lawyers were enrolled in such programs. Over time though, primarily during the past few decades, there has been a push by law school administrators, so much so that LLM programs are now often designated primarily for international students (Hartig & Lu, 2014; Hupper, 2015). Over the span of the past several years in particular, these LLM programs have seen increasing enrollment with approximately 19,800 students enrolled in 2019 compared to roughly 11,100 enrolled in similar programs only six years earlier (American Bar Association, 2019). Typically lasting one-year in length, LLM programs provide international students with an overview of the U.S. common law system and opportunities to take coursework in specialized topics of students’ interest (e.g., international tax law, intellectual property, etc.). Many times as well, the instruction in these courses has a supplemental (and secondary/CBI-like) focus on language use, in which L2 instruction is delivered either in-class or through supplemental course materials such as workbook exercises or homework readings. Unlike K-12 CBI contexts in which students are typically somewhat uniform in terms of both their ages and prior experiences, those L2 English students who enter LLM programs are comparatively quite diverse, representing a broad spectrum of L1s, prior educational experiences, and prior work and life experiences. Relatedly, as Baffy (2017) notes, many LLM students have different plans following the completion of their respective programs, such as: returning to their home countries to seek employment, returning home to continue with an existing position (that is temporarily on hold while pursuing the LLM degree), remaining in the U.S. to sit for the bar exam, or continuing with their education in pursuit of an SJD. Set against the backdrop of LLM programs, there have been a number of L2 studies conducted, though admittedly, very few when compared to studies conducted in other STEM 16 fields and social sciences. Within applied linguistics in particular, studies involving LLM programs have tended to fall within one of the categories of genre-based research described earlier in this chapter—that is, with researchers examining the written discourse of different genres through the adoption of text analyses and corpus-based methods. Such studies have consisted of rhetorical move analyses of legal genres including cases and opinions (e.g., Bhatia, 1993; Hafner, 2010), and, a number of corpus-based studies that have analyzed features of lexis or lexical bundles in legal genres such as office memoranda, opinions, and more (e.g., Breeze, 2013; Hartig & Lu, 2014; Tománková, 2016). These studies typically are framed from an ESP perspective, and the researchers’ primary objective has been to highlight important genre- specific discourse or lexico-grammatical features that may be of potential use to those L2 learners enrolled in LLM programs (see Fanego & Rodríguez-Puente, 2018 for a review). Apart from work in applied linguistics, specifically within the field of law, there have been a number of books and instructional manuals published by legal writing scholars that target either teachers of or students in LLM programs who are working with legal writing. In terms of instructors, multiple texts have been created for the purpose of offering pedagogy-related tips to legal writing instructors through providing examples of: sample syllabi, suggested lesson plans, and general advice on teaching and grading ESL students’ writing (e.g., Oates & Enquist, 2011; Thornton, 2014), and also, books that provide information on ethics, conventions involving professionalism, and even detailed examples of PowerPoint slides and assessment materials (e.g., Weresch, 2009). Likewise, for students who are learning to write different legal genres, textbooks have been created that provide detailed information on: general characteristics of legal writing, legal reasoning, and commonly used legal genres with detailed, annotated examples (e.g., Chew & Pryal, 2016; Edwards, 2006; Oates, Enquist, & Francis, 2018), along with 17 textbooks that provide tips on exam preparation and reading and note-taking strategies (e.g., using graphic organizers) for improving literacy (e.g., McGregor & Adams, 2008; Piccard, 2017). Despite the broad range of topics covered by these books, whether intended for practitioners or students, what nearly all share in common is that they tend to provide extensive information on one particular written legal genre: the office memorandum. The office memorandum. When it comes to bridging the gap and transitioning between law school and real world practice, one of (if perhaps not) the most important genres for students to master is the office memorandum (i.e., office memo). According to a study by Chew and Pryal (2015), in a survey of 160 employers, 91% of those surveyed expected new law student hires to be able to write an office memo both “proficiently” and “with minimal supervision” (p. 6) upon walking through the company’s doors. Relatedly, when applying for jobs, office memos often can serve as an applicant’s writing sample that can be included as part of their application packet (Oates, Enquist, & Francis, 2018). For LLM students, too, as many hope to stay and work in the U.S. post-graduation, being able to produce the office memo is critical. Therefore, as mentioned, due to its importance in the legal profession, a significant focus of many textbooks and introductory legal writing courses has tended to center on assisting students in learning how to write the office memo genre. The office memo is a predictive legal document that is written in response to a specific legal issue(s) involving a client (Chew & Pryal, 2016). Most often, office memos are intended for internal use by a law firm in order to help advise the firm on a particular course of action (e.g., whether to take an issue to court, to settle), though occasionally, clients may also request to view specific versions or sections of a memo. The genre is also intended to be objective in nature. As Piccard (2017) has stated: "The internal office memo should not argue on the client's 18 behalf, but should instead be an honest evaluation of the merits (if any) of the client's position in the legal dispute” (p. 100). Therefore, the office memo is typically a formal document, though the level of formality may vary slightly from firm-to-firm (Edwards, 2006). However, that being said, the genre itself is fairly formulaic. For both L1 and L2 English learners who are seeking instruction in how-to-write an office memo, regardless of the textbook they select, students will most likely find a very similar suggested formatting and organizational pattern, which consists of sections that include the: 1. Heading, 2. Question(s) Presented, 3. Brief Answer, 4. Fact Statement, 5. Discussion, and 6. Conclusion (and Recommendation) (e.g., Chew & Pryal, 2016; Edwards, 2006; Oates et al., 2018; Piccard, 2017). The ‘Heading’ section includes the writer listing basic elements such as identifying the date, the requesting attorney, the author of the memo, and the issue at hand. Next, the function of the ‘Question(s) Presented’ section is to provide the reader with the specific legal question(s) the writer has been asked to respond to, which are related to a specific set of facts/circumstances in relation to a client(s). As Edwards (2006) notes, this section also can serve the dual purpose of reminding a busy supervising attorney of the specific issues he/she has assigned to the writer, who is typically a more junior attorney. Following the ‘Question(s) Presented’ is the ‘Brief Answer’ section. In this section, the writer often provides concise and direct responses to the legal questions listed in the ‘Question(s) Presented.’ This, again, can also serve to help the busy supervising attorney on whether or how closely he/she should read the explanation and analyses that follow later on. In the next section, the ‘Fact Statement’ (or ‘Facts’), the writer outlines the actions of involved parties that are typically not in dispute (e.g., on what date a party mailed a letter, made a phone call, on what date someone was injured and how, other specific actions that occurred, etc.). Importantly as well, because this section lists objective facts, those facts may 19 either favor the firm’s client or the opposition (Oates et al., 2018). Following the presentation of the ‘Fact Statement’ is the ‘Discussion.’ In this section, the writer lays out his/her analysis that led to the specific answer(s) provided in the ‘Brief Answer’ section earlier. This ‘Discussion’ is accomplished by integrating numerous elements, including the writer analyzing the legal questions in relation to the facts of the case, and importantly, by analyzing those questions and facts in relation to previously established or common law that may come in the form of case law, statutes, and more (Piccard, 2017). In the final section, the ‘Conclusion (and Recommendation),’ the writer may concisely restate the findings and projected outcome of the client’s case. In specific circumstances as well (such as office memos that involve issues related to contracts), the writer may provide recommendations regarding the inclusion and/or amendment of contract language, the course of action that should be taken, or more (Edwards, 2006). Crucially, as Oates, Enquist, and Francis (2018) have noted, for novice writers, the process of composing an office memo can be extremely difficult due to the complex skills that are required in order to produce the document, stating: To write an effective memo, you need to know how to do legal research; how to analyze and synthesize statutes, regulations, and cases; how to construct arguments; how to evaluate the merits of different arguments; and how to write about complex issues clearly and concisely. (p. 131) Thus, in order to write effective office memos that will serve the genre’s communicative purpose of synthesizing important facts and issues along with predicting future outcomes, one must possess highly developed literacy and writing skills, and also, a working knowledge of the genre’s intended audience, purpose, and conventions. 20 Metacognition Theory Origins, scope, and importance. As discussed, although there is a need for researchers to explore both genre-based teaching and learning in a variety of contexts such as those in LLM programs, there is also a need to understand how L2 learners’ genre knowledge develops over time in these contexts (Polio, 2017). In this particular area, there have only been a small handful of studies that have attempted to trace learners’ developing genre knowledge (e.g., Negretti, 2012; Negretti & Kuteeva, 2011; Yasuda, 2011; Yeh, 2015). Notably as well, is that three of these studies (with the exception of Yasuda, 2011) have adopted the theory of metacognition as a lens for understanding longitudinal changes in L2 learners’ cognitive development. The first tenets of the theory of metacognition were originally put forward by Flavell (1979). However, since that time, multiple researchers have further developed the theory into a robust and multifaceted theory (and model) that can be used for the purposes of understanding the nature of cognition and awareness, and, how individuals may use that awareness when performing a variety of tasks (e.g., Brown, 1976, 1978; Flavell, 1979; Pintrich, 2002, 2004; Schraw & Dennison, 1994). Broadly speaking, metacognition is one’s own awareness of their thinking or thought processes (Flavell, 1979). Schraw and Dennison (1994) have further defined the construct of metacognition as "the ability to reflect upon, understand, and control one's learning” (p. 460). As mentioned, the associated model of metacognition is multifaceted, and scholars today generally agree upon two primary distinctions (or sub-components) of metacognition, which include the constructs of metacognitive knowledge and metacognitive regulation (see Figure 1 for a synthesized depiction of the metacognition model as described in: Brown, 1976, 1978; Flavell, 1979; Pintrich, 2002; and Schraw & Dennison, 1994; depicted in Negretti & McGrath, 2018). The former concept, metacognitive knowledge, refers to the general 21 awareness that one has regarding his/her own cognition (Pintrich, 2002). Conversely, metacognitive regulation refers specifically to the aspect of control, which is to say, how individuals manage their metacognitive knowledge and assert regulation over a number of sub- processes (Schraw & Dennison, 1994). Figure 1. Metacognition model Within the constructs of metacognitive knowledge and metacognitive regulation, there are also multiple sub-components that are recognized by scholars. Metacognitive knowledge is theorized to consist of three sub-orders, including: declarative knowledge, procedural knowledge, and conditional knowledge. As Negretti and McGrath (2018) explain, these three concepts refer to “what we know (declarative knowledge), how to apply it (procedural) and why it is relevant to the current learning conditions (conditional)” (p. 15). However, again, it is important to note that these three concepts are all associated with individuals’ conscious knowledge and what they say they know about their awareness, or as Schraw and Dennison 22 (1998) state, individuals’ abilities to verbalize their knowledge of “about,” their knowledge of “how,” and their knowledge of “why” and “when” (p. 114). Learners’ metacognitive regulation, on the other hand, refers to how learners use their conscious knowledge to engage in a number of sub-processes. Most notably, this includes the three processes of planning (before a task), monitoring their performance (during a task), and evaluating (after a task) their own performance (Negretti, 2012; Schraw, 2009). Upon introducing the concept into the field of educational psychology, Flavell (1979) originally theorized that metacognition should play an important role in numerous domains related to comprehension and performance, including in the: “oral communication of information, oral persuasion, oral comprehension, reading comprehension, writing, language acquisition, attention, memory, problem solving, social cognition, and, various types of self- control and self-instruction" (p. 906). To date, Flavell’s predictions regarding the critical nature of metacognition have tended to be highly accurate, as the theory has since played a prominent role in illustrating the relationship between individuals’ awareness and their comprehension and/or performance on a variety of tasks. Over the years within L1 research alone, numerous intervention studies have been conducted, showing that in the classroom, some instructional aids such as checklists may promote metacognition and facilitate more efficient problem-solving (e.g., King, 1991), and, that explicit instruction targeting metacognitive knowledge may positively impact aspects of one’s divergent thinking (e.g., van de Kamp, Admiraal, Drie, & Rijlaarsdam, 2015). Furthermore, non-intervention studies have often shown that those individuals who exhibit greater metacognition surrounding a given task tend to outperform their peers in number of areas, such as: mathematical problem solving (e.g., Dignath & Buttner, 2008; Garofalo & Lester, 1985), reading comprehension (e.g., Pintrich, 2004; Wang & Chen, 2013; 23 Woolley, 2011), writing performance (see Harris, Santangelo, & Graham, 2010), and more (for a review of metacognition research in the field of Education, see Hacker, Dunlosky, & Graesser, 2009). Metacognition theory in SLA and for understanding writing development. Within the domain of SLA, metacognition has also played an important role. However, although some SLA researchers have adopted the model of metacognition depicted earlier in Figure 1 to explore various phenomena (e.g., Bui & Kong, 2019; Negretti & Kuteeva, 2011), most L2 researchers have chosen to explore metacognition in a much broader or more generalist sense—typically, by operationalizing metacognition in relation to one’s general awareness of task-related strategies (e.g., Ong, 2014; Taki, 2016; Wenden, 1998). Despite this, like those L1-based studies described earlier, research within SLA also appears to suggest the importance of metacognition in facilitating various facets related to L2 learning. For instance, studies have suggested there are advantages for those exhibiting increased metacognition in areas that include: reading comprehension and lexical retrieval (e.g., Carrell, 1989; Schoonen et al., 2002), listening comprehension (e.g., Li, 2013; Vandergrift, 2005, 2006), writing performance (e.g., Qin & Zhang, 2019; Schoonen, Van Gelderen, Stoel, Hulstijin, & Glopper, 2011; Wei, 2020), and also, in receiving and retaining corrective feedback (e.g., Bui & Kong, 2019; Sato & Loewen, 2018). Clearly, the abundance of research speaks to the idea that metacognition is integral in understanding L2 comprehension and production. It is precisely because of this that the theory has recently begun to be adopted for examining longitudinal development, especially in genre studies within the field of L2 writing. As previously mentioned, Polio (2017) has noted that there is a lack of research exploring the development of L2 learners’ genre knowledge over time, as prior studies exploring L2 development have tended to examine the acquisition of linguistic 24 features through the use of timed-writing tasks (e.g., Bikowski & Vithanage, 2016; Bulté & Housen, 2014; Crossley, Kyle, & McNamara, 2016; Storch, 2009; Yoon & Polio, 2017). Additionally, for multiple reasons, scholars have advocated for the use of metacognition theory in exploring L2 writing development, with researchers such as Hacker et al. (2009) even suggesting that, as an action, “writing is applied metacognition” (p. 155). Due to the complex interplay of cognitive, social, and textual elements involved in the teaching and learning of written genres, Negretti (2017) has further suggested that using metacognition theory for exploring L2 learners’ genre development may be one way to address scholars’ calls (e.g., Hacker et al., 2009; Ortega, 2012) for providing a more holistic view of the dynamic interplay among such elements. If we consider metacognition theory for the purposes of understanding L2 genre learning, we can use the theory and its tenets (i.e., metacognitive knowledge, metacognitive regulation, and their respective sub-orders) to explore the state of individual learners’ genre awareness, and importantly, how that awareness evolves over time. In other words, we can use the framework for understanding how metacognitive genre awareness develops (Negretti & Kuteeva, 2011). For example, if we take the genre of academic research articles for illustrative purposes, by using this genre, a researcher could explore whether a graduate-level L2 writer is able to discuss his/her awareness of the genre in relation to the three sub-components of metacognitive knowledge: (e.g., declarative) – Can the individual discuss what he/she knows about the genre’s audience, its purpose, the organization, the type of language that is typically used in the genre, etc.?; (e.g., procedural) – Can the individual discuss how he/she would generally go about writing an academic research paper, including how he/she would format and organize the paper, integrate content and research, etc.?; and, (e.g., conditional) – Can he/she discuss when/why 25 certain task conditions affect the writing itself, such as how a specific topic may affect the paper’s content or organization, how the journal’s audience may affect content and language use, etc.? By extension, this, too, can also be used to explore those sub-components of metacognitive regulation, or, how learners use their genre awareness for the purposes of planning (prior to writing a research article), monitoring (while writing), and evaluating their performance (post- writing). Despite its potential, only a few SLA researchers have adopted metacognition theory specifically for the purposes of examining L2 learners’ development of metacognitive genre awareness (e.g., Negretti, 2012; Negretti and Kuteeva, 2011; Yeh, 2015). In one such study by Negretti and Kuteeva (2011), over the span of three weeks, the researchers explored learners’ development of metacognitive knowledge. In their study, the participants consisted of eight undergraduate pre-service English teachers, who were enrolled in an academic reading and writing course in Sweden. During the three weeks, Negretti and Kuteeva used an ESP genre analysis approach (Swales, 1990, 2004) to teach the pre-service teachers how to analyze and deconstruct the genre of academic research articles in applied linguistics. Using a combination of data sources that included observations, student reflections, and student text analyses, Negretti and Kuteeva illustrated that although all of the participants were able to develop declarative and procedural knowledge of genre-relevant aspects of the articles, very few of their participants developed conditional knowledge (i.e., knowledge of when/why certain articles differed in features across various subjects and domains). In further investigating why this conditional knowledge did not develop uniformly across the participants, the researchers suggested that only those students who were able to provide more finer grained, in-depth analyses of the texts were the ones who also exhibited conditional awareness. 26 In another study by Yeh (2015), the researcher examined the development of learners’ metacognitive regulation using 16 EFL writers in an academic writing course. Taking place in a computer-assisted language learning context, participants were trained in an online course management system to analyze academic texts and to study the rhetoric of research article introductions. Like Negretti and Kuteeva’s (2011) study, Yeh also adopted an ESP genre analysis approach in teaching the students. Over the span of one academic semester, Yeh collected data from students in three cycles, having learners: write three drafts of academic research proposals, complete online actions logs that recorded students’ online comments in analyzing both published academic texts and their peers’ texts, and finally, respond to an open- ended questionnaire that was used to understand students’ reactions to applying their genre knowledge. In the results, Yeh reported that when it came to metacognitive regulation, all 16 students in the study remarked that the planning stages did not necessarily “lead to well written texts” (p. 491). Therefore, this led Yeh to explicate that although the course instructor’s teaching/training assisted students in developing their planning in academic writing, the students struggled in monitoring these principles while writing, and they also required collaborative help and feedback from their peers, particularly during the stage of evaluating. The final study to adopt metacognition theory for exploring L2 genre learning was conducted by Negretti (2012). Negretti examined how 17 beginning academic writers developed rhetorical consciousness in a community-college composition course. Importantly, as well, unlike those studies by Negretti and Kuteeva (2011) and Yeh (2015), Negretti (2012) attempted to connect learners’ general awareness with how they used that awareness in regulating (i.e., monitoring and self-evaluating) their writing performance. During the semester-long study, students were assigned four writing tasks (e.g., a research paper), and the course instructor 27 focused on raising students’ awareness to different aspects such as audience, purpose, and research and writing strategies. As students completed each writing assignment during the semester, Negretti had the students engage in writing reflective journals (five entries per assignment), which elicited students’ writing task strategies, how they specifically used them when writing (i.e., monitoring), and how they evaluated their own writing performance. Negretti reported that learners’ metacognitive awareness, and especially that of conditional awareness, appeared to be strongly interconnected to self-regulation. In particular, Negretti suggested that conditional knowledge “help[ed] students know how to adapt their strategic choices to the specific requirements of the task and why” (p. 170), which in-turn, later fed back into learners’ awareness. Thus, Negretti theorized that students’ monitoring was closely related to students’ general awareness and how they perceived the writing task itself. All three of the studies described above—Negretti and Kuteeva (2011), Yeh (2015), and Negretti (2012)—highlight the complex nature of learning to write in a particular genre, and they suggest that when it comes to the development of learners’ genre knowledge, not all facets will develop uniformly at the same rate. This is to say: students may quickly excel in one area such as planning, but then, they may face struggles in evaluating their own performance in relation to the audience or discourse community’s expectations (e.g., Yeh, 2015). Conversely, it may be the case that one facet of metacognitive knowledge such as conditional knowledge may be more closely tied to aspects of metacognitive regulation like monitoring (e.g., Negretti, 2012). These aforementioned studies show the potential of adopting metacognition theory for understanding the intricate nature of L2 learners’ metacognitive genre awareness. Importantly, too, is that these studies can provide practitioners with practical implications. For instance, in the case of Yeh (2015), one implication for an L2 writing teacher would be to acknowledge that early on, 28 students will likely struggle in evaluating their own work. Therefore, the teacher might wish to build in group or peer review and feedback activities to help facilitate the building of this awareness. Though metacognition theory appears to be suitable for exploring L2 learners’ development of metacognitive genre awareness, it is important to note that the few prior studies that have adopted the model for examining changes over time have been conducted in a remarkably unified context. In all three of the aforementioned studies: (1) the instructors are linguistically trained, typically with advanced degrees in applied linguistics or teacher training in English as a Second/Foreign Language (ESL/EFL); and, (2) the students are taught academic genres-only (i.e., typically academic research papers); and finally, (3) the students often are explicitly taught using a genre-based ESP approach, in which students sometimes even read the works of Swales and practice applying rhetorical move analyses to published academic research articles. Additionally, in their explorations into learners’ development of metacognitive genre awareness, with the exception of Negretti (2012), prior studies have only explored awareness by examining sub-sections of the model. For instance, in Negretti and Kuteeva (2011), the researchers only explored the component of metacognitive knowledge, and in Yeh (2015), the researcher only explored metacognitive regulation. Yet, all of the aforementioned studies suggest that metacognitive genre awareness does not develop uniformly; and, when both components of the theory are explored as in Negretti (2012), the findings highlight how important it may be to understand not only which facets of the model develop, but also, what connections (or, disconnections) exist between general awareness and the regulation of performance in sub-processes like planning, monitoring, and evaluating. Thus, it is crucial to 29 understand both how this knowledge develops, and, to what extent it relates to actual writing performance. The Current Study The current dissertation is motivated by a number of gaps in the SLA and L2 writing literature, along with recurring calls for research related to furthering scholars’ understandings of both the teaching and learning of written genres in L2 contexts. To summarize these needs, they include the following three (3) areas: (1) the need for more studies to explore existing practices in L2 genre teaching and pedagogy, especially through the adoption of ethnographic approaches, as such explorations may assist SLA researchers in further understanding the current state of the field and the full scope of genre-based pedagogy (e.g., Johns et al., 2006; Paltridge et al., 2016; Russell, 2001); research is particularly needed in a greater variety of CBI contexts, as those studies involving CBI have most often been conducted in K-12 and STEM settings (see Lyster, 2017 and Tedick & Wesley, 2015 for reviews); (2) the need for studies to explore the development of L2 learners’ genre knowledge through the adoption of longitudinal designs, as much research has tended to examine L2 development using timed-writing tasks (see Polio, 2017); the adoption of metacognition theory is especially apt for such purposes, as prior studies adopting the theory have highlighted the intricate nature of developing metacognitive genre awareness (e.g., Negretti, 2012; Negretti and Kuteeva, 2011; Yeh, 2015); and finally, (3) the need for future studies to explore not only the development of L2 learners’ metacognitive genre awareness in different contexts, but also, to do so by examining 30 the full metacognition model and the relationship between learners’ general awareness and performance, as prior studies have been limited to exploring sub- sections of the model (e.g., Negretti and Kuteeva, 2011; Yeh, 2015) and/or have been highly contextual in their use of ESP approaches (e.g., Negretti, 2012; Negretti and Kuteeva, 2011; Yeh, 2015). Therefore, in attempts to address these issues, the current dissertation explores phenomena related to L2 genre teaching and learning in an LLM law school setting, as a law instructor teaches L2 English learners how to research and write the professional legal genre of the office memo. This dissertation is guided by the following three research questions (RQs): RQ1: How does an instructor in a content-based legal research and writing course attempt to raise genre awareness surrounding the office memo? RQ2: How does students' metacognitive genre awareness of the office memo develop over the span of a 15-week academic semester? RQ3: To what extent do students' self-reports involving their metacognitive genre awareness reflect their writing performance as perceived by the course instructor? 31 CHAPTER 2: METHODOLOGY Context and Participants Several months prior to the start of the study, I approached administration officials at Hutz Law School (pseudonym)—a premier law school located in the Midwestern United States—about potentially conducting research on the law school’s premises, including the prospect of attending classes as an observer and collecting data involving a teacher and his/her students. Following multiple discussions with Hutz’s administration, I was granted permission by both the law school and an instructor of the LLM’s legal research and writing course to attend classes and to collect data during the fall 2019 semester. As mentioned, Hutz is a law school located in the Midwestern U.S., and admissions are highly competitive. The school offers a number of degree program options for students such as a Juris Doctor (JD), a Master of Laws (LLM), and a Doctor of Juridical Science (SJD), along with additional programs and activities for enrolled students such as moot court competitions, publishing opportunities through various law school-based journals, other clinical programs, student organizations, and more. Participants in the current study included both the primary instructor for Hutz Law School’s LLM legal research and writing course and multiple students (N = 6) who were enrolled in the LLM program. Additional information surrounding the course, the course instructor, and the LLM student-participants is described below. The legal research and writing course. During the LLM program, international students are provided with opportunities to take classes alongside other (typically American) students enrolled in the JD program, yet the LLMs are also given the opportunity to enroll in an optional legal research and writing course, which is specifically designated for LLMs-only. This legal research and writing course was designed and is currently taught by Professor Meyer 32 (pseudonym), and the course has the stated goal of introducing LLMs to “how American lawyers research and analyze legal problems and communicate their analysis to senior attorneys and clients” as well as focusing on “what senior attorneys in the U.S. expect of junior lawyers” in the areas of legal analysis, written and oral communication, and research (excerpts taken from course syllabus). The legal research and writing course is offered only once-per-year during the fall semester, and LLMs can elect to enroll in the course while simultaneously taking other classes at Hutz. The course is divided into two sections. The first section is eight weeks and introduces students to legal research and analysis in the American common law system, with special emphasis placed on conducting research and learning to write the office memo genre; it culminates in students completing one office memo assignment. The second section of the course, which is seven weeks, expands on the skills learned during the first section, and it provides students with an additional opportunity to continue practicing their legal research and analysis skills in the drafting of a second office memo. This second half of the course also has a small, supplemental focus on oral communication, which briefly provides LLMs with “the chance to practice orally communicating [their] analysis of a problem to a supervising attorney” (excerpt from course syllabus). Students can select to take the legal research and writing course for one credit (i.e., the first eight-weeks only) or two credits (i.e., the full semester). Those who select the two-credit option for the course conduct research for and write a total of two office memos as part of the class. At Hutz Law School, most LLMs who select the two-credit/full semester option do so because they plan to remain in the U.S. after completing the LLM program in order to take the bar examination. This completion of two academic credits helps fulfill part of the eligibility criteria to sit for the bar in certain states such as New York, which stipulate a 33 minimum of two law school credits “in legal research, writing and analysis” must be met (see “The New York State Board of Law Examiners,” n.d.). Professor Meyer. The instructor of the legal research and writing course, Professor Meyer1, teaches one section of the LLM legal research and writing course at Hutz Law School. Meyer, who is a Caucasian, American male in his 50s, practiced law for a few years at a large firm directly after completing his JD. After working at the firm, Meyer then worked in private practice, where he also worked for a number of years. During our introductory interview, Meyer mentioned that he first became interested in teaching as a potential career during law school, adding: I had always been told I was a good legal writer, and I always enjoyed that aspect of it (Interview #1). While running his own practice, when an adjunct position became available at a different nearby law school, Meyer decided to teach one class in order to see if he liked the role. Finding that he enjoyed teaching, when a full-time position became available at another law school in the Midwest, Meyer jumped at the opportunity. Since that time (and at the time the current study began), he had been teaching at Hutz Law School for nearly two decades, serving as a professor of legal research and writing, offering different sections that are intended for L1 (i.e., mainly JDs) or L2 English speakers (i.e., LLMs). Apart from self-study in the form of reading various academic law journals, books, and attending legal writing conferences, Meyer stated that he had no formal teacher training and no formal training in working with L2 learners. LLM students. Prior to the start of the fall 2019 semester, I contacted all 22 of the LLM students enrolled in Professor Meyer’s legal research and writing course via email. The 1 Some of the details surrounding Professor Meyer’s background and experiences are left intentionally vague to help protect his identity. 34 recruitment email did not reveal the full details of the study, yet it stated that I was interested in learning more about LLM students’ general experiences throughout their legal programs and training, including various issues related to L2 English learning such as reading and writing. For those who were interested, the email invited prospective participants to complete an online background questionnaire (see Appendix A) prior to beginning their fall semester coursework, and the survey also included a section for individuals to volunteer to participate in the case study, which would occur throughout the fall semester in the form of multiple interviews. In total, 14 LLM students completed the background questionnaire and volunteered to participate in the case study. Of those 14 individuals, six (N = 6) LLM students were ultimately selected to participate in the case study. These six students were chosen based on purposive sampling (Duff, 2008) so that the students represented a broad and diverse participant pool (e.g., representing different genders, L1s, etc.) with a range of experiences (e.g., those with formal work experiences post-law school, those who had recently graduated, etc.). The decision to adopt purposive sampling was made to further explore the potential influences of such aforementioned factors on individuals’ reported metacognitive genre awareness, especially since prior studies have suggested that the acquisition of genre knowledge is multifaceted and that performance may be susceptible to individual factors such as a learner’s content familiarity, linguistic differences, and even L1-based knowledge such as a student’s rhetorical expectations (e.g., Gao, 2012; Tardy, 2005, 2009). Below, I describe each of the six case study participants in further detail. Additionally, Table 1 provides a more concise description of the six participants (note: pseudonyms are used throughout this study for all participants’ names). Aashna. Aashna was one of the two youngest LLM participants in the current study. At 23 years old, Aashna was an L1 Hindi speaker who had formally studied English for 17 years in 35 her home country of India. She entered her law school program in India shortly after finishing high school. After finishing law school, she had directly enrolled in the LLM program at Hutz Law School. During our first introductory interview, Aashna made it clear that she was worried about her age and lack of experiences relative to her LLM peers, stating: Just a precursor to Professor Meyer’s class, I have literally no idea what I'm getting into... Like I literally don't know because I have just graduated from law school in India, and law school there is very different...so I'm literally like youngest person in my class. I'm 23. I know nothing about anything. (Interview #1) Aashna had no prior professional work experiences, and she had no experiences in reading or drafting office memos during law school. She stated that most of her writing had been for testing purposes, which consisted primarily of memorizing specific Indian law statutes and then describing or restating them verbatim. However, Aashna did have several experiences writing for law review journals—(scholarly work in which writers often summarize multiple laws and rulings, and then argue a specific point of view in attempts to summarize different positions and/or persuade readers that a law should be changed [see Delago, 1986]). Her personal interests involved areas of law such as antitrust and competition laws. After completing the LLM program, she hoped to stay in the U.S. and work for a few years. Then eventually, she hoped to return to India and share the knowledge she had gained from her time spent in the U.S. Jing. Jing was a female student from southwest China who had studied her L2 of English for 11 years. During her legal education program in China, Jing had experiences in reading a variety of legal documents, including: answers or complaints, law review articles, office memos, and more. In terms of writing, she also had experiences in drafting law review articles. After completing her legal education program, Jing worked as a legal specialist for two 36 years at an international law firm in southern China. At the firm, she focused on cash markets, compliance issues, and advising clients of different rules and regulations pertaining to the Hong Kong Stock Exchange. During the course of her work, Jing stated that she occasionally came into contact with English office memos (she estimated approximately once-per-month), but she was unsure whether those memos from her firm reflected those produced in the U.S. context. Despite her work experiences, her primary interests were in public interest and human rights law. Following the LLM program, Jing hoped to pass the bar and then return to China, stating: In the international law firm, if I want to be an associate, then they need us to have a foreign bar degree. So…the most reason I attend the program is to get to take the bar exam—to sit on the bar. So after graduate, I mean if hopeful it is like smoothly…I will do the New York Bar, and I will return to XXX [city] and to be an associate. (Interview #1) Table 1 Case study participant bios Name (pseudonym) Aashna Age Home country India 23 Jing 24 China Luisa 29 Mexico Paul 29 Germany Yuki 30 Japan Years of L2 English study 17 11 13 7 7 Yurui 23 China 15 Additional background post-law school and experiences involving office memos* • Recently graduated from law school; no experiences reading or drafting memos • Worked as a legal specialist for two years; a few formal work experiences in reading memos • Worked as a legal research associate for two years; a few work experiences in reading and drafting memos • Served as a law school research assistant for three years; no experiences in reading or drafting memos • Served as an attorney for nearly five years; some formal work experiences in reading or drafting memos • Recently graduated from law school; 2 experiences reading or drafting memos *All participants’ law school and post-law school experiences occurred outside of the U.S. Luisa. Luisa was an L1 Spanish speaker from Mexico. At 29 years old, she had formally 37 studied English for 13 years. She received her legal education in Mexico, and during that time, she had experiences both reading and drafting many types of legal documents, including answers or complaints, appellate briefs, client letters, and more. Post-graduation, she also gained experience by working as a research associate in Germany for two years, where she primarily researched topics related to investment and commercial arbitration, as she so described: So basically, [I was] a second-class associate in which you are doing the job of an associate but paid less, if that makes sense… The first thing they told me to do was like to write a commentary of certain rules of arbitration that they were gonna publish. So I had to draft a commentary… For example, that was my first task, and I did that for like a month. Then…as I say… I was doing working as an associate, but I was also at the same a trainee, so I had to do both things, mainly just researching. (Interview #1) Luisa indicated that she had some familiarity with office memos due to her work as an associate, but she, too, was unsure whether memos in the U.S. context were similar to those she had encountered in Germany. After two years of work, Luisa decided to enroll in the LLM program at Hutz Law School. Following the LLM program, she hoped to pursue an SJD and then practice law in the U.S. before eventually returning home to Mexico. Paul. Paul was an L1 German speaker who had studied English for only 7 years prior to enrolling in the LLM program. Having received his legal education in Germany, Paul stated that during his schooling he was exposed to relatively few types of legal documents, mainly limited to reading cases, statutes, and law review articles. During this time, he primarily engaged in writing for the purposes of school exams and also for writing law review articles. Post- graduation, Paul served as a legal research assistant for three years under a professor at a law school in Germany. There, he researched various issues, although much of it aligned with his 38 own interests in comparative law and constitutional law. In the future, after completing the LLM program, Paul was still unsure of what he wanted to do, although he was considering pursuing a career in academia, stating: I haven't really figured it out yet… After finishing law school, we have to do some practical training, and there we work for judge and we work for a prosecutor and we work in the administration, and then you have to pass the bar, basically. And then you can decide if you want to become a judge or pursue an academic career or be an attorney. But it's very much all up in in the air right now. I guess at this point, I really would favor the academic area, but it's really really hard to get into that… It's another four or five years of research and writing articles. And then, you only have a chance. (Interview #1) Yuki. At 30 years old, Yuki was the oldest of the six case study participants. Like Paul, Yuki had studied English for only 7 years in his home country of Japan prior to joining the LLM program, where he had also served as an attorney for nearly five years. He described his job: I work for XXX, which is one of the biggest law firm in Japan…a corporate law firm. So there are many various kinds of law fields, and we have many groups, but I only belong to that one group which is often deal with the financial transaction… So in that transaction, it is not litigation, so we prepared some agreements for the financial transaction. For me…maybe 95% of my working time is sitting in front of the PC and receiving agreements, for example: loan agreement, some purchase agreement, for the real estate… That's the main in my job. (Interview #1) Prior to his professional work, Yuki also had relatively little exposure to different types of legal documents during his education, and he mostly read cases and statutes for the purposes of preparing for examinations. Yuki was unsure of what an office memo was, yet he stated that he 39 had many experiences in reading or drafting internal documents for his firm. Post-LLM program, Yuki’s law firm was sending him to Australia for one year to work before he would ultimately return to Japan to continue practicing law. Yurui. Yurui was the final case study participant, and along with Aashna, he was one of the two youngest students in the study. At 23 years old, Yurui was an L1 Chinese speaker, and he had studied English for 15 years prior to joining the LLM program. Yurui came from Eastern China, where he also received his legal education. While there, he read many different types of legal documents, including: answers or complaints, appellate briefs, client letters, law review articles, and more. During his program, he also stated that he had the opportunity to intern at two law firms in China, where he primarily conducted research yet also had a couple of low- stakes experiences in reading and drafting office memos, stating: I write memos, basically, I think it's about compliance issues…corporate compliance issues and mergers and acquisitions…[but] I was told that my supervising lawyers would help me to modify my memos, so I was not so stressful for that. (Interview #1) Following the completion of his legal education, Yurui applied directly to the LLM program via a cooperative agreement between his law school and Hutz Law School. After completing the LLM program, Yurui was considering pursuing an SJD in the areas of civil and commercial law, yet he had not finalized his plans. Researcher Positioning Because one of the primary research questions (i.e., RQ1) involved understanding the pedagogical practices of a content-based instructor through the adoption of a classroom-based ethnographic approach (e.g., Bloome, 2012), it was necessary for me to attend each weekly meeting of the legal research and writing course. Therefore, knowing that I would be a class 40 attendee and a visible presence throughout the semester, I attempted to establish a partial-emic (i.e., insider) perspective throughout the duration of the study (see Duff, 2007 for more on establishing emic perspectives in qualitative, classroom-based research). As I was a doctoral student and a relatively experienced instructor at the time of data collection, this duality of my own status helped me in establishing relationships with both the LLM students and Professor Meyer. In addition, prior to becoming a doctoral student, I had previously worked at a law school for three years, where I served the law school dean, faculty, and university administrators in tasks involving grant writing and communications; this also assisted me in establishing connections with the participants. Thus, during our interactions, I attempted to make connections with students surrounding life as a graduate student and the various challenges they faced during their program; I also attempted to connect with Professor Meyer in terms of understanding challenges related to teaching research and writing. Although the course instructor and students were paid for each interview they participated in, as De Costa (2015) has suggested, I attempted “to treat the [research] exchange as being more than a transaction” (p. 249). Specifically, I attempted to achieve this by offering to serve as a researcher-as-resource (Sarangi & Candlin, 2003). For instance: for the course instructor, I offered to provide feedback regarding students’ perceptions of different facets of the course along with the creation of a “white paper” document, or a condensed version of the research findings; and for the students, I offered to provide them with individualized feedback on their writing following the completion of the study, an offer which multiple students accepted. Data and Data Collection Timeline In order to address the three research questions (RQs) in the present study, in terms of data, a number of sources were collected pertaining to the legal research and writing course over 41 the span of the fall 2019 semester. Table 2 highlights the specific data that were used to address each RQ, respectively. Furthermore, Table 3 subsequently outlines the data collection timeline for the entire study’s duration. In what follows, I provide a more detailed description of the data that were collected. Artifacts. Throughout the duration of the study, numerous legal research and writing course-related artifacts were collected, which contributed to the investigation of RQ1 involving the instructor’s approach to pedagogy and building awareness of the office memo genre. These artifacts included various documents such as: 2 course syllabi (i.e., 1 syllabus for each section of the class; 6 pages in total) along with 71 different course readings and handouts (477 pages in total), all of which were posted in a digitally accessible format on the legal research and writing class’s online course management system. Table 2 Research questions (RQs) and corresponding sources of data RQ# RQ1 Focus of question Pedagogical practices of the law instructor (i.e., his techniques for building genre awareness) Sources of data • Artifacts (i.e., syllabi, course readings, classroom assignments/materials) • Field notes from classroom observations • Interviews with instructor • Visualization reflection • Background questionnaire • Interviews with students • Stimulated recalls with students and their office memos • Interviews with students • Stimulated recalls with students and their office memos • Interviews with instructor (including his memo rankings) RQ2 Development of students’ metacognitive genre awareness RQ3 Relationship between students’ self-reported metacognitive genre awareness and writing performance as perceived by the instructor Field notes. In addition to collecting course artifacts to address RQ1, I attended the legal 42 research and writing class in person throughout the academic semester. During each class session—which occurred once-per-week, lasting two hours in length—I took extensive field notes using a word processing program (see Copland, 2018 for more on the use of observational field notes in qualitative research). For taking field notes, I created a word document. In this document, prior to each class, I wrote a brief paragraph that described the previous night’s homework assignments. Then, underneath that paragraph, for each day of the class I created a table with three columns. The first column was labeled “timing” to list when each new in-class activity began and how long it lasted. The second column was labeled “activity,” in which I created a title to designate the type of activity occurring (e.g., “exercise with case XXX” or “discussion on proofreading memos”). The final column was labeled “description/notes,” in which I took extensive notes on the professor’s pedagogy, the materials he used, students’ interactions during those activities, and more. In total, 34 pages (13,051 words) of field notes were taken during the in-class observations. Semi-structured interviews (course instructor). As shown in Table 3, I interviewed the legal research and writing course instructor, Professor Meyer, on three separate occasions. Each interview lasted approximately 1-to-1.5 hours in length, and Meyer was compensated with $30 for each interview he completed (i.e., $90 in total). For the first semi-structured interview, which occurred prior to the beginning of the semester, the goal was to learn more about Meyer’s background, including his professional work and teaching experiences, but also, to learn more about the legal research and writing course itself, along with Meyer’s perspectives regarding teaching the class, the LLM student population, and more. The second semi-structured interview with Meyer occurred midway through the semester after students had submitted their first office memo assignment for the course. For semi- 43 structured interview #2, the goal was twofold: (1) to further understand specific pedagogical practices and decisions made by the instructor (e.g., decisions regarding reading topic selections, general course design, etc.); this aided in the investigation of RQ1. Additionally, (2) to have Meyer evaluate and rank the six case study students’ office memos in terms of overall effectiveness, which contributed to the analysis of RQ3 regarding the connection between students’ metacognitive genre awareness and their texts’ perceived quality. Prior to beginning semi-structured interview #2, Meyer had graded the case study participants’ first office memo assignments. In the interview, Meyer was asked to imagine that he was a senior attorney at a law firm who had assigned the office memo. Then, Meyer was asked to rank the six memos (against each other) in terms of overall effectiveness (i.e., as a senior attorney considering all aspects of content, organization, language, legal analysis, etc. in relation to the functional purpose of the office memo genre). After ranking the six memos, Meyer then discussed the strengths and weaknesses of each memo in detail, including why he ranked them as such in relation to the other memos. This combination of understanding the instructor’s pedagogical practices plus having him rank and comment on the memos was repeated again for semi-structured interview #3 with Meyer, which occurred after students had submitted their second/final office memo (see Appendix B for the full list of interview questions for Interviews #1-3 with Meyer). See Table 3 for the respective sequencing of these semi-structured interviews. Visualization reflection. The final data to be collected for addressing RQ1 was a visualization reflection. In various studies that involve the use of drawings by study participants, participants are often asked to illustrate something related to their learning and/or classroom activities and then to subsequently describe or reflect upon their illustration. Such techniques have widely been used across multiple disciplines with different populations, including studies 44 examining: the testing of children in K-12 settings (e.g., Winke, Lee, Ahn, Choi, Cui, & Yoon, 2018), the teaching practices and identities of adult teachers (e.g., Weber & Mitchell, 1996), adult L2 learners (e.g., Wargo & De Costa, 2017), and, with students in content-based science classrooms at various ages/levels (e.g., Stieff, 2017). The use of participant-produced drawings have such purported benefits as serving as scaffolds for participants and promoting metacognition and reflection, which ultimately has the potential to positively influence learning outcomes (Lowe & Ploetzner, 2017; Naug, Colson, & Donner, 2011). For the visualization reflection in the current study, near the end of the third semi-structured interview, I asked Meyer to consider his approach to teaching the legal research and writing course and the LLM students. Then, I provided Meyer with multiple pens and a blank sheet of paper and asked him to draw/illustrate his approach to teaching the class. After drawing, Meyer was asked to orally describe the picture he had produced. Students’ office memos. As discussed, during the legal research and writing course, students were assigned two office memo projects, which served as their only two writing assignments for the course. One office memo was due midway through the semester in Week 8, and the second memo was due at the end of the semester in Week 15. The scope and topics related to these two office memos are discussed in greater detail in relation to RQs 1-2 in the Findings section of this dissertation. However, the six focal students’ memos submitted for these two assignments totaled 109 pages of writing, but they varied in length, ranging from 6-12 pages (for memo 1) and 7-14 pages (for memo 2). Students’ memos were written using Microsoft Word and were submitted simultaneously to the course instructor for grading, but also, to me for use in the stimulated recall sessions for addressing RQs 2-3. Semi-structured interviews and modified stimulated recall sessions (students). As shown in 45 Table 3, three semi-structured interviews were conducted with students during the semester. These three interview sessions were carefully designed in order to understand students’ previous writing experiences, education, and existing background knowledge concerning the office memo genre (Interview #1), but also, as to how their knowledge of the office memo genre developed over the span of the semester (Interviews #1-3). In order to understand this development of metacognitive genre awareness, a fixed list of questions was created, and I asked students these same questions during all three of the interview sessions. The questions were created in reference to the metacognition model described earlier in the Literature Review, and specifically, these questions (see Appendix C) were drafted in relation to the metacognitive knowledge component of the model (i.e., declarative, procedural, and conditional knowledge of the office memo). For example, one declarative question that was asked during all three of the interviews was: “What is the purpose of writing an office memo? Why would someone write one?” Thus, the goal was to understand how learners’ responses to such fixed questions evolved over the span of 15-weeks. In addition to assessing students’ metacognitive knowledge (i.e., their general awareness) of the office memo, during interviews #2 and #3, a modified stimulated recall was conducted with each student following the completion of his/her office memo. As Gass and Mackey (2017) have stated, typically, a stimulated recall should be conducted immediately following a task’s completion. However, in the current study, due to the extended nature of the office memo writing tasks themselves combined with the busy schedules of the participants, logistically, this was not an option. Therefore, the current study adopted a modified stimulated recall (see Brooks & Swain, 2009 for a related ‘augmented’ stimulated recall), in which all recall sessions were scheduled and completed within 24 hours of students finishing their respective memos. Like the 46 questions created for all three interviews to assess students’ metacognitive knowledge of the office memo genre, questions were also created to assess students’ metacognitive regulation when writing their memos (i.e., their planning, monitoring, and evaluating during the writing process). Thus, the use of the term ‘modified’ stimulated recall in the current study refers to the fact that learners’ were asked to reflect on various phases of their writing processes (i.e., before, during, and after writing). For example, after being asked a declarative question such as “Who reads office memos (i.e., who is the audience)?”, participants would then be asked a follow up stimulated recall question about monitoring, such as: “Was there a time you thought about the audience while writing and you changed or added something because you were thinking about them? If so, can you give me an example(s) from your memo?” Table 3 Timeline for data collection Week Pre-course (week prior) Data collected • Interview with instructor • Background questionnaire with students • Interviews with students Additional information • #1 of 3 • #1 of 1 • #1 of 3 Week 1 Week 8 Week 10 Week 15 Post-course • In-class observations begin (including collection of artifacts) • Students’ office memos submitted • Interviews with students • Modified stimulated recall with students • Interview with instructor (including rankings of first memos) • In-class observations conclude • Students’ office memos submitted • Interviews with students • Modified stimulated recall with students • Interview with instructor (including rankings of second memos) • Visualization reflection • #1 of 2 • #2 of 3 • #1 of 2 • #2 of 3 • #2 of 2 • #3 of 3 • #2 of 2 • #3 of 3 • #1 of 1 Each combined semi-structured interview and stimulated recall session lasted 47 approximately one-hour. For each session, case study participants were compensated with $30 (i.e., $90 in total for all three sessions). To see a complete listing of the semi-structured interview and stimulated recall questions for all three sessions, please see Appendix C. Data Analysis All interview and stimulated recall sessions were audio-recorded. Those recordings were then converted into .mov files and uploaded into a private YouTube channel available only to the researcher, where they were automatically transcribed. The subsequent transcriptions were inspected for accuracy and corrected as necessary. Interview and stimulated recall transcripts, along with field notes (44 pages of notes in total), were then uploaded into MAXQDA (VERBI Software, 2019), which is a qualitative and multi-media data analysis tool. Additionally, although many artifacts were collected during the semester-long course, only selected course artifacts (e.g., the syllabi and some readings, activities, and class handouts) were uploaded into MAXQDA for analysis, as the entire artifact dataset constituted 483 pages of material in total, some of which were not pertinent to the current study. RQ1. As discussed, in order to address RQ1 concerning the instructor’s approach to teaching legal writing and raising genre awareness, I adopted a classroom-based ethnographic approach (e.g., Bloome, 2012) in the collection of data. Thus, through the collection of a variety of data sources over an extended period of time, I attempted to provide a thick description (Geertz, 1973) of the field site (i.e., Meyer’s classroom) and to triangulate the findings by using multiple sources (e.g., the instructor’s visualization reflection, interview data, his in-classroom practices, assigned readings, etc.). Once the data were uploaded into MAXQDA, I independently coded those data using an inductive two-cycle coding approach (e.g., Miles, Huberman, & Saldaña, 2014). In adopting this 48 analytical approach, an initial round of open coding was conducted involving the relevant data sources, in which I assigned codes to strings of text based on the themes that emerged from the data. Then, in the second round, those codes were subsequently grouped according to similar themes. For example, during the data collection process, Meyer made the following three comments, respectively: [Comment #1, made during Interview #1]: The focus has to be on the audience. I mean, who is it that you're writing to, and what is it they need? Considering in particular that, you know—so many readers of legal writing, they're busy people. So you have to get to the point quickly… [Comment #2, made in-class during Week 2]: Explain why you’re organizing it the way you’re doing it. You’re trying to anticipate what your audience wants… You just gotta tell them what you’re doing. [Comment #3, made during Interview #2]: Trying to write as a lawyer is so broad—I mean, it covers such a range of topics. There aren’t like… ‘If you can master the proper use of these five or ten phrases, you can sound convincingly like a lawyer.’ There's nothing like that. Thus, in the two-cycle coding approach, for the first round of coding, I coded comment #1 with the code “Audience – Fronting/Forecasting Conclusions,” since Meyer was discussing the importance of students being able to understand their audience and how that audience affected “get[ting] to the point quickly” by stating conclusions outright; for comment #2, I coded the string as “Audience – Fronting/Forecasting Organization,” as Meyer was again discussing the audience’s impact on writing, but this time, as to how a writer could chose his/her organization but still needed to spell out this information for the reader; and then, for comment #3, I coded it 49 as “Philosophy – No language instruction,” since Meyer was addressing why he chose not to provide any in-class language instruction when teaching his legal research and writing course. After these codes were assigned in the first round, during the second round of coding, comments #1 and #2 were grouped together due to exhibiting similar coding-themes, while comment #3 was not included with these two and was instead placed into a different category. RQ2. Again, RQ2 involved individual case studies in which I tracked the development of six students’ metacognitive genre awareness. I first independently coded a small subset of the data through a combination of deductive and inductive coding (e.g., Paltridge & Phakiti, 2015; Polio & Friedman, 2017). Deductive coding was used since, prior to the study, specific questions were designed to elicit students’ metacognition surrounding the office memo in accordance to the metacognition model (i.e., students’ declarative, procedural, and conditional knowledge, along with their planning, monitoring, and evaluating abilities). However, inductive coding was also employed in the data analysis, since students’ comments might be idiosyncratic in nature within the model itself. For instance, when responding to a declarative question, one student might exhibit awareness of the differences in memos between L1 and L2 contexts; meanwhile, another student might not respond with such a cross-cultural comparison and instead may exhibit some other sub-type of declarative awareness. The coding scheme was first developed by the researcher through an initial reading of multiple interview transcripts. Once the coding scheme was solidified, a second coder received training on the metacognition model and then independently coded 11.11% of the student-dataset (the coder-training sheet along with specific examples can be viewed in Appendix D). When coding for the six different types of metacognition in students’ transcripts as shown in Appendix D, any occurrence or instance of a student showing metacognitive genre awareness (e.g., 50 declarative, procedural, etc.) was marked simply as being present (see Kessler, in press, for a related example). Inter-coder reliability was sought for RQ2 as I hoped to obtain a simple raw frequency count for the number of metacognitive comments that students were able to make throughout the 15-week course. If a code appeared more than once in a single transcript, it was counted as such for the purposes of seeing which respective areas students’ developed the most and/or least awareness in. A comparison of the codes revealed inter-coder reliability of 82.93%, after which, we discussed and resolved any differences. Following the establishment of inter- coder reliability, I independently coded the remaining dataset. RQ3. Finally, for RQ3 involving the relationship between students’ metacognitive genre awareness and their texts’ perceived quality, the qualitative codes and raw frequencies obtained during the analysis of RQ2 were compared against comments and rankings made by the course instructor. As mentioned, during interviews #2 and #3 with the course instructor, Professor Meyer was asked to rank the six case study participants’ office memo texts in order from most- to-least effective, and, to provide additional evaluative comments regarding why he gave such rankings. As with RQ1, I independently coded the instructor’s comments using an inductive two-cycle coding approach (e.g., Miles et al., 2014). Thus, to evaluate RQ3, I analyzed two areas: (1) I compared Meyer’s evaluative comments with/against his own rankings in order to examine the extent to which his evaluations and rankings aligned; and (2) I then compared Meyer’s effectiveness rankings with the total number of metacognition comments that were self- reported by LLM students when discussing their own memos (i.e., the comments they made during interviews/stimulated recalls #2-3). 51 CHAPTER 3: FINDINGS Below I report the findings for the three RQs in the current study. First, I showcase the pedagogical practices of the legal research and writing course instructor, Professor Meyer, including information regarding his philosophy of teaching LLM students and how this translates into course design and his classroom techniques for building students’ reading skills and raising genre awareness (RQ1). Secondly, following the presentation of the instructor’s beliefs and practices, I highlight the development of the six LLM case study participants in terms of their reported metacognitive genre awareness and its evolution over the span of the academic semester (RQ2). Finally, I present information on the relationship between students’ self-reported metacognitive genre awareness and their perceived writing performance (RQ3). RQ1: Law Instructor’s Pedagogical Practices in a Content-Based LLM Writing Course In illustrating the findings for RQ1 on the practices of the legal research and writing course instructor, I present the findings in three areas in relation to: (1) the overarching structure of the course, including the content, topics, and activities; (2) Meyer’s personal philosophy of teaching; and lastly, (3) how Meyer’s philosophy is connected to his in-classroom practices and strategies for building students’ reading skills and raising awareness of the office memo genre. Legal research and writing course structure. As mentioned in the Methodology chapter, Meyer’s legal research and writing course followed a 15-week academic semester, and the course was split into a one-credit and two-credit bearing option. However, a majority of the LLM students (both in the current study and in prior years) have tended to select the two-credit option because they eventually plan to take the bar examination in the U.S. and to remain in the country to practice law. To view a synopsis of the various topics, content, and activities that were covered during the legal research and writing course, see Table 4 below. 52 Table 4 Overview of schedule, topics, and activities for legal research and writing course Week Topic(s) Intro to course, American court system, and common law Description of in-class activities -Welcome and overview of course -Analysis of case #1 How common law develops, and intro to office memos -Analysis of case #2 -Intro to office memo: Organization How common law develops, and researching and writing memos -Analysis of case #3 -Info on how to organize a memo and how to conduct legal research Conducting legal research, and intro to first memo assignment -Intro to memo / research assignment #1 -Info on how to conduct legal research -Practice in using legal research databases Approved cases for memo #1, and how to write a memo No class -Discussion of 8 cases for use in memo #1 -Analysis of sample office memo outline -Research list for memo #1 due this week -Individual conferences with students Reflecting on research process, and research-related ethics -Discussion of research process and ethics -Office memo #1 due this week Reading statutes, and style / proofreading in office memos -Intro to memo / research assignment #2 -Practice reading and interpreting statues -Discussion of memo #1 common errors -Analysis/critique of sample office memo 1 2 3 4 5 6 71 8 11 T/GW* T GW/T T T T T T T GW T T T T GW/T T GW/T T T T T T GW/T 9 More on researching memos, and info on oral presentations -Discussion of research process and tips -Intro to oral presentation assignment -Research list for memo #2 due this week 10 Approved cases for memo #2 -Discussion of 11 cases for use in memo #2 Students’ oral presentations -Mock oral presentations to senior attorney GW/T 12 No class 13 No class (due to holiday) 14 Info on using memos as writing samples, plus professional emails and ethics -Individual conferences with students -Discussion of using a memo as a writing sample for a job application -Analysis of professional email writing -Activity on ethics and conflicts of interest 15 No class (finals week) -Office memo #2 due this week *T = Teacher-led lecture or discussion; GW = Group work and/or small group discussion 1End of 1-credit course option As shown in Table 4, the legal research and writing course focused on a number of topics, many of which were discussed repeatedly throughout the span of the 15-week semester. 53 Primarily, these recurring topics included discussions of six central themes, which were related to: (1) understanding the U.S. common law system; (2) interpreting/understanding various readings in the form of case law, statutes, etc.; (3) conducting legal research; (4) writing office memos; (5) delivering information orally to one’s supervisor; and finally, (6) contemplating ethics-related issues. Although these six themes were integral components of the course, based on my own in-class observations and field notes, Meyer tended to devote a majority of the in- class sessions to the specific topic of (2)—that is, assisting LLM students in reading-based issues of interpreting/understanding various readings in the form of case law, statutes, and more. While building students’ reading skills appeared to be the main focus of most of the in-class sessions, two other topics did receive a substantial amount of attention, and those were the topics of (3) conducting legal research, and (4) writing office memos. In the next sections, I explore how Meyer’s own philosophy of teaching LLM students legal writing impacted a myriad of factors related to classroom pedagogy, including: his selection of topics; the context and environment he attempted to foster in his classroom; and subsequently, how Meyer’s own views on teaching manifested into actual classroom techniques/practices related to the style and delivery of information, in building reading skills, and in raising awareness of the office memo genre. Professor Meyer’s philosophy of teaching L2 legal writing. As discussed, near the end of the study’s data collection, I asked Meyer to participate in a visualization reflection activity. Since the activity occurred at the conclusion of the legal research and writing course, it was appropriate for the purposes of having Meyer to reflect on multiple aspects of his teaching, including the course itself and the broad scope of his own approach and philosophy to teaching LLM legal writing. Thus, I begin by presenting the results of this visualization reflection first, as 54 they provide insights into Meyer’s beliefs about his role as an instructor and the nature of teaching in general—and, crucially—as to how those beliefs later manifested into actual classroom practices. In the visualization reflection activity, Meyer was asked to consider his approach to teaching LLM students and legal writing in general, and then, to draw a picture that subsequently represented his approach. After completing the drawing, Meyer described the illustration in detail (see Figure 2 to view the illustration that he produced). As shown in Figure 2, there are two primary components that make up Meyer’s illustration, which include: (1) a trumpet/megaphone in the clouds, and (2) a ‘Rube-Goldberg-ish device’ at the bottom of the page, with the LLM students being situated in between both of these images. In terms of the illustration, Meyer described it as follows: There's two aspects to this: this [i.e., pointing to the trumpet/megaphone] is the voice from heaven…actually, the trumpet from heaven or trumpet from the clouds. But it's also sort of like a combination of the trumpet but also the cheerleader—like the megaphone. So it's kind of like that aspect of it… So that's the downward facing aspect of this and the sense of knowledge proceeding from above. But then there's also the support aspect… Meyer further explained: So I guess what I'm trying to say is… I'm sort of envisioning my role. I suppose that, in some ways…these are embodying me… But that's, I guess, it's kind of in one sense capturing how I see myself—I mean, as someone who is, you know, an expert in the area. I mean you are providing information from above, right? It's not the ‘sage on the stage,’ but it is. I mean, there's nothing wrong with learning from authority. That's how learning progresses in many respects. But also, there's the cheerleader aspect that you’re also helping the students from the ground up. 55 As Meyer described in his illustration, he envisioned his role as a legal writing instructor as being situated above that of his LLM students—that is, as a figure of authority in the classroom and as the primary disseminator of knowledge. However, along with this position of authority also came the unique dual role of being a motivator or “cheerleader,” and specifically, as a figure that could help foster students’ development through providing positive encouragement. Beyond viewing himself as an authority and key contributor in encouraging his students, Meyer also viewed his role in terms of the importance of understanding his LLM students and their respective abilities. When describing the second part of his illustration, Meyer explained: So there's like three different components of this [i.e., pointing to the bottom half/’Rube- Golberg-ish device’ in the drawing]… [There] are three different images that correspond to three different types of students. But you could also say that it's three different images that correspond to different skills that might apply to any particular student… So the first of these bottom images, it’s intended to be kind of like a stepstool. And the idea is that there are some students for whom progress just comes very slowly. And you try to move them from point-to-point-to-point… It takes a while. He then moved on to discuss another part of the image involving the ladder: The second of these—I'm going to skip to the ladder—that's the idea then again…we're always trying to advance upward in all of these scenarios. But other students can make those more extended climbs. So it's not just, you know, raising a foot or two off the ground, but it's moving from step-to-step, story-to-story, stage-to-stage…or, you know, from a beginner level, to intermediate level, to a high level. 56 Figure 2. Professor Meyer’s visualization reflection Meyer then discussed the third part of the image, the spring: 57 And then the third image here is supposed to indicate the spring—so which is more like…the student who just suddenly makes the leap. I mean, [the student] who makes maybe the intuitive jump, or just on this particular skill, just knows what he or she is doing, and so boom!—just kind of like jumps ahead… Maybe [he/she] doesn't need that sort of individual iteration of the various points. In following up on his explanation of the three components of the steps, spring, and ladder, I asked Meyer about his labeling of the “Rube Goldberg-ish device,” and, whether the image itself depicted the idea that he, as a teacher, felt his role was to identify which type of person each LLM student was. Meyer replied: It’s meant to capture that idea… I mean I've got here, there's no like ‘Rube Goldberg-ish device’…so you’re certainly trying to do that, right? I mean to the extent it's possible, you can adjust the way you approach any individual student based on your assessment of where they stand or where their abilities lie… And then, that may in turn influence the way you approach them. But some of it also is driven by them—the effort that they put into it—the skills, the abilities, the characteristics they bring to it… I don't in any way mean to suggest that this is like 100% driven by me. I mean the ultimate takeaway or the ultimate goal of all this is the students learn the skills… So this is trying to capture my role, but…the students play a critical role in this themselves, too. Thus, Meyer viewed himself as a key figure of authority in the classroom and as the primary disseminator of knowledge (in the form of the trumpet/megaphone image in the illustration). Importantly though, Meyer also regarded his other primary role as an L2 instructor to be that of discovering his students’ individual capabilities and needs. Meyer’s own ability to recognize students’ individual capabilities, he felt, was critical—especially since there was no 58 one-size-fits-all approach for him to treat everyone equally with the aid of some ‘Rube Goldberg-ish device.’ Even though he regarded this as an important aspect of being a teacher, Meyer felt that it was not solely his responsibility, as he stated that the students, too, needed to be held accountable for their own learning. Connecting beliefs with pedagogy in the teaching of the office memo. In conjunction with Meyer’s visualization reflection in which he spoke to his own philosophy of teaching, Meyer also provided information regarding the type of environment he was attempting to create in his LLM legal research and writing class. And, because Meyer was both an experienced lawyer and teacher, many of his personal views on teaching were informed by these prior first- hand experiences. In particular, when discussing his philosophy and approach to teaching LLMs legal writing (and specifically, in teaching the office memo genre), Meyer tended to discuss three central themes. These themes consisted of the needs for: (1) building students’ reading skills, (2) raising awareness of the office memo genre, and (3) balancing a real-world context with LLM students’ current needs. In what follows, I describe Meyer’s beliefs related to each of these three aforementioned themes in detail, including how these beliefs were connected to Meyer’s actual course design and classroom practices. Building students’ reading skills. As discussed, one of the primary foci of numerous in- class sessions was related to building LLM students’ reading skills. Meyer saw this as integral and as deeply interconnected to students’ ability to write an effective office memo. One of the rationales he provided for reading being a primary focus was related to students’ backgrounds: Because so many of them are coming from non common-law countries, that [is] one of the things that we're also trying to get across in this class: what's different as a common law lawyer as compared to a civil law lawyer, which most of them have experience with 59 in their home countries. So especially some of the earlier classes in particular, the first two or three of the semester…although there is a writing focus in those classes as well, we spend a lot of time just walking through the common law analysis of a series of cases and how judges and advocate professionals would…interpret these cases. As Meyer referenced, in the first few class sessions—(the overview of which can be seen earlier in Table 4)—students’ backgrounds were primarily in civil law. From his own experiences in working with LLM students, he anticipated that nearly all of the LLMs would struggle in interpreting cases in the U.S. common law system, in which laws typically are not dictated from one central authority, but instead, are based on legal precedent from a variety of prior legal or judicial rulings. Because of this, Meyer structured the first three sessions of the course accordingly, designing the lessons so that students would read only one case per week, gradually building upon each case and subsequently adding layers of complexity to the issues at-hand. For example, prior to attending the first class, for homework, Meyer provided students with background readings on the U.S. common law system, how courts make decisions, and a one-page sheet with tips for taking notes when reading cases (e.g., extracting/writing down integral information such as: the case name, the court that decided the case, the facts, the issue, the court’s holding). Then, for the first class in Week 1, students read a case called Slocum vs. Food Fair Stores (truncated name), which was related to a legal issue involving the intentional infliction of emotional distress (IIED). After discussing various important features of the case at length during Week 1’s class, then during Week 2, students read another IIED case called Korbin vs. Berlin. In this second case, some of the details were similar when compared with Slocum vs. Food Fair Stores (e.g., some of the facts and the jurisdiction/location of the case, which took place in Florida), yet some of the details were quite different (e.g., some of the facts and the 60 court’s holding). Finally during Week 3, students read one more IIED case called Dominguez vs. Equitable Life. Again, in class, Meyer and the students discussed this case in detail, and then, they compared it to the two prior cases they had read. As before, some details of the cases were similar while others differed. Thus, Meyer designed the course in this way in an attempt to facilitate LLM’s reading skills related to understanding the intricacies of U.S. common law, how it develops, and how a variety of factors may subsequently affect the interpretation of law and the courts’ holdings. In addition to designing the course in a manner that he felt would gradually build students’ reading skills, Meyer used a variety of techniques in the classroom for attempting to foster those skills. For instance, when discussing cases individually (e.g., Korbin vs. Berlin in Week 2), Meyer often showed each case on the classroom’s projector using MSWord. While discussing the case in front of the class, Meyer frequently would use his cursor to highlight key elements of the reading (e.g., the court that decided the case, specific facts, the issue, the court’s holding, etc.). As he stated, he often adopted this strategy for attempting to raise students’ awareness of certain key points and furthering class discussions: Sometimes we want to highlight that critical language in the case so that we can then focus the students’ analysis for classroom discussion. Beyond this specific highlighting technique, Meyer also attempted to help students make sense of the complex and multifaceted nature of their readings by providing them with a graphic organizer. During Week 3’s class when Meyer and the students were comparing all three of the IIED cases, Meyer helped guide the LLMs though how to use this graphic organizer for the purposes of understanding similarities and/or differences among the cases (see Figure 3 below). Later in the course as well during Week 5, Meyer provided the students with another blank 61 graphic organizer for assisting students in their readings of cases for the first office memo assignment. At multiple times throughout the course, too, Meyer encouraged the LLM students to create their own graphic organizers, which he felt could assist both students and practicing lawyers in keeping track of and making sense of the cases they had read. Figure 3. Screenshot of graphic organizer for comparing cases As mentioned, despite the title of the course itself being focused on legal research and writing, many of the class sessions were devoted to building LLM students’ reading skills. Though reading was a key component, Meyer viewed reading as strongly connected to students’ success in learning to write the office memo genre. In particular, he also viewed a focus on reading as an opportunity to focus students’ attention on legal writing in general, and, to focus students’ attention on the process of applying case law to (hypothetical) clients’ issues— 62 something that would be integral for the purposes of writing an office memo later on: As we've gone through those first three Florida cases, one of the things we've tried to do is talk about how the cases are written or just how the judges have written them and what works and what doesn't in that regard. And we also provide them with some materials, and we start thinking about: ‘Okay, if we were going to do it, whatever our conclusion may be…how would we organize that?’ You know, ‘how would we structure a big- picture analysis of that way?’ We do some basic walk-through of that. So we're trying to get some of the writing and communicative skills in there too along with the legal method. Meyer further discussed how his focus on building reading skills was directly tied to students’ future ability to write the office memo, and specifically, how it was intended to help build towards the first office memo assignment: We spend a lot of time discussing...I'm trying to pull out the various analytical lessons about how to read it and what's actually going on… And, as is typical…the LLM students usually have a lot of questions, especially those coming from civil law jurisdictions… Then we continue that…we go through more cases…continuing to strengthen our consideration of what the law actually is and then how it applies to the client’s circumstances. And all that has prepped them for what they're gonna do with their first memo. Raising awareness of the office memo genre. Although the primary focus of the in-class sessions appeared to be centered on building LLM students’ reading skills, the second area that Meyer devoted the most attention to was that of raising students’ awareness of different facets related to the office memo genre. In particular, when learning to write the office memo, Meyer 63 personally felt the most integral component was teaching students to understand the genre’s audience—and, by extension, the audience’s purpose for reading the memo: The focus has to be on the audience. I mean, who is it that you're writing to, and what is it they need? Considering in particular that, you know—so many readers of legal writing, they're busy people. So you have to get to the point quickly, not beat around the bush, [and] not belabor on points. But you also have to give them sufficient guidance and information that they can feel confident in the conclusions that you reach. Meyer then further described how the audience and their purpose for reading the memo affected specific aspects of the writing itself. In particular, he mentioned how the audience and purpose affected characteristics of the memo such as language use and organization: [Language in legal writing and the office memo should be] analytical. Precise. Concise. Not flowery. Organized—and it has to be conscious organization. Conclusion first—you know, most important points first. You're not building to the conclusion. You're not wanting to keep the reader on pins and needles until page 10, no. The reader should know on page 1 what your conclusion is, and then back it up. Thus, in teaching the office memo to his LLM students, Meyer felt that the students needed to fully be aware of these considerations. However, in addition to these factors, he also expressed that the genre itself was fairly formulaic in some respects, and, that in order to write it effectively, students needed to learn the memo’s “basic components”: There's several basic components of it. One of the critical things they need to be able to learn is how to give a precise and quick conclusion. You know, what you would see sometimes described as a Brief Answer or Short Answers…just the quick nutshell summary of what your takeaway point is. That first is absolutely critical. 64 The second part is that a typical office memo, at least a more complex one, it's going to have some sort of description of the underlying factual scenario... And so there has to be some grasp of that and how to put it together objectively… An office memo is, to the extent possible, suppose to be purely objective… And then there is the basic normal, maybe even formulaic in some respects, approach to legal analysis once you're actually into the memo [i.e., the Discussion]: ‘This is the law. This is me analyzing the law. I'm applying it to our facts.’ There's a basic straightforward paradigm to follow that most legal readers will expect and that we want to emphasize that students make sure they know how to convey… Most legal audiences want to see it. To summarize, for LLMs learning to write the office memo, Meyer believed the key considerations to be those of: audience, purpose, general organization, and other genre-specific features of language (e.g., using objective and concise language). Therefore, based on these beliefs, Meyer incorporated these elements into his legal research and writing course. Additionally, while some of these elements were addressed during in-class sessions with the students, a large majority of this information was delivered outside of the classroom in the form of supplemental homework readings. Audience. In terms of in-class pedagogy, Meyer overtly discussed the importance of audience in 5 of the 11 total in-class sessions (i.e., during Weeks 2, 3, 5, 9, and 14). He frequently attempted to stress to the LLMs the importance of thinking about their audience when writing. During class lectures as well, he discussed a number of issues related to audience, including: how the audience may affect a memo’s organization, content, headings, and more: Explain why you’re organizing it the way you’re doing it. You’re trying to anticipate what your audience wants…You just gotta tell them what you’re doing. (from Week 2) 65 When you’re preparing an office memo, it can have several audiences…and also, quite a lifespan. Your audience, of course, would be your senior partner, but also your colleagues are going to be reading it, and this memo might be used in your office on this legal issue for years to come…which is why you need to include the date. Whoever is grabbing onto your memo three years from now knows that it was current as of XXX [date]. (from Week 3) We’re trying to anticipate the way a typical reader would think about it. (from Week 5) [Discussing internal and external audiences for a memo]: Internal refers to the firm. It refers to your boss… How well you read directions is going to be tested…whether you’re a trusty researcher. Externally, your next audience is the client…the outside of the firm person that is going to be receiving the advice or information based on the work you’ve done… The client needs to feel that he/she can rely on your advice. (from Week 9) Clearly, audience was an important consideration that Meyer stressed throughout his teaching. In the final class during Week 14 as well, Meyer again emphasized that when it came to writing the office memo genre, he felt that: “Everything boils down to audience.” Structure/organization. Apart from considering the memo’s audience, in multiple class sessions, Meyer also occasionally spent time focusing students’ attention on the general structure/organization of the office memo. During 4 of the 11 in-class sessions (i.e., Weeks 2, 3, 5, and 10), Meyer devoted class time to show LLMs a sample memo outline via the classroom projector. For instance, in Week 10 when the students were in the midst of working on their second office memo assignment, Meyer presented them with a sample memo outline (see Figure 4 below). He then spent time discussing how students might go about organizing the different sections of their memos. For example, when presenting the Facts section, Meyer discussed that 66 students might wish to order the facts in chronological order, or, they may also wish to order the facts based on relevant themes that emerged, depending on which made the most sense in students’ estimations. Figure 4. Screenshot of sample memo outline During an interview, as Meyer further explained to me, he felt that showing students specific examples of the memo’s general organization was beneficial: I think one of the best ways though to demonstrate and how to teach—I think, almost anything frankly—is by providing samples. I mean, we can talk through issues…but I'm a firm believer that when the student sits down to write whatever they're writing, that almost the more samples the better in terms of giving them a range of ways to approach a particular writing project… [That way, students can] talk about questions that they have, and then, they have those samples to build off when they’re doing their own projects. However, as mentioned, although Meyer did spend some time in class discussing how to 67 write the memo—especially involving those factors of the memo’s audience and organization—a majority of the in-class time was reserved for the purposes of reading-based discussions. Outside of class though, there were a large number of readings that focused on how to write an office memo, and these readings provided students with information on a broad range of topics. One of the primary and most comprehensive documents that Meyer provided to the LLMs came early on in the course during Week 2. This document was a lengthy 30-page manual with a large amount of detail on the office memo, including information related to: key features of the office memo (e.g., its purpose and predictive nature); the audience and their purpose for reading the document; the general elements of a memo’s organization (e.g., Facts, Questions Presented, etc.); and other issues related to outlining a memo and organizing concepts within each of the aforementioned sections (see Figure 5 for a screenshot of the manual’s table of contents). This manual was intended for students’ external uses, and it was not utilized or discussed during the class. Language. As referenced earlier, during in-class sessions, Meyer showed students a sample office memo outline on multiple occasions and walked them through potential ways of organizing their respective texts. However, outside of class, he also provided students with a sample full-length office memo, which he personally annotated (see Figure 6 for a screenshot of this annotated memo). These annotations provided LLMs with information that generally highlighted either (a) sentence-level discourse, which showcased sentence-level rhetoric and the function of that rhetoric (e.g., “This sentence serves as a roadmap, keeping the reader focused…”), or (b) paragraph-level discourse, which explained the function of a particular paragraph (e.g., “This thesis paragraph contains the memo’s overall practical conclusion…”). 68 Figure 5. Screenshot of Meyer’s office memo manual Finally, it is worth noting that despite the class being reserved for L2 English learners, considerations surrounding English language use were never explicitly discussed during in-class sessions. Meyer, too, mentioned this during one of our interviews, saying: “I mean, I’m not trained in English as a second language.” Therefore, he expressed that he did not feel comfortable focusing on language instruction. Relatedly as well, Meyer felt that language instruction was not one of the intended foci of the course. Because of this, all of the information regarding English language support and instruction were delivered to LLMs through out-of-class homework readings. These types of language-focused readings tended to be assigned during the later part of the course, with LLMs receiving various language-focused readings in 3 of the 15 total class weeks on topics such as: (1) rhetoric and language-based considerations for writing 69 (e.g., in Week 8, students received a homework reading on how to move from old-to-new information when constructing sentences); (2) editing and revising one’s work (e.g., in Week 10, students received a reading on how to edit their writing to improve flow and to be more concise); and, (3) understanding key language-based features of legal writing (e.g., in Week 10, students received a reading on commonly used transition words, auxiliary verbs with associated prepositions, and on other concepts such as nominalization and parallelism). Figure 6. Screenshot of Meyer’s annotated full-length memo In summary, Meyer considered audience, general organization, and genre-specific features of language to be key considerations for his LLM students in learning to write an effective office memo. Therefore, Meyer integrated all of these elements into his legal research and writing course, yet he presented them to his students in different ways. Figure 7 further illustrates how these themes were recycled throughout the weekly classes during the semester. In particular, while elements related to audience and organization were primarily addressed during in-person class sessions, all of these elements were also addressed outside of the classroom through homework readings—and especially, those features that were related to 70 rhetoric and English language use. Figure 7. Recurring themes of audience, organization, and language in weekly schedule Note: 1First office memo assignment due; 2second office memo due Balancing a real-world context with LLMs’ current needs. The final theme that emerged from the data in relation to Meyer’s pedagogical practices was related to his desire to construct a relatively authentic context in his classroom. Meyer hoped to recreate the context of a U.S. law office to the extent that was possible with his LLMs, and he occasionally attempted to remind students of this context during class, as he described to me: And sometimes when we get into discussions, I…[remind students]: ‘Okay, in an American law office’… I mean, that's sort of the context of this entire class. We're simulating an American law office. [I ask]: ‘If you were working for a partner in a law firm, how would you provide them with what they are expecting to see?’ In attempting to create this context—which again, was tied to the concept of understanding the 71 audience and the audience’s purpose for reading the office memo—one of the things that Meyer stressed was that, because of this American law office context, this constituted a main reason he decided not to focus on English language instruction in his teaching: Trying to write as a lawyer is so broad—I mean, it covers such a range of topics. There aren’t like… ‘If you can master the proper use of these five or ten phrases, you can sound convincingly like a lawyer.’ There's nothing like that… [Additionally,] from the reviewer’s perspective…ultimately what's most important is what the content is. I mean if it reads perfectly, but it's wrong, it's like so what? I mean, ‘your grammar is great, but it's a terrible analysis.’ As Meyer explained, both law instructors and lawyers in American law offices might be willing to overlook some English language or grammatical issues. However, they would expect that a newly hired lawyer understands the main tenets of the office memo genre and legal analysis: I'm not an expert in foreign languages. I mean, but…we know you’re not a native speaker, so we're not looking for fluency or perfect style, perfect English. We don't get that from our native speakers either. But what I want is, as I go through here [in your memo], I don't want there to be any surprises as we go from point-to-point, paragraph-to- paragraph—where each paragraph is taking me. Where is it leading? What are the steps in the analysis? Apart from attempting to foster an authentic law firm environment by focusing on issues such as legal analysis and ignoring minor issues of English language use, Meyer also attempted to invoke this environment by instituting oral presentations into his syllabus design. In particular, during the second half of the semester, students were assigned to give mock oral presentations to a senior attorney (i.e., to Meyer) in small groups before beginning to write their 72 second office memo assignment. Meyer explained why he gave LLMs this assignment, and, how he felt it mimicked an authentic, real-world task: We divide them into teams and then each of the teams gives a presentation to us as a senior attorney—you know, sit down across the desk…and just give me your, you know: ‘What do you think? What’s your take?’… Then, as the senior attorney, I'm trying to fire stuff, questions at them that a real supervising attorney will ask… I don't necessarily want to know all the details of the case. I want to know what this means for the client… We do that in class, and everybody has to attend because part of it too is like, you're learning from your colleagues, right? You know, as someone sitting up in front of me being grilled…because I'll ask some of the same questions to every group. But also… that way, everybody learns from each other and gets those different perspectives on ‘I hadn't even thought about that,’ or ‘I didn’t realize the importance of that.’ In addition to these benefits, Meyer felt that the oral presentation task was effective for positively influencing the quality of students’ subsequent memos, not only from his perspective, but also from what LLMs had told him as well: By forcing them to sit down and essentially think about what their memo is gonna look like in an outline and prepare to discuss it—you know, to orally talk about it—that in turn…leads to a better first draft… And I've actually stolen it now from this [LLM] course and taken it into my JD course where I do it… Some students have actually come back to us later and told us that they like it. I’m not saying everybody does…but I mean, we've heard back from students in previous years that they thought it did actually help them to create the better first draft. Thus, through instituting this mock oral presentation, Meyer felt that the authentic activity had 73 numerous benefits, ranging from providing students with opportunities to learn from one another to producing better quality memos. To the extent possible, it was Meyer’s goal to mimic the context of an American law office, thereby providing his LLM students with authentic experiences for their benefit. However, Meyer was also acutely aware that there were certain sacrifices or trade-offs he had to make in terms of authenticity in order to adequately support his LLM students’ needs. One of the trade-offs that Meyer made was related to editing case readings for the students. Meyer explicitly referenced this in class on multiple occasions that students were reading an “edited version” of a case, so I asked him to further discuss what he typically edited and why he chose to do so. Meyer explained: So a couple things: one is that we often edit the cases, and some of that is simply to exclude material that is extraneous to the cases…or to the issues they're writing about. And I go back and forth on that one because, on the one hand…in the real world, obviously when you read cases, no one gives you edited cases, right? And especially for students whose English isn't as fluent, I mean, it's already difficult sometimes for them to completely grasp what's going on in the cases…just as a result of their English… So we do it actually to make it a little bit easier for them. And actually, also from experience…I didn't use to edit the cases nearly as much, and then students would just get hung up on the wrong things. Apart from editing the cases that the students read, another area in which Meyer was careful in considering his LLM students’ needs was with the selection of the topics for their office memo assignments. For the first office memo assignment, the issue concerned legal malpractice. In the hypothetical scenario, a married couple had previously approached a law 74 firm about filing a suit related to a car accident they were involved in. The lawyer at the firm agreed to take the couple’s case. However, the lawyer then failed to submit vital paperwork and documents to the courts on time as promised, which eventually resulted in the clients’ case being dismissed on summary judgment (i.e., a court decision made without a trial). For the second office memo assignment, the issue concerned a potential contract issue involving third-party beneficiaries. In the scenario, the law firm’s client was leasing a building from the city in order to run a small theatre company. The client was concerned with the question of: if a theatre patron slips and falls in the parking lot during the winter months, would the theatre patron be able to sue the city? As the lease was soon to expire and the client was considering renegotiating the lease, he wanted to know if/how this third-party beneficiary issue might affect the contract negotiations. In terms of the first memo topic involving the issue of legal malpractice, Meyer explained why he selected it: The first memo, which is the Pennsylvania malpractice memo, I chose that topic…one because I just think it's really interesting from an ethical perspective, and that's also one of the ways we can expose students to some of the ethical issues... Because in that scenario, the clients are suing the lawyers for malpractice for screwing up prior representation. He then went on to describe his rationale for selecting the second memo topic: Because a lot of students in the LLM program are coming from a business background and corporate background, I wanted to sort of switch up some of the subject matter of the memos… That’s why the second memo—which is a lease interpretation issue—that's exactly why I added them. Because you've got a contract we're interpreting—it's a 75 business focus. Although different, Meyer felt the two topics complemented each other nicely, and that students would be able to apply the skills they had learned from the first memo to that of the second: And some of it's a question of transferability, right? Transferring what you did on this memo in Pennsylvania dealing with, you know, legal malpractice—now you're in Michigan dealing with contract interpretation and statutory interpretation and third-party beneficiary… It seems completely different or partially different…well it's actually not. So transferring what you've started to learn on that first memo to the second memo. As Meyer expressed, he selected the memo topics not only because he felt they would be accessible and interesting to the LLMs, but also, specifically with memo one, because the topic had added pedagogical value in raising important ethical issues involving the profession. In forgoing the context of an American law office in order to meet the needs of his students, the final area in which Meyer did so was though the implementation of approved case lists for the students’ two office memo assignments. Prior to each office memo assignment being due, Meyer allowed the LLM students to work either individually or in pairs to conduct background research for the office memo assignments. Students were encouraged to conduct their research using a variety of sources such as consulting the Hutz Law School librarians, secondary law sources (e.g., legal encyclopedias, legal periodicals), and primary sources (e.g., statutes, cases) by using online databases. When conducting this research, students were also encouraged to keep detailed records of their research processes by filling out a document called a “Sample Research Plan and Log” (see Figure 8 for a screenshot of the log). In this research log, students would document their research processes, indicating information such as: their research question(s); specific key words and search terms they felt would be useful in the executing 76 searches in online databases; and also, by keeping a log of whom they spoke to (e.g., a librarian, a peer), specific actions they took, and what the results of their inquiries were. However, despite spending such a large amount of time in documenting their research processes for the memo assignments, students were not permitted to use their research lists when writing their actual memos. Instead, the LLM students spent time discussing their research lists/logs with Meyer during individual conferences, where Meyer provided the students with detailed feedback on their research processes. Then, when it finally came time to write their office memos, Meyer gave students an “approved case list,” which consisted of multiple cases (8 cases for memo #1, and 11 cases for memo #2) that students should use in writing their office memos. In a semi-structured interview with Meyer, I inquired as to why he instituted this practice; specifically, I inquired why he felt that students should not use their own research lists when composing their office memos. He responded: I mean…in the real world, no one's gonna come in and say ‘Okay. Now here's the 10 approved cases.’ I mean, you're writing your memo based on what you found, but here's where we're not trying to track the real world precisely. The idea is that once we've done the research and you have the opportunity to practice that skill, and we've had the opportunity to discuss maybe where or how you went off track a little bit in doing the research… Or maybe even just, you know: ‘You selected these cases, and I selected these. Reasonable minds could maybe differ in some respects.’ But then when it comes time to writing a memo, we want to be working from a common baseline. I mean if we're in class trying to talk about the issues in the cases, and we're having to talk about 30 of them…you just can't do that… Simply, again, just to put everybody on that same baseline…it’s for pedagogical reasons. 77 Figure 8. Screenshot of research plan and log Meyer further explained his rationale: I know from experience, frankly, that oftentimes students are not going to get a lot of these cases. And part of it is like, I want them to have that ‘ah ha’ moment of: ‘This is really going to take a lot of time to dig in and find this stuff’… It's like a vivid demonstration that ‘I got something wrong,’ right? You know, it's: ‘Meyer gave me six cases, and I only got three of them. That must mean I'm crappy at research,’ right?... It's 78 like some sort of demonstrable reflection of a skill that they're doing in law school. When otherwise, in law school, especially in their first semester here in American law school, you're not getting a whole lot of direct feedback on how well you're doing. To summarize, in his teaching, Professor Meyer attempted to create the authentic context of an American law office to the extent that was possible. This meant providing his LLMs with minimal feedback on their English language use, and, providing LLMs with an oral presentation opportunity to simulate the real-world experience of researching and presenting an issue to their boss. Conversely, Meyer felt it was necessary to forgo the authentic context in other situations in order to adequately address his students’ needs. He accomplished this by: modifying students’ readings; by carefully selecting topics that were of interest and of dual-pedagogical value; and, by adopting an approved case list so that the LLMs could receive feedback on their research skills, and, so that they could all work from a common baseline as a class. RQ2: The Development of L2 Learners’ Metacognitive Genre Awareness In presenting the findings for RQ2 surrounding the development of LLM students’ metacognitive genre awareness, I remind the reader that in the current study, there were three data collection points throughout the academic semester at: the Pre-course stage (i.e., one-week prior to the course start date), Week 8 (following the submission of the first office memo assignment), and finally in Week 15 (after the submission of the second office memo; please refer to Table 4 earlier for a high-level overview of the course structure). In what follows, I highlight the development of the six case study participants firstly at an individual level, by including detailed examples of how each student’s awareness of the office memo developed respectively. After providing details at this individual level, secondly, I provide a more holistic overview of the six participants as a group, by highlighting those areas in which the students expressed the greatest 79 metacognitive genre awareness and development as a cohort. Individual level: An accounting of each student’s metacognitive development. I now turn to the findings for the development of LLM students’ metacognitive genre awareness at an individual level. Before proceeding, the reader should note that even though students expressed metacognition in numerous areas, I have chosen only to highlight select areas for each respective student, as providing examples and quotes for each of the student’s code categories described below would undoubtedly become a laborious, painstaking task for the reader. The areas for each student were selected to highlight (1) areas in which individual writers experienced growth, but also (2) to showcase as many areas of metacognition (and their respective sub-codes) as possible for the reader. Aashna. As one of the two youngest participants in the study (and in the LLM class at- large), Aashna entered the LLM program immediately following the completion of her law degree in India, and she stated that she had no prior experiences in working with office memos during her schooling. This was reflected in my pre-course discussion with her as well, in which, when prompted to answer questions about the office memo, Aashna was able to provide relatively few and vague declarative comments about the genre (see Table 5 to view Aashna’s development of metacognitive genre awareness). One such comment concerned a tenuous understanding of the memo’s audience, as Aashna stated: Like…[it’s for] the associates, partners, or even administrative staff. However, when further questioned about additional features of the memo such as whether she knew if there was a typical organizational pattern, etc., Aashna was unable to answer: I have no idea [laughs]. Brutally honest—I don’t know. Although Aashna possessed little knowledge of the genre at the beginning of the class, by 80 Week 8 after completing her first memo, her metacognitive genre awareness had expanded in multiple directions. In terms of declarative awareness, Aashna’s knowledge became more solidified when it came to discussing the memo and its audience, purpose, and organization: I think an office memo in an American law context is only, well, mostly for the partners, senior associates, and people in the law firm… A general structure of an office memo is that you start off with the Facts of the case… The next phase is generally the Question Presented and like the Short Answer for it…[etc.] Table 5 Aashna’s development of metacognitive genre awareness of the office memo Comments produced (#) Pre-course 4 Week 8 19 Metacognitive Knowledge • Declarative • Audience • Purpose • Declarative • Audience • Org • Purpose • Procedural • Org • Conditional • Audience affects content • Audience affects lang • Memo section affects lang Week 16 24 • Declarative • Audience • Org • Purpose • Procedural • Integrating cases • Org • Conditional • Audience affects content • Audience affects lang • Memo section affects lang Metacognitive Regulation • Monitoring • Monitoring • Audience and lang • Audience and lang • Evaluating • Formatting • Lang • Org Key: Diff/Sim = Differences/Similarities; Lang = language; Org = organization In addition to declaratively describing the organization, Aashna was able to further explain how (i.e., procedurally) she generally went about doing so in her own memo. Relatedly, too, she could express conditional knowledge such as how the audience might affect different facets of the memo, such as the content: 81 If it’s like the first office memo they just sent to you, then the facts might be longer because you’re introducing everything… And if it’s the second office memo, then as if I were a senior attorney…you may not include them all—maybe just explain ‘Okay. This is where we are.’ While Aashna showed expanding awareness in all three components of metacognitive knowledge, during the stimulated recall with her office memo, she was unable to demonstrate any planning prior to writing or evaluating in relation to her writing performance. For instance, when asked to evaluate her own performance on the first memo, Aashna simply replied: [Referring to Professor Meyer]: I’m hoping that I’ve met his expectations? [laughs] Despite showing no clear evidence of planning or evaluating, Aashna was able to show a limited ability to monitor while writing, and she was able to recall specific instances during her writing process. When recalling such instances, she described that while writing, she thought about her audience (i.e., a senior attorney) and the types of language that this attorney would be able to comprehend: I was writing the word ‘contracts claim, contracts claim’ way too many times in my office memo, and I just wanted a substitute because it was sounding very repetitive. And then the word ‘assumpsit’ popped up—so that's basically a contracts claim. And I don't need to explain it in an office memo that’s going to a senior attorney because they will know what it means. As Aashna showed obvious progress between the pre-course and Week 8 stages, she developed even further metacognitive genre awareness by the conclusion of the legal research and writing course in Week 16. Notably, Aashna further developed awareness in the regulatory category of evaluating, as she could now critically reflect on various facets of her own writing 82 performance. Although she was unable to provide reflective comments in Week 8, she was now able to do so in Week 16. Specifically, Aashna was able to critically reflect on different elements of writing her second memo, and she was able to evaluate it in reference to her first memo assignment in areas that included formatting, language use, and organization: Just the formatting of how the memo should look is better in the second one… [And also]…like the Short Answer and the Question I think is better in the second one than the first one. I think it's more concise in the second one. And the conclusion is more concise in the second one… I think I've become slightly more efficient, even though I’m not super confident… I think, compared to the first memo I wrote, in the second one my structure was much better, like the approach of taking—citing the high-authority first, and then going in a chronological order. By the course’s end, Aashna exhibited the development of metacognitive genre awareness in relation to all facets of the metacognition model with the exception of the regulatory aspect of planning, which she never discussed during our three sessions together. Jing. Following the completion of her legal education program, Jing briefly worked as a legal specialist for two years at an international law firm in southern China. Jing stated that she had occasionally come into contact with office memos at her firm—(she estimated approximately once-per-month)—but she was unsure of whether those memos were similar to those produced in the U.S. context. Like Aashna, at the pre-course stage, Jing could provide very few comments about the office memo (see Table 6 for an overview of Jing’s development), though she was able to articulate facets of declarative awareness related to the purpose and the audience’s expectations: If there is a request from a client, you issue a memo to help solve their questions to 83 clarify on certain issues for the client… Because they [i.e., attorneys] are also very busy, so you need to write it in a simple way and to address the issue very quickly and directly. Table 6 Jing’s development of metacognitive genre awareness of the office memo Comments produced (#) Pre-course 4 Week 8 30 Week 16 26 Metacognitive Knowledge • Declarative • Audience • Audience’s expectations • Purpose • Declarative • Audience • Audience’s expectations • Lang • Purpose • Procedural • Integrating cases • Org • Conditional • Audience affects content • Audience affects lang • Memo section affects lang • Topic affects org • Declarative • Audience • Audience’s expectations • Lang • Purpose • Procedural • Integrating cases • Conditional • Audience affects content • Memo section affects lang • Topic affects org Metacognitive Regulation • Planning • Planning • Graphic organizer • Oral presentation • Monitoring • Monitoring • Audience and lang • Lang • Evaluating • Content • Formatting • Planning • Audience and content • Evaluating • Content • Planning Key: Diff/Sim = Differences/Similarities; Lang = language; Org = organization Halfway through the semester by Week 8, Jing was able to express metacognitive genre awareness in relation to all six components of the metacognition model. In addition to showing declarative awareness (e.g., audience, purpose, etc.) and procedural awareness of how she might typically go about organizing a memo, Jing was now able to express a wide-ranging awareness of conditional aspects of the genre as well, such as how different audiences might affect 84 language use. At one point in particular, Jing mentioned that either an attorney or a client could potentially read an office memo, yet she stated that she wrote her memo assignment with an attorney in mind. I further inquired about this, asking if she were to send her memo to a client instead, if/how it might affect her language use. Jing responded: If I'm sending this memo to a client, I think we would be more careful about using those ‘highly possible’ or strong ‘it is highly likely’ [phrases] or something like that…because the client maybe rely on that. We may can tell the client: ‘Our view is to stick to the contracts or tort claim,’ but if we write in the paper, it may be used as evidence for future suit. Beyond indicating conditional awareness of how the audience might affect her language use, Jing expressed conditional awareness of the idea that different parts of the office memo itself (i.e., different sections) also could potentially affect a writer’s language use: So for in Discussion part, you first need to analyze the legal issue… It needs more like formal and legal language. And, but in the second part of Discussion when you analyze the case facts, it's less formal, I think, to like describe those facts. And I think those are the different parts. Apart from exhibiting a full range of awareness related to metacognitive knowledge, Jing showed an ability to discuss how she regulated that awareness when planning, monitoring, and evaluating. In one particular example, unlike Aashna who did not plan prior to beginning writing, Jing described her process of planning, as she adopted the use of Professor Meyer’s graphic organizer before writing her first memo: Actually, I didn’t spend much time in writing. But for reading the approved materials for the case list and the facts and the depositions, I think that took me like three or four hours 85 to do that. And I need to analyze…and I need to fill in the case chart so I can move on to the drafting. So preparation is the most part took me time. And once I started, it didn’t took me that long. During the actual writing process, too, Jing showed evidence of monitoring, specifically as she was able to recall aspects of how her audience affected her language use. In one example, while writing, Jing recalled that Professor Meyer had told the class they needed to remember to add their predictions about the potential outcome of the case. In considering this direction, Jing pointed to a passage in her first memo, showing me that she had added a specific sentence as a result of thinking about Meyer’s comment while she was writing: Professor said…‘your partner need more your position on this issue’… [It’s] in the second paragraph of page 5. It's: ‘In light of the above, it is highly likely that the current court will adopt a gist of action doctrine.’ So, um…actually in this part…this was kind of a summary part of my drafts, and so I added it… So I said ‘it is highly likely that the current court will adopt.’ By the end of the course at Week 16, Jing again was able to express metacognitive genre awareness related to all six facets of the model. Interestingly, she made fewer comments and addressed a fewer number of areas during the final interview and stimulated recall. Based on the codes that arose from this final session, it did not appear that Jing developed any new awareness beyond what was already mentioned during our Week 8 session, with only one exception arising within the category of planning. Unlike the first office memo in which Jing prepared for her writing by using Meyer’s graphic organizer, in writing the second office memo, Jing used the in- class oral presentation as her preparation. She remarked: [When] professor acted as the senior partner and we talk him—presented to him our 86 structure and our basic logic of analyze discussion…he ask us some questions. And I think that meeting is very helpful in preparing for the memo because, first, that meeting help to make us prepare earlier than what we usually will do. And second, in the meeting with him he ask us some questions like he is really the partner, and he needs to make sure the analysis is complete. So I think that meeting…made me prepare the whole structure more efficiently. As mentioned though, apart from this particular facet of planning, no additional areas of metacognitive genre awareness appeared to be novel during our final Week 16 session. Luisa. At 29 years old, Luisa was one of the oldest students in the legal research and writing course. After completing her legal education in Mexico, she had worked as a legal research associate for two years, where she gained some experience with reading and drafting office memos. This was made clear in our pre-course interview as well, as Luisa was able to provide declarative information about the genre’s audience and purpose, and, she was able to exhibit procedural and conditional knowledge related to the genre (see Table 7 for an overview of Luisa’s development). For instance, Luisa provided conditional information about how a specific client might affect different aspects of a memo’s content: It will depend a lot on the clients. I was working with some… They were like: ‘Okay. As long as I can read it fast, it doesn't matter.’ But we have some clients that are super picky in the way they want to have presented their things. So they will tell you, for example: ‘Okay. Can you please write in all the memos our whole name?’… Like their internal rules, that was like a thing. In the pre-course interview as well, Luisa was able to provide some general information about how (i.e., procedurally) she would go about writing a memo, filling in the different sections in a 87 first draft as quickly as possible: So first I would think: ‘Okay. What does it have to have inside?’ Then I will just draft it like very quickly, like the points, or I will make like a skeleton argument… Then like, I will go through it again. I will see if it has logic, if a step is missing… Table 7 Luisa’s development of metacognitive genre awareness of the office memo Comments produced (#) Pre-course 7 Week 8 24 Week 16 25 Metacognitive Knowledge • Declarative • Audience • Purpose • Procedural • Integrating cases • Org • Conditional • Audience affects content • Audience affects lang • Declarative • Audience • Audience’s expectations • Diff/sim between countries • Lang • Org • Purpose • Procedural • Integrating cases • Org • Conditional • Audience affects content • Audience affects lang • Topic affects org • Memo section affects lang Metacognitive Regulation • Planning • Graphic organizer • Outlining • Monitoring • Audience and lang • Audience and org • Lang • Evaluating • Content • Lang • Planning • Declarative • Audience • Audience’s expectations • Diff/sim between countries • Lang • Org • Procedural • Integrating cases • Org • Conditional • Audience affects content • Audience affects lang • Audience affects org • Topic affects org • Planning • Outlining • Monitoring • Audience and content • Audience and lang • Lang • Evaluating • Planning Key: Diff/Sim = Differences/Similarities; Lang = language; Org = organization Luisa was able to provide a great deal of information about the office memo at the pre- course stage of the study, but her knowledge continued to evolve over the span of the first eight weeks of the semester. By Week 8 in particular, Luisa had come to the realization that the 88 memos she was exposed to during her previous work experience were different from those memos produced in the U.S. context. Therefore, she was able to expound on some of these differences across cultural contexts: A lot of things for me were pretty obvious because I was already working…but some of the things, like, I didn't know exactly at least how it is in the U.S. to structure an office memo. I think…if I remember correctly, I told you… Oh yeah! I have done office memo before, but in my work experience, the office memos that I prepare normally were going to the client… Like this structure is, I would say, is completely different [in the U.S.]. Because, at least the memos that I used to know, depending on the question is how you will like build the memo. Here for everything, it’s the same. Apart from Luisa’s understanding of the memo changing in relation to how features differed depending on the context, she was able to clearly express how she regulated that knowledge when planning, monitoring, and evaluating her own writing in Week 8. For instance, when planning for the first memo task, Luisa stated she used the graphic organizer provided by Meyer. In addition to using the organizer though, she had another planning strategy that involved physically moving around and categorizing the cases to place them into an outline: So what I did…? So I have printouts of all the cases. As I was reading them, I identified portions that I found it could be helpful for me, so I underlined them. And then, I categorize the cases. So I put all the cases that will be like in favor of what I want to say, cases there will be against what I want to say… And then I kind of put like a header cue with them. Beyond planning, while writing, Luisa expressed that she consistently thought about multiple audiences: the first audience was related to whether or not she was addressing Meyer and the 89 organizational scheme he had taught them in class, but, the second audience was related to whether her own language was appropriate for a broad, non-legal readership: I was thinking always if Professor Meyer would like it. I don't know if that would be like the approach when you are actually crafting an office memo… [But] when I was drafting this, like…[I was thinking]: ‘Am I being clear enough?… I'm having the sections that they told me about.’ Like I was really worried about the structure... And I always think, like: ‘If I will give this to my boyfriend’—he's an engineer—‘will he understand what I'm doing?’ Because if I can explain it to a mortal, I can explain it to everyone. By the end of the course in Week 16, Luisa was able to produce only one more comment about the office memo than she had in Week 8. Like Jing, Luisa did not discuss as many areas as she had during Week 8. Despite this, her knowledge continued to evolve. In our final session together, she spoke about this in detail, and she was able to provide me with general declarative information about her understanding of the memo’s audience: Like I said, I didn't know before, like… I told you I thought that the office memo was the thing that I was supposed to give to the client. Relatedly, Luisa expressed that she understood that the audience might conditionally affect the organization of a memo, yet she still did not fully comprehend why memos were structured this way in the U.S.: For me, it’s an additional step anyway. Because in the end, ultimately, this office memo will be given to the client in a different structure. Anyway, it doesn’t seem so necessarily. Like, why do we need to have it? Maybe in the U.S., it’s a culture. Thus, although it appeared that Luisa’s knowledge of the office memo expanded into numerous 90 domains over the span of the semester, by the end of the course, she still had lingering questions about why certain practices existed in writing the genre in the U.S. context. Paul. Paul completed his legal education in Germany, and post-graduation, he served as a legal research assistant for three years under a law school professor. During this period, Paul stated that he was never exposed to office memos. Therefore, prior to beginning the LLM program, he possessed minimal knowledge of the genre (see Table 8 for an overview of Paul’s development). Yet, Paul did display declarative knowledge related to components of the memo’s audience, purpose, and the general type of language that a writer might use: It has something to do with like a suggestion for your employer about how a legal dispute should be solved… I assume it's very much similar to Germany. The language used for legal briefs or your documents is much more formalistic and has certain terms that are not used in everyday language. Though Paul possessed minimal awareness of the office memo at the outset, by Week 8, he was able to produce more comments than any of the other case study participants, and, he made remarks that corresponded to every section of the metacognition model. Like Luisa, Paul was able to compare the memo to other genres and legal writing he was more familiar with in Germany, and he made a general declarative statement expressing such comparisons: Well, I think I've learned a lot, taking into account that I didn't really know what an office memo was in the beginning. But I think…I figured out that it is similar to something you would write to your senior attorney in German to figure out how much—how likely it is for a plan to succeed in court. During the class, because Paul was able to discover some similarities between the U.S. office memo and those types of writing he was familiar with in Germany, he was able to explain how 91 (i.e., procedurally) one should use language in writing a memo, and, how he followed his own intuition in writing the memo just as he would in his native country: Table 8 Paul’s development of metacognitive genre awareness of the office memo Comments produced (#) Pre-course 3 Week 8 31 Metacognitive Knowledge • Declarative • Audience • Lang • Purpose Metacognitive Regulation • Declarative • Audience • Audience’s expectations • Diff/sim between countries • Lang • Org • Purpose • Procedural • Org • Lang • Performing task the same as in home country • Rhetoric • Conditional • Audience affects content • Audience affects lang • Topic affects org • Planning • Graphic organizer • Outlining • Monitoring • Purpose • Evaluating • Audience’s expectations • Content • Org • Planning Week 16 32 • Declarative • Audience • Audience’s expectations • Diff/sim between countries • Lang • Org • Purpose • Procedural • Integrating cases • Conditional • Audience affects content • Audience affects lang • Planning • Oral presentation • Monitoring • Audience • Audience and content • Audience and lang • Evaluating • Audience’s expectations • Content • Lang • Org • Planning Key: Diff/Sim = Differences/Similarities; Lang = language; Org = organization I would never write like ‘in my opinion’ or ‘I think that.’ I don't know if that applies here as well. But I would refrain from it and be like most neutral in a way that just… It's not 92 me, a person, who is stating an opinion, but rather like a document telling you about the facts and the rules of it… I just did it how I would do it in Germany, I think. Like Luisa, Paul used the graphic organizer in Week 8 that Professor Meyer provided the students, and Paul also went a step further in engaging in outlining for planning for his memo. While composing the memo, too, Paul illustrated that he was engaged in monitoring. He recalled that while writing his first memo’s Discussion section, he was actively thinking about the purpose of the memo genre and how he needed to do a better job in illustrating the rule of law and applying it to the facts of the current case. He pointed to a specific passage in his first memo to explain to me how this consideration actively caused him to insert new content: So I came up with the rule and said [to myself]: ‘You need a specific structure’… And usually, I would just argue everything that I can to support that ‘this is a specific instruction because that is our client.’ But I tried I think—to look down at some points… I say [in my memo on p. 7] that: ‘Even though it might be specific instruction, it is unimportant in this case because it is not the course for the damages that they’…[etc.]. Paul not only showed the ability to monitor, but he was also quite reflective in evaluating his own writing. He reflected on both the objectivity of his content/legal analysis and his writing in relation to the audience’s expectations: I think that the most troubles I had… Yeah. I think my mindset is more like finding the best arguments for the clients without being…like…obviously bending the law…. [laughs]. I pretty much interpreted everything most favorably from the clients’ perspective, if I think about it. Maybe that's something Professor Meyer will tell me about—that I’m not impartial enough in applying these moves. 93 By the end of the legal research and writing course, Paul again produced the greatest number of comments of all the six LLM participants. Similar to Jing and Luisa, though his comments covered all six areas of the metacognition model, he discussed fewer areas in Week 16 than in the Week 8 session. Despite this, Paul displayed a broad range of metacognitive knowledge in relation to the office memo. Moreover, when it came to using that knowledge in his actual writing, he engaged in planning, monitoring, and evaluating. Paul mentioned that he used Meyer’s oral presentation task in order to plan for his second office memo. Also, by the end of the course, he was able to cogently discuss how he engaged in using his genre knowledge to monitor his writing. For instance, while composing, Paul’s audience was a senior attorney, and he thought of the knowledge that a senior attorney would possess. This caused Paul to remove some content in his memo since he felt his audience was experienced and would already be familiar with some of the details he was discussing: I discussed a large concept of contracts that is very familiar to any attorney, so I shortened that paragraph quite significantly… That would be in on page 3 and… oh no, it would be the fourth side. With the first paragraph starting with ‘However, in the case the contract is ambiguous’… This is something that applies to all contracts. Beyond being able to discuss monitoring his own writing, Paul was able to critically evaluate his performance. He evaluated his writing in numerous ways, but one such way was that he compared his second memo to the first memo assignment, and specifically, as to how he was more efficient in researching and planning for his second office memo: I think it was easier for me this second time around to apply the cases to our facts… I did that last in the last part in the first memo and didn’t spend enough time on really going back to the cases and looking into the facts and really try to compare them to our facts in 94 the case. At the outset of the course, Paul appeared to posses little knowledge of the office memo. However, by the end of Week 16, he appeared to be one of the strongest of the six LLM students, as he was highly capable of vocalizing his metacognitive genre awareness. Yuki. The oldest of the six case study participants, Yuki was also by far the most experienced professionally. Prior to joining the LLM program, Yuki had worked as an attorney for nearly five years at a law firm in Japan, where he regularly prepared loan agreements and purchase agreements for his company. Despite his extensive experience, in our pre-course discussion, Yuki stated that he was rather uncertain of what an office memo was and/or how to write one (see Table 9 for an overview of Yuki’s development). However, he did express some general declarative knowledge of the audience and the purpose of the genre: I'm not sure what the office memo means in Japan…but I guess the office memo is something like…it is not for the client, but only for colleague or bosses. I think the purpose of office memo, and this is my understanding, is it is an internal memo to discuss internally and make evidence for the internal decision process. Though he could verbalize little at the start of the study, after completing his first office memo in Week 8, Yuki expressed metacognition in relation to all six components of the model. Within metacognitive knowledge, his declarative knowledge expanded to include awareness of the audience’s expectations. Relatedly, because he had prior work experiences, he was able to compare the memo to the types of legal writing he had previously done during his job. Yuki was able to express declarative comments about differences and similarities related to the frequency with which associates would produce an office memo in the U.S. context versus in Japan: [Unlike in Japan], here maybe, if we deal with the cases, we have to make office memo 95 everytimes…so the organization and accuracy of the memo is expected higher than in Japan. Table 9 Yuki’s development of metacognitive genre awareness of the office memo Comments produced (#) Pre-course 4 Week 8 22 Metacognitive Knowledge • Declarative • Audience • Language • Purpose Metacognitive Regulation • Declarative • Audience • Audience’s expectations • Diff/sim between countries • Lang • Purpose • Procedural • Integrating cases • Conditional • Audience affects content • Audience affects lang • Audience affects org • Topic affects org • Planning • Graphic organizer • Monitoring • Audience • Audience and lang • Evaluating • Content • Org • Planning Week 16 23 • Declarative • Audience • Audience’s expectations • Lang • Org • Purpose • Procedural • Integrating cases • Org • Conditional • Audience affects content • Audience affects lang • Planning • Oral presentation • Monitoring • Audience • Audience and content • Lang • Evaluating • Content • Planning Key: Diff/Sim = Differences/Similarities; Lang = language; Org = organization In addition to explaining cross-cultural differences, Yuki was able to describe many features related to his conditional awareness. One such example consisted of explaining how a change in the memo’s audience from an attorney to a client might affect both the language and the organization: Because the clients are not professional lawyer, so we have to use not so formal, not so difficult terms. And, we have to organize more clearly, not completely advanced. If it is 96 not for the partner, we need simple sentence… After producing his first office memo, Yuki also showed that he could use his awareness in regulating his writing. Like some of the other students, Yuki adopted the graphic organizer for planning. Additionally, he was able to discuss how he actively used his genre awareness when monitoring. Unlike some of the other students who thought about a senior partner when writing their memo, Yuki thought mostly about the client while wring his, remarking: I think when I write and was writing the memo, I thought about the client. But this is very natural for me because when I was in Japan and I worked as a lawyer, I usually… when I make some products…it is it is supposed to be for the client, so I have to always have to think about the client and how they feel. When I further inquired as to whether this monitoring of the client audience affected specific language in his memo, Yuki pointed to a specific passage, indicating that he changed it as a result of consciously considering his audience: When I mention the cases…page 3. It is not so short… the heading [that starts with] ‘the daily approach’… When I last wrote this, I wrote this paragraph by one sentence. But…I shorten it [for the client]. By the end of the course in Week 16, Yuki again made comments that expressed his metacognitive genre awareness in relation to all six components of the model. One of the areas in particular in which his knowledge evolved was in his understanding of the audience. As shown, originally in his first memo, Yuki stated that he wrote the memo particularly with the client in mind. However, in his second memo, Yuki discussed how he tried to balance both audiences—and, he addressed a declarative understanding of the legal audience’s expectations: I think…the reader who have some professional skills expect me to mention the 97 important fact for analyzing this case and the general statute and cases which is advantage to the clients, and also, the cases which are disadvantaged to us. I think this elements is expected to be mentioned at least by the readers who has some professional knowledge of the law. Apart from discussing his newfound awareness of the audience’s expectations for an office memo, Yuki went on later in the stimulated recall to discuss how he actively used this awareness in monitoring certain aspects of his writing such as the memo’s heading: At the beginning in my memo, I have to write ‘to whom’… So in this, I write ‘Professor Meyer.’ But it is usually written partner's name, so in the beginning, I realize that this is for the partner or my boss. This memo will be important to my boss or my partners’ time, so…I think about that. And then during writing the memo, I usually think about. Relatedly, Yuki was further able to discuss how he tried to balance the two audiences of the lawyer and the client. In one such example, he addressed how when monitoring and thinking about the client’s needs, this caused him to add specific content to his second memo: But lastly, I write about the suggestion or the current issue…last page [on p. 7 of the memo]. So it is also for the clients, but it is… My solution is what kind of language will be added to the current contract agreement. So this matter is a very professional or lawyer’s task. Thus, even though Yuki was the most experienced professionally of the six case study participants, throughout the 16-week period of data collection, his understanding of the office memo genre continued to evolve in multiple respects. Yurui. Yurui was the final case study participant. Like Aashna, Yurui was 23 years old and one of the youngest participants in the study. Yurui had completed his legal education in 98 China and joined the LLM program at Hutz Law School immediately afterwards. While pursuing his legal education though, Yurui had the opportunity to intern briefly for a local law firm, where he had the chance to write a couple office memos for his employer. Therefore, at the outset of the data collection, Yurui expressed the most metacognitive genre awareness of all six of the LLM participants (see Table 10 for an overview of Yurui’s development). From the beginning, he was able to showcase declarative, procedural, and conditional awareness of the genre in multiple categories. For instance, Yurui could provide detailed information of how (i.e., procedurally) he would go about conducting research and integrating cases into an office memo: Let's begin with the source of law. I think like, you have to find the relevant sources of law, right? And they may be statutory laws, case laws, administrative regulations…and I think some region also looks look at the constitution of law… And if there's no such directly relevant case law or statutory law, maybe I would also look for some scholarly publication also… But, I think many clients do not like scholarly published articles because it do not have direct legal effects. Yeah, so maybe if there is also another directly applied the statutory law or case law, I will put it in first and adjust the scholarly publications at the bottom of the reasoning part. Also in the pre-course stage, Yurui was able to express some declarative knowledge of the differences and similarities of office memos produced in China versus in the U.S. context: In Chinese style memos, I think we tend to put the answer at the end of the memo—not give a short answer before. I don't know why, but I think U.S. memos should be more straightforward for clients. And…and in U.S…memos, I think, is much more organized, I would say. Or, it requires more logical analysis because you have to sometimes to divide a question into different parts. 99 Week 16 28 • Declarative • Audience • Audience’s expectations • Purpose • Procedural • Integrating cases • Conditional • Audience affects content • Audience affects lang • Audience affects org • Memo section affects lang • Topic affects content • Topic affects org Table 10 Yurui’s development of metacognitive genre awareness of the office memo Comments produced (#) Pre-course 10 Week 8 22 Metacognitive Knowledge • Declarative • Audience • Audience’s expectations • Lang • Purpose • Procedural • Integrating cases • Org • Conditional • Audience affects content • Audience affects lang • Audience affects org • Memo section affects lang • Declarative • Audience • Diff/sim between countries • Lang • Org • Purpose • Procedural • Integrating cases • Conditional • Audience affects content • Audience affects lang • Audience affects org Metacognitive Regulation • Planning • Planning • Graphic organizer • Graphic organizer • Monitoring • Lang • Evaluating • Content • Org • Planning • Monitoring • Audience • Evaluating • Audience’s expectations • Content • Formatting • Lang • Planning Key: Diff/Sim = Differences/Similarities; Lang = language; Org = organization Clearly, Yurui already possessed a wealth of metacognitive knowledge at the study’s outset. By Week 8, after writing his first office memo, he expressed many of the same ideas that he had discussed during the pre-course interview. However, one novel area not expressed during his pre-course interview concerned his conditional knowledge, and in particular, how he felt that the section of the office memo potentially affected the language that a writer used: 100 For the Fact part, it's definitely different from the Discussion part. The Discussion may have more legal English, like of phrases, but the Fact part… I will say it's not really like the article we read in news or website. I think like… the Facts part is more like the charges. In addition to this new development, during the stimulated recall, Yurui showed evidence of planning, monitoring, and evaluating. For instance, like some of his peers, he adopted the use of Meyer’s graphic organizer for the purposes of planning to write his memo: Mmm… It is helpful when I read the case, and like yeah, I just put the relevant information in the chart. When I start to write the memo… sometimes I always look back to the chart. While writing his first office memo as well, Yurui commented that he was actively monitoring and thinking about a specific homework reading that Meyer had provided to the class earlier. This reading was related to subtleties in meaning of English language use involving reporting verbs: And when writing, [I remembered] like the grammar stuff… The court did not ‘assert’ or ‘argue,’ but court only ‘states, notes,’ or ‘maintains’… I use that word once [i.e., argues], and…it is wrong because court did not argue anything…because a court is supposed to be neutral. When I write, I didn't know the world ‘argue’ has such a specific implication, so I ignore that. But…it makes sense, so [I] corrected it. By the end of the semester in Week 16, Yurui continued to show signs of development, especially in the areas of conditional knowledge and in evaluating his own writing performance. In terms of the former, after completing the second memo assignment, Yurui was able to provide information on how the specific nature of the legal topic being addressed might conditionally 101 affect the organization—or, that is, what sections the writer might choose to include in his/her memo: In the second memo, we are asked to make some suggestions for the future action. But for litigation cases [in the first memo], it is not necessary or possible for that. And because this…some of us may include the Executive Summary [section]. In terms of regulating his performance, as mentioned, Yurui also exhibited reflective skills in terms of evaluating his own writing. For instance, by Week 16, he was able to further reflect on his memo writing, expressing evaluations related to areas such as his research and planning processes and even his general formatting of the memo genre: Firstly from the research part, I learned how to scrutinize whether my case list is complete or not. It is the most important knowledge I didn't know before. And to include to find the best authority for my intended office memo is also very important. Of course [I made] some improvement… I would say I made fewer—there were like fewer technical mistakes in my second memo. Like, I just forgot insert page number or to change the dates, and some technical problems like [that in my first memo]. Despite entering the course with a wealth of background knowledge, by the end of the semester, it appeared that Yurui had developed new metacognitive genre awareness in multiple areas. Group-level comparison: Overview of the six focal participants’ development. Now that all six case study participants have been discussed at an individual-level, I shift the reader’s attention to the group as a whole. In particular, I wish to highlight two primary areas, both of which are related to the previous section. These areas include: (1) the 15 most frequently mentioned areas/codes of metacognition that recurred throughout the entire dataset (see Table 11); and, (2) an overview of the six categories of the metacognition model, and how the LLM 102 cohort progressed in relation to these categories throughout the academic semester (see Table 12). Specifically, Table 11 and Table 12 highlight the total number of comments that students were able to make within each category when discussing the office memo during our semi- structured interviews and stimulated recalls. Table 11 Top 15 most frequently recurring metacognition comment codes Rank Metacognition category Subdomain Code category Integrating cases Purpose 1 Metacognitive Knowledge Declarative Audience 2 Metacognitive Knowledge Procedural 3 Metacognitive Knowledge Declarative 4 Metacognitive Knowledge Conditional Audience affects content 5 Metacognitive Knowledge Declarative Audience’s expectations 6 Metacognitive Knowledge Conditional Audience affects language Planning 7 Metacognitive Regulation 8 Metacognitive Knowledge Declarative Language 9 Metacognitive Regulation 10 Metacognitive Knowledge 11 Metacognitive Knowledge Conditional Memo section affects language 12 Metacognitive Knowledge Conditional Audience affects language 13 Metacognitive Knowledge Conditional Topic affects organization 14 Metacognitive Knowledge Declarative Diff/sim between countries 15 Metacognitive Regulation Evaluating Procedural Organization Planning Graphic organizer Evaluating Content Total Comments 30 29 26 23 23 22 21 19 18 13 11 11 10 9 8 As Table 11 shows, the most frequently recurring comments made by the six case study participants pertained to students discussing their metacognitive knowledge. Comments surrounding metacognitive knowledge accounted for 12 of the top 15 categories that were mentioned. Within this category of metacognitive knowledge, comments related to students’ declarative and conditional knowledge constituted the most-mentioned subdomains (5 for the declarative subdomain, and 5 for the conditional subdomain), as LLM students frequently discussed both what they knew about the office memo genre and when/why certain factors related to the genre might subsequently affect their writing. Moving aside from these subdomains, in examining the individual code categories, 5 of the top 15 codes were related to 103 some aspect of the office memo’s audience (e.g., declarative knowledge of the audience; conditional knowledge of how the audience affects language). Building on these most frequently recurring codes, Table 12 goes on to highlight a high- level overview of the six categories of the metacognition model, and, how students’ comments were distributed across these six categories during the full academic semester. As Table 12 suggests, at the pre-course stage of data collection, a majority of the LLM students exhibited relatively little awareness of the office memo genre. In particular, Aashna, Jing, Paul, and Yuki were only able to make relatively few and vague declarative comments about the nature of the genre (described in detail earlier). These four participants indicated no knowledge of how (i.e., procedurally) one would write an office memo and when/why (i.e., conditionally) certain factors such as audience, topic, etc. might affect one’s memo writing. Apart from these four participants, only two individuals—Luisa and Yurui—were able to exhibit any procedural or conditional awareness related to writing the office memo genre at the pre-course stage. By midway through the semester in Week 8, all six students had developed metacognitive genre awareness related to metacognitive knowledge, and they were able to produce remarks that reflected all three corresponding facets of declarative, procedural, and conditional knowledge of the memo. Week 8 also represented the first time in which stimulated recalls were adopted to extract information related to students’ metacognitive regulation. During those recalls, students discussed how they used their awareness when writing the office memo in terms of planning, monitoring, and evaluating their performance. As the data show, with the sole exception of Aashna (who was unable to express the aspects of planning and evaluating), all five of the other LLMs exhibited the capabilities of discussing these three aforementioned regulatory functions. 104 Table 12 Overview of the LLMs’ development of metacognitive genre awareness of the office memo Participant Week 8 Metacognition exhibited Pre-course Total Total comments (# of comments) comments Week 16 Metacognition exhibited (# of comments) Total comments Declarative (10) Procedural (3) Conditional (5) Declarative (7) Procedural (2) Conditional (8) Declarative (8) Procedural (8) Conditional (4) Declarative (7) Procedural (5) Conditional (4) Declarative (7) Procedural (5) Conditional (4) Declarative (5) Procedural (4) Conditional (10) Group total Subtotals Declarative Procedural Conditional ---------- Monitoring (1) Evaluating (5) Planning (1) Monitoring (2) Evaluating (6) Planning (1) Monitoring (3) Evaluating (1) Planning (1) Monitoring (3) Evaluating (12) Planning (1) Monitoring (4) Evaluating (2) Planning (1) Monitoring (1) Evaluating (7) 44 Planning 27 Monitoring 35 Evaluating 24 26 25 32 23 28 158 5 14 33 Metacognition exhibited (#)1 Aashna Jing Luisa Paul Yuki Yurui Declarative ---------- ---------- Declarative ---------- ---------- Declarative (2) Procedural (3) Conditional (2) Declarative ---------- ---------- Declarative ---------- ---------- Declarative (6) Procedural (1) Conditional (3) Group total Subtotals Declarative Procedural Conditional 23 4 5 4 4 7 3 4 10 32 Declarative (9) Procedural (3) Conditional (5) Declarative (9) Procedural (3) Conditional (8) Declarative (8) Procedural (3) Conditional (5) Declarative (9) Procedural (7) Conditional (4) Declarative (7) Procedural (1) Conditional (5) Declarative (6) Procedural (3) Conditional (5) ---------- Monitoring (2) ---------- Planning (2) Monitoring (3) Evaluating (5) Planning (2) Monitoring (3) Evaluating (3) Planning (3) Monitoring (1) Evaluating (7) Planning (1) Monitoring (2) Evaluating (6) Planning (1) Monitoring (1) Evaluating (6) Group total Subtotals Declarative Procedural Conditional 48 Planning 20 Monitoring 32 Evaluating 19 30 24 31 22 22 148 9 12 26 1Note: Metacognitive regulation was not assessed during the pre-course stage since there was no memo task for students to perform. 105 Lastly, during the final data collection session in Week 16, all six of the case study participants again were able to produce remarks reflecting that they possessed awareness at the level of metacognitive knowledge related to declarative, procedural, and conditional knowledge of the office memo. Likewise, in terms of metacognitive regulation, Aashna was again the sole exception, not showing any evidence of planning prior to writing. All five of the other LLMs were able to express the capacity for discussing the three regulatory functions of planning, monitoring, and evaluating their writing performance. In examining the progress of the LLMs as a group, as time progressed during the semester, the LLMs’ metacognitive genre awareness grew exponentially in terms of the overall number of comments they were able to produce (i.e., from 32, to 148, to 158 comments, respectively). However, when examining the specific categories within metacognitive knowledge and metacognitive regulation in which this development occurred (as also shown in Table 11 earlier), as a group, the LLMs appeared to be more adept in discussing their metacognitive knowledge versus how they used that knowledge in regulating their performance (100 versus 48 comments in Week 8, and 106 versus 52 comments in Week 16). Additionally, while the group as a whole was able to express more comments over time in general, within each respective category, development did not progress in a linear fashion. Within metacognitive knowledge, as a group the LLM students’ procedural and conditional knowledge of the office memo increased during each data collection period. Yet, the LLMs’ declarative knowledge increased dramatically between the pre-course and Week 8 periods, yet it appeared to decease slightly between Weeks 8 and 16, as students produced fewer declarative comments (and in fewer respective areas) surrounding the office memo genre. Conversely, within metacognitive regulation, as a group the LLM students’ regulatory control of monitoring and evaluating 106 increased from Week 8 to Week 16, yet their comments appeared to decrease slightly in terms of planning from Week 8 to Week 16. RQ3: The Relationship Between Students’ Metacognitive Genre Awareness and Perceived Text Quality Building on RQ2, I now present the findings for RQ3, which was an exploratory investigation into the relationship between students’ self-reported metacognition and the perceived quality of their office memo texts. As discussed in the Methodology chapter, following Meyer’s grading of the LLM students’ memos, he was asked to rank those memos from the six case study participants. Meyer was told to imagine himself as a senior attorney who had assigned the memos, and then, he was asked to rank the six participants’ memos in terms of overall effectiveness, holistically considering all aspects of content, organization, language, and legal analysis in relation to the functional purpose of the genre. After ranking them, Meyer discussed the strengths and weaknesses of each student’s memo in detail, including why he prescribed such rankings. This ranking procedure occurred twice—once after students’ first memo assignment and again following the completion of their second memo assignment. In what follows, I begin by presenting Meyer’s rankings and comments on the students’ memos, including those recurring themes in his evaluative comments. After establishing the themes on which Meyer based his evaluations, I compare Meyer’s rankings to students’ self-reported metacognitive genre awareness. Professor Meyer’s comments on students’ first memo assignment. Table 13 highlights both Meyer’s rankings and his comments regarding the strengths and weaknesses of the six LLM students’ first office memos. Unsurprisingly, those students whose memos were ranked higher tended to receive a greater number of positive comments from Meyer. Upon a 107 closer examination of Meyer’s comments, two broad recurring themes could be seen, which were related to the perceived importance of LLMs’ abilities: (1) to understand the legal reader/audience and that audience’s need for an appropriate amount of background information and content (i.e., case law) and legal analysis; and (2) to use appropriate language. In regards to the first theme of understanding one’s legal audience and their needs for adequate background information and analysis, Meyer made both positive and negative comments on students’ memos related to this theme. For instance, with Paul’s first memo, this can been observed with Meyer voicing that Paul appeared to heed Meyer’s in-class direction in considering the reader’s/audience’s needs with his remarks that “I’m not telling him to expand” in the same manner that he typically would for most of the other LLMs. However, this strength of Paul’s writing, in turn, became a criticism of Aashna’s first memo. In particular, Meyer felt that the audience was not provided with enough background information in her memo in order to comprehend the full scope of the issue. Meyer also partly attributed this to Aashna’s relative novice status and inexperience in dealing with the professional legal audience. This was apparent in Meyer’s comments that “this student is young” and “the reader is not really going to get a sense of how it applies.” In a related comment involving Luisa’s memo, Meyer again addressed the issue of the audience’s needs, but in a slightly different respect. In particular, Meyer felt that Luisa was considering her audience’s needs and including the appropriate amount of case law and analysis, but he felt that the manner in which she presented this information in her memo showed only a partial understanding of the busy legal reader’s desires to see the conclusions fronted or forecasted. This could be observed with Meyer’s comments of: “[She] makes me wait too darn long for it…Don’t build to it. Put it at the start.” In terms of the second theme of using appropriate language, Meyer’s comments with this 108 theme only surfaced when discussing two students: Yuki and Yurui. Specifically, Meyer felt that both of these students had some “fluency” issues—which, even though Meyer stated he attempted to ignore and instead focus on their content and analyses—Meyer still felt these language issues slightly hindered their memos’ overall effectiveness. This was evident in Meyer’s comment of “it just affects your ability to accurately describe what it is you’re saying.” In his comments, Meyer pointed to a specific language-based issue that both Yuki and Yurui exhibited, which was related to the use of verbs and time in their office memos. As Meyer discussed, in the situation that students were provided with for the first memo (i.e., the legal malpractice issue in which a couple was suing their former legal counsel), the memo was supposed to be written with the understanding that the suit was already in progress. However, Meyer noted that in their writing, both Yuki and Yurui often referred to the fact that their clients “want to sue,” leaving Meyer to question whether the two students were making a minor language-based error, or, whether they were not comprehending the context of the issue at-large. These two main considerations—that is, (1) the audience’s need for enough content/analysis, and (2) language use—caused Meyer to rank the LLM students’ first office memo assignments accordingly. When discussing these rankings as well, Meyer ultimately split the six memos into two tiers: (1) the top tier of the three memos from Paul, Jing, and Luisa, and (2) the lower tier of the memos from Yuki, Yurui, and Aashna. In reference to the top tier of students, Meyer discussed what made these memos stand out, specifically in reference to the first theme of the audience: They’re giving the senior partner, I think, a lot of what they senior partner would need to know. 109 Table 13 Professor Meyer’s evaluative comments on the first office memo assignment Rank Student Eval1 Comments 1 2 3 Paul Jing Luisa + + – + + – + – +/– 4 Yuki 5 6 Yurui Aashna + – – + – – + – “This read very well… Once we get into the body of law, very nice job explaining what’s going on.” “On so many of these memos, I’m telling [students to]: ‘Expand, expand, expand! The reader needs more. The reader needs more.’ …I’m not telling him to expand. He’s doing what he should.” “It was very sparse in the Fact section.” “I liked her memo… She does a nice job of good introductory sentences that really help keep the reader focused.” “I tell the students in class too: ‘You’ve got to take a stand… You gotta give us a prediction.’ …And she did a nice job of that.” “More case examples. I want more examples of how this works. …I’d say she’s got more than…[Aashna], but she needs more.” “She did a pretty decent job with the Facts.” “It was kind of sparse and unfocused… There’s still more you gotta do…in terms of explanation of the law, the illustration of the law and how the courts have actually applied…and then, applying it to us.” “One of the things she does [well is to say]…‘Here’s the takeaway point from this particular authority…’ [But she] makes me wait to darn long for it. And this is something I tell them about readers all the time, right? Tell me at the start: ‘Why are you bothering telling me about this case?…Don’t build to it, don’t build to it... Put it at the start.’” “He did a good job thinking about ‘here are these different lines of analysis’… He did a nice job of explaining what those were.” “This student in particular has some English fluency issues…but that, I think directly has some influence on—it just affects your ability to accurately describe what it is you’re saying. I know he’s thought about this. …I’m not too worried though…it’s not a focal point.” “He’s getting the approach—he’s getting what you have to do, but he’s maybe not doing as much of it [as he should].” “He’s got the components—he’s got the basics of what they need.” “It might be more of a reflection of language issues. It’s not that he doesn’t understand it. He clearly understands…[For example], he’ll say like ‘Our clients want to sue this particular law firm.’ And I’m circling [in his paper]… ‘We already have sued them, you know? That’s why we’re here.’” “He has a lot of what I refer to as ‘indefinite reference’… You know, ‘This means that’…where it's just not clear what he's referring to.” “I often remind the students just to provide some basic background principles before you jump right into the specific issue. Like, ‘What’s the background the reader needs?’…I thought she did a pretty good job of that.” “This student is young—basically right out of undergrad… [I need] more in terms of explaining the particular law and bring in the case examples and illustrate the rule, and then, applying them. …The reader is not really going to get a sense of how it applies.” 1Indicates whether evaluation comment was positive (+) and/or negative (–) in scope However, in reference to the lower tier, Meyer concluded that these three students were not 110 providing the reader with enough background information and analysis to show how they ultimately reached their conclusions: I, at least, know what your take is. What I’m not feeling is confident… I get that your conclusion is ‘we are likely to win’ or ‘we are likely to lose.’ But, if you’re not giving me enough to let me see how you get there, I don’t have the trust in that… You gotta show me how you got there. From the senior partner’s perspective, that’s what I wanna see. Professor Meyer’s comments on students’ second memo assignment. Table 14 highlights Meyer’s rankings and evaluative comments on the LLMs’ second office memos. Like the first office memo, students whose memos were ranked higher tended to receive a greater number of positive comments from Meyer. Also similar to the first office memo is that there were multiple recurring themes in Meyer’s evaluative comments. Meyer again discussed the importance of LLMs’ abilities to (1) understand the legal reader/audience and that audience’s need for an appropriate amount of background information and content (i.e., case law) and legal analysis. In the second memo though, he no longer discussed the theme of language use, and instead, he focused on LLMs’ need to (2) adopt an appropriate organizational scheme. In regards to the first theme related to the audience’s need for sufficient content (i.e., including case law) and analysis, this theme once again surfaced in the form of both positive and negative comments that Meyer made. For example, in complementing Paul’s memo, Meyer simply remarked: “It was very thorough… he’s got lots of case law.” Relatedly, Meyer made positive comments on Yurui’s and Jing’s second memos with his discussion of how both writers adequately addressed their legal reader’s needs. With these two students’ memos, this was apparent in Meyer’s comments of: “[Yurui] not only did a nice job of setting out all the pertinent 111 principles, but also showing the reader how that works,” and, with Jing: “that’s real-world lawyering.” Again though, these positive attributes involving the legal reader/audience became criticisms levied against those memos that were ranked lower. This was clear in Meyer’s negative remarks on Yuki’s memo with: “He’s getting the basic ideas of what we’re talking about, but it doesn’t always transfer so well.” This was also evident in his remarks on Aashna’s memo with: “You really need to add a couple more cases here.” Table 14 Professor Meyer’s evaluative comments on the second office memo assignment Rank Student Eval1 Comments 1 2 Paul Yurui 3 Jing 4 5 6 Luisa Yuki Aashna + + + + – + + – +/– +/– + +/– – – – “Very comprehensive… It was very thorough. He’s got lots of case law.” “He also made a lot of good number of analytical suggestions.” “This was quite good… Organized the memo as it should be.” “Not only did a nice job of setting out all the pertinent principles, but also showing the reader how that works—you know, not just ‘here’s the abstract principles’ but also ‘here’s how the courts have applied it.’ That helps the reader immensely in an American legal system.” “For some students [like Yurui], learning when to cite… [What he’s saying] is accurate, but he needs to provide authority to it. It’s just expected in the American law office.” “So I thought that she did a pretty good job... Her organization is fine.” “One of the things she did a nice job of… One of the things we wanted in this assignment was… ‘What does all this mean as a practical matter for the client?’ …She did a really nice job with that. That’s a critical part of lawyering. …That’s real-world lawyering.” “There’s some places, she could use more case law… In a real-world scenario, you want to give the senior partner—the reader—as comprehensive a sense of the law as you can.” “It’s passable, but substantively there’s a lot more she could do.” “She gets it. She knows what’s going on. But it doesn’t necessarily transfer to the written product.” “She’s got the basic organization structured correctly.” “He’s getting the basic ideas of what we’re talking about, but it doesn’t always transfer so well.” “The organization was really muddled. …The organization of this memo should be driven by what the Michigan Supreme Court has said.” “There are some substantive analytical things [that could be improved].” “You want to structure Questions Presented as law plus facts: What’s the legal issue in the context of these facts?... She didn’t really include that factual part… That reflected itself later in the analysis.” “You really need to add a couple more cases here. This is unclear.” 1Indicates whether evaluation comment was positive (+) and/or negative (–) in scope 112 In regards to the second evaluative theme involving LLM students’ need to adopt an appropriate organizational scheme, this, too, surfaced as a source of strength or weakness for students’ respective memos. Meyer mentioned the texts’ organization when discussing four of the six participants’ memos. On three separate occasions, Meyer positively discussed the manner in which students had organized their memos. This was clear with simple comments that Meyer made on students’ memos like Yurui’s such as “[he] organized the memo as it should be,” and also, with those comments on Luisa’s, such as: “She’s got the basic organization structured correctly.” Apart from these complements though, Meyer mentioned that not all of the students had organized their memos appropriately. In particular, Meyer singled out Yuki’s memo, as Meyer felt that the organization was “muddled” and should have been driven by certain internal considerations involving the legal issue at-hand. When ranking students’ second office memos, Meyer noted that in particular, Yurui had made a substantial amount of progress between the first and second office memo assignments, as Meyer remarked: “he made quite a jump this time.” Beyond this individual student though, Meyer also reflected on the progress that he felt the six LLM students had made collectively as a group. Specifically, Meyer reflected on his own in-class pedagogical focus of stressing the importance of a writer understanding his/her legal audience and their needs. Due to this concerted pedagogical effort to direct students’ attention to the audience, Meyer felt that the LLMs had been able to make progress in addressing their legal readers’ needs, stating: One of the things I think students did get better at…putting aside the substance of the analysis. The things we can do as a communicator—as a writer—to make our analysis easier to follow… The idea of like leading with conclusions; the idea of pulling out principles from a case…not to build to that… That’s something that I emphasize a lot in 113 this class because it’s critical to professional writing… Students are getting better at that. And that, to me, is a big deal… And that is a skill. That, to me, is an important skill in communicating legal analysis in an American professional setting. Comparing Meyer’s rankings with students’ self-reported metacognition. Clearly, when assessing/ranking the effectiveness of the six LLM students’ first and second office memos, Meyer considered a wide range of factors that were related to writing the genre. These factors included (but were not limited to): understanding the various needs of the professional legal audience (e.g., fronting conclusions for the busy reader, making predictions or practical recommendations, etc.); incorporating sufficient content (e.g., facts, case law, etc.) and legal analysis showing one’s steps; using appropriate language (e.g., verb tenses in the correct time); adopting a logical organization (e.g., a general organization for the memo itself, and, a sub- organizational scheme conditionally driven by the topic); and more. Meyer attempted to raise students’ awareness of such factors throughout the course in various in-class sessions and through the assigning of supplemental homework readings. As such, as was shown earlier in RQ2, the six LLM students’ developed metacognitive genre awareness in many of these areas throughout the legal research and writing course. Sometimes students developed similarly, but other times, they developed quite differently in terms of the respective rate and/or area of development. Thus, I now compare Meyer’s evaluations/rankings with students’ reported metacognitive development during the semester. Table 15 below shows the rankings that Meyer assigned to students’ office memos. In addition, Table 15 illustrates the total number of comments that the LLM students were able to produce when discussing their awareness of the office memo genre during Week 8 and Week 16, respectively. These comments are separated into three categories, which include: metacognitive knowledge, 114 metacognitive regulation, and metacognition in general. Table 15 Professor Meyer’s memo rankings and students’ total metacognition comments Memo Metacognitive Metacognitive # 1 1 1 1 1 1 2 2 2 2 2 2 Student Name Meyer’s Holistic Ranking* Knowledge Regulation (# of comments) (# of comments) Paul Jing Luisa Yuki Yurui Aashna Paul Yurui Jing Luisa Yuki Aashna 1 2 3 4 5 6 1 2 3 4 5 6 20 20 16 13 14 17 17 19 17 20 16 18 11 10 8 9 8 2 15 9 9 5 7 6 Metacognition (Total # of comments) 31 30 24 22 22 19 32 28 26 25 23 24 *Note: 1 = highest-rated and 6 = lowest-rated memo in terms of effectiveness As Table 15 shows, only two of the case study participants—Paul and Aashna— maintained the same rankings across both memo assignments, with Paul’s memo being ranked as the most effective and Aashna’s being ranked as the least effective. Apart from these two individuals though, for the most part, the other four LLMs maintained relatively similar positions between the two assignments. However, as previously noted, Yurui appeared to make a large amount of progress between the first and second assignments, with his first memo ranking fifth in terms of overall effectiveness and his second memo ranking as the second most effective. When examining the total number of comments that the LLMs produced compared with Meyer’s rankings, in first inspecting the category of students’ reported metacognitive knowledge (i.e., students’ general awareness), there does not appear to be a clear relationship between the total number of comments LLMs produced and the rankings that Meyer assigned. Yet, when moving on to the category of metacognitive regulation (i.e., students’ reported abilities to use 115 their awareness when performing the task), a more recognizable pattern starts to emerge. With some exceptions, those students who were more able to discuss how they used their awareness during planning, monitoring, and/or evaluating tended to receive higher effectiveness rankings. This pattern becomes even clearer when combining the subcomponents of metacognitive knowledge and metacognitive regulation into the holistic concept of metacognition. With the sole exception of Aashna’s second office memo assignment, the number of metacognition comments produced by the LLMs corresponds to Meyer’s holistic effectiveness rankings quite uniformly. This was also reflected in other ways at an individual level across the first and second memo assignments as well. For instance, although Yurui’s memo was ranked fifth in the first memo assignment, it was subsequently ranked second by Meyer during the second memo assignment. This positive shift in Yurui’s effectiveness ranking is supported not only from Meyer’s own perspective with his comment of “he made quite a jump this time,” but also, from the general increase in the number of comments that Yurui was able to make compared to his LLM peers. 116 CHAPTER 4: DISCUSSION In this section, I expound on those findings presented in Chapter 3. Each of the three RQs is addressed individually, and I discuss them in relation to previous SLA literature when applicable. Then, the chapter closes with a brief discussion of some of the limitations of the current study. RQ1: Law Instructor’s Pedagogical Practices in a Content-Based LLM Writing Course As discussed in Chapter 1, in the area of L2 genre-based teaching and research, one of scholars’ primary interests to date has been to explore how instructors approach the teaching of genres and raising genre awareness with their L2 learners (e.g., Flowerdew, 2016; Harman, 2013; Mahboob, Dreyfus, Humphrey, & Martin, 2010; Myskow & Gordon, 2010; Racelis & Matsuda, 2013; Tribble & Wingate, 2013; Yigitoglu & Reichelt, 2014). Some studies have focused on describing teachers’ selections of course materials and activities (e.g., Myskow & Gordon, 2010; Yigitoglu & Reichelt, 2014), while others have attempted to show how teachers’ practices impact the development of students’ genre awareness and writing (e.g., Harman, 2013; Hultin & Westman, 2018; Tardy, 2015). Despite the apparent abundance of research, most of these studies have occurred within the classrooms of ESL/EFL teachers who are adopting specific ESP or SFL approaches. This uniformity has led researchers to call for more investigations into the classroom practices of other L2 instructors (e.g., Leki et al., 2008; Yigitoglu & Reichelt, 2014), and especially, involving those contexts occurring in CBI and LLM programs since they have remained relatively unexplored (Baffy, 2017). Thus, in the current study, RQ1 examined the beliefs and pedagogical practices of a law instructor in an LLM legal research and writing course, as the instructor taught the office memo genre to a group of L2 English learners. The connection between beliefs and pedagogy. Numerous studies within the domain of 117 L2 writing have examined the beliefs of instructors and how those beliefs affect classroom pedagogy. These studies have tended to focus on phenomena such as corrective feedback though, and, to what extent teachers’ beliefs regarding feedback align with the types/amount of feedback they provide to their learners (e.g., Lee, 2009; Yang & Gao, 2013). No studies (to my knowledge) have directly examined the connections between L2 writing instructors’ beliefs and their practices involving genre pedagogy. Yet, of those corrective feedback studies, Lee’s (2009) work is particularly noteworthy as her findings showed that teachers’ beliefs can have a powerful influence on their actions in the classroom; likewise, although teachers may sometimes openly state they believe one thing, an observer’s analysis of their actual practices may reveal a mismatch in this regard. In terms of Professor Meyer’s practices in the current study, it is interesting to note the connections between his beliefs about teaching and how those beliefs manifested into actual classroom practices. When completing his visualization reflection, Meyer stated that he viewed his role as a teacher in terms of the importance of two things, as being: (1) a disseminator of knowledge (which came in the form of the trumpet/megaphone image in his illustration), and (2) as someone whose role it was to discover his students’ individual capabilities and needs (which came in the form of the steps, spring, and ladder images in his illustration). In terms of being the primary disseminator of knowledge, this was clearly apparent in numerous ways. For instance, Table 4 shown earlier provided the reader with an overview of the legal research and writing course’s weekly schedule, topics, and in-classroom activities. In reviewing this table, one can see that during these various lessons and in-class activities, a large majority of the content was teacher-led or teacher-driven, with only occasional group work occurring in some of the class sessions. Noticeably, this heavy, teacher-fronted discussion and 118 presentation-style of delivery reflected Meyer’s beliefs that it was his primary role to deliver knowledge to the LLM students. Meyer specifically mentioned this during his visualization reflection as well, stating: It's not the ‘sage on the stage,’ but it is. I mean, there's nothing wrong with learning from authority. That's how learning progresses in many respects. At face value, this statement may appear to be somewhat of a contradiction. However, as Meyer further clarified, it was his belief that teaching and learning should consist of a variety of practices involving lectures and activities. However, Meyer felt that not all learning could occur from activities alone. Instead, he believed that the acquisition of knowledge many times required knowledge to be passed down from others who possessed greater knowledge or authority. In regards to Meyer’s other belief that it was his role to discover his students’ individual capabilities and needs, this also was somewhat apparent in his practices. It could be witnessed through one action in particular, which was Meyer’s conferencing with the LLM students. During Weeks 6 and 12 of the course, Meyer structured the syllabus so that the LLMs would have time to discuss their office memo research processes with him in lieu of a typical in-class session. As Meyer saw it, these conferences were an opportunity to get to know his students and their abilities, yet also to provide them with individualized feedback. This was evident as he discussed why he instituted these research conferences into the syllabus: It's like some sort of demonstrable reflection of a skill that they're doing in law school. When otherwise, in law school, especially in their first semester here in American law school, you're not getting a whole lot of direct feedback on how well you're doing. While these two beliefs regarding his own role were made clear during the visualization reflection, Meyer’s pedagogical practices also revealed additional aspects which he felt were 119 important in teaching LLMs legal writing. In particular, three recurring themes emerged during my in-class observations and examination of course artifacts, which I later inquired about during an interview with Meyer. Meyer expressed that with his students, he felt it was essential to: (1) build students’ reading skills, (2) raise awareness of the office memo genre, and (3) balance a real-world context with LLM students’ current needs. As an observer in the classroom, of particular interest was Meyer’s emphasis on building students’ reading skills, as this dominated a majority of the class sessions. Meyer, too, later acknowledged that he felt reading was essential, and without it, he believed that his students would be unable to write an effective office memo. He attributed the source of this belief to his years of personal experience in working with LLMs and understanding their previous law experiences: So many of them are coming from non common-law countries, that [is] one of the things that we're also trying to get across in this class: what's different as a common law lawyer as compared to a civil law lawyer.2 Interestingly, like Lee’s (2009) study, which highlighted both the connections and mismatches between teachers’ beliefs and practices—from an observer’s perspective in the current study—Meyer’s belief involving the importance of building reading skills seemed to be both in-keeping with yet simultaneously at-odds with his goal of (3), balancing a real-world context with LLM students’ current needs. For instance, in order to help facilitate reading development, Meyer engaged in editing students’ case law readings. This practice, too, he attributed to his prior experiences with LLMs, stating: From experience…I didn't use to edit the cases nearly as much, and then students would just get hung up on the wrong things. 2 As a reminder, in civil-law countries, the legal system is dictated by codes, statutes, principles, etc., rather than being dictated/influenced by legal precedent from prior judicial opinions. 120 This specific editing practice appeared to directly support Meyer’s belief in the importance of building reading skills. Yet, it was also Meyer’s belief that he should not provide his LLM students with language support in-class. For this, the primary reason he provided was due to his desire to establish the context of an American law office. As he explained, lawyers in an American law office might be willing to overlook some of the LLM’s English language or grammatical issues in their memo writing; thus, the practice of forgoing in-class language support appeared to align with an authentic American law office context and how lawyers might perceive L2 writing. However, this ‘no-language support’ belief also appeared to be at odds with Meyer’s goal of building reading skills. In particular, it seems that devoting some attention to language function and/or form during class reading-based discussions likely would not only assist LLMs in developing reading skills (see Nation, 2009), but also in developing their reading in conjunction with their writing skills (e.g., Harman, 2013). In addition, during the span of my interviews with the LLM students, a couple of them remarked that they wished there was some focus on English language use during class (although, they were not overtly specific about the types of language instruction they wished to see). The instructor’s practices compared to approaches in legal writing, CBI, and L2 genre-based pedagogies. As discussed, Professor Meyer’s personal beliefs were that a teacher should serve as the main disseminator of knowledge and should identify his/her students’ individual capabilities and needs. Likewise, in teaching the LLMs, Meyer felt it was essential to: (1) build students’ reading skills, (2) raise awareness of the office memo genre, and (3) balance a real-world context with LLM students’ current needs. I now turn to a more focused discussion of (2)—that is, Meyer’s practices related to the teaching of writing and raising awareness of the office memo genre. As mentioned, Meyer stated that apart from self-study in the form of reading 121 various academic law journals, books, and attending legal writing conferences, he had no formal teacher training and no training in working with L2 learners. However, many of his pedagogical practices reflected other established pedagogies in the spheres of legal writing and CBI, and less so, that of L2 genre-based pedagogies. In terms of legal writing scholarship, many of Meyer’s practices mirrored those presented in published guidebooks and other works. For instance, teachers’ manuals such as those by Weresch (2009) have suggested that instructors should integrate discussions of ethics and professionalism into their legal writing courses. This was something that Meyer adopted in his practice, as he not only devoted class time to such discussions during Weeks 7 and 14 of the course, but he also integrated an ethics-related issue into the topic of students’ first office memo assignment itself (i.e., with the issue involving legal malpractice). In addition to teachers’ manuals, as mentioned in Chapter 1, there have been multiple textbooks created that are intended to provide students and teachers with detailed information regarding general characteristics of legal writing, legal reasoning, and commonly used legal genres with annotated examples (e.g., Chew & Pryal, 2016; Edwards, 2006; Oates et al., 2018). Meyer’s practices closely reflected the content presented in many of these textbooks, as he chose to include homework readings throughout his course that focused on general characteristics of legal writing and legal reasoning, some of which were even directly sampled from these aforementioned authors’ textbooks. Meyer engaged in other practices, too, that have been promoted as effective by legal writing scholars—methods that are specifically aimed at assisting students in building reading skills and with researching and writing office memos. For example, Meyer provided his class with note- taking strategies, and he modeled and encouraged students’ use of graphic organizers (for examples, see McGregor & Adams, 2008; Piccard, 2017). Additionally, he crafted his own 122 office memo with annotated examples (shown earlier in Figure 6) with the intention of helping raise students’ awareness to different features of the office memo (for additional examples, see Chew & Pryal, 2016; Oates et al., 2018). Clearly, Meyer’s practices closely reflected many of those within the sphere of legal writing scholarship. Although Meyer maintained that he had no training in working with L2 learners, some of his practices did reflect suggestions by scholars in CBI as well. In CBI, content is seen “as a resource for learning language” (Stoller & Grabe, 1997, p. 78). Thus, language is typically viewed as a secondary focus. This sentiment was echoed in Meyer’s own belief that language instruction was not the goal of his legal research and writing course, yet Meyer still provided his LLMs with some nominal language support—though, as discussed, this support did not come during class sessions but instead came in the form of out-of-class homework readings. Likewise, Kasper (2000) has highlighted that even though a CBI instructor’s focus is typically on the course content itself, teachers do often attempt to engage in a variety of instructional practices that will help facilitate L2 acquisition for their learners such as: editing course materials so that they are more accessible; presenting information in a variety of aural and visual formats, such as utilizing visual aids like videos and graphic organizers; and more. In his own teaching, I observed Meyer engaging in all of these practices. Interestingly though, as mentioned, Meyer attributed the source of these practices not to formal training, but instead to his first-hand experiences in working with the LLM student population over the years. Though Meyer’s pedagogical practices were clear reflections of established pedagogies within legal writing scholarship, and, to some extent within CBI, the relationship between his strategies for building genre awareness are much less apparent when compared to L2 genre- based pedagogies such as ESP, SFL, and the New Rhetoric approach. Meyer’s own approach to 123 teaching the office memo, in general, does not appear to align with any one of the three aforementioned pedagogies. If anything, his pedagogy perhaps resembles a mixture of the three. For instance, in teaching the office memo genre, Meyer placed the foremost stress on the importance of writers understanding their audience, or as he succinctly put it during a Week 14 lecture: “Everything boils down to audience.” This emphasis on audience and understanding the social nature of one’s discourse community is a tenet shared by all three genre-based pedagogies (see Paltridge, 2014 for a review). In addition to audience, Meyer emphasized that students needed to understand the basic organizational structure of the memo, as he occasionally directed students’ attention to its general structure/organization (during 4 of the 11 in-class sessions in Weeks 2, 3, 5, and 10). Meyer even devoted class time to showing LLMs a sample memo outline via the classroom projector and highlighting a common organizational structure (see Figure 4 shown earlier). Unlike in an SFL approach in which a teacher might adopt a “teaching-learning cycle” (e.g., Rothery & Stenglin, 1994) to first model and deconstruct the organization of the text, Meyer instead gave students an explicit pre-prepared organization as a guide. Thus, this practice perhaps most closely resembles an ESP approach to teaching. Yet, unlike ESP, there was no further step involving the teaching of “rhetorical moves” and/or formulaic language (which Meyer was explicitly against). Unlike SFL as well, there was no subsequent joint construction stage. Conversely, and still related to the organizational structure of the office memo, Meyer engaged in aspects of raising genre awareness that also might closely align with SFL. After providing students with the outline for a standard memo (in Figure 4), Meyer then further indicated that beyond this general organizational scheme, there also might be a different sub- 124 organization that is required. Importantly, he mentioned that this sub-organization might be quite fluid or conditional in nature because it might be dependent on factors such as the law itself (i.e., how the law is written or structured in terms of whether there are multiple parts/components that need to be addressed). Meyer’s attempts to raise students’ awareness to the conditional effects of the topic on the organization can be observed in SFL approaches, too, as SFL scholars have tended to emphasize the importance of understanding genres in relation to other integral factors such as context, topic, etc. (e.g., Halliday, 1994; Halliday & Matthiessen, 2004; Martin & Rose, 2008). RQ2: The Development of L2 Learners’ Metacognitive Genre Awareness In conjunction with the need to explore the genre pedagogies of L2 writing instructors, there is also a need for SLA researchers to further understand the development of learners’ genre knowledge over time (see Polio, 2017). As discussed, there are relatively few studies within this domain. However, of particular interest is the small handful of studies that have examined the development of L2 learners’ genre knowledge using metacognition theory (Flavell, 1979), as such inquiries have highlighted the highly complex nature of developing of metacognitive genre awareness (e.g., Negretti, 2012; Negretti and Kuteeva, 2011; Yeh, 2015). Thus, in the current study, RQ2 adopted metacognition theory for the purposes of exploring the longitudinal development of six L2 English speakers as they learned to write the office memo genre during an LLM legal research and writing course. The connection between instruction and awareness. Prior to discussing the development of the LLM students’ metacognitive genre awareness, it is worth briefly highlighting the connections between Meyer’s pedagogical practices and their subsequent impact on students’ awareness. As mentioned, some prior L2 genre studies have attempted to show how 125 teachers’ classroom practices have affected the development of students’ genre awareness and writing itself (e.g., Harman, 2013; Hultin & Westman, 2018; Tardy, 2015). Like those prior studies, in the current study there were many instances in which the connection between Meyer’s teaching and students’ awareness were quite clear. One vivid illustration of the effects of Meyer’s pedagogy on students’ learning can be seen in Meyer’s belief—as discussed in the previous section—that LLMs must understand their audience when writing the office memo. As shown, Meyer expressed this idea repeatedly throughout the course (in 5 of the 11 total in-class sessions during Weeks 2, 3, 5, 9, and 14). As such, this emphasis he placed on audience clearly affected students’ awareness. This was illustrated in Table 11, in which the top 15 recurring thematic codes were presented based on students’ self-reports of their metacognitive genre awareness. As Table 11 showed, the most frequently recurring comment code involved students expressing their declarative knowledge of the audience. Additionally, a further examination of the individual code categories revealed that 5 of the top 15 codes were related to some aspect of the office memo’s audience. These findings are somewhat representative of other authors’ works including Tardy (2015), who investigated an undergraduate research methods class in environmental science. In exploring how the course professor attempted to build students’ literacy and reading skills, Tardy noted that many of the students tried to imitate their professor by repeating the professor’s words orally during their interviews with Tardy. In the current study, too, Meyer’s continued stressing of different facets related to understanding the genre’s audience appeared to penetrate different levels of students’ conscious awareness. Because of this, during our interactions, students consistently discussed aspects of the memo’s audience, reporting high-levels of declarative and conditional awareness. Beyond the influence of audience—though much less frequent—were the impacts of 126 Meyer’s pedagogy at a linguistic level. Although Meyer’s in-class teaching did not appear to be a determining factor in affecting students’ linguistic development, his choices involving language-based homework readings did appear to have some occasional influence. One instance in which this was apparent was with the case study participant Yurui. As noted, during one of our interactions, Yurui discussed how his understanding of the reporting verb to argue had changed. As he stated, this development in his understanding was the result of a homework reading he had completed, which then influenced his awareness and avoidance of the verb when writing his office memo: And when writing, [I remembered] like the grammar stuff… The court did not ‘assert’ or ‘argue,’ but court only ‘states, notes,’ or ‘maintains’… I use that word once [i.e., argues], and…it is wrong because court did not argue anything…because a court is supposed to be neutral. When I write, I didn't know the world ‘argue’ has such a specific implication. Somewhat relatedly, in a classroom-based ethnography by Harman (2013), Harman highlighted how an 5th grade classroom instructor’s practices imbued her ESL learners with a metalinguistic awareness surrounding the types of linguistic choices that were available to them when writing. Harman stated that the instructor’s approach to teaching the genre encouraged her L2 learners to both borrow (from their readings) and play with different lexical patterns when writing. In the current study, though the appropriation of language was clearly not to the same extent as reported in Harman’s, there were occasional instances in which Meyer’s selection of course materials/readings did appear to promote students’ linguistic awareness and their subsequent adoption of specific language while writing. The development of the LLMs’ metacognitive genre awareness. Over the span of the 127 15-week legal research and writing course, all six of the LLM students’ developed new facets of metacognitive genre awareness. Though some of the participants began at slightly different starting points, by the end of the course they were able to express a wide-range of awareness related to metacognitive knowledge (i.e., general awareness) and metacognitive regulation (i.e., the ability to report how they used that awareness). One important finding in relation to the LLM students’ individual trajectories is that their awareness did not appear to develop uniformly—that is, although some students seemed to develop awareness rather quickly within one respective area of the metacognition model, other students did not. For instance, in comparing the developmental sequences of Aashna and Paul, both LLMs appeared to start at relatively similar places in terms of their pre-course awareness of the office memo genre. At this preliminary stage, both students possessed minimal declarative knowledge concerning the memo’s audience and purpose. By the time the second round of data collection occurred during Week 8, both Aashna and Paul self-reported a number of new developments. However, Paul was clearly able to exhibit a much greater, more complex metacognitive awareness of the genre. This could be seen not only in Paul’s ability to discuss his metacognitive knowledge, but also in his ability to comprehensively report how he used that awareness in regulating (or in planning, monitoring, and evaluating) his own writing. Though Paul showed an impressive range of regulatory abilities, conversely—even when specifically asked during the interview/stimulated recall—Aashna was unable to discuss how she used her awareness for the purposes of planning and evaluating her writing of the office memo. This finding regarding the uneven nature of development is reflected in other prior L2 studies involving metacognitive genre awareness. In Negretti and Kuteeva’s (2011) three-week study, the authors examined pre-service English teachers’ development of metacognitive 128 knowledge in relation to learning academic research articles. In their findings, Negretti and Kuteeva illustrated that even though all of their participants were able to develop declarative and procedural knowledge of genre-relevant aspects, only a few of their participants developed conditional knowledge. Likewise, in Yeh’s (2015) study on the development of EFL learners’ metacognitive regulation involving academic research articles, Yeh reported that many EFL students would quickly excel in one area such as planning, only to then face struggles in evaluating their own performance. One of the potential underlying factors, which might be responsible for affecting learners’ differing rates of development, is related to students’ prior experiences. Beyond those aforementioned studies that specifically adopted metacognition theory, other studies pertaining to L2 genre learning have revealed important insights. Research by Tardy (2005, 2009) and Gao (2012) has shown that the acquisition of genre knowledge is multifaceted and a skill that is gradually built up over time. Additionally, their works point to the conclusion that genre knowledge and performance may be highly susceptible to individual differences and other factors such as a learner’s content familiarity, linguistic differences, and even L1-based knowledge. In the present study, too, there were some cases in which the impact of LLM students’ prior experiences was clearly visible on the types of genre awareness they acquired. With Luisa and Yuki in particular, by Week 8 of the semester, both learners were able to discuss their declarative knowledge concerning the differences and similarities between office memos in the U.S. versus in their respective home/L1 contexts. This, unmistakably, was the result of both students having professional work experiences in law offices prior to joining the LLM program. Meanwhile, Aashna, who had no prior work experiences and had only recently graduated from 129 law school, never developed this type of declarative awareness during the course of the study. Although the relationship between prior experience and genre awareness was sometimes quite apparent, in other cases, the reasons behind why students developed awareness differentially was far less clear. For instance, in comparing the developmental trajectories of Jing and Luisa, before joining Hutz’s LLM program, both women worked in law offices for two years, and each stated they had a few brief experiences in working with office memos. Despite (seemingly) similar starting points, backgrounds, and years of L2 English study, Jing consistently produced a greater volume of comments in a number of different areas for both memo assignments. In attempts to understand a potential explanation for this finding, it is important to note that only Negretti’s (2012) work has explored the connection between learners’ general awareness and how they use that awareness in regulating their writing performance— (i.e., those studies by Negretti and Kuteeva, 2011 and Yeh, 2015 only explored subsections of the metacognition model). In her study, Negretti (2012) reported that learners’ metacognitive knowledge appeared to be strongly interconnected to their self-regulation. In particular, Negretti suggested that these different types of knowledge both fed into and affected one another, or essentially, that greater metacognition in one area led to greater metacognition in another—(i.e., awareness fed awareness). However, in the current study, no such relationship could be observed. For instance, in examining the total number of comments that the LLMs produced and the respective areas in which they produced them (as shown earlier in Tables 12 and 15), there does not appear to be any clear association between one type of awareness and another. This is to say: an increase in self-reported metacognitive knowledge (and its subcategories) did not appear to result in an increase in metacognitive regulation (or any subcategories), or vice-versa. Therefore, unfortunately, when removing the potential impact of one’s experience on 130 development, the reason behind why some students like Jing seemed to outpace others like Lusia remains a mystery. Thus, I return to this issue again in Chapter 5 in the discussion of future research directions. RQ3: The Relationship Between Students’ Metacognitive Genre Awareness and Perceived Text Quality As discussed in the previous section, there is a need for studies to explore the development of L2 learners’ genre awareness in different instructional contexts. In addition to exploring genre learning, through the adoption of metacognition theory as a lens, there is also a need to examine the full metacognition model, as prior studies have primarily explored sub-sections of the model only (e.g., Negretti & Kuteeva, 2011; Yeh, 2015). As all of these studies have suggested though, metacognitive genre awareness does not develop uniformly. Additionally, when both components of the theory are explored as in Negretti (2012), the findings highlight how important it may be to understand not only which facets of awareness develop, but also what connections (or disconnections) may exist between general awareness and the regulation of performance. Thus, it is crucial to understand how this knowledge develops and to what extent it relates to actual writing performance. Therefore, RQ3 of the current study was an exploratory investigation into the relationship between LLM students’ self-reported metacognitive genre awareness and their office memos’ text quality as perceived by the course instructor. Greater metacognitive genre awareness may lead to greater text quality. Prior to discussing the relationship between students’ metacognitive genre awareness and perceived text quality, as mentioned, it is important to note that RQ3 was exploratory in nature, and is, by no means, claiming to be an exhaustive attempt at correlating the two. A large body of quantitatively-oriented research has examined the relationship between students’ metacognitive 131 strategies and their task performance (see Sato, in press), yet no studies have examined the relationship between students’ metacognitive genre awareness and their perceived text quality. Thus, for examining the relationship between awareness and performance, students’ self reports were compared against Professor Meyer’s holistic rankings and Meyer’s evaluative comments on the memos. From an analysis of Meyer’s comments provided during my interview with him, it was clear that when evaluating and ranking students’ memos, Meyer considered a broad range of criteria that included the memo’s content, legal analysis, organization, language use, and more. In examining Meyer’s rankings (presented earlier in Table 15), it is important to note two things: The first is that, although the development of students’ metacognitive genre awareness sometimes appeared to be facilitated by their prior experiences, Meyer’s holistic rankings did not appear to have any clear relationship with students’ prior experiences. This is to say: those students with the most professional/work experience did not receive higher holistic rankings on their memos. For instance, Yuki clearly had the most experience, yet his memos were ranked as third and fifth most effective, respectively. Likewise, Yurui, who had only held a brief school- based internship (and was one of the two youngest case study participants), had his memos ranked fifth and then second, respectively. Secondly, just as experience did not appear to have any clear impact on students’ writing performance, the number of years that the students had studied L2 English also did not appear to have any clear influence. While Aashna had studied English for 17 years (longer than any participant in the study), Paul had studied English for only 7 years. Yet, Aashna’s office memos were consistently ranked last in terms of perceived quality, while Paul’s memos were consistently ranked as the best of the six office memos. Thus, in terms of the relationship between metacognitive genre awareness and performance, in comparing Meyer’s rankings with the total number of comments produced by 132 the LLMs, there did not appear to be a clear relationship between LLMs’ reported metacognitive knowledge (i.e., students’ general awareness) and the rankings they received. However, a more recognizable pattern began to emerge when examining the number of comments students produced within the category of metacognitive regulation (i.e., students’ abilities to report how they used their awareness in the writing task itself). Ultimately, this pattern became even more apparent when students’ memo rankings were compared with their metacognition in general (i.e., the total number of comments students produced involving their metacognitive knowledge plus their metacognitive regulation). There was only one exception in this regard, which came with Aashna’s second office memo assignment, which did not fit into the overall pattern. These findings suggest that metacognitive knowledge, on its own, may not be a strong indicator of one’s actual performance. This is simply to suggest: just because someone can discuss what he/she knows regarding a specific genre, this does not necessarily mean that he/she can produce that genre (or, more accurately, produce it well). Likewise, though students’ self- reported abilities to regulate their writing performance appeared to be a stronger indicator of their texts’ perceived quality, even stronger was when students could discuss both their general awareness (metacognitive knowledge) and how they operationalized that awareness (metacognitive regulation) together. Though exploratory, the findings of RQ3 are consistent with scholarship within the domains of L1 and L2 research involving metacognition. Within L1 research alone, scholarship has shown that those individuals who exhibit greater metacognition surrounding a given task tend to outperform their peers in number of areas. This is not only limited to writing performance (e.g., Harris et al., 2010), but it has also been shown in numerous other areas, such as in: mathematical problem solving (e.g., Dignath & Buttner, 2008; Garofalo & Lester, 1985) 133 and reading comprehension (e.g., Pintrich, 2004; Wang & Chen, 2013; Woolley, 2011). Likewise, specifically within the field of L2 writing, recent scholarship has suggested that there is a strong connection between learners’ metacognition and their L2 writing performance (e.g., Qin & Zhang, 2019). In a further examination of Table 15 (as mentioned during the discussion of RQ2), one observation of interest is that greater self-reported metacognitive knowledge did not consistently appear to result in greater metacognitive regulation, or vice-versa. While this finding is somewhat puzzling in the context of the current study, perhaps it may be related to Negretti’s (2012) theorizing that different types of metacognition might subsequently feed into other types—(e.g., in Negretti’s study, students who exhibited greater conditional awareness also tended to exhibit greater monitoring capacities). While Negretti has theorized that there might be potential connections between metacognitive knowledge and metacognitive regulation, Yeh’s (2015) study has further complicated this notion, showing that within each of these respective areas, there may be disconnections within the subcategories. For instance, when it came to metacognitive regulation in Yeh’s study, all 16 of the EFL learners remarked that the planning stages of writing did not necessarily “lead to well written texts” (p. 491), suggesting that the sub- components of the model may not be interconnected. In the current study, Negretti’s (2012) findings were not reflected in the data. However, Yeh’s (2015) findings did appear to be supported, as the LLMs who reported greater planning and/or monitoring abilities did not consistently report higher levels of evaluating. Limitations Throughout the duration of this study, many important insights were gleaned into the nature of discipline-specific genre pedagogy and its impact on L2 learning and performance. That being 134 said, this study is not without its limitations. In terms of the connections between RQ1 and RQ2, the adoption of both a classroom-based ethnography and a case study design had clear strengths in terms of affording a robust perspective on Professor Meyer’s pedagogy, and as to how Meyer’s practices impacted his LLMs’ genre awareness. Yet, the adoption of the corresponding methods also meant that I was unable to bear witness to any of the L2 teaching and learning that occurred outside of Meyer’s classroom. Though, in many instances, students’ development of metacognitive genre awareness could be directly traced to Meyer’s pedagogical practices, there were other instances in which development could not be accounted for. Thus, it is likely the case that some of the genre learning that took place occurred beyond Meyer’s classroom and those readings that he assigned (e.g., in the form of peer-to-peer learning, via self-study, the use of technology, etc.). In my own work (e.g., Kessler, 2020; Kessler, Solheim, & Zhao, 2020), I have called for richer investigations into the connections between students’ in-and-out-of-classroom practices. Thus, in the current study, although the findings can account for what students heard and did in the presence of their teacher, what students learned (and how they learned it) beyond the school setting cannot be accounted for. Likewise, the LLM students were not asked if or whether they engaged in additional self-study beyond the course materials, so it is not possible to estimate the extent to which (if at all) students engaged in such actions. In terms of RQ2 and RQ3, one of the chief limitations concerns the nature of the methods that were used. Although I asked the LLM students the same series of questions at three different time points throughout the study, this practice might be criticized in two regards. Firstly is that the awareness that students developed and/or exhibited may, in some cases, be due to the interview questions themselves. For instance, in drawing on SLA literature involving language testing, numerous studies have shown that the tests themselves may have a positive 135 influence on students’ learning outcomes (for a review, see Fulcher & Davidson, 2016). Thus, it is possible that the questions I posed during Interviews #1 or #2 subsequently influenced students’ awareness of specific phenomena, thereby causing them to pay more attention to certain aspects than they would have otherwise. Secondly is that students may have possessed greater awareness than they actually expressed. In terms of the interview questions, because the question-list was finite and limited to a one-hour session, certain aspects of students’ genre awareness simply may not have been captured. In a study by Negretti and McGrath (2018), the authors remarked that future studies exploring metacognition should not solely rely on self- reported interview data. Therefore, I heeded this advice and adopted a secondary source of data in the form of modified stimulated recalls with the students and their texts, which followed the completion of each office memo assignment. This too, though, has potential drawbacks. Gass and Mackey (2017) have suggested that stimulated recalls may be most effective when immediately following a task’s completion. However, due to the extended nature of the memo writing tasks themselves combined with the busy schedules of the LLM participants, logistically, this was not an option. Therefore, the modified stimulated recalls occurred later on, and all were completed within 24 hours of students finishing their respective memos. As the data in the current study show, the LLMs clearly were able to remember some of what they were thinking while writing. However, this extended time-delay may have caused some participants to forget certain relevant aspects of their thought processes. Finally, as mentioned, the investigation of RQ3 was exploratory in nature. Some of the criticisms may again be related to the methods used to address the question. For one, inter-coder reliability was sought for the purposes of quantifying the number of comments that students produced. In this coding scheme, it is possible that a student made one declarative comment 136 regarding the office memo’s audience in the beginning of the interview, yet the student made a very similar comment again near the end of the interview. This type of comment was counted twice in coding. This decision to count any/all comments (even if occurring in the same interview) was made because obtaining reliability on the general occurrences of the six- metacognition categories (i.e., declarative, procedural, conditional, etc.) could reliably be assessed by two coders. However, attempting to have two coders further distinguish between subcategories and to make decisions about what should or should not be counted (due to its presence earlier in the interview) could not reliably be assessed. Likewise, although inter-coder reliability was sought for the students’ comments, inter- rater reliability was not sought for Meyer’s own holistic rankings of the memos. This may raise questions regarding the objectiveness of Meyer’s rankings and as to whether or not they are representative of other legal readers or merely idiosyncratic to Meyer alone. Multiple surveys have been developed specifically for assessing metacognitive strategies in areas such as L1 reading (e.g., Schraw & Dennison, 1994) and L2 writing (e.g., Qin & Zhang, 2019), yet no such instruments or guidelines currently exist for assessing metacognitive genre awareness and/or connecting that awareness to writing performance. Thus, the reasons these qualitative methods were adopted in the current study for RQ3 are due to multiple factors, including: (1) the small sample size that was available for longitudinal (and qualitative) research; (2) to conduct an exploratory investigation into the types of comments that students would produce when discussing their metacognitive genre awareness; and finally, (3) to examine the extent to which those comments and themes were present (or not) in the evaluative comments of the primary reader/gatekeeper of the target discourse community. Although a larger scale study with an established questionnaire and multiple raters might have yielded interesting insights for RQ3, 137 this, again, was not a possibility in the current study. I return to this issue once more in the next chapter when discussing future research directions. 138 CHAPTER 5: CONCLUSION As discussed in the previous chapter, although the current study is not without its limitations, there are still many important insights that can be obtained from the findings. Therefore, in this final chapter, I provide the reader with an overview of these implications, and I discuss them in reference to their potential impact on current SLA theory, pedagogy, and future research directions. Implications and Future Research Directions Interdisciplinary engagement and research are necessary for understanding the scope of genre pedagogies. Despite the abundance of research into the classroom practices of L2 writing instructors, as mentioned, most of these studies have occurred within the classrooms of ESL/EFL teachers who are adopting specific ESP or SFL approaches. In response to scholars’ calls for more investigations into those practices of educators in different instructional contexts (e.g., Leki et al., 2008; Yigitoglu & Reichelt, 2014), the current study highlights the importance of such investigations. Looking firstly at the instructor’s pedagogical practices as a whole, like other studies within the domain of L2 writing (e.g., Lee, 2009), the classroom-based ethnographic investigation into Meyer’s pedagogy showed how teachers’ beliefs can have a powerful influence on their actions. In some instances, there was a strong connection between beliefs and practices. In other instances though, those beliefs and practices did not necessarily align. This further illustrates the importance of teachers engaging in reflective practices. As Farrell and Ives (2015) have noted, by engaging in reflective practices on one’s own teaching, this can help educators to not only understand the (dis)connections between their beliefs and actions, but also, this can help them to further understand how their actions may subsequently impact their 139 learners’ experiences in the classroom. Also related to Meyer’s pedagogy, of particular note is that apart from self-study in the form of reading various books, academic law journals, and attending legal writing conferences, Meyer stated that he had no formal teacher training and no training in working with L2 learners. Many times, as well, Meyer attributed his practices involving L2 writing pedagogy to his personal experiences rather than from (self-)education. Interestingly though, many of the practices and techniques that Meyer adopted in his teaching would also be considered ‘good practice’ by those educators and trainers involved in language pedagogy (e.g., Kasper, 2000). For example, Meyer edited students’ readings, promoted the use of graphic organizers to promote reading skills, and more. This, then, begs the question: How much formal training and education is actually necessary in order to develop best practices? Likewise, which of these best practices can one learn ‘naturally’ from hands-on experience, and which of these practices may not develop from experience? For instance, Meyer stated that from his experience, early on, he learned that he should edit LLMs’ readings; however, even after nearly two decades of teaching, he still felt uncomfortable addressing language-related issues with his L2 learners. This may be a potential area of interest for future researchers, as longitudinal approaches could be adopted for shedding light on content or CBI instructors’ practices, and, for tracing the evolution and emergence of their pedagogies over time. In future studies, researchers may wish to follow a group(s) of CBI instructors in different contexts over a period of multiple years, documenting the emergence and development of different techniques/approaches, and importantly, what spurred those techniques and how they came to be realized—(i.e., were they prompted by something the teacher saw in the classroom? Something the teacher read in students’ writing? Something the teacher learned 140 during self-study? Etc.). In terms of understanding these CBI teachers’ development over time, one potential analytical lens that holds untapped potential is that of identity. While there is a wealth of literature devoted to the topic of L2 teacher identity and how identity subsequently influences teachers’ actions in the classroom (see Norton & De Costa, 2018 for a review), there is limited scholarship that addresses the role of teacher identity specifically when it comes to L2 writing and genre pedagogy (e.g., Lee, 2013). Thus, this too, is an area of interest for future studies, as researchers might adopt identity for understanding the various connections between CBI instructors’ beliefs and practices, and how they evolve. Perhaps the most important implication arising from the examination of the legal writing instructor’s practices is related to the need for increased interdisciplinary engagement between L2 writing scholars and other fields. As discussed, the field of L2 writing itself is still somewhat in its infancy, as it only began to gain traction in the early 1990s (Matsuda & De Pew, 2002). Thus, as described, many of the investigations into L2 writing pedagogies have tended to be quite insular in nature in the sense that applied linguists (i.e., researchers) are often observing other applied linguists (i.e., as teachers)—and sometimes, are even teaching students who are studying within the domain of applied linguistics. Yet, the current study involving Meyer’s LLM classroom shows that interdisciplinary engagement may be highly beneficial for all parties involved. In terms of Meyer’s pedagogy, he often engaged in practices that many L2 writing instructors would not be familiar with, yet also might find quite intriguing. For instance, Meyer’s use of research logs (shown earlier in Figure 8), research conferences, and his subsequent ‘approved case list’ for writing the office memo genre might be of interest to some L2 writing instructors who are teaching L2 writing or in English for Academic Purposes contexts. Conversely, even though Meyer stated that one cannot “sound convincingly like a 141 lawyer” by simply mastering a set of English phrases, an ESL/EFL teacher, in turn, might wish to engage with Meyer about why this belief (though true) should not prohibit him from addressing language in his classroom, as doing so can help to build students’ reading skills (see Nation, 2009) and facilitate reading-writing connections (e.g., Harman, 2013). In the current study, even though the LLM students were advanced in their L2 English proficiency, it was clear that some of the students still needed additional language-based support. Likewise, during our interviews, some of the LLMs specifically expressed this, stating that they wished there was some focus on language use during the in-class sessions. In sum, these findings show that there is a need for even more investigations into those practices of educators in different instructional contexts, and by extension, a need for L2 writing scholars to engage in an interdisciplinary exchanging of ideas. The acquisition of genre knowledge is multifaceted and susceptible to individual differences. Metacognition theory has been adopted far less frequently within the domain of L2 writing research when compared to other mainstream theories such as Sociocultural Theory or Skill Acquisition Theory (see Polio & Kessler, 2019 for a review). The current study though, like those before it (e.g., Negretti, 2012; Negretti and Kuteeva, 2011; Yeh, 2015), showcased the theory’s potential for assisting SLA researchers in understanding longitudinal perspectives regarding learners’ developing genre awareness. Importantly as well, through the lens of metacognition theory, the findings of this dissertation illustrated the complex interplay of cognitive, social, and textual elements involved in the teaching and learning of written genres (Negretti, 2017). In terms of the connection between classroom pedagogy and students’ budding awareness, there were many instances within the current dataset in which students’ awareness of 142 genre-specific characteristics such as features of audience, organization, etc. clearly could be attributed to the professor’s teaching. In many ways, this is reflective of other studies within L2 writing literature as well (e.g., Harman, 2013; Tardy, 2015). Despite this, as mentioned, there were numerous instances in which the LLMs developed awareness that could not be identified as a product of the teacher’s instruction. Thus, future studies need to delve further into the complex processes of genre teaching and learning by extending their lenses to observe phenomena beyond the four walls of the classroom. In an effort to understand genre teaching and learning both in- and-out of the classroom context, future researchers may wish to adopt additional methods beyond those used in this study. For instance, although I adopted stimulated recalls with students’ texts, this could be enhanced through the addition of screen-recordings of students’ real-time writing. In Kessler (2020), the use of both screen recordings along with stimulated recalls led to particularly unique insights, especially pertaining to the myriad of digital tools, materials, and social connections that students drew upon to mediate their writing processes. The implementation of new and/or additional methods may provide L2 researchers with insights into the types of materials that students engage with during the actual writing process; likewise, this use of an additional stimulus may help facilitate discussions during the stimulated recalls themselves (see Gass & Mackey, 2017). Another important finding related to the students’ development of genre awareness is that when the six case study participants were compared, their awareness developed neither uniformly, nor at the same rate. Research by Tardy (2005, 2009) and Gao (2012) has also suggested that the acquisition of genre knowledge is multifaceted and gradually accumulated over time. Crucially, too, is that while some LLM students seemed to develop awareness rather quickly within one respective area of the metacognition model, other students did not (also 143 shown in Negretti & Kuteeva, 2011; Yeh, 2015). This finding holds particular significance for L2 writing instructors. Many times, teachers may think of their students as empty vessels that are to be filled with knowledge. This was reflected in Meyer’s belief that, as a teacher, “you are providing information from above.” Yet, as the LLMs showed, this is not really the case. When entering their respective classrooms, students oftentimes bring with them a wealth (or dearth) of knowledge, prior experiences, L1-based expectations, and a host of individual differences that can affect their acquisition of genre knowledge. Thus, teachers need to be cognizant of this fact, and they should attempt to activate their learners’ existing background knowledge early (and often) in attempts to help draw learners’ awareness to their preexisting knowledge, biases, and how such prior experiences may affect characteristics of their writing in the target L2 context (see Tardy, 2005, 2009 for a related discussion). For researchers as well, this realization that genre knowledge is highly complex and susceptible to individual differences (ID) is noteworthy. Within SLA, recently there has been a swell of interest involving the impact of IDs (i.e., learner-specific characteristics) on L2 learning. Metacognition, too, has been a target of interest, as ID researchers have examined the influence of learner motivation, feedback-seeking behaviors, and self-efficacy on metacognition (for a review see Sato, in press). However, most of these studies have tended to be quantitative in nature, and they have typically focused on the relationship between learners’ metacognitive strategies and performance. In the context of the current study, in some instances it is clear as to how preexisting knowledge and other experiences affected learners’ development of new knowledge. Yet, it still remains relatively unclear as to why some students developed exceedingly quickly in certain areas of metacognition while others did not. Therefore, future research may wish to tackle this issue through the adoption of approaches/methods that can 144 capture the cognitive, social, and textual elements involved in the learning of genres. Specifically, ethnographic perspectives may hold particular potential in this vein, as they may enable researchers to examine both the emergence and impact of certain IDs on students’ metacognitive genre awareness over time (see De Costa, Kessler, & Gajasinghe, in press). Greater performance may require the mingling of different types of metacognition. A large body of quantitatively-oriented research has examined the relationship between students’ metacognitive strategies and their task performance (see Hacker et al., 2009 and Sato, in press), yet no studies have examined the relationship between students’ metacognitive genre awareness and their perceived text quality. Thus, although RQ3 was largely exploratory in nature, some important insights may be gleaned. For one, it appeared that Meyer’s holistic rankings of students’ texts were unrelated to students’ prior experiences or their respective L2 proficiencies; however, there did appear to be a relationship between his rankings and students’ self-reported metacognition. Though no literature exists involving the intersections of genre awareness and performance from which to draw comparisons, this finding in the current study is tangentially consistent with L1- and L2-based literature reporting that a greater awareness of metacognitive strategies leads to better writing performance (e.g., Harris et al., 2010; Qin & Zhang, 2019). Importantly though, there did not appear to be a clear relationship between LLMs’ reported metacognitive knowledge (i.e., students’ general awareness) and the rankings they received. However, students’ self-reported metacognitive regulation (i.e., students’ abilities to discuss how they used their awareness in the writing task) was more closely related—and, even more so was that of students’ self-reported metacognition in general (i.e., knowledge + regulation). Thus, these findings appear to suggest that learners who are able to both discuss what they know and how they use it may fare better in producing texts of higher quality. If 145 future research in this domain holds true, in terms of pedagogy, this may suggest that teachers should work to activate learners’ existing background knowledge of the target genre before learners begin writing. This could be accomplished through the use of brainstorming sessions (if the writing will be completed in-class) or through the use of questionnaires/prompts (for out-of- class assignments). In this way, L2 writers can reflect not only on what they know (metacognitive knowledge) about the genre, but also how they will actively use that awareness in the upcoming writing task (metacognitive regulation) in planning, monitoring, and evaluating their own performance. Lastly, in terms of future research directions, if scholars choose to adopt similar qualitative methods for examining the development of L2 learners’ genre knowledge and its relationship to writing quality, researchers may wish to adopt methods such as concurrent think- aloud protocols in obtaining the instructor’s evaluations. In the current study, due to the lengthy nature of the writing assignment itself (memos were sometimes 10 pages in-length), I had to adopt retrospective interviews with Meyer to have him discuss why he prescribed such rankings on students’ memos. However, assuming the texts themselves are not as lengthy as those in the current study, concurrent think-aloud protocols may provide more in-depth insights into the genre-specific features that the readers/raters pay attention to when assessing students’ writing (for a further discussion of retrospective versus concurrent think-alouds, see van den Haak, De Jong, & Jan Schellens, 2003). Finally, although the current study was qualitative in scope, future researchers may wish to adopt quantitative methods to address the relationship between genre awareness and writing performance. Clearly, there is a rich tradition in this vein within both educational and SLA research. However, in order to do so effectively within this tradition, researchers must not only 146 obtain access to larger L2 participant pools and multiple raters, but more importantly, they also must develop and establish valid and reliable quantitative measurement tools (e.g., questionnaires) for assessing L2 learners’ metacognitive genre awareness, as none currently exist within the SLA literature (see Winke, 2014 for a related discussion on establishing measurement tools in SLA and using techniques such as structural equation modeling). 147 APPENDICIES 148 APPENDIX A Background Questionnaire (LLM Students) Male 1. First name: ___________________ 2. Last name: ___________________ 3. Gender: Female 4. Age: ___________________ 5. Email address: ___________________ 6. What is your home country? ___________________ 7. What is your first (native) language? ___________________ 8. If English is not your first language, how many years have you studied English? (If this Prefer not to disclose does not apply, write 'N/A'). ___________________ 9. Do you speak any additional languages? If so, list them here. (If not, write 'N/A'). ___________________ 10. What was your undergraduate major (your main focus of study)? ___________________ 11. Where did you receive your legal education? List the 1) university name, and 2) country name (e.g., Harvard University, USA). ___________________ 12. What type of legal system did you study during your legal education program? (e.g., common law, civil law, etc.) ___________________ 13. What types of legal documents did you read during your legal education program? (select all the apply) ___ Answers or complaints ___ Appellate briefs ___ Black letter law ___ Cases ___ Client letters or Opinion letters ___ Demand letters ___ Law review articles ___ Office memos ___ Statutes ___ Other 14. What types of legal documents did you create (or write) during your legal education program? (select all the apply) ___ Answers or complaints ___ Appellate briefs ___ Client letters or Opinion letters ___ Demand letters ___ Law review articles ___ Office memos ___ Other 15. After you finished your legal education program, did you: ___ Apply directly to this (Michigan) LLM program ___ Work professionally in a law-related career ___ Work professionally in a non-law related career ___ Other (please specify) 149 16. If you worked professionally in a law-related career… a. What was your job title? ___________________ b. How long did you work there? ___________________ c. What were your duties/responsibilities? ___________________ d. What types of legal documents did you read during your job? (select all the apply) ___ Answers or complaints ___ Appellate briefs ___ Black letter law ___ Cases ___ Client letters or Opinion letters ___ Demand letters ___ Law review articles ___ Office memos ___ Statutes ___ Other e. What types of legal documents did you create (or write) during your job? (select all the apply) ___ Answers or complaints ___ Appellate briefs ___ Client letters or Opinion letters ___ Demand letters ___ Law review articles ___ Office memos ___ Other 17. After you complete this LL.M. program, what are your plans? (Example: return to your home country to practice law; stay in the U.S. and work; etc.) ___________________ 150 APPENDIX B Semi-Structured Interview Questions (Professor Meyer) Interview #1 of 3 Questions Background and Education 1. How long have you taught legal writing? 2. After law school, did you practice law prior to becoming a teacher? 3. How did you become interested in teaching? 4. Have you had any formal teacher training? 5. Have you had any formal training specifically in teaching L2 learners? 6. How do you approach the teaching of legal writing? What are you trying to instill in your students (also, What do you personally think is important)? 7. Can you describe a typical day in your classroom? Legal Writing & The Office Memo 8. How would you describe legal writing in general? What are some of its characteristics? What is important? Thinking about writing an office memo now… 9. What are the basic elements of an office memo that students need to learn? 10. What characteristics make an effective office memo? 11. How would you describe your method of teaching the office memo? What’s your approach? 12. What is the biggest challenge in teaching the office memo? The Students 13. What is the biggest challenge that the students face in learning the office memo? (Is it the research? Is there one section that is most difficult?) 14. Thinking about your past students’ progress over the semester: In your experience, where do students have the most success in memo writing? 15. Thinking about students’ progress over the semester again: Where do they make the least amount of progress? 16. Is there anything else you think I should know about the LLM students based on your experiences as a teacher? Interview #2 of 3 Questions 1. Can you explain how you have chosen to structure the first half of the class (i.e., the first credit hour)? Explain your reasoning behind why you structured the class the way you did. a. Discuss why you selected these various topics over the course of the first few weeks. 2. Has this syllabus evolved over time, or has it pretty much stayed the same since you began teaching the class? 3. I noticed you often reference in class that you’ve somehow edited the 151 documents/readings students are given. Can you give me a few examples of something you’ve edited and how? 4. Based on your interactions with students in class, how do you feel about their comprehension of the readings? a. I.e., Do you feel there were any particular readings they struggled with? b. If so, how did you attempt to address this in class? 5. For the office memo: You had students do research on their own, and then, you gave them an "approved case list" to use in their actual memo. a. Can you explain why did you decide to structure it this way? Why not let students use their own research lists? b. (How do you think this helps students?) c. (Does this help you?) d. (What do you think would happen if you let students use their own?) 6. For office memo #1: Why did you select this specific topic? (i.e., the contract vs. tort claim in Pennsylvania) 7. For office memo #2: Why did you select this specific topic (i.e., contractual liability)? 8. For office memo #1: Let’s discuss the 6 focal students' memos individually: a. [Students’ names here] 9. For each memo: a. What was your general impression of this memo? b. What were the strengths? What were the weaknesses? 10. As a senior partner: Please rank the 6 memos in order in terms of effectiveness. a. Please explain why you gave the rankings you did. Interview #3 of 3 Questions Pedagogy 1. Can you think of any specific manuals or books that you have read that have informed how you structure or approach the teaching of your legal writing class? 2. In addition to teaching research and writing skills, would you say that part of the aim of the course is also to teach reading and/or literacy skills? a. (If yes): Can you describe your approach? b. In terms of reading, what do you think is important for students to know (i.e., do they need to learn specific skills for reading case law, etc.)? 3. I noticed that when you present a case brief to students in class, you often highlight different parts of the case on the projector. For example: you often highlight where the case location or jurisdiction (i.e., what court). a. Can you explain why you do this? b. What other parts do you try to point out to students in your teaching? 4. Visualization reflection: This will sound a bit odd, but here's what I want you to do: Think about your approach to teaching legal research and writing. Draw a picture that represents your approach to teaching the class. It can have words in it. It can be anything: for example, maybe a flowchart, a tree with branches, a pyramid, etc. a. Describe the picture. 5. For the office memo: You had students do research on their own, and then, you gave them an "approved case list" to use in their actual memo. 152 a. Can you explain why did you decide to structure it this way? Why not let students use their own research lists? b. How do you think this helps students? c. Does this help you? d. What do you think would happen if you let students use their own? Writing – Office Memo 6. What was your general impression of students’ performance on this memo compared to office memo #1? 7. For office memo #2: Let’s discuss the 6 focal students' memos individually: a. [Students’ names here] 8. For each memo: a. What was your general impression of this memo? b. What were the strengths? What were the weaknesses? 9. As a senior partner: Please rank the 6 memos in order in terms of effectiveness. a. Please explain why you gave the rankings you did. 10. Does any student(s) in particular stand out to you as having made the most progress? (If so, who?) 153 APPENDIX C Semi-Structured Interview and Stimulated Recall Questions (Students) Interview #1 Questions about the office memo. In this interview, I have some follow up questions I would like to ask you: In the initial online background questionnaire, I asked you if you know anything 1. What is the purpose of writing an office memo? Why would someone write one? [Declarative] 2. Who reads office memos (who is the audience)? [Declarative] 3. Do you think that the audience affects how you write the memo? [Conditional] a. If so, can you give me a specific example? 4. Tell me about the structure/organization of your office memo. Is there a general structure? [Declarative] a. How would you go about writing/organizing an office memo? [Procedural] 5. Do you think the readers who read an office memo have any expectations? [Declarative] a. If so, what are those expectations? 6. Can you tell me anything about the language that is used in an office memo? Describe it. [Declarative] a. Does it ever change when you are writing, or does it always stay the same? [Conditional] 7. If you have written an office memo before: a. What is the easiest part of writing an office memo for you? The most difficult? [Open: Could be Declarative/Procedural/Conditional] Interview/Stimulated Recall #2 and #3 Questions Can you open your 2nd office memo assignment on your computer? I’ll ask you to refer to it during our conversation. 1. What is the purpose of writing an office memo? Why would someone write one? [Declarative] 2. Who reads office memos (who is the audience)? [Declarative] 3. Do you think that the audience affects how you write the memo? [Conditional] a. If so, can you give me a specific example? b. Was there a time you thought about the audience while writing and you changed or added something because you were thinking about them? [Monitoring] i. If so, can you give me an example from your memo? ii. What if the audience changed? Would that change your memo at all? [Conditional] 4. Tell me about the structure/organization of your office memo. Is there a general structure you followed in writing yours? [Declarative] a. Did you use/change your office memo structure from the example Professor Meyer provided you? (Why/why not)? [Procedural/Conditional] 154 5. Professor Meyer gave you specific cases to use for this office memo assignment. I want you to tell me about using them in your writing, specifically: a. How did the cases Professor Meyer selected compare to those you found in your research? [Procedural/Evaluating] i. Was there a lot of overlap? Was anything different? If so, what? [Evaluating] b. Since the beginning of the class, did you learn anything new about researching or integrating cases into your writing? [Open: Could be Declarative/Procedural/Conditional] i. If so, can you give me an example from your office memo? 6. Take me back to before you started writing your office memo. Did you do any planning before writing? [Planning] a. If so, what did you do? 7. Do you think the readers who read an office memo have any expectations? [Declarative] a. If so, what are those expectations? b. In your memo, do you think you met those expectations? Why/why not? [Evaluating] 8. Can you tell me anything about the language that is used in an office memo? Describe it. [Declarative] a. Does it ever change when you are writing, or does it always stay the same? [Conditional] b. If it changes, can you give me an example from your office memo? [Conditional/Monitoring] 9. What was the easiest part of writing the office memo for you? The most difficult? [Open: Could be Declarative/Procedural/Conditional] 10. Last question: Reflect on the beginning of the class to now: What new information have you learned about creating an office memo? It can be anything. [Open: Could be Declarative/Procedural/Conditional] 155 APPENDIX D Coder Training Sheet Background: You will be provided with transcripts from two interviews that occurred between me (Matt Kessler) and two research participants. In the interviews, the participants discuss their experiences involving a class they took on legal research and writing. Specifically, they discuss writing a type of legal document called an office memo. The office memo is typically assigned by a more senior attorney as a task for a junior attorney. The office memo’s purpose is for a lawyer in a firm to analyze a client’s case in relation to established law. The office memo is predictive in the sense that the writer’s job is to try to predict how the court will rule if the client’s case is brought to court (i.e., Will they win? Will they lose? Should the law firm even take the case? How should the law firm advise the client? Etc.). Directions: Read the 2 interview transcripts and code them in relation to the students expressing genre awareness of the office memo (e.g., the student is expressing knowledge of the office memo genre such as the intended audience, the purpose of the office memo, the organization, the specific language that should be used, etc.). In order to code instances of genre awareness, you will assign 1 of 6 codes based on the following framework: Metacognitive knowledge Metacognitive regulation 1. Declarative – WHAT the learner says 4. Planning – The learner discusses some she knows aspect of planning for the writing task 2. Procedural – HOW the learner says she 5. Monitoring – The learner says she does something (Think: “This is how I do it.”) 3. Conditional – IF a condition changes, the learner describes how/why she would change something is/was consciously thinking about something specific while writing 6. Evaluating – The learner critically reflects on some aspect of her own writing Examples: 1. Declarative: -“Initially I really had no idea what the purpose is of an office memo. But I think the basic purpose is to give an impartial and good opinion or prediction of what are the chances of the client to win.” 2. Procedural: -“[In an office memo], I would never write like ‘in my opinion’ or ‘I think that’.” -“I just did it how I would do it in Germany, I think.” 156 3. Conditional: -“If the audience was the clients, you would have to write it a little bit more from a layman’s perspective—to explain something small. A senior attorney would know the basic concepts.” 4. Planning: -“I started with basically outlining it on a piece of paper. Then I had a basic structure of it for a discussion section, how I would want to discuss state courts or federal courts and which cases…” 5. Monitoring: -“When I was writing, I had Professor XXX in mind…” 6. Evaluating: -“I pretty much interpreted everything most favorably from the client’s perspective, if I think about it. Maybe that’s something Professor XXX will tell me about later—that I’m not partial enough.” 157 REFERENCES 158 REFERENCES American Bar Association. (2019). 2019 standard 509 information report data overview. Retrieved from https://www.americanbar.org/content/dam/aba/administrative/ legal_education_and_admissions_to_the_bar/statistics/2019-509-enrollment-summary- report-final.pdf Anderson, J. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press. Baffy, M. (2017). Shifting frames to construct a legal English class. Journal of English for Academic Purposes, 25, 58-70. Bawarshi, A. S., & Reiff, M. J. (2010). Genre: An introduction to history, theory, research, and pedagogy. West Lafayette, IN: Parlor Press. Bazerman, C. (1997). The life of genre, the life in the classroom. In W. Bishop & H. Ostrom (Eds.), Genre and writing: Issues, arguments, and alternatives (pp. 19-26). Portsmouth, NJ: Boynton/Cook. Bazerman, C. (1988). Shaping written knowledge: The genre and activity of the experimental article in science. Madison, WI: University of Wisconsin Press. Bhatia, V. K. (1993). Analysing genre: Language use in professional settings. New York, NY: Longman. Biber, D., Connor, U., & Upton, T. A. (Eds.). (2007). Discourse on the move: Using corpus analysis to describe discourse structure. Amsterdam: John Benjamins. Bikowski, D., & Vithanage, R. (2016). Effects of web-based collaborative writing on individual L2 writing development. Language Learning & Technology, 20(1), 79-99. Bloome, D. (2012). Classroom ethnography. In M. Grenfell, D. Bloome, C. Hardy, K. Pahl, J. Roswell, & B. Street (eds.), Language, ethnography, and education: Bridging New Literacy Studies and Bourdieu (pp. 8–26). New York, NY: Routledge. Breeze, R. (2013). Lexical bundles across four legal genres. International Journal of Corpus Linguistics, 18(2), 229-253. Brooks, L., & Swain, M. (2009). Languaging in a collaborative setting. In A. Mackey & C. Polio (Eds.), Multiple perspectives on interaction: Second language research in honor of Susan M. Gass (pp. 58-89). New York: Taylor & Francis/Routledge. Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of 159 metacognition. In R. Glaser (Ed.), Advances in instructional psychology (pp. 77-165). Hillsdale: Erlbaum. Bui, G., & Kong, A. (2019). Metacognitive instruction for peer review interaction in L2 writing. Journal of Writing Research, 11(2), 357-392. Bulté, B., & Housen, A. (2014). Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing, 26, 42-65. https://doi.org/10.1016/j.jslw.2014.09.005 Cammarata, L. (2009). Negotiating curricular transitions: Foreign language teachers’ learning experience with content-based instruction. The Canadian Modern Language Review, 65(4), 559-585. https://doi.org/10.3138/cmlr.65.4.559 Caplan, N. A., & Farling, M. (2016). A dozen heads are better than one: Collaborative writing in genre-based pedagogy. TESOL Journal, 8(3), 564-581. https://doi.org/10.1002/tesj.287 Carrell, P. L. (1989). Metacognitive awareness and second language reading. The Modern Language Journal, 73(2), 121-134. https://doi.org/10.1111/j.1540-4781.1989.tb02534.x Chew, A. Z., & Pryal, K. R. G. (2016). The complete legal writer. Durham, N.C.: Carolina Academic Press. Chew, A., & Pryal, K. R. G. (2015). Bridging the gap between law school and law practice (January 1, 2015). http://dx.doi.org/10.2139/ssrn.2575185 Copland, F. (2018). Observation and fieldnotes. In A. Phakiti, P.I. De Costa, L. Plonsky & S. Starfield (Eds.), The Palgrave handbook of applied linguistics research (pp. 249-268). London: Palgrave Macmillan. Crossley, S. A., Kyle, K., McNamara, D. S. (2016). The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 32, 1-16. https://doi.org/10.1016/j.jslw.2016.01.003 Dalton-Puffer, C., Nikula, T., & Smit, U. (2010). Language use and language learning in CLIL classrooms. Amsterdam, The Netherlands: John Benjamins. De Costa, P. I. (2015). Ethics in applied linguistics research. In B. Paltridge & A. Phakiti (Eds.), Research methods in applied linguistics: A practical resource (pp. 245-257). London, England: Bloomsbury. De Costa, P. I., Kessler, M., & Gajasinghe, K. (In press). Ethnography in individual differences research. In S. Li, P. Hiver, & M. Papi (Eds.), The Routledge handbook of second language acquisition and individual differences. New York, NY: Routledge. Delago, R. (1986). How to write a law review article. University of San Francisco Law Review, 160 20, 445-454. Devitt, A., Reiff, M. J., & Bawarshi, A. (2004). Scenes of writing: Strategies for composing with genres. London, U.K.: Longman. Dignath, C., & Buttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3, 231–264. https://doi.org/10.1007/s11409-008-9029-x Duff, P. A. (2008). Case study research in applied linguistics. New York, NY: Routledge. Duff, P. A. (2007). Qualitative approaches to classroom research with English language learners. In J. Cummings & C. Davison (Eds.), International handbook of English language teaching (pp. 973-986). Boston, MA: Springer. Edwards, L. H. (2006). Legal writing: Process, analysis, and organization (4th ed.). New York, NY: Aspen Publishers. Engeström, Y. (1987). Learning by expanding: An Activity Theoretical approach to develop- mental research. Helsinki, Finland: Orienta-Konsultit Oy. Engeström, Y. (1999). Activity theory and individual and social transformation. In Y. Engeström, R. Miettinen, & R.-L. Punamäki (Eds.), Perspective on Activity Theory: Learning in doing: Social, cognitive and computational perspectives (pp. 19-38). New York, NY: Cambridge University Press. Fanego, T., & Rodríguez-Puente, P. (2018). Corpus-based research on variation in English legal discourse. Philadelphia, PA: John Benjamins. Farrell, T. S. C., & Ives, J. (2015). Exploring teacher beliefs and classroom practices through reflective practice: A case study. Language Teaching Research, 19(5), 594-610. Fitzsimmons-Doolan, S., & Stoller, F. L. (2018). Issues in syllabus design: Content-based syllabi. In A. Faravani (Ed.), Issues in syllabus design (pp. 111-124). Sense Publishers. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new era of cognitive- developmental inquiry. American Psychologist, 34(10), 906–11. Flowerdew, L. (2016). A genre-inspired and lexicogrammatical approach for helping postgraduate students craft grant proposals. English for Specific Purposes, 42, 1-12. Fulcher, D., & Davidson, F. (2016). The Routledge handbook of language testing (1st ed.). New York, NY: Routledge. Gao, L. (2012). Investigating ESL graduate students’ intercultural experiences of academic English writing: A first-person natation of a streamlined qualitative study process. The 161 Qualitative Report, 17(24), 1-25. Garofalo, J., & Lester, F. K. (1985). Metacognition, cognitive monitoring, and mathematical performance. Journal of Research in Mathematics Education, 16(3), 163-176. Gass, S. M., & Mackey, A. (2017). Stimulated recall methodology in applied linguistics and L2 research. Routledge: New York, NY. Geertz, C. (1973). The interpretation of cultures: Selected essays. New York, NY: Basic Books. Hacker, D. J., Keener, M. C., & Kircher, J. C. (2009). Writing is applied metacognition. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of metacognition in education (pp. 154-172). New York, NY: Routledge. Hafner, C. A. (2010). A multi-perspective genre analysis of the barrister’s opinion: Writing context, generic structure, and textualization. Written Communication, 27, 410-441. Halasek, K. (1999). A pedagogy of possibility: Bakhtinian perspectives on composition studies. Carbondale, IL: Southern Illinois University Press. Halliday, M. A. K. (1996). Literacy and linguistics: A functional perspective. In R. Hasan & G. Williams (Eds.), Literacy in society (pp. 339-376). London, Longman. Halliday, M. A. K. (1994). An introduction to functional grammar. London, U.K.: Edward Arnold. Halliday, M. A. K. (1978). Language as social semiotic. London, U.K.: Edward Arnold. Halliday, M.A.K., & Matthiessen, C. M. I. M. (2004). An introduction to functional grammar. New York, NY: Routledge. Hanzawa, K. (2018). The development of voice onset time (VOT) in a content-based instruction university program by Japanese learners of English: A longitudinal study. The Canadian Modern Language Review, 74(4), 502-522. https://doi.org/10.3138/cmlr.2018-0196 Harman, R. (2013). Literary intertextuality in genre-based pedagogies: Building lexical cohesion in fifth-grade L2 writing. Journal of Second Language Writing, 22, 125-140. http://dx.doi.org/10.1016/j.jslw.2013.03.006 Harris, K. R., Santangelo, T., & Graham, S. (2010). Metacognition and strategies instruction in writing. In H. S. Walters & W. Schneider (Eds.), Metacognition, strategy use, and instruction (pp. 226-256). New York, NY: Guilford Publications. Hartig, A. J., & Lu, X. (2014). Plain English and legal writing: Comparing expert and novice writers. English for Specific Purposes, 33, 87-96. 162 Hultin, E., & Westman, M. (2018). The reuse of semiotic resources in third-year children’s writing of sub-genres. Journal of Early Childhood Literacy, 18(4), 518-544. Hyland, K. (2012). Bundles in academic discourse. Annual Review of Applied Linguistics, 32, 150-169. Hyland, K. (2007). Genre pedagogy: Language, literacy and L2 writing instruction. Journal of Second Language Writing, 16, 148–164. Johns, A. M. (2008). Genre awareness for the novice student: An ongoing quest. Language Teaching, 41, 237-252. Johns, A. M. (2002). Genre in the classroom: Multiple perspectives. Mahwah, NJ: Erlbaum. Johns, A. M., Bawarshi, A., Coe, R. M., Hyland, K., Paltridge, B., Reiff, M. J., & Tardy, C. M. (2006). Crossing the boundaries of genre studies: Commentaries by experts. Journal of Second Language Writing, 15, 234-249. Kasper, L. F. (2000). Content-based college ESL instruction: Theoretical foundations and pedagogical applications. In L. F. Kasper (Ed.), Content-based college ESL instruction (pp. 3-25). Mahwah, NJ: Lawrence Erlbaum. Kessler, M. (In press). A text analysis and gatekeepers’ perspectives of a promotional genre: Understanding the rhetoric of Fulbright grant statements. English for Specific Purposes. Kessler, M. (2020). Technology-mediated writing: Exploring incoming graduate students’ L2 writing strategies with Activity Theory. Computers and Composition, 55. https://doi.org/10.1016/j.compcom.2020.102542 Kessler, M., Solheim, I., & Zhao, M. (2020). Can task-based language teaching be “authentic” in foreign language contexts?: Exploring the case of China. TESOL Journal. King, A. (1991). Effects of training in strategic questioning on children’s problem-solving performance. Journal of Educational Psychology, 83, 307–317. Kuteeva, M., & McGrath, L. (2013). The theoretical research article as a reflection of disciplinary practices: The case of pure mathematics. Applied Linguistics, 36(2), 215-235. Krashen, S. (1985). Input hypothesis: Issues and implications. New York, NY: Longman. Lee, I. (2013). Becoming a writing teacher: Using “identity” as an analytic lens to understand EFL writing teachers’ development. Journal of Second Language Writing, 22(3), 330- 345. https://doi.org/10.1016/j.jslw.2012.07.001 Lee, I. (2009). Ten mismatches between teachers’ beliefs and written feedback practices. ELT Journal, 63(1), 13-22. https://doi.org/10.1093/elt/ccn010 163 Leki, I., Cumming, A. H., & Silva, T. (2008). A synthesis of research on second language writing in English. Kentucky: Routledge. Li, W. (2013). A study of metacognitive awareness of non-English majors in L2 listening. Journal of Language Teaching and Research, 4(3), 504-510. Doi:10.4304/jltr.4.3.504- 510 Long, M. H. (1996). The role of linguistic environment in the second language acquisition. In W. Ritchie & T. Bhatia (Eds.), Handbook of Second Language Acquisition (pp. 413-468). San Diego, CA: Academic Press. Lowe, R., & Ploetzner, R. (2017). Learning from dynamic visualization: Innovations in research and application. Boston, MA: Springer. Lu, X., Yoon, J., & Kisselev, O. (2018). A phrase-frame list for social science research article introductions. Journal of English for Academic Purposes, 36, 76-85. Lyster, R. (2017). Content-based language teaching. In S. Loewen & M. Sato (Eds.), The Routledge handbook of instructed second language acquisition (pp. 87-107). New York, NY: Routledge. Macken-Horarik, M. (2002). ‘Something to shoot for’: A Systemic Functional approach to teaching genre in secondary school. In A. Johns (Ed.), Genre in the classroom: Multiple perspectives (pp. 17–42). Mahwah, NJ: Lawrence Erlbaum. Mahboob, A., Dreyfus, S., Humphrey, S., & Martin, J. R. (2010). Appliable linguistics and English language education teaching: The scaffolding and literacy in adult tertiary environments (SLTE) project. In A. Mahboob, & N. Knight (Eds.), Appliable linguistics: Texts, contexts, and meanings (pp. 185-199). London: Continuum. Martin, J. R. (1984). Language, register, and genre. In F. Christie (Ed.), Language studies: Children’s writing: Reader (pp. 21-30). Geelong: Deakin University Press. Matsuda, P. K., & De Pew, E. K. (2002). Early second language writing: An introduction. Journal of Second Language Writing, 11(4), 261-268. Martin, J. R., & Rose, D. (2008). Genre relations: Mapping culture. London, U.K.: Equinox. McGregor, D. B., & Adams, C. M. (2008). The international lawyer's guide to legal analysis and communication in the United States. New York, NY: Aspen Publishers. Miles, M. B., Huberman, A. M., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage. Miller, C. R. (1984). Genre as social action. Quarterly Journal of Speech, 70, 151-167. 164 Myskow, G., & Grdon, K. (2010). A focus on purpose: Using a genre approach in an EFL writing class. ELT Journal, 64(3), 283-292. Doi:10.1093/elt/ccp057 Nagao, A. (2019). The SFL genre-based approach to writing in EFL contexts. Asian-Pacific Journal of Second and Foreign Language Education, 4(6), 1-18. https://doi.org/10.1186/s40862-019-0069-3 Nation, I. S. P. (2009). Teaching ESL/EFL reading and writing. New York, NY: Routledge. Naug, H. L., Colson, N. J., & Donner, D. G. (2011). Promoting metacognition in first year anatomy laboratories using plasticine modeling and drawing activities: A pilot study of the “blank page” technique. Anatomical Science Education, 4(4), 231-234. https://doi.org/10.1002/ase.228 Norton, B., & De Costa, P. I. (2018). Research tasks on identity and language education. Language Teaching, 51(1), 90-112. https://doi.org/10.1017/S0261444817000325 Negretti, R. (2017). Calibrating genre: Metacognitive judgments and rhetorical effectiveness in academic writing by L2 graduate students. Applied Linguistics, 38(4), 512-539. doi:10.1093/applin/amv051 Negretti, R. (2012). Metacognition in student academic writing: A longitudinal study of metacognitive awareness and its relation to task perception and evaluation of performance. Written Communication, 29(2), 142–179. Negretti, R., & Kuteeva, M. (2011). Fostering metacognitive genre awareness in L2 academic reading and writing: A case study of pre-service English teachers. Journal of Second Language Writing, 20, 95–110. Negretti, R., & McGrath, L. (2018). Scaffolding genre knowledge and metacognition: Insights from an L2 doctoral research writing course. Journal of Second Language Writing, 40, 12-31. Oates, L. C., & Enquist, A. (2011). Just memos: Teacher's manual (3rd ed.). New York, NY: Wolters Kluwer Law & Business. Oates, L. C., Enquist, A., & Francis, J. (2018). The legal writing handbook: Analysis, research, and writing (7th ed.). New York, NY: Wolters Kluwer. de Oliveira, L. C., & Lan, S. W. (2014). Writing science in an upper elementary classroom: A genre-based approach to teaching English language learners. Journal of Second Language Writing, 25, 23-39. Ong, J. (2014). How do planning time and task conditions affect metacognitive processes of L2 writers? Journal of Second Language Writing, 23, 17-30. 165 Ortega, L. (2012). Epilogue: Exploring L2 writing-SLA interfaces. Journal of Second Language Writing, 21(4), 404-415. Paltridge, B. (2014). Genre and second-language academic writing. Language Teaching, 47(3), 303-318. Paltridge, B., & Phakiti, A. (2015). Research methods in Applied Linguistics: A practical resource. New York, NY: Bloomsbury Publishing. Piccard, A. M. (2017). U.S. legal writing for international lawyers and law students. Saint Paul, MN: West Academic Publishing. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385-407. Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory Into Practice, 41(4), 219-225. Polio, C. (2017). Thinking allowed: Second language writing development: A research agenda. Language Teaching, 50(2), 261-275. Polio, C., & Friedman, D. A. (2017). Understanding, evaluating, and conducting second language writing research. New York, NY: Routledge. Polio, C., & Kessler, M. (2019). Teaching L2 writing: Connecting SLA theory, research, and pedagogy. In N. Polat, P. MacIntyre, & T. Gregersen (Eds.), Research-driven pedagogy: Implications of L2A theory and research for the teaching of language skills. New York, NY: Routledge. Qin, L., & Zhang, L. J. (2019). English as a foreign language writers’ metacognitive strategy knowledge of writing and their writing performance in multimedia environments. Journal of Writing Research, 12(2), 393-413. doi: 10.17239/jowr-2019.11.02.06 Racelis, J. V., & Matsuda, P. K. (2013). Integrating process and genre into the second language writing classroom: Research into practice. Language Teaching, 46(3), 382-393. Rothery, J., & Stenglin, M. (1994). Spine-chilling stories: A unit of work for junior secondary English. Sydney: Metropolitan East Disadvantaged Schools Program. Russell, D. R. (2001). Where do naturalistic studies of WAC/WID point? A research review. In S. H. McLead , E. Miragilia, M. Soven, & C. Thaiss (Eds.), WAC for the new millennium: Strategies for continuing writing across the curriculum programs. Urbana, IL: NCTE. Russell, D. (1997). Rethinking genre in school and society: An activity theory analysis. Written Communication, 14, 504-554. 166 Samraj, B. & Monk, L. (2008). The statement of purpose in graduate program applications: Genre structure and disciplinary variation. English for Specific Purposes, 27, 193-211. Sato, M. (In press). Metacognition and metacognitive instruction. In S. Li, P. Hiver, & M. Papi (Eds.), The Routledge handbook of second language acquisition and individual differences. New York, NY: Routledge. Sato, M., & Loewen, S. (2018). Metacognitive instruction enhances the effectiveness of corrective feedback: Variable effects of feedback types and linguistic targets. Language Learning, 68(2), 507-545. DOI: 10.1111/lang.12283 Sarangi, S., & Candlin, C. N. (2003). Trading between reflexivity and relevance: New challenges for applied linguistics. Applied Linguistics, 24(3), 271-285. Schoonen, R., Van Gelderen, A., Stoel, R. D., Hulstijin, J., & De Glopper, K. (2011). Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, 61(1), 31–79. Schoonen, R., Van Gelderen, A., De Glopper, K., Hulstijin, J., Snellings, P., Simis, A., & Stevenson, M. (2002). Linguistic knowledge, metacognitive knowledge and retrieval speed in L1, L2 and EFL writing: A structural equation modeling approach. In S. Ransdell & M.-L. Barbier (Eds.), New directions for research in L2 writing (pp. 101- 122). Boston, MA: Springer. Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460-475. Stieff, M. (2017). Drawing for promoting learning and engagement with dynamic visualizations. In: R. Lowe & R. Ploetzner (Eds.), Learning from dynamic visualization: Innovations in research and application (pp. 333-356). Boston, MA: Springer. Stoller, F. L., & Grabe, W. (1997). The six t’s approach to content-based instruction. In M. A. Snow & D. M. Brinton (Eds.), The content-based classroom: Perspectives on integrating language and content (pp. 78-94). New York, NY: Longman. Storch, N. (2009). The impact of studying in a second language (L2) medium university on the development of L2 writing. Journal of Second Language Writing, 18(2), 103-118. https://doi.org/10.1016/j.jslw.2009.02.003 Swales, J. M. (2004). Research genres: Exploration and applications. Cambridge: Cambridge University Press. Swales, J. M. (1990). Genre analysis: English in academic and research settings. Cambridge: Cambridge University Press. 167 Swales, J. M. (1981). Aspects of article introductions. Aston ESP Research Reports, No. 1 Language Studies Unit, The University of Aston at Birmingham. Republished by University of Michigan Press 2011. Swales, J. M., & Feak, C. B. (2012). Academic writing for graduate students (3rd ed.). Ann Arbor, MI: University of Michigan Press. Swales, J., M., & Feak, C. B. (2000). English in today’s research world: A writing guide. Ann Arbor, MI: University of Michigan Press. Taki, S. (2015). Metacognitive online reading strategy use: Readers’ perceptions in L1 and L2. Journal of Research in Reading, 39(4), 409-427. https://doi.org/10.1111/1467- 9817.12048 Tardy, C. M. (2016). Beyond convention: Genre innovation in academic writing. Ann Arbor, MI: University of Michigan Press. Tardy, C. M. (2015). Bending genres (or, when is a deviation an innovation?). In N. Atremeva & A. Freedman (Eds.), Trends and traditions in genre studies. Edmonton, CA: Inkshed Publications. Tardy, C. M. (2011a). The history and future of genre in second language writing: A North American perspective. Journal of Second Language Writing, 20, 1–5. Tardy, C. M. (2011b). ESP and multi-method approaches to genre analysis. In D. Belcher, A.M. Johns, & B. Paltridge (Eds.), New directions in English for specific purposes research (pp. 145-173). Ann Arbor, MI: University of Michigan Press. Tardy, C. M. (2009). Building genre knowledge. West Lafayette, IN: Parlor Press. Tardy, C. M. (2005). ‘It’s like a story’: Rhetorical knowledge development in advanced academic literacy. Journal of English for Academic Purposes, 4(4), 325-338. Tedick, D. J., & Wesley, P. A. (2015). A review of research on content-based foreign/second language education in US K-12 contexts. Language, Culture and Curriculum, 28(1), 25- 40. DOI: 10.1080/07908318.2014.1000923 The New York State Board of Law Examiners. (n.d.). Rules of court / BOLE. https://www.nybarexam.org/Rules/Rules.htm#520.6 Thomas, W., Collier, V., & Collier, K. (2011). English learners in North Carolina, 2010. Raleigh, NC: North Carolina Department of Public Instruction. Retrieved from http://esl.ncwiseowl.org/resources/dual_language/ Thornton, J. B. (2014). U.S. legal reasoning, writing, and practice for international lawyers: Teacher's manual. New Providence, NJ: LexisNexus. 168 Tománková, V. (2016). Lexical bundles in legal texts corpora: Selection, classification and pedagogical implications. Discourse and Interaction, 9(2), 75-94. Tribble, C, & Wingate, U. (2013). From text to corpus—A genre-based approach to academic literacy instruction. System, 41(2), 307-321. van de Kamp, M. T., Admiraal, W., Drie, J., & Rijlaarsdam, G. (2015). Enhancing divergent thinking in visual arts education: Effects of explicit instruction of metacognition. British Journal of Educational Psychology, 85(1), 47–58. van den Haak, M., De Jong, M., & Jan Schellens, P. (2003). Retrospective vs. concurrent think- aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 5, 339-351. Vandergrift, L. (2006). ‘It was nice to see that our predictions were right’: Developing metacognition in L2 listening comprehension. The Canadian Modern Language Review, 58(4), 555-575. https://doi.org/10.3138/cmlr.58.4.555 Vandergrift, L. (2005). Relationships among motivation orientations, metacognitive awareness and proficiency in L2 listening. Applied Linguistics, 26(1), 70-89. https://doi.org/10.1093/applin/amh039 VERBI Software. (2019). MAXQDA 2020 [computer software]. Berlin, Germany: VERBI Software. Available from maxqda.com Vygotsky, L. S. (1978) Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Wang, J.-R., & Chen, S.-F. (2013). Exploring mediating effect of metacognitive awareness on comprehension of science texts through structural equation modeling analysis. Journal of Research in Science Teaching, 51(2), 175-191. Wargo, J. M., & De Costa, P. I. (2017). Tracing academic literacies across contemporary literacy sponsorscapes: Mobilities, ideologies, identities, and technologies. London Review of Education, 15(1), 101-114. DOI: https://doi.org/10.18546/LRE.15.1.09 Wenden, A. L. (1998). Metacognitive strategies in L2 writing: A case for task knowledge. In J. E. Alatis (Ed.), Georgetown University roundtable on languages and linguistics 1991: Linguistics and language pedagogy: The state of the art (pp. 302-322). Washington, DC: Georgetown University Press. Woolley, G. (2011). Reading comprehension: Assisting children with learning difficulties. The Netherlands: Springer. 169 Weber, S., & Mitchell, C. (1996). Drawing ourselves into teaching: Studying the images that shape and distort teacher education. Teaching and Teacher Education, 12(3), 303-313. https://doi.org/10.1016/0742-051X(95)00040-Q Wei, X. (2020). Assessing the metacognitive awareness relevant to L1-to-L2 rhetorical transfer in L2 writing: The cases of Chinese EFL writers across proficiency levels. Assessing Writing, (Early View). https://doi.org/10.1016/j.asw.2020.100452 Weresh, M. H. (2009). Legal writing: Ethical and professional considerations: Teacher's manual (2nd ed.). Newark, NJ: Matthew Bender & Company. Winke, P. (2014). Testing hypotheses about language learning using Structural Equation Modeling. Annual Review of Applied Linguistics, 34, 102-122. Winke, P., Lee, S., Ahn, J. I., Choi, I., Cui, Y., & Yoon., H.-J. (2018). The cognitive validity of child English language tests: What young language learners and their native-speaking peers can reveal. TESOL Quarterly, 52(2), 274-303. doi: 10.1002/tesq.396 Yang, L., & Gao, S. (2013). Beliefs and practices of Chinese university teachers in EFL writing instruction. Language, Culture, and Curriculum, 26(2), 128-145. Yasuda, S. (2011). Genre-based tasks in foreign language writing: Developing writers’ genre awareness, linguistic knowledge, and writing competence. Journal of Second Language Writing, 20(2), 111-133. Yeh, H.-C. (2015). Facilitating metacognitive processes of academic genre-based writing using an online system. Computer Assisted Language Learning, 28(6), 478-498. Yigitoglu, N., & Reichelet, M. (2014). Using a genre-based approach for writing instruction in a less commonly taught language. Language Awareness, 23(3), 187-202. https://doi.org/10.1080/09658416.2012.742906 Yoon, H.-J., & Polio, C. (2017). The linguistic development of students of English as a second language in two written genres. TESOL Quarterly, 51(2), 275-301. https://doi.org/10.1002/tesq.296 170