MMWW W llWin]lljfllllljfllfllifllfllylll LIBRARY Michigan State University This is to certify that the dissertation entitled ' THE EFFECTS OF EXPLICIT INSTRUCTION ON THIRD GRADE LOW GROUP STUDENTS' METACOGNITIVE AWARENESS OF READING LESSON CONTENT AND THEIR PERFORMANCE ON ADDITIONAL MEASURES OF READING ABILITY presented by Michael Steven Meloth has been accepted towards fulfillment of the requirements for Ph.D Educational Psychology degree in Date April 4, 1288 unn.--1a:_....- . - r, In - r .1 0427" MSU LIBRARIES .—3—_ RETURNING MATERIALS: Place in book dFEE—to remove this checkout from your record. FINES will be charged if book is returned after the date stamped below. ' .- H -a I“; .. r \ft '_' s: .‘.- u ,9», i “an ' '- fi. t 33;: 3 s 3,13" .‘ 1V. .2 --" t ’\ L4 xi , r T .‘E‘ I 1 —‘ THE EFFECTS OF EXPLICIT INSTRUCTION ON THIRD GRADE LOW GROUP STUDENTS' METACOGNITIVE AWARENESS OF READING LESSON CONTENT AND THEIR PERFORMANCE ON ADDITIONAL MEASURES OF READING ABILITY BY Michael Steven Meloth A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Counseling, Educational Psychology, and Special Education 1988 5670;3in ABSTRACT THE EFFECTS OF EXPLICIT INSTRUCTION ON THIRD GRADE LOW GROUP STUDENTS' METACOGNITIVE AWARENESS OF READING LESSON CONTENT AND THEIR PERFORMANCE ON ADDITIONAL MEASURES OF READING ABILITY BY Michael Steven Meloth This study extends current research in metacognitive awareness and instruction by (a) simultaneously examining the metacognitive awareness of reading lesson content at three different levels, (b) identifying the changes in levels of metacognitive awareness over time, and (c) examining the association between metacognitive awareness and performance on other reading outcome measures. The subjects and data for this study were low-group readers from twenty third grade classrooms. Students in ten classrooms received explicit instruction in reading throughout the academic year while students in the control classrooms received no such instruction. Students were interviewed six times during the year immediately following reading instruction and asked to respond to three levels of questions regarding lesson content. Each level of metacognitive awareness was examined to determine whether recipients of explicit instruction were rated significantly higher than control students. The trends of these ratings across the six interviews were then analyzed to determine the ways in which metacognitive awareness changed over time. ‘Finally, the variance attributable to levels of metacognitive awareness was removed from scores on the additional outcome measures. The between-group differences on these measures were then examined to determine whether metacognitive awareness contributed significantly to reading performance. At the end of the year there were nearly significant differences favoring the treatment group for Level 1 ratings and highly significant differences favoring the treatment group at Level 2 and Level 3. These significant differences were first apparent at the fourth observation. Significant linear trends were found at each level. There were no significant differences between groups on four of the five additional reading measures once the variance attributable to metacognitive awareness was removed. There are three implications of this study: (1) more explicit instruction with low group third grade readers should produce greater Level 1 responses, (2) poor readers' metacognitive awareness is not easily affected by explicit instruction even when such instruction is specifically designed to promote such awareness, and (3) the findings were consistent with a hypothesized model that metacognitive awareness mediates cognition during instruction. Copyright by MICHAEL STEVEN MELOTH 1988 This work is sponsored in part by the Institute for Research on Teaching, College of Education, Michigan State University. The Institute for Research on Teaching is funded primarily by the Program for Teaching and Instruction of the National Institute of Education, United States Department of Education. The opinions expressed in this dissertation do not necessarily reflect the position, policy or endorsement of the National Institute of Education. (Contract No. 400-81-0014) ACKNOWLEDGEMENTS As with many acknowledgements found in doctoral dissertations, there are many individuals who are deserving of praise. The members of my committee, Jere Brophy, Joe Byers, Chris Clark, and Gerry Duffy, all helped me over the past year in conceptualizing and finally completing a piece of work that, when compared to my original proposal, looks as if it is a completely different document. It is. Specific thanks go to all members (except Gerry Duffy) who never asked me when this document would be finished. Jere Brophy and Ellen Strommenn are to be thanked for their insightful comments on particular aspects of the original work, particularly when I began to think that metacognitive awareness was not what I had originally thought it was. Special thanks go to Chris Clark who, although I did not want to admit it at the time, forced me to re-think, and thereby more deeply understand, the construct of metacognitive awareness and its relation to the many, slippery aspects of human cognition. I cannot begin to thank Joe Byers, who also served as the chair of my program committee, for his knowledge and explicit instruction of the analysis and interpretation. He helped me turn what I had come to believe were an uninteresting and uninterpretable set of correlations to what has become an interesting and exciting study. Gerry Duffy, who only kept after me because of his concern for my future plans, was unfailing in his support throughout this dissertation. While I sometimes thought him too adept at finding theoretical and conceptual inconsistencies, thereby causing me to rewrite, and rewrite again, he gave me the skills to think through a problem, the confidence to believe the dissertation was an important one, and the motivation to move on to my professional career. Continued appreciation go to Gerry Duffy and Laura Roehler for many additional reasons. In my association with them while a member of the Teacher Explanation Project I was never anything other than a full, contributing member. I was given opportunities to develop ideas, investigate issues, and learn how to conduct research without ever losing sight of two major facts--we must treat teachers and students with respect and dignity and insure that our research gives something of.value to those who help us conduct it. Gerry and Laura’s professionalism in these areas will serve as a model for my future associations with graduate students, classroom teachers, and young learners. Second, and equal to the above, is the friendship I developed with Gerry and Laura. They are two individuals, who by their very nature, bring out the good in others and the opportunity to speak and work with them on a continuous basis may be the one thing I will miss most upon leaving Michigan State. Much of the same goes for the additional members of the project staff, Cass Book, Joyce Putnam, and Eva Sivan, with whom I worked for quite some time. They were a joy to work with and I look forward to continuing our friendship. Last, but not least is Linda Vavrus, who, in 1982, mentioned to me that she needed someone to work temporarily on the project. This chance encounter led to my association with the Teacher Explanation Project and, ultimately, what is found in this dissertation. There are many others in the College of Education who deserve to be mentioned, too many to mention, but I extend my thanks to all. Two, however, standout. Mary Mowry always had a kind word and helpful comment to everyone who came by her office, even me, even when I did not deserve it. Andy Porter, who helped me through, make that got me through, the most difficult single hoop in graduate school. That he sought me out and offered his help meant more than I could describe. Thanks also to Karlene, who helped organize this mess into something presentable. Special thanks go to my parents, Bud and Peggy, who never once asked me when I was going to graduate and always asked if I was enjoying myself. For a kid who almost flunked out of high school, I hope I made them proud. And finally, special appreciation goes to Cheryl Winter, whose friendship I value above all others. Since meeting her in the library three years ago she has been a constant friend and confidant, bringing, sometimes pulling, the best from me. I know I'll see her again. vi TABLE OF CONTENTS List of Tables . . . . . . . . . . . . . . . . . . . . xiv List of Figures. Chapter One Introduction and Purpose Background. Definition of Metacognitive Awareness Metacognitive Awareness of Reading Lesson Content Explicit Instruction Emphasizing Metacognitive Awareness of Reading Lesson Content Summary of the Background Statement of the Problem Research Questions Significance of the Problem Definition of Terms Metacognitive awareness Metacognitive knowledge Awareness Declarative, procedural, and conditional knowledge Metacognitive awareness of reading lesson content vii 10 12 14 16 16 17 17 17 17 I; Explicit instruction Levels of metacognitive awareness of reading lesson content Limitations Limitation 1: Definition of the construct of metacognitive awareness Limitation 2: Metacognitive knowledge not reflected in metacognitive awareness Limitation 3: Generalization of findings Limitation 4: Language differences among students Limitation 5: The deletion of questions at Level 2 and Level 3 Limitation 6: Interviewer-Rater agreement Limitation 7: Differences in interviewer style Summary of Chapter One Organization of the Remainder of the Dissertation Chapter Two Review of Relevant Research Section One: Definitions and Distinctions of Metacognitive Awareness Definition of Metacognitive Awareness The Distinction Between Metacognitive Awareness and Metacognitive Knowledge The Distinction Between Metacognitive Awareness and Cognition . The Distinction Between Metacognitive Awareness and Cognitive Awareness viii 18 18 19 34 36 The Distinction Between Metacognitive Awareness and Routinized Knowledge The Use of Verbalizations in the Assessment of Metacognitive Awareness Summary of Section One Section Two: Explicit Instruction For Metacognitive Awareness Instruction and Metacognitive Awareness. Characteristics of Explicit Instruction for Metacognitive Awareness Summary of Section Two Summary of Chapter Two Chapter Three Design and Methods Background Subjects Teachers Students Procedures Used to Collect Awareness Interview Data. Collecting the Awareness Interview Data. Scoring Awareness Interview Data Training Raters to score the data. Rationale for scoring deleted questions at Level 2 L and Level 3 . Conditions necessary for using the scoring system in this dissertation ix 4O 4O 41 62 66 Data Collected on Additional Reading Outcome Measures .. Supplemental Achievement Measure (SAM) Collecting the SAM Data. Scoring the SAM Data The Graded Oral Reading Paragraph (GORP) Collecting the GORP Data Scoring the GORP Data. Concepts of Reading Interview. Collecting the Concepts Interview Data Scoring the Concepts Interview Data. Stanford Achievement Test (SAT). Collecting the SAT Data. Scoring the SAT Data The Michigan Educational Assessment Program (MEAP) Collecting the MEAP Data Scoring the MEAP Data. Data Collected on Explicit Instruction and Classroom Management. Collecting the Teacher Data. Training raters to score the data. Scoring the Teacher Data Data Analysis. Research Questions 1 Through 3: Analysis of Levels of Metacognitive Awareness 69 72 73 73 74 75 76 76 76 77 77 78 79 79 79 80 81 82 83 87 87 Research Question 4: Analysis of Changes and Trends Research Question 5: Analysis of Additional Outcome Measures. Summary of Chapter Three Chapter Four Results Introduction Analysis of the Three Levels of Metacognitive Awareness Summary of the Findings for Levels of Awareness. Analysis of Changes and Trends in Metacognitive Awareness Changes and Trends for Level 1 Changes in Level 1 ratings across time Trends for Level 1 ratings across time Changes and Trends for Level 2 Changes in Level 2 ratings across time Trends for Level 2 ratings across time Changes and Trends for Level 3 Changes in Level 3 ratings across time Trends for Level 3 ratings across time Summary of the Findings for Changes and Trends The Relationship between Metacognitive Awareness Ratings and Performance on the Additional Outcome Measures. Level 2 Stepdown F-tests xi 88 in»- 112 Level 3 Stepdown F-tests Summary of Between-Group Differences when Variance Due to Awareness Interview Ratings was Removed. Summary of Chapter Four. Chapter Five Conclusions and Implications. Introduction Discussion and Conclusions of Research Questions 1 Through Research Question 3 Discussion of the Findings of Research Questions 1 Through Research Question 3 Conclusions of Research Questions 1 Through 3. Summary of Research Questions 1 Through 3. Discussion and Conclusions for Research Question 4 Discussion of the Findings of Research Question 4. Conclusions Regarding Research Question 4. Summary of Research Question 4 Discussion and Conclusions of Research Question 5. Discussion of the Findings of Research Question 5. Conclusions Regarding Research Question 5 Explanation of the GORP Findings Summary of Research Question 5 Implications of the Findings of this Dissertation An Emerging Model of Instruction and Metacognitive Awareness Metacognitive Awareness as a Mediational Process Implications for Future Research 112 113 113 115 115 117 119 P“ l‘ ) F.) r.) r.) 4.- FA I‘ J \J 128 129 132 135 136 136 137 142 Summary of Chapter Five. Appendix A Question Format for Student Awareness of Reading Lesson Content Interviews. Appendix B Awareness Interview Rating Criteria Appendix C Supplemental Achievement Measure(SAM) Appendix D Modified Graded Oral Reading Paragraph (GORP). Appendix E Concept of Reading Measure Appendix F Criteria for Explicit Instruction Rating Appendix G Management Principles Rating Criteria References xiii 153 157 159 170 LIST OF TABLES TABLE gags 3.1 Means and Standard Deviations for Management Principles . . . . . . . . . . . 85 3.2 Means and Standard Deviations for Explicit Instruction . . . . . . . . . . . . 86 4.1 Observation 6 Means and Standard Deviations for Level 1, Level 2 and Level 3 Awareness Interview Ratings . . 97 4.2 Univariate Analysis of Variance for Level 1, Level 2, and Level 3 Awareness Interview Ratings at Observation 6 . . . . . 98 4.3 Means and Standard Deviations for Level 1, Level 2 and Level 3 Awareness Interview Ratings . . . . . . . . . . . . . 101 xiv 4. FIGURE 1 D.) LIST OF FIGURES Level 1 Awareness Interview Ratings Across Six Observations Level 1 Awareness Interview Ratings and Linear Trends Across Six Observations Level 2 Awareness Interview Ratings Across Six Observations Level 2 Awareness Interview Ratings and Linear Trends Across Six Observations Level 3 Awareness Interview Ratings Across Six Observations Level 3 Awareness Interview Ratings and Linear Trends Across Six Observations Model of Explicit Instruction and Metacognitive Awareness of Reading LESSON Content XV PAGE 103 104 P... O 0‘: 1C7 108 109 138 CHAPTER ONE INTRODUCTION AND PURPOSE There are an increasing number of studies examining the explicit instruction of poor readers (Duffy, Roehler, Sivan, Rackliffe, Book, Meloth, Vavrus, Wesselman, Putnam, & Bassiri, 1987; Paris, Newman, & McVey, 1982; Paris & Jacobs, 1984; Paris, Cross, and Lipson 1984; Hansen & Pearson, 1983; Palincsar & Brown, 1984). Many of these studies directly teach poor readers to improve their metacognitive awareness of reading. Such awareness is generally described as the readers' verbalization or demonstration of declarative, conditional, and procedural knowledge about reading (Paris, Wixson, & Lipson, 1983). By directly informing poor readers of (a) what the important content of the lesson is (declarative knowledge), (b) why it will be beneficial to learn (conditional knowledge), and (c) how to use lesson content in a strategic manner (procedural knowledge), their metacognitive awareness should be enhanced and other areas of reading should be improved. The results of these studies are encouraging for two reasons. First, regardless of whether they were conducted independent of classroom contexts or within naturally occurring classroom environments, explicit instruction has improved poor readers' metacognitive awareness of (a) what 2 successful comprehension entails, (b) why certain reading skills are beneficial, and (c) how to successfully use reading skills before, during and after reading. Second, and just as important, there is evidence that explicit instruction affects performance in other areas of reading (e.g., Duffy, et al., 1987; Paris, et al., 1984). However, additional research is needed before the effects of explicit instruction are fully understood. The above findings only indicate that explicit instruction effected poor readers' metacognitive awareness of reading and reading performance. No study has examined the extent to which different levels of poor readers' metacognitive awareness of reading lesson content are affected by explicit instruction or the extent to which performance on other outcome measures, independent of metacognitive awareness of reading, is affected by explicit instruction. To further understand the effects of explicit instruction on levels of metacognitive awareness of reading lesson content, its development over time, and the relationship between levels of such awareness and performance on other measures of reading, such questions need to be addressed. This dissertation addresses these questions. Background The following section, divided into three parts, provides background for this dissertation. Because readers' metacognitive awareness of reading lesson content 3 is the focal point of this study, this section begins with a general definition of metacognitive awareness. This will be followed by the specific definition of metacognitive awareness of reading lesson content used in this dissertation. Finally, explicit instruction which focuses on improving poor readers' metacognitive awareness of reading lesson content will be discussed. Definition of Metacognitive Awareness The study of metacognitive awareness is in an embryonic stage and, as such, the definitions of this construct are tentative and may vary quite markedly from one study to another. As Paris (1987) has suggested, these definitions often fail to distinguish metacognitive awareness from metacognitive knowledge and incorrectly attribute performance on tasks to metacognitive knowledge without actually measuring such knowledge. However, despite these problems, enough similar elements in the writings from this field can be drawn together to provide an initial definition of metacognitive awareness. The definition of metacognitive awareness used in this dissertation in based on these writings. Flavell (1976) writes that metacognitive awareness refers to a self-awareness of what an individual knows about both his or her cognitive processes and the ability to control that knowledge through the "active monitoring 4 and consequent regulation and orchestration of these processes...usually in the service of some goal ( 1976, p. 232)". Brown (1978; 1980) agrees with Flavell (1976; 1981) that an awareness of the knowledge an individual possesses and an awareness of how to regulate this knowledge are the primary elements of metacognitive awareness: A very basic form of self-awareness is the realization that there is a problem, of knowing when you know and when you do not. If an unfortunate [learner] does not recognize that he or she has failed to understand an important point, he or she cannot initiate a course of action to rectify the gap in knowledge (Brown, 1980, p. 458). Thus, a conscious awareness of cognitive knowledge and regulation of cognition are both important components of metacognitive awareness. Jacobs and Paris (1987) further refine the definition of metacognitive awareness by stating that it is: any knowledge about cognitive states or processes that can be shared between individuals. That is, knowledge about cognition can be demonstrated, communicated, examined, and discussed...Thus, it is reportable, conscious awareness about cognitive aspects of thinking (Jacobs & Paris, 1987, p. 258). Flavell (1976; 1981) and Brown (1976; 1980) often describe the knowledge involved in the awareness of cognition as declarative knowledge and the knowledge involved in the regulation of knowledge as procedural knowledge. Paris, et a1. (1983) suggest that in order for metacognitive awareness 5 to be demonstrated or communicated, a third category of knowledge, conditional knowledge, is necessary. Declarative knowledge involves an awareness of what is known by an individual, such as an awareness that activating relevant prior knowledge facilitates learning. Procedural knowledge involves an awareness of how to use knowledge that is possessed, such as how to use strategies that assist in retrieving relevant prior knowledge when learning. Conditional knowledge involves an awareness of when and why particular prior knowledge should be activated, such as why one particular strategy will be more likely to result in learning than other possible strategies. There are two implications of these three categories of knowledge. First, declarative, procedural, and conditional knowledge provide a concrete delineation of the cognitive knowledge that is necessary for learning. Second, these categories of knowledge can be directly demonstrated and communicated to students. Drawing together the work by Flavell (1976; 1981), Brown (1978; 1980), and Paris and his colleagues (Paris, et al., 1983; Jacobs and Paris, 1987), the following definition of metacognitive awareness, commonly used in the work cited above, is the definition used in this dissertation: metacognitive awareness is knowledge at a conscious, verbalizable level reflecting an individual's awareness of (a) the knowledge and abilities that are possessed 6 (declarative knowledge), (b) how to use knowledge and abilities (procedural knowledge), (c) why the knowledge and abilities are important to possess (conditional knowledge), and (d) the ways in which declarative, procedural, and conditional knowledge can be controlled or monitored so that learning can occur. This definition distinguishes metacognitive knowledge from metacognitive awareness (Jacobs & Paris, 1987). Metacognitive knowledge is knowledge of cognition and regulation of cognition that may or may not be available to conscious recall. Metacognitive awareness, in contrast, is knowledge that Brown, Bransford, Ferrara, and Campione (1983) call "statable" knowledge, i.e., that metacognitive knowledge which is at a conscious level. Conscious awareness suggests that there may be some "threshold" of awareness, below which knowledge cannot be reported. Declarative, conditional and procedural knowledge below the "threshold" cannot be verbalized by the individual. Above the threshold is conscious awareness, where a subject can report or verbalize knowledge. It is the knowledge above this threshold that is of interest in this dissertation. The advantage of defining metacognitive awareness as declarative, procedural, and conditional knowledge at a conscious level is that one gains a valid and reliable, albeit, conservative, indicator of knowledge possessed. By its very definition, only that which can be verbalized is 7 measured. Metacognitive knowledge, on the other hand, must be inferred through performance on some other measure, and, as a consequence, represents an indirect measure of knowledge that is possessed. Metacognitive Awareness Of Reading Lesson content The above definition of metacognitive awareness refers to learning in general. Various terms are used when investigating different domains of knowledge. For example, metacognitive awareness of reading is used when studying the knowledge an individual possesses about reading in general (Brown, 1980), metacomprehension is used to examine awareness of the processes of comprehension (Markman, 1981), metamemory is used when investigating awareness of encoding and retrieval strategies (Flavell & Wellman, 1977; Brown, 1978), and metacommunication is used when studying awareness of verbal and/or nonverbal interactions in social situations (Flavell, 1981). Despite the different terms, each includes the elements described in the above general definition of metacognitive awareness. In this dissertation, the domain of interest is reading lesson content. Consequently, metacognitive awareness of reading lesson content is defined as knowledge at a conscious, verbalizable level reflecting an awareness of (a) the knowledge and abilities that are learned from reading 8 lessons (declarative knowledge, or what the lesson was about), (b) how to successfully use what was learned during instruction (procedural knowledge, or how to use the lesson content correctly), (c) why the knowledge and abilities learned from the lesson are important to possess (conditional knowledge, or why the lesson was important), and (d) the ways in which declarative, procedural, and conditional can be controlled or monitored so that improved reading abilities are acquired. Explicit Instruction Emphasizing Metacognitive Awareness of Reading Lesson Content It is important for poor readers to be metacognitively aware of reading lesson content. When students are aware of what the lesson is about, why it is important, and how to use the information presented during instruction, improved reading ability is likely to result (Roehler, Duffy, & Meloth, 1986; Paris, Wixson, & Palincsar, 1987). Unfortunately, a common characteristic of poor readers is that they lack an awareness of the What, Why, and How of the lesson (Roehler, Duffy, & Meloth, 1986). Consequently, teachers need to insure that poor readers are provided with instruction that will improve their metacognitive awareness of lesson content. Studies emphasizing metacognitive awareness have been successful in directly teaching poor readers' to become more 9 metacognitively aware of reading and reading lesson content (Duffy, et al., 1987; Paris, at al., 1982; Paris, at al., 1984; Hansen & Pearson, 1983; Palincsar & Brown, 1984). However, they provide only an initial description of the effects of such instructional interventions on poor readers' metacognitive awareness and performance on additional reading outcome measures. What is lacking from these studies is a more detailed examination of the development and impact of metacognitive awareness. Specifically, it is important to examine (a) how soon recipients of explicit instruction demonstrate different levels of metacognitive awareness, (b) the ways in which these levels of metacognitive awareness change over time, and (c) the extent to which metacognitive awareness affects other reading outcomes. This dissertation was directed toward investigating these important areas. Summary of the Background Metacognitive awareness, defined as declarative, conditional, and procedural knowledge which can be accessed and verbalized, is assumed to play an important role in learning. Metacognitive awareness of lesson content is the ability to verbalize what the lesson was about, when to use what was learned, why it is useful to learn, and how to apply lesson content successfully. Poor readers whose metacognitive awareness is not developed during instruction may not acquire the knowledge and skills that are associated 10 with improved reading abilities. Explicit instruction should have a powerful impact on poor readers' metacognitive awareness. When instruction explicitly informs students of the declarative, conditional, and procedural knowledge embedded in lesson content, poor readers can increase awareness of lesson content which, in turn, improves their ability to successfully comprehend text. However, only initial data are available regarding the ways in which metacognitive awareness develops and affects reading growth. Examining different levels of poor readers' metacognitive awareness of reading lesson content, the changes in levels of metacognitive awareness over time, and the extent to which metacognitive awareness affects performance on other reading outcome measures will provide a more detailed and in-depth understanding of the impact of metacognitive awareness on learning to read. Statement of the Problem The purpose of this dissertation was to determine (a) whether poor readers who receive explicit instruction are more likely to demonstrate metacognitive awareness of reading lesson content than their control-group counterparts when provided with only very general post-lesson interview prompts regarding what the lesson is about, why it is important to learn, and how to use what was learned effectively, as opposed to being provided with either a moderate amount of 11 post-lesson interview prompts regarding the What, Why, and How of the lesson or with extensive cuing regarding reading lesson content, (b) how quickly poor readers acquire such awareness, and (c) the extent to which metacognitive awareness contributes to performance on additional reading outcome measures. To determine how much cuing recipients of explicit instruction require before demonstrating metacognitive awareness of reading lesson content that was significantly greater than poor readers who did not receive such instruction, awareness was assessed by uSing a multidimensional interview measure comprised of three levels of questions regarding awareness: Level 1-- General Awareness, Level 2--Specific Awareness, and Level 3--Awareness With Stimulus Materials. The differences in levels of metacognitive awareness of reading lesson content between poor readers in treatment classrooms (explicit instruction) and poor readers in non-treatment classrooms at the end of the instructional intervention were then examined to determine whether recipients of explicit instruction demonstrated metacognitive awareness with fewer interviewer prompts than students who did not receive such instruction. To examine how metacognitive awareness of reading lesson content changed over the course of the intervention, the interview measure was administered following six observed reading lessons. The first observation where significant 12 differences between groups at each of the three levels was identified. This finding indicated the time required before recipients of explicit instruction demonstrated improved metacognitive awareness of reading lesson content. The trends in ratings of metacognitive awareness were then examined to determine whether, for example, poor readers levels' of metacognitive awareness continued to improve throughout the course of the intervention or increased to a certain extent and then leveledoff. To investigate the extent to which metacognitive awareness contributed to poor readers' performance on other measures of reading ability, the differences between treatment and non-treatment groups on five additional reading outcome measures where significant differences favored treatment students were re-analyzed. In this analysis, the variance due to metacognitive awareness at each level was removed from each of these measures. The results indicated whether the subsequent differences between the two groups became non-significant once the variance attributable to metacognitive awareness ratings was removed. Research Questions Given the current interest in metacognitive awareness and the assumption that other reading abilities will improve by promoting changes in metacognitive awareness, this dissertation investigated the following general research 13 question: What are the differences at each level of metacognitive awareness between recipients of explicit instruction and those who did not receive such instruction, what are the changes in levels of metacognitive awareness over time, and how does metacognitive awareness affect performance on other measures of reading? The following specific questions were addressed to investigate this general research question. With the exception of the trend analyses (Research Question 4), only metacognitive awareness ratings from the end of the study (Observation 6) were used because (1) these final ratings reflect the effects of the year-long instructional intervention and (2) these ratings were obtained at approximately the same time the additional outcome measures were administered. The first three research questions examine differences between the treatment and non-treatment groups at each of the three levels. The fourth research question examined the trends of awareness ratings across time. The fifth research question examined the extent to which the variance attributable levels of metacognitive awareness contributed to scores on the additional outcome measures. 1. Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 1 Awareness ratings? 2. Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between 14 students in treatment and treated-control classrooms for Level 2 Awareness ratings? 3. Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 3 Awareness ratings? 4. What is the trend in the ratings received at each of the three levels of metacognitive awareness of reading lesson content across the six observations? 5. If the variance due to metacognitive awareness ,ratings at those levels where significant differences between groups were found is removed from the five additional outcome measures where significant differences were also found, do the differences between groups on the additional outcome measures become non-significant? Significance of the Problem The findings of this dissertation are significant for three reasons. First, while previous studies in this field have used only a single, unidimensional metacognitive awareness measure, in this study poor readers' metacognitive awareness of reading lesson content was examined at three different levels. As such, a more detailed understanding of‘ the ways in which instruction affects metacognitive awareness is offered. By assessing metacognitive awareness at three different levels, this study examined whether poor readers who receive explicit instruction demonstrate significantly greater metacognitive awareness when (a) presented with only very general post-lesson interview prompts regarding what the 15 lesson is about, why it is important to learn, and how to use what was learned (Level 1), (b) provided with a moderate amount of post-lesson interview prompts regarding the What, Why, and How of the lesson (Level 2), or (c) when provided extensive post-lesson interview prompts regarding reading lesson content (Level 3). The second reason the findings of this dissertation are significant is its examination of changes in poor readers' metacognitive awareness over time. Virtually all instructional studies emphasizing metacognitive awareness examine pretest-posttest differences. Consequently, no data are available to indicate how poor readers' metacognitive awareness may improve over time. This dissertation examines the differences in responses at each of the three levels of metacognitive awareness of reading lesson content between groups at six different points during a year-long intervention study. Such an analysis identifies the length of time necessary before explicit instruction affects poor readers' metacognitive awareness. The trends of metacognitive awareness ratings were indicate whether responses at each level of metacognitive awareness continued to improve over time or if poor readers' awareness increased to a certain extent and then remained constant or decreased. The third reason the findings of this dissertation are significant is the examination of the extent to which metacognitive awareness is associated with performance on 16 five additional measures of reading ability. Five measures of reading ability were selected because poor readers in the larger study who received explicit instruction performed significantly better on each of these measures than poor readers who did not received such instruction. While other studies have found strong, positive correlations between metacognitive awareness and reading performance, no instructional study has examined whether it is the improved metacognitive awareness that significantly associated with performance on other reading outcome measures or if some other component(s) of the instructional intervention affect improved performance on these outcome measures. This study examined the extent to which ratings of metacognitive awareness are associated with poor readers' performance and, by extension, the role that metacognitive awareness plays in reading. Definition of Terms This dissertation has used several terms in its discussion. The major terms are defined below. Metacognitive awareness: Knowledge at a conscious, verbalizable level reflecting an individual's awareness of (a) the knowledge and abilities are possessed (declarative knowledge), (b) how to use knowledge and abilities (procedural knowledge), (c) why the knowledge and abilities are important to possess (conditional knowledge), and (d) the 17 ways in which declarative, procedural, and conditional knowledge can be controlled or monitored so that learning can occur. Metacognitive knowledge: Knowledge about cognition and the knowledge which regulates cognition. This knowledge contributes to the ways in which an individual activates background knowledge and carries out cognitive activities when learning, reading, or eliminating disruptions to comprehension. Metacognitive knowledge is not at a conscious level and is not verbalizable. Awareness: The ability to demonstrate, verbalize or communicate what is known by an individual. Declarative, procedural, and conditional knowledge: Three categories of knowledge that contribute to the acquisition of metacognitive awareness. Declarative knowledge, for example, is knowledge that a particular reading skill has been learned and can be used efficiently and effectively. Procedural knowledge involves an understanding of how to apply a reading skill correctly and modify the skill when needed. Conditional knowledge is knowledge of why a particular skill is important to learn or when a particular skill should be applied. Metacognitive awareness of reading lesson content: Knowledge at a conscious, verbalizable level reflecting an awareness of (a) the knowledge and abilities are learned from reading lessons (declarative knowledge, or what the lesson 18 was about), (b) how to successfully use what was learned during instruction (procedural knowledge, or how to use the lesson content correctly), (c) why the knowledge and abilities learned from the lesson are important to possess (conditional knowledge, or why the lesson was important), and (d) the ways in which declarative, procedural, and conditional can be controlled or monitored so that comprehension can occur. In this dissertation, metacognitive awareness of lesson content is assessed by interviewing students immediately following their reading lessons. Explicit instruction: Instruction which directly informs students of (a) what the important content of the lesson is, (b) why it will be beneficial to learn, and (c) how to use the information in a strategic manner. This information is intended to direct readers' attention toward the salient lesson content and insure that the appropriate connections between what is taught about reading and what is learned about reading agree. Levels of metacognitive awareness of reading lesson content: A level refers to the extent to which poor readers are able to report their metacognitive awareness in response to a set of interview questions. A level does not imply that the metacognitive awareness reported in response to the questions posed at that level is of different amount and/or quality--an individual possesses metacognitive awareness regardless of the questions posed but the questions vary in 19 the extent to which they focus students attention to, and verbalizations of their awareness. Sets of questions vary in the degree to which poor readers' awareness is directed toward the declarative, conditional, and procedural knowledge of reading lesson content. Level 1--General Awareness questions asked the student to tell the interviewer everything that he/she could remember from the reading lesson. Level 2--Specific Awareness questions specifically asked the student to respond to (a) what the lesson was about? (declarative knowledge), (b) why it is important to know and when you would use the lesson information? (conditional knowledge), and (c) how do you use or apply what you learned? (procedural knowledge). Level 3--Awareness With Stimulus Prompts were identical to Level 2 but students were allowed to review materials and examples from their lessons when responding to Level 3 questions. Limitations There are seven limitations to the study: the construct of metacognitive awareness, the extent to which metacognitive awareness reflects metacognitive knowledge, the generalization of the findings, the language ability of the subjects, the possible lack of independence among the three levels of questions, the deletion of questions at Level 2 and/or Level 3, and the possible bias in interview data due 20 to interviewer style. Each will be discussed in turn. Limitation 1--Definition of the construct of metacognitive awareness: Due the short history of research in this field, an agreed-upon definition of metacognitive awareness and the manner in which it should be assessed have not yet been established. One impediment to such a definition is that it can be difficult to distinguish between those processes that are meta and those that are cognitive. Another difficulty is that the "threshold" of metacognitive awareness, i.e., the point at which an individual is consciously aware of his or her metacognitive knowledge, may be a function of the stimulus used to evoke verbalizations. A weak stimulus (e.g., Level 1) may never evoke conscious awareness while an extremely strong stimulus (e.g., Level 3) may evoke conscious awareness. In addition, but related to the above, the extent to which metacognitive awareness reflects the metacognitive knowledge that is not available to conscious awareness, regardless of the strength of the stimulus, is not known. Consequently, there can be a discrepancy between the metacognitive knowledge that is possessed by an individual and metacognitive knowledge that can be demonstrated or verbalized (metacognitive awareness). Limitation 2--Metacognitive knowledge not reflected in metacognitive awareness: This limitation is related to Limitation 1. No assumption is made in this dissertation regarding the extent to which the measure of metacognitive 21 awareness used in this study reflects the metacognitive knowledge possessed by an individual. Some degree of metacognitive knowledge may go unreported and, hence, not reflected in reports of metacognitive awareness. Since knowledge that is not reportable can still be used when performing on the additional measures of reading included in this dissertation, the extent to which the assessment of metacognitive awareness fails to capture metacognitive knowledge goes unreported but which contributes to performance is not known. Limitation 3--Generalization of findings: The subjects were the low-reading group students of 20 third-grade teachers who volunteered for this study. The curriculum was the reading skills taught by these teachers to their low- group readers as prescribed by the basal reading textbook mandated by the school district. The treatment model was the explicit instruction of basal reading skills as strategies. The approach to assessing metacognitive awareness has not been used with students of other reading abilities. No claim is made regarding the effectiveness of this treatment model for non-voluntary third-grade teachers, for higher level reading groups, or for other curricula areas in these or other third-grade classrooms. Additionally no claim is made regarding the effectiveness of this instructional model for other grade levels. Finally, it is not known whether the metacognitive awareness ratings of poor readers in the 22 treatment group reflect the ratings that would be given to readers who possess average or above average reading abilities. Limitation 4--Language differences among students: All students in this study were in the third grade and reading at or below the second-grade level at the beginning of the 1984-85 school year. The students had been identified as low-ability readers by their schools in accordance with district guidelines. These reading groups tended to be heterogeneous in nature, and included low-aptitude children, mainstreamed special education students, children whose primary language was not English, etc.. These low-group readers may have had deficits in their language base, their expressive language abilities, and/or other language-related problems. The validity and reliability of the interview measure which required participants to verbalize their responses in this study may be affected and could result in a biased assessment of metacognitive awareness of lesson content. Limitation 5--The deletion of questions at Level 2 and Level 3: A fifth possible limitation of this dissertation stems from the interview procedures used to investigate metacognitive awareness. As is noted in Chapter 3, the interviewer judged each student's responses and determined whether that subject gave an exemplary response to the question (i.e., a rating of 4) for one or more of the three 23 knowledge categories: declarative, conditional, and procedural knowledge of lesson content. If the interviewer judged the response as exemplary for Level 1 or Level 2, the question(s) at the subsequent level(s) for that knowledge category were not asked. This was done because of a concern that asking the same question at a subsequent level might signal to the subject that his/her response was not correct. When questions were deleted, however, interview questions (and the opportunity to respond) at each subsequent level were dependent on the response at the previous level. Consequently, an exemplary rating at Level 1 or Level 2 would indicate that a ceiling effect was present and that the student's "threshold" of awareness was reached. As a result, ratings at the subsequent level(s) would not provide an accurate reflection of metacognitive awareness. However, only 123 questions (3.796%) out of a possible 3,240 questions (540 interviews x 3 Level 2 questions x 3 Level 3 questions) were deleted. This low percentage of deleted questions suggests that few ratings of responses were dependent on the previous rating and, consequently, the dependence of the questions is not likely to affect the outcomes of the study. Limitation 6--Interviewer-Rater agreement: This limitation is also related to the interviewers' decisions to delete a question based on their judgment of the quality of the subject's response. The interviews were independently rated by trained raters who were not involved in the 24 interviewing of students. If the responses that resulted in the deletion of subsequent questions at Level 2 or Level 3 were not rated as high awareness (i.e., a rating of 4) by the raters, then there is no way of knowing whether the student could have given a better response (a higher rating) to that same question at the next level. As in Limitation 5, this may affect the interpretation of the findings. However, the raters assigned a rating of 3 or 4 to 90.10 % of the questions deleted by the interviewer. Consequently, the interviewers were accurate judges of students' responses during the interviews. Limitation 7--Differences in interviewer style: There were 6 interviewers, all members of the project staff, who were trained in the use of the awareness interview format. Training included the correct phrasing and ordering of the interview questions and the use of follow-up probes to encourage students to respond. Some staff members interviewed students in treatment classrooms, others interviewed students from treated-control classrooms, and others interviewed students from both. However, each interviewer had his or her own style. Some may have been more adept at helping the students feel comfortable during the interview, more effective at knowing when to probe and when to move on to the next question, and/or more able to avoid leading questions and probes that signal to the student the desired response. Even though all interviewers were 25 given training, systematic differences in interviewer style, particularly those who interviewed students from only treatment or only treated-control classrooms, may have inadvertently resulted in biased responses from students. Summary of Chapter One There are an increasing number of studies examining the direct instruction of poor readers. Many of these studies concern the explicit instruction of metacognitive awareness of reading. Such instruction informs poor readers of (a) what the important content of the lesson is, (b) why it will be beneficial to learn, and (c) how to use lesson content in a strategic manner. By including this information in reading instruction, both metacognitive awareness and performance in other areas of reading should improve. However, there are three areas in this research that have not been adequately addressed. First, research has not examined whether poor readers who receive explicit instruction are more likely to demonstrate metacognitive awareness of reading lesson content than their control-group counterparts when provided with only very general cues regarding what the lesson is about, why it is important to learn, and how to use what was learned effectively, or that being provided with either a moderate amount of cuing regarding the What, Why, and How of the lesson and/or being provided with extensive cuing regarding reading lesson 26 content is necessary before differences in metacognitive awareness become apparent. Second, no data are available from instructional studies which indicate the ways in which levels of metacognitive awareness change over time. Finally; no data are available to indicate whether poor readers' performance on other measures of reading is strongly associated to metacognitive awareness, independent of the type of instruction they received. Explicit instruction in this study was primarily directed toward improving poor readers' metacognitive awareness of how to use repair strategies while reading. Poor readers in treatment classrooms received explicit instruction while poor readers in the treated-control classrooms did not. Metacognitive awareness of reading lesson content assessed at three different levels through an interview measure which varied in the extent to which poor readers' awareness was focused toward the metacognitive content of their lessons. Interview data were collected at six different points during the instructional intervention to identify the changes in these levels of metacognitive awareness of reading lesson content across time. The extent to which metacognitive awareness ratings were associated with performance on additional reading outcome measures was assessed by removing the variance attributable to metacognitive awareness ratings from the scores on five additional reading measures. 27 Organization of the remainder of the dissertation The organization of this dissertation is the following: Chapter 2 will discuss the theoretical basis for examining metacognitive awareness of lesson content and the role of instruction in facilitating such awareness. Chapter 3 will describe the design of the study, including the subjects, dependent measures used to assess metacognitive awareness of lesson content, reading achievement, and explicit instruction, and the statistical analyses employed to analyze the data. Chapter 4 will present the findings of the study. Chapter 5 will discuss the results and offer implications of the findings to metacognitive awareness, reading achievement, and instruction of reading skills in the early grades. CHAPTER TWO REVIEW OF RELATED RESEARCH Studies investigating the effects of explicit instruction on metacognitive awareness and performance are relatively new, and the characteristics of metacognitive awareness and explicit instruction are complex. Therefore, this chapter provides an in-depth discussion of (a) metacognitive awareness as distinct from, but related to, other components of thinking and (b) the way in which explicit instruction can be structured to improve poor readers' metacognitive awareness. To do so, this chapter is divided into two major sections. The first section provides a discussion of metacognitive awareness and its distinctions from metacognitive knowledge, cognition, cognitive awareness, and routinized knowledge. While the definition of Ametacognitive awareness is tentative, given the short history of research in this field, it represents an integration of current research on this construct. In addition, although the definition of metacognitive awareness of reading lesson content was given in Chapter One, it is also included here in order to provide a context for the discussion of these distinctions. The second will discuss explicit instruction for metacognitive awareness. This section particularly emphasizes research on effective instruction, the general structure of explicit instruction for improving metacognitive 28 29 awareness of reading lesson content, and the specific characteristics for explicit instruction. Section One: Metacognitive Awareness--Definitions and Distinctions The formal definition of metacognitive awareness of reading lesson content was given in Chapter One. This definition will be re-stated here to provide a context for the ensuing discussion of the distinctions between metacognitive awareness and other elements in mental 'processing. In doing so, this section is divided into six subsections: (1) the definition of metacognitive awareness of reading lesson content, (2) the distinction between metacognitive awareness and metacognitive knowledge, (3) the distinction between metacognitive awareness and cognition, (4) the distinction between metacognitive awareness and cognitive awareness, (5) the distinction between metacognitive awareness and routinized knowledge, and (6) the accuracy of describing metacognitive awareness through verbal reports. Definition of Metacognitive Awareness Brown, Bransford, Ferrara, & Campione (1983) write that it is often difficult to distinguish between those processes that are meta and those that are cognitive. For example, it is unclear whether the term metacognition should be used when 30 describing the ways in which an individual identifies the purpose of a task or the ways in which an individual recognizes which prior knowledge is necessary when solving a problem. In addition, Paris (1987) suggests that there is an additional problem in understanding this construct: a failure to distinguish between the metacognitive knowledge that is possessed by an individual and metacognitive knowledge that can be demonstrated or verbalized (metacognitive awareness). Despite these problems, it is generally agreed that metacognitive knowledge can be "statable, in that one can reflect on the cognitive processes involved and discuss them with others" (Brown, et al., 1983, pp. 107). Consequently, a major criteria of metacognitive knowledge is that it is available at a conscious level. Hence, one is always consciously aware of such knowledge, if only for a brief, fleeting moment. As Brown, et al. (1983): Several theorists from quite disparate schools agree that the most stringent criteria of [a subjects'] understanding involve the availability of knowledge to consciousness and reflection, thus permitting verbal reports (pp. 108) The formal definition of metacognitive awareness of reading lesson content is closely tied to an awareness of what is taught during instruction: metacognitive awareness of reading lesson content is defined as knowledge at a conscious, verbalizable level reflecting an awareness of (a) the knowledge and abilities that are learned from reading 31 lessons (declarative knowledge, or what the lesson was about), (b) how to successfully use what was learned during instruction (procedural knowledge, or how to use the lesson content correctly), (c) why the knowledge and abilities learned from the lesson are important to possess (conditional knowledge, or why the lesson was important), and (d) the ways in which declarative, procedural, and conditional can be controlled or monitored so that improved reading abilities are acquired. The inclusion of an awareness of declarative, conditional, and procedural knowledge (Paris, et al., 1983) in this definition of metacognitive awareness is essential. Instruction which makes one aware of these three knowledge categories, for example, assists the student in attending to, and learning about, the important lesson content communicated by the teacher. Declarative knowledge represents knowledge of what the lesson is about, conditional knowledge represents knowledge of why the lesson content is important to learn, and procedural knowledge is knowledge of hgw to use the lesson content in a flexible, strategic manner. When readers are aware of these three knowledge categories, they are more likely to acquire an understanding of (a) the important cognitive knowledge and skills presented during the lesson, (b) how to use the lesson content when reading, (c) why learning the lesson content will be beneficial, (d) the ways in which this knowledge can be controlled or monitored so 32 that comprehension can occur. The Distinction Between Metacognitive Awareness and Metacognitive Knowledge The distinction between metacognitive awareness and metacognitive knowledge is straight-forward. Metacognitive knowledge refers to the knowledge information and skills possessed by an individual and the knowledge of how to control or regulate that knowledge (Brown, 1980). When confronted with a learning task, metacognitive knowledge plays a vital role in activating and implementing the relevant cognitive knowledge and strategies necessary for completing the task as well as monitoring progress on the task (Borkowski, Carr, & Pressley, 1987). . Metacognitive awareness is simply metacognitive knowledge that is at, or available to, recall, verbalization, or demonstration. As Flavell (1976) writes, "metacognition refers to one’s knowledge concerning one's own cognitive processes and products or anything related to them" (p.- 232). Flavell (1979) further states that the knowledge involved in metacognition "consists primarily of knowledge or beliefs about what factors or variables act and interact in what ways to affect the course and outcomes of cognitive enterprises" (p. 907). Metacognitive awareness, then, is the ability to make such metacognitive knowledge "public". Thus, metacognitive awareness is "reportable, conscious awareness 33 about cognitive aspects of thinking" (Jacobs & Paris, 1987, p. 258) and such knowledge can then be "shared, described, or demonstrated explicitly" (Paris, Jacobs, & Cross, 1987, p.237). The demonstratable/verbalizable component that distinguishes metacognitive awareness from metacognitive knowledge is extremely important in that such demonstrations or verbalizations are more direct measures of the declarative, conditional, and procedural knowledge possessed by an individual. If an individual can describe his or her metacognitive knowledge, that individual is aware of such knowledge. This description can then be evaluated to determine the extent to which the individual possesses metacognitive awareness. In studies of metacognitive knowledge, on the other hand, the assessment of metacognitive knowledge is inferred through performance on a task which, theoretically, requires the use of metacognitive knowledge. It is assumed that the better the performance, the greater the metacognitive knowledge. For example, assume a student is trained to use the rule or strategy of "re-reading the sentence when comprehension is disrupted". According to Brown (1980), such a strategy is metacognitive in nature as it involves recognizing when and how to take corrective action when failures to comprehension are detected by the reader. The student is then given several passages in which words or 34 phrases are intentionally difficult to understand. If improved performance is found, then it is concluded that the individual activated and implemented the metacognitive knowledge compatible with the type of disruption. The above example assumes that there is a causal link between instruction and performance. With rigid experimental control, such an inference might be correct. Even so, metacognitive knowledge is not directly measured. Further, in many ecologically valid studies, where "training" occurs through instruction in uncontrolled classroom settings, performance can be subject to a variety of other factors such as classroom organization (Doyle, 1986), student motivation (Weiner, 1979) and/or language proficiency (Fillmore & Valadez, 1986). Consequently, inferring the use of metacognitive knowledge without direct evidence that such knowledge was the primary reason for improved performance is subject to error. By more direct measurement, through metacognitive awareness, such inferences can be reduced and more valid conclusions regarding metacognitive awareness and the training, particularly in classroom-based studies, can be drawn. Distinctions Between Metacognitive Awareness and Cognition The difference betWeen metacognition awareness and cognition is also straight-forward. The knowledge possessed 35 by an individual consists of facts, rules, skills, principles, concepts, strategies, scripts, and so on (Gagne, 1977; Bransford, 1979). Cognition involves the activation and application of this knowledge. For example, reading an unfamiliar passage may require the individual to go back and re-read sections that are confusing or that contain words which are not recognized. Such an action involves cognition in that the individual uses a rule or strategy ("If the meaning of the passage is not known, then go back and look for the appropriate information") to comprehend the passage. Metacognitive awareness involves the verbalization of, for example, why it is necessary to go back and look for the appropriate information when the meaning is not known, or how to skim the passage until the information is found. In sum, cognition is the act of re-reading or skimming. Metacognitive awareness is the description of, in this example, why or how such cognitive activity will be successful. Cognition does not need to occur at a conscious level; an individual may re-read a passage without consciously realizing that such a rule or strategy is being used. When an individual is aware that such a rule/strategy is necessary, the individual can be said to be metacognitively aware of such a rule or strategy. 36 The Distinction between Metacognitive Awareness and Cognitive Awareness The distinction between metacognitive awareness and cognitive awareness directly follows from the above discussion. Cognitive awareness is the demonstration or verbalization of cognition while metacognitive awareness is the verbalization of metacognitive knowledge. Using the above example, an individual who was observed to re-read a passage may state that "I have to re-read the passage". In this situation, the individual is aware that such a rule or strategy is necessary. However, the individual is not metacognitively aware that such a rule/strategy is necessary unless he or she can also state the declarative, conditional, and procedural knowledge involved in using such a rule or strategy. Thus, if the individual states "I know that re-reading a passage is helpful when I do not understand certain words or the main idea and that I need to re-read the portions that I find confusing", then the individual is metacognitively aware, i.e., is conscious of the knowledge. The distinction here is subtle but very important. An individual who is aware that he or she is re- reading a passage (cognitive awareness) may not necessarily be aware of why re-reading is important or what event during reading caused the individual to re-read (metacognitive awareness). Assuming that verbalizations of cognitive awareness infer metacognitive awareness can lead to significant errors in 37 assessment. However, even metacognitive awareness can, over time, become routinized and unavailable to conscious recall. The Distinction between Metacognitive Awareness and Routinized Knowledge Possessing metacognitive awareness at one point in time does not necessarily mean that such knowledge will always be at the conscious level. Through repeated practice or rehearsal, much of what is learned eventually becomes routinized or automatic and unavailable to conscious recall (Ericsson & Simon, 1984; Craik & Lockhart, 1974). Over time, such metacognitive knowledge is integrated into existing schemata and is activated without conscious effort (LaBerge & Samuels, 1974; Bransford, 1979). For example, when a very young child first learns to re-read unfamiliar or difficult words or sentences, he or she may be very consciously aware of when and how to re-read (Baker & Brown, 1984). At this point, the child possess a strong metacognitive awareness that re-reading helps in comprehension. Over time, with repeated practice and the development of more proficient comprehension abilities, these same words or sentences do not need to be re-read because the individual's sight-word recognition has been routinized (Flavell, 1978; Gough, 1984). Consequently, what had been an activity which required a great deal of metacognitive awareness (re-reading unfamiliar words to comprehend their meaning) becomes automatic, 38 unconscious, and routinized. The Use of Verbalizations in the Assessment of Metacognitive Awareness The above does not imply that metacognitive awareness represents all the metacognitive knowledge possessed by an individual or that what is verbalized is reflects an unbiased report of what is known. The debate on the use of verbalizations as data is long-standing (e.g., Nisbett & Wilson, 1977; Cavanaugh & Perlmutter, 1982; Ericsson & Simon, 1980; White, 1980). Metacognitive awareness can be considered as a sub-set of metacognitive knowledge in that there may be some metacognitive knowledge that is not available to conscious recall (Fischer & Mandl, 1984). However, as Ericsson & Simon (1984), Yussen, Mathews, & Hiebert (1982), and Meichenbaum, Burland, Gruson, & Cameron (1979) have written, verbalizations can be valid and reliable indices of knowledge when children are asked to respond to questions (a) regarding difficult and recently learned tasks and information, (b) that soon follow an event (e.g., reading lesson), (c) which are followed by undirected probes (e.g., "Tell me more about that"), (d) which are directed toward the event/information of interest in order to reduce the demands on verbalization, particularly for young children, (e) that avoid hypothetical scenarios and vague questions, and (f) assess only those responses that are directly related to the 39 event/information of interest. When such conditions are met, even young children can accurately report their metacognitive knowledge about reading (Garner, 1987). In addition, there has been some concern that verbalizations represent simple recall of information, not metacognitive awareness (Cavanaugh & Perlmutter, 1982). For example, if a subject is taught to when and how to use the strategy of re-reading when comprehension is disrupted, and then that subject is asked to describe a strategy for improving comprehension, the response (description of re- reading) might simply reflect recall of what was taught. However, one important criteria is necessary if these responses can be described as metacognitive awareness and not simple recall--assessing metacognitive awareness through verbalizations differ from simple recall if these verbalizations are associated with differences in performance. If the subject can only recall what was taught, then there should be no improvement in performance on tasks which require the use of the strategy. In this case, the .subject's response only indicates that he or she remembered what was taught and did not incorporate the information into his/her schema for "dealing with comprehension disruptions". Conversely, if the subject's performance on tasks that require the use of the re-reading strategy, then the subject's verbalizations reflect metacognitive awareness. 40 Summary of Section One This section provided the definition of metacognitive awareness of reading lesson content. Metacognitive awareness is the verbalizable knowledge regarding declarative, conditional, and procedural knowledge while learning from instruction. The importance of verbalizing metacognitive knowledge was emphasized because such verbalizations make the thinking involved in metacognitive awareness "public" and therefore more easily and accurately assessed. Several distinctions were also made regarding metacognitive awareness, metacognition, cognitive awareness, cognition, and routinized knowledge. Finally, the use of verbalizations in assessing metacognitive awareness was discussed. Section Two: Explicit Instruction for Metacognitive Awareness There is a growing recognition that instruction should place greater emphasis on promoting metacognitive awareness (Roehler, Duffy, & Meloth, 1986; Chipman, et al., 1985; Paris, Wixson, & Palincsar, 1986). Such instruction, often termed "explicit instruction", informs students about the declarative, conditional, and procedural knowledge of lesson content in the assumption that such knowledge will improve students' metacognitive awareness. The purpose of this section is to briefly discuss explicit instruction emphasizing metacognitive awareness of reading lesson 41 content. To do so, this section will consist of two subsections: (1) effective instruction and its relationship to metacognitive awareness, and (2) the characteristics of explicit instruction. Instruction And Metacognitive Awareness Until recently, research on teaching was concerned primarily with the relationship between instructional behavior and student achievement. Remarkable progress has been made in identifying effective instructional practice that is associated with improved cognitive abilities. After reviewing numerous studies of research on teaching, Brophy and Good (1986) find that students achieve more in classes where teachers play an active role in organizing and 'communicating lesson content. They indicate that such active, direct instruction is effective when the teacher: presents information and develops concepts through lecture and demonstration, elaborates this information in the feedback given following responses to recitation or discussion questions, prepares students for follow-up seatwork activities,...monitors progress on assignments,...and follows up with appropriate feedback and reteaching if necessary (p. 360-361) In short, effective instruction is characterized by teachers who assume the major responsibility for informing students of the important lesson content and design lessons accordingly: The teacher carries the content to the students 42 personally rather than depending on the curriculum to do so...There is a great deal of teacher talk, but most of it is academic rather than procedural or managerial, and much of it involves asking questions and giving feedback rather than extended lecturing (Brophy & Good, 1986, p. 360-361). The basic conclusion of Brophy and Good (1986) and others (e.g., Rosenshine and Stevens, 1986; Roehler, et al., 1986) is that effective instruction requires subject matter to be carried to students during instruction by (a) identifying what is important to learn and (b) the manner in which to explicitly communicate this content to students. When teachers fail to do so, according to Rosenshine and Stevens (1986): students run the danger of not attending to the right cues, or not processing important points, and of proceeding on to later points before they have done sufficient elaboration and practice (p. 379). In the field of reading, there is growing evidence that teachers need to identify and explicitly communicate important lesson content to poor readers. Such instruction needs to identify the metacognitive components of lesson content and communicate this content by emphasizing what the lesson is about, why it is important to learn, and how to use the information successfully (Duffy, et al., 1987; Paris, et al., 1983; Forrest-Pressley & Gillies, 1983). Classroom- 'based studies have found that when instruction emphasizes metacognitive awareness of reading lesson content, readers 43 derive greater benefit from instruction. For example, Duffy, et al. (1987) and Paris & Jacobs (1984) find that students in classrooms where instruction emphasized an awareness of how and when to use various reading strategies improved their metacognitive awareness of reading and performance on a variety of reading outcome measures. Recent reviews of training and instructional studies support these finding (Paris, et al., 1987; Forrest-Pressley & Gillies, 1983; Baker & Brown, 1984). Instruction that explicitly communicates important lesson content enhances readers' metacognitive awareness and such awareness is associated with improved strategy use during reading. In sum, there is growing evidence that instruction should be directed toward improving poor readers' metacognitive awareness and that such instruction also benefits reading performance. However, due to the short history of such instructional practice, the manner in which lesson content is explicitly communicated varies from study to study. The following describes the essential characteristics of explicit instruction directed toward improving poor readers' metacognitive awareness of reading lesson content. 44 Characteristics of Explicit Instruction for Metacognitive Awareness Explicit instruction is intended to increase poor readers' metacognitive awareness of what the lesson is about, why it is important to learn, and how to use the information when reading as well as to improve poor readers performance in other areas of reading. The instructional characteristics of explicit instruction reflect an integration of findings from research on teaching (e.g., Brophy & Good, 1986; Rosenshine & Stevens, 1986; Clark & Peterson, 1986) and cognitive psychology (Baker & Brown, 1984; Flavell, 1981; Wagoner, 1983). Roehler, et al. (1986) and Duffy, et al. (1987), drawing upon research on teaching, and Paris, et al. (1986), drawing upon studies in cognitive psychology, identify several of the essential characteristics of explicit instruction. One characteristic involves planning and task analysis of the cognitive and metacognitive outcomes of the lesson. When planning a lesson, teachers examine these outcomes and decide how the curriculum can be organized so that students learn what the lesson is about, why it is important to learn, and ‘ how to use the content when reading. Objectives are then set and advance organizers are formulated which emphasize the cognitive and metacognitive outcomes of the particular lesson as well as the continuity between the present lesson and past and future lessons (Bloom, 1976; Hartley & Davies, 1976). 45 During instruction, teachers provide explicit explanations of what the lesson is about, why the content is important to learn, and how to use the content successfully. The What, Why and HOW'iS referred to throughout the duration of the lesson in order to insure that the appropriate "cues" mentioned by Rosenshine and Stevens (1986) are attend to by students and the learning that results from attending to these cues meet the goals of the lesson. Instruction also includes detailed demonstrations and modeling of when lesson content is to be used and how to apply content when reading. Demonstrations and modeling of when and how to use the lesson content appropriately serve to "make visible the inviSible” thinking associated with using lesson content (Duffy, et al., 1987; Brainin, 1985; Meichenbaum, 1985). The provision of practice, corrective feedback, and on- going assessment during and after instruction is also an important characteristic of explicit instruction. During the lesson the teacher insures that there is sufficient opportunity for each student to demonstrate his or her understanding of what is being learned. Such practice during instruction provides the teacher with the opportunity to conduct an on-going assessment of each students' progress and allows for immediate corrective feedback to the student before the lesson ends. Such practice has the added benefit of allowing students to model and demonstrate how to use the lesson content to others in the reading group so that 46 students can listen to, and learn from, the ways in which their peers use the lesson content. Thus, the teacher monitors the progress of students during the lesson and, consequently, the teacher can regulate how much instructional time needs to be devoted to meet the objectives of the lesson (Duffy, et al., 1987). In sum, explicit instruction informs students of the value and benefit of what is to be learned and how to use lesson content in a manner that will improve their reading ability. The outcomes of the lesson and the ways in which the lesson content can be effectively used are clearly and explicitly communicated and demonstrated by both teacher and student so that students become metacognitively aware of lesson content. Teachers also conduct an on-going assessment of the progress made by individual students so that they can be sure that all students understand what the lesson is about, why it is important to learn, and how to use the content successfully. Metacognitive awareness of lesson content is enhanced because teachers' emphasize such awareness throughout the lesson. Summary of Section Two Until recently, effective instruction was described in terms of teacher behaviors with little emphasis on those behaviors which promote metacognitive awareness of reading lesson content or the ways in which effective instruction 47 also benefits metacognitive awareness. Instruction identified as effective typically assumes the responsibility of informing students of what is to be learned in order to insure that students attend to important lesson content and not trivial details. More recently, explicit instruction has built upon findings from research on teaching. Such instruction "carries" the content of the lesson to students and insures that important lesson content is attended to. In addition, it places a greater emphasis on the metacognitive aspects of lesson content so that students become aware of the declarative, procedural, and conditional knowledge inherent in their lessons. Objectives, interactive instruction, practice, and feedback are all directed toward informing students of what the lesson is about, why it is important to learn, and how to use lesson content in a manner which will result in learning. Summary of Chapter Two The purpose of this chapter was twofold. First, metacognitive awareness was distinguished from other forms of thinking. Second, the characteristics of explicit instruction were described. Metacognitive awareness was distinguished from metacognitive knowledge, cognition, cognitive awareness, and routinized knowledge. Specifically, metacognitive awareness of reading lesson content, closely tied to an awareness of what is taught during instruction was 48 defined as the ability to verbalize the knowledge and abilities that are learned from reading lessons, how to successfully use what was learned during instruction, why the knowledge and abilities learned from the lesson are important to possess, and the ways in which declarative, procedural, and conditional can be controlled or monitored so that improved reading abilities are acquired. The characteristics of explicit instruction include planning and task analysis based on the desired cognitive and metacognitive outcomes of the lesson, explicit communication of the declarative, conditional, and procedural knowledge of the lesson content, modeling and demonstrations of when and how to use lesson content, and practice and on-going assessment of student progress. CHAPTER THREE DESIGN AND METHODS This chapter details the data collection and analysis for this dissertation. The first section describes the background to the study. The second section describes the subjects who participated in the study. The third section describes the procedures for collecting metacognitive awareness data and the scoring of the data. Imbedded within the third section are two sub-sections: the rationale for using this scoring system and the conditions that needed to be met if this scoring system was to be an accurate assessment of metacognitive awareness of lesson content. The fourth section provides a description of each of the additional outcome measures used to assess student reading achievement, the procedures used for collecting these data, and the scoring of the data. The fifth section provides a description of the procedures for collecting and scoring data on explicit instruction and management principles. The final section describes the statistical procedures used to analyze the research questions posed in this dissertation. Background This dissertation was conducted as part of a larger 4- year research project (The Teacher Explanation Project, Institute for Research on Teaching, Michigan State 49 50 University). The primary purpose of the larger study was to investigate (a) whether training teachers in explicit explanation would improve their instruction of reading skills as comprehension strategies for third-grade low-group readers and (b) whether such training improved low-achieving students' performance on a variety of reading measures. The larger study was a variation of the typical treatment-control experimental design in that both groups received training in the effective management principles identified by Anderson, Evertson, and Brophy (1979) while the treatment group also received training in explicit explanation of basic reading skills. Classroom observation data and student interview data were collected at six different times during the 1984—85 academic year. Additional data on student achievement were collected at the end of the school year and in the fall of the 1985-86 academic year. This dissertation was specifically designed to extend the findings of the larger study by providing an in-depth examination of three levels of metacognitive awareness of reading lesson content, the development of metacognitive awareness over time and the relationship of metacognitive awareness to performance on a variety of outcome measures. 'This dissertation does not duplicate findings of the Teacher Explanation Project published elsewhere (e.g., Duffy, et al., 1987). The method for examining student metacognitive awareness 51 .of reading lesson content and the role that awareness played in reading achievement differs from the larger study in two ways. First, in the larger study, students were interviewed following their reading lessons and asked three levels of questions designed to assess their metacognitive awareness of the lesson content. The ratings used in the larger study consisted of the sum of the highest ratings for each of three categories (What the lesson was about, Why it was important to learn, and How to use lesson content effectively) regardless of the level at which the rating was given. In this dissertation, the ratings given at Level 1, Level 2, and Level 3 levels were examined. Level 1 questions were intentionally general in nature. Level 2 questions were more specific than Level 1 and intentionally directed students' attention to the metacognitive knowledge of interest in this study. Level 3 questions were the same as those at Level 2 and allowed students to refer the materials, workbooks, and story selections used during their reading lessons. Each successive level further focused students' attention on the metacognitive knowledge of interest. As such, responses at each level provide a more indepth understanding of the effects of explicit instruction on poor readers' metacognitive awareness of reading lesson content than did the larger study. The second unique aspect of this dissertation was that it examined changes in levels of metacognitive awareness over 52 time. Virtually all instructional studies examine pretest-posttest differences. However, such an analysis indicates only that instruction had an effect on metacognitive awareness. In this study, metacognitive awareness was assessed at six different points in time, to gain a greater understanding of the ways in which metacognitive awareness changes in response to explicit instruction. The third unique aspect of this dissertation was the way in which the contribution of metacognitive awareness to performance on other measures of reading ability was examined. In the larger study, significant differences between treatment and treated-control classrooms were found on five of these additional reading outcome measures. However, since a major goal of explicit instruction was to improve both metacognitive awareness of reading lesson content and reading performance, the between-group differences on these additional outcome measures may have been confounded by the effects of metacognitive awareness. This dissertation examined the performance on these additional measures independent of metacognitive awareness ratings. To do so, the variance due to metacognitive awareness of reading lesSon content was removed from the scores on each of the five outcome measures. Such an analysis provides a more detailed understanding of the ways in which metacognitive awareness affects performance and, 53 conversely, the extent to which explicit instruction directly affected performance on these additional measures. Subjects The subjects in the study were (1) 19 third-grade teachers employed by an urban school district in the midwest and one third-grade teacher from a neighboring suburban school district, each were randomly assigned to Treatment or Treated-Control Groups, and (2) the students in these teachers' low reading groups. Teachers All teachers were volunteers who responded affirmatively to a canvassing of all third-grade teachers in the district during the previous spring and summer and who had reading groups using either Skylights or Towers editions (second-grade difficulty) of the Houghton-Mifflin basal reading series. The latter criterion was necessary to insure that the low groups in all the classrooms were relatively equal in reading ability. Teachers in the treatment group received training in the explicit instruction of reading skills as comprehension strategies and in effective classroom management principles. The treated-control group teachers received training only in the use of the effective management principles. Originally, all 20 teachers were volunteers from the 54 Lansing School District. They were then randomly assigned to either the treatment or treated-control group. The one teacher from the neighboring suburban school district was recruited and assigned to the treatment group when one teacher from the treatment group became ill in mid-September (prior to the beginning of training) and dropped out of the study. All of the teachers received modest remuneration for participating. Students The students in the study were those assigned to the participating teachers' low reading groups. Participating students all returned a permission form signed by their parent(s) or legal guardian. The criteria used for assigning students to the low reading group was a combination of district policy and teacher judgement. These criteria included (a) each student's reading level at the end of the second grade (as determined by their end-of—year basal placement), (b) standardized test scores administered to all students at the end of the second grade (1984 Stanford Achievement Test scores), (c) second grade teacher recommendations for reading placement, and (c) third grade teachers' observations at the beginning of the 1984-85 school year. The project staff made no recommendations regarding low group placement. As a result, the individuals in the low group represented a typical range of reading difficulties. 55 Thus, included in the low reading groups were immigrant children with severe.second-language difficulties, mainstreamed students with particular reading or learning problems or mild behavioral disorders, and students of low intellectual ability. The average number of low group students per classroom across all 20 classrooms was 7.40 (SD = 3.79) with a range from three to sixteen students. The mean number of students in the treatment classrooms was 7.10 (SD = 3.25) and the mean number of low group students in the treated-control classrooms was 7.70 (SD = 3.92). Procedures Used to Collect Awareness Interview Data Awareness Interview data were collected through interviews with participating students. There were nine different interviewers, all members of the project staff. Each was assigned the responsibility for observing and interviewing at least two of the twenty classrooms. All interviewers had participated in a previous study using an earlier version of the interview format (Duffy, Roehler, Putnam, Wesselman, Book, Vavrus, Meloth, 1986). All interviewers received additional training in the use of the interview format designed for use in this study. 56 Collecting the Awareness Interview Data This dissertation uses an interview measure that examines students' metacognitive awareness at three different levels: Level 1--General Awareness, Level 2-- Specific Awareness, and Level 3--Awareness With Stimulus Prompts. Students were interviewed regarding their metacognitive awareness at all three levels immediately following their reading skills instruction: Level 1-- General Awareness, Level 2--Specific Awareness, and Level 3--Awareness With Stimulus Materials. A maximum of five students from each classroom's low reading group were administered the Awareness Interview following each of the six observed lessons throughout the school year. These interviews were given at approximately one-month intervals beginning in October, 1984 and ending in May, 1985. Three of the interviewed students from each classroom were termed "target" students. These students were randomly selected at the beginning of the school year and interviewed following every observed lesson. The two additional "non-target" students were randomly selected from the remaining pool of low-group students in each classroom prior to each observation. Only five students were selected because of the time it took to individually interview each student (approximately 10 to 15 minutes for each student). The number of students interviewed following any particular Observation varied due to the size of the 57 classroom's low reading group. Seven of the classrooms included five or fewer students in the low reading group (four classrooms in the treatment group and three classrooms in the treated-control group). It was not expected that fewer than five students in some classrooms would present a problem in the analysis of the awareness data for the following reasons: (1) teachers were randomly assigned to groups, (2) students were randomly selected from the low reading group for interviews (when more than five students were present in the reading group), (3) the mean number of students interviewed per classroom across all six observations was 4.52 out of a maximum of five, (4) the mean, standard deviation, and range in class size in treatment classrooms (Mean = 7.10, SD = 3.25, range = 3 to 12 low group students) did not differ significantly from those in treatedj control classrooms (Mean = 7.70, SD = 3.92, range = 4 to 16), and (5) there was never more than one student absent during the interviews from low groups in classrooms where there were five or fewer students. The interviews took place in a variety of settings (e.g., in a corner of the room, in an empty classroom or resource room, in the hallway outside the classroom). All interviews were conducted following the end of the lesson. Each interview began with a brief "warm-up" about the students' interests, plans for vacations, and so on, in an attempt to help students feel comfortable about talking with 58 the interviewer. Following this warm-up, the interviewer began asking the Level 1, Level 2, and Level 3 questions. These interviews were audio-taped and transcribed. The format of the Awareness Interview can be found in Appendix A. The following is a brief description of the interview format. Level 1--General Awareness: The interviewer asked the student to "Tell me everything you remember about the reading lesson" (or "Tell me everything you learned today in the reading lesson"). This question was always the first one posed to students. The only probes used at this level were designed to encourage the student to verbalize as much as they could remember (e.g., "Tell me more", or "What else did you learn?"). Level 2--Specific Awareness: There were three questions asked at Level 2. These were intended to focus students' attention toward the three metacognitive knowledge categories (declarative, conditional, procedural). These questions were asked immediately after the Level 1 question. The questions were the following: "What was the lesson about?", "When would you use what you learned?", and "How do you do it?". Probes designed to encourage students to articulate their responses to each of the three questions were used (e.g., "Tell me more about why the lesson was important", "Can you give me an example of how to do it?"). To avoid encouraging repetition of responses and thereby inadvertently signaling to the student that the information 59 they gave for the Level 1 question may have been insufficient or incorrect, the interviewer paid close attention to the student's response. If, in the interviewer's judgement, a student indicated a high level of metacognitive awareness of lesson content for any of the three categories in their response to the Level 1 question, the interviewer did not ask: a specific question relating to that category at Level 2. Thus, if in the interviewer's judgement, a student gave an exemplary response to the When, Why, or How categories, he specific question reflecting that category was deleted at Level 2. However, in order for a question to be eliminated, the responses in Level 1 must have indicated, in the interviewer's opinion, the optimal degree of awareness as described in the Student Metacognitive Awareness of Lesson Content rating criteria as listed in Appendix B. Level 3--Specific Questions with Stimulus Materials: These questions were the same as those posed at Level 2. In addition, students were allowed to use the actual lesson materials (worksheets, examples from the board, etc.) included during that day's reading lesson. The use of the stimulus material was intended to further focus students' attention toward the What, Why, and How categories. This optimized the opportunity to respond for students who either possessed poor awareness of the reading lesson content, could not remember particular aspects of the lesson, or who were shy when responding to direct questions from the observers. 60 Again, if the interviewer judged that the student had given an exemplary response for one or more of the three specific questions at Level 2, questions pertaining to those categories were not asked at Level 3. Scoring Awareness Interview Data All Awareness Interview transcripts were assigned a code number and any information which might suggest that transcripts were from treatment (explicit instruction and effective management principles training) or treated-control (effective management principles training) classrooms was deleted. Training raters to score the data. In order to score the transcripts accurately and reliably, four graduate students received approximately ten hours of training in rating lesson interview transcripts used in the 1983-84 study. The format of these training transcripts was similar to the transcripts rated in this study. Two project staff members conducted the training. Raters were first trained to identify where questions pertaining to Level 1, Level 2 and Level 3 began in the transcript. They were then trained to rate student responses for each of the three categories for Level 1, then to rate responses to Level 2 questions, and then Level 3 questions. If the interviewer did not ask a specific question (e.g., What, Why or How) at Level 2 or Level 3, the rating received for that category was left 61 blank. Students could receive a rating between 0 and 4 for their responses to each of three categories, with a score of 0 reflecting an absence of metacognitive awareness of lesson content and a rating of 4 indicating an exemplary awareness of the lesson. The total awareness ratings for each level were then derived by summing the ratings for each of the three categories (e.g., what, why, how). The highest total awareness rating possible was 12 (4 points maximum by 3 categories). Inter-rater reliability, using Pearson Product-Moment Correlation coefficients, was based on the sum of ratings at each of the three levels. When the reliability for each of the three levels was consistently above .80, the raters were considered ready to rate the interview transcripts used in this dissertation. After inter-rater reliability was achieved using the training transcripts, raters were grouped into two rating teams. Each team received one set of Awareness Interview transcripts from the first observation. Transcripts were randomly distributed between teams. When the teams finished rating the first set of transcripts, they received the interview transcripts-from Observation 2. Raters continued in this manner until all transcripts had been rated. Two teams were used to insure inter-rater reliability within teams as well as reliability between teams. To insure 62 within-team reliability, the two members of each team separately rated each lesson interview transcript assigned to them and then met together to resolve any discrepancies in their ratings. Discrepancies that could not be resolved were taken to the two project members who conducted the training for resolution. To insure between-team reliability, 25% of each of the six sets of Awareness Interviews were rated by both teams. Following ratings of each set, reliabilities were computed. If this ongoing reliability fell below .80, raters met with members of the project staff for additional training and re-rating of the transcripts. The reliability throughout the study was .84. Rationale for scoring deleted questions at Level 2 and Level 3. When a question at Level 2 and/or Level 3 was deleted, indicating an exemplary response for one or more of. the three categories, the raters coded the student's response to that question as a blank. For the purposes of this dissertation, blanks were later assigned the rating given to the student's response for the same category (i.e., What, Why, and How categories) at the previous level. Blanks were replaced with the rating given at the previous level because it was believed that asking students to repeat a response at each of the three levels when a sufficient response had already been given might inadvertently lead the student to conclude that his or her response was incorrect or insufficient. Changing responses to repeated questions is 63 not an uncommon occurrence (Ericsson & Simon, 1984; Hundeide, 1985). Therefore, the repetition of a question could result in a student changing his or her answer, and the changed response could conceivably be rated lower than the original response particularly if the original response was an exemplary one (e.g., a rating of 4). As this dissertation was not interested in the strength of a student's response, that is, how certain the student was that his or her response conformed to the question posed by the interviewer, questions were deleted when the interviewer believed the student's response was exemplary and all blanks were assigned the rating given at the previous level. This interview format scoring system also benefited those students who did not display a high level of awareness. Questions at Level 2 and Level 3 were only deleted if the response was judged to be exemplary. If an insufficient response for any of the three categories was given at Level 1 or Level 2, then those questions were repeated at the next level. Students who may have misinterpreted the questions posed by the interviewer, but could display their awareness when their attention was more directly focused on the information of interest, were given the opportunity to respond to additional questions at Level 2 or Level 3. Other students, with poor metacognitive awareness, would not be likely to be rated very high (i.e., a 3 or 4) at any of the three levels and consistently low awareness ratings at each 64 of the three levels would reflect this poor awareness. The following illustrates an example of the scoring system used in this dissertation. The first two examples reflect the rating given students when questions were deleted and blanks were inserted for responses at subsequent levels. Student A ratings: Level 1--What = 2, Why = 4, How = 1, Total = 7, Level 2--What = 2, Why = _, How = 2, Total = 4, Level 3--What = 4, Why = , How = 3, Total = 7, Overall rating = 18 (The Blank categories represent questions deleted during the interview) Student B ratings: Level 1--What = 0, Why = 0, How = 0, Total = 0; Level 2--What = 2, Why = 1, How = 4, Total = 7; Level 3--What = 4, Why = 3, How = , Total = 7 Overall rating = 14 (The blank catEgories represent questions deleted during the interview) Student C ratings: Level 1--What = 0, Why = 0, How = 0, Total = 0; Level 2--What = 1, Why = 0, How = 0, Total = 1; Level 3--What = 1, Why = 1, How = 0, Total = 2; Overall rating = 3. Student A's response to the Level 1 question indicates that she possesses a moderate to above average awareness of the lesson content, with a particularly strong awareness of why the lesson content was important to learn. Consequently, the "Why" question was not asked at Level 2 and Level 3. Conversely, Student B's response to the Level 1 question was extremely weak and he was therefore asked all questions at 65 Level 2. Because his response to the How category question at Level 2 was very strong, he was not asked the same question at Level 3. Student C's ratings are low throughout all categories and levels and as a result she was asked all questions possible. Now consider the same students' awareness ratings when the responses to the deleted questions are substituted for the blanks: Student A ratings: Level 1--What = 2, Why = 4, How = 1, Total = 7; Level 2--What = 2, Why = 4, How = 2, Total = 8; Level 3--What = 4, Why = 4, How = 3, Total = 11; Overall rating = 26. Student B ratings: Level 1--What = 0, Why = 0, How = 0, Total = 0; Level 2--What = 2, Why = 1, How = 4, Total = 7; Level 3--What = 4, Why = 3, How = 4, Total = 11; Overall rating = 18. Student C ratings: Level 1--What = 0, Why = 0, How = 0, Total = 0; Level 2--What = 1, Why = 0, How = 0, Total = 1; Level 3--What = 1, Why = 1, How = 0, Total = 2; Overall rating = 3. Student A gave an optimal response to the Why category question at Level 1 and the original blanks at Level 2 and Level 3 were replaced with the rating she received at Level 1. For Student B, all What category questions and Why category questions were asked at Level 2 and Level 3 but because of an exemplary response at Level 2 for the How category question, this question was deleted at Level 3 and 66 the rating from Level 2 was inserted. Thus, even though Student B's awareness was extremely low at Level 1, he was allowed ample opportunity to display his awareness. As the questions became more specific, the student could recall and verbalize the important lesson content. Student C's ratings remain unchanged as she did not have any questions deleted from her interview. As a result, student's ratings for each of the three levels provides an accurate picture of their awareness of lesson content. The risk of misdirecting some students' responses is reduced when questions are deleted from the interview. Others, who have not "caught on" to the intent of the question posed by the interviewer, are provided more opportunity to respond. Those possessing very poor. metacognitive awareness reflect this in their consistent1y_ low ratings. Consequently, the ratings received by students through the use of this interview system and rating scale should be accurate representations of student metacognitive awareness of lesson content. Conditions necessary for using the scoring system in this dissertation. While the above rating system appears to be an effective approach to assessing student awareness, two conditions had to be met before the scoring system was to be used in this dissertation. First, there must not have been an excessive number of questions deleted at any one level. The opportunity to respond at Level 2 and/or Level 3 is 67 dependent on the responses given at the previous level(s). If a great many students gave exemplary responses for any of the three categories at Level 1, for example, then a large number of questions would be deleted and it would not be possible to determine whether the same response would have been given at the subsequent level(s). The convention adopted in this dissertation (i.e., the deletion of questions) was that there should be less than a ten percent deletion rate. Second, the questions asked at Level 2 and Level 3 are dependent on the accuracy of interviewers' \ judgments of student responses. As a result, the interviewers could delete questions at Level 2 and Level 3 when responses were not exemplary. The ratings of these interviews, then, would be subject to bias because students were not given ample opportunity to display their awareness. Thus, it was necessary for this dissertation that a high percentage of the responses upon which deleted questions were based received exemplary ratings (e.g. a rating of 3 or 4). The convention adopted for the interviewers' judgement of responses was that the ratings for responses judged to be exemplary should not fall below 90%. Both conditions were met: few questions were deleted during the interviews and those that were deleted received exemplary ratings. Throughout the entire study (Observations 1 through 6), a total of 540 interviews were conducted. For any one student, the maximum possible questions that could 68 have been deleted during any one interview was six (i.e., three questions at Level 2 and three questions at Level 3). Thus, the total number of questions that could have been deleted for all students across all six observations was 3,240 questions (540 interviews by three Level 2 questions by three Level 3 questions). Only 123 (3.796%) questions were- intentionally deleted from the interviews. Of these deleted questions, 77 were declarative knowledge questions (what the lesson was about), 33 were conditional knowledge questions, and 23 were procedural knowledge questions (7.129%, 3.055%, 2.129%, respectively). A greater proportion of questions were deleted at Level 3 (99 questions, 6.111%) than Level 2 (1.296%). Of the 123 deleted, 118 questions (94.10%) were . rated as 3 or 4. Of these 118 questions, forty were rated as 4 and seventy-eight were rated as 3. Thus, out of 3,240 questions rated, the low number of deleted questions suggest that student responses were free to vary across the three levels and interviewer judgments were accurate in assessing the quality of responses. The observation where the greatest number of questions were deleted was Observation 6. Eighty-five students were interviewed with a total of 510 possible questions deleted (85 students by three questions at Level 2 by three questions at 3 Level). A total of 23 questions (4.511%) were deleted. Of these deleted questions, 15 were from the What category (declarative knowledge), seven were from the Why category 69 (conditional knowledge), and one was from the How category (procedural knowledge). The percentage of deleted questions were 8.823%, 4.112%, and 0.588%, respectively. A greater proportion of questions were deleted at Level 3 (18 questions out of a possible 255, or 7.058%) than Level 2 (5 questions, 1.961%). Of the twenty-three questions deleted, 91.30% (21 questions) were rated as 3 or 4. Of these twenty-one questions, seven were rated as 4 and fourteen were rated as 3. Thus, out of 510 questions rated, the low number of deleted questions suggest that student responses were free to vary across the three levels, ratings at one level were not overly dependent on ratings at the previous level, and interviewer judgments were accurate in assessing the quality of responses. Consequently, ratings at one level of metacognitive awareness were not overly dependent on ratings at previous levels and, as a result, the scoring system used in this dissertation was expected to give an accurate picture of student metacognitive awareness of lesson content. Data Collected on Additional Reading Outcome Measures There were five additional outcome measures of reading ability included in the analysis for this dissertation: the Supplemental Awareness Measures--Process Items (SAM-- Process), the modified Graded Oral Reading Paragraph (GORP), the Concepts of Reading Interviews (or Concepts Interviews), the Stanford Achievement Test--Word Study Subtest (SAT--Word 70 Study), and the Michigan Educational Assessment Program-- Reading Subtest (MEAP). All data were aggregated by classroom and, as reported elsewhere (Duffy, et al., 1987) all five measures indicated significant differences favoring the treatment group. Two of the measures, Supplemental Awareness Measure (SAM--SAM Process) and the Graded Oral Reading Paragraph (GORP), were specially prepared measures of students' awareness of the strategy they believe to be necessary to successfully comprehend a particular text. A third measure, the Concepts of Reading Measure, was also specially prepared as a more general measure of student's metacognitive awareness of the components of successful reading comprehension. The other two measures, the SAT--Word Study and the MEAP, were not specially prepared. They are more commonly used measures in the assessment of reading achievement. These measures require the student to demonstrate an understanding of word parts, correct word meanings, main ideas, sequencing of events in a passage, and so on, with the assumption that some degree of metacognitive awareness is necessary to arrive at the correct answer. The three target students selected at the-beginning of the year were administered the specially-prepared measures-- the SAM--Process, Concepts Interviews and the GORP. It was not possible to administer these measures to all low-group 71 students in the twenty classrooms because of the time it took to administer each of these measures (approximately 15 to 30 minutes for each student). However, because these target students were randomly selected, they are representative of students from their low-reading group. These five measures are described below in the order of their similarity to the type of knowledge and skills taught by treatment teachers in the study. The closest in similarity is the SAM which required the student to select a specific strategy or process that could be used to solve a problem in reading. This type of activity was a major emphasis in the treatment group. The SAT and the MEAP, on the other hand, required the student to correctly identify various base words and their prefixes or the important content in a passage. Two other reading outcome measures, the SAT—- Comprehension Subtest and the SAM--Content Items, were included in the larger study but were not used in this dissertation because no significant differences between groups had been found (Duffy, et al., 1987). Consequently, performing stepdown F-tests, as described in Chapter One and in this chapter under Data Analysis, would not be of value because the stepdown F-tests were intended to remove the variance due to-metacognitive awareness in order to determine whether the results would indicate non-significant differences between groups. 72 Supplemental Achievement Measure (SAM) The Supplemental Awareness Measure, consisting of a set of two questions for each skill (e.g., compound words, main idea, etc.), were especially designed for use in the larger study. Each set of items corresponded with reading skills lessons taught during the study. The first question, termed the SAM--Content Item, was typical of reading skills items found in basal textbooks, the MEAP and the SAT, and required the student to choose, for example, the two words that made up the compound word or the sentence that best described the main idea. The second question, the SAM--Process Item, asked the student to select from a set of strategies the one that he/she used to figure out the answer to the SAM--Content Items. Collecting SAM the Data. The SAM items were administered to the three target students a maximum of three times, beginning in March, 1985. Each administration contained different items. The number of items each student was given depended on the number of skills lessons covered in each classroom (each teacher covered the content at his or her own pace). After a period of three to four weeks, when several new reading skills had been taught, SAM's were administered to the target students. Because of the number of skills covered during the school year and time involved in administering the test (approximately 15 to 20 minutes per student for each administration), usually no more than 15 73 sets of items were given to students during any one administration.’ Appendix C provides an example of a typical pair of SAM items. Scoring the SAM Data. A simple percentage correct score was derived for the SAM--Content and SAM--Process measures by dividing the number of correct answers to each by the total number of items administered for each part. A multivariate analysis of variance (MANOVA), using the Content and Process items as dependent measures was used to analyze the data. Significant differences favoring treatment classrooms were found (F(2,17) = 6.688. p = .0072). The SAM-—Process portion was found to be significant (F(1,18) = 13.331, p = .002). The SAM--Content portion of the measures was not significant (F(1,18) = 0.18, p = .674). While there is no guarantee that the strategy the student selected in the second question was the one actually used by the student when responding to the first question, it did require the student to identify a particular strategy that could be used to correctly answer the first question. In this way, the SAM's indicate whether the student can recognize specific reading strategies for solving commonly encountered reading comprehension problems. The Graded Oral Reading Paragraph Measure (GORP) The GORP, an "on-line" measure used by many reading specialists (Clay, 1972; Wixson & Lipson, 1986), was 74 administered as a pre- and post-test to all target students. This measure provided an assessment of students' strategic reading behaviors, e.g., the strategies that were used to figure out unknown words and eliminate disruptions while reading. Two components were included in the GORP: student's self-corrections while reading and their responses to two embedded words in the paragraph ("grub" and "uncovered"). The GORP passage, listed in Appendix D, was selected from the third grade version of the Houghton Mifflin Placement Test. This particular paragraph was selected from this placement test beCause all students in the study used the Houghton Mifflin basal text series, none had previously read the passage, and two of the project staff members, based on their expertise as reading specialists, judged the passage to be of sufficient difficulty for students in the study. Collecting the GORP Data. The GORP was individually administered to the target students. Each testing session began with a brief warm-up by asking students to identify approximately 30 sight words provided by the Houghton Mifflin Placement Test. The test administrator then showed each student the first of the embedded words ("grub) and asked the student to pronounce it and use the word in an original sentence. If the student mispronounced the word, the test administrator provided the student with the correct pronunciation. The same procedure was used for the second embedded word ("uncovered"). No students used the word 75 "grub" (an insect) in a manner similar to the "grub" was used in the paragraph, although most students knew the meaning of the word "uncovered". Following this, the test administrator asked the student to read the paragraph aloud and remember what he or she had read. As the student read, the test administrator recorded the instances of self-corrections and hesitations. After the student was finished, the test administrator questioned the student about his or her self-corrections. These questions were intended to elicit how the student knew a self-correction was necessary and what strategy was used to make the correction. There were between two and five self- corrections per student. Finally, the test administrator asked the student to define the word "grub" and how he or she figured out the meaning (since it was not known prior to. reading the passage). The test administrator then asked the student how he or she would figure out the meaning of the word "uncovered" if it was not one of the words that were self-corrected. Scoring the GORP Data. Using the audio-tapes of the testing session, the test administrator noted (1) the number of strategies each student described for both the self- corrections and the two embedded words, (2) whether each of these strategies focused on word meaning or word recognition, and (3) the percentage of each student's strategies judged to I reflect strategic reading behaviors. To establish inter- 76 rater reliability, twenty percent of the GORP transcripts were rated by both test administrators. The reliability was 0.82. A one-way univariate analysis of variance, using the aggregate of the three components listed above as the dependent measure, revealed no differences between students in treatment and the treated-control groups for the pre-test (F(1,18) = 0.236, p = .879). Significant differences on the post-test favoring treatment classrooms were found (F(1,18) = 37.268, p < .001). Concepts of Reading Interview The Concepts of Reading measure indicates a general awareness of the metacognitive components of reading. Collecting the Concepts Interview Data. The target students interviewed at the end of the year were asked four questions: What do good readers do?, What is the first thing you do when you read?, What do you do when you come to a word you do not know?, and What do you do when you come to a sentence you do not understand? Each interview took approximately ten to twenty minutes. These questions were designed to provide the student with an opportunity to describe his/her knowledge of strategic reading before and during reading. Scoring the Concepts Interview Data. A four step procedure was used to rate responses to the Concepts Interviews using procedures suggested by Ericsson and Simon 77 (1984). First, a content analysis of responses to each question from a random sample of sixteen interviews (26.67%) was conducted by four staff members. This allowed the staff members to identify the range of responses made by the students. Second, the staff members discussed the responses to the four interview questions to identify possible categories of general reading concepts. Next, ten categories were decided upon and a seven-point scale was constructed (See Appendix E). Finally, the four staff members examined the responses given in sixteen interviews to check whether the Likert scale differentiated between responses for the ten categories. Two of the staff members then divided the Concept Interview transcripts and rated them using the Likert scale. To establish inter-rater reliability, twenty-five percent of the transcripts were rated by both staff members. Reliability was 0.81. Significant differences favoring treatment classrooms were found (F(1,18) = 6.011, p = .025). Stanford Achievement Test (SAT) The Stanford Achievement Test is a standardized reading measure given to all students in the Lansing School District. Collecting the SAT Data. The SAT--Comprehension Subtest and Word Study Subtest were administered in May of 1984 and served as a pretest. Both subtests were administered again in May of 1985 and were used as a posttest measure. The comprehension sub-test consists of a set of short paragraphs 78 and two or three questions that require the identification of important content contained in the paragraph (e.g., main ideas, sequence of events). The word study skills subtest assesses students' ability to identify prefixes, suffixes, base words, and word parts using word attack skills. Scoring the SAT Data. The data used in this study were scaled scores. Tests were analyzed by the school district and copies of individual student scores were provided to the larger study. Scaled scores were used because they represent standardized scores based on national norms. Scaled scores were derived by dividing the number of correct items by the number of items answered by each student. These were felt to be more representative scores than raw scores because raw scores reflect the number of correct items out of all possible items. Thus, raw scores are not indicative of the number of attempted items, only those answered correctly. A one-way multivariate analysis of variance, using the two subtests as dependent measures revealed no significant difference between groups (F(2,18) = 1.288, p = .301) for the 1984 SAT Subtest. A multivariate analysis of covariance, using the 1984 subtests as covariates revealed significant differences favoring treatment classrooms on the 1985 post- test (F(2,16) = 4.16, p < .05). The univariate F-tests revealed significant differences between groups for the Word Study Subtest (F(1,16) = 13.149, p = .002), but not the Comprehension subtest. 79 The Michigan Educational Assessment Measure (MEAP) The Michigan Educational Assessment Program is a non- standardized criterion-referenced test developed for use in all Michigan public schools (additional MEAP tests are also given to 7th and 10th graders). The format and test items are similar to those included on the SAT and more closely reflects whether students attain the grade level reading skills objectives set by the state of Michigan. Collecting the MEAP Data. The MEAP is a state-wide test and all fourth grade students in Michigan are required to take the test at the beginning of the academic year. Thus, all students in the study were administered the MEAP. The MEAP test was administered when the students in this study first entered the fourth grade (October, 1985) and is considered a test of maintenance. Scoring the MEAP Data. Scores for each student were provided by the school district. These scores reflected the number of criteria passed on each of 34 reading criteria in the MEAP. A passing score for these criteria was 75 percent correct. Significant differences favoring treatment classrooms on the MEAP were found (F(1,18) = 5.723, p = .029). 80 Data Collected on Explicit Instruction and Classroom Management In addition to the Awareness Interviews and the additional measures of student achievement described above, data were also collected on explicit instruction and effective management principles. Both treatment and treated- control groups received training in effective classroom management principles described by Anderson, Evertson, & Brophy (1979). These principles have been found to improve student time-on-task and are correlated with improved achievement. Training was intended to insure equal time was spent learning basic reading skills in both groups. Treatment teachers also received training in explicit explanation of reading skills using a staff development program developed by project members. The major component of explicit explanation is to inform students of what the lesson is about in a detailed manner, why the reading skill taught during the lesson is an important one, when the skill can be used in "real" reading situations, and how to use the skill as a strategy to assist in comprehension. Because they were given this information, students were expected to become more metacognitively aware of strategic reading actions that would be beneficial under a variety of conditions. There were four advantages of both groups receiving training. First, both groups are provided with information 81 which will help to develop instructional management behaviors which increase engaged learning. Such instructional behaviors have been associated with improved reading achievement. Second, because both groups received training in effective management, any differences between students favoring the treatment group should be more directly attributable to the different type of instruction (i.e., explicit instruction), and not due to an unexpected by-product of explicit instruction such as time-on-task because both groups should be equal on that variable. As a consequence, any student differences between groups may be more attributable to the effects of explicit instruction training. This design has the added benefit of providing greater statistical power in identifying the impact of explanation on student outcomes because the management principles have been associated with improved reading achievement. Collecting the Teacher Data. The nine staff members of the Teacher Explanation Project were assigned two to three teachers each. The first observation occurred in early October, 1984. Treatment teachers were observed 12 times throughout the 1984-85 school year at approximately two to three week intervals. Treated-control teachers were observed six times during the year at approximately four to six-week intervals. Each of the six observed treated-control lessons coincided with every other observed treatment lesson, 82 beginning with the first (baseline) lesson. The additional observations for the treatment group were considered monitor lessons and were used to monitor the teachers' progress and provide feedback on their explicit instructional behaviors. These observations were not included in the analysis of the data. Explicit instruction data and management data were obtained during these observations. The lessons taught during these observations were the same lessons the students were interviewed about during the Awareness Interviews. Following the random assignment to treatment and treated-control groups in October of 1984, baseline observations of teacher explanation and classroom management principles were conducted. Following this, each group received training in two-hour blocks one afternoon per week for four consecutive weeks. Both groups continued to receive additional two-hour training sessions in approximately one-month intervals. The criteria for explanation and management can be found in Appendix F and Appendix G, respectively. Training raters to score the data. The procedure for rating teacher transcripts was similar to the rating of Awareness Interviews. Six raters, all graduate students, rated transcripts used in previous studies of explicit instruction. The possible ratings of teacher explicitness ranged from 0 to 44 (4 points maximum for each of 11 categories of explanation). Training consisted of five one 83 to two hour sessions in which staff members modeled how to rate the transcripts and supervised raters as they rated successive, identical transcripts. When reliability among raters was consistently above .80, raters began to rate the transcripts collected for the larger project. The six raters were then divided into three teams of raters. Each team was given one-third of the transcripts and each member individually rated his or her transcript set. The team members then met to resolve any discrepancies. Twenty-five of the transcripts were rated by all three team members to insure inter-rater reliability. This reliability was .81. Scoring the Teacher Data. The observed lessons for all 20 teachers were audio-taped and transcribed. When observations were conducted, the observers used the management principles checklist, consisting of 18 categories, to record the extent to which these principles were present during the lesson. Observers coded each of these principles as either "present" or "absent" during their classroom observations. This same lesson was audio-taped, transcribed and rated for eleven dimensions of explicit instruction. As reported elsewhere (Duffy, et al., 1987), the results of this training were successful. A one-way analysis of variance (ANOVA) procedure was used to examine the use of management principles. There were no differences between the ' treatment and treated-control teachers in their use of these 84 principles at any observation during the course of the study (See Table 3.1). To examine explicit instruction, a one-way ANOVA was performed on the initial (baseline) ratings and analysis of covariance (ANCOVA) procedures, using the ratings for Observation 1 as the covariate, was conducted to examine ratings at each of the five subsequent observations (ANCOVA was used to increase the power of the F-test). There were no significant differences between treatment and treated-control teachers at the beginning of the study on their ratings of explicit instruction (F(1,18) = 3.578, p = .061). Significant differences were found favoring the treatment group at Observation 3 (F(1,17) = 24.369, p < .001) and continuing through Observation 6 (F(1,17) = 6.118, p = .024). The means and standard deviations for each observation can be found in Table 3.2. These results suggest that students in the treatment and treated-control classrooms experienced equal time in reading skills instruction and that students in treatment classrooms received explicit instruction which placed a greater emphasis on metacognitive awareness of lesson content. Thus, any differences between groups in the student metacognitive awareness ratings and the additional outcome measures discussed in this dissertation can be attributed to the effects of explicit instruction. 855 Table 3.1 Means and Standard Deviations for Management Principles ¢ Lon E Observat;:n : ' 7 iii; a Group Observation 1 Observatzcn 2 Observaticn 3 Cbse 1 (3D? I (SEE l iSE) X :53) (It a; C‘ ('1 Treatzent 31. E SE 76 53 09 31 20 ".33 ".33 '19.46. (20 30) 29 79j 415 13; .21 81 21.39 Ireatec-Ccntrcl 51.36 68.40 16.70 32.90 33.40 37.3: {19 ‘8) (24 .2} £21.38: 13 45, :19.37. f1'.S=. thal 51.34 68.55 ’ .35 :2.35 3..19 :. 43 615.51: {23.16) 25 78. :5 72: :1 39; .2..:.; Table 3.2 Means and Standard Deviations for Explicit Instruction Ratings Grcup observation 1 Observaticn 2 Observatl n 3 abservazlcn 1 335 IVEELCE 3 Observe 1 (SD) 1 (SD) 1 (SD; 1 2501 l {59) Treataent 13.10 :1.60 19.90 -1..o 1:.50 7.7: 15.50) i8.01; '9 00} 15 953 5.55 6.99? Treated-Central 9.50 11.03 10.30 10.30 9.10 9.10 (2.431 (5.62) {2.41) 23.13: (3.55) {3.14: Total 11.45 14.3 13.10 15.30 13.35 .3.43 £4.45} (7.34) 1"6.18) i‘ 55. {6.63 15.9 87 Data Analysis The overarching research question for this dissertation was: What are the differences at each level of metacognitive awareness between recipients of explicit instruction and those who did not receive such instruction, what are the changes in levels of metacognitive awareness over time, and how does metacognitive awareness affect performance on other measures of reading? A variety of analysis procedures will be used to answer each of the specific research questions. Each of the specific questions and the procedures used to answer these questions will be described below. Research Questions 1 through 3: Analysis of Levels of Metacognitive Awareness The first three research questions concerned the differences between groups in ratings of metacognitive awareness of lesson content. The same analysis procedures was used to answer each of these three research questions. These questions were: Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 1 Awareness ratings? Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 2 Awareness ratings? 88 Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 3 Awareness ratings? All three questions were examined using a single one-way multivariate analysis of variance procedure (MANOVA), with the three levels as dependent measures. The MANOVA procedure was used instead examining each level using a univariate analysis of variance procedure (ANOVA) because of its greater power when the null hypothesis of no differences between means is rejected. If this MANOVA was found to be significant, the three individual univariate F-tests were then examined in order to determine which of the three levels contributed to the significant multivariate effect. Researcthuestion 4: Analysis of Changes and Trends The fourth research question concerned the changes in Level 1, Level 2, and Level 3 Awareness Interview ratings across the six observations and trends of these-awareness ratings across the six observations. The research question was: What is the trend in the ratings received at each of the three levels of metacognitive awareness of reading lesson content across the six observations? To investigate this research question, individual 89 repeated measures MANOVAs were performed for Level 1, Level 2, and Level 3 ratings. The ratings at each observation served as dependent measures. The analyses provided information regarding (1) the observation at which significant between group differences were first apparent, (2) whether there was a significant trend in the ratings for each of the three levels across the six observations, and (3) the type of trend that was present (i.e, linear, uadratic, cubic, quartic, or pentic). The trend analysis prov1des information regarding whether (a) the interaction between group membership (treatment and treated-control) and time changed at a different rate across the six observations, (b) the averaged ratings across all observations were different between groups, and/or (c) significant changes in awareness ratings across time (observations) were present irrespective of group membership. The findings for Level 1, Level 2, and Level 3 Awareness Interview ratings will be reported separately. However, as the trend analyses are complex to describe, the following four-step "decision frame" will describe the reporting of the trends in a step-by-step manner. The first step in examining trends is the examination of Bartlett's test of sphericity. This statistic examines the within-cell correlations (i.e., the correlations among the 12 cells--six observations by two groups). If this statistic is significant, then there are significant within- cell correlations. Consequently, the 90 multivariate F-tests for group by time interactions, averaged effects for group, and effects for time, can be examined. If any of these multivariate F-tests are significant, then further examination of the appropriate univariate F-tests, as described under Steps 2 through 4, is warranted. If this F- test is non-significant, then the requirement that significant multivariate F—tests be present before examining the individual univariate F-tests is needed. The second step in interpreting the results of the trend analysis is to examine the interaction effect. The multivariate F-test for the interaction is first examined (assuming a significant Bartlett test). If the multivariate F-test is significant, then the individual univariate F- tests which indicate the function of the trend (e.g., linear, quadratic) can be examined. If the Bartlett test is non-significant, the multivariate test is by—passed and the univariate interaction trends are examined. If significant univariate F-tests are found, the change in the ratings between the two groups changes at a different rate across time. If a significant interaction effect is found, the averaged ratings assessing differences between groups and the trend(s) for time cannot be reported as they are confounded due to the significant interaction. The third step involves the examination of the between- group differences for averaged ratings across the six observations. This analysis can only be used if the interaction between group and time is 91 not significant. This analysis is analogous to a one-way ANOVA but uses the averaged ratings (i.e., the grand mean) of the six observations for each group. If significant differences between groups are found, the results indicate that one group's ratings were, on average, greater than the other group's ratings. Finally, the fourth step involves the analysis of the trends across the six observations irrespective of group membership. This analysis is also appropriate when no group. by time interactions can be found. However, it is not dependent on finding significant differences between the averaged group ratings. If the Bartlett test is significant, the multivariate test is examined. If this proves to be significant, then the individual univariate F- tests are examined. If the Bartlett test is non- significant, then the multivariate F-test is ignored and the individual F-tests are examined. This analysis indicates whether the ratings for the two groups changed over time and, if so, what the function of the trend was like (e.g., linear, quadratic). Research Questions 5: Analysis of Additional outcome Measures The final research question investigated the amount of variance attributable to the awareness ratings in each of the five additional outcome measures. The research question was: If the variance due to metacognitive awareness ratings at those levels where significant 92 differences between groups were found is removed from the five additional outcome measures where significant differences were also found, do the differences between groups on the additional outcome measures become non-significant? To investigate this research question, stepdown F-tests were performed using only those levels at Observation 6 where significant between-group differences were fouid. The stepdown procedure removed the variance due to awareness ratings from the scores of the five outcome measures. The results of this analysis will indicate the extent to which explicit instruction affects performance on the additional measures of reading independent of metacognitive awareness ratings. Summary of Chapter Three This chapter has described the methods and procedures used in this dissertation that are intended to answer the research questions. Low-group third-grade students from twenty classrooms and the teachers from these classrooms participated. Up to five students in each classrooms were individually interviewed following each of six observed lessons throughout the 1984-85 school year. These students were asked three levels of questions designed to assess their metacognitive awareness of reading lesson content. Four additional reading outcome measures were administered toward the end of the 1985 academic year and one additional measure 93 was administered early fall of the following school year. To analyze the data, multivariate analysis of variance (MANOVA) procedures were performed on ratings for the three levels of metacognitive awareness. This analysis examined whether there were differences at each of the three levels. A trend analysis was then conducted to examine the changes in metacognitive awareness ratings over the course of the school year. Finally, each of the five additional outcome measures was then examined after the variance due to metacognitive awareness ratings was removed. This analysis revealed the extent to which metacognitive awareness affected performance on these additional measures. CHAPTER FOUR RESULTS Introduction The purpose of this study was to examine third grade poor readers' levels of metacognitive awareness and the relationship between metacognitive awareness and performance on additional measures of reading performance. The general research question that guided the formulation of the specific research questions and the procedures used to answer these research questions was the following: What are the differences at each level of metacognitive awareness between recipients of explicit instruction and those who did not receive such instruction, what are the changes in levels of metacognitive awareness over time, and how does metacognitive awareness affect performance on other measures of reading? To answer the research questions, data were collected on metacognitive awareness of reading lesson content at six different points during the school year. These data were obtained through an interview measure which assessed student metacognitive awareness at three different levels: Level 1--General Awareness, Level 2--Specific Awareness, and Level 3--Awareness With Stimulus Prompts. Each level was designed to elicit responses regarding metacognitive awareness of lesson content and each successive level intentionally focused students' attention more directly toward what the 94 b.» :1 .l flu 0f in th; 95 lesson was about, why the lesson was important, and how to use the lesson content successfully. Data were also collected on four additional measures of reading performance near the end of the school year and on one measure in the fall of the following school year (i.e., the fourth grade). This chapter presents the major findings of the study. The findings are organized into three sections corresponding to each of the research questions posed in Chapter One. The first section describes the results of the first three research questions regarding between-group differences at Observation 6 for Level 1, Level 2, and Level 3 awareness ratings. The second section examines the differences in metacognitive awareness at each of the six observations and the trends of these changes across time. The third section examines the relationship between metacognitive awareness ratings at Level 1, Level 2 and Level 3 ratings and performance on the additional measures of reading. Analysis of the Three Levels of Metacognitive Awareness In this section, the results of the analyses examining differences between levels of metacognitive awareness ratings of students whose teachers received training in explicit instruction (treatment classrooms) and students whose teachers did not receive such training is presented. The three following research questions are: Are there significant differences at Observation 6 in metacognitive awareness of reading lesson 96 content ratings between students in treatment and treated-control classrooms for Level 1 Awareness ratings? Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 2 Awareness ratings? Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 3 Awareness ratings? To investigate the above three questions, a one-way multivariate analyses of variance (MANOVA), using the three levels of awareness at Observation 6 as dependent measures was performed. The MANOVA revealed a significant main effect favoring the treatment group (F(3,16) = 10.528, p < .001). To determine which of the levels were significantly different, the individual univariate F-tests were then examined. These F-tests revealed significant differences for Level 2 (F(1,18) = 19.899, p < .001) and for Level 3 (F(1,18) = 22.276, p < .001). Near-significant differences were found for Level 1 ratings (F(1,18) = 3.513, p = .077). Table 4.1 gives the means and standard deviations of the ratings at Observation 6. Table 4.2 provides information regarding the univariate ANOVA's. Summary of Findings for Levels of Awareness The results indicate that the significant main effect found in the MANOVA analysis was attributable to significant differences favoring the treatment group at Level 2 and Level 97 Table 4.1 Observation 6 Means and Standard Deviations for Level 1, Level 2 and Level 3, Awareness Interview Ratings Level 1 Level 2 Level 3 X (SD) X (SD) X (SD) Treatment 2.497 5.250 5.945 (1.078) (1 00) (1.246) Treated 1.602 2 432 0 Control (1.058) (1 196) (1 3) Total 2.049 3.841 4.517 (1.136) (1.995) (1.970) 98 Table 4.2 Univariate Analysis of Variance for Level 1, Level 2 and Level 3 Awareness Interview Ratings at Observation 6 It. ' . 7 hill 01 511! CI M831} 5183“ T v- . = . ' a. I:- :qua.ss 3:113:85 .CEUEI'E :cuare :--ll..'. ii": a- Q Q I A q_ u so.” .. ' .- 61-11465" LIFO!" 281;!!ch ..I'3’.' :"Jcl‘u: I: : p 9 1 ‘ An A, a 1"... a p -- LEVEI L '05 .3 :19 - 3.3 . ..9 3.3. E .V ' A no no I, a a1 a q -nr 1: 1- 151:. . 39.715 -5.9.5 :9 . S 1.39: .:.39: 003 ‘I a :‘ 0‘. a, H 0 ‘4' I:- a any-[.- ‘upa- Le‘l'e- J :CISUJ 3b 9]. 4L'5J' 105V .6. -i: {“1 Degrees of Ereedonz Between = 1; Error 2 15 99 3, suggesting that treatment group students were not rated as possessing greater metacognitive awareness of reading lesson content unless specific post-lesson interview prompts (Level 2) and'specific post-lesson interview prompts with stimulus materials (Level 3) were used. Analysis of Changes and Trends in Metacognitive Awareness The fourth research question concerned differences in awareness ratings between groups for each of the three levels and for awareness ratings across all six observations. The formal research question was: What is the trend in the ratings received at each of the three levels of metacognitive awareness of reading lesson content across the six observations? To investigate this research question, two sets of MANOVA's were performed for Level 1, Level 2 and Level 3 Awareness ratings. The first MANOVA in each set was a simple one-way MANOVA which used the total ratings (sum of the three categories) at each of the six observations as dependent measures. This analysis revealed the observation at which significant between group differences were first apparent. The second MANOVA was a repeated measures trend analysis procedure which examined whether there was a significant trend for each of the three levels and for overall ratings and the type of trend that was present (i.e, linear, quadratic, cubic, quartic, or pentic). 100 Changes and Trends for Level 1 Changes in Level 1 ratings across time. The analyses for Level 1 examined the ratings between treatment and treated-control groups in response to the general question "Tell me everything you can remember about the lesson you just had". Table 4.3 gives the means and standard deviations for ratings at each of the six observations. The results of the first MANOVA examining the differences between groups on awareness ratings at each of the six observations was not significant (F(6,13) = 1.530, p = .244). Trends for Level 1 Ratings across time. The trend analysis revealed that the Bartlett Test of Sphericity was non-significant (p = .459). Therefore the multivariate F-tests were ignored. There were no significant group by. time interaction effects. Consequently, the main effect for the averaged ratings across observations between groups and the univariate trends for time can be examined. A significant main effect for group revealed that the treatment group's averaged ratings for the six observations was greater than the averaged ratings for the treated-control group (F(1,18) = 8.784, p = .029). This analysis also indicated that there was a significant linear and quadratic trend for“ time (F(1,18) = 14.292, p = .001, and F(1,18) = 8.642, p = .009, respectively). These results suggest that the ratings for both groups at Level 1 tended to increase across the six 1(31. Table 4.3 Means and Standard Deviations for Level 1, Level 2, and Level 3 Awareness Interview Ratings Group Ozservatzcn 1 Cbssrvazzan 2 Observation 3 Observation 4 CoservatLon 5 Observatzon 6 Z :59) X 550) Z (59? 1 {SE} 3 591 Z 552i ”roarzert 1.545 1.385) '.512) 1.6461 1 4~.; 1.0'3: 36761 - 1.375 3. 37 s 359 4.373 4.320 5._50 1.514; (2 7C201 1 76:, 11.33‘: 11 9491 {1.593 Level 1 3.355 3 «'5 5.015 2.187 5.427 ‘ 345 11.557. (1.. . .1.532) 11.227 1.553.1 ..:..= Tre‘2ed-Con2rol Level 1 .372 672 1.150 ..052 1.515 . :.2 (.653) 1 553) ‘.9éii .9791 ‘1. 93: 1 3.5 Level 2 2.399 2.251 2.929 1 479 3.333 2.422 1 104) (1.6331 11.5381 1 338. 51.591 :1 19*f Level 3 3.198 3.“2 4.250 3.553 4.1-5 3 008 (.999: :1. 1 1 1.975) '1.255; '1 :0 11 45’: Total Level 1 1.079 .983 1 1 1.435 1.337 2 349 (.669) (.787) 1 1 .356} {1.2."' xl.136: Level 2 2.381 2.691 3.640 3.67 3.952 3.841 {I 436) (1 556) :1 ’71) 41. . 1 (1.717) {1.955} Level 3 3.542 3.55 ‘ 1 (x: O“ on *4 H “3- 1 n N “a N b— --J «I A ...e .3)- (1' fl 5 0‘ (AI " H (I! 102 observations. Figures 4.1 and 4.2 provide the ratings and trends of the two groups across observations at Level 1. Changes and Trends for Level 2 Changes in Level 2 ratings across time. The analyses for Level 2 examined the ratings between treatment and treated-control groups in response to the three specific questions: "What was the lesson about?" (declarative knowledge), "Why was it important to learn?" (conditional knowledge, and "How do you use what you learned?" (procedural knowledge). Table 4.3 gives the means and standard deviations for ratings at each of the six observations. The results of the first MANOVA examining the differences between groups on awareness ratings for each of the six observations was significant (F(6,13) = 3.798, p = .021). Students in the treatment group were rated significantly greater than students in the control group beginning at Observation 4 and continuing through Observation 6. Trends for Level 2 ratings across time. The trend analysis revealed that the Bartlett Test of Sphericity was non-significant (p = .517). Therefore the multivariate F- “tests were ignored. A significant linear trend for the interaction of group by time was found (F(1,18) = 4.972, p = .039). This trend, as seen in Figure 4.3, indicates that the ratings for treatment group at Level 2 increased at a steady rate across the observations while the ratings treated- «M 3:21.); 103 _.u..::O.1C.T ”imam; ll t-_m:-_.-:m®._.r Ill rfu\ OJ ‘_ Cosme, some)” w \J o A d _ mcozmiombo 5m 3904 35:5 26.50:: wmmcocmsd. F .954 3‘ 339m [ICU-H-—CO') 104 0cm; (m1 Ucmfi. .I..+..| _O:_¢_OQ-U®HMQ; ll cozmiomgO v m EoEEoC I.) _ L mcozmiomoo 1m mwouo< mnco; some... new 3:22“. wmocmcmyi F .96.. N... 930E GZCUH--CO) 105 control group received at Level 2 remained relatively constant. Figure 4.4 provides a display of the Level 2 ratings with the addition of the linear trends. Changes and Trends for Level 3 Changes in Level 3 ratings across time. The set of analyses for Level 3 examined the ratings between treatment and treated-control groups in response to the three specific questions and the use of stimulus materials. Table 4.3 gives the means and standard deviations for ratings at each of the six observations. The results of the first MANOVA examining the differences between groups on awareness ratings for each of the six observations was significant (F(6,13) = 3.124, p = .040). Students in the treatment group received significantly higher Level 3 ratings than students in the control group for Observation 4 and Observation 6. Trends for Level 3 across time. The trend analysis revealed that the Bartlett Test of Sphericity was non- significant (p = .653) and the multivariate F-tests were ignored. A significant linear trend for the interaction of group by time was found (F(1,18) = 7.210, p = .015). This trend indicates that the Level 3 ratings for treatment group increased at a steady rate across the observations while the Level 3 ratings for the treated-control group remained relatively constant. Figures 4.5 and 4.6 provide a display of Level 3 ratings and linear trends, respectively. 106 LO LO 6:50-839; I.) EoEbmo; (4| Cosme. .mmflO e m... (N A J . _ mcozmimmoo 1m m.mo.o< 3:2me 28.52:. mmocoasd m .96.. m.v m.:a.u 0 LO (0 [ICU-H-—-CO) 107 use: (mT new; LT _o:_..oo-.oo$o.F 1| EmEEofi I.) c o . 3.... .630 v a _ d 10 L.) (If) (\1 V‘- O ——1 CO mcozmiomoo rm $0.04 wocmfi 50c... 0cm. mmczmm mmmcmumsq m .653 TV 930.“. CCCUH°—CO') 108 3.5.00-8..3: IT. EoELmofi LI :9 . 6.... 530 m. m. a. a. m _ 4 q d _ Q l _. 1 m - Hm // \L..( L / L (1. V 1 m _\ I my N mcozmimmoo x.@ 3904 35:3“. 3232:. mmmcocmsq m .m>m... we 939“”. [ICU-H-—CO'J 109 cam: ImT pro; LT. _.n..-.LcoQ-.oome.e I.I C O . Hm...» .m mQQ 3085?» II m m. .u. m. N e L T 4 _ O L P 1 m m 6 Mom wcozmiomno Em mmo.o< mace; Soc... ocm moczmm mmmcocmsa. m .961. o... 059.... USCG-Hm-CO') 110 Summary of the Finding for Changes and Trends For Level 1, no significant differences were found between groups at any of the six observations for Level 1 ratings. However, by the end of the year (Observation 6), the differences between groups approached significance. The results of the trend analysis revealed that ratings for both groups increased marginally over time, with treatment group ratings increasing at a rate that was slightly greater than ratings for the treated-control group. For Level 2, significant differences favoring the treatment group were found beginning at Observation 4 and continuing throughout the remainder of the study. For Level 3, significant differences favoring the treatment group were found for Observation 4 and Observation 6. The results of the trend analysis for Level 2 and Level 3 ratings revealed a significant group by time interaction, indicating that treatment group students improved in their ratings throughout the course of the study at a rate greater than students in the treated-control group. Taken together, the results at all three levels revealed similar patterns: ratings for treatment group students improved gradually over the course of the study while ratings for treated-control group students remained relatively constant. 111 The Relationship between Metacognitive Awareness Ratings and Performance on the Additional Outcome Measures The final research question investigated the relationship between metacognitive awareness ratings and performance on five outcome measures used in this study: the Supplemental Awareness Measure--Process (SAM--Process), the Concepts of Reading Interviews, the Graded Oral Reading Paragraph (GORP), the 1985 Stanford Achievement Test--Word Study sub-test (SAT--Word Study), and the Michigan Educational Assessment Program reading sub-test (MEAP). As described in Chapter Three, in the larger study these were the measures for which significant differences favoring the treatment group were found. Of interest in this dissertation was whether these significant differences remained once the effect of metacognitive awareness was removed. The formal research question is the following: If the variance due to metacognitive awareness ratings at those levels where significant differences between groups were found is removed from the five additional outcome measures where significant differences were also found, do the differences between groups on the additional outcome measures become non-significant? To answer this question, two sets of five stepdown F- tests were used. The ratings for Level 1 were not used in this analysis because of the non-significant MANOVA reported 112 under Changes for Level 1 (F(6,13) = 1.530, p = .244). The first set used the Level 2 ratings for metacognitive awareness of lesson content and each of the five outcome measures. The variance due to Level 2 ratings was removed from each individual outcome measure. The second set was similar to the first except that Level 3 ratings for metacognitive awareness were removed from each of the five outcome measures. Level 2 Stepdown F-tests: The first set of stepdown F- tests revealed that when the variance due to Level 2 metacognitive awareness ratings was removed, non—significant differences were found for the SAM--Process measure (F(1,17) = 1.199, p = .289), the Concepts Interviews (F(1,17) = 0.747, p = .399), the SAT--Word Study measure (F(1,17) = 1.041, p = .322), and the MEAP (F(1,17) = 0.069, p = .796). The differences between groups for the GORP measure remained significant (F(1,17) = 14.979, p = .001). Level 3 Stepdown F-tests: The second set of stepdown F- tests revealed that when the variance due to Level 3 metacognitive awareness ratings was removed, non-significant differences were found for the SAM--Process measure (F(1,17) = 1.250, p = .279), the Concepts Interviews (F(1,17) = 0.031, p = .863), the SAT--Word Study measure (F(1,17) = 0.836, p = .373), and the MEAP (F(1,17) = 0.001, p = .992). The differences between groups for the GORP measure remained significant (F(1,17) = 16.533, p < .001). 113 Summary of Between-Group Differences When Variance Due to Awareness Interview Ratings was Removed The results indicate that when the variance due to Level 2, Level 3 and Awareness ratings was removed from the scores on the five additional outcome measures, differences between groups became non-significant for all additional outcome measures except the GORP. This suggests that metacognitive awareness of reading lesson content is an important component of performance on four of these five measures. Summary of Chapter Four The results of the analysis indicate that Awareness Interview ratings for the treatment group were significantly higher than ratings for the treated-control group for Level 2 and Level 3, and these ratings approached significance at Level 1 by the end of the year (Observation 6). Ratings for treatment group students at Level 2 and Level 3 tended to increase at each of the six observations, while the ratings for the treated-control group did not increase significantly. Treatment group ratings for Level 1 awareness ratings increased steadily across the six observations but improvement in ratings were not significantly different between groups. The results also indicated that when the variance due to Level 2 and Level 3 Awareness ratings were removed from the 114 scores on the additional outcome measures, significant between-group differences remained only for the GORP. All other between-group differences became non-significant, suggesting that Awareness Interview are consistent with a model of student mediation whereby instruction directly effects metacognitive awareness and awareness, in turn, affects performance on the other measures of reading performance included in this study. CHAPTER FIVE CONCLUSIONS AND IMPLICATIONS Introduction The purpose of this study was to examine third—grade low-group readers' levels of metacognitive awareness of reading lesson content and performance on additional reading outcome measures. The general research question addressed in this dissertation was the following: What are the differences at each level of metacognitive awareness between recipients of explicit instruction and those who did not receive such instruction, what are the changes in levels of metacognitive awareness over time, and how does metacognitive awareness affect performance on other measures of reading? Three weaknesses in instructional studies emphasizing metacognitive awareness guided the formulation of this general question. First, such studies have used only unidimensional measures of metacognitive awareness and have not examined the extent to which different levels of poor readers' metacognitive awareness of reading are effected by explicit instruction. Second, available research generally examines pretest-posttest differences; virtually no data are available to indicate the ways in which levels of metacognitive awareness change over time. Third, studies of explicit instruction often correlate measures of metacognitive awareness and performance on other reading 115 116 outcome measures but no data are available to indicate the extent to which performance on other outcome measures is dependent on metacognitive awareness of reading. Five specific research questions addressed the above weaknesses. The first three specific questions examined differences in metacognitive awareness of reading lesson content ratings between students in treatment and treated- control classrooms at three different levels. The fourth research question examined the changes in levels of metacognitive awareness ratings across time. The fifth research question examined the extent to which metacognitive awareness ratings contributed to performance on other measures of reading. This chapter will discuss the findings, implications, and future directions of research on explicit instruction and metacognitive awareness of reading lesson content. To do so, this chapter contains five major sections. The first section is a discussion of the findings and conclusions of the first three research questions. The second section is a discussion of the findings and conclusions of the fourth research question. The third section is a discussion of the findings and conclusions of the fifth research question. The final section discusses the implications of the findings in two ways. First, implications for construction of a theory of explicit instruction and learning to read are detailed. Second, the implications raised by the findings regarding 117 future research on explicit instruction emphasizing metacognitive awareness and on the role metacognitive awareness plays in learning to read are presented. Discussion and Conclusions of Research Question 1 Through Research Question 3 The first three specific research questions were the following: Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated—control classrooms for Level 1 Awareness ratings? ‘ Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 2 Awareness ratings? Are there significant differences at Observation 6 in metacognitive awareness of reading lesson content ratings between students in treatment and treated-control classrooms for Level 3 Awareness ratings? Discussion of Findings of Research Questions 1 through 3 At Level 1, students were asked to respond to a general question about their reading lesson ("Tell me everything you remember about your reading lesson."). This level examined whether recipients of explicit instruction could report their metacognitive awareness of reading lesson content with a minimal amount of post-instruction prompting by the 118 interviewer. Nearly significant differences between treatment and treated-control groups were found at Level 1. At Level 2, where the questions posed following instruction were specifically directed toward what the lesson was about, why it was important to learn, and how to use the lesson content when reading, ratings for students who received explicit instruction were significantly greater than for students in treated-control classrooms. In addition, Level 2 ratings for the treatment group (X = 5.250) improved markedly from Level 1 ratings (2.497, an increase of 2.753). The ratings of students who did not receive explicit instruction (X = 2.432) improved only slightly from Level 1 (X = 1.602, an increase of 0.83). These findings suggest that when specific prompts were provided, only those who received explicit instruction were able to provide an adequate description of their declarative, conditional, and procedural knowledge of lesson content. Responses at Level 3, where students were provided with lesson materials and examples in addition to specific questions about the What, Why, and How of the lesson, mirrored the findings of Level 2; significant differences in favor of treatment students' Awareness Interview ratings were found. There was a marked increase in treatment students' ratings when compared to Level 1 (Level 3 ratings = 5.945, an increase of 3.448) but only a moderate improvement for treated-control students' ratings (Level 3 ratings h 3.088, 119 an increase of 1.486). The changes between Level 2 and Level 3 ratings for both groups were minimal and nearly identical (treatment group increase = 0.695, treated-control group increase = 0.656). In sum, although students in the treatment group possessed greater metacognitive awareness that students in the treated-control group, (a) it appears that responses at Level 2 and Level 3 more accurately reflected the differences in such awareness than responses at Level 1 and (b) that providing stimulus materials during the awareness interview (Level 3) had little effect on the metacognitive awareness reported by students in either group. Conclusions of Research Questions 1 Through 3 The results of the first three research questions indicate that poor readers who receive explicit instruction are more metacognitively aware of lesson content but that such awareness was not easily demonstrated in response to the Level 1 question. One conclusion that can be drawn from these findings is that explicit instruction was not strong enough to provide students with sufficient information so that they could display their awareness at Level 1. Support for this conclusion is found in the stability of ratings between Level 2 and Level 3 and the relatively low ratings received by teachers for explicit instruction. The mean ratings for treatment students at Level 2 and Level 3 120 were similar, with an increase in ratings between Level 2 and Level 3 of only 0.695. A similar pattern in Level 2 and Level 3 ratings was found for treated-control students (3.088 and 2.432, respectively, an increase of 0.656). Thus, both groups displayed a minimal, but similar, increase in ratings between Level 2 and Level 3. Thus, instruction was explicit enough to result in only a slight change in ratings between Level 2 and Level 3, but quite dramatic differences between Level 1 and Level 2.‘ This suggests that explicit instruction did have a powerful impact on metacognitive awareness but that the Level 1 question may have been too general to assess the differences between treatment and treated-control students. Further support is found when the explicit instruction ratings received by'teachers are examined. As described in Chapter Three, the explanation ratings for Observation 6 indicated that treatment teachers were more proficient in explicit instruction (x = 17.70, SD = 6.89) than their treated-control group counterparts (X = 9.10, SD = 3.44). However, the maximum possible rating that could be received for explicit instruction was 44. Thus, while treatment teachers' ratings of explicit instruction were almost twice that of treated-control teachers, the mean rating for treatment teachers was less than half of what could have been received. These ratings were high enough to effect treatment group students responses to the specific What, Why, and How 121 questions included in the interview (i.e., at Level 2 and 3), but not sufficient to effect responses to the more general question posed at Level 1. Consequently, Level 1 questions do not indicate a lack of metacognitive awareness and it is conceivable that greater Level 1 ratings would have resulted if treatment classroom instruction had been even more explicit than they were in this study. Summary of Research Questions 1 Through 3 In sum, nearly significant differences were found between groups at Level 1. At Level 2 and Level 3, the similar, highly significant ratings suggest that little is gained by asking specific questions in conjunction with stimulus materials; there were virtually no changes in the ratings for the treatment or treated-control group between Level 2 and Level 3. The lack of significant findings at Level 1 is likely due to the relatively low ratings of explicit instruction received by treatment teachers in the study. Discussion and Conclusions for Research Question 4 The fourth research question concerned the changes in ratings across time and the trends of these changes: What is the trend in the ratings received at each of the three levels of metacognitive awareness of reading lesson content across the six observations? 122 This research question examined (a) the between-group differences for the three levels at each of the six observations and (b) the trends of the changes in metacognitive awareness ratings over time. Discussion of the Findings of Research Question 4 For Level 1, the results of the analysis revealed that although the differences between groups were non-significant, the ratings for the treatment group increased in relatively small, but consistent increments across the six observations. IntereStingly, while no significant differences at any of the six observations were found for Level 1, there was a significant effect for the averaged ratings across the six observations (averaged ratings for the treatment and treated- control groups were 1.697 and 1.160, respectively) and there were significant linear and quadratic trends across time. All of these findings favored the treatment group. Thus, when the averaged ratings were compared, treatment students were rated higher than treated-control students. In addition, the trends suggest that ratings for both groups at Level 1 tended to increase during the year. Interestingly, the mean ratings for the treatment group remained similar between Observation 1 and Observation 3 (ratings increased from 1.287 to 1.310, or 0.023 points) but increased markedly from Observation 3 to Observation 6 (increase from 1.310 to 123 2.497, or 1.187 points). There was only a modest increase for the treated-control group from Observation 1 to Observation 3 and between Observation 3 to Observation 6 (0.279 and 0.452, respectively). For Level 2, a pattern similar to that of Level 1 was found; treatment students were rated higher than treated— control students throughout the year. In addition, significant differences favoring the treatment classrooms were found beginning at the fourth observation and continuing through the sixth, and final observation. In addition, significant linear trends for the Group X Time interaction were found. This finding indicates that Awareness Interview ratings for students in the treatment group increased throughout the year while the ratings for students in the treated-control group remained relatively constant. Similar results were found for Level 3 ratings although significant differences favoring the treatment classrooms were found at the fourth and sixth observations only. The non-significant differences at Observation 5 appear to be the result of an increase of ratings for the treated-control group, while the ratings for the treatment group remained essentially the same at Observation 5. The treated-control group's ratings declined at Observation 6 to a level equal to their Observation 4 ratings. In addition, significant linear trends for the Group X Time interaction were found. As with 124 Level 2, these findings indicate that Awareness Interview ratings for students in the treatment group increased throughout the year while treated-control students' ratings remained relatively constant. In addition, Level 3 ratings at each observation for both treatment and treated-control students were slightly higher than ratings at Level 2, suggesting that when students are asked questions about their reading lesson and are provided stimulus materials (Level 3,, they demonstrate slightly greater metacognitive awareness than when asked questions alone (Level 2). Conclusions Regarding Research Question 4 Two conclusions are drawn from these findings: (1) if provided with more explicit instruction throughout the year, treatment group students' Level 1 ratings could become significantly greater than treated-control students' ratings and (2) improved metacognitive awareness in response to explicit instruction is not easily effected in naturalistic classroom studies. Evidence for the first conclusion stems from the trend of metacognitive awareness ratings at each of the three levels and the ratings of explicit instruction across the six observation reported in Chapter 3. Level 1 ratings for the treatment group remained relatively unchanged until the fourth observation at which time the ratings began to increase steadily. Thus, while the question posed at Level 1 125 may have been too general for an accurate assessment of poor readers' metacognitive awareness, the significant differences in averaged ratings and the significant linear and quadratic trends indicate that awareness at this level is affected by instruction and that the treatment group students did improve their ratings between Observation 3 and Observation 6. In addition, as discussed in Conclusions of Research Questions 1 through 3, the ratings of explicit instruction (see Table 3.2) were far below the maximum possible rating of 44. If treatment teachers had been more explicit throughout the year, it seems likely that Level 1 ratings could have mirrored the trends found for Level 2 and Level 3. A second conclusion is that improved metacognitive awareness is not easily acquired by the population of this study, i.e., low-group third-grade readers. The treatment group required approximately twenty weeks (i.e., until Observation 4) before their Level 1 ratings began to increase and before their Level 2 and Level 3 metacognitive awareness ratings were significantly greater than ratings for the control group. Thus, if an improvement in metacognitive awareness is of interest in instructional studies, short-term interventions may give false impressions of the efficacy of the instructional intervention: poor readers, who commonly display deficiencies in metacognitive awareness, may require a great deal of instructional time before their awareness improves. One reason that such time may be necessary is that 126 acquiring metacognitive awareness is more difficult than acquiring cognitive abilities. Poor readers can learn numerous facts and rules about reading. For example, they are aware that the word "frostbite" is a compound word (fact) because it is comprised of (a) two words that (b) stand alone and (c) use the meaning of both words to form the compound word (the rule). Many different, individual compound words can be learned by poor readers, thereby increasing their vocabulary and sight word recognition, as much of this learning requires memorization. To become metacognitively aware of the knowledge communicated is a much more complex process because it requires students to recognize that all compound words contain similar characteristics, that there is an important reason that such words are used or an appropriate time when these words can be used, that there is a procedure for both identifying compounds words when reading and for constructing your own compound words, and that such words are found in a variety of content areas (math, science), and contexts (in school, cereal boxes, newspapers). Consequently, metacognitive awareness requires a more generalized understanding than does simply learning individual facts or rules, and may therefore require a great ' deal more time before such awareness is demonstrated. Consequently, Level 1 treatment group awareness ratings may become significantly greater than control group ratings if two conditions are present: (1) teachers are more 127 proficient in explicit instruction (the explicit instruction ratings in this study were well below the maximum possible), and/or (2) poor readers receive explicit instruction for a period of time greater than one academic year. Summary of Research Question 4 In sum, metacognitive awareness is amenable to change under the appropriate conditions and poor readers can become more metacognitively aware. However, it is necessary for these readers to receive continuous exposure to explicit instruction which emphasizes what the lesson is about, why it is important to learn, and how to use lesson content correctly, before substantial changes can be made. Discussion and Conclusions of Research Question 5 The fifth research question concerned the contribution of levels of metacognitive awareness to poor readers' performance on the five additional reading outcome measures: If the variance due to metacognitive awareness ratings at those levels where significant differences between groups were found is removed from the five additional outcome measures where significant differences were'also found, do the differences between groups on the additional outcome measures become non-significant? 128 This research question examined the between-group differences on the five additional measures of reading once the variance due to levels of metacognitive awareness at Observation 6 was removed from the scores on these measures. Only the variance attributable to awareness ratings at Level 2 and Level 3 were used in this analysis as no significant difference was found for Level 1. Removing the variance due to Level 1 ratings would not effect the analysis. Discussion of the Findings of Research Question 5 Prior to the removal of the variance attributable to Level 2 and Level 3 Awareness Interview ratings, significant differences favoring the treatment group were found for all additional performance measures. When the variance attributable to Level 2 ratings was removed and the between- group differences were re-examined, the differences between the two groups became non-significant for four of the five measures: the SAM-Process measure, the Concepts of Reading Measure, the SAT-Word Study subtest, and the MEAP. The between-group differences for the GORP measure remained unaffected (i.e., significant differences favoring the treatment group remained). Identical results were found when the variance attributable to Level 3 ratings were removed. These findings indicate that explicit instruction during reading skills instruction results in improvement in a variety of areas of comprehension. However, directly 129 attributing such performance to explicit instruction alone is unwarranted. Poor readers who receive explicit instruction apparently mediate such instruction based on their metacognitive awareness. Therefore, poor readers in the treatment group were better able to (a) identify a specific strategy that will assist in comprehension (SAM-- Process), (b) describe what procedures are necessary when given a story to read or when comprehension is disrupted (Concepts of Reading), (c) decode individual words (SAT-- Word Study), and (d) answer questions about a short passage (the MEAP) because they possessed greater metacognitive awareness of lesson content. Explicit instruction alone does not affect improved performance; metacognitive awareness of reading lesson content also plays a significant role. Conclusions Regarding Researchgguestion 5 There is one major implication of the findings of the fifth research question: differences in poor readers' performance on additional measures is due, in large part, to improved metacognitive awareness of reading lesson content. Such awareness, in turn, impacts a variety of reading abilities which are important for comprehension. Metacognitive'awareness has been associated with reading performance in a variety of studies. However, virtually all of these studies have been descriptive in nature and the effects of instruction were not investigated. Of the few 130 recent instructional studies that have taught poor readers to become more metacognitively aware, and subsequently assessed performance in other areas of reading, none have attempted to examine whether performance improves independent of metacognitive awareness. In this study, poor readers were taught to become more metacognitively aware of reading lesson content during their regular reading skills instruction. No attempt was made to directly teach these students to select, from a set of options, a specific strategy that would be appropriate for a given reading situation (SAM--Process) or to describe a general, strategic concept of reading (Concepts of Reading). Poor readers in both the treatment and treated- control groups were directly taught to decode individual words (SAT-—Word Study) and, to a much lesser extent, to answer questions about a short passage (the MEAP), but the words and passages were not the same as those contained in the SAT or MEAP. Consequently, if metacognitive awareness is an independent component of reading that is not associated with performance, or at least is weakly associated with performance, then removing the variance attributable to metacognitive awareness should have had little effect on performance and the differences between treatment and treated-control groups on the performance measures should have remained unaffected. The data from this study suggest that this is the case. While the findings are correlational 131 in nature, the data are consistent with a model of student mediation where metacognitive awareness, produced by explicit instruction, effects performance on the additional reading outcome measures. In sum, the results suggest that improving poor readers' metacognitive awareness might be a necessary precondition for successful performance on the reading tasks included in this dissertation. If an individual is not aware of what was learned during a reading lesson, for example, a strategy for predicting outcomes or decoding particular words, then it is not likely that the individual would recognize test items on the SAM, SAT, or MEAP that required the use of such strategies. Hence, poor readers in the treatment group were provided with information intended to improve both their metacognitive abilities (declarative, conditional, and procedural knowledge) and cognitive abilities (reading skills), but their improved cognitive performance was affected by their metacognitive awareness. The instructional intervention emphasized metacognitive awareness of basic reading skills. Therefore, it is not possible to conclusively determine from the data in this dissertation whether explicit instruction directly affects metacognitive awareness which, in turn, directly affects performance. As noted in Chapter One, a basic assumption is that explicit instruction can increase both metacognitive awareness of reading lesson content and performance in a 132 variety of areas which are important for improved reading comprehension. Explicit instruction provides poor readers with specific lesson content, and through such content, emphasizes the importance of the declarative, conditional, and procedural knowledge, unlike many general "thinking skills" programs (cf., Segal, et al., 1985) which emphasize metacognitive awareness (or some other form of strategic problem-solving) independent of reading content. Explanation of the GORP findings. The findings of the GORP revealed that the between-group differences remained unaffected when the variance attributable to metacognitive awareness was removed. Such findings are difficult to interpret but there are two possible reasons for this finding: (1) the measure assessed some factor "X" which is independent of metacognitive awareness but was affected by explicit instruction and (2) the knowledge measured by the GORP was the same as that measured through Awareness Interviews, not an independent reading ability. Regarding the first possibility, there may be some factor or component acquired from explicit instruction that is not associated with metacognitive awareness. That is, the knowledge that is needed for successful performance on the GORP may have been an unexpected by-product of explicit instruction which is independent of metacognitive awareness. The existence of some unknown factor that remained unaffected once the variance attributable to metacognitive 133 awareness was removed seems plausible. A major goal of the larger study was to improve students' ability to comprehend text by teaching specific reading skills as strategies. It was expected that such instruction would transfer to other areas of reading (e.g., Concepts measure, SAT). Teachers were trained and assessed solely on the basis of how explicit they were in teaching students about the declarative, procedural, and conditional knowledge of lesson content. Consequently, it is not known if the training also produced particular instructional behaviors which inadvertently assisted teachers in communicating information that was beneficial for performance on the GORP. Such instruction would not be captured by ratings of explicit instruction nor by ratings of metacognitive awareness. However, a second explanation for these findings emerges when the format of the GORP measures is closely examined. In part, the GORP measure was intended to assess whether students would use reading strategies while actually engaged in reading. To assess whether students did so, unknown words were embedded into the passage and students were asked to read the passage aloud. The number of self-corrections were noted and students were then asked to verbalize why they made self-corrections when reading and how they knew that such self-corrections were necessary. Students were also asked to define the unknown words embedded in the passage and to describe how they knew the meaning of the words. The 134 descriptions provided by students indicated whether they were aware, for example, that a problem existed while they were reading (e.g., that they did not pronounce a word correctly) and how to figure out the meaning of an unknown word encountered in the text. When viewed in this way, the knowledge assessed by the GORP measure is quite similar to that assessed by the . Awareness Interviews in that students were asked to describe why they re-read miscued words and to describe how they knew the meaning of the unknown words embedded in the passage. Thus, it may be a measure of metacognitive awareness of reading in general and not a measure requiring the use of some unknown factor. In essence, treatment students' performance on the GORP may have been precisely what was expected--by receiving explicit instruction throughout the year, students' metacognitive awareness of reading in general improved and as a result, they were more able to determine what strategies they possessed, when these strategies could be used during reading, and how to apply these strategies flexibly and independently. Further support for this possibility can be found when the size of the significant differences treatment and treated-control groups is examined. As indicated in Chapter Pour, the between-group differences on the GORP measure was highly significant (greater than .001). If the measure is a proxy measurement of metacognitive awareness, then it is 135 quite possible that the Stepdown-F procedure removed only a small proportion of the variance attributable to metacognitive awareness; that which was reflected in Awareness Interview ratings. A significant proportion of variance (i.e. metacognitive awareness of reading) remained in the scores on the GORP and, therefore, the between-group differences remained. If this second possibility is correct, then the effects of explicit instruction may be even more profound than first thought. In effect, treatment group students received instruction that was not only sufficient to affect metacognitive awareness of reading lesson content, but such instruction also affected students' more global ability to comprehend text independent of instruction--precisely the goal of the instructional intervention. Summary of Research Question 5 The results of Research Question 5, while correlational in nature, suggest that explicit instruction may not directly affect performance on the measures used in this dissertation, with the exception of the GORP measure. Once the variance due to Level 2 metacognitive awareness and Level 3 metacognitive awareness was removed from these measures, the differences between the treatment and control group became non-significant for the SAM--Process measure, the Concepts of Reading measure, the SAT--Word Study measure, and the MEAP 136 measure. Consequently, it might be plausible that explicit instruction directly affects metacognitive awareness of reading lesson content. Such awareness, in turn, affects performance of reading on four of the five reading outcome measures. The results of the GORP, which remained unchanged when the variance due to metacognitive awareness was removed, could be due to the presence of some other unanticipated factor that was facilitated by explicit instruction. However, just what this factor is, and why significant differences remained once the variance due to metacognitive awareness was removed from the GORP remains unknown. Implications of the Findings of this Dissertation Two sets of implications are suggested by this study. The first relates to a model of instruction; the second relates to future research. An Emerging Model of Instruction and Metacognitive Awareness One major implication of this dissertation is that the relationship between instruction and performance (in this case, the ability to comprehend text) is a complex one. The specific findings suggest that a emerging model of instruction which has as its core the role of metacognitive awareness provides an initial representation of this complex relationship and more adequately explains how instruction can 137 improve poor readers' ability to understand what they read. This section briefly describes this model. Metacognitive awareness as a mediational process. Two major findings of this dissertation informed the construction of the model shown in Figure 5.1. First, the results of Research Questions 1, 2, 3, and 5 suggest that there is a strong association between explicit instruction and metacognitive awareness and between metacognitive awareness and performance on four of the five reading outcome measures used in this dissertation. These findings are consistent with a model of student mediation which posits that learning (i.e., performance) is not directly effects by instruction but that some internal mental process interprets and transforms instruction and that this transformation is what is assessed by performance measures (Shulman, 1986). In this dissertation, instruction directly affects metacognitive awareness of reading lesson content and such awareness, in turn, mediates (transforms) performance on the reading outcome measures. Second, as suggested by Research Question 4, explicit instruction cannot be expected to immediately affect the metacognitive awareness of poor readers. To acquire such an awareness, teachers must present instructional information explicitly so that poor readers can (a) attend to important lesson content, (b) understand why such content will assist them in comprehending text, and (c) understand how to use 1.323 Figure 5.1 Model of Explicit Instruction and Metacognitive Awareness of Reading Lesson Content Explic1t InstructLCn a: ogn; tive 1.4are. ‘ Ute 39301 ng 365533 50215; Hera‘ogn...ve awareness Cene-a. ( Information Condo zcazed I during inatrnction: ( I I I Declarative iIHOWlé I . - - —_ A- “A Conditional knowledg- Prccedural knowledge to improve: Knowledge of cognition - .. ... .. w__..__..__.. Regulation of cognition .-—.—m- ——.———.. Students attend to: Decl aret.' ve 3ncv edge i:SSOR CCUCEEC Condition 1 Knowles ; 'ESSC" :ontent Procedural knowle _ 1‘ R .A .32? lasz: n vknb But. a “Q 'A nfiPl“? tLo:5.o 1.-..19 i g'::r;“ 3.:rsr.:c '-~uv~. ''''''''' .5. 6L0 H an ' V -- va"'; .'n r ‘ .: -3.-a1.~.4 -'-~3C".:ci:» HQ g a: .quw'-n;~ ~v331t&:3 . D-on-r .L‘“: - ' | o '9'fl‘lz-“u 9.4:. .Iygq'd sat-I; oanvihb;. a-.0v or - fir I ":ornation Intended one: awareness _--.-.._-._ _ ...__..—._.__ ..__.__~_-_-_-__._ ___,-- Knowledge of ccgnzzin regarding reading lesson content The ability to regulate cognition during reading lessons at; litv tor mo la :e AA n 91“!- vv .u-'u.uu .1 turn, Knowledge of and regulation of. cogn;t1c: reen.ts in :aproved read;ng performance .. ._...._..- a”. ____---.__..__..-___. 4, 139 lesson content under a variety of conditions (e.g., during interactive instruction, seatwork, reading for pleasure, etc.). Consequently, it takes a significant amount of time for poor readers to recognize the importance of attending to such knowledge during instruction and using knowledge during and after their reading lessons. In sum, the findings of this dissertation suggest that in order to understand how poor readers learn from instruction, it is useful to conceptualize metacognitive awareness as a mediational process which links what is taught about reading with what poor readers learn from reading lesson content. The construct of mediation or mediational processing as a variable which links external stimuli (e.g., instruction) to performance is not new (see, for example, Hull, 1930). Mediation during instruction, as it is conceptualized here, is simply the process by which students become aware of (a) what instructional stimuli is important to attend to, (b) the purpose or goals of the lesson, and (c) what relevant background knowledge should be activated so that learning about reading will result. Shulman (1986) concurs that mediation is an important element in learning from instruction because recent research indicates that: The learner does not respond to the instruction per se. The learner responds to the instruction as transformed, as actively apprehended. Thus, to understand why learners respond (or fail to respond) as they do, ask not what they were taught, but what sense they rendered of what they were taught. The consequences of teaching can only be understood as a function of what that teaching 140 stimulates the learner to do with the material (p. 17) Similarly, Winne (1982, p. 15) also believes that understanding what happens between the information communicated during instruction and what is ultimately learned about that information is important because: [If] one accepts that this assumption of inherent cognitive activity and strategic purpose accurately characterizes learners, a question follows logically about how learners learn from instruction: what relations exist between these naturally occurring cognitive processes and the instructional events that an instructor creates to approach the goal of acquiring predefined content (i.e., the curriculum)? In essence, mediation of instruction is a process by .which salient lesson content is attended to, the goals of instruction are identified, and the appropriate background knowledge is activated. What information is attend to, what purpose of the lesson is perceived, and what background knowledge is activated, determines what sense is made of instruction. And the quality of instruction, in this case, the explicitness of instruction, determines whether the sense students' render is in agreement with the sense intended by the teacher. In the model presented here (see Figure 5.1), metacognitive awareness of lesson content is hypothesized to play a major role in such mediation. When poor readers are metacognitively aware of lesson content, they are aware of 141 what the lesson is about, which should assist them in identifying salient lesson content. When poor readers are more aware of why the lesson is important, it should help them recognize the purpose of the lesson. Finally, when poor readers are aware of how to use lesson content successfully, they should be more able to activate the relevant background knowledge which will assist them in successfully performing the tasks required during the lesson. Figure 5.1 provides an overview of the model. In the middle portion of the model, poor readers are seen as mediating instruction based on their awareness of the ID U 1 U1 0 :3 declarative, procedural, and conditional knowledge of 1 content. In doing so, they can activate appropriate cognitive knowledge and effectively regulate their cognitive activity during the lesson. However, during any one lesson, a great amount of information is presented and poor readers, as evidenced by the findings of this study, are not immediately aware of the importance of such knowledge. Through consistent instruction which emphasizes declarative, conditional, and procedural knowledge (the left-hand portion of the model), poor readers become increasingly aware of what the lesson is about, why it is important to learn, and how to use lesson content successfully. The learning that results contributes to their knowledge of cognition and their ability to regulate their cognition during reading instruction. Over time, such awareness of lesson content leads to the ability 142 to successfully perform reading tasks requiring the knowledge and skills gained from instruction. In sum, the findings of this study suggests that one valuable approach in understanding how to promote improved reading abilities for poor readers is to recognize that metacognitive awareness is not simply another component of comprehension, similar to the ability to, for example, decode words or describe main ideas in a passage. Instead, metacognitive awareness may be an important link between what teachers intend their poor readers to learn, and the knowledge that these readers gain from instruction. Implications for Future Research There are six implications for future research on metacognitive awareness and explicit instruction: (1) the relationship between metacognitive awareness and explicit- instruction for readers of average and above average ability, (2) the correspondence between individual components of explicit instruction and metacognitive awareness of reading lesson content, (3) the effects of explicit instruction on the acquisition of the declarative, conditional, and procedural knowledge that comprises metacognitive awareness, (4) the role of other factors, particularly motivation, in mediating instruction, (5) the need to experimentally establish metacognitive awareness as a mediator of reading performance, and (6) the extent to which such mediation --1’M' :p . _ -. " agar. 143 during reading instruction affects performance in other content areas. Regarding the first implication, it is not known whether average and/or above average readers who receive explicit instruction would have been rated at or near the maximum rating of 12. The findings indicate that there were nearly significant differences at Observation 6 for Level 1, highly significant differences at Level 2 and Level 3, and treatment group ratings at all three levels were approximately twice ‘ '_ yam. bum-F... “w u l" the ratings received by control students. However, treatment group ratings at Level 1 was, Level 2 and Level 3 were less than half the maximum possible rating of 12 (2.497, 5.250 and 5.495, respectively). Thus, it could be argued that the intervention did not effect metacognitive awareness ratings when compared to the total possible rating of 12. However, when compared to poor readers who did not receive such instruction, the intervention did have a very positive effect on awareness ratings for treatment group students. Given the low reading group population used in this study, it may be unrealistic to expect such students to be ratedat or near the maximum possible of 12 within the time span of this study. As this study did not include readers of average or above average ability, it is not known whether readers of different abilities would have been rated at or near the maximum of 12 had they received explicit instruction. Further research which "norms" the Awareness Interview across 144 a variety of reading ability levels is warranted. Regarding the second implication, further research is needed to indicate which components of explicit instruction appear to have the greatest impact on poor readers' levels of metacognitive awareness. In this study, only between-group differences in levels of metacognitive awareness were examined. It was assumed that the significant differences between treatment and control students at level 2 and Level 3 were due to the significantly greater explanation ratings received by treatment teachers in the larger study. However, treatment teachers were rated far below the maximum possible rating and there was a great amount of variability among them in their ability to communicate reading lesson content explicitly (see Table 3.2). It could be that some teachers were more adept at explicit instruction in general (as reflected in the total ratings given in Table 3.2), others may have been more adept in some components of explicit instruction (e.g., modeling, linking the present lesson to past and future lessons), while still others may not have included some components of explicit instruction at all. From the data reported in this study, there was no way of determining whether high total ratings of explicit instruction (the sum of the eleven components as listed in Appendix F) or high ratings of certain individual components of explicit instruction contributed significantly to treatment students' metacognitive awareness ratings. 145 Therefore, research needs to investigate whether there are certain individual components of explicit instruction which significantly contribute to improved metacognitive awareness at each of the three levels and whether the high ratings in one or more components might result in (1) significantly greater ratings at the three levels, and (2) significantly different ratings earlier than the fourth observation (approximately twenty weeks into the study). To do so, it would be valuable to train teachers in explicit instruction and then intensively examine the instruction of individual teachers (perhaps observing one or two lesson per week) over a period of several months. Students could be interviewed regarding their metacognitive awareness of lesson content following each lesson. Fluctuations in the extent to which different components of explicit instruction were included in individual lessons could then be compared to their students' ratings of metacognitive awareness of lesson content. The third area of research suggested by the findings of this dissertation is closely related to the second: the ways in which explicit instruction affects poor readers' declarative, conditional, and procedural knowledge at each of -the three levels. The primary interest of this study was in the differences between treatment and control readers' ratings at the three levels of metacognitive awareness, not the extent to which declarative, conditional, and procedural knowledge contributed to these ratings. Thus, it is unknown 146 whether some components of explicit instruction may be significantly related to these three knowledge categories while other components of explicit instruction may not. Although it may seem intuitively logical that certain components of explicit instruction should be related to one or more of the three knowledge categories which comprise metacognitive awareness (e.g., modeling how to use the strategy should be correlated with procedural knowledge), such data are not available. If such associations are found, then greater information regarding the ways in which explicit instruction affects metacognitive awareness is gained. Consequently, educators are provided with a more indepth understanding of the correspondence between what teachers do and say during instruction and what students learn. The fourth area of further research suggested by the findings of this study concerns the cbntribution of other factors, particularly motivation, in the mediation of instruction. It is not known if poor readers' motivation was directly affected by explicit instruction. Since a major intent of explicit instruction is to provide poor readers with lesson content that will improve their awareness of what the lesson is about, why it is important to learn, and how to use the content successfully, it is possible that such instruction may simultaneously affect poor readers' motivation to attend to, and make sense of, lesson content. If this is the case, then subsequent performance is mediated 147 by both motivation and metacognitive awareness. Further, such a relationship between motivation, metacognitive awareness, and performance suggests that the construct of metacognitive awareness may not be independent of motivational factors. Consequently, future research needs to conceptualize the effects of explicit instruction on motivation, the relationship between motivation and metacognitive awareness, and the ways in which both contribute to the mediational processing that occurs during reading instruction. The fifth area of research that needs to be pursued is a causal relationship between metacognitive awareness and reading performance. In this dissertation, a strong correlational association between ratings on the Awareness Interview and performance on the additional outcome measures was found. These results are consistent with the model of mediation discussed in the above section but they do not confirm that the link between instruction and performance "passes through" metacognitive awareness. Such a causal link, where improved metacognitive awareness is directly associated with performance, is still needed. In doing so, the roll of metacognitive awareness as a mediator of learning from instruction would be more fully understood. The sixth area of research that is suggested by the findings of this dissertation concerns the extent to which improved metacognitive awareness transfers to other content 148 areas. Poor readers in treatment classrooms improved their metacognitive awareness of reading content and such awareness may have contributed to performance on the additional outcome measures. An important extension of such findings would be to examine whether poor readers, by attending to the declarative, conditional, and procedural knowledge of reading lesson content are also more metacognitively aware of such ' "films—.117 knowledge in other academic areas. If so, then these readers, by virtue of experiencing instruction which emphasizes the importance of the What, Why, and How of lesson content, might also be able to recognize, and attend to, such knowledge in other academic areas regardless of whether instruction is explicit in these content areas. Historically, such transfer effects have been difficult to establish (Chipman, et al., 1985; Brown, et al., 1983), possibly because subjects either do not receive long-term instruction or do not receive instruction which consistently emphasizes metacognitive awareness. Recent evidence (e.g., Palincsar & Brown, 1984) suggests that improved performance I H-“W tux \ fl in other areas can be found if instruction emphasizes the development of metacognitive abilities. Additional research needs to determine whether poor readers' metacognitive awareness which is produced by explicit instruction results in attending to the declarative, conditional, and procedural knowledge inherent in other content areas. In sum, the findings of this dissertation suggest that 149 four areas need to be addressed in order :: further understand the role of metacognitive awareness of reading lesson content, explicit instruction, and performance on a variety of reading tasks. First, future research must more closely examine the individual components of explicit instruction and the ways in which these components affect . metacognitive awareness at the three levels. Second, a more detailed examination of the effects of explicit instruction on the acquisition of the declarative, conditional, and procedural knowledge that comprises metacognitive awareness is needed. Third, the role of other factors, particularly motivation, and how they interact with metacognitive awareness in the mediation of instruction is needed. Finally, it is necessary to determine the extent to which such instruction emphasizing metacognitive awareness transfers to other content areas. Summary of Chapter Five 7.2!! No significant differences were found between groups at Level 1 although significant differences favoring the treatment group were found for Level 2 and Level 3. It is suggested that lack of significant findings at Level 1 appears likely due to the combination of the general nature of the Level 1 question and the relatively low ratings of explicit instruction received by treatment teachers in the study. The similar ratings at Level 2 and Level 3 suggest 150 that little is gained by asking specific questions in conjunction with stimulus materials as the results for the ratings at Level 2 and Level 3 were virtually identical. The ratings at all three levels improved over the course of the study. Significant differences favoring the treatment group appeared at the fourth observation for Level 2 and Level 3. While no significant differences were found at Level 1, the ratings improved slightly at each observation. Regarding the changes of metacognitive awareness of reading lesson content over time, the findings suggest that such awareness is amenable to change when poor readers receive explicit instruction extending over a considerable amount of time, in this case, approximately twenty weeks. The results of Research Question 5 suggests that explicit instruction may not directly affect performance on the measures used in this dissertation, with the exception of the GORP measure. Once the variance due to Level 2 metacognitive awareness and Level 3 metacognitive awareness was removed from these measures, the differences between the treatment and control group became non-significant for the SAM--Process measure, the Concepts of Reading measure, the SAT--Word Study measure, and the MEAP measure. Consequently, explicit instruction has a direct affect on metacognitive awareness of reading lesson content. Such awareness, in turn, may affect performance of reading on four of the five reading outcome measures. 151 The results of the GORP, which remained unchanged when the variance due to metacognitive awareness was removed, could be due to the presence of some other unanticipated factor that was facilitated by explicit instruction. However, just what this factor is, and why significant differences remained once the variance due to metacognitive awareness was removed from the GORP remains unknown. ' ”7m,“ 7 The model proposed in this dissertation was an outgrowth of the findings, particularly for Research Questions 4 and 5. The model described metacognitive awareness as a mediating process by which poor readers, in response to explicit instruction, become more metacognitively aware of reading lesson content which, in turn, mediates their performance in other areas of reading. Finally, the findings of this dissertation suggest that future research needs to address four areas in order to more fully understand the relationship between metacognitive awareness of reading lesson content, explicit instruction, and reading comprehension. First, future research must more closely examine specific components of explicit instruction and how these components affect metacognitive awareness at the three levels. Second, an examination of the effects of explicit instruction on the acquisition of the declarative, conditional, and procedural knowledge that comprises metacognitive awareness is needed in order to determine the extent to which these three knowledge categories are affected 152 by instruction. Third, the role of other factors, particularly motivation, and their interaction with metacognitive awareness in the mediation of instruction is needed. Finally, the transfer of the declarative, conditional, and procedural knowledge gained from reading instruction needs to be examined in order to determine the extent to which such instruction emphasizing metacognitive L awareness contributes to performance in other content areas. ”a. 'm1u_ . APPENDICES qmv'hi \J . .‘v - A. APPENDIX A Question Format for Student Awareness of Reading Lesson Content Interviews Introduction and Warm Up (FOR NEW STUDENTS ONLY) SAY: "Hello. How are you today? May name is What's your name? SAMPLE QUESTIONS - WARM UP 1. How was your summer? Did you watch the Olympics? Do you like sports? What's your favorite sport? What other things do you like doing? What about in school? What do you like to do in school? What's your favorite subject? Is there a subject you don't like? How do you like reading? or 2. Well, , I bet you do some neat things outside of school, like sports or hobbies. Is there something that you really like to do? What do you most like to do in school? What's your favorite subject? What don't you like to do? How do you like reading? PICTURE - SAY: Take a look at this picture. This is a house, this a school, and these are four piles of things kids read. Each pile gets smaller and smaller. The big pile means "I read a lot." The second pile says "I read sometimes." The third pile means "I read just a little bit." And this last pile means "I read almost nothing." I want you to draw a line from the picture of the house, we'll pretend it's your home, to how much you read.‘ Say: at home, I read a lot; or, at home, I read sometimes. Good. Now do the same for school. I read a lot at school, or I read almost nothing at school. That's good, too. 153 “PM?” ".1 I?" L...'.s.‘ PICTURE - FOLLOW-UP What do you read at signs, mags, books, What do you read at Do you read because 154 QUESTIONS home? (cereal boxes, comics?) school? you like to, street or because someone tells you to? E. INTRO TO RESEARCH QUESTIONS (Student's Name), now I have some other kinds of questions to ask. These questions have to do With the reading lesson you just had. During this entire school year I'm going to be visiting your class's reading lesson. You'll see me watching your teacher, and listening to all of you kids in your reading group. After the lesson, I'll be talking to some of . you, just like I'm doing now. The questions are L short, and your answers tell us how the teacher . taught. At the end of the year, we put all of the answers together, and then I think we'll find out the best way for teachers to teach reading. Do you have any questions? OK, let's begin. . w o...p—. E. . n..'- u . Introduction for students who had been interviewed before. II. SAY: Hi (Student's Name). remember when I asked you some questions? today we're going to talk again. First with general question) How are you today? Do you Good. Well (begin III. General Question - Level 1 (Either question can be used.) ' e1 1. Tell me everything you remember about the lesson you just had. 2. Tell me all that you can about the lesson I just saw. m :3-"-'_‘.' . Iii t. c Tell me more. Was there anything else? Tell me more about that. PROMPTS (use prompts frequently): RESPONSES: Good. That answer showed you're really thinking. IV. You are really trying hard, thanks. OK, I can see you are thinking. These are acceptable responses to pupil's EFFORTS. They may be used at any time. Do not evaluate correctness of pupil's answer. Specific Questions - Level 2 (The questions under what, how and why are not in hierarchical order. More than one question may be used, but not all have to be used.) What? 1. Can you tell me what the lesson was about? 2 What were you learning to do in the lesson today? 1 How did you do it? 2. How did you know what to do? 3. How do you decide (what to do)? 4 Pretend your best friend is sick today and s/he didn't come to school. How would you teach your friend about ? 5. How did you find the right answer? Why? 1. Why do you need to be able to do that? 2. How would learning that help you? 3 You know how to , when would you use it? If student says in learning or in reading, ask: How would it help you in reading? Specific Questions - Level 3 These questions are asked in conjunction with the presentation of a concrete example. In order to maintain the consistency of interview style, all interviewers should ask these questions, even if s/he thinks the student has answered previous questions adequately. '.' II AIIQ-3fi.:" whet h (Ii-.13 . It'. 1" 156 SAY: Good. 'Now, this is an example of the work you were doing in class today. Could you tell me A. What? 1 What were you learning to do in the lesson? 2 What did your teacher teach you in your reading lesson today? 3. What is this worksheet about? B. How? 1. Can you tell me how you got the answer? 2. How did you ? (whatever the pupil has learned) 3. Show me what you would do to get ? C. Why? 1. What will learning about help you do? 2. How might this help you in reading? VI. Closing Give positive feedback for effort. Suggestions: Good. You've really been helpful, thanks. Thanks for helping me to find out more about teaching reading. See you again. APPENDIX B Awareness Interview Rating Criteria Determine student awareness by judging student response to the interview questions posed at each level and all subsequent elaborating probes which the researcher may have used in conjunction with each question. The criteria for student awareness follow. 1. A highly rated response to the question about "what" was being taught must include a specific reference to the process involved in completing the task and an example: 0--No awareness (student does not know, is inaccurate or supplies a response that does not make sense). 1--the response is a non-specific reference to the task ("We are learning about words."). 2--the response refers to the name of the specific task which can be done successfully if the process is applied correctly or is an example of what can be done ("We are learning gu words."). 3--the response includes a specific reference to the process being learned ("We are learning how to sound out 93 words."). 4--the response includes a specific reference to the process and an example ("We are learning how to sound out 23 words, like out."). 2. A highly rated response to the question about "why" or "when it would be used" must specify both the context in which it will be useful and what he/she is able to do in that context." 0--no awareness or includes no reference to the specific task ("I' ll get smarter" or "it'll help me when I grow up. "). 1--the response is not specific to the task but is related to reading language generally ("I' 11 read better. "). 157 158 2--the response refers to an appropriate general category but not to the specific use for what was taught ("I can sound out words better."). 3--the response includes specific reference to what he/she will be able to do but not the context in which it would be useful ("I can sound out 23 words."). OR specifies the context in which it would be useful but not what he/she will be able to do ("I can use this when I come upon an unknown word in my book."). 4--the response includes both what he/she will be able to do and the context in which it is useful ("When I come upon an unknown 93 word in my library book, I'll be able to sound it out."). A highly rated response to the question about "how do you do it" must include an example of how one does the mental processing associated with successful completion of the task or an appropriate sequence of steps to be followed. O--no awareness. 1--the response is not specific to the mental processing to be used ("I'll sound the word out."). OR is merely an example that does not illustrate conscious understanding of the mental processing to be used ("loud"). 2--the response refers to features to attend to but not to the way they are used in doing the mental processing ("I say, 'l-o-u-d'"). 3--the response identifies some of the features to attend to and some understanding of the mental processing ("If I see a word that has 23 in it, I say the sound of 93."). 4--the response includes a sequence of the mental processing or a specific example of the mental processing (when I meet an unknown word such as loud, I think first...and then...etc.). -n-aoaruuu—‘pu-tuuu 1 APPENDIX C Supplemental Achievement Measure SAM-Content Item Researcher reads directions orally: Read the sentence. Decide what the base word is for the underlined word. Jan and Sandy were planning a special trip to the sea in the summer. (Student reads sentence) Researcher reads directions orally: Now choose the base word for the underlined word. Put an X before the correct answer. plane planned plan (Student marks the answer) SAM-Process Item Researcher reads directions orally: I am going to read a question and four possible answers. Choose the best answer. Put an X before the best answer. You just chose a base word; how did you decide which base word was the right one for the underlined word in the sentence? I looked for the word that looked most like the word in the sentence. ' I just knew what the base word was. I took off the ending and that helped me find the base word that would make sense. I thought about the sea and that was a clue that helped me choose the base word. 159 APPENDIX D Modified Graded Oral Reading Paragraph (GORP) Note: Adapted from w. Durr. (1983). Placement test, Houghton Mifflin Reading Series. Boston: Houghton Mifflin. 1. earth hour fire egg catch . 3‘- 2. king pass act milk blew § 3. touch form plane eight reach r 4. thick base warm tale final 5. port fresh train women spoon 6. check island complete notice usual When the young skunks were eight weeks old, the mother skunk took them on their first hunt. It was at night. Skunks hunt at night and sleep in the day. The young skunks followed along behind their mother in a single line, their bushy tails held up high. Skunk Baby was the last in line. The moon was shining down through the threes. The mother skunk stopped by a log. With her sharp, strong claws, she dug at the rotting wood. She uncovered some small grubs and snapped them up. Skunk Baby tasted a fat grub and licked his lips. Suddenly the skunks heard a strange noise at the other end of the log. A round, bristly-looking animal walked past. The mother skunk did not even look at the old porcupine. She was not afraid of him. He was not an enemy. She gave her young 3 sign to follow her. And off the family waddled down the path and toward the pond. 160 161 From the pond came the song of the frogs. Under rocks and leaves, crickets rubbed their wings together, making a cheerful, chirping sound. The frogs' singing grew louder. The skunks were almost at the pond. Suddenly there was a soft, swishing sound overhead. A great horned owl swooped down. The owl was a dangerous enemy! The mother skunk stamped her front feet. Her family quickly scrambled under a thorny bush. ‘ am? The branches were so full of sharp thorns that it was impossible for the owl to land. Soon it hooted and flew away. When she was sure it was safe, the mother led her family to the pond. They walked to the edge and drank the cool'water. ' APPENDIX E Concept of Reading Measure 1. involves intentionality involves intentionality to decode to get meaning 2. involves effort involves no effort g 3. is unsystematic is systematic g 4. is self-directed is other-directed 5. involves problem- does not involve problem- solving solving 6. uses skills/rules uses skills/rules to to gain meaning decode 7. is for the purpose of does not involve getting getting meaning meaning 8. involves no conscious involves conscious processing processing 9. involves selection among involves no selection strategic processes among strategic processes 10. involves the idea that that printed material is printed materials tell to be decoded in segments the reader something All categories were rated on a seven-point scale: 7 = strategic response, 1: non-strategic response. 162 APPENDIX F Criteria for Explicit Instruction Rating Information Presented about the Strategy Rate how explicit the teacher is in informing students that the task to be learned is a strategy for solving a problem encountered in reading. O--the teacher makes no statement about what is to be learned (total absence of ..). 1--the task is named/labeled but there is little information beyond "we will learn about prefixes..." 2--the task is named/labeled and there is some elaboration beyond "we will learn about prefixes..." 3--the task is described as an adaptive, flexible strategy ("we will learn how to...") but it is not an exemplar. 4--an exemplary presentation of the task is an adaptive, flexible strategy to solve a problem encountered when reading. Rate how explicit the teacher is in informing the students that the strategy is useful as they read. 0--there is no statement of where the skill would be used (total absence of...). 1--the teacher only mentions that the skill is generally useful or useful in reading but does not specify why or when. 2--the usefulness of the task is related to the future ("when you get in sixth grade...") or is vague or general in stating why or when it is related to particular text ("it helps you get information..."). 163 “I. r“- :5 51%. . ‘ "a 164 3—-the immediate usefulness of the skills is illustrated with a specific reference to a particular example but it is not an exemplar. 4—-an exemplary statement of the immediate useful- ness of the skill in reading connected text in which one or more concrete examples are used to illustrate. Rate how explicit the teacher is in telling students how to decide which strategy to select for use when encountering a problem in reading. O--there is no mention that students will have to select a strategy to solve the problem (total absence of...). 1--the teacher mentions that this skill can be used to solve a problem but provides no additional information. 2--the teacher mentions that this skill can be used to solve a problem and provides some information about how to choose the appropriate strategy. 3--the problem situation is explicitly specified and-how to select an appropriate strategy is emphasized but it is not an exemplar. 4--an exemplary statement of how to recognize that problem exists and how to select the appropriate strategy. Rate how explicit the teacher is in telling students how to perform the strategy to solve the problem when reading real text. 0--there is no explanation of how to perform the strategy (total absence of...). 1--there is an explanation but it is stated as a rule to be memorized or as a procedure to be ‘ recalled and no examples are provided. 2--the teacher talks about the rule and/or procedure as routine to be applied without variation and examples are provided. 3--the teacher shows students how to follow mental steps and a sequence in a flexible, adaptive VT "3 K? -. . ’fir- - -I as’AJ - in. 165 'manner but it is not an exemplar. 4--an exemplar description in which the teacher shows students how to follow mental steps and a sequence flexibly and adaptively when perform- ing the strategy. II. The Means Used to Present the Information 1. Rate how explicit the teacher is in introducing the lesson. O—-the teacher makes no introductory statements or overview regarding the lesson (total absence of...). 1--the teacher makes an introductory or overview statement about what is to be learned, but does not mention why or how. 2--the teacher makes an introductory or overview statement about what is to be learned and either why or how (but not both). 3--the teacher shows students how to follow mental steps and a sequence in a flexible, adaptive manner but it is not an exemplar. 4--the teacher makes an exemplary introductory or overview statement about the strategy to be learned, the "real text" situation in which it will be applied and what to attend to when using it. 2. Rate how explicit the teacher is in modeling for students the mental steps in identifying the problem, selecting the strategy, and applying the strategy. O--the teacher does not model how to do the task at any point in the lesson (total absence of...).. 1--the teacher models the procedural use of a rule. 2--the teacher models the steps to be followed as a procedure but does not make the invisible use of a rule. 4. 166 3--the teacher models mental steps in using the strategy adaptively (makes the invisible visible) but used artificial text samples or otherwise is not an exemplar. 4--the teacher provides an exemplary model of how to use mental steps in applying the strategy adaptively to a sample of natural connected text. Rate how well the teacher shifts the instructional interaction from teacher regulation of the strategy to student control of the strategy. O--the teacher does not provide any guided practice (total absence of...).. 1--the teacher requires the students to provide answers to tasks which presumably call for the use of the skill (in a recitation or assessment mode). 2—-the teacher moves from teacher regulation to student regulation but the emphasis is on answers rather than student mental processing. 3--the teacher moves from teacher regulation to student control and emphasizes student mental processing rather than answers, but it is not an exemplar. 4--the teacher provides an exemplary series of trials which are characterized by increased student mental processing, but much teacher assistance early in the lesson, by teacher monitoring of students use of mental processes, and by making reference to the monitoring of student responses in asking for subsequent responses. Rate how well the teacher elicits responses which require students to verbalize how they arrive at their answer. ' "t man-2|»?! “ g _____IJ; i_ 167 O--the teacher does not elicit student responses to the skill of the task (total absence of. .). 1--the teacher elicits right answers and does not require students to state how they know the answer. 2--the teacher requires students to state how they got answers but focuses on procedural recall rather than knowing how to get the answer. 3--the teacher requires students to explain how they got the answer but has individual students g verbalize individual steps rather than having ' each student verbalize all the steps, or other- wise fail to be an exemplar. q,’ 4--the teacher's elicitations are exemplary, requiring each student to verbalize all the mental steps used in applying the skill strategically. _"_"'“\' i Rate how well the teacher brings closure to the observed lesson (or lesson segment). O--there is no evidence of closure to the lesson (total absence of...).. 1--the teacher ends the lesson but makes no summary statement about the skill being taught. 2--the teacher makes a summary statement but does not include all information (the what, the why and the how). w :’-' . H {fig-sf... ~ . an..- . 9 . 3--the teacher ends the lesson with a summary statement about what was learned, why it was learned and how to do it (but does so without student involvement or otherwise fails to be an exemplar. 4-—the teacher provides exemplary closure by involving students in summarizing and/or in reviewing, or in using the skill strategically in natural connected text, or by reminding them that it is in such natural connected text that the skill will be used. III. Intra- and Inter-Lesson Cohesion 168 Rate how successful the teacher is in bringing a sense of cohesion to the lesson. O--there is no recognizable sequence or cohesion within the lesson.(total absence of...). 1--the teacher's lesson has some evidence of a logical sequence but there are frequent inconsistencies and breaks. 2--the teacher's lesson reflects a logical progression but contains some inconsistencies or breaks in lesson focus or breaks in activity flow. 3--the lesson has structure, is consistent, is .4 focused and flows smoothly but is not an exemplar. 4--the teacher provides a lesson which is exemplary in terms of internal structure, consistency, focus and flow. Rate how successful the teacher is in communicating a sense of cohesion with past and future lessons. O--there is no recognizable connection to past and future lessons (total absence of...). 1--the teacher refers to past lessons but makes no reference to future lessons or refers to future lessons but makes no reference to past lessons. "2} 2--the teacher refers to past and future lessons but there is little evidence of cohesion. :7..— 3--the teacher refers to past and future lessons, achieves some cohesion across lessons, but it is no exemplar. 4—-the teacher provides an exemplary lesson in terms of its cohesion across lessons. APPENDIX G Management Principles Rating Criteria Teacher provides a standard and predictable signal to get attention? Teacher faces class with small group while students face away? Overview of what is to come is provided? New words and sounds are presented before Story is read? Students repeat new sounds or words until said satisfactorily? Teacher presents information? Teacher works with individual students as they ' practice? Teacher uses a pattern for turn taking? Teacher occasionally questions a student about another student's response? Teacher calls on volunteers only when personal experiences or opinions are related? When call outs occur, teacher reminds the student that everyone gets a turn and he/she must wait? Teacher avoids leading or rhetorical questions? Teacher provides: I. wait time for questions? 2. feedback about incorrect answer? 3. answer if answer can't be reasoned out? and 4. clues if answer can be reasoned out? —To:am79_ 71“"fi i}. t (5.. Teacher makes sure all students hear and understand correct answers? Teacher provides praise in moderation? Teacher provides specific criticism and specification of correct alternatives? 169 LIST OF REFERENCES LIST OF REFERENCES Anderson, L., Evertson, C., & Brophy, J. (1979). An experimental study of effective teaching in firs t-grade reading groups. Elementary School Journal, 22, 193-223 Baker, L. & Brown, A. L. (1984). Metacognitive skills and reading. In P.D. Pearson (Ed.), Handbook of reading research (353—394). New York: Longman. Bloom, B. S. (1974). Time and school learning. American Psychologist, £2, 682-689. Borkowski, J.G., Carr, M., & Pressley, M. (1987). "Spontaneous" strategy use: Perspectives from Metacognitive theory. Intelligence, 11, 61- 75. Bransford, J. D. (1979). Human cognition: Learning, understanding, and remembering. Bellmont, CA: Wadsworth Publishing Company. Brainin, S. S. (1985). Mediating learning: Pedagogic issues in the improvement of cognitive functioning. In E. Gorden (Ed.), Review of research in education, (Vol. :12) (121-156). Washington, D. C.: American Educational Research Association. Brophy, J. E. & Good, T. L. (1986). Teacher behavior and student achievement. In M. C. Wittrock (Ed), Handbook of research on teaching (Third Edition) (328-375). NY: Macmillan. Brown, A. L. (1978). Knowing when, where and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in instructional psychology (77-165). Hillsdale, NJ: Lawrence Erlbaum Associates. Brown, A. L. (1980). Metacognitive development and reading. In R.J. Sprio, B.Bruce, & w. Brewer (Eds.), Theoretical 170 Li 171 issues in reading comprehension (pp. 453-481). Hillsdale, NJ: Lawrence Erlbaum Associates. Brown, A. L., Bransford, J. D., Ferrara, R. A., & Campione, J. C. (1983). Learning, remembering, and understanding. In J. H. Flavell & E. M. Markman (Eds.), Carmichael's handbook of child psychology (Vol. 3). New York: Wiley. Cavanaugh, J. C. & Perlmutter, M. (1982). Metamemory: A critical examination. Child Development, 23, 11-28. Chipman, S. E., Segal, J. W., & Glaser, R. (1985). Thinking and learning skills: Research and open questions (Vol. 2). Hillsdale, NJ: Lawrence Erlbaum Associates. Clark, C. M. & Peterson, P. L. (1986). Teachers' thought processes. In M. C. Wittrock (Ed.), Handbook of research on teaching (Third Edition) (255-296). New York: Macmillian Publishing Company. Clay, M. (1972). Reading: The patterning of complex behavior. Aukland, New Zealand: Heinemann Educational Books. Craik, F. I. M. & Lockhart, R. S. (1972). Levels of Processing. Journal of Verbal Learning and Verbal Behavior, 11, 671-684. Doyle, w. (1986). Classroom organization and management. In M. C. Wittrock (Ed.), Handbook of research on teaching (Third Edition) (392-431). New York: Macmillan. Duffy, G. G., Roehler, L. R., Sivan, E., Rackliffe, G., Book, C., Meloth, M., Vavrus, L., Wesselman, R., Putnam, J., & Bassiri (1987). Reading Research Quarterly, 22, 347-368. Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis. Cambridge, MA: MIT Press. Fillmore, L. w. & Valadez, C. (1986). Teaching bilingual learners. In M. C. Wittrock (Ed), Handbook of research on teaching (Third Edition) (648- 685). New York: Macmillan. Fischer, P. M. & Mandl, H. (1984). Learner, text variables, and the control of text comprehension and recall. In H. Mandl, Stein, N. L., & T. Trabasso (Eds.), Learning and comprehension of text (213-254). Hillsdale, NJ: Lawrence Erlbaum Associates. 171 Flavell, J. H. (1976). Metacognitive aspects of problem solving. In B. C. Resnick (Ed.), The nature of intelligence (231-235). Hillsdale, NJ: Lawrence Erlbaum Associates. Flavell, J. H. (1978). Metacognitive development. In J. M. Scandura & C. J. Brainerd (Eds.),Structural/ process theories of complex human behavior. The Netherlands: Sijthoff and Noordhoff. Flavell, J. H. (1979). Metacognition and cognitive monitoring. American Psychologist, 34, 907-911. Flavell, J. H. (1981). Cognitive monitoring. In W.P. 5 Dickson (Ed)., Children's oral communication skills ” (35-60). New York: Academic Press. Flavell, J. H. & Wellman, H. M. (1977). Metamemory. In R. V. Kail & J. w. Hagen (Eds.), Perspectives on the development of memory and cognition (pp. 3-34). T_ Hillsdale, NJ: Lawrence Erlbaum associates. Forrest-Pressley, D. L. & Gillies, L. A. (1983). Children's flexible use of strategies during reading. In M. Pressley & J.R. Levin (Eds.), Cognitive strategy research: Educational applications (pp. 33-156). NY: Springer-Verlag. Gagne, R. (1977). The conditions of learning, (Third Edition). New York: Holt, Reinhart, & Winston. Garner, R. (1987). Metacognition and reading comprehension. Norwood, NJ: Ablex Publishing Corporation. Gough, P. (1984). Word Recognition. In P.D. Pearson (Ed.), Handbook of Reading Research (pp. 225-254). New York: Longman. “it: I. Hansen, J. & Pearson, P. (1983). An instructional study: Improving the comprehension of fourth-grade good and poor readers. Journal of Educational Psychology, 15, 821-829. E I-“ J... Hartley, J. & Davies, I. K. (1976). Preinstructional strategies: The role of pretests, behavioral objectives, overviews, and advance organizers. Review of Educational Research, 46, 239-265. Hull, C.L. (1930). Knowledge and purpose as habit mechanisms. Psychological Review, 31, 511-525. 172 Hundiede, K. (1980). The origins of the child's replies in experimental situations. Laboratory of Comparative Human Cognition, 2(1), 15-18. Jacobs, J. & Paris, 5. (1987). Children's metacognition about reading: Issues in definition, measurement, and instruction. Educational Psychologist, 22, 255-278. LaBerge, D. & Samuels, S. (1974). Toward a theory of automatic information processing in reading. Cognitive Psychology, 6, 293-323. Markman, E. (1981). Comprehension monitoring. In w.P. Dickson (Ed)., Children's oral communication skills (pp. 61-84). New York: Academic Press. Meichenbaum, D. (1985). Teaching thinking: A cognitive- behavioral perspective. In S.F. Chipman, J.W. Segal, & R. Glaser, R. (Eds.). Thinking and learning skills: Research and open questions (Vol. 2) (pp. 407-426). Hillsdale, NJ: Lawrence Erlbaum Associates. Meichenbaum, D., Burland, 8., Gruson, L., & Cameron, R. (1979). Metacognitive assessment. Paper presented at the conference on the Growth of Insight, Wisconsin Research and Development Center, Madison, WI. Nisbett, R. E. & Wilson, T. D. (1977). Telling more than we know: Verbal reports on mental processes. Psychological Review, 25, 231-259. Palincsar, A. S. & Brown, A. L. (1984). Reciprocal teaching of comprehension fostering and monitoring activities. Cognition and Instruction, 2, 117-175. Paris, S. (1984). Improving children's metacognition and reading comprehension with classroom instruction. A paper presented at the annual conference of the American Educational Research Association, New York. Paris, S. G. (1987). Meta-Metacognition: Does metacognition immobilize or enable proficient reading?. Paper presented at the annual conference of the National Reading Conference, St. Petersburg, FL. Paris, S. G. & Jacobs, J. E. (1984). The benefits of informed instruction for children's reading awareness and comprehension skills. Child Development, 55, 2083-2093. '— 173 Paris, S. G., Wixson, K. K., & Lipson, M. Y. (1983). Becoming a strategic reader. Contemporary Educational Psychology, 2, 293-316. Paris, 8. G., Cross, D., & Lipson, M. (1984). Informed strategies for learning: A program to improve childrens' reading awareness and comprehension. Journal of Educational Psychology, 26, 1239-1252. _— Paris, 5. G., Jacobs, J. E., & Cross, D. R. (1987). Toward an individualistic psychology of exceptional children. In J. Borkowski & Day, J. (Eds.), Intelligence and cognition in special children: Comparative approaches to retardation, learning disabilities, and giftedness (pp?— 215-248). NY: Ablex. Paris, 8. G., Newman, R. S., & McVey, K. A. (1982). Learning the functional significance of mnemonic actions: A microgenetic study of strategy acquisition. Journal of Experimental Child Psychology, 22, 490-509. Paris, S. G., Wixson, K. K., & Palincsar, A. S. (1986). Instructional approached to reading comprehension. In E. Rothkopf (Ed.), Review of research in education (pp. 91-128). Washington, D.C.: American Educational Research Association. Roehler, L. R., Duffy, G. G., & Meloth, M. S. (1986). What to be direct about in direct instruction: Content-only versus process-into-content. In T. E. Raphael (Ed.), The contexts of school-based literacy (79-96). NY: Random House. Rosenshine B. & Stevens, R. (1986). Teaching Functions. In M. C. Wittrock (Ed), Handbook of research on teaching (Third Edition) (376-391). NY: Macmillan. Segal, J. W., Chipman, S. E., & Glaser, R. (Eds.), Thinking and learning skills, Volume 1: Relating instruction to research. Hillsdale, NJ: Lawrence Erlbaum Associates. Shulman, L. s. (1986). Paradigms and research programs in the study of teaching: A contemporary perspective. In M. C. Wittrock (Ed), Handbook of research on teaching (Third Edition) (3-36). NY: Macmillan. Wagoner, S. A. (1983). Comprehension monitoring: What it is' and what we know about it. Reading research Quarterly, 22, 328-346. 174 Weiner, B. (1979). A theory of motivation for some classroom experiences. Journal of Educational Psychology, 22, 3-25. White, P. (1980). Limitations on verbal reports of internal events: A refutation of Nisbett & Wilson and of Bem. Psychological Review, 22, 105- 112. Winne, P. H. (1982). Minimizing the black box problem to enhance the validity of theories about instructional effects. Instructional Science, 22, 13-28. Winne, P. H. (1985). Steps toward promoting cognitive achievements. The Elementary School Journal, 22, 673-693. Wixson, K. K. & Lipson, M. (1986). Reading (dis)ability: An interactionist perspective. In T.E. Raphael (Ed.), 222 contexts of school-based literacy (pp. 131-148). New York: Random House. Yussen, S. R., Mathews, S. R., & Hiebert, E. (1982). Metacognitive aspects of reading. In W. Otto & S. White (Eds.), Reading expository material (pp. 189-218). New York: Academic Press. 175