‘3‘"; m‘ eutwvaw‘w 1‘3: ‘3'» a :9 := '2 1.3“" ‘5’? n n .. ".9 hang»- 75%;. mun-{us who“ n Ana 1 r .. I a ‘7’ ,.. M... . ,.. 2.2.. ENS" .-_ 2 n4 M- tut...” 4, ~ r. q u. 58:. - ”sac". : ("w-.43“ :' ( :5 £31233? ”I"! J G“ u ‘0 c .. ”'5.” 3,. L}. a ”d ’.;l\o‘1‘in‘ . urn“. .. a...” '; ..': . .1 9 L- L 2.25:.“ ‘_. ., L J“. A‘ ’ 1..) .4 l .' ”Aw. . ...- .m.‘ L ‘vvf 1 ~ rum" ~. {I .J'." v" . 35': am. ., n :9“: .;.. .1. ' ”In 1.}.F.:,..x;:‘z F. -1. ; w , .2 .1 5:. ‘.v‘- * ~ .7 .c/ .n I 1"- u ‘ \‘a ”4‘5; If .Khv‘v _.,.: r' A" ujt 1, ~10- B . , r'. ‘7, . : Elf-{Lync- 11‘ ~ “893‘: ‘ $331.“ , my: > ,. .u nn ’9‘“ {1. 42;: , ‘ ‘ J‘r .4 g. - i: 4232"?” "Hi; :55! ,1 . ”2:, 4,43: »: 3.1; 1‘: 3r .- . «- —A"p,|p. , “,4... p’... ‘ ”1', ‘ i'gé“? "31:29 . "””'f. L. éflfiuv": . -~. v 1w - .. 132:”‘5‘ 12'“ "W .2 -. -: waning,“ I. ' 1' I n 1'". a“ I «H g ”(r ’2 u- :;"‘:C,..’... .< pm (I. fluff! . . .u- w- y. «(37" 9:23}, .7 'Afirmzrr ., n: 2;»: .2 :3 253;." 2-2; Fae-17' ., I¢ y’a‘v. - I‘h 0.1-4" «_ .1 . u éhf ‘ ' a, ‘ 25:3“ amt-:41: r wurnnjv-‘f‘r‘ne‘n‘i .. «jr‘y...» «#136 {-13% ”1' .n . v ‘7‘? II... 2.3.5:: l ""5.‘ "'11.(1 “v: “’1 3.3:“ ‘1» u umém ’w ‘V. ”mart“ ""2: "W .. . . .0. 1.1..‘1‘1‘3: a. It»? W 61,-“? awry- 23".} .1 man. 2..., - 5g..,.~.“~'5...:::::£,.:1* "'r - . :‘Q‘I‘ 112:!“ aazrr‘r If"; ”my; - 1&5 IV :ay’lwh'zwm F ~- ”K1-.. .i- ' u... fiwiflj 3:32. " 1:.- 2"“ "" {3? ... . £1.35?" ‘7. '3“.‘1':‘..'3}:ld€‘u . 3;? ”ii/9.x 2,11,.mvm‘m" . 5;}; "EL! - 4n ’41:", ,1." .... z . '2 1 ' . . 2m .5; ‘1'“ 229* mm, ., . a 9 w.1mfl2“’w .. . fi THE“ 77fl/793 nsnnsmuam ||l||l(|l|HH||iHIUHHIIUIHI‘ Wt. llllllllHlll 3 1293 00774 9868 TI LIBRARY Michigan State 1 University L .J ‘ This is to certify that the dissertation entitled TEACHERS' INTERPRETATION AND USE OF STATE ACHIEVEMENT TEST RESULTS presented by Rossi J. Ray-Taylor has been accepted towards fulfillment of the requirements for /o/)~D degreein ac/deif0/1 6/ [Oxycfioiw gW/tt 3% Major professor/ Date fl/W/ifig/ AISU is an Affirmative Action/Equal Opportunity Institution 0-12771 PLACE IN RETURN BOX to remove this checkout from your record. TO AVOID FINES return on or before date due. DATE DUE DATE DUE DATE DUE “Wrath uwfié KEY 27 200? L.__ MSU I: An Aflirmdtve Action/Equal Opponunlty Institution emu-n39 1 TEACHERS’ INTERPRETATION AND USE OF STATE ACHIEVEMENT TEST RESULTS By Rossi J. Ray-Taylor A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Counseling, Educational Psychology and Special Education 1991 .. weft? .2 39"; Cd/ 7‘ TEACHERS' INTERPRETATION AND USE OF STATE ACHIEVEMENT TEST RESULTS BY Rossi J. Ray-Taylor ABSTRACT This is the study of teacher descriptions of their use and interpretation of one kind of test data, state mandated assessment data, the Michigan Educational Assessment Program (MEAP). Through two Focus Groups and six Interviews I heard the descriptions and stories of third and fourth grade teachers. I set these in the context of time, location and local culture to find the meaning in the teachers' wdrds. This study took place in Lansing, Michigan, the state capital and 'home of the Michigan Department of Education. Lansing has a long history of local control and practices a form of site-based management. These contextual factors are described as the background against which this study takes place. This study uncovers four major themes related to teacher use of state assessment test results. The words of these third and fourth grade teachers and one principal are woven together to find contrasts and similarities. 1‘ Rossi J. Ray-Taylor The descriptions of the teachers make vivid the frustration teachers feel with pressure to improve test scores and little direction as to how. Teachers described the pressure of too much to teach in too little time. They worry about teaching to the test, but are undlear about what are appropriate test preparation practices. Practical lessons are learned from this study about how school administrators and policy makers can view state assessment testing and the capacity of these tests to bring about reform in schools. They are cautioned about the wisdom of test reform leading curriculum reform. Results of this study lead to concern that the pressure to improve test scores may actually undermine curriculum and erode valuable instructional time. Suggestions are made to test designers about ways to design tests to be more useful to teachers and to increase the likelihood that improved teaching and learning will result from their USE. Copyright ROSSI J. RAY-TAYLOR 1991 DEDICATION This dissertation is dedicated to my family: to my parents for giving me the foundations of ambitions, a love of learning and perseverance; and to my sisters for their confidence and support. I also say a special thank you to my husband, Gary, and our sons, Mathis and Marshall, for their sacrifices, patience, love, and encouragement . iii ACKNOWLEDGMENTS The assistance of the following individuals is gratefully acknowledged. Each provided special expertise, guidance, and thoughtful questions and comments. -- Christopher Clark, Ph.D., my chairperson, for his guidance and support. -- Susan Florio-Ruane, Suzanne Wilson, and Ralph Putnam, my committee, for their insightful comments and discussion. -- David Donovan, Ed Roeber, and Geraldine Coleman for their assistance and openness regarding MEAP. -- Pat Petersen and Jane Faulds for their support and willingness to share their practices and to facilitate data collection. -- Mary Keno for leading the Focus Groups so expertly. -- Marge Crawford for making the video tape of Focus Group One. -- Barbara Reeves for final typing and formatting. -- Kathy Wagner for her assistance along the way. -- Gary Lynn Taylor for computer-consultatiOn. -- Richard Halik for his unwavering support and patience. -- My many friends and colleagues with the Lansing School District for their confidence and interest. Finally, I thank the teachers and principal who shared their views and experiences with me. I appreciate their taking time from their crowded days to talk with me to help me understand MEAP testing. iv TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES . CHAPTER ONE: INTRODUCTION What Is This Study Trying to Learn About? Why Michigan, Why Lansing? . . . Research Questions . The Researcher . Organization of This Document CHAPTER TWO: REVIEW OF THE LITERATURE Teacher Use and Perceptions of Test Data . How Teachers View Tests . . Other Influences on Test Data . Looking at the Findings Together . Teacher Thinking and Standardized Testing . . How Teachers Use Test Results in Classrooms . Early MEAP Studies . Recent Perceptions of MEAP MEAP and Public Policy MEAP and the Curriculum . Local Control . . . . External Standards for Teachers . CHAPTER THREE: METHOD: ASKING TEACHERS TO TELL THEIR OWN EXPERIENCES . Design and Procedure . Design. . . Focus Groups . Other Approaches Considered . ll 13 14 14 15 l7 19 22 23 24 25 28 28 3O 30 33 Site . Population and Sample The Focus Group Technique Teacher Interviews . Principal Interview Data Analysis CHAPTER FOUR: PRESENTATION AND ANALYSIS OF THE DATA History of MEAP . . . . Description of Current MEAP Tests . MEAP Mathematics Test . MEAP Science Test . MEAP Reading Test . Report Formats . . State Expectation for the Use of MEAP Results . The Context of MEAP in Lansing . . Lansing' 3 Focus on Improving MEAP Scores Effective Schools . . . . . Analysis of the Data . How the Data Were Analyzed . Strategies Used . Writing Narratives . The Relationships among the Themes, Storyboard, and Data Themes . . . . . . . . Elaboration of the Themes . Perceived Usefulness: They Don' t Tell Me Anything . Test Results Looked at as a Unit Teaching to the Test MEAP is leading to a state- wide curriculum . Results are confusing: What do all the numbers mean? . . Teachers Are Overwhelmed, and MEAP Aggravates This Condition . . Not enough time Too much to teach Too much testing . . The Pressure Is High, but the Stakes Are Low No consequence . . . . . . . . . . Storyboard The Data Teacher Use of MEAP . . Goal Setting, Affirmation, and the Big Picture Summary. . . . . . . . . . . . . . . vi 35 37 40 42 43 A3 48 48 49 50 52 53 54 55 59 60 63 63 65 67 67 69 7O 70 72 73 74 75 76 76 77 78 78 8O 82 85 85 9O 92 Teacher Preparation of Children for MEAP . Teacher Access to the Test Content Test Content and the Curriculum . District and Teacher Activities to Prepare for MEAP . Putting MEAP Preparation to Perspective . A Difficult Item--Contrasting Responses . Valoria Decides If and How to Respond to Difficult Items . MEAP and the Curriculum Mathematics . Reading . Science . . State- wide Curriculum . . . Time, Testing, and the Crowded Curriculum . Teachers Analyze MEAP Items Rose' 3 Story: Responding to Unrealistic Concepts Pressure to Score Well on MEAP . . . . . Within School Pressures . Pressure and Stakes . . . Bill's Story: Responding to One Teacher' s Low Score . Contrasting School Responses to MEAP Results . Contrast 1: A Teacher and Principal's Perspective on Low Scores . . Contrast 2: Multiple Teacher Perspectives on Declining Scores Contrast 3: The Contrast Between Two Low Scoring Schools . CHAPTER FIVE: LESSONS LEARNED AND THEIR IMPLICATIONS . Overview . . Lessons Learned . Lessons Learned about Curriculum MEAP and the Curriculum . Proliferation of State Achievement Testing Competing Curricula . . Should MEAP Influence Curriculum? . MEAP Has a De Facto Influence on Curriculum There Is No One Best Way . . State Control of Local Curriculum and Instruction . Distorting Practice and Curriculum . . Teaching to the Test/Teaching to the Curricuium . Lessons Learned about the Meaning of MEAP . Pressure and Stakes . . . Importance of MEAP to Teachers . Teachers Put MEAP into Perspective . Lessons Learned about MEAP Use, Interpretation, and Increasing Scores . vii 93 93 95 96 97 98 101 102 102 106 108 111 112 114 117 118 120 122 125 126 127 127 128 129 129 129 130 130 130 133 134 135 136 137 138 139 141 141 144 144 145 Intended Use and Interpretation of MEAP Mandates to Increase Test Score Support for Increasing Scores . . Conclusions: The Call to Action in the Lessons . The Practical . The Study of Testing The Stories Told Results and Implications APPENDICES A: Discussion Group (Focus Group) Questions . B: Interview Questions C: Letters of Invitation D: Assurance Form . E: Profiles . F: MEAP School Sample Packet REFERENCES viii 145 147 148 149 150 151 152 154 158 158 159 162 165 166 173 182 LIST OF TABLES Positive Effects of MEAP Ranked by Frequency of Response: Five Most Frequent Responses . . School MEAP Ranking--Reading Category 3 MEAP Ranking--Reading Category 3 . MEAP School Results MEAP Skill Areas Assessed (Mathematics and Science) MEAP Mathematics and Science Tests Categories of Achievement Criteria . . . . . . . . MEAP Essential Skills Reading Test Content . MEAP Reports Available to School Districts . Local Conditions and District Responses to Testing . ix 21 34 39 45 51 52 56 57 143 4.1 4.2 LIST OF FIGURES Page Relationships among Essential Goal Statements, MDE Core Curriculum, MEAP, and Lansing School District Curriculum Statements . . . . . . . . . . . . . . 61 1003 . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 CHAPTER ONE INTRODUCTION Demands for educational accountability and need for a response to the spate of reports lead by A_flg§igg_§t_§1§k have resulted in nearly every state in the United States adopting some form of state achievement testing. These tests differ in their consequences. Some are high stakes and determine high school graduation for the students taking them; others are lower stakes tests. Regardless of the level of stakes, state policy makers expect some action to result from state assessment tests. Policy makers anticipate that the test results will be used and interpreted by teachers to the betterment of public education (e.g., Michigan Department of Education, 1990). Given the nature of public schools, they reason, it is up to teachers at the local schoolhouse level to respond constructively to these tests. 5 S ud b u This is a study to learn how teachers interpret and use one kind of test data, state assessment data. One cannot assume that all tests given by teachers are used by or are useful to teachers. There are many reasons why some tests are used and useful and why other tests are not. ...—n—--.r This study looks at state assessment tests because the use of these tests has grown dramatically in the past decade. This study asks teachers in Lansing, Michigan to talk about their use and interpretation of state assessment results. The Michigan Educational Assessment Program (MEAP) is a reoccurring event in the lives of Michigan teachers and school children. First administered in January 1970 in grades 4 and 7 in reading and mathematics, MEAP is now administered in three subject areas at six grade levels. (Occasional pilot and voluntary tests are administered in other subject areas such as writing, social studies, and health, etc.) All general education public school students in Michigan pass through the MEAP terrain several times throughout their educational careers. The development of MEAP was part of a larger state effort to enact an instructional accountability model for local school districts and schools. Michigan policy makers expect school district personnel to take action to improve low MEAP results (Michigan Department of Education, 1990). State compensatory education funding (Article 3 of the state school aid act) is awarded to districts using a formula based on MEAP results in reading and math. For the past 2 years districts maintaining gains in MEAP scores have been rewarded through extra funds in the state school aid act. Schools that receive Article 3 funds and do not improve their test scores are required to submit improvement plans. (At one time they received financial penalties as well.) MEAP results are often published in local newspapers with comparisons made between school districts and between individual schools. 0—- MEAP drives curriculum. To score well on the test districts change the format or sequence of instruction to better reflect items and objectives tested on MEAP. Many local school districts align their curriculum with MEAP objectives (Ingham Intermediate School District, 1986). The new MEAP reading test, along with state reading objectives, for example, has been a catalyst in districts adopting a whole language approach to reading. MEAP results are reported in a variety of formats. The tests and report formats have changed over the years reflecting advances in test data analysis technology, to ease interpretation, to conform to current practice and research in the subject areas tested, and, most recently, to reflect the different philosophy of mastery inherent in the new reading MEAP. W In the history of state assessment testing in Michigan we see a number of interesting characteristics. Michigan has a long history of state assessment testing. Michigan's assessment testing program began in 1970 and has gone through revisions to reflect recent research and practice in the subject matter tested. Michigan has long held policies in support of local control of its K-12 school districts. More than half of the dollars raised to support public education are raised through property taxes levied by the local school district (Citizens Research Council of Michigan, 1990, p. 3). Local districts establish their own curriculum and graduation standards. Locally elected school boards determine the policy and hold responsibility for running Michigan's schools. School district staff are hired by each local school district, which determines the wages, duties and working conditions of its employees. Michigan policy has begun to change. For example within recent years school administrator and teacher certification requirements have been adopted by the state legislature. School districts have been offered monetary incentives to adopt state high school graduation requirements and to adopt a state core curriculum (Public Act, 25. 1990). State incentives have also been used to lower class size in grades K-3, and to encourage local school districts to offer foreign language classes at the middle and elementary school levels (State School Aid Acts, 1987, 1988, 1989, 1990). Michigan represents an interesting case of a fluctuating mixture of local and state level control. Within Michigan Lansing reflects a similar mixture of controls. In the late 1960's Lansing adopted a model of school level decision making called ”responsible building autonomy“. At one point this model allowed each school to determine its own curriculum, adopt its own textbooks, hire staff after minimal screening by the district personnel division, and maintain a building based budget (Lansing School District memorandum; Iverson, 1991). Over the years Lansing has retreated from some of the characteristics of its autonomy model (for example, textbooks are now adopted and purchased centrally to allow for better cost management and uniformity of instructional expectations). Lansing staff are in the process of reaffirming the meaning of autonomy in the wake of site based management, district management and budget concerns and in the context of its commitment to the effective schools process. In Chapter 4, I describe a decade-long effort of Lansing staff to improve test scores and how specific activities related to improving MEAP scores were developed and carried out. Any of these features may not be unique to the Lansing School District but they combine in Lansing to provide a backdrop for this study that reflects many of the major policy issues facing schools today. These observations along with my long history with the Lansing School District and close work with MEAP from administration, analysis, and use perspectives give a powerful vantage point from which to conduct and interpret this study. Teachers receive MEAP results a few weeks after it is administered, typically in early October. Once teachers unpack the box, open the envelopes and spread the results on the table, how do they make sense of the results and integrate MEAP results into their lives and work as teachers? This question is the focus of this study. ese es This study explores two major questions: (a) How do teachers interpret and use one kind of test data, state assessment tests; and (b) How do teachers make sense of the MEAP results and integrate them into their lives and work? W The questions raised in this study came from my early interest in the use of data for decision making in the education setting. As a program evaluator who compiled masses of data for policy makers and practitioner review I was curious as to the subsequent use of data in of :1: SC St po t0 ass its many forms and purposes. Over time that curiosity narrowed to consider how test data are used by teachers with particular interest in how instruction, pedagogy and curriculum are shaped by tests. Job changes required me to design, implement and monitor educational programs. My professional responsibilities grew to analysis of government policy and lobbying government officials on behalf of the local school district, grant writing and most recently administration of the school district's state and federal program office that manages the state, federal and grant funded programs of the school district. In this position my staff and I are responsible for the design and administration of millions of dollars of programs to improve student school performance. We rely on plans developed by principals and school staffs (presumably determined in part by test data). This cabinet level position allows input on policy and practice level decisions. I have had a long and close association with MEAP and other test data. As an evaluator I conducted several meetings with school staffs to review their test data, as an administrator I go out to a school to assist with their MEAP testing, as a cabinet member I an audience to the presentation of MEAP and other test results, as Director of State and Federal Programs I direct resources to meet student needs and improve student achievement, as a lobbyist I talk with legislators about legislative issues including testing issues, as a parent I review the MEAP results for my son and for his school. These many perspectives led to the design and conduct of this study. These perspectives lead me to conclude that the answers to the questions raised in this study do not lie with administrators, or policy makers or test designers but rather with teachers. The answers to these questions will be addressed to policy makers, test designers, testing coordinators, in the hope that what is learned here will provide perspective, insight and empirical data useful to those who effect life in classrooms so greatly. r an 2 s 0 en The data for this study come from two focus groups and five individual teacher interviews and one interview with a principal. Chapter Two presents a review of the literature related to the topics of this study. It is not intended to be exhaustive but to give the readers a grounding in the areas that relate most directly teachers' use of state assessment data. Studies that describe test use and specifically state assessment test use and MEAP are cited. In Chapter Three, I present the design and methods used and the assumptions and precedents on which they are based. I recount the decisions and rationale that led to the particular design employed in this study as well as choices considered and rejected. Chapter Four tells what I have found. First I give contextual background needed to understand this study. Here you will find details about the Lansing School District and MEAP. Themes are described that are a synthesis of the contextual and teacher data. Next I present and analyze the data. I use teachers' words and stories to tell how they use and interpret assessment test data. In addition to the teachers you will hear my voice often in this chapter as I analyze the teachers' talk and interpret their words( Chapter Five summarizes the lessons that I have learned. Here I discuss and make connections among the themes, the literature, contextual information, and the talk of the teachers from Chapters Two and Four. Recommendations based on this study are found here as well. Recommendations are addressed to policy makers, test designers, curriculum specialists, teacher educators, central office administrators and principals. This chapter gives a closer look at the methodology employed in this study. Here I will describe what I learned about method; what went well, and what did not. Finally, I describe implications for further research for both the subject of this study and its methods as well. I made the deliberate decision to write this study in the first person and to include myself prominently in the text. This is done to make as visible as possible the lens through which this study was conducted. All names of people, schools and communities other than Lansing are pseudonyms. CHAPTER TWO REVIEW OF THE LITERATURE This chapter summarizes research on the following topics: teacher use and perceptions of test data, influences on test use, studies of MEAP, assessment testing and public policy, MEAP and the curriculum and external standards for teachers. This chapter sets the context for.this study and connects the study to what others have learned about teachers and tests. WW Rudman, Kelly, Wanous, Mehrens, Clark and Porter (1980) provide a perspective of assessment that reviews the years 1922-1980. Although they review assessment tests in general, much of what they summarize is useful when considering how teachers make sense of MEAP. Rudman et al. state that ”there is a growing sense of need today to link assessment with teaching and to make instructional decisions more rational and less intuitive” (p. 2). They go on to say that . it seems that those who exhibit considerable skepticism of assessment techniques in general, and standardized tests in particular, tend to view testing and teaching as two separate events which have little or no relationship one to the other (Bhaerman, 1977; Wise, 1978; Shulman, 1978) (p. 2). 10 For teachers the practice of standardized testing is not connected with their daily instruction and is therefore an intrusion. Rudman et al. conclude that for whatever reason . there appears to be a widespread belief that teachers are skeptical about the values of external assessment information yielded by tests not devised by the teachers themselves (Perrone, 1977) (p. 3). Is this skepticism borne out in teacher descriptions of their use of MEAP results? One plausible explanation of why teachers are skeptical about use of tests lies in the extent of their knowledge about the test and the test results. A series of studies by Stetz (1977) and Stetz and Beck (1978, 1979) surveyed teachers and administrators and found that teachers viewed themselves as ”knowledgeable about testing, interested in the results yielded by tests, somewhat anxious about the results, and generally supportive of testing” (p. 4). While teachers feel comfortable with test results and their ability to understand them, Yeh (1978) reports that only 502 of the teachers sampled were able to correctly interpret two standard scores commonly used in reporting standardized achievement test results (percentile ranks and grade equivalents). Olejnik (1979) reported similar results and concluded that "in spite of self-reported confidence it appears that non-measurement specialists needed additional assistance in the interpretation of standard scores” (Rudman et a1., 1980, p. 4). Rudman and associates go on to describe Stetz's study asking Stanfggg A;h1gggmgn§_1e§t users the types of scores they found most useful for assessment purposes . 11 One would like to assume from this that those who showed such a strong preference for these two standard scores understood what they signified, but Olejnik's study does give one some pause (p. 15). Perhaps the Rudman et al. and Olejnik judgments are a bit harsh. If we look at use of test data from the teacher's angle we may find that their sense making and thinking about test results are adequate and perhaps even superior for the judgments and contexts with which they engage. However we know little about the judgments, use and sense making about specific test results from teachers themselves. Much of the early literature about teachers and tests is based on surveys of non-teachers. Some of the findings of these studies contradict teachers' self reports. Stetz and Beck (1978, 1979 and 1980) demonstrate that the perceptions of measurement specialists (National Council on Measurement in Education members) of teachers' perception and practice of use of test results do not reflect the reality of teacher perception or practice. Although, here again, survey results may mask many nuances of meaning that could be drawn from the informants, the results do indicate that decision makers (in this case testing analysts and designers) may not have an accurate perception of the practice of those on whom they rely to administer, interpret and act on test results. 322W Stetz and Beck's (1978) survey of 3300 teachers shows that fewer than one fourth of the respondents felt that ”too much" standardized testing takes place. Further, half of the teachers surveyed replied 12 that "some" use was made (with nearly 101‘ responding ”considerable”) by teachers of standardized achievement test results. This finding does not indicate how test results were used; clearly an important question left unanswered. Three-quarters (772) of teachers surveyed by Stetz and Beck responded positively to using test data for measuring "student growth,“ 65% for planning instruction, 261 for evaluating teaching methods, and 19% for evaluating teacher performance. Although teachers indicated that these are appropriate uses for test results, we have no idea whether they or their colleagues use tests for these purposes. Testing assumptions that teachers were disinterested in, pressured by or not knowledgeable about tests, Stetz and Beck compared teachers' and measurement specialists' perceptions on these factors and found that teachers reported they are more ”interested in, calm, comfortable, and knowledgeable about standardized tests" than the measurement specialists thought they would be. Further, 601 of the teachers reported desire for further measurement training, but only 352 of the teachers wanted to increase use of criterion-referenced tests. This information leads one to conclude that when asked general survey questions regarding “Standardized tests" teachers rate themselves quite comfortable, Confident, competent and satisfied with their use. Perhaps equally important for this study, teachers report that they tThink that it is desirable to make use of standardized tests for 1Instructional purposes. These findings leave many unanswered questions about how teachers use specific test results in their practice of teaching and planning . 13 e u c s e t U Yeh and colleagues at the Center for the Study of Evaluation (Yeh et al. , 1978) conducted a nationwide survey drawn from a sample of large school districts (over 50,000 students) and followed up with interviews of teachers and administrators in five California school districts. Among other findings Yeh concludes that teachers felt that parental concerns about test results influenced the direction of their school's testing programs. An important finding by Yeh is that student test performance influenced teachers' attitudes towards testing practices. Staffs of schools whose children did poorly on tests were least likely to use test results. Conversely, in schools with high test scores teachers tended to view tests as useful. Understanding this phenomenon is important to understanding the extent to which tests can be used to drive change in schools. Yeh goes on to report that teachers with more knowledge about testing were more likely to claim usefulness for all types of tests. Yeh gives clues about the value teachers place on testing. Teachers considered the time spent in required testing compared with the benefits received from such testing as falling between ”about right” and "Slightly too much." Teachers' experience was positively correlated w1th this response. Teachers with more than 16 years experience were more likely to respond that too much time was spent in relation to the benefits received than were these teachers who had fewer than 16 years ellEpsrience . 14 Teachers with more teaching experience were more likely to report 'using test results from norm-referenced tests than were less-experienced teachers (Yeh, 1978). Finally, when asked about factors considered most important when choosing commercially prepared tests teachers valued clear format, similarity to class material and accurate prediction of achievement. oki a d s T the Yeh's findings help us to understand the conclusions of Stetz and Beck (1978). Student test performance, teacher knowledge of testing and teaching experience each seem to shape teacher attitude toward and use of test results. These findings could lead to a powerful explanation of patterns of continued poor or high test results in schools. If schools with poor results and less experienced and less test-wise teachers systematically make less use of tests in curriculum and instructional planning it follows that their instruction could move further away from the test, resulting in still poorer test performance. Conversely, those schools with strong test performance and more experienced teachers who are more knowledgeable about testing could move toward even further improved scores. Is this evidence that testing is driving curriculum? Perhaps that equation is too simple an explanation for what we seem to see evidenced in these findings. e e nd t n rd e s i The work of Stetz and Beck (1978, 1978) and Yeh (1978) along with studies by O'Regan, Airasian and Madaus (1979), Taylor (1970), and the 15 Ingham Intermediate School District (1986) study of MEAP, provide some illuminating insights as to teacher thought related to the value of testing. But can we get still closer to understanding how teachers make sense of test results once received? How do they think about them and decide what to do about curriculum, instruction, materials or the tests themselves? WWW Rudman et a1. (1980) find that there is an insufficient amount of research available regarding “how teachers use test results in their classrooms” (p. 20) and a ”lack of accurate and abundant descriptions of test practice" (p. 21). One step toward constructing this needed body of information is to listen to teacher talk about test results. This study steps back from some of the assumptions and prescriptions to be informed by teachers themselves as to how they make sense of and respond to one set of test results--the Michigan Educational Assessment Program (MEAP). MEAP is viewed by policy makers in Michigan as an important component in the State's educational infrastructure. It seems reasonable then to use MEAP results as the vehicle to understand teacher thinking about testing. Of interest to this study Rudman et a1.(l980) draw two conclusions (among several others) regarding assessment and teaching. One conclusion addresses teachers' sensitivity to the information they use in instruction. The Rudman et a1. review found conflicting evidence about teacher sensitivity to the reliability of the information they receive and their ability to adjust their judgments of students based on 16 test results. Work by Shavelson, Cadwell and Izu (1977) shows that teachers differentiate among the data they receive and use them differentially depending upon the instructional issue they confront. Conversely, other researchers (Pedulla, Airasian, Madaus, & Kellaghan, 1977; Morine-Dershimer, 1979; Sorotzkin, Flemings, & Anttonen, 1974; Beggs, Mayer & Lewis, 1972) report that teachers' perceptions of students' behavior are stable and not much influenced by data when the new information seems to contradict what they have observed. Taken together these seemingly contradictory findings lead to questions about how teachers use and weigh test results differentially based on decisions to be made and in conjunction with other information. They may imply a complex teacher thinking schemata rather than a simple decision tree. A second conclusion by Rudman et a1. synthesized from review of numerous studies (Dusek & O'Connell, 1973; Shavelson, Cadwell & Izu, 1977; Airasian, 1980; Brophy & Good, 1970) is that teachers hold accurate perceptions of what their students' academic achievement test scores and aptitude scores will be (Rudman et al., 1980). Taken together the findings of this section lead one to question our understanding of how teachers make sense of test results. These findings indicate a complex model of test data use influenced by teacher experience, knowledge of testing, student performance and parental concerns. Teachers use test results differently depending on the instructional issues at hand and claim to value their use for measuring student growth and for planning instruction. Teachers may hold accurate perceptions of their students' test performance derived from other 17 sources. Finally, policy makers seem to hold a largely inaccurate view of teacher perceptions of tests. We now move from the general field of assessment data to specific consideration of the MEAP. W In the mid-1970's to early 1980's several studies were conducted in Michigan to review and monitor the use of MEAP. Among the earliest of MEAP studies is a report by the National Education Association and the Michigan Education Association and conducted by House, Rivers and Stufflebeam (1974). This study investigated the use of MEAP by classroom teachers and the length of time between test administration and test results availability (among other topics). Other early studies of MEAP investigated attitudes toward MEAP. Over the years the Michigan Department of Education (MDE) has worked to shorten the time between testing and return of results, improve the reporting format, and provide support services to ease in test interpretation. A study by Aquino (1978) investigated teacher attitudes toward MEAP and the uses teachers made of the results. Aquino surveyed 4th-grade classroom teachers and seventh-grade English and math teachers using cluster sampling techniques. Aquino concluded that there was a problem with the timing of the receipt of the results by teachers. In 1978 for districts surveyed it took between 8 and 16 weeks to return results; by 1990 it took approximately 4 weeks. Second, he found that teachers did not receive all the reports intended for teacher use. Although the term comfort was not well defined, Aquino found that teachers felt less "comfortable” about the 18 reading test results than math. However, since at the seventh grade level separate groups responded to the comfort question, the results could be a reflection of the teachers' background and training or comfort with test results in general. Approximately 80% of each of the three population groups surveyed found the information received was worthwhile. And all three groups "placed more value on 'information received' than on the value of the results for 'making instructional decisions'” (p. 81). This finding leads one to ask What use is the test valued for if not for making instructional decisions? Large percentages of the teachers surveyed had studied their assessment results, and more fourth grade teachers felt "well prepared" or ”very well prepared" than did seventh grade English teachers in making use of the assessment test results (p. 83). Aquino found that two-thirds of the fourth grade teachers and fewer than half of the seventh grade math and English teachers attempted to use the results to provide students with special instruction on objectives not mastered on the test. These findings call for greater understanding of specifically how teachers understand and use MEAP results. Aquino's study does not provide information that can lead to a clear explanation of his findings. Any number of factors, perceptions, or attitudes could lead to the levels of comfort and use of MEAP Aquino reports. Aquino found that few teachers were Igguirgg to change their instruction based on the 1974 MEAP results and that few teachers were "criticized” as a result of low student scores on the MEAP. This finding is interesting when compared with later studies that describe the large effects of publication of MEAP results in newspapers and l9 comparisons between schools. The principal (or assistant principal) was identified to be the person who most encouraged teachers to use MEAP results. Consistent with the Rudman et a1. findings about the accuracy of teacher perception of test scores, Aquino concluded that the greatest impact of MEAP was to "confirm previous judgments teachers have had about individual student skills” (p. 84). Finally, he concluded that there was no overwhelming opposition to MEAP and that indeed teachers favored gradual expansion to other subject areas and grade levels and making the test optional. If optional, half of the fourth grade and two-thirds of the seventh grade teachers reported that they would want their classes to take the test. Aquino's data are now clearly old and MEAP tests, report formats, return of results to teachers and support for use of results have all improved. Further, self report survey methods may bias the data toward anticipated ”acceptable" responses. The study encountered some problems with data collection from urban public school teachers. 'These facts notwithstanding, this study does shed some light on the fact that teachers perceive that they use MEAP data (albeit to confirm other perceptions of students) and that they feel comfortable in using the data. These returns seem modest when compared with invested time, expense and high expectations for MEAP use held by state policy makers. W In 1986 the Assessment Resource Network (funded by the Michigan Department of Education through the Ingham Intermediate School District Department of Planning and Evaluation) conducted a survey of 416 v 01 5‘ pr 0P De 1’9. sir res Var enc be of dist 20 Michigan school districts (representing almost 791 of the K-12 districts in Michigan). The survey was to determine testing-related needs of Michigan school districts. Like some of the other studies reviewed by Rudman et a1. and cited here, the distribution of this survey's respondents was primarily to school district administrators. The survey results shed some light on testing practices in Michigan and confirms some of the notions described earlier. When asked to identify five staff positions most responsible for routine administration of testing, one is not surprised to find that teachers were ranked first, with principals and assistant principals ranked second. The survey contained open ended questions asking respondents to identify positive and negative effects of MEAP they observed in their school district. The responses to these questions were compiled but not analyzed. I grouped similar responses and developed frequency and rank order information using the Assessment Resource Network (ARN) listing of responses. (See Table 2.1.) The most frequently cited positive effect was the use of MEAP for curriculum review/revision/focus/and alignment (frequency of response - 56 of 180 responses made). Educators and policy makers who want the test to drive curriculum may regard this finding as somewhat encouraging. Yet given Yeh's findings that test data are used and valued differently depending on student performance on the test one must be cautious. The use of MEAP for review, revision, focus and alignment of curriculum may reflect the responses of the one third of Michigan districts with highest test results. The second most frequent response was the positive effect of MEAP on public relations, pride and community awareness (frequency of response - 32). The third most 21 TABLE 2.1 Positive Effects of MEAP Ranked by Frequency of Response: Five Most Frequent Responses RANK RESPONSE FREQUENCY 1 Curriculum review/revision/ 56 focus/additions/alignment 2 Public relations/pride/ 32 community awareness 3 Focus on specific skill 19 attainment/outcomes/mastery teaching 4 Awareness or objectives/needs 18 5 Student achievement has improved 14 Total Number of responses made - 180 SOURCE: Ingham Intermediate School District, 1986. frequent response (frequency of response - 19) is the positive effect of MEAP on focus on specific skill attainment, student outcomes and mastery teaching. Although it is difficult to draw conclusions based on frequency of response to open ended questions, the fact that the number of respondents who cited specific skill benefits is interestingly low considering that an issue that led to MEAP's creation was insuring individual student mastery of basic skills. However, when specific skill benefit is viewed with the fourth ranked response, the positive effect of teacher awareness of objectives and instructional needs (frequency of response - 18), one can conclude that in a number of districts teachers are more aware of student and classroom instructional 22 needs because of exposure to this set of state developed objectives (i.e., MEAP test items). Twenty-two years after its first administration we find MEAP still with us and strongly embedded in the framework-of public policy in Michigan. Testing format, items, and objectives have changed. Grade levels tested and subject areas have been expanded. MEAP now includes tests in writing, science, social studies and health, as well as reading and mathematics. Printed material, video tapes and workshops are available to assist in interpretation of MEAP data. Test results are routinely published in newspapers with comparisons made between schools, school districts and with previous years. Realtors even refer to MEAP results of communities and schools when showing homes to prospective buyers (Ornstein, 1990). MEAP results were part of annual education reports required of every Michigan school and district beginning in the spring of 1991. The context of MEAP has changed. This study investigates how teachers construct meaning from, and think about use of MEAP data. ~Earlier studies have established that teachers report that they ”use” MEAP for some purposes. But none have dug deeper into the question to determine how. ub c In 1986 Schwille, Porter, Alford, Floden, Freeman, Irwin and Schmidt reviewed state policy and control of curriculum decisions in seven states, including Michigan. They found that neither the legislation creating MEAP nor the State Department of Education dictated that MEAP results should be used in any specific way. Although never 23 formally stated in legislation or State Board of Education policy there is clear evidence that MEAP is viewed by legislators, State Board members and State Department of Education officials as serving a pivotal role in shaping curriculum in Michigan. The fact that the legislature requires annual reports by schools and their districts to the public on several school quality factors including MEAP results (PA 25 of 1990) and that continued low scores on MEAP trigger the Michigan Department of Education to require a plan for MEAP score improvement evidence a strong role for MEAP in Michigan's schools. The 1991-92 school aid act contains a requirement that students must score 752 on the 10th grade MEAP to receive a state endorsed high school diploma (State School Aid Act, 1991). Along with this evidence the ARN (1986) study shows that MEAP has grown to be viewed as having a specific role as a tool in analysis of curriculum and in influencing public relations in Michigan school districts. Given the data compiled here and the findings of this study policy makers and educators can address the following questions; Can MEAP reasonably drive curriculum? Should it? Does it now? W There are, arguably, a number of official or formal curricula for the schools in this study. Formal curricula are stated through the locally developed curriculum statements, through the content of the textbooks used, through the curricula that underlie the standardized tests employed by the Schools (specifically MEAP and the Stanford Achievement Test). Added to this list is the newly adopted State Core Curriculum. The late 1980's and early 1990’s have witnessed the 24 beginnings of a subtle shift away from Michigan's long commitment to local (community level) control of school decision making to increasing legislative and executive level initiative toward creating change in schools through state action. We see in this study the tension between a view of the teacher as arbiter of classroom instruction and teacher perception that a state wide curriculum is in the offing. The enacted curriculum consists of the teachers' responses to these formal curricula and other factors as well. What happens when the teacher stands before her students with her door shut is this enacted curriculum. In this study teachers talk about the role of MEAP testing on the enacted curriculum. W Stevenson and Baker (1991) describe the American educational system as characterized by local control. , The administrative mandates as to what teachers should teach are varied and weaker (than the centralized French educational system) and there is a greater diversity in the textbooks that are available for teachers to use and in the types of training available for teachers (p. 2). Teachers in the decentralized American educational system have considerable autonomy and discretion in how they handle classroom instruction and learning. They often modify the official or standard curriculum to meet their needs or those of their students. Even within a single school teachers may differ greatly in the amount of material they cover, the particular topics they cover, the amount of time they spend on instruction, and their use of curricular materials for the same subject (Cohen et al., 1990). 25 Stevenson and Baker (1991) note the paradox of education reform in the 1990's. The development of state accountability systems and the publication of annual school report cards that publicize students scores on standardized tests may have increased pressure on teachers to teach the 'test curriculum.’ While such pressure may reduce instructional differences, they are coupled with a movement to allow schools to develop the measures on which they will be held accountable and to restructure schools to allow teachers to exercise greater professional judgment about their work (p. 2). a 0 ac When viewed together the MEAP reading, mathematics and science tests and the Essential Goals and Objectives for Reading, Minimal Performance Objectives for Mathematics and Essential Performance Objectives for Science Education articulate the State of Michigan standards for education in these subjects. These standards are set by the State Board of Education of Michigan, an elected governmental body external to and independent from local elected boards of education. By setting these standards it is expected that educational achievement in Michigan will improve. Porter (1989) considers the pros and cons of setting external standards for teachers. Externally set standards for student achievement not only specify what content is held to be most worthwhile, they specify criteria for judging adequate teaching and learning. Standard setting, therefore, has potential for connecting each of the two main pieces of the goal, good teaching and worthwhile content (p. 345). Porter also describes another point of view. Telling teachers what to do through state and district policies is seen as antithetical to empowering teachers and 26 strengthening the teaching profession. External standard setting represents one piece of telling teachers what to do (p. 345). This study looks at the tension teachers express between these two ways of thinking about the role of control in teaching. FUhrman and Elmore (1990) state the issue clearly. Statewide testing and state mandates for increased local testing are also affecting school curricula. Teachers, principals, and superintendents cite testing as a key influence on what is taught (Koretz, 1988; OERI Accountability Group Study, 1988; Fuhrman and Elmore, 1990, p. 85). A logical next question to consider is the nature of the influence of these external standards on schools and teaching. Many of the state policy objectives were expressed as mandates or rules, as opposed to efforts to build local capacity, and thus they reinforce the image of an increasingly obtrusive state presence (Fuhrman and Elmore, 1990, p. 85). In classrooms this pressure can be seen in a number of ways. Teachers mention a variety of ways in which testing has influenced what they cover and how they cover it: teaching test taking skills and awareness, reviewing for the test, changing sequencing of content, incorporating concepts of the test and emphasizing weak areas identified by the test (Fuhrman and Elmore, 1990 p. 85). In two recent articles, Smith (1991a, 1991b) drew on teachers' descriptions to understand the effects of and preparation for external tests. Among the effects of testing on teachers, Smith (1991a) found that pressure to improve test scores led to a narrowing of the uoucH use H macho .900» v 30H>uoucn o seaeueueu use H Guano couch N macho .500» H sowvuoucH m one a soseuoueu stum 6H 0H0“ h.¢ + 0.0H + ~o.oNM undue-Ha soyu oQEJuj> o.hH ¢.hH o.Hn m.mv o on an mu OH nH N xcdm oooH m.o + m.ou + 0.0H + -.Hn. uoHuucHa sou» ousoHuo> AnHoozuu nn «0 n uyzgunnxnuluznhunax |_n=annfi=:u A¢d=a Afixxnum e.o~ en m.m~ Hm e.Hn «H n.e~ e” e.Hm .u on v o mmmm mama xcom Hoonumv oriuuo gas-H coca: oeHe vocivou vHIHuHOOO . deacon v.0NI «.mHl «.mN vchn oosoHuo> OIHOOE mm on «N mH N manages Haste 35 assumptions about teachers and testing. I hypothesized that the results of such a study would be broad but not very deep. I also considered a think aloud study to be conducted as teachers reviewed their MEAP data but it seemed that this was not a sufficiently natural or representative task. A think aloud activity contrived just for the study would not give information useful to answer the questions I had. Due to time constraints I was unable to conduct a year long series of Interviews. In hindsight such a design would probably give interesting information. Classroom observations following Interviews in the fall as teachers review for MEAP and even throughout the year (as they receive the results, and as they prepare for SAT, or CMS) could allow one to follow up on themes found in this study. Again, time constraints made such a study impractical. £15; VThis study takes place in the Lansing School District. Lansing is the capital city of Michigan. The largest employers in the area are Buick-Oldsmobile-Cadillac and other automotive related industries, state government, Michigan State University and the school district. The diversity of employment has buffered the community from the worst effects of Michigan's economic downturns. But recent layoffs of state employees and continued recession strain the local economy. Visitors from the east coast marvel about the green lawns, mature trees, and flowers that adorn even the poorest neighborhoods. There is a marked absence of tenement style housing and much of the housing for the lowest income people is single family houses subdivided into multiple 36 substandard units. Like much of the mid-west housing is still affordable for many, although even here those who are poor or just starting careers find it difficult to buy a home. Like most urban centers the school district's population is poorer and browner than that of the surrounding communities. Yet a substantial middle and working class population make up the majority of the school district's families. The Lansing School District, a medium large urban school system, has a student population of approximately 25,000. The school district student population is racially and ethnically heterogeneous, with racially balanced student enrollments in all schools. Lansing School District administrative offices are located in downtown Lansing, in fact within a 3 block radius of both the Capitol building and State Department of Education, an easy walk for state or federal program auditors and news reporters looking for a local perspective on an education story. With access to the local newspaper, legislators read as much about Lansing schools as they do about their home districts' schools. Many legislators' staff and staff of the State Department of Education live in the Lansing School District community. They are our friends, relatives, neighbors, taxpayers and the parents of Lansing's students. Lansing schools come under close scrutiny. Lansing schools are neat and orderly places to learn. It is noteworthy that there are no graffiti stained walls, broken windows or glass covered playgrounds. As I visited schools, even at the end of active school days, the playgrounds, halls and classrooms were free from trash and debris. 37 The Lansing School District has a long history of site based management called responsible building autonomy. The 20 year experience has taught the educators of the community much about site based management, its possibilities and limits. Lansing teachers are euphemistically called veteran teachers. Most have taught more than 20 years and have continued their educations to and beyond the masters degree level. Teachers in the school district are experienced participants in research studies. They have taught through the days of objective referenced tests, elaborate testing and record keeping systems designed to diagnose learning difficulties and monitor instruction. A slowing of the 15-year decline in student enrollment has allowed the district to hire a small number of new teachers during the past three years. One of these new teachers took part in this study. Many innovations have found their way to Lansing classrooms recently. In addition to effective schools and school improvement programs, many teachers have been trained in Instructional Theory Into Practice and other effective instruction techniques. Assertive discipline and clinical supervision are popular, along with cooperative learning. A new reading program started in the schools two years ago and is being followed in 1991-92 by a new math program and new approaches to science and social studies. W The participants in this study were third and fourth grade teachers. The teachers were all practicing teachers in the Lansing 38 School District. The range of teaching experience for the participants_ was 2-36 years and most were very experienced with MEAP testing. (See Appendix E, Profiles, for a summary of teacher demographic information.) All 150 third and fourth grade teachers in the school district were invited to participate in the study. Teachers were asked to participate in one of the two study activities (Focus Group or Interview). The participants were balanced so that they were roughly representative of high, average and low performing MEAP schools. All 33 elementary schools in the district were ranked using MEAP reading results. Alternate schools on the list were then invited to participate in either the Interview or the Focus Group activity. This process was used to increase the likelihood that actual participants represent high, average and low scoring schools. Other sampling techniques were considered. However, since participation was voluntary and I anticipated a small number of participants I decided that this procedure would maximize the number of participants while still maintaining a sample fairly representative of school performance on MEAP. In developing the rank ordered list of MEAP performance, the results in MEAP reading, math, and science were reviewed. Math and science results did not show sufficient variability in scores between schools to make ranking meaningful. The ranked listing of reading results was developed by computing individual school variance from the district average in category 3 (mastery of both the informational and fiction sections of the reading test) for the 1990 and 1989 test administration. (See Table 3.2.) The variances were combined and ranked. School performance over the two years was fairly stable with 39 o 'VQ'N'OOOO’U’WO HHHN I I N ' O O O H H H O... I++I ,0 0 O O o N H N 00 H I|+ FIDO ooe ONH NH” + '9' 0 0 Q N O I I 0 O ,4 N H +++++++++++++++ 0 O O N 000000 ”QOHHQHQOHQW.H e o 0 ' H 0.0m ~m.0flq coon nouns-Ha IOHh oosoHuo> 0'!""OI‘OONOON‘MFFN‘FMIDOMFOMOI‘IDQM O D H § 5 E a g g 5 IUOOIEOIHbKo-I ‘30fluoflflufll-OD>B OHOHquOD ‘ n #18340 0-33 I 02H; a .Nn .Hn .on .ON .hN .0“ .0" .NN .0H .0H .mN .NH .ON .HN .OH .HH .nN .OH .OH .0 .ON .0H 0 N.¢H I o.bH N.VH I o.hH O.NH I «.0H 0.0 I v.0u n.0H I 6.0" 5.0 I m.mN 0.h I 0.nN n.m I a.mu «.0 I o.mN h.OH I n.0N N.nH I 0.0H 0.0 + b.Hn 0.NH I 0.0H h.h + $.0n 0.0 I 0.mN H.h I H.VN 0.0 I h.vn 0.0 + 0.5m o.n I m.hN H." + n.nn n.v I $.0N N.H I o.on 0.0 + h.hn H.o + n.Hn 0.5 + b.0n o.m + O.Hv 0.0 + 0.0M H.NH + n.nv H.HH + n.nv 0.0N + h.Hm m.mH + h.om 0.0H + 0.0m n.NN + 05.nm ~N.Hnu uaoH UOHHuIHD Shh gd Nd> 5‘ :5 g 3’ g: s 5 3 flNOOHhOIHbKv-l IZOQOKHQBD>338N>UN= UHOHquOO .nn .Nn .GN .VN .hN .HN .0N .mH .Nfl .0" .HM .VH .on .0 .ON .mN .nn .NH .hH .nH .0H .0H .HH .0H .0H .0 .0 .0 .N .n .0 .H h.vnl .nn 0.0nI .Nn 0.6NI .Hn m.HNI .On 0.0NI .QN «.mHI .ON n.0HI .hN 0.0HI .0N n.0HI .mN b.0l .vN 5.0I .nN 0.5I .NN h.hl .HN h.0I .ON 0.0I .fiH v.ml .oH 0.HI .hH 0.0 .0H m.“ .0H n.n .0H 0.0 .nH 0.6 .NH h.h .HH o.o .OH 0.o .m m.HH .0 0.0H .h 0.0H .0 n.0n .m «.mu .0 n.nu .n v.5n .N H.nv .H Mcdxcdfl H1309 40 the exception of one of the schools included in this study which fell from being at the district average to well below. Third and fourth grade teachers were selected as the focus of this study because they have the most experience in~MEAP preparation, administration, and analysis. Both third and fourth grade teachers are responsible for teaching all subjects tested by MEAP, unlike in middle and high school where teaching duties are typically departmentalized. MEAP reading and math tests are administered in October of the fourth grade and thus reflect instruction which has occurred during previous years. MEAP science testing (a newer test) is administered in October of the 5th grade and thus reflects science instruction in grades 1 through 4. The third grade teacher is the last to have responsibility for teaching the content covered by MEAP reading and math. It is the fourth grade teacher who must administer MEAP, receive the results and carry out the instruction of these children for the remainder of the school year. It is the fourth grade teacher who is the one with the opportunity and responsibility to remedy based on MEAP results. The 4th grade teacher is also the last teacher to be responsible for science instruction before the MEAP science test is given in the fall of 5th grade. These third and fourth grade teachers, it was hypothesized, have the potential to tell much about how they use and make sense of MEAP results. WW Teachers who indicated that they were willing to participate in the Focus Group stage of this study participated in one of two group 41 discussions. One session included three third and fourth grade teachers representing two different schools. The other session included the four third- and fourth-grade teachers from a single school. At the beginning of the session I outlined the study and the purpose of the Focus Group. Teachers were reminded that their participation was voluntary and confidentiality assurances were reviewed. Focus Group procedures were explained to the participants. A discussion facilitator, Mary, was used to frame the discussion and to free me so that I could observe and take notes on the proceedings. I prepared a script of questions and the questions and the Focus Group procedure were reviewed with Mary. These questions were raised by Mary to serve as prompts to discussion. During the Focus Groups copies of actual MEAP results (with school and student identifying information removed) were available. Copies of actual MEAP reading and math tests and the MEAP testing handbook published by the State Department of Education were also available for review (see Appendix F). The Focus Groups were video and audio taped. My notes served as the primary documentation of the proceedings and were augmented by the video and audio tapes. These data records were used to code and analyze the discussions. The purpose of this activity was to gain insight from teachers in a non-threatening format where teachers are viewed to be the experts and knowledgeable informants. This format allowed for discussion by teachers and produced a richer description than surveys or one-on-one interviews for this stage of the inquiry. 42 Teacher Ingezxigws Seven teachers responded that they would participate in the Interview activity. Two later dropped out due to scheduling problems. Participants represented a balance of high, average and low scoring MEAP schools. The participating teachers were contacted and an Interview date, time and location were arranged. The Interviews took between 60 and 90 minutes each and were audio taped. I also took notes during the Interviews. I opened the Interview by reminding the teacher that their participation was voluntary and reviewing and securing signatures on the confidentiality assurance forms. The purpose of the study was explained and shared with the teacher in writing. Questions for this Interview stage were drawn largely from analysis of data from the Focus Group phase of this study, as well as from the literature. The Interviews were designed to be an informal means of verifying and probing issues raised during the Focus Groups. ‘The teachers who participated in the Focus Groups were not invited to participate in the Interviews. It was thought that teachers would feel freer to respond to the Interview questions if they did not feel that they were "on record" as advocating a position or as holding a particular opinion expressed during the Focus Groups. During the Interview teachers were shown MEAP results and samples of the various reporting formats, as well as the actual reading and math test and the MEAP Handbook. 43 c a e w Following the teacher interviews one principal was selected to be interviewed. This principal was selected because teacher descriptions during the Focus Group stage led me to hypothesize that the principal held strong views of how MEAP should be used and interpreted and that these views were shared with staff. Further, the two staff members who participated from the this principal's school differed in their feelings about the principal's expectations. The principal was asked to participate, he consented and I conducted a 45-55 minute Interview with the principal. This stage helped to verify that the vision held by these teachers and their view of the principal's motivation matched the principal's intent. It also helped add to the interpretive context for the descriptions given by all the teacher participants. Wale The transcripts from this study offered rich descriptions of teacher use and interpretations of MEAP results. Because Interview questions were open ended and informal, teachers made connections not anticipated prior to the study. The task of the analysis of this data was to find these connections, identify patterns and anomalies and to look for stories, if they exist, to be used in understanding and describing the data. The treatment of this data is intended to provide grounding for subsequent studies. From the transcripts the data analysis was developed. This approach proved to be the best way to portray the data and maintain the integrity of teacher thought and feelings. In addition to the data 44 collected through the Interview and Focus Group process descriptive data were collected about the participants and their schools. The current MEAP results for each of the schools represented in the study were also compiled. (See Table 3.3.) These data are used to help set the context for the study. 45 one; 8385:. 0' Ii .ioossgaggisogsoflgogaueai uaflgaggagguHeESHeauou3§ 8888 .Suuoalguoflkgtlsozaefluaifleofl 00 goo . H bones-u I Hive-Mound Calhoun-0 guac- Hg figscngausiigaig .0 80819-35380!!! 3 goo so no no: .8383- Hague-85 so in! bonus-H»: .00 0.30300 0903 HEHquounH I. use .am so gavel— »uoauOueHuoo .40 Eula-o no is! goon-Hae- 3 08.3.0 .H 8001.!— 0:353 50 50 0.00 0.00 0.H0 0.50 5.H0 0.00 0.0 H.0 0.00 0.00 0.0 0.50 0‘ 00 00 0.00 0.0H 0.50 0.00 0.00 0.00 5.0 0.H 0.00 0.00 H.HH 0.5H N 05 05 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0.H 0.50 0.00 0.HH 0.0H h H0 H0 0.00 0.00 0.50 0.H0 0.00 0.00 0.0 0.0 0.00 0.00 0.0H 0.00 I 50 00 0.H0 5.00 0.00 H.00 0.00 0.50 0.0 0.0 0.00 0.H0 0.5H 5.00 01'000 00 H0 0.00 0.50 0.00 0.00 0.00 0.00 0.0 0.0 0.5H 0.00 0.5H 0.00 IIIIIIA 00 00 0.00 0.50 0.00 0.50 0.00 H.00 0.0 0.0 0.00 0.00 0.00 0.00 8 00 00 5.00 0.00 0.00 0.H0 0.00 0.00 0.0 0.0 0.H0 0.00 0.0H 0.00 > 00 00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0.H 0.00 0.00 0.00 0.00 0 00 00 0.00 0.00 0.00 0.H0 0.00 0.00 0.0 0.0 0.00 0.00 0.00 0.00 B 00 00 0.00 0.00 0.H0 0.00 0.00 0.00 0.0 0.0 0.00 0.00 0.H0 0.0H 0 00 H0 0.00 0.00 0.00 0.00 0.00 5.H0 0.0 0.0 0.00 5.H0 0.0H 5.H0 .0900 0800 00 00 0.H0 0.H0 0.00 0.00 5.00 0.00 0.0 H.0 5.00 0.50 0.H0 0.0H 0 00 00 0.0H 0.00 0.00 0.05 0.00 0.50 H.5 0.0 5.00 0.00 0.0H 0.00 0 00 00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 H.0 0.00 0.00 5.00 0.00 0 00 00 0.00 0.00 0.H5 5.00 0.50 0.00 0.H 0.0 0.00 0.00 0.00 H.00 O 05 00 0.00 0.H0 0.00 0.00 0.00 0.00 0.0 0.0 0.00 0.00 0.00 5.00 I 50 50 0.00 5.00 0.00 5.00 0.00 0.00 5.0 5.0 0.0H 0.00 0.H0 0.50 I 00 00 5.00 0.00 0.00 0.05 0.00 5.00 0.0 0.0 0.00 5.50 0.00 0.50 IODIQII 00 00 0.H0 0.00 0.00 0.00 0.00 0.00 H.0 0.00 0.0H 0.00 H.00 0.00 a 50 00 0.00 0.00 0.50 0.H0 5.00 0.00 5.0 0.0 5.00 0.00 0.50 0.00 I 00 00 0.50 0.00 0.00 0.05 0.00 0.50 0.H 0.0 0.00 0.00 5.00 0.00 h 50 00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0.H 0.00 0.00 0.00 5.50 0 00 00 5.50 0.00 0.00 0.00 0.00 0.50 0.H 0.0 0.00 0.00 0.00 0.H0 0 00 H0 0.00 0.H0 0.00 5.50 0.00 0.00 0.0 0.0 0.00 0.00 0.00 5.00 0 00 H0 0.00 0.H0 H.50 0.00 H.00 5.0H 0.0 0.0 0.00 0.00 0.00 0.H0 0 00 00 0.H0 0.00 0.50 0.00 0.00 H.50 0.5 0.0 0.00 0.00 0.00 0.00 0 00 00 0.00 0.00 0.00 5.00 0.00 0.00 5.0 0.0H 5.00 0.00 0.00 0.00 0 50 00 0.00 0.00 5.00 0.50 0.50 0.00 0.0 0.0 0.00 0.00 0.00 0.00 O 50 00 H.00 0.00 0.00 0.05 0.H0 0.0H 0.0 0.0 0.00 0.50 0.00 5.H0 .00 I00 00 00 0.00 0.00 5.00 5.05 0.H0 0.0H 0.H 0.0 0.00 0.00 0.00 5.00 0 00 00 0.00 0.00 0.05 0.05 5.00 0.00 0.0 0.0 0.00 0.00 0.00 0.00 00.000I00 00 50 0.00 0.00 0.05 0.00 0.00 0.5H 5.H 0.0 0.00 0.00 0.50 5.00 4 0000 0000 0000 0000 0000 0000 0000 000H 000H 0000 0000 0000 0000 000H 0000400 .300 .0000 0009. a nluulhiu 0N 0.00-H00 00 0000-000 0 nluuifldu. .00 0000.000 0 00808 0 A‘BDF 0 00808 0 0 ON 0” Emma UZHQ4HNO quqHHVHm .H‘Heflzmmmm A30 :02“ 928000002 20506900 2003050 000.0 EM“ 020 E0 46 MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM (MEAP) 4TH GRADE REPORT 1990 IflLlifllfii 4 OF STUDENTS IN CARGO!!! 4 O! m TOTAL STUDENTS (N) TESTED Hot 8 Chanqs [ITIRIRG 428 Stats Goal Iron ananxns 1981 1987 1988 1989 1990 86 - 9O 81 to 90 A 62.3 77 94.79 75 100.09 63 97.09 67 91.59 59 9 9 9 9 9 +29.2 Dssrfisid RA na 75.89 66 88.99 54 88.29 34 95.59 44 9 9 9 9 9 In I IA NA 93.29 59 94.49 71 91.39 69 88.49 69 9 9 9 9 9 IA 31: at. 70.3 64 84.29 76 93.89 64 96.69 58 98.29 57 9 9 9 9 9 +27.9 c 66.7 93 90.59 63 89.29 65 94.29 52 85.19 67 9 9 9 9 9 +18.4 .D 56.3 16 80.89 26 63.2 19 96.79 30 80.09 30 9 9 9 9 +23.7 I 60.7 56 95.29. 42 79.59 44 80.99 47 88.49 43 9 9 9 9 9 +27.7 r 56.7 67 87.39 55 90.49 52 86.99 61 88.19 42 9 9 9 9 9 +31.4 O 75.09 36 84.29 38 93.69 47 96.89 31 80.89 52 9 9 9 9 9 + 5.8 I 66.1 56 95.19 41 92.19 38 93.89 48 88.79 53 9 9 9 9 9 +22.6 I 50.8 59 84.19 63 81.09 58 92.39 52 75.49 57 9 9 9 9 9 +24.6 J 60.4 53 96.69 29 97.79 44 95.09 40 92.99 56 9 9 9 9 9 +32.5 t 67.5 40 100.09 30 81.59 27 92.39 26 91.99 37 9 9 9 9 9 +24.4 L 66.7 36 88.69 35 100.09 40 97.49 39 90.69 32 9 9 9 9 9 +23.9 deuood 62.5 48 95.09 60 88.99 45 88.69 44 88.49 43 9 9 9 9 9 +25.9 I 55.3 47 91.79 24 73.3 45 85.29 27 75.79 37 9 9 9 9 +20.4 I 53.7 82 80.59 77 83.89 80 81.79 93 80.59 77 9 9 9 9 9 +26.8 O 80.39 61 92.79 41 95.79 47 88.99 54 95.29 63 9 9 9 9 9 +14.9 P 64.7 51 100.09 29 100.09 41 94.99 39 95.59 44 9 9 9 9 9 +30.8 9 51.2 43 96.99 32 97.69 42 100.09 36 92.99 56 9 9 9 9 9 +41.7 I 56.7 30 89.79 29 92.39 26 84.49 32 92.19 38 9 9 9 9 9 +35.4 Pins lidqs 63.6 44 94.79 38 82.69 46 90.29 41 67.4 43 9 9 9 9 + 3.8 8 66.7 36 91.99 37 88.99 36 96.09 50 82.99 35 9 9 9 9 9 +16.2 r 61.4 44 85.29 54 82.29 45 90.99 44 84.29 38 9 9 9 9 9 +22.8 0 In sa 81.09 42 83.39 42 80.89 52 75.59 53 9 9 9 9 9 an v 55.2 87 94.19 68 93.89 80 90.69 85 87.99 66 9 9 9 9 9 +32.7 I 52.1 48 87.39 55 92.69 54 90.99 55 72.0 50 9 9 9 9 +19.9 Lsnsuss 49.1 55 87.29 39 81.59 54 68.6 51 76.19 46 9 9 9 9 +27.0 Ottawa ua an 93.39 45 69.8 53 87.99 58 78.79 47 9 9 9 na 1 76.29 21 85.79 35 79.19 43 92.79 41 75.69 41 9 9 9 9 9 - .6 r In an 80.09 60 87.39 63 77.89 72 71.4 70 9 9 9 9 an I 54.9 71 66.0 50 92.39 65 73.6 53 72.2 54 9 9 +17.3 an 38.5 39 76.09 25 75.09 36 80.99 47 63.8 47 9 9 9 9 +25.3 80288: Tbs stats goal for math sxpscts that 754 of ths studsnts will psrtorn in Catsqory 4: i.s., achisvo 75-1005 of ths objsctivos. Starrsd nunbsrs indicats that ths stats goal was sst or surpasssd. "In" indicatss that a school had no fourth grads in 1981, ths district basslins ysar for math rssuits. lhsn looking for trsnds in ths data, kssp in mind that ths sna11sr ths group, ths nor. usight ths psrtormancs of any individua1 carriss. fisssarch and Evaluation Sarvicss Dscsnbsr 6, 1990 ‘47 MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM (MEAP) 5TH GRADE REPORT 1990 £3<::J:ralsnc:25 4 or STUDENTS IN carzcon! 4 or TflngOTAL STUDENTS (617135130 zuwsnxuc STE GRADERB 1969 1990 a 83.1* 59 56.3 64 Deertield 58.2 55 65.9 41 6 , 66.1 12 63.1 65 lie at. 12.4 56 15.6. 45 c 66.3 63 56.9 56 0 36.0 25 55.6 ‘ 36 r 13.3 45 10.1 41 r 64.4 45 63.6 56 6 56.1 46 61.1 36 a 19.4. 34 61.4 46 1 54.5 55 54.0 50 J 10.2 41 66.1 36 x 55.0 20 53.6 26 L 54.1 31 51.6 45 Redwood 50.0 40 56.1 43 n 65.0 40 63.6 22 a 64.2 61 49.4 61 o 41.9 46 41.5 53 p 69.0 42 60.0 45 9 63.0 46 66.4 36 a 50.0 46 50.0 36 Pine Ridge 38.0 50 46.2 39 6 36.4 33 40.6 32 r 53.5 43 51.3 39 o 54.2 46 50.0 46 v 46.1 11 49.3 11 I 46.3 56 56.9 56 Leneuee 43.6 46 42.3 52 Ottawa 36.0 50 41.6 46 x 33.3 39 54.3 35 r 31.0 13 51.6 66 6 53.6 52 34.0 41 AA _ 41.4 36 34.6 46 NOTES: The etate goal for ecience expecte that 75‘ of the etudente will perform in Category 4: 1.6., achieve 7S-lOOt of the objectivee. Starred numbere indicate that the etate goal vae net or eurpaeeed. When looking for trende in the date, keep in mind that the smaller the group, the more weight the performance of any individual carriee. Reeearch and Evaluation Servicee December 6. 1990 CHAPTER FOUR PRESENTATION AND ANALYSIS OF THE DATA In this chapter I present and analyze the data of this study. I first provide contextual data that help the reader to understand the information presented here. The background information includes the history of MEAP, a description of MEAP and its tests, and information about the Lansing School District. Next I present several major themes and minor themes for thinking about teacher use and interpretation of state assessment data. These themes were drawn from viewing the data in its totality and context. Against these themes the summary of data will be set. I then present analysis of the raw data obtained from the teacher and principal sessions. What you will see in this chapter are summaries of these data and analyses. W The Michigan Educational Assessment Program was first administered in January, 1970, as a norm-referenced test in the areas of vocabulary, reading, English usage, and arithmetic (Michigan Department of Education, 1990). The test was originally designed to support a six- step state-wide accountability model. The six step model, described in a 1972 Michigan Department of Education position paper included: 1) 48 49 identification of common goals, 2) development of performance objectives, 3) assessment of needs, 4) analysis of delivery systems, 5) evaluation, and 6) recommendations for improvement. Experience from the first four years of MEAP use led State Department of Education officials to conclude that the test was not adequate to describe the ”status and progress of Michigan basic skills education” (Michigan Department of Education, 1990, p. 1). Curriculum specialists and teachers from throughout the state were convened in 1971 and given the charge to develop performance objectives in the basic skill areas. The resulting objectives were reviewed by grade level commissions, the elementary and secondary education council and were finally approved and adopted by the State Board of Education. Objective referenced tests were developed to measure attainment of these goals. Beginning in 1976 the reading and mathematics objectives and tests were revised with assistance from the Michigan Reading Association and the Michigan Council of Teachers of Mathematics. The revised objectives were adopted by the State Board of Education in 1980 and test items were then developed to support the new objectives. WW MEAP consists of reading and mathematics tests administered during the fifth and sixth week of school in grades 4, 7 and 10. The MEAP science test is administered at the same time to students in grades 5, 8 and 11. The tests are designed to measure student skill attainment from their previous years of schooling. That is, the MEAP 4th grade reading 50 and math tests are designed to measure students' knowledge and skills learned in grades K-3. MEAP tests are un-timed "power tests." Students receive test booklets and answers must be marked by the student on a separate answer sheet. All students are tested except for some special education students and some limited English speaking students. We; The current MEAP mathematics test was field tested twice and has been in use on a state-wide basis since the 1980-81 school year. A new mathematics test is scheduled to be administered in the fall of 1991. The current math test is made up of two parts, the core math test and the correlated test. The fourth grade test contains 84 items testing 28 core objectives. The core objective areas for 4th grade mathematics are: numeration, whole numbers, fractions, metric measurement, non-metric measurement and geometry. In addition 24 items are used to test 8 correlated objectives. The correlated objectives are: numeration, whole numbers, geometry, and decision making and problem solving. (See Table 4.1 for all areas tested in grades 4 and 5.) The correlated objectives are not included in the proportions report. To pass an objective a student must answer correctly two of the three items that test the objective. School, district and state mathematics results are reported as the proportion of students who attain mastery. The state expectation is that 752 or more of the students should pass 751 or more of the objectives. That is, 75! of the students would perform in Category 4 (the highest category). 51 Table 4.1 MEAP Skill Areas Assessed (Mathematics and Science) r de u a e a Skill Area Nggber gf Objectives CORE Numeration Whole Numbers 1 Fractions Metric Measurement Non-Metric Measurement Geometry NHNNNO Sub-total: 28 CORRELATES1 Numeration Whole Numbers Geometry Decision Making and Problem Solving F‘P‘U‘P‘ 00 Sub-total: 1Attainments on the mathematics-correlated objectives are not included in the proportions report. Gr v e e §LLLI_AI§§ b b v 5 Life Science 5 Earth and Space Science 4 Physical Science 6 Science Process 13 Science, Technology, and Society 2 TOTAL: 30 52 MEAP Scienge Iest Development of the MEAP science tests began in the 1983-84 school year under the supervision of MDE staff with assistance from selected science educators from around Michigan. An item try out occurred in 1984, with pilot testing in 1985. The state-wide use of the science test began in the fall of 1986 at the 4th, 7th, and 10th grade levels. In the fall of 1989, the administration of the science test was moved to grades 5, 8, and 11. At the fifth grade level, 30 objectives are tested on the 90-item test. Areas tested in science include life sciences, earth and space science, physical science, science process, and science, technology, and society. (See Table 4.2.) Science results are reported in the same manner as mathematics, the state goal being that 751 of the students pass 751 of the objectives. Table 4.2 MEAP Mathematics and Science Tests Categories of Achievement Criteria Category 4 Students who attain 3/4 or more of the objectives (highest) Category 3 Students who attain 1/2-3/4 of the objectives Category 2 Students who attain 1/4-1/2 of the objectives Category 1 Students who attain 1/4 or fewer of the objectives SOURCE: MEAP figgdbggk, 1990, p. 27. 53 e d es Prior to 1989 the MEAP reading test had been a test of basic skill attainment and was reported in a manner similar to mathematics and science. In 1989 a new version of the test was used measuring "essential skills" and based on a new definition of reading grounded in a meta-cognitive model. During the 1989 and 1990 administration of the test students took two reading tests (the essential skills test and an abbreviated version of the basic skills test). The use of the two tests allowed school districts and the state department of education to make comparisons with previous years' tests. The essential skills test is described by MDE as a higher standard of reading performance than the basic skills version. It grew out of the 1982-83 review of the basic skills reading test. At that time the reviewers (representatives of MDE staff and the Michigan Reading Association) concluded that the definition of reading upon which the basic skills reading test was based did not reflect current theory and research in reading (MDE, 1990). Based on that review a new definition of reading was adopted and revised objectives written. The new MEAP essential skills reading tests were written to reflect the changed definition and objectives. The MEAP flgnghggk (MDE, 1990) describes the test as a “broad-gauged measure which reflects the goals of reading instruction as closely as possible.” There is no one-to-one correspondence between the items and reading objectives. Instead three major categories (reflective of characteristics of a good reader) are tested by the essential skills reading test: 54 -- constructing meaning -- knowledge about reading, and -- attitudes and self-perceptions. The format of the test has changed as well. The older basic skills version of the test contained brief reading passages and students answered three or four questions based on each passage. The new essential skills test contains two lengthy reading selections. The first is a complete story; the second is a portion of a chapter of a subject area textbook. The 1990-91 version of the test included a passage from a social studies textbook. Illustrations and pictures appear to be as they originally appeared in the textbook. In the 1990-91 version of the MEAP reading test students answered 46 items on each of these reading passages. Items testing topic familiarity are also included. These items ”evaluate a student's knowledge of the super-ordinate concepts that are important to the understanding of the reading selections" (Michigan Department of Education, 1990, p. 5). In this section of the test students identify -- characteristics or attributes of each concept, -- examples of each concept, and -- relationships among concepts. See Table 4.3 showing reading test content. W MEAP test results are reported in several formats. Some are standard; a school district receives these reports automatically. 55 Others are optional and must be ordered by the school district. MEAP reports are sent to the superintendent, the school principal, and classroom teacher by MDE. Table 4.4 shows the reports available and which are received by the Lansing School District. Rather than receive the racial-ethnic and gender reports developed by the State Department of Education, Lansing School District testing staff develop their own reports analyzing the MEAP results by racial-ethnic group, by gender, and by mother's education (an indication of family social economic status). The MEAP flgndbogk contains sample report formats. These samples and actual MEAP results for selected schools were used with the teachers and principal in this study. They appear as Appendix F. WM In a section entitled ”Using MEAP Test Results” the 1990-91 MEAP flagghggk describes the expected use of MEAP results as follows: The main use of MEAP test results is to focus on student achievement. By examining the results and developing and implementing a curriculum plan, a school takes significant steps toward improving student achievement. This effort necessitates a total staff involvement. Each staff member should be committed to this improvement process and take responsibility for his or her participation in it. It is critical that the building principal be personally involved and allow and encourage all teachers and support staff to participate as well. (Michigan Department of Education, 1990, p. 7) This quotation is subtitled “Effective Schools" and its focus mirrors the effective schools philosophy. It should be noted that the MDE staff are strong supporters of effective schools/ school improvement efforts as articulated by Lezotte, Edmonds, Brookover, Sizer and others. The 56 Table 4.3 MEAP Essential Skills Reading Test Content G a u figment STORY SELECTION Beyond Text Text Intersentence Topic Familiarity Knowledge about Reading Student Self Report -- Performance -- Effort -- Interest INFORMATIONAL SELECTION Beyond Text Text Intersentence Topic Familiarity Knowledge about Reading Student Self Report -- Performance -- Effort -- Interest SOURCE: MEAP flangpggk, 1990, p. 6. l2 14 ll 12 14 57 Table 4.4 MEAP Reports Available to School Districts STANDARD REPORTS Individual Student Report Classroom Listing Report School Summary Report (includes Proportions Data) District Summary Report (includes Proportions Data) Test Item Analysis--Mathematics and Science (classroom, school, district levels) Special Education Report Parent Pamphlet OPTIONAL REPORTS RECEIVED BY LANSING Feeder School Report No Racial-Ethnic Report No* Gender Report No* Research Summary I Report No Research Summary II Report No *Lansing research and evolution develop their own reports analyzing MEAP results by racial-ethnic group, gender, and mother's education. MDE has established an office and staff who focus on school improvement and restructuring. The language of effective schools and school improvement appears in many MDE documents. The MEAP flgnghggk describes use of MEAP results by three levels: school level use, individual student use, and district level use. In 58 the portion of the MEAP Handbook that describes school level use the MEAP Hgngbggk authors state ”. . . use of MEAP results at the building level provides the best means of impacting student achievement” (p. 8). The MEAP flgnggggk asks and answers the question ”Why use MEAP?" saying: -- MEAP focuses on student achievement. -- MEAP test results provide valuable information on the status of essential skills education in schools. -- MEAP helps match skill needs with curriculum and instructional programs. -- MEAP helps teachers plan group and individual instruction. The figndbook goes on to state that . . . ”School personnel should use MEAP test results to help: -- identify the skill needs of students tested; -- review the curriculum and instructional programs; -- establish instructional priorities for the school year; and -- involve parents in the educational process" (p. 8). The figudhggk describes the responsibilities of the school team to include using MEAP results to explore teacher expectations of students, identify problems and needs, conduct curricular and instructional review to determine current strategies used to teach skills tested, and make decisions, set goals, and develop a plan to address needs (Michigan Department of Education, 1990, p. 8). It is important to the remainder of this study to note that specific expectations have been articulated to teachers for the use of DUQKP results. In Lansing each school has a teacher who has the duties of designated test coordinator. The test coordinator receives the MEAP him, but it is not known if the W is read and it is doubtful 59 that the contents of the flanghggk are shared with fellow teachers in any detail. The Context of MEAP in Lansing Lansing School District administrators take MEAP seriously. Each year central office staff are deployed to each school to assist with the administration of the test. Guidelines have been developed as to appropriate ways central office staff can assist. For example, staff teach 3rd graders in combination third/fourth grade classes while the 4th graders take the test, take those students who finish early and lead them in an activity or monitor the stragglers who have not yet finished when the rest of the class is ready to go on to other things. Yearly workshops are held on proper test administration for all teachers who will give the test. Once the results are returned to the district presentations are made to the Superintendent's Cabinet and building administrators and the Board of Education reviewing MEAP results over a period of years and identifying trends, concerns, and highlighting gains. The review session with the Board of Education is reported by the local newspaper. The MEAP data are disaggregated by race, gender and mother's education (an indication of socio-economic status of the family). Staff from the research and evaluation office prepare an analysis of each school's test data profile including MEAP tests, the Stanford Achievement Test and the locally developed Curriculum Monitoring System (CMS). Meetings are scheduled with each school's staff and these test data are reviewed. This elaborate system for support of test data has evolved over two decades. 60 Each content area of the curriculum is reviewed once every five years. Curriculum steering committees, made up primarily of classroom teachers, with subject area consultants and others review the curriculum for the subject areas (e.g., elementary mathematics). Teachers look at the fit between the curriculum and currently administered tests, including MEAP. They revise locally developed curriculum statements and consider their fit with the newly developed state core curriculum statements. They consider, review and recommend needed books, support materials and staff development. Their findings are referred to the Instructional Council (designated by contract with the teachers' union and made up of a majority of teachers appointed by the union) and parents, students, and administrators. Recommendations go from the Instructional Council to the Superintendent. The new reading, science and math MEAP tests have been reoccurring topics on the respective steering committee agenda. Testing steering committees for elementary and secondary levels play a similar role with the entire district testing program. They, too, review MEAP tests and their results. MEAP objectives come from the MDE Essential Goal Statements, as do the Michigan Core Curriculum Statements. Figure 4.1 shows the relationships among the MEAP, Essential Goal Statements, Core Curriculum, and Lansing School District Curriculum. We In 1982, following a newspaper article that proclaimed "City Low in Urban Scores” (Ihg_L§n§1ng_§§g§g_lggrn§1, February 21, 1982, p. 1B), the Superintendent established a district priority to improve the MEAP 61 ESSENTIAL GOAL STATEMENTS / \ MDE CORE CURRICULUM . MEAP LANSING SCHOOL DISTRICT CURRICULUM STATEMENTS The Lansing staff also consider the following in developing curriculum statements: -- Stanford Achievement Test -- Textbooks -- Other subject areas -- Articulation with higher education -- National subject matter association recommendations -- Review of current literature and practice Figure 4.1. Relationships among Essential Goal Statements, MDE Core Curriculum, MEAP, and Lansing School District Curriculum Statements. 62 performance of Lansing schools. Schools' staff were trained in test taking skills to share with students. Each school was required to improve MEAP scores by at least 10 points by the following fall and to submit a plan to the central office outlining how they would do so (Superintendent's memorandum, March 12, 1982; Assistant Superintendent's memorandum, March 23, 1982). Focus was drawn to how to best administer the test and the yearly testing workshops and the practice of sending central staff out to assist in MEAP testing was put into place. A MEAP ”Bird" visited each school and met with students to explain the purpose of the test and test taking skills. The MEAP bird was a local invention of Lansing School district staff. The ”bird" is a college student who dresses in a bird costume ('MEAP-MEAPt'). Later in 1988, a leap year, the MEAP bird evolved into a MEAP frog. Most recently the MEAP frog mutated into a MEAP turtle. Schools whose students scored well on MEAP received recognition from the Superintendent and Directors of Elementary and Secondary Education. They received staff breakfasts to celebrate their success. Schools that did not score well were required to submit plans for improvement. Not only was overall achievement reviewed, but the ”gap" between minority and majority student achievement as well. Schools were required to submit plans for how they would reduce/eliminate this gap. Many attribute the improvement in Lansing's MEAP scores, at least in part, to these efforts. Although schools are no longer required to submit MEAP improvement plans to the central office, the superintendent's 1991-92 fall opening 63 school address to administrators and the 1991-92 Instructional Division goals call for improvement in MEAP and Stanford Achievement Test scores. f e i 001 Over a five year period ending in the 1991-92 school year the Lansing School District trained Effective Schools/school improvement MDE teams in each of its schools. The process as employed in Lansing is based in the Edmonds and Lezotte model of school improvement and organizational development. This is important to this study because although most schools represented were in their first year of training some were in their 3rd, 4th or 5th year. All teachers referred to their school's effective schools team as a venue for use of MEAP data. The effective schools process is an omnipresent part of the culture of this school district. ANALYSIS OF THE DATA This study is based on teachers' reflections and narratives. I have taken advantage of my role (as an insider in some respects) in the school district to understand and reflect on the talk of these teachers. In this study Focus Groups and Interviews were used as the primary data gathering devices. As I planned the design of this study I considered Focus Groups to be a special form of Interviews; one in which participants engaged in guided conversation among themselves. Van Manen (1990) describes the purpose of Interviews in the following way. I find these purposes served by the Focus Groups as well. 64 In hermeneutic phenomenological human science the Interview serves very specific purposes: (1) it may be used as a means for exploring and gathering experiential narrative material that may serve as a resource for developing a richer and deeper understanding of a human phenomenon, and (2) the Interview may be used as a vehicle to develop a conversational relationship with a partner (interviewee) about the meaning of an experience. (p. 66) In this study Focus Groups and Interviews were used for both of the purposes described by Van Manen. Initially Focus Groups and Interviews were used to learn how teachers use state assessment data. Here I anticipated learning from teachers their views about the importance of the test, the extent they viewed the test as measuring valid content and about the pressure teachers felt to achieve high test scores. But this study also provides the narrative material to paint a portrait of the issues being studied. Like a portrait the Focus Group and Interview data provide background, shading and perspective. Van Manen (1990) gives guidance for review of text to isolate themes. His approaches parallel those used in this study to analyze the data and, eventually, to identify themes. Van Manen describes three approaches. (1) In the holistic reading approach we attend to the text as a whole and ask, What sgntentious phras sg may ggptgrg thg funggmgnt a1 We then try to express that meaning by formulating such a phrase. (2) In the selective reading approach we listen to or read a text several times and ask, Eng; stgtement(§) g; ph;§§g(s) segm tic ent e These statements we then circle, underline, or highlight. (3) In the detailed reading approach we look at every single sentence or sentence cluster and ask, Eh§§_ggg§_§h1§_§gnggngg_gg 1‘1‘C 3.... l‘ '! 1°“.‘Q'! ! !' figsgxihggl (Van Manen, 1990, p. 93). 65 H w the a W e Audio tapes were made of each Focus Group and Interview session as well as written notes. In the first Focus Group, a video tape was made as well. Transcripts were made from the audio tapes and my notes. These transcripts were reviewed in three ways. First I read the transcripts to see what other questions or comments they raised. I noted these. Next I read the transcripts considering them line-by-line to make note of the major or recurring issues, contradictions or affirmations of other teachers' talk, and comments that most closely addressed the questions asked in the Interview and the questions of this study. Marginal notes were made. Next I read the transcripts again, this time looking for the big themes, the outstanding questions, and other insights that came to mind. During this reading I mulled over what I read and reflected on its possible meaning. I looked at other data, for instance the actual MEAP results for the school in question. I looked across Interviews to see if connections could be made across grade levels, school performance, teacher's experience, or school's poverty rate. Again marginal notes were made. I read the transcripts another time to identify quotations that represented major ideas on themes. Here I looked for quotations that clearly stated ideas, thoughts, or feelings that were representative of more than one person in the study, offered a different perspective or were so compelling as to warrant further exploration. These statements were highlighted. As I prepared to write narrative accounts of the transcripts I developed a common outline that encompassed the two Focus Groups and six Interviews. The common outline approach was used as a means of organizing the data 66 for the reader and for myself. It proved to be a valuable step from which to look across Narratives. These three approaches to reviewing the transcripts involved countless re-reading of the transcripts. From this process I developed the Narratives of the Focus Groups and ‘Interviews. These Narratives were a way of re-telling what the teachers told me. They include contextual and demographic information to help the reader understand the teacher and school. Once the Narratives were written I discovered that they were too lengthy to be included in the body of the study and that connections needed to be made for the reader to get the fullest meaning from the data. I next wrote the story of what I learned. I looked across the two Focus Groups and six Interviews and drew on the other data presented in this study. I read the Narratives several times, and as needed referred to the transcripts. Once again I made marginal notes and reflected on what I read. To write the story I used a device I have seen used in advertising and in the development of media presentations: a storyboard. The storyboard told the story that I heard in the data on the left side of the page and gave the “picture," in this case the place where the evidence supporting the story could be found in the transcripts, on the right side of the page. This approach proved useful as a means of organizing masses of data. The storyboard gave a visual outline for preparing the analysis of data presented here in Chapter Four. I moved across the boundaries of the Focus Groups and Interviews to tell you what teachers told me and each other and to reflect on their words and stories. I combined teachers' talk with contextual information about their schools (MEAP results and demographic information) to present a situated picture of teachers' 67 beliefs about interpretation and use of MEAP in their thoughts and actions. I used the teachers' own words as much as possible. W While reviewing and analyzing the data of this study, I considered how to demonstrate the varying perspectives of the teachers and principal. In some cases these perspectives converged to display a shared meaning of a context or experience. In other cases these perspectives showed marked differences in the ways the teachers viewed the events or circumstances of interest in this study. I used these examples to give depth to the data presented and to the conclusions drawn. One strategy used to add depth to the text was to triangulate the data presented. I cross-checked the observations of a participant with those of another. The purpose in this strategy was not to validate the perspective of one participant with others, but to explore how perspectives can differ and lead to differing conclusions and responses. A second strategy that I used was to look for contrasts. Contrasts allowed me to display and explore ways that experiences and other conditions led to differing actions and explanations by teachers and to make vivid to the reader the complexity of reaching understanding of teacher use and understanding of test results. W As I approached writing what I heard and learned from the teachers I considered how to maintain the teachers' voice and still present a 68’ document that met acceptable standards for scientific inquiry. Two criteria are important. They are apparency and verisimilitude. Connelly and Clandinin (1990) use these terms to describe criteria of qualitative research. Apparency is the support for the claims and notions raised in the text made clearly and convincingly to the reader. In this sense the writer must display the data and set the context. Apparency is accomplished in this study by using teachers' words frequently, and by carefully describing a detailed contextual backdrop against which this study is situated. Verisimilitude refers to the plausibility of the claims made. The reader asks, "Do the claims carry the appearance of being true?" Verisimilitude in this study is achieved by drawing the reader into the world of the teachers through description and context. I have placed the Narratives in time and location. I have shown the background, shadow and shadings that give full texture to the words of the teachers. The method used to analyze the data and to write the Narratives allowed me to look at the data as a whole and to look at the particular. The challenge was to convey the vision of the whole while using detail to sharpen the image and give the sense of apparency. Narrative explanation derives from the whole. We noted above that narrative inquiry was driven by a sense of the whole and it is this sense which needs to drive the writing (and reading) of narrative. Narratives are not adequately written according to a model of cause and effect but according to the explanations gleaned from the overall narrative . . . (Connelly and Clandinin, 1990, p. 7). Since my voice is heard often in this study it was important to identify the "multiple I's" (Connelly and Clandinin, 1990, p. 9). I did this in Chapter 3, but I remind the reader that they will hear ”me," the 69 researcher, and ”me,“ the school district administrator, frequently throughout the analysis of data sections. e l t n h s amon the Themes Sto board a d ata The reader may be helped by seeing the relationship among the themes, the storyboard, and the data presented in this study. As stated earlier the themes were a way for me to look across the data to describe the big ideas of this study. The themes are presented early in this chapter so that the reader can use these themes to organize the data that follow. The storyboard was a device that I used to organize the story told to me by teachers. I present it here to give the reader a preview of the data that will follow and to show how this story is related to the themes. THEMES In this section I will describe themes that I discovered in the analysis of the data of this study. Van Manen (1990) writes about themes as a means of conveying the essential meaning of experiences. The notion of theme is used in various disciplines in the humanities, art, and literary criticism. In literature, "theme" refers to an element (motif, formula or device) which occurs frequently in the text. The term "theme” is often applied to some thesis, doctrine or message that a creative work has been designed to incorporate (Van Manen, 1990, p. 78). Themes are used in this study to give order and structure to data Presented. The themes are a means of organizing the messages drawn from the data. Four major themes and six minor themes are used here to describe teacher use and interpretation of MEAP data, in some cases they 70 are expanded to describe state assessment testing in general. The themes were developed from my synthesis of the contextual and teacher data compiled for this study. The themes are: MAJOR THEME ONE: Perceived Usefulness MEAP Results are Not Useful--They Don't Tell Me Anything MINOR THEME: Test Results are Looked at as a Unit MAJOR THEME TWO: Teaching to the Test MINOR THEMES: MEAP is Leading to a State-wide Curriculum Results Are Confusing-~What do all these Numbers Mean? MAJOR THEME THREE: Teachers are Overwhelmed, and MEAP Aggravates this Condition MINOR THEMES: Not Enough Time Too Much to Teach Too Much Testing MAJOR THEME FOUR: The Pressure is High, But the Stakes are Low ELABORATION OF THE THEMES Pergeiveg Usefulness; Thgy Dog't 1611 Me Anything With one exception the teachers who participated in this study made this statement strongly and frequently. But as each Interview Progressed teachers identified numerous ways in which they put MEAP to some use. Teachers use MEAP as one assessment with which they set school 80618. They review MEAP along with other achievement data during 71 effective schools meetings and consider this information as they develop effective schools mission statements and plans. Most teachers referred to MEAP as a yardstick by which they measured their school's performance compared to other schools in the district and area. And some even use MEAP to assure themselves that they are doing something right in their instruction. MEAP science results did seem to ”have a lot to say” to the teachers. Perhaps this is because the science test is new, or because Gathucators and the media have begun to talk more about science teaching. But perhaps it is because teachers find little else that is acceptable and assessable to guide elementary science instruction. The MEAP science test poses an interesting cause and effect question. Do teachers find science testing more informative due to the contextual pressure to attend to and improve science teaching, so they look to the MEAP test for curriculum guidance? (We will see evidence of this in fried's Interview.) Or, do teachers feel pressured by poor MEAP science performance thus causing them to seek new ways to improve instruction to result in higher scores? We will see both scenarios in the data Presented later. Since teachers conclude that MEAP does not tell them anything, I am Prompted to ask of the data what is it that teachers expect MEAP to tell. Do the teachers' conclusions mean that MEAP does not tell anYthing unique, that it does not tell anything in a timely fashion, that MEAP does not provide an accurate picture of class or student performance? Teachers seem to be asking of MEAP diagnostic questions; what skills does Joey still need?, In which group should I place Janice? 72 MEAP is not used for individual student decision making. Teachers refer to other measures to identify and diagnoses student needs (e.g. , teacher made tests, unit tests, and the Curriculum Monitoring System). Teachers talk about and seem to think about MEAP results as a classroom or grade level package. And their use of MEAP seems aimed at that intact classroom unit . Wynn The teachers studied viewed MEAP results in aggregate. They tend not to refer to individual student performance during the Interviews unless a specific Interview question posed led them to do so. The teacher talk about MEAP was in terms of whole school generalizations (e.g. , high, low, improving scores). In reading and science there was little focus on individual test items. I suspect that this is because teachers are least familiar with these items (reading and science are the two newest tests). I wonder if the test booklet had not been available during the Interview if teachers would have talked even less about student performance on particular items. Teacher testing traditions encourage us to look at tests to tell sPecific things about a child. But in spite of the MEAP W Suggestions for using MEAP results for individual student decision Ina~1<1ng purposes, MEAP is not used or designed to be used in that way. MEAP data do not come to teachers at a time when the results would be helpful to plan for individual children. MEAP results are not cross re:Eerenced with major teaching materials, except as individual districts may do this task themselves, thus making it hard to relate test 73 performance to classroom lessons or strategies. The objectives on which the MEAP tests are based are not easily available to teachers thereby making individual planning from the results nearly impossible. The fact that teachers think that the MEAP is intended to be used for individual student diagnosis and that instead they look at MEAP results as classroom unit information contributes to the view that teachers hold that MEAP is not used. Teacher expectation is for MEAP to assist in individual student diagnostic decision making. It does not, so teachers conclude that it is not of use to them. c n the Tes Teaching to the test is among the strongest themes found in the Focus Groups and teacher Interviews. Teachers are concerned about teaching to the test. Some think that other teachers in other districts teach to the test so they feel at disadvantage when compared with these schools. Other teachers are careful 13th to prepare their students in particular ways for MEAP for fear of teaching to the test. The teachers interviewed differed greatly in their description of what was appropriate test preparation and what was not. This leads to some POtential dilemmas. What happens when teachers and administrators or decision makers disagree about what constitutes teaching to the test? If teachers define appropriate test preparation (teaching to the test) more narrowly than their principal or central office are they viewed then as less than aggressively pursuing high achievement? If the principal or Central office staff view teaching to the test narrowly they may conclude that teachers are cheating or are unethical. Despite the fact 74 that concern for teaching to the test is universally held there is no common definition of what constitutes teaching to the test short of giving the students actual test booklets to practice prior to the test. The school district offers examples of appropriate and inappropriate ways to prepare for tests but the message does not seem to have been clearly received by teachers. The sanctions for cheating are weak. The most flagrant example of teaching to the test would at most warrant a Visit to the principal from the supervisor of the research and evaluation office with a repeated offense resulting in a letter in the teacher's personnel file. So, if fear of sanctions does not raise this as a strong issue among teachers, commendably teachers' sense of professional ethics does. Teachers model what they teach their children: ”You don't cheat on tests.” e -w . Some of the teachers described MEAP as leading to a state-wide curriculum. That development would not be welcomed by most of the teachers in this study, for two reasons. First teachers feel that their curriculum is constrained by the volume of formal curriculum from many sources (e.g. , textbooks, local curriculum statements, various standardized tests) and by lack of tiii-line to teach what is expected. Second, teachers conclude that as Professionals thgy should determine the content, method, scope and Sequence of instruction for their students. This, of course, relates to the notion of teaching to the test. Teachers feel pressure to have t:l'leir students score well on MEAP. But to accomplish this it seems that the curriculum taught needs to be closely aligned with the content of 75 the test. Teachers know little about the curriculum implied by and tested by MEAP. Because the old math test had been used so often and aligned fairly well with the popular textbooks this was not as great an issue for math as it will be with the first use of the new MEAP math test in the fall of 1991. Teachers know little about either the MEAP science or the reading curriculum. A state-wide curriculum cannot be achieved successfully as the result of a few hours of inservice training to a few teachers in a school district. WW Teachers have difficulty making sense of the numerous MEAP reports. They take away from MEAP reports only the big picture: ”We did good," ”We're ok," We're low." Beyond those broad conclusions teachers find the reports of results confusing, especially in reading. Teachers seem to be looking for a call to action; some clear diagnostic and prescriptive statements specifying what they should do to improve student performance. The MEAP math and science tests are somewhat responsive to this need (given that the teacher accepts the underlying curriculum tested). The new MEAP reading test does not offer prescriptive information. The reading Behool profile is compiled in a manner unlike the more familiar math and 8C- 1ence results. Math and science reports indicate the percent of 8titadents passing ranges of objectives. Teachers can think of the reSults in terms of the number of students performing in high, medium high, medium low and low categories (defined by the percent of obj ectives passed). Teachers can also look at results by objective to See which objectives their students failed. Fourth grade teacher 76 Valoria, for example, found that her students failed the zero multiplication facts (Interview 1). Reading results, in contrast, are no longer reported in a similar manner as a result of the movement away from a skill based reading test. Edith (Focus Group 2) did find it helpful to know that boys did less well on the fiction section of the test and her school staff responded by purchasing fiction books of interest to boys. But when describing this same finding for her school, E Valoria rhetorically asked whether that finding was telling a teacher anything that she didn't already know (p. 23). That is, she heard no call to action in the same gender-different results. Teachers find it hardest to draw meaning from the reading test results. I don't find anything informative about the new reading. 334--what good does that number do? It doesn't tell me anything. Maybe it is supposed to. . . . What does it tell you other than they are good or bad? At least in math you can look at it, and some schools can say they can't fill in the number in the sequence. . But I don't see anything to work on in the reading. (Valoria, Interview 1, pp. 21-22) Teachers describe themselves as "overwhelmed.” The word was used spontaneously by many of the teachers and in many contexts. Teachers describe themselves overwhelmed by time pressures, testing, curriculum changes and individual student academic and social needs. They describe their students as overwhelmed as well. Ngg_gngggh_§img. The concern for time was raised on several occasions throughout the Interviews and Focus Groups. Teachers said that there is not enough time to cover the curriculum. Three or four 77 weeks of review for MEAP in the fall followed by more than a week of MEAP testing erodes needed instructional time. Teachers do not have sufficient time to get to know what their colleagues at other grade levels actually teach. The concerns about time may have been highlighted by the teachers in this study because the Focus Groups and Interviews were conducted during the last few weeks of school, a period when teachers feel the press of what they have not yet accomplished with their students. During these weeks teachers were also conducting Stanford achievement testing and local Curriculum Monitoring System testing and trying to squeeze in spring field trips and special year-end events. It is no wonder that lack of time was a major concern and was related for these teachers to the issue of their use and interpretation of MEAP results. But the time of year that these Interviews were conducted should not obscure the fact that teachers feel pressured by lack of time. Anyone who listens to teachers hears this concern no matter the time of year. Igg_mggh_§g_§g§§h. Strongly related to the notion of not enough time is the notion that there is too much to teach. Valoria described a typical school day (Interview, p. 20). In her view there wasn't enough time to cover the local district formal curriculum let alone meet formal or informal curriculum demands from other sources (e.g., MEAP). In this light one may conclude that it isn't any wonder that MEAP does not influence the curriculum more; there may be too little flexibility with which to work. One gets the sense from the teachers interviewed that the school year is a race to squeeze it all in, to cover it all. Other 78 researchers (e.g., Porter, et al., 1979) have raised the issue of coverage for exposure versus teaching for understanding. For the teachers of this study the pressure from an overloaded curriculum is relentless. Igg_mugh_£g§£ing. In the teachers' view MEAP, the Stanford Achievement Test and Curriculum Monitoring System tests cover similar content and produce similar results. Jerri described MEAP results as very consistent with other assessment tests. For teachers this is a major weakness of MEAP. In reading and math MEAP provides little that is unique and therefore worth the expenditure of instructional time and resources. Again the time of year of the Interviews may have effected this perception. Also the testing program in Lansing is extensive. Every-year, every-pupil testing with the Stanford Achievement Test and the extensive fall and spring Curriculum Monitoring System testing place a heavy burden on the 900 available hours of student instruction. es ur 5 Hi h but the akes e Low Teachers felt pressure to do something for MEAP. But what? Fourth grade teachers looked for answers in their four weeks of review for the test. Third grade teachers talked about their efforts to incorporate experiences like those the students would encounter on the test into daily classroom activities (e.g., ”strongly agree" questions), but major attempts by the individual teachers to realign the curriculum seemed inappropriate and beyond their control. Teachers described their attempts to prepare students for the science MEAP test as most closely 79 resembling curriculum change. I would argue, however, that the changes described may be better defined as changes in activities than a true change in curriculum. And perhaps of more importance, the changes described by the teachers do not reflect a unified view of science curriculum. The preparation for MEAP reading and math were more in the realm of activity changes or additions to student activities than changes in the curriculum or pedagogy used in the classroom. These I would classify to be more tinkering than real change; but that may be appropriate. Pressure to improve MEAP scores is felt by the teachers and the principal and in their estimation by the students as well. Some principals try to protect their teachers from the pressure and help them focus on the school's goals and the big picture. -Others use the test results to push their teachers to excellence. Although MEAP has high visibility in the local press, probably more so than the Stanford Achievement Test because only MEAP allows cross district comparisons, there is no perceived pressure from the students' parents. One could postulate that in part this is a consequence of how the results are reported. Since most students fall in the top category and nearly all in the top two categories in math the likelihood of a parent seeing lower results for their child is remote. Parents may interpret that since their child is performing in the top one or two categories they are above average. Since parents only encounter MEAP reading and math three times in a child's school career (fourth, seventh and tenth grades) they have little experience from which to draw 80 iconclusions. All of this of course presumes that the parents actually see the results at all. N9 ggnsgguegge. The issue of stakes raises the question If scores .are bad, so what? If they are good, so what? It is safe to say that ‘there are no consequences for individual students resulting from low ldEAP scores. High scores can earn the student a certificate signed by the Governor. In Lansing, consequences for individual students come as the result 63f performance on a combination of assessment indicators, primarily the Stanford Achievement Test. Is the consequence of low scores greater for teachers? The description by the principal Bill of his transfer of the third grade teacher was based in part of MEAP scores but heavily on the Stanford results and fall and spring Curriculum Monitoring System as well. Bill implied that the change in grade level assignment was intended to reduce the teacher's negative impact on students' and school test scores. While a move may solve the problem for MEAP, placed at any other grade level, this teacher will continue to effect Stanford and CMS test results. Aieda as a new teacher says from her view teachers' careers are Imade or broken by test scores (Interview 5, p. 65). Jerri, a veteran teacher from the same school, contrasted MEAP performance before she cane to teach at Deerfield school with current results, concluding Proudly that, "We're doing something right" (Interview 2, p. 27). 81 One way to view the issue of the stakes of a test is to describe stakes in terms of the consequence for students. As noted above the 4th grade MEAP results hold low stakes for students. But the notion that "the stakes are low" deserves a second and broader look. For teachers in the Lansing School District, the consequence of sustained poor MEAP performance is relatively minor. Serious consequences for teachers (e.g., change in assignment, loss of job) would necessitate that numerous factors be considered, not just the scores on one test. There are other, less visible consequences to poor MEAP performance. These consequences may have profound and lasting effect. Low scores on a state assessment test like MEAP can affirm opinions that the public may hold about a particular school or about urban schools or public school and their students and teachers in general. Basing these opinions on the scores of one kind of test, even if viewed over time, may lead to a skewed picture of schools and their effects. Teachers in this study were concerned about this possibility. Low scores can lead to a climate of increasingly centralized control of curriculum and instructional practices. . Pressure from low scores could cause teachers and curriculum leaders to infuse the curriculum with quick fixes that may undermine the integrity of the entire curriculum. Although teachers in this study question the content of the MEAP in some areas, for the public at large and even those teachers with less contact With MEAP the test carries the presumption that it is appropriately measuring worthwhile content. 82 To conclude that the consequences of MEAP are low, based on the immediate consequences for students and teachers, may be a shortsighted ‘view. The consequences can be broad and profound. STORYBOARD The following "storyboard” displays the story line for the analysis ‘of'data section of this study and names the teachers whose words contribute to each conclusion. “Teachers say that they do not use MEAP . . . Edith Valoria Doris Jerri Lynn Rose .and they say that they don't use MEAP for individual student diagnosis or planning . Valoria Jerri Doris But teachers do use MEAP for other purposes, such as . comparisons to other schools . Ted Bill curriculum review: Ted as a curriculum bench-mark in science: Ted as confirmation that what they are doing is OK: Jerri to assist in goal setting: Edith for science planning: Katheryn Jerri Karen and as part of the big picture along with other data. Jerri Bill Valoria 7Peachers find MEAP results consistent with the other tests given--SAT Innd CMS: Jerri Karen It is unclear how well the local curriculum and MEAP match . 83 in science . . . Bill Ted Rose Lynn in math . multiplication: Valoria measurement: Rose application: Katheryn in reading . . Valoria Katheryn Teachers ask ”How will the mathematics curriculum and MEAP fit?” Katheryn Sheila Carol Teachers feel that there are other things more important than the things tested on MEAP . Aieda Rose Ted Teachers ask ”Who determines what is on the test and in the curriculum?" Ted Rose Bill Teachers conclude that there is too much testing . Edith Ted and there is too much to teach . Ted Lynn Sheila Edith Teachers worry about a state-wide curriculum . Edith Aieda Teachers resent having to spend the first few weeks of school reviewing for MEAP . . . valoria Rose but do find some positive aspects to MEAP preparation: Ted At one time the District's response to MEAP was really crazy . Valoria Teachers don't get to see the MEAP test and don't know what's on it . Valoria Jerri Rose Teachers are very concerned about teaching to the test . 84 what does it mean? Sheila Edith Rose Bill Jerri Valoria examples . Valoria Bill Rose Teachers feel a lot of pressure to do well on MEAP . pressure comes from comparisons with other schools . Rose Ted (good effect) Aieda Dawn Lynn Sheila and from the principals and others . Dawn Rose Karen Jerri Aieda Bill Valoria Teachers find tests frightening and intimidating . Ted Aieda they are overwhelmed: Ted and ”uptight”: Katheryn But if there were no MEAP where would be the pressure to improve? Dawn Ted There is even concern about teachers being evaluated using MEAP results: Jerri Bill Critical decisions will be made about individual students based on their MEAP: Aieda There is a contrast in how the teachers at Deerfield view testing . Jerri Aieda Upon looking at the MEAP test teachers think that the informational reading selection is too hard: Valoria the reading selections are too long . Katheryn Valoria Aieda Rose the fictional story was too juvenile and inappropriate . Aieda Edith ? there are too many questions . Edith Lynn Valoria Aieda 85 In math . multiplication is tested too early: Valoria the 1008 chart is a bad item . Valoria Edith Overall the test is too hard: Valoria the test is simple: Bill When the MEAP results come back they don't tell me anything: Valoria What is MEAP measuring? Edith Lynn It's too hard to understand: Valoria One problem may be that MEAP is given too early/too late in the fall: Valoria But we really don't need MEAP: Lynn Teachers wonder who was involved in the test construction and hope that teachers were involved . Rose Ted Bill In reading the children lack needed life experience and prior knowledge to do well on the MEAP . . . Valoria Bill Rose THE DATA Ieache; Usg 9f MEAP With one exception (Ted, Interview 4) teachers say that they do not use MEAP results. "I don't think it (MEAP results) says anything. I don't use the results. I don't know anybody that does (Interview 1 p. 14).“ Valoria's comments are typical of the teachers interviewed. The thought that MEAP does not "tell" anything pervades the teachers' talk and raises the question of what is it that MEAP is expected to tell. Knowing more about this may help us understand teachers' use of MEAP. 86 Evidence from these Interviews leads me to conclude that teachers think that MEAP should provide information that will help them to diagnose students' needs. Perhaps this is because the MEAP flgngbggk describes MEAP use in this manner. But I did not see evidence that teachers had seen or were familiar with the MEAP Handbook. It could be that teachers expect to use MEAP in this way because this is the most useful service that the MEAP data in principle can provide. Or it could be that the tests that teachers have most contact with (teacher made tests, unit tests and textbook tests) provide diagnostic information, so teachers expect MEAP to serve this function also. I don't use the results at all. If they came in earlier that would help. Because you would have to look at them one-on-one and as a teacher I don't have much one-on-one time. One for each kid--I'd have to make out a separate lesson plan--I'm going to work on this with this kid, this with this kid, etc. Years when I have a student teacher I do a lot more of that. But I do it from the unit test. (Valoria, Interview 2 p. 25). Like Valoria, Jerri makes it clear early in the Interview that she does not see much of the MEAP results and does not use them diagnostically. I don't see a lot of the results right away. They are not used diagnostically for me" (Jerri, Interview 2, p. 26). It is interesting to note here that when asked how she uses MEAP Jerri raises the concept of diagnostic use of MEAP. As with Valoria, I see in Jerri's statement evidence that she thinks that she is expected to or should use MEAP for this purpose. Jerri states that she does not use them to ”. . . ask are they (the kids) low in this or high in this” (p. 26). 87 Rose (Interview 3) also says that she doesn't use MEAP results. "The MEAP results aren't helpful at the time they come back (late November).” By then she has decided what she plans to teach, has sized up her classroom and performed her own diagnostic testing with her students. I would hope that before the tests come back I would have some idea of where my curriculum plans are going for the year. It's not like I use MEAP scores to decide what to teach and when (Rose, pp. 31-32). The comments of each of these teachers tells me that there is a shared perception among the teachers interviewed that MEAP is not used and that it is not useful for the important diagnostic decisions that these teachers must make. These teachers expect more from MEAP than it delivers. But I found that teachers do use MEAP for some things. Ted, a 4th grade teacher with 22 years experience (Interview 4 ), felt that MEAP helped in giving a view of how his school and district compared with the rest of the state. He said, ”I get a handle on how they (his students) are doing compared with other kids across the state” (p. 50). In fact, Ted reported that he uses MEAP more than the Stanford results, and finds that he likes how the MEAP results are reported. I can get a handle on.how they are doing compared with other kids across the state. I look at the MEAP more than I do the SAT (Stanford Achievement Test). I think that it shows our strengths and the areas we need to focus on. . I look at what I can do as far as instruction goes to take a look at that part of the curriculum (p. 50). Ted describes looking at his instruction and the curriculum to address areas of weakness he identifies in his students' MEAP results. He also describes using MEAP as a bench-mark for his science 88 instructional planning. By looking at the MEAP science test Ted gleans what is expected of 5th graders in science. He uses this yardstick to look at his own curriculum and judge how well he prepares his students. Ted claims this has led to change in.what he teaches and "it's helped me focus on what am I going to do in science.” Ted went on to tell how MEAP results helped him to focus his thinking and planning for science teaching: I have found something interesting that happened this year and its getting me to look at my science curriculum; The fact that science (MEAP test) is at the 5th grade. Also I feel we need to look at science curriculum in Lansing. I need some direction there. I don't like what we are doing or not doing. I've found that it's not really teaching to the test--but what do they expect 5th graders to know? And then I take a look at if I'm doing anything to help my kids become prepared for that. Some changes have been made in the science curriculum for the better, because I know that they were going to be tested next year. But also because I think that's an area I think gets easily pushed aside and it shouldn't . . . . I haven't really looked at the test to say this is what I have to teach. But it has helped me to focus on what am I going to do for science education. What are the things that I think are important? What's the process or the content? (Ted, Interview 4, p. 51). Teachers' talk reveals that MEAP has had effects on their science teaching. Teachers report being motivated by the results on the science MEAP to look critically at their science teaching and to attend workshops and classes on science instruction. Science has changed 'cause of poor MEAP results a few years ago. . . . There are materials for science as a result of MEAP test results. . . . Nobody seemed to care about science till the MEAP science test. For a long time there was a push on what was tested (reading and math) (Katheryn, Focus Group 1, p. 6). 89 Jerri (Interview 2 ) describes an example of how her use of MEAP results led her to find ways to change her teaching. Jerri realized that students scored low on MEAP science magnetism items. . I didn't do a great deal with magnets before and then I noticed that we were down in magnets so I took an LCC (Lansing Community College) class in the AMES materials. So they gave us some materials on magnets, so now we have a center on magnets. Atmosphere (as in weather): I tried to look at the whole atmosphere a little more. . Measurement and geometry; I did a little reading this summer. . . . But this is only one part. MEAP to me isn't the all inclusive (Jerri, Interview 2, p. 28). As a result of her students' MEAP scores Jerri took a class in teaching science at the local community college and incorporated methods and materials from the class in her science instruction. Student performance on one set of objectives, magnetism, led Jerri to a remedy that may effect her total curriculum and instruction in science. I contrast this example that Jerri gave with her statement that she does not use MEAP diagnostically. This example helps me to understand for what purposes MEAP is valued, and how broad or narrow is her concept of diagnosis. Jerri's response to low attainment on magnetism items also stands as an example of two kinds of responses that teachers can make to missed items. (Setting aside for the moment the possibility that teachers may choose to do nothing about missed items.) One response to low MEAP performance leads teachers to seek wide-reaching remedies like major changes in instructional content or materials. Are such responses to a few items missed on a state assessment test appropriate? Teachers could argue, yes, because MEAP is only a representative sub-set of expectations in a complex subject area, so it is reasonable to 90 generalize from this set to conclude that other areas in the curriculum need to be addressed based on the poor performance on a few isolated objectives. This view is grounded in an acceptance of the objectives and the curriculum tested by MEAP. Alternately,.teachers could also argue that such far reaching responses to one set of items on one test is an over-reaction and leads to a curriculum constantly changing in reflexive reaction to shifts in test items and test performance. An alternative Jerri could have exercised was to find a remedy targeted toward the specific items missed, that is, magnets. Such a remedy would have limited the "cure" to the topic of need identified by the test and would have preserved the remainder of the science curriculum that may have been working adequately. The disadvantage of this solution is that it could lead to a patchwork of such “fixes" that could undermine the integrity of the overall curriculum. During the Interviews teachers offered many examples of their responses to missed items. Most of them fell into this second category of remedies that narrowly attack specific items. t a d t Teachers described their uses of MEAP results to include goal setting and confirmation that the teaching practices that they use work. Jerri says, ”If I see a low one, I zero in on it. What am I doing wrong?" On items where all students show mastery, she concludes, ”What I'm doing must be right" (Jerri, Interview 2, p. 27). Teachers viewed MEAP as a part of the ”total” or "big picture." MEAP, Jerri said, is '. . . one of the things in the total picture" (p. 91 27). The total picture analogy was repeated often throughout this Interview and others. By total picture Jerri means broad conclusions about school effectiveness. ”It's a total picture, my evaluation, my observation, my looking at SATs, my just kind of looking at the whole thing" (Jerri, Interview 2, p. 29). MEAP and other elements of the testing program are for Bill (principal at Pine Ridge) only one part, albeit an important part, of the profile of his school and students. A I always look at any testing program as just one piece or part of putting together a complete picture of the student and in looking at the complete (set of) students that will tell me what the school looks like. It's only one part but it does provide teaching strategies for all the teachers. We like to take a look at MEAP as being primarily lst, 2nd, 3rd and part of 4th grade levels. So there is some responsibility for all of these people. If you do well in it, there are some teaching implications, if you do poorly on it there are teaching implications. (Bill, Interview 6, p. 68). Bill goes on to describe the other elements that make up the profile of his school and students. You have to take a look at the child's classroom performance. (It should) be a reflection of his daily work. YOu'd.have to look at homework assignments. You also have to look at parental expectations because if this is not consistent with teacher expectations sometimes you have a problem. You have to look at readiness level of the child. That is one of the things that I think some of the teachers are missing and I try to preach that message all the time. Just because a child is in the 4th grade does not mean that he is going to be performing at the 4th grade level and if he is not, teach the child what he doesn't know whatever that is. Some kids may be below, some may be above. Wherever they are is where you have to teach them (Bill, Interview 6, p. 69). 92 Bill puts these elements together to paint a picture of the ”whole child." What is described here is a balanced view of MEAP as only one piece of the picture of a child's or a school's achievement. Teachers find the other sources of information about assessment to be fairly consistent with MEAP (Jerri, Interview 2). But if the consistency between measures reassures teachers that MEAP is assessing important content, it is also frustrating to the teachers who feel the testing is redundant with information that they receive from other sources, that are more timely, less time consuming and more easily interpreted than MEAP. 51mm I see in the data presented support for the idea that when asked to assess the usefulness of MEAP, teachers respond that they do not use it and do not find MEAP useful. But I also found that when teachers talk about MEAP and the school's curriculum or about MEAP items or other very specific topics they list a number of practical ways in which they use MEAP results. These uses however, do not include the teacher-expected functions of MEAP use for individual student diagnosis. But one may argue that the examples of MEAP use described by teachers as a result of low science scores, for instance, are also an important use of assessment data. 93 eac e r dr 0 Fourth grade teachers spend the first four weeks of school preparing students to take the MEAP test in reading and math. They resent having to spend instructional time this way. Valoria's efforts to help her students score well on MEAP include extensive review in the fall and focus on specific items she has found to be difficult for students. She describes the first few weeks of school in her classroom as settling the students into work and review. Rose If it weren't for MEAP I'd start a lot more relaxed and easy going and kind of warm up with fun things, active things. Instead, now I have to settle them down quickly . and remind them . . . how to add, subtract, and how to multiply. I have to do all this in the first 3 weeks of school. The first week of school we work, work, work or my test scores won't be good. The principal will be angry, and my school district will be angry. I think that is a lot of pressure on those poor little 4th graders (Valoria, Interview 1, p. 26). tries to make the best use of her time by keeping her review for general and focused on determining her student's skill level. I don't think that the results are the ”end all.” Rather than spend 4 weeks preparing for what might be on the MEAP--I don't keep a copy around; I refuse to teach to the test--I do a general review. I think it is more important to ascertain what these incoming kids do know and how they preform for me. I'm probably going to use thg; information more than sheet after sheet of statistics. I'm going to use what I know they can put down on paper and what they can read and show me and take it from there (Rose, Interview 3, pp. 47-48). W test . Teachers feel pressure to prepare their students to take the MEAP But as I listened to teachers it became clear that they did not know clearly what would be on the test. Teachers do not have access to 94 the test before it is given. This is a test security precaution. During these late spring interviews I showed the teachers a copy of the 4th grade MEAP reading and math test. Each of the teachers spontaneously leafed through the test booklet and made comments about the items. Valoria's experience with the MEAP test was typical of the teachers interviewed. I don't get to look at the test itself, unless I look over the students' shoulders. One of the problems that I have is trying to get (understand) the results. I don't understand one from the other. I don't understand what it means. . . . Some of the areas I don't even understand. My printout this time I thought was really confusing. (Valoria, Interview 1, p. 14). Reviewing the results with a copy of the test in hand could help teachers understand the results. Without access to the test that opportunity is not available to teachers. During the Interview Valoria paged through the test noting and commenting on specific items. In her view not making the test available to teachers was an effort to keep teachers from teaching to the test. ”We should have at least one (copy of the MEAP test). But I know that would cause people to teach to it" (p. 24). Jerri a third grade teacher (Interview 2) says ”I've never laid eyes on this” ( p. 29). She says this as she looks through the pages of the MEAP test her students took in the fall. (She has seen other editions over the years.) She has heard from her colleagues that it is long, and she opines that some of the items are too “tricky” for measures as basic skill attainment. 95 Jerri suggests, "Maybe they should give 3rd grade teachers an overview (of MEAP). That would probably be teaching to the test" (p. 30). Jerri raises the issue of teaching to the test here as reason for not sharing an overview of the test with the third grade teachers who are preparing students who will take the test the next year. d e u Rose (Interview 3) doesn't know what is on the 5th grade science MEAP, which presumably she prepares her 4th graders to take next year. She hears from the 5th grade teachers that it is hard. If there is some expectation that 4th grade teachers prepare students to take the 5th grade MEAP, Rose doesn't know what to prepare them for. It could be assumed that issue has been taken care of for her by the formal curriculum which was developed with MEAP and other tests in mind. But Rose says that she doesn't use the formal curriculum or text. She has no way to know how well she is preparing kids for performance on the MEAP science test except by examining the scores her former students will receive next year or by review of former years' scores. If we look back at the Ottawa School science scores we see that they are below the district average (47.8! compared to 551 district-wide in the highest category). The need for reform of curriculum and instruction is clear, but the means are absent. In the data presented here I do not see adequate understanding of the implied curriculum underlying MEAP for teachers to prepare their students to perform well. Teachers prepare for MEAP by teaching 96 test-taking skills and providing activities aimed at a few particularly difficult test items. Disgzig; and Teacher Activities are or MEAP Valoria describes the district's historical response to MEAP as being "really crazy.” The 1982 call by the superintendent to improve MEAP scores by 10 points led to celebrations, banners, breakfasts for staff of high scoring schools and a wide variety of activities to prepare students to take the MEAP test. Many of the preparatory activities dealt with the mechanics of testing. Teachers were inserviced on how to administer the test. Publications were prepared by the research and testing office to aid in preparing teachers to give the test. The ”MEAP Bird" was commissioned and students were taught test taking skills. Valoria felt that these activities prompted people to teach to the test. I know schools where the principal had the "MEAP Moment." They'd do one MEAP objective every single day. And they'd teach right to the test. And the kids did better. Well give me a break! Of course they did! . . . But does it really tell you how smart the kids are? Have you really improved that much? (Valoria, Interview 1, p. 19). As teachers describe their preparation for MEAP fourth grade teachers' talk focused on the four weeks of preparation for the test in the fall. Each teacher interviewed spontaneously raised her concern about teaching to the test. Sheila, a third grade teacher, talks about the difficulty of trying to 32; teach to the test. I gave a trial for the new math MEAP last year. Its very difficult. . . . We try not to teach to the test but if they are going to use graphs and calculators then you have to do some graphs in class (Sheila, Focus Group 2, p. 11). 97 As in many of the examples given by teachers, Sheila's definition of teaching to the test is very broad and precludes many appropriate instructional activities. M e aratio i to Fe ect v Jerri says that when she makes changes in what she teaches or how she teaches, it is not just for the test. Jerri explains how she may augment the math examples given in a textbook to promote a deeper understanding of a concept by her students. But it isn't just for the MEAP, I do it for the kids. It's one of the things in the total picture. I do not teach to the test. I would be doing most of this anyway, because they need to know how to tell time or subtract (Jerri, Interview 2, p. 27). What does it mean then to prepare for MEAP? Is preparation different for Jerri, a third grade teacher, than for the 4th grade teachers interviewed? The burden is on first through third grade teachers to introduce and teach the content of the 4th grade MEAP tests (although the major part of this burden rests with the third grade teacher). The only responsibility that can be reasonably assumed by 4th grade teachers--with only four weeks of school before the test is . given--is to prepare students for the logistical skills of taking the test (e.g., how to fill in the answer sheets), and to review skills that students were taught in lst through 3rd grades. Jerri leads me to conclude that preparation for MEAP is not as important as the many other things that she considers when planning instruction. One plausible approach to preparing for MEAP is to teach well and let the MEAP scores fall where they may. This seems a practical response considering that 98 Jerri doesn't get to see the test and hasn't seen the state curriculum on which it is based. Jerri must consider many factors in getting the school year off to a good start, including the social system of the classroom, the textbooks, other tests, and local.curriculum statements. Time-consuming explicit preparation for MEAP competes with a host of immediate priorities. W In separate interviews Valoria and Rose, both fourth grade teachers, also looked through the MEAP reading and math tests page by page and made several comments about what they did in response to specific items in the test. Both referred to a difficult set of math items that they refer to as the 100's chart. Valoria points to the set of three items and says, "My kids haven't seen a chart like this since they were in lst grade and learning to count” (p. 18). The set of items contains three 10 x 10 grids. The first two rows of cells in the grids are numbered from 1 to 20. The far right column of cells is numbered 10, 20. 30 . . . 100. The rest of the cells are blank. In each grid there are three or more contiguous boxes shaded and students are asked to identify the set of numbers that belong in the shaded area (see Figure 4.2). Most of them missed this because they haven't any idea what they want. . . That's really hard for the kids unless you do it in advance with the kids (Valoria, Interview 1, p. 18). 99 OBJECTIVE AID EXAMPLE ITII FOR FOURTH GRADE MITIIINTICI 16-2 Given a hundred chart with the first twenty numbers and multiples of ten filled in, the learner will write in any portion of the chart as requested.“ Which numbers belong in the shaded area? 11 12 13 14 15 16 17 18 19 20 30 70 100 A 29, 39, 49, 59, 69, 79, 89, 99, 109 3 911, 912, 913, 914, 915, 916, 917, 918, 919 C 91, 92, 93, 94, 95, 96, 97, 98, 99 D 901, 902, 903, 904, 905, 906, 907, 908, 909 * 'raken tr:- Michigan State Board of lducatim, Michigan lducation Assessnnt Progru handbook, 1990-91, p. 45. 100 Valoria has found a partial solution to her students' difficulty by posting a large chart in the fall in preparation for MEAP and covering squares randomly with magnets. Her students think this is ”baby work," but it has helped them to understand how to respond to the test items. Rose also cites the 100's chart as a difficult set of test items for her students. She feels, however, that to present something similar to her students for practice would be teaching to the test and, therefore, wrong. I watch the kids die on this page year after year. They just fall apart on this one. (Rhetorically . . . ) now is this a critical skill? I've always felt that I should not present them with something similar or that would be teaching to the test. And they haven't seen anything like this in their math books at least not in the 4th grade and evidently not in the lower grades either. They are thrown by it each time. So is this a math test or a reasoning test? (Rose, Interview 3, p. 36). I am interested in the contrast here between the way in which Valoria and Rose approach a set of items their students find difficult on MEAP. Valoria practices a similar activity with her students so they don't do poorly on an item that she views as confusing and tricky. Rose decides not to drill her students with an activity similar to the 100's chart because to present something similar would be teaching to the test. She also questions whether the skill tested by this item is critical. Rose's concept of teaching to the test limits the test preparation activities she is willing to use with her students. Is this example typical of the decisions that teachers make on behalf of their students? Is it coincidence that the MEAP scores of Rose's school are consistently lower than those of Valoria's? 101 V c d ow t e o to fficult It ms In my Interview with Valoria she spoke about two other specific examples of items she finds difficult for her students. Valoria finds the emphasis on multiplication on the math test to be out of sequence with Elm Street's curriculum. She feels that multiplication is tested too early. Multiplication by zero stumps her students. She believes this is because 3rd grade introduces multiplication but does not include zero facts. It (multiplication by zero) is extremely easy by the time you get through 4th grade. They just start getting their facts in the last month of the 3rd grade. It's kind of hard because they get all these problems on multiplication and they have only been exposed to the (non-zero number) facts. Studies show that they are testing things way too early. Multiplication facts shouldn't be at the third grade level. They shouldn't have to memorize until 5th or 6th grade, and then they wouldn't have any problem. They're shoving them down their throats in the 3rd and 4th grades (Valoria, Interview 1, p. 17). Valoria turns the pages of the booklet and comes across a test item on thermometer reading. She points out that the scale on the thermometer is in units of two (each mark representing two degrees). See, in our book these numbers don't stand for two, so the kids are always putting this as 63 and this would be 53 and they don't know what to do (Valoria, Interview 1, p. 19). In response to this difficulty, she has begun to use examples of thermometers with various scales as part of her instruction. Valoria says that her response to items testing thermometer reading and the 100's chart, along with teaching the words ”greater than” and "less than,” are ”the only ones I think that you have to teach to” (Valoria, Interview 1, p. 19). 102 It is interesting to contrast Valoria's response to student difficulty on these two math items (the zero facts and the 100's chart). To Valoria it seemed inappropriate, both from a cognitive readiness perspective and from concern that she would be teaching to the test to call for deeper coverage of multiplication facts, especially zero facts. However, posting of the chart and allowing students to practice identifying numbers under the covered squares was seen as an appropriate preparation for a difficult item. This contrast highlights the difficulty teachers wrestled with as they responded to MEAP results. Using the chart to clarify a confusing item seemed a response that would not violate the ethic against teaching to the test; perhaps because it was trivial, and had no long term effect on learning or pedagogy. However, to change the sequence or scope of introduction to multiplication facts to produce greater MEAP performance was viewed as teaching to the test and wrong; both from an ethical standpoint and pedagogically. Teacher understandings of the ethical and pedagogical meaning of teaching to the test tell a great deal about the impact of state assessment tests on curricular reform in local school districts. Teachers' views about what constitutes appropriate test preparation may confine the effect of MEAP on curriculum to tinkering rather than significant reform. Cur Mathematics. Much of the discussion by the teachers interviewed was about MEAP and its relationship to the curriculum. When considering curriculum in 103 schools one must consider both the formal and the enacted curriculum. The formal curriculum consists of curriculum statements, textbooks and other instructional materials. Other standardized tests, teacher preferences, inservice workshOps, time, tradition and many local factors influence the informal curriculum. Teachers gave mixed answers to how well MEAP matched the school district curriculum. It seems that from the teachers"point of view, the Lansing mathematics curriculum matches MEAP fairly well. This is not surprising because the MEAP math test is an edition that has been used for a number of years and is familiar to the teachers and curriculum specialists. There has been sufficient time for the local curriculum to be aligned to the MEAP test. MEAP is consistent with the mathematics textbook used in Lansing and the strong computational approach that Lansing schools have taken in mathematics. Lansing's MEAP mathematics test scores are above the state goal. Yet even with this strong alignment teachers find areas of concern in the math test. As noted above, Valoria finds the emphasis on multiplication on the math test to be out of sequence with Elm Street's curriculum. She believes that multiplication is tested too early. It is extremely easy by the time you get through 4th grade. They just start getting their facts in the last month of the 3rd grade. . . . Multiplication facts shouldn't be at the third grade level (Valoria, Interview 1, p. 17). Valoria said that she doesn't think that the test is bad, "I just don't see what you use it for." For Valoria, the math achievement information offered by from MEAP is not worth the time taken from her instruction. 104 Jerri cited examples of how she changed her instruction in measurement and geometry to improve areas identified as weak by MEAP results. Rose notes that geometry is a topic in which her students perform poorly. But she concludes that geometry is less important than other topics that she spends more time on, specifically whole number operations. I t Rose 3. . I," I know that my kids will be low in measurement and geometry. But to me those things may be less important than some other things (Rose, Interview 3, p. 32). described her students' performance on metric measurement in a similar way. She is concerned by the mixed messages she is receiving about metric measurement. test We always fall down in metric measurement. (Rhetorically) Is metric measurement the most important thing? This new math test that's being proposed--Did you read about it in the newspaper? (the new national math test recently piloted) It's all in inches and feet. We've never been allowed to have inch rulers in this building. These kids don't even know what this means (pointing to a test item) they have all been brought up on metric (Rose, Interview 3, p. 35-36). Considering the mis-match between the state and proposed national led Rose to ask who decides what the curriculum is. Who decides what goes on the state test? And do other states have tests that are similar to these? If so, who . would make up a national test and have it be at odds with the states'? Will we have to change our state test to go along with the national or would one be done away with? And then would we teach to the national test? (Rose, Interview 3, p. 36). These questions are critical to understanding teacher use of test results. They reflect the problems teachers experience with mis-match 105 between the various tests administered, between the test and enacted curriculum, and between the tests and textbooks. Katheryn, a teacher of third grade, is concerned that even when students do well on math tests she remains uncertain that they can make real life practical applications. I have a concern with math. You would like the kids to be able to do some real life applications if they do well on the test. There is talk about the new math MEAP; (teachers say) there will be more story problems. What can be done for the new math MEAP to help teachers? We may need to change the way we teach (Katheryn, Focus Group 1, p. 6). In four short months (three of which are summer vacation) Katheryn's students will take the new MEAP math test. Teachers of third graders seem to know about the test through what they hear about it from fourth grade teachers. The primary reasons for implementing the new MEAP math test are to test more appropriate math concepts than the former math test and to encourage a constructivist approach to teaching mathematics (Michigan Department of Education, 1990). Why then do teachers like Katheryn have to speculate that they may have to change how they teach? Sheila had the opportunity to pilot-test the new MEAP mathematics test to be used in the fall of 1991. She reported to the others that it is very difficult. Edith also talked about what she anticipated to be part of the new math test. There are more calculators on the test. We got some (calculators) from the PTA and the district bought some. We need base ten blocks, we just found that out a few weeks ago that it is on the test and needed (Edith, Focus Group 2, p. 11). 106 I am surprised that I do not hear teachers comment about how instruction will be changed to accommodate graphing, calculators, and base ten blocks. Perhaps these props fit smoothly into the math curriculum as it is taught at Redwood with no special thought needed for accommodation. BaLding The MEAP reading test and the Lansing reading curriculum are both relatively new: 1991 is the second year for each. When I got the results I was shocked. The students hadn't had the benefit of the change in instruction (the new reading approach). The kids were not used to the number of questions in a section. There were 46 questions in one reading section. It was intimidating. The results pointed out that using this as a measure needs some work. The results pointed out the need for some work (by teachers) in comprehension (Katheryn, Focus Group 1 pp. 1-2). Katheryn draws several implications from.her review of the test results. She reflected that there had not been enough time for the effects of the new reading program to show in students' test performance, and that her students need more instruction in reading comprehension. She found the test to be too long and intimidating for her students; and that the test design itself is flawed by excessively long reading selections, numerous questions and difficult format. As I sat with Valoria she looked through the test booklet. Several of the items cause her concern. Despite the relatively high scores of her students on the MEAP reading test and her earlier comments that she did not worry about her students' test scores because they perform high on MEAP, she says the informational reading material is hard for children. “I thought the expository (informational reading section) was 107 hard for the beginning 4th graders. It wouldn't have been for ending 4th graders" (Valoria, Interview 1, p. 15). This section of the test called on the children to read a long factual passage taken from a social studies textbook, and answer questions related to the passage. Valoria thought that the passage was from a 4th grade social studies textbook. At the time of the testing, students would not have been exposed to the longer passages with smaller type and fewer pictures that characterize the 4th grade social studies textbook. Valoria cited the fact that her school has a progressive librarian who is up to date on meta-cognitive approaches to reading as one reason that she doesn't worry about their MEAP reading scores. ”Our kids would do fine no matter the test" (Valoria, Interview 1, p. 14). She finds the new MEAP reading test, designed to reflect the meta-cognitive, whole language approach to reading more difficult to teach to. It's not an easy test to teach to anymore. With the other test (the old version of the MEAP reading test) there were more specific skills you could teach to improve your score. I think it (the new version of the test) is a better test. (Valoria, Interview 1, p. 14) It is not clear if she draws this conclusion because the new MEAP is no longer skill-oriented or because it is now more difficult to teach to the test. Valoria believes that her students lack the experience assumed by many of the questions on the test. Her students haven't been many places. "No one has been on a plane,“ she says, ”or can name a state park" (Valoria, Interview 1, p. 15). She is frustrated that the parents' and school's lack of money make it difficult to sponsor field 108 trips to compensate for this lack of experience. Rose and Bill also raise the issue of student lack of test-related experience. Science Science shows the largest difference between the curriculum underlying the MEAP test and what is taught in Lansing classrooms. Alignment is poorest here, and so are test scores. Ted credits the MEAP science test with getting him to look at his science curriculum. By looking at the MEAP science test he infers what is expected by the State Department of Education of 5th graders in science. He uses these inferences.to analyze his own curriculum to judge how well he prepares his students. This has led to some change in what he teaches and ”it's helped me focus on what am I going to do in science" (Ted, Interview 4, p. 52). Ted told how MEAP results helped him to focus his thinking and planning for science teaching. I have found something interesting that happened this year and its getting me to look at my science curriculum; The fact that science (MEAP test) is at the 5th grade. Also I feel we need to look at science curriculum in Lansing. I need some direction there. I don't like what we are doing or not doing. I've found that--its not really teaching to the test--but what do they expect 5th graders to know? And then I take a look at if I'm doing anything to help my kids become prepared for that. Some changes have been made in the science curriculum, for the better because I know that they were going to be tested next year. But also because I think that's an area I think gets easily pushed aside and it shouldn't. . . . I haven't really looked at the test to say this is what I have to teach. But it has helped me to focus on what am I going to do for science education. What are the things that I think are important? What's the process or the content? (Ted, Interview 4, pp. 51-52) Ted described drawing from a seminar on agriculture in the classroom, his interest in ecology, the environment and waste management 8., 109 and a science education program from the State Department of Education to develop his own science program. ”I don't know if I would have done that if I didn't feel pressure to do something. That's one way it's (MEAP) had an effect on what I do" (Ted, Interview 4, p. 52). Lynn looks at MEAP science results in another way, "On science I think that if they do well they have picked it up somewhere else, cause we don't do that much science" (Lynn, Focus Group 2, p. 10). This observation raises many questions about the enacted science curriculum at Redwood school and, taken with Ted's comments, it raises questions about the elementary science curriculum in the whole district. It is not clear from Lynn's statement whether the science textbook is thought to be inadequate to support the amount of science instruction desired by Lynn, or if the amount of time prescribed for science instruction is too little, or if teachers spend so much time covering high priority subject areas that little time remains for science instruction. But whatever the meaning, it suggests that elementary students' opportunity to learn science is compromised. Rose doesn't know what is on the 5th grade science MEAP, for which she, Ted, Lynn and the other 4th grade teachers in this study prepare students to take next year. Rose worries about her science teaching. Since she doesn't like the district-adopted science book she doesn't use it. Rather, like Ted, she has developed her own science curriculum, borrowing bits from here and there. In science I've never felt that the textbook was that good in the first place. I've kind of developed my own. Probably it doesn't hit right along with MEAP. I'm not - at .i 110 going to fit my whole science program to the 5th grade MEAP. To me that would be teaching to the test” (Rose, Interview 3, p. 33). I asked Bill, principal at Pine Ridge, if he thought the new science MEAP has had an impact on what is taught and emphasized in science. Bill was pessimistic. It has some implications. I can only speak about the teachers in this building. They are concerned about the science, but a lot of them are more concerned about their scores in reading and math. The teachers that I talk to are a little bit disenchanted with our Houghton Mifflin science program and they think that it is really not lending itself to our kids' science achievement. I also know this, we have far too many teachers, particularly at the elementary level, who do not enjoy teaching science. We've got to face that. That's why Dan Mathis (pseudonym for the school district's science curriculum specialist) has gone to a plan that more departmentalizes it. If you have a particular person on staff who has a proficiency in science, that person may end up teaching it to all the boys and girls at maybe two or three different grade levels . . . (Bill, Interview 6, pp. 74-75). i v-" I asked Bill if MEAP science testing was going to have an influence on what was taught in science. Bill replied, This school district is going to be hard pressed to make some impact there. For too many years, if you look at what our core curriculum consisted of, it was reading and math. And I still take a look and say if our reading scores or our math scores are at or exceeding our district's or the state's expectations we're OK. Science will just have to be pulled along. It would be nice to get those scores up, but if it doesn't happen I don't think anybody is going to be alarmed about it. Reading and math are different. They'll attack those. I don't think the testing in science will make too much difference (Bill, Interview 6, p. 75). Bill's views contrasted with those expressed by most of the teachers interviewed who now felt pressure to improve science as well as reading and math. We see that Ted has invested his time in constructing an informal science curriculum based on his interests, workshops, and 111 what he has gleaned from the MEAP science results. Jerri took a science workshop series designed for teachers. It is interesting to note that from the teachers' descriptions the science curriculum content of these workshops were quite different from one another. Rose developed her own science curriculum. And Lynn observes that at her school they don't teach much science. I am led to two thoughts. First, despite the fact a that teachers feel compelled to improve science teaching Ted, Jerri and Rose pursue improvement through three different curriculum paths. The , results are ”hit or miss” attacks on MEAP scores and will make it difficult for the school district overall to improve science scores. Second, to improve science teaching we will need to go beyond the hope that what gets tested gets taught. The teachers and principal all share the environment of the Lansing School District. They have some degree of shared experience by virtue of the formal curriculum and materials and directives to improve MEAP test scores. This section demonstrates how teachers and the principal, faced with similar circumstances, draw differing meanings from the environment and react to the environment in various ways. Even among the teachers who share a similar job role, there is a difference in how each perceives and approaches the problem of low science test scores. -w G r The teachers in this study ask whether MEAP is leading to a state- wide curriculum. ”It's going to lead to a state-wide curriculum," says Edith. "The textbooks may not emphasize what MEAP emphasizes, so even if you do a good job you may not hit the test (Edith, Focus Group 2, 112 p. 9). The thought of a statewide curriculum is not a happy one. Aieda finds some merit in a state-wide curriculum, but is concerned that with tests like MEAP driving curriculum, educators may loose sight of the other important purposes schooling serves. Schools aren't just here for academics. We are also here for socialization and it's been that way since Horace Mann. But we've kind of forgotten that and focus just on the academics (Aieda, Interview 5, p. 59). Aieda sees a portfolio approach to assessment as more satisfying and beneficial than multiple choice assessment instruments. i e a d e C owd Ted is concerned when tests dictate what happens in the classroom and dictate what is important. . It's not just math and science and reading that are important; it's how that fits into everything else. One of my concerns is that tests can dictate what happens in the classroom and dictate what is important. Every year something does not get done because of testing. I've felt it a lot this year. Everything is new right now; reading, science, math. I'm not as laid back as I used to be. I'm not as art-oriented as I used to be because this is what the state wants us to do. I feel intimidated by tests (Ted, Interview 4, p. 56). Ted teaches less art than he used to because of MEAP. Ted describes more and more additions to the already crowded curriculum. There is not enough time. It is interesting to note that despite Ted's earlier comments.that he is more relaxed with MEAP than in previous years he finds tests in general intimidating. I see new programs developed or instated with the rationale that this can be part of what you're doing now; this can be part of the writing or reading program. But it takes time. What is going to go? What is going to be not done? (Ted, Interview 4, p. 56). 113 In Ted's statements I heard several concerns; pressure to cover a wide-ranging curriculum, pressure to make changes in the way he teaches reading, math and science, and the pressure of time. The teachers said in many ways that the curriculum is over-crowded. They are expected to cover too much too fast. They reported that they have been unable to spend the time needed to develop concepts adequately. .Reviewing for MEAP in the fall adds to these feelings of time pressure and resentment. Lynn: So much is being pushed on the curriculum. Where is the time? Sheila: You have to leave concepts so quickly. You're constantly reviewing. You are lucky if you can put a few problems on the board for them to do as review. Lynn: It's unfair. Edith: The science curriculum committee has come to the conclusion that there is too much in the science curriculum. I hope that they carry through with reducing the content covered in the grade levels. Math needs to do it too. There is too mush. (Focus Group 2, p. 1) Teachers also concluded that there is too much testing. Doris: The test tells you stuff that you already know from daily work. Lynn I don't think that we would loose anything if there were no MEAP. Edith: Next week we give CMS (Curriculum Monitoring System), last week we did SAT (Stanford Achievement Test); it's the same stuff on all of them. CMS is even better because it's given at the beginning and end of the year, you can see growth. There is a lot of doubling up. I'm tired of tests this year. (Focus Group 2, p. 9) 114 Although Ted is most positive and straightforward in describing his use of MEAP results to shape his instruction he has some concerns. . . . I don't know if it's this year, or that I'm getting older, but I find myself overwhelmed with the testing that we do in Lansing this year. I know that we have done MEAP, SAT's, CMS but since spring break I've lost about 2 months of instruction because of test administration and I keep thinking "it's what we did last year but it didn't feel the same." I don't understand that. Of all the tests we take I seem to look at MEAP more than SAT. (Ted, Interview 4, p. 51) Teachers feel that the demands are too much; too much testing, too much to teach and too little time to accomplish it all. Teacher na e te Throughout the interviews the teachers said that they did not have opportunity to look at the MEAP test. As we talked they looked through the MEAP test booklet and made comments about specific items. They frequently looked for specific items that they knew were hard for their students and pointed these items out to me. The 100's chart was most frequently mentioned as a difficult item. Teachers also cited portions of the new reading test that they found difficult for students. I have questioned the test questions. I don't worry about it (the MEAP test) cause its a bad question. Remember the 1009 chart? . . . . They always ask us to review the MEAP but there are things like the 100s chart that we complain about but that they don't change. Remember the garage sale story? Someone must have thought it was funny but I thought it was inappropriate for the kids. (Edith, Focus Group 2, pp. 8. 12) Aieda looked through the test and skimmed the fictional story about Oink and Auntie Min. The drawings that accompany the story show that 115 Oink and Auntie Min are pigs. Aieda didn't feel that a story about fictional characters named Oink and Auntie Min would interest her students. They would view the story as too juvenile and not read it. (To) a lot of kids in the 4th grade .-. . someone named Oink and Auntie Min it's not real life to them. They don't live in sheltered little worlds where the sidewalks are all perfect. So they don't want to read about Oink and Auntie Min. So they don't (read the story). . . . I can't believe this! (flipping through the pages) My kids would get bored “ with this. They would read the questions and then--are there more?-- 46 questions about a story that is on the 2nd grade reading level? I know some kids that won't answer them. It's an insult to their intelligence when you're in the 4th grade. Even in 3rd grade, that stuff is OK occasionally, but they are more mature. They are ready to read stories about boys and girls and how they live; mysteries. They like fables. I can't wait to see the new MEAP. (Aieda, Interview 5, p. 60) Lynn and Edith reach conclusions similar to Aieda's. Lynn: All of these items are on one story?! Forty-six questions?! This is ridiculous! Edith: The kids get tired and just start marking them (randomly). (Focus Group 2, p. 12) The test scores of children may tell more about their stamina or interest in the task than about their reading ability. Aieda offers a perspective on the reading test that has not been stated by the other teachers interviewed. The others focused primarily on the length of the reading selection and the number of questions that had to be answered. Aieda's observation about the low interest level of the selection could also explain why some students have difficulty with this section of the test. For these students the scores may not be a reflection of their reading performance, but of their low interest in the material or their unwillingness to continue to comply with the demands of a long test that is of little interest. This is an example of how the insights and 116 observations of practicing teachers can inform test design, test use and the interpretations that can be drawn from the results. Could this notion lead to a hypothesis that the reading selections included in the test may not be of uniform interest for all students and that there may be some systematic ways in which interests differ, hence biasing the test? The effect of interest may be more pronounced on the new MEAP reading test than on the old with its shorter passages and fewer questions per passage. The reading test score may be a function of students' compliance with reading a long and boring passage and answering numerous items about it. This could systematically bias the test results in favor of those students whose compliance is high (i.e., are not independent, confident thinkers) and whose interests best match the reading selections. In this connection Valoria says, ”I thought the expository (informational reading section) was hard for the beginning 4th graders. It wouldn't have been for ending 4th graders” (Interview 1, p. 15). Jerri (3rd grade teacher) hasn't seen the MEAP reading test but has heard from her colleagues that it is very long, and she hears that some of the items are “tricky" as measures of basic skill attainment. She feels that tricky items are better suited for norm-referenced tests. Teachers share what they have learned from experience with MEAP, they talk about the MEAP items and their opinions of their students' responses to these items. As Jerri looked through the test, she stated that those items asking students to "choose the best answer'I are difficult. She worked on this with her students last year in preparation for the test. She H's. 'I 117 felt similarly about questions asking students if they "strongly agree, agree, etc." "It's too hard,” Jerri said. As we talked she wrote down the exact wording of these items for use with her class next year. Bill (a principal) is concerned when students don't meet the MEAP expectation of 751 of the objectives mastered. Looking at MEAP you are talking about minimal objectives. It blows my mind when a child can't achieve 75% mastery of those objectives. Because it really is a simple exam. (Bill, Interview 6, p. 69) Bill's conclusion that the test is simple contrasts with that of the teachers interviewed, who cited the long reading passages, numerous questions and confusing items as evidence that the test is difficult. It is not clear that Bill has looked at the most recent reading test, which departs from the notion of being a basic skills test and instead is called an essential skills test, or if he refers to the science MEAP that teachers view as not testing content covered in their classrooms or if his comments are directed toward the math MEAP that still carries the title basic skills test. Bill's reference to 751 mastery may mean that he is referring to the math test or to the former version of the reading test. WW Rose told an interesting story that taught her to put the MEAP reading test into perspective and caused her to look at the test items critically. Rose described a year when her students did not do well in response to the section asking the students to identify an author's purpose. The school staff met and spent hours discussing strategies to 118 improve student performance in this area. As a capstone experience for this unit the teachers invited a local professor and author to talk with the students. When the students asked how he came up with his ideas, how he developed his themes, what did he have in mind when he began to write . . . the author responded "I didn't have a message. I just got this crazy idea in my head" (p. 37). Rose concludes from this experience that the test items for author's purpose were ”artificial” and didn't reflect anything in the real world. Rose used this story to illustrate that some times the items on the test are not realistic or do not measure meaningful concepts. res u e W Teachers described numerous sources of pressure to have students score well on the MEAP. The fact that MEAP results are printed in the local newspaper with comparisons made between local school districts was a frequently mentioned source of pressure for teachers and one they felt was unfair. Rose says that comparisons of MEAP performance with other school districts lead to unfair comparisons and generalizations. I've heard of people who move to a school district rather than another based on MEAP results. I've personally spoken to a number of people in the Stonyville area (pseudonym for a small rural and suburban community where Rose and her family live) who have said that. And Stonyville has consistently had high-high MEAPs. But you can't compare Lansing's MEAPs with Stonyville's MEAPs. They have a stable population who have been there all their lives. Family stability and the importance of education may be entirely different there. . . . The average people try to rank apples and oranges and try to compare us with someplace 119 in the UP (Upper Peninsula of Michigan, made up largely of rural areas and small towns and villages). They can't do that. (Rose, Interview 3, p. 33-34) MEAP scores do not tell the whole story. Later I asked Rose what she would tell policy makers about MEAP. She returned to the newspaper comparisons made between school districts. I would tell them not to print up Class A and Class D school districts together (Class A and Class D distinguishes the size of school districts for pairing of their athletic teams. She uses this to distinguish between Lansing, a large Class A district and Stonyville, a small Class D district.) I think it is misleading. I've talked to 3 people this week who claim that they've moved to another district because they have the best MEAP scores over long term. But I'm not sure that means they have the very best schools. Its a small school and they can't offer a lot of the curriculum activity that they offer in a large school district. (Rose, Interview 3, p. 48) Lansing may offer greater opportunities for a comprehensive curriculum than smaller school districts. But MEAP results don't reflect that. Teachers felt unfairly compared with suburban districts and noted the high mobility rates for Lansing students, which they conceded varied even among Lansing schools. Lynn observes that, Kids transfer from school to school. They are not in the same school through the 5th grade. . . . Its unfair to be listed with Heatherwood, Grants Creek and other (wealthy suburban) districts. (Lynn, Focus Group 2, pp. 9, 12) The suburban school districts that Lynn mentions are much smaller than Lansing. Sheila adds that, "Even in Lansing there are so many schools and they are so different . . .' that comparisons are inappropriate. Aieda felt strong pressure to push her students to perform well on the MEAP. 120 They (test results) are what make or break you. . What do they publish in the paper? They publish my MEAP results or Stanford results. (Aieda, Interview 5, p. 65) Aieda was concerned that publishing test results in the newspaper leads to generalizations and assumptions about schools that may be harmful in light of current discussions about ”schools of choice.” At both the state and local level there has been much discussion about schools of choice. Versions of the plans being considered would allow parents to select any school for their child from among public schools within a school district. Some plans expand the territory open for schools of choice to entire counties. Katheryn, Karen and Dawn talked about making MEAP score comparisons with other school districts and among schools within the Lansing school district. We make sure to look at where our building is compared to other schools. . . . I wonder how we are doing compared to kids in other parts of the state and country? (Katheryn, Focus Group 1, pp. 3-4) Comparisons seem an important benchmark to Katheryn. To Dawn they are viewed as unfair. W - e sures Another source of pressure to score well on MEAP comes from principals and other administrators. Katheryn, Karen and Dawn described the process for the review of MEAP results at their schools. Our principal meets with grade level teams and with individual teachers to discuss MEAP results. . . . The principal highlights what we do well on in MEAP. We make sure to look at where our building is compared to other schools. The principal asks what he can do to help improve the scores; (can he get) materials, observation in your room, model a lesson, ”What help do you need?" He makes 121 sure that you know about the weaknesses and that he expects and supports improvement. (Katheryn, Focus Group 1, pp. 2-3) Bill's (Pine Ridge principal) response to MEAP test scores was different from Katheryn's and Dawn's. Katheryn seemed to view the focus on test results as fairly constructive. "MEAP has led to more talking about academics. If we're talking about academics it's good" (Katheryn, Focus Group 1, p. 4). Dawn expressed frustration with the emphasis on MEAP and other tests. Pressure is put on teachers to excel, but some kids just can't get it. What can you do? . . . When we get the results you still feel that we are getting beat over the head about the scores. . . . If the message from the principal is to support you I feel it is contradictory to want you to be as high as other schools in the district. (Focus Group 1, pp. 3, 4-5) Karen describes how the principal of her school handles the issue of presenting test results without demoralizing his teachers. Our's is a tricky situation. Our scores are usually low but you have to keep the morale up. The principal tries to word things in a positive fashion. We have had three principals and all have treated the results the same. The teachers get discouraged. Their attitudes come out as . 'what can we do they are so low to begin with'. The teachers are less and less interested in test scores. (Karen, Focus Group 1, p. 3) As I showed earlier, Ottawa School has low MEAP scores. I wonder if Karen's observation is similar to Yeh et a1. (1978): that those with higher scores are more inclined to use test results than those with lower results. Do teachers become so frustrated with continued failure that they choose not to focus their energy on the test but on the overall performance of the child in the areas they see as important, or on things that they think that they have some hope of changing? Al 122 s d a es Aieda's experience with her principal and MEAP was different than experiences reported by Karen and Dawn. She (the principal) is someone who guides you. Adrienne (the principal) nevg; says ”Your MEAP results were low, you need to . . .” She will come in and say ”at Deerfield we are trying to. . . . Have you ever tried this? Have you tried that?" She's an instigator. Just like a teacher is an instigator. She is my teacher. . . . I would be disappointed if she were to say ”your test results are low, I expect to see improvement.” . . . I don't think that they (the test scores) are real important to Adrienne. It (the test results) shows something but it doesn't show everything to her. (Aieda, Interview 5, p. 64) Aieda went on to say that in spite of her principal's low key approach the teachers at Deerfield are very anxious and concerned about test results. Aieda said that teachers talk about test results in staff meetings. They say, ”That's a good teacher: you should see her test results" (Interview 5, p. 65). Aieda also cited the attention teachers give to review and preparation for testing as evidence that teachers at her school are very concerned about testing. "This schoOl is very concerned about test results. It's very important to them” (Interview 5, pp. 62-63). The principal seems to make an effort to not add to the pressure of testing, yet a school culture of pressure related to standardized test performance exists. Despite Aieda's commitment to pay attention to the non-academic aspects of schooling, her concern with the MEAP test itself and concern for the appropriateness of tests as a fair way to document effective teaching and student growth, Aieda felt strong pressure to perform well on the MEAP. They (test results) are what make or break you. . What do they publish in the paper? . . . they publish my MEAP results or Stanford results (Aieda, Interview 5, p. 65). 123 Jerri, who teaches 3rd grade at Deerfield, doesn't find the principal's role to be key in shaping the teachers' use of MEAP results. When we hear about MEAP its when evaluation services comes out and (with our involvement in) effective schools this year we looked at MEAP and did some disaggregation (of the test results by race and gender). (Jerri, Interview 2, p. 29) I asked if the principal shaped teacher use of MEAP; and Jerri answered, ”No.” This indicates that the pressure that teachers feel related to MEAP does not primarily originate from the principal. In Interview 5 Aieda (who also teaches at Deerfield), reported that the principal tries to dispel some of the pressure of testing. Yet we see in Jerri's words that pressure still remains. When I asked Bill, the principal of Pine Ridge, if teachers at his school feel pressure to do well, he responds emphatically: Oh yes. There is pressure put on the principal to do well too. There is pressure on central administrators to make sure that we do well. I'm sure that the pressure was passed on to them by the Board of Education. There is a domino effect. (Bill, Interview 6, p. 70) Unlike principal Adrienne who tries to minimize the feeling of pressure from tests, Bill seems to try to keep the test performance prominent in his teachers' thinking. Valoria talks about the pressure she faces at the beginning of the school year. If it weren't for MEAP I'd start a lot more relaxed and easy going and kind of warm up with fun things, active things. Instead, now I have to settle them down quickly . and remind them . . . how to add, subtract, and how to multiply. I have to do all this in the first 3 weeks of school. The first week of school we work, work, work or my test scores won't be good. The Principal will be angry, and my school district will be angry. I think that is a lot of pressure on those poor little 4th graders. (Valoria, Interview 1, p. 26) 124 Teachers did identify a positive side to the pressure that they feel to improve test scores. Science has changed 'cause of poor MEAP results a few years ago. . . . There are materials for science as a result of MEAP test results. . . . Nobody seemed_to care about science till the MEAP science test. For a long time there was a push on what was tested (reading and math). (Katheryn, Focus Group 1, pp. 4, 6) Low scores in science, in the view of all the teachers, led to greater " PJ‘H - I'l attention to and emphasis on science instruction. Dawn made an interesting observation. She believes that in some school districts, parents' high levels of expectation puts pressure on the schools to excel. In other districts, . . . The parents will push the curriculum, but if you took away MEAP in Lansing where would the push be? Would we be increasing our science, the new reading, etc., or would there be more affective?. (Dawn, Focus Group 1, p. 5) In Lansing, where few parents take an active role in education, where would the impetus for more rigorous curriculum come from if not from publicizing MEAP results? This perspective casts pressure as beneficial to some schools. Ted credited the MEAP science test with getting him to evaluate his science curriculum. By looking at the MEAP science test he inferred what is expected of 5th graders in science. He used this yardstick to look at his own curriculum to see if he prepares his students. Its led to some change in what he teaches, and Ted says, ”It's helped me focus on what am I going to do in science” (p. 52). Ted told how MEAP results helped him to focus his thinking and planning for science teaching. 125 Some changes have been made in the science curriculum, for the better, because I know that they were going to be tested next year . . . I don't know if I would have done that if I didn't feel pressure to do something. That's one way it's (MEAP) had an effect on what I do. (Ted, Interview 4, pp. 51-52) ' to ° Re ea ' w c Bill, the principal at Pine Ridge, told a story to illustrate that MEAP has brought some changes in his school. Analysis of the fall's MEAP scores showed a dramatic drop in reading and math from last year. He asked the school district evaluation staff to assist him in investigating further. Subsequent analysis showed inconsistency between students' prior test result patterns and current 4th grade MEAP scores. The students who exhibited the drop in scores were from a particular third grade classroom, and follow-up with spring Stanford and CMS testing bore out a pattern of decline in test scores. When these youngsters were in the 2nd grade a lot of them were in the 80-90 percentile on SAT. There should be some kind of correlation between high SAT scores and achieving 752 mastery on MEAP, but once they take the test they drop off dramatically. We were following those kids. And it's in one teacher's classroom. And as a result of that I have had a decision to make to allow that to continue or to move the person to another grade. I made the decision to move the person to another grade. . . . For me to allow that to continue would have been total irresponsibility on my behalf. . . . This building since I've been.here has always scored high in MEAP. . . . We went from 851 of the objectives in math to 671. That's too much of a drop off. . It's right here staring me in the face. You try not to use one piece, you have to take a look at all of it, that's why I had to wait to see the SAT results and now the CMS results: (they show) the same thing. (Bill, Interview 6, pp. 73-74) I did not hear from Bill the other things that he has done to assist this teacher to improve her instruction or whether school 126 district resources have been brought to bear to assist the teacher. I didn't hear any of the background information that led to placing the teacher at this grade level in the first place. Bill did not say whether be anticipated that the teacher would be more successful at the new grade level. I would need to know more about these issues to draw a more complete picture of use of the tests results. Considering only the information that Bill has offered, it is worth mentioning that the act of assigning a teacher to a different grade level may solve one test score problem but also create other learning and performance problems for the students of that teacher. Since all Lansing students are tested by Stanford and CMS at every grade level every year, scores on these tests may well continue to drop for the teacher's students at the new grade level also. Bill's story shows that he felt pressure to respond to dropping MEAP scores, how he discovered that the problem was attributable to one teacher and class by verifying the drop in student performance using other test data, and how he chose to make management and staffing decisions to improve (he hopes) student performance as measured by MEAP. CONTRASTING SCHOOL RESPONSES TO MEAP RESULTS I contrasted two sets of MEAP results, the 1989 and 1990 scores, comparing teachers' and principals' responses to changes in those scores. The question that I want to pursue here is how those interviewed make sense of and respond to change in their students' MEAP scores over time. 127 Contrast ° Tea e a 's Pers ective o w 0 es During Focus Group 1 Katheryn, a fourth grade teacher at Pine Ridge, said that she had been shocked by her students' MEAP performance. She concluded that it would take more than a year for the new reading program to positively effect MEAP reading scores. Interview 6 shows that, looking at the same results as Katheryn, a drop in the MEAP reading and math results for Pine Ridge caused Bill to investigate further. He gathered confirming evidence via other test scores and as a ‘result changed a teacher's grade level teaching assignment. The reading and math scores for that school dropped; science scores went up in 1990. In response to an Interview question regarding the likelihood of low science scores propelling the school district toward improved science instruction, the principal made a statement that sheds light on the interpretations that he brings to MEAP science results. Bill felt that the school district culture valued achievement in reading and mathematics more than achievement in science. Although higher science scores are desirable no one will be really upset over low science scores. We see in this example difference in the conclusions that Bill, the principal, and Katheryn, the fourth grade teacher, draw from the same test scores. on s ' lti a s ect v s In Elm School scores also dropped in reading from 511 to 31.6%, a .20.l.point drop in the percent of students performing in the highest category. For Pine Ridge the loss was 13.1 points. The Interview with 128 Valoria, the 4th grade teacher at Elm (Interview 1), did not reflect a concern for the decline in scores. She attributed the change to a slower group of students entering 4th grade this year. The school's mathematics and reading scores were good and even the decline in reading left the school with a reading score well above the district average. The principal may have viewed this decline differently and may bring another explanation to this score. But the overall fact that this school does well seems to temper any concern about a one year drop in scores . e C t twe w 8 Consider now Lenewee and Ottawa schools' MEAP results. Table 3.3 shows similar scores in reading, mathematics and science for these two schools. In her Interview Rose talked about social and environmental factors effecting children's achievement. Two implied factors were poverty and mobility (some argue that in urban schools these two factors are closely related). Aid to Families with Dependent Children (AFDC) rate is an often used measure of poverty level of a school. The AFDC rate for Ottawa School is 29.1% compared to a rate of 40.4% for Lenewee. The district-wide range is 54.82 to 17.32. Ottawa and Lenewee rank 15th and 5th respectively. Lenewee's poverty rate was much higher than Ottawa's, yet Ted did not refer to poverty related factors in his Interview. In fact, Ted makes virtually no reference to factors outside his own instruction, curriculum and materials as influences on students' MEAP performance. CHAPTER FIVE LESSONS LEARNED AND THEIR IMPLICATIONS mm As I began this study I wanted to learn about teacher use of state assessment data. I wanted to learn from teachers themselves, in their own words. I asked the teachers of the school district that I know best, the Lansing School District. To their words I added the local context (e.g., Lansing's site based management involvement, the "MEAP Bird", school demographic data) to situate these narratives for the reader. Through this process of asking and listening, reflecting and rethinking I have learned the lessons described in this chapter. W Much of what I have learned was not conspicuous as I framed my questions and started this study. I have learned about big issues like teaching to the test and the relationship of tests to the curriculum. I have learned about how innovations in schools sometimes mis-match one another and the resources available to teachers. I have learned that sometimes we say that we intend to empower teachers while we strip them of control of the most essential element of their profession, instruction. I have heard teachers say without hesitation that they do 129 130 not use MEAP and I have heard them describe a half dozen ways MEAP shapes what teachers do in classrooms. These lessons will be described in the pages that follow. LESSONS LEARNED ABOUT CURRICULUM State assessment programs test curriculum that has been judged to be worthwhile and, therefore, is expected to be taught in schools. Local curricula, textbooks, instructional materials and teacher instructional practices are the vehicles for conveying curriculum to students. As teachers talked about MEAP they had much to say about the connection between testing and the curriculum. WW Win; The most serious consequence of teacher concern about teaching to the test is that the concept as understood or misunderstood by Lansing teachers precludes any meaningful use of MEAP data. If a goal of state assessment testing is to provide guidance to curriculum and instruction in schools then it seems that two conditions are requisite. First good curriculum and good instruction must be defined. The State Department of Education has expended much time of its staff and of educators across Michigan on this matter. The results are the Essential Skills Statements for reading, science and now mathematics published by the MDE. Second, information must be shared widely with teachers and principals. This step, I submit, has not happened. There is virtually no understanding of what the state's new expectations are for 131 mathematics education, aside from the use of manipulatives and calculators. Such information gives a shallow understanding of the curriculum changes the state expects and may doom the new mathematics to failure. As early reports from the California_Mathematics Frameworks studies show (Educational Evaluation and Policy Analysis, 1990), teachers with superficial understandings of curriculum will adapt and reshape the innovation to conform to what they know (the old ways). Teachers will inadvertently overlook or omit critical elements of the innovation and construct their own, perhaps combining incompatible practices. The result is teachers frustrated that test scores are low and no one seems to know how to improve them. The experience with reading and science MEAP tests show the same flaw in state and local teacher preparation. I feel sympathy for those teachers taking workshops and classes and trying to patch together a science program, hoping that it will improve their MEAP science scores. The new reading and mathematics and science curriculums are bigger than the issues being addressed by hands-on science or cooperative learning workshops. They are grounded in a different way of thinking about how one learns these disciplines and what it means to know. They look at the content of the discipline in new ways and focus the definition of teaching accordingly. MEAP results will only have meaning for teachers when set against a well-defined curriculum. Teachers don't see the curriculum in which MEAP is rooted. They are hesitant to teach the concepts that they have gleaned from the test because they believe that would be teaching to the test. These concepts and activities are never fit together for teachers into a whole that is part of the state core curriculum or essential 132 skills or articulated with textbooks. No wonder MEAP doesn't tell teachers anything. Standardized testing in Michigan and in Lansing have followed a pattern of proliferation described by Mathison (1991). Mathison cites Madaus' view that assessment is a means . . . by which policy initiatives are implemented. Testing has spread to all levels of schooling and to most subjects, serving a multiplicity of purposes-- accountability, student assessment, student certification, and program evaluation. (p. 201) When proliferation in testing is considered with teacher and principal statements that they are uncertain what purpOse all this testing serves, one must ask if the multiple purposes are clearly articulated at the policy maker level, and if those purposes are clearly communicated to others. The evidence presented here suggests that they are not. Mathison (1991) describes the logic propelling state mandated testing programs. The logic is simple: Testing certain content in certain ways will results in an alignment of classroom practices with the official view of what and how subjects should be taught--what Popham calls a ”curriculum magnet.” (p. 201) But the teachers and principal ask who decides. Each teacher described scenarios where they viewed themselves as appropriately the primary determiner of what was taught and how. They were the ones who made daily assessments of student needs and developed or combined materials and methods to meet what they viewed as the appropriate curriculum. 133 W Which curriculum? In Lansing, curriculum is tested by MEAP, the Stanford Achievement Test and the local Curriculum Monitoring System (CMS). Clearly, another strong statement of curriculum is explicitly made by the textbooks used. Although the Lansing School District adopts the textbooks that best match its view of appropriate curriculum, once adopted the district does not have the resources to mix and match across several texts. For example, the 5th grade math text may present most areas well but do a less than adequate job in the areas of metrics, geometry and probability, or omit these areas altogether. If this is the case these areas are less likely to become part of the implemented curriculum: what is actually taught in the classroom. Lansing carefully articulates curriculum statements for each subject area and grade level. They are reviewed on a five year cycle. Major investment of classroom teacher and subject area specialists' time is involved. These statements form the framework for the report card design and are sent to parents. In Michigan MEAP is viewed by some as a potential determinant of what is taught in schools. The State Department of Education recently adopted a State Core Curriculum that every district in Michigan is to adopt or identify and report local variations from this curriculum. This hodgepodge of curriculum is not unique to Michigan and contributes to the sense of lack of direction and focus in schools. If testing can drive curriculum and textbooks drive curriculum, and specific curriculum outcome statements drive curriculum where is the Issac-“'7" 134 synthesis that puts this all together in a whole cloth that makes sense to teachers who must implement a curriculum? The influence of MEAP on the informal curriculum is mediated by the extent that teachers judge that it is ethical and permissible to include concepts from a test in the curriculum of the classroom. That in turn is determined by the extent to which the teachers see MEAP as representing a curriculum that they choose to adopt and one that offers something better than the curriculum that they have available in the school district formal curriculum or in textbooks. We see this in the teachers' use of the science MEAP to guide their science instruction and indicate areas where professional development was needed. This, however, requires that the teachers know the implicit curriculum underlying the MEAP test, and at the very least know the objectives ' being tested by MEAP. uen e Should MEAP have even such a strong potential to influence curriculum and practice in Michigan schools? Aside from policy and philosophical concerns about the appropriate level in the system at which school curriculum ought to be articulated, one must ask if in MEAP we have seen sufficient depth, breadth and specificity to drive the entire curriculum for the Michigan schools. Not only must a curriculum be pedagogically and developmentally sound and content accurate, it must also be supported by instructional materials, staff training and resources of time and class configurations. We don't see this in MEAP. Bill (the principal) was concerned that sufficient training of teachers 135 had not preceded the new reading program adopted by schools as a response to the new MEAP reading test. Bill and Rose saw the new reading curriculum practiced in Lansing as inadequate to accommodate children with little prior knowledge about the content of the reading material, or limited experiences or with social or developmental lags. Some students had too little for teachers to build upon; these students fell behind. Some of the teachers felt strongly that elements of the traditional basal skill-based curriculum need to be retained. Teachers argue for retention of a skill based approach from a curriculum standpoint, but their need for information for understanding and remedying the needs of students suggests a need for skill based approach from an assessment standpoint as well. It could be the case that neither view of reading (skill based or whole language approach) is sufficient to describe, guide or shape teachers' day-to-day practice. W This study demonstrates that MEAP has a de facto role in defining the curriculum in Lansing schools. This claim is best supported by the response to the new science MEAP test and the newer version of the reading test. A new math MEAP test has been developed and was used in the fall of 1991 (after the data collection for this study). In science we are beginning to see mixed evidence regarding MEAP's influence on the development of curriculum as individual teachers and schools develop their own enacted science curriculum in response to their low MEAP scores. Some teachers deliberately considered the content of the MEAP test as they developed their science curriculum, others were unaware of 136 the content of the MEAP science test and were guided by other factors as they developed their science activities. In reading we also have a relatively new MEAP test that, along with other factors, is shifting reading instruction to a meta-cognitive approach. In math the old test focused on isolated skill development. Only two teachers in this study mentioned preparation for the new math MEAP test. e e est Wa Some time ago Philip Jackson posed the notion that there is no one best way to teach. That notion applies to curriculum and its assessment as well. It could be that part of the tension we see in teacher use of MEAP comes from their resistance to viewing any curriculum as the one best way. Appreciation of this principled resistance to imposition of monolithic curriculum is critical to understanding teacher construction of enacted curriculum and to the variety of messages teachers take from and uses they make of MEAP. I was recently struck by the relevance of John Goodlad's words in this connection. Ours is a much-schooled society. The large percentage of children, youths and young adults in the formal system of elementary, secondary and higher education tells us only part of the story. In most parts of the country, courses in almost anything and everything are close at hand. And if one cannot or prefers not to enroll in a course, there are books magazines, cassettes and tapes on virtually anything one wants to learn. (Goodlad, 1984, p. 130) Here lies some of the uneasiness that some educators find in the new curricula as they understand them. Yes, schooling and both formal and informal education are all around us. But for many taking a class, 137 reading a book or even listening to a how-to tape would be the last way they would seek to solve a problem or to figure out how to do something new. For these people the new curricula may not fit very well. If children are not disposed by prior experience, modeling or other learning to unlock the hidden curriculum of schools they are at a disadvantage in learning in the new ways we are defining literacy and instruction in reading, math and science. This conclusion is not an indictment of the meta-cognitive approach. In fact, at its best, it can make mysteries of learning visible and accessible to those who have traditionally had the least school success. Unfortunately, that is not what teachers in this study see happening in their schools now. So teachers with a weak grasp of the new expectations for their teaching are frustrated when their students seem even more lost; the students haven't had the experience, the children don't even know what a castle is, they only read at a first grade level, and the MEAP scores reflect it. The culprit here is not the test, or the curriculum or the students or the teachers; the culprit may be how all this is communicated to and understood (or misunderstood) by teachers and their administrator supporters. The consensus is weak concerning what it means to teach and learn, and which learning is essential, and what uses of assessment data make sense. MW Teachers and the principal raised concern about state control of the local curriculum when they asked, Who writes the test? They advocate that classroom teachers_be directly involved in the writing of the test 138 and thus shift the control to those most sensitive to life in classrooms. A shift in the test from a minimum skills test to a more comprehensive competency test was instituted at the state level and is not clearly realized by the practitioners who teach children, administer MEAP and interpret the results. The state has used MEAP performance to sanction schools and districts and to reward them through funding. Teachers asked if MEAP is leading to a state curriculum. The comments of most of the teachers participating in this study suggest that would not be a desirable occurrence. 8 o P a tice nd Cu r c Valoria's perception that the change of the starting date of the school year was made by local school districts to improve MEAP scores suggests distortion of the social process of schooling to maximize performance on the test. Striving to improve scores on the test may by definition change or corrupt the phenomenon being measured. By necessity the measure draws more attention to and makes it possible for others to put on more pressure to change those things that most directly and immediately improve performance on the MEAP. This study documents this phenomenon in the teachers' responses to the 100's chart and the extensive MEAP review practiced in the fall of the fourth grade. Each teacher's response to preparing students for higher MEAP test scores, while having the potential to improve test performance, may distort the curriculum as well. 139 e e ea u It is difficult to determine whether teachers believe that important decisions are made as a result of MEAP results. One teacher expressed concern about teachers being evaluated based on the MEAP. Clearly all were aware that the test results were of some interest to the public at large, and to parents of school children in particular. For one teacher the principal's decision to change her teaching assignment was directly related to declining test results. _Whether or not all teachers in this study identified important decisions to be made on the basis of the test results, they all described their concern about teaching to the test. There was some difference in defining what teaching to the test is (what is permissible or appropriate, and what is not). All cited examples they had witnessed that they would consider to be teaching to the test (and thus inappropriate). The example of increased attention toward science curriculum reform and workshops on science teaching due to MEAP science results is perhaps a laudable outcome for the student and teacher time and money expended on.MEAP testing. Although one may caution that if real change is to be made in science instruction then the context or culture of the school and district must support this emphasis through allocation of time, resources and demonstrated by what is rewarded and sanctioned. A contrasting image is the teachers' response to the 100's chart. Classroom instructional time is spent teaching children how to address this item. One may argue that from a mathematics perspective it does not further the students' understanding of mathematics and that indeed if the concept of pattern or number sequence is truly the concept :‘l’.flus "' 140 needed, it can be taught more effectively using other strategies and targeted toward those students who need this skill. Has MEAP, at least in the instance of this item, distorted what is taught in mathematics in our classrooms? Perhaps several minutes of instruction about the 100's chart is not worth a philosophical battle when the tradeoff is better MEAP scores and less pressure. Teaching the 100's chart is a reasonable and practical response by teachers who have no other options. (They feel disenfranchised from input on the construction of the test itself.) VHF!“ But someone needs to raise the larger question: could our schools' curriculum become flooded with these practical responses to tests, thus drowning out the other curriculum content (art, music, physical education, social studies, etc.) in favor of disjoint I'skills" needed to do better on a test? Ideally achievement tests are designed to test student performance on a stated curriculum. The dilemma here is that since there is no one curriculum for schools in Michigan, standardized tests, including MEAP, must be designed based on some consensus view of curriculum. This consensus view is not communicated or widely known (only constructed for the purpose of the test). Teachers then must ferret out the curriculum from the test items and results (they can't see the items for fear of compromising test security), and teach to it. Not only for MEAP, but Stanford and the textbooks as well. This iterative process of shaking up the curriculum to improve performance on the ever moving target of standardized tests is a perplexing way to run schools and classrooms. Overlay that with the view held by some that the professional nature of teachers and the notion of academic freedom leaves the individual 141 teacher as the arbiter of what occurs in his or her classroom. This puzzle doesn't fit together. In describing a similar positive response by New York teachers to science testing's role in moving science instruction toward more science and a more hands-on approach, Mathison (1991) speculates, . that in only a few years, when the test has become the curriculum, teachers will realize that their authority to make curricular decisions has been seriously limited. In the face of much talk about teacher professionalism, these teachers will see their professional judgment eroded by the testing process. (Mathison, 1991, p. 208) Lansing teachers have begun to raise this question now. LESSONS LEARNED ABOUT THE MEANING OF MEAP The following set of lessons taught me about the ways that teachers describe their relationship to the MEAP test and its results. Teachers talked about the pressure that they feel to have their students score well on the test. They also talked about the importanCe of the test in their lives as teachers. The teachers were able to share with me their thoughts about putting MEAP, as well as other standardized tests, into perspective. Whales Corbett (1990) offers an interesting topology upon which to map the findings of this study. Corbett uses Merton's (1968) topology of individual adaptation to societal expectation to develop the following assumptions about policy implementation: 1) school districts respond variably to policy initiatives in terms of how they implement the policy and 142 the policy's effect on school district and building operation, and 2) the variability of local response is largely the product of differences in the district's organizational, political, economic, and cultural contexts. To focus solely on whether a policy's intentions are achieved without also assessing the nature of such locally influential factors as communication patterns (organization) resource availability and allocation (environment) . . . can yield misleading conclusions about the policy's effectiveness and misguided attempts to alter it. (p. 204) For Corbett political factors are the amount of pressure put on educators to improve student performance on minimum competency testing. Cultural factors are the local perception of the importance of the minimum competency testing results as inputs into significant decisions as key determinants of the types of responses districts make to a state mandate to implement its testing program. These factors can be used in conjunction with any number of sources of information. In this study I use them as one way of describing use of assessment data. Corbett uses district as the unit of analysis--this study moves a step inward to the teacher as unit of analysis. Corbett concludes that, . . the conceptual framework seemed to be a useful topology of district responses to mandated statewide testing programs. It allows for the highlighting of important qualitative differences in ways the districts, or parts of districts, approached implementing the tests. Some systems embraced the purpose of the testing programs and acted on that goal appropriately (conformity). While other systems did not accept the program but behaved as if they did (ritualism); and some systems accepted the program's goal but invented new ways of acting in accordance with it (innovation), while other systems substituted a new goal for the program and then devised ways to reach the new goal (rebellion). (p. 214) Application of this topology to this study's teacher level analysis proves a useful way of describing how teachers view the climate within which they must use MEAP. 143 Table 5 . 1 Local Conditions and District Responses to Testing PRESSURE TO IMPROVE PERFORMANCE HIGH LOW HIGH Innovation Conformity (Rebellion) STAKES LOW Ritualism Retreatism Source: Corbett, 1991; p. 207. Throughout this study I have been concerned that teachers not be blamed for factors beyond their control. We have seen in this study that MEAP testing design precludes teachers from seeing the test. Teachers describe having little knowledge of what the tests purport to measure. They feel constrained by lack of time and materials and lack of coherent and well-defined curricula that reflect the expectations of MEAP tests and that give a clear call to action. To continue to focus on educators as the solution to problems generated by the testing policy is akin to blaming students for failing to learn from bad teaching. Whether a state can devise a testing program that engenders local conditions more conductive to reform is questionable. Any attempt to influence local behavior through the imposition of state standards related to student learning outcomes is likely to be unsuccessful, primarily because such standards do not indicate what actions are needed to improve the situation. (Corbett, 1990, p. 214) 144 In this study we see teacher frustration with MEAP for exactly the reasons that Corbett describes. Teachers are at a loss as to what to do to remedy low scores. I or ance o o a h There are mixed findings as to whether Lansing teachers view MEAP results to be important. The formal ceremonial trappings surrounding the test suggest strongly that it is important. The results are published in the newspaper, with comparisons made to other school districts, presentations are made to the Board of Education, and Evaluation Office staff conduct meetings with each schools' staff every year to review the test results. Administrators are assigned to come to the schools to help with the testing. Yet one teacher reports not even looking at the results until a meeting much later in the year. Many teachers interviewed felt unclear as to the purpose MEAP serves or what theresults mean. All expressed feelings of pressure to do something. Fourth grade teachers felt compelled to use the first weeks of the school year to prepare their students to take the test. That pressure to do something seemed directed at review or additions to the curriculum for the entire class prior to the test. There were no teacher references to remediation for individuals or topics after review of the results of the test. c i o e v The teachers interviewed do not describe MEAP as effecting student life choices. On the one hand they say that it is important that the 145 students do well, but on the other hand their comments suggest that good MEAP performance is more for the benefit of the school or district as a whole than it is for direct benefit to the students. There have been efforts by the state legislature to tie high school graduation or endorsement of diplomas to performance on the 10th grade MEAP reading and math tests, but these efforts are not widely known or of concern to elementary teachers and were adopted by the legislature after the Focus Groups and Interviews took place. Some of the frustration of the teachers interviewed relates to mixed messages about purposes that the test is intended to serve and concern that testing is getting out of control. Several teachers referred to the need to view the MEAP test as part of a more complete and comprehensive picture of education. LESSONS LEARNED ABOUT MEAP USE, INTERPRETATION, AND INCREASING SCORES WM MEAP was originally designed as an assessment of the minimal reading and math performance of student in Michigan public schools. The use of the data by the media, state department officials and the legislature demonstrate intent to use MEAP as an evaluation instrument to measure local school and school district performance in the areas , tested. The department moved from defining MEAP objectives as a minimal skills assessment to now defining the test's content as essential skills (implying more than the minimum). Teacher descriptions of MEAP continue to use the terminology of ”minimal" skill. The teachers hold mixed 146 opinions as to whether the test is "too hard" or appropriately suited for the grade level tested. The fact that every pupil is tested and the form of the reporting format suggest that individual pupil and classroom unit planning may be appropriately based on MEAP. The MEAP Handbook states that individual student planning is an intended use of MEAP (Michigan Department of Education, 1990). However, the timing of the results make such planning and decision making impractical. Teachers describe other sources of information readily available to them that support instructional decision making in a more timely and comprehensive manner than MEAP. Maybe it is appropriate to put to rest any expectation that MEAP will provide help in individual student decision making and planning. The same may be true for individual classrooms. Freed from the need to provide data upon which to make decisions about every 4th, 7th, and 10th grade student in the areas of reading and math in Michigan, the MDE test designers could more realistically consider sampling procedures for MEAP testing. Sampling may free some teachers, school districts and students from the burdens of one more test. This may yield one source of flexibility as school districts wrestle to cram more into the school year. Using a sample of students to collect MEAP data may also help to communicate to teachers that there is no expectation that MEAP results be the basis of individual student decisions, thus relieving some teacher frustration with the test. More concise targeting of the test to specific state purposes clearly stated and communicated to local school districts and teachers could assist in answering the question for “is? ' 147 teachers: ”What is all this testing for?" The trade off is this-~will sampling carry the same credibility with teachers to signal need for change as does every student MEAP testing? Clearly science MEAP has led to at least the perception of need for change. The teachers interviewed described their heightened concern for science instruction and curriculum, their perceived pressure to do something to improve science, and their initiation of responses-~taking classes in science teaching methods, revising curriculum and instruction, etc. Would this have happened if only selected classrooms, or school districts were tested? WW Superintendents, principals and others often challenge their staff to increase students' test scores. The response is often test taking skills training for students and more scrutiny of test administration. These responses to testing were predicted to occur by Madaus (1988). We saw each of these responses practiced in Lansing schools. The real work of increasing test scores, assuming that the test reliably measures important student learning, is to increase the learning of all students. This is often the unstated mandate and, unfortunately, without sufficient direction and support it is the most difficult to achieve. Rather than focus on the test, the focus needs to be on instruction and learning. The test only reflects these. How then do we shift the spotlight for teachers from the need to score 10 more points on the test to enlightened analysis of the curriculum, instruction and learning in the classrooms? My interest as a parent is not that the district MEAP scores go up, but that my child learn more in 148 school. My interest as an administrator should not be that overall test scores go up but that as test scores rise all children are learning what we as a school district, and as a community think is important. Is the call for 10 more points on MEAP a convenient shorthand way of calling for this improvement in learning? If so this study shows that teachers are not hearing the call. Teachers say again and again that MEAP does not tell them anything. Do we call for improvement in this way to raise debate over issues like the best way of teaching and what learning is and how we know learning when we see it? At some level these questions are unavoidable. Who asks these questions, with what evidence? How can we move improvement in learning to the foreground? S s n S o 3 Teachers are frustrated by trying to increase or even hold their ground in test performance. They are groping for ways to improve test scores. Perhaps this is a necessary experience to enable teachers to respond to the ever-changing environment of schools. They learn and are better able to find their own instructional solutions in the future. But perhaps this is just a convenient abdication of administrator and instructional leadership. Perhaps it is not the teachers' responsibility to find the necessary time and information for change. Perhaps the time should be given through reconfiguration of the school day, week or year. Perhaps the information that teachers need for change should be provided. Not by one-shot cafeteria style inservice (where each teacher picks whatever satisfies their appetite for the moment) but by scholarly and practical review of the best practice in 149 education set in the varying contexts and local conditions teachers face. Hand holding, patience, reflective support for the change expected of teachers is needed with feedback for teachers to question, modify and think about possible new approaches. As information is presented the policy makers, as well as practitioners, must keep in mind the big picture. . . . How do testing, curriculum and instructional innovations all fit together across a teacher's day and year? . . . How do they fit into a classroom already crowded with attempts at innovation? . . . How do they fit for students as they matriculate through the education system, crossing grade level and school structure boundaries? These questions should be the "stuff" of improving schools, improving test scores, and curriculum development and planning. To consider schooling in this manner is an expensive proposition. Staff development, teacher ”think time”, and time for collaboration cost dearly. Overburdened administrators are hard pressed to stop what they are doing to ferret out the best that has been said on instructional issues and to pass that along to teachers in a coherent manner that reflects local context and needs. But that is what is needed to support and administer schools. CONCLUSIONS: THE CALL TO ACTION IN THE LESSONS Teachers in this study expected MEAP to give a call to action--to tell them to do something. Through the lens of the teachers' perceptions MEAP does issue a call to action. That call will be made clear here. In the narratives, themes and discussion of this study I found practical- and practice-oriented implications. I also found a 150 second category of implication from this study regarding testing in general and specifically state assessment testing and MEAP. e r ca The talk of the teachers in this study echo the findings of Haladyna, Nolan and Haas (1991) related to test score pollution. With little consensus among educators and test designers, teachers have no direction to distinguish appropriate test preparation practices from cheating. Teachers described avoiding appropriate test preparation activities for fear of teaching to the test. Teachers also described spending precious classroom instructional time drilling students in activities that had weak connection to worthwhile and realistic curriculum. Teachers will be helped by thoughtful and practical discussion of what activities are ethically and legally permissible to prepare students for increased test performance. There needs to be consensus reached among teachers, administrators and test designers on this topic, beginning in Lansing and going far beyond. The connection between the MEAP test and the curriculum on which it is based must be made accessible to teachers. All teachers, not only the 4th grade teachers, need to be aware of how MEAP reflects a well planned state curriculum and how that curriculum matches the locally adopted curriculum and available teaching materials and instructional practices. Those who pressure teachers for increased test scores need to be aware that increased scores without improved system-wide instruction and learning mean little. The focus of administrators, teacher educators 151 and politicians needs to be on improved classroom practice and support for integrating solutions for the many instructional and social demands put upon teachers. u s n As we have become more sophisticated in our ability to produce and score tests, we are in danger of the test score becoming the artifact of interest rather than the learning the test purports to measure. Stronger connections need to be made for the users of tests so that the relationship between the test and the implied curriculum is apparent. Test makers could produce detailed test score analysis reports that not only show student mastery of items on the test but also point out the underlying curriculum concept and give suggestions as to instructional materials and strategies to use to improve student learning. Such a report by the test makers could move the spotlight beyond the scores in isolation, and could set the test scores in a context of worthwhile curriculum and successful instructional practice. Test makers should consider the wisdom of using high pressure or high stakes tests to lead curriculum and instructional change. The use of the revised MEAP reading test occurred before many teachers had a firm grasp on the new teaching methods needed to teach reading with a whole language, constructivist approach. Many teachers were not adequately trained for this radical change in how reading instruction is now viewed. Many classrooms did not have ample reading materials to support this new approach. Teachers were not trained in how to make this approach to reading work for those students who came to school with 152 limited experience in the areas on which the new reading model is based. It seems unfair then to base a test on a reading model teachers were unprepared to implement. Test makers must be aware of the biasing effects of long tests and tests with content of low interest to specific population sub-groups. Test makers need to be careful not to confound measurement of reading skill with student interest, perseverance and patience. W What stories do these teachers tell? The teachers interviewed each said that they did not use MEAP. In a sense this is true. Clearly they did not use it in a way they thought the state level policy makers expected. They did not use it in the same manner as other tests. MEAP did not inform unit or yearly planning in reading or mathematics in the main. And most assuredly MEAP did not inform decisions about individual students (diagnostic decisions). And why should it be expected to do so? The new reading curriculum and practice were prescribed by the local district. The new meta-cognitive reading model and accompanying trade and textbooks were expected by local policy makers (administrators and the school board) to be taught. Further there was a generalized notion that the new reading MEAP tests support this approach, at least in concept. In mathematics MEAP did not necessarily match the scope and sequence of the locally adopted math textbook or curriculum statements. But since the test itself was so skill oriented it is fairly simple to 153 plug isolated math skills and objectives into the overall local curriculum (e.g., 100's chart and metric measurement). From teachers' descriptions we have some glimpse of why science MEAP may be more successful in influencing science instruction in Lansing. Nearly universally the teachers interviewed expressed discontent with science instruction. Many were disillusioned with the science textbook, the lack of time devoted to science and their own discomfort with the subject matter. Further, perhaps more than reading and math, the field of science from which an elementary program can draw is so vast that left on their own it is unlikely that there would be much overlap between a test independently developed by the state and the local formal or enacted science curriculum. Study of earth science and ecology will not prepare a student for a science test that has physics and biology as its focus. Since local direction in science instruction was thought to be weak teachers looked to the MEAP science test as a guide for local curriculum design. Short of legislating its use MEAP in itself stands little chance to be used more than as described here unless several things are made much more explicit. First the curriculum underlying MEAP must be succinctly stated and communicated to teachers (and to parents). It must be clearly stated that MEAP no longer addresses minimal expectations. To the extent that the new state core curricula match MEAP and are stated in brief, measurable language this purpose may be served. This issue is also partially resolved by helping teachers become more familiar with the test items themselves. The items are the only vehicle with which teachers can view state curriculum expectations in reading and math 154 (however, test items like the 100's chart are poor substitution for clear curriculum outcome statements). The state must clearly state its purpose in MEAP testing and expectations from schools. Over the years MEAP has become loaded down with the transient expectations of policy makers. Its role as an accountability device and as a means to reform teaching and learning are not one and the same. MEAP as a tool to certify graduates, close ineffective schools, award compensatory education funds, reward academic growth, evaluate programs and practices, and communicate to the general public, may force the test to serve schizophrenic purposes and communicate mixed messages to the teachers expected to administer and use them. As one teacher aptly stated, '. . . tests have their limitations. You have to decide what you want from your testing program” (Edith, Focus Group, p. 13). d ns During the course of this study I probed the thinking of teachers to determine how teachers make use of, make sense of and understand MEAP data. The methods selected for this study were chosen to provide insight and to provide a description from the teachers' perspective. This study found nuances of teacher understanding that were not apparent through survey research methods. The results were grounded in the local context and practice (changing curriculum and textbooks, etc.). I found that teachers bring much information to their analysis and use of test information. The teachers of this study ”knew” which test items were confusing and poorly written (e.g., 100's chart) or tested in 155 a manner that did not match their textbook or instruction (e.g., thermometers). They knew which items had not yet been taught (e.g., the zero multiplication facts) and which were trivial in their view. The teachers called on knowledge about their students as individuals and as a group to frame a context for interpretation of MEAP results. They recognized the ”slow” classes and even traced such classes of low achievers on yearly standardized test results as they matriculated through the grades. The teachers of this study at times consciously_ chose not to alter sequence, method or content of instruction to match the test but rather waited for the test to be revised. Each of these practices was evidenced through the teacher interviews. In some cases it is reasonable to suspect that teaching methods may not be powerful enough to effect the magnitude of change needed to improve test scores to acceptable levels. If a teacher does not possess teaching strategies adequate to teach reading for understanding and problem solving in the manner tested on MEAP then he/she may not be able to identify how he/she should teach to improve test scores. Bill cautioned that without significant change in the school district culture of allocation of resources, sanctions and rewards, science instruction is not expected to change much no matter the science results on MEAP. Despite admonitions to believe that "all kids can learn,” we saw evidence that some teachers believe that limitations of the child's prior experience, teacher time and methodology doom some students to perform poorly on tests. Teachers interpreted their students' performance against the context of change experienced in their school. [1 156 New curriculum expectations and practice in reading and science were specifically cited as leading to changes in test scores on MEAP. While listening to teachers make sense of MEAP results I heard teachers actively calling on confirming or conflicting student performance information from other sources. End of unit textbook tests, teacher made tests, as well as standardized test results were cited to confirm or to refute MEAP findings. Until recently much of the research on use of test data was conducted at the administrator or counselor level. Most of the data collected at the teacher level was through survey techniques. This study brings to bear qualitative research methods on the question of how teachers make sense of MEAP results. Findings of this study may influence MEAP designers about practical issues to consider when designing the test and/or its reports. Findings may assist principals and district level administrators and instructional support staff and testing staff as they anticipate use of MEAP data by teachers. Findings could lead those who anticipate use of standardized tests to effect change to plan for and enhance the use of other factors teachers find valuable in constructing meaning from test results. Despite many of the teachers' statements that they did not use MEAP results, findings show that teachers make efficient use of MEAP results restricting their use to those areas where they are uniquely and parsimoniously suited to be of assistance. Findings led to a portrayal that shows teachers as "smarter" about use of tests than previous studies have concluded. Survey methods have not looked at teacher sense making and thinking about specific test results against the backdrop of 157 the extent to which the teachers buy into what is being tested, that is, the extent to which teachers see the underlying curriculum area theories and pedagogy as appropriate. An out-of-date test or one out of step with teacher perceptions of curriculum theory and pedagogy may provide data that is valued little. This study has shown that teachers do describe their use of the test in the context of its measuring the right content. There is some perceived match or mis-match between testing, curriculum and teaching described by the teachers particularly in the area of science. These results have also assisted in a better understanding of the idea (and its limits) that testing can be used to drive, change or improve curriculum and teaching practice. This study builds on a body of research that describes use of test data in general and the use of MEAP data in particular. It adds to a deeper understanding of both of these issues. It adds to a foundation of work on how testing is viewed and used in the practice of teaching. . APPENDICES APPENDIX A DISCUSSION GROUP (FOCUS GROUP) QUESTIONS DISCUSSION QUESTIONS These questions are provided only as prompts to initiate your discussion. Feel free to discuss issues which you think are more important to the understanding of this topic. I. WHAT DO MEAP RESULTS "TELL" YOU ABOUT THE TEST? . STUDENTS? . CURRICULUM? . SCHOOL? II. DO MEAP RESULTS GIVE INFORMATION WHICH IS HELPFUL TO YOU IN INSTRUCTIONAL/CURRICULUM DECISIONS AND PLANNING? E.G., HOW/WHETHER TO CHANGE INSTRUCTION, MATERIALS, SEQUENCE, SCOPE, ETC.? III. OVER THE YEARS TO WHAT EXTENT HAVE YOU USED MEAP RESULTS? HOW? E.G., SHARING WITH PARENTS, INDIVIDUAL STUDENT PLANNING, EVALUATION OF INSTRUCTIONAL MATERIALS/METHODS, ETC.? 158 APPENDIX B INTERVIEW QUESTIONS INTERVIEW QUESTIONS I. WHAT DO MEAP RESULTS ”TELL" YOU ABOUT THE TEST? . STUDENTS? . CURRICULUM? There is a curriculum implied by MEAP on which the test items are based, what are your thoughts about that curriculum? Is it appropriate, worthwhile . . . in reading? . . . in math? . . . in science? . SCHOOL? Are there other things that tell you more than MEAP? What are they? . I What questions do MEAP results lead to? Are there things about your school, your students, your school district that one needs to know to interpret your particular school's test results, e.g., the reading teacher was out most of the preceding year with an illness or the third grade teacher was a first-year teacher or we were short of math textbooks and that hampered our instruction, etc. 159 160 II. DO MEAP RESULTS GIVE INFORMATION WHICH IS HELPFUL TO YOU IN INSTRUCTIONAL/CURRICULUM DECISIONS AND PLANNING? E.G., HOW/ WHETHER TO CHANGE INSTRUCTION, MATERIALS, SEQUENCE, SCOPE, ETC.? What are some examples of ways in which MEAP is helpful? If none, why? What is more helpful? If your MEAP scores were low, what would you think and do? Is that different for reading, math, or science? Do you have confidence in MEAP results? Do the results really show what students know? Are MEAP results informative as you plan for individual or groups of students? If yes, how? If not, why? What is more informative? If MEAP is helpful in planning decisions, what levels of planning weekly, daily, term, unit, etc.? III. OVER THE YEARS TO WHAT EXTENT HAVE YOU USED MEAP RESULTS? HOW? E.G., SHARING WITH PARENTS, INDIVIDUAL STUDENT PLANNING, EVALUATION OF INSTRUCTIONAL MATERIALS/METHODS, ETC.? What are some of the variety of ways you have used the test results? How has your use of MEAP changed over the years and why? What other things have influenced your use of MEAP test results? Are any particular MEAP data more helpful than others, e.g., reading, math, or science; different reporting formats, e.g., individual student reports, classroom reports, reports for parents, etc.? Some think that MEAP should and does determine curriculum in local classrooms, does it? Should it? IV. VI. VII. 161 IF I WERE TO VISIT YOUR CLASSROOM THIS MONTH, ARE THERE THINGS THAT YOU COULD POINT OUT ABOUT YOUR TEACHING THAT ARE A RESULT OF MEAP? . . . HAS MEAP CAUSED ANYTHING TO BE DIFFERENT IN YOUR ROOM IN THE LAST YEAR? . . . OVER THE LAST FEW YEARS? IN YOUR EXPERIENCE WHAT IS THE INFLUENCE OF THE PRINCIPAL IN F1 SHAPING HOW TEACHERS USE AND INTERPRET/RESPOND TO MEAP? WHAT IS THE INFLUENCE OF OTHER TEACHERS IN YOUR SCHOOL? IF YOU COULD TELL THE PEOPLE WHO MAKE DECISIONS ABOUT MEAP TWO THINGS, WHAT WOULD THEY BE? ARE THERE OTHER THINGS THAT YOU WOULD LIKE FOR ME TO KNOW ABOUT MEAP? APPENDIX C LETTERS OF INVITATION M E M 0 R A N D U M TO: Third and Fourth Grade Teachers 9 FROM: Pat Petersen, Research and Evaluation Service ddsfilfl. RE: Attached DATE: 11 April, 1991 As part of her doctoral program, Rossi Ray-Taylor is conducting a study' of how teachers interpret. and use MEAP' results. Her proposal has been reviewed and approved by the district Elementary Research Review Committee. As you decide whether you want to participate, there are a couple of issues that you may want to consider. The results of this study will be of particular interest to me as I weigh the expectations we hold of staff for reviewing and. using' MEAP information against the teacher‘ time, effort. and. benefit to instruction which results. Staff at the Michigan Department of Education have also expressed interest in the results. Participation will give a unique opportunity for Lansing teachers to express their views and experience with MEAP. The bottom line is that, like all research studies, your participation is voluntary. Thank you for reading through the attached material for the details. cc: Alda Henderson Elementary Principals 162 163 Dear Third and Fourth Grade Teachers, This letter is to request your participation in a research project designed to learn more about how teachers interpret and use MEAP results. I am inviting third and fourth grade teachers to be involved in this important study. The results of this study will be shared with staff and policy makers at the Michigan Department of Education responsible for design and administration of MEAP. The study is to fulfill partial requirements for a Ph.D. in Educational Psychology. The committee guiding my research consists of M80 professors actively engaged in studies of education policy making and teacher decision making. They represent backgrounds in the subject areas of mathematics, language arts and social science. I am asking your participation in an interview about MEAP results. Participation will take approximately 1 to 1 1/2 hours of your time, and will be scheduled at your convenience between May 6 and May 22. You will be interviewed about the ways in which you use and interpret MEAP results and whether you find MEAP beneficial to your instruction. No special preparation for participation is needed on your part. I think, however, that following the session you will gain some useful insight about this important topic. Participation gives a rare opportunity to express the first hand experience of teachers to policy makers and researchers at the state and national levels. This study is based on the belief that it is important for policy makers and researchers to understand teacher opinions and thinking about events that effect teacher and student lives so greatly. My committee and I are excited to have this opportunity to describe teacher's views and to share them with those who make policy and shape practice in schools. The names of participants will be kept confidential unless you choose to disclose your participation. Video and audio tapes will be used for data analysis but not for reporting purposes. PGP credit is available for your participation. Your participation is voluntary. I hope that you choose to participate in this important study. Please complete the attached response form and indicate whether you can participate. If you have any further questions, please do not hesitate to call me at 374-4410 (work) or 372-3151 (home). A ' 5&4 ’6,27/V Rossi Ray-Taylor' £77 APPENDIX D ASSURANCE FORM ASSURANCE OF VOLUNTARY PARTICIPATION AND CONFIDENTIALITY I understand that my participation in this study is voluntary and that I may withdraw from participation at any time. I further understand that my participation in this study by name or by school will not be disclosed without my written permission. I have the opportunity to request that. my' participation be disclosed for the purpose of obtaining Lansing School District Professional Growth Credit. I may also request that my participation be disclosed to the persons I have listed below. I am assured that all video and audio tapes of research sessions will be used only for data analysis purposes and will not be used for any reporting purposes unless my written permission specifically granting such permission is obtained. My signature indicates that I have been assured confidentiality of participation in this study as described above. Printed Name A Date Signature Exceptions I request that it be disclosed that I participated in this study for the sole purpose of obtaining PGP credit. I request that my participation in this study be shared with the following person or people: 165 APPENDIX E PROFILES Teachers: School: Grade Level: Years at this grade: Years teaching experience: School rank: Years in effec- tive schools: Other grade levels taught: Karen Ottawa Third 30 29 K-6 PROFILE FOCUS GROUP 1 Katheryn Pine Ridge Fourth 15 22 2.3.5.7. 8,9.college 166 Dawn Pine Ridge Third 15 22 K-12 Teachers: School: Grade Level: Years at this grade: Years teaching experience: School rank: Years in effec- tive schools: Other grade levels taught: Edith Redwood Fourth 23 15 2.3 167 PROFILE FOCUS GROUP 2 Doris Fourth 36 1.2.3 Lynn Third 1,2,6 Sheila 2nd/3rd Teacher: School: Grade Level: Years at this grade: Years teaching experience: School rank: Years in effec- tive schools: Other grade levels taught: 168 PROFILE INTERVIEW l Valoria Elm Fourth 23 23 15 4th/5th combination Teacher: School: Grade Level: Years at this grade: Years teaching experience: School rank: Years in effec- tive schools: Other grade levels taught: 169 PROFILE INTERVIEW 2' Jerri Deerfield Third 25 K,l.2.reading Teacher: School: Grade Level: Years at this grade: Years teaching experience: School rank: Years in effec- tive schools: Other grade levels taught: 170 PROFILE INTERVIEW 3' Rose Ottawa Fourth 22 29 3rd/4th.5,6 Teacher: School: Grade Level: Years at this grade: Years teaching experience: School rank: Years in effec- tive schools: Other grade levels taught: 171 PROFILE INTERVIEW 4' Ted Lenewee Fourth 22 28 1st/2nd.4.5th/6th Teacher: School: Grade Level: Years at this grade: Years teaching experience: School rank: Years in effec- tive schools: Other grade levels taught: 172 PROFILE INTERVIEW 5' Aieda Deerfield Third 6-8 APPENDIX F MEAP SCHOOL SAMPLE PACKET gfig‘gilg llllllllllllilllll ton—oz ace—toned .=_ I100. fl-a‘: Obi-DD». a pluUl-L Us: Ouc(x 20:.Uw4mw 4. 5° LII”: Clov- Dada- L III 93.0 .5.» 2.: x. 82.. 2.253: 2: o» 23!... on 68.88.. :86 8.35 to Loan! 1.3 min “.3 min .5 ad» 12 n6» .5 U: nan-“5P3 wflfiflun.§4n“”oruafl 9n n“: HUM" MUM” .E at... s of“. u..— E In bub Dunn ans .5 nag—uh 0:: a! :5 g 00: 3:— !» £23.39 0 ON M mm b 0' In: A h I on u no Nun l. .Wfiphunww“ Hannah—fl “1U. 39 U: ofiu Run 5.8 .5. . {on n.0u néc .t: . .. . :35. . . . . . . . . . . U: a»: mg: on.- g aha-$.03 U: .29. Du ".MH ".M" M.“ .N a; .3 03. B- ump: 9' M." ".MM M.“ .fil 2535 6.. R! :3: 22255 :3 R! :2: 8 52! 28:03: >5 .5 3032 a . 13 2: 2383?. 52225.2:— 253: 9552.523 22833 :35 so :6. on: n .3335: 52 .hzouuu .uozsfiouzun no hxguzlugun 0.33:2.» lllllllllllllll . . . As; .2 or“. .|= may! . . . n: a... on: an a... nun son 2 can .8 on 35m» 22255 394; on e... 0% 0.: 3.62 ea 83 in! .2:- 3 9.9. {on o... 8 52.2 .58 .523; in» £535 5.. R! :3: .8... .2... 9563. 2....» 8 503: _ 23533 a: . 22833 #225535 2:25: 2.232528 0.25 05 .o 3.30: >. . n 02:25. .50: munuaxbzx «:3 532.30: 69:5 . . . :38 a: a: .5: 5! . . . 3225595 ”5.... a» e o n no .. u on S e o o nu. u 3 <0 a... To» . 1: 3.62 ea :2 .3. 6:1 3 a... 9.3 fl: . S 2 .. ............ 22255. :3 a! :2: £232... :3 8.. :2: “.3 u... a .3: .5 «was: . 23533 2: a 5!... .t: 22533. :zafiszfifi 3:25: 2.22.528 2283mm :8; on 3 n. . I... .1: R has... .. Caucasus: 2.3» 3 on . :2 .. 8:239:00 «no... .350 can .28» 05:00:. acacia-:00 cooiaom 2:23:33. .= i b u 5: u 5.8 .9. on I 3‘25; >35¢opu<3..¢ofiz§¢a Enos! w . .2 > a... m3 9... 8» n u n u on w a ed an “S n3 9... o: k." «on > o 3 m“ a“. w...“ 3 um "um um 9.: a...» «a w w a... 3d n... :u a... «mu s; m2 2 p o... "3 Tu com o... emu 1n 2: . . u < 5.. .3 a... mom a; "3 m6 men x mm a 2. a Sn 2n 9. :n 9.: SN #3 a < u 9. a: o; o: in IN To“ 8” c.» 03 a”. :m own o: £8 on 8: 2: «.3 “8 u a me» o n 2» a 2 8n Isl-65>. ‘0 pl-U‘: '00. 3:10- zouhuwawm >¢o~m .83.! 2‘00! 0.: .5 1‘ 41.4.1 5.828. ouovnm Ibunuhnun Bug 38.8 gun: as :83 .5353 ..1. .- - .w .mq¢<=z=m soozom «oodlooon . . iii-E . . slul III at: 5 «who: dome 89.00.... «c2533. c2653 coo—5.:— 173 §32g1§£§3uufig ONOIO .hnuh 2:» 2. cu»: 9.0.515“ at» O» 0.5-Dunn- : 63.82.. :86. III... 3 .352 1.8 a... 1% u... .5 5: 9.8 ad. .5 U: ofiflnwfifi undercut wuss—W! or: 38.5 at OJ“ h.nu O.¢N .5 as; .1 or: .0: in! an 0.0" mi. 0.0" .5 . v.3.“ 2:- a! .9: Hanan: .wwww “MEWS—Elu- 9 an n at 0 on .51! 0 mu 0 00 n o .Evn B >Lfi>h_nawwnaus.wffluwnwfl vi“ as u: 0.0 5.3 Tun .c: n.nn >5» ode E: _ g o 9.0 min >3. .5 :33 ll .03 .B- i=5 n— 9.9 min To. .35 U: E... as: :2 ea 3: 3: U: .2... o... T: min. (at >3 min 9.3 .If £585 :3 8: so... . £335 :3 8: .5... 8 505: 3.3532. >3 8 50a! 2 a... 2: :35 sum 223:5...qu 2:25.. 3.82828 2283“.“ :67. 5 RR .8. .. gamut: nz< .53.: £025.33: no anuxinua 9.53:5 . . . .3“- g or: .R— Eu! . . . a a... 8: 7 c on o 8 o a 2 o 3 c on o o OUhnwb ”Eugm nwgn o 0.0 n.nn 5‘0 . a; g :3 .E in»: a 0.9 6.00 0.9; to lug .Eho» Enos: :4: tan-humanm .8.— en! _ in: AZOnhouJum >fl~ tum—“ugh“ 6.. an! .8»: «now 95.3: 2.2». 23533. #222522. 2:25: 02.82528 03cm 2: .o 2:53. >— 1... .. 2.35.. 53¢ menmnxozz <>5 8 .35: U I... 22533 :zoCEzEz. 2:25: 3.52.528 23533. :85 In... an... H. .5: .. tunings: 8.5.. E; 2 3 . .9. .. SacconEoo «no... .350 can «23m 055.22 3:25:30 coozaom n.:a:o=a.o¢ .= i r u 0.5 I 5!. .o; o I 3.21.8.1! 2.05:3: 525.. 0.0“ I Sun .5. ad I «9339:! 5.99.3.3: 1.5.5.. 0 cc 5.: w z 0 o.u0u I :5: DJ.— I gg! 28533:: tau-l h .8u I IS: ”an I gig: £85525..- .505; n > a.“ m: . o... 3. u u 9.: mu m m . 0.0 emu ad uou 9.0 02 m6 no“ > o ”H” “m m". a" 3 um 3. a“ 060 add (a w W 9.: saw 0.0 or». 06— an" H.0 m9“ I ,._. nut ”cu on. can one an» nut con 9.: 0.0 U 1 A} can n.v mom n.c ~ou n.c men _ n < 0 . 0 en a: o a 3n n 0 non m o :n a.» z. 1. o: o... :u 4.. 3. .. 0.3 a: 0.9 an Tu o: 15 gm . 8: 8: .5. S. o... n: .5 2. 1. a. Ilozfl-adx Out-03>. >0 blunt-s IlOUln—ua‘g But-63>- ». pluvl-t .300- fl-4<0l 2 Incl». .0 plug-- :00. D...“ zo~»0w.._wm 4¢<2=Dm JOOIOW «3.732 69.02.... «c0643.... 5:323 coo—5.:— 175 an. . 2 ”A w .9830... :8... g angina m. a. 3 2.. 2 no. .22 2.2 .22 m... 2. . a .2, 2. m” 325 22.2.. . 8n! .3 22.8 £5820 ugfiu no no on 3 sun-8...... a; a6 a... .6 . w... . w o O O O C :> n ¢ 9 o o a o o a m c >0 9.2 a... 1. n6 . aw . . . . . » ERREEH 5.5.5855 u... <0 in ‘u 0’" coon mung! .32. ..+¢ofi..a..o.§c.. 5.. (5.?» 1. .2?» IIII ...II a: m. . 'Mri ‘ ’3’“: J '2‘ 917T..\.." ”2.3 «~. .‘_ I t: :12. I- -1.-'-2\__..4~ .- .- “IL m .‘fiffi'w‘f w—fi'JfiN-H'L‘J'u _--_ _ 1 — “n- 2?. Mn: ”M s-{mn' 1‘1”" > '99-'11: . . _ ‘ "fir-x . y. a 1 , If I ~3.~ I W~ . ' ,IK ' ', . 4” w". ‘ , 33v - \ . ‘ ““".J :”P . 3 .. 8 839:. So... on v .3 .o - 4 850:. 39! g: 350:. 39! 8 98 .3332! Bug g"; B unplug.- 00.3.7. E a “uni! .Ufih 535...» .8 has?! :52. g S .5 sun .gnouk Duos.“ I a 3 i .t'.‘ 2099 222232221212 gun Zn In; bk! 3:; 3 km“ Inns (8 v 0’ GO (8 A 0: 4:5 buOnaIm 263 (“out .mbwm Cues g5: . .fit. ..2 .2 «a . argue .22 agfi¥um .. ;.2 2.... .22 2.2.23... . _ 2 L8. 3.»... . Q... .. OIhOu NInOd Olsen ”can - ——c‘—. -——. 0% Gaolnn Inga gnu. gas 32.! .6.“ aug- e ‘31 monh¢ "‘95:? :lea |%. .. ’ V-n. V;- I. A) __d— avg} ‘- f-‘me. .4! u; 333833388 883383383333 88 $3 3 33 38833888 . .__ ! :- ~. $3.15. q 'V v‘ \ f , ‘ J_ ‘ . V _ 6 , . ’~ n ‘ '\ 5" v ‘ _~. 7 f' '-;‘ €13 . L214 -,.~ “I r. g.» .rggg 8. an; 39! I O x c .9: 049! I C a u 355 39... 09 u a .9522 39! OnV..( .ll‘ uneasy; g: .aug 0490. 80> 8 as :Unbgg 3% g: B nuuxwg 3.38 E a 5g! . a» E! a bngfiummmm S .5 as .088! a; 950 g >=hzwc (Iv-0‘ 0 I I I CC‘( g E ‘i’ 3 f. R I. ’22:§¢€§2384‘fl. a .: ..“§ 92223222 1 .“mmm ‘1 l t I III g ‘ ”— ag ago E! 4.. w r a F“. L .2 .1 w .4... ,1. m m w. m i r 395 3.88 ._. 8m! 0033 828. W“ 5.0853 figs-ubChu so no nI 9. .550 ... a." 9.. a... a... _ 2; U0 . . . . a :> 0 an I u o o h n w ¢ . >0 0..» «.0 «.8 "in . mm 1 b _.n U( . (0 80.- !n 0:“ 20¢ Bangs. ”4...". “Imus... .33. = .1 n a...” ,. _ {JIIIIW ....8..8 8?: -8352 .889 .g E: g 203 30g. * e K .1 _ munp¢«¢Ohm hawk 1: gland; hxommm othmH... zoomm <40 « «=58 ,.. 9.88.. Q09: 38:22 585...... 88.838 cogs...L again 179 2.2. 5.... .._.|.v nu: it's-“>0... 1.2%;- n £23386.- 3 2.823 23:3... :2 s annulus a. ‘82 .8380“: s bio-uhidun Przugp- .2 -5 (Olg- pg Us; A! gs; 98> .- .3.— Cu- 5 . ~u\6— {0 Italian 30:05u. 39—13805:— §O on 3‘ giant! g.- ‘hglhg .- ZOFUuJu. 02.3: :0: :0; ans“; run... 0.3: I :1- nOOO 00:. H... =1 ‘2” you“... hmomux hzwaahm a<=nn>mnzH :35: La.» .85. J8.“ two-cu. H85. bug-.— in!“ add—>33: 33:9 . 3... g Coin-— . .83 is 23 >1: "all... oz~a .2 .2 .: .quuiuo-zusi n p. .8 .3 .3 1.9.21.6... «-2 u <2 .2 .= 2. “9.“...93-229: u > c «2 . .2 . e2 8 «.23. .83.... 2a... 72 .. ‘. .3 )8 QB ammfiuflows .22.!!! a-.. .. u . a 8 1 no . 3 ‘82. 8 2228... 7.2 .. a 2 a 2 n S 22.3 7.2 :2 .1. _.u _.o 89:25! .2: 7.2 .. u a 2 ¢ 2 . 2 83.222 . ~22 p . n . u . 2 am 2!! .35.: 0.2 a: _« a «a _« a S .Sflu... 2.5.3.“... .3. n > 0‘ on...“ .v..¢—.l< ”.0“ p . 2 . 2 . o. . . . .. . - a o: .3 o: u.¢.<-ol< 0-: p «a. .2. .3 32.13.... -2 p a .2 .3 .2 I...38...EEJ-2 -— p a . 3 . 2 . I 2.522.. 8 so - 2 -o I o: is» 105 . Engage. 0|: I‘d p . 2 . 2M . 2 .93... 205.. 3:!" a... p . 2 . a . 2 !..38s. 5...... a. . 2 a-.. D o In 0 On 0 In .figuoun sun! U o a no: u u o 2 < 2 a 2 9.388.. R. .3 . 2 «-2 .3 on. .8 «$2382.! “.2 -2 ‘ g ‘ > < d . 2 . «a 82.32 .a 202. .k! a; p .2 .2 .2 32.5.8.2-88 o.. > .2 .2 .2 .8. .:8 8.3. $2 > ~ . 2 . 2 . 2 125.5: 2? 2:... o... p u . u . 2 . 2 895. .225. :2 - 2.9: T: If .u . o . 2 . o 3.2.. 22?- ...53 72 a . o 3 E89 2.33. .52.. 33. u... > n . - . 8 . u was... .22 2.8 .6. > n . no . co . no 3 =6. .22 .88 a-.. I538. To: 5. 8.3 89.8 tnYnEiflfll 8.2“. r...» Bauunumnifl 5.252821%» 98' u 2-. “mg .35 .8282.“ i...)- monhnazH «5.4a. . _. .. 180 3: 0g .00... thug. n 28:08 3 a g .83. £003: .53an ; 5.5588! 388 . . ma0... 0... .0 0.. 0.0.. .02... .23 .z. .0... 3.... .8. .6; 05:00.3 .0500. 90.2... 50» 0.0.. 8 00.30 0.0) 000.20.... 0...... .260. 2.3.. .8. 8...... o... 5 0.3.8. .0: 0.0.. .0... 80. 9:000. .32 0... 00 30.8000 .2... 03... .2. 0.2. 30> "20:02:22. 22.5 .00.: 00 :00 09.0.59... 0.0 .6... 0:0 .000. 000.. .2 .003 .000. .0. 0. .20..) .0000 50. 0. 3:22.080 .0000 0... .0 0: 6.1.0.0... .0 20:2. 0 000. 0. .000... 0.20 50>.000E 00:00.00 9:000. .00..0 :0 30.00 .0 «an... 0.80 ( 08:00.00 0.: 000.2000: 000 000. 0.20 50> .000... 02.» 5.0.00.0:00 00.00.28 2 00:00.0. 00.0.0. 00.000 0.30 .000» .0 0.8. ( .22....0. 00.000. 0... 0... .0 8.0 :0 0.8.9.2008 93.02.0000 0.0.10 50.. .0. .0000 2.82.2.8838 8. 8.... 65:»... S... .000. 000 000060. :00 )0: .0 05.00.... :00 05 2 30.000! 93.03.0000 00:00. 00:00.0. 93000. 0 00.05.2000: .0... .50... .0... 5.00.0. .0000: 9.00020. .20.... .080. 80000.0. 0. 0.50. 0.05 0.... 00:00.0. .000...E.0.£ c0 .03 .050 0... 000 E20 0 .0... 000 05:00.0. 9:000. 03:09:50... 0... 000.2000: 0... .0 0.. =0! 30.. .0000 2.0.3000 0.20 50> 5.0 .00. 9:000. 0(a) 05. 02403 hunk 62.03.. nth :0 album 904.20 :30) #5.; 5 : 2.... .0. .. >9!‘§§!2£§0 .8... .8... .8... $825.... .0 80.0.0.0 0.00030 2 .0. $000.0 30.4 .02....5. .- 2.. $.83... .1.» 0 80:23:02.. 5er .2 .80. 033. .22 38.8.0 . 1’“ fl. 0. ... .30 . . v! i. l . .I .3. v- in 900.30.08.08. ‘ ‘T'TjIUII' '32358338' 80:03: 08—31 80$ (0; «Pusan: run». >. HONCMNNI. ~.M.O~~fl .......:w “I? 300—. 02.n m burn: (a: .333 5a: .8... /: m guy—h 6200.000. 381.08 ._ 2.... 2.0 .3 82.8 2.028. .32 3.8.... o... 8 :2. .5 8. 8.... .953... .02 2...... .3 0...... "8.5.... I... 02.3... .. 6030: 0.00.0000 0800.0. .0 3:000... 2.. a... 850.. 8.2.0.0 9.2.2.... 50.5.0... $08... 5... 9500003000 9:000 000 .0502. 26 20.00.. 2. o. 2... 25...... 0:20.58... 30.00... . 08.0.0060. 000 .0620» .020 030000. a. £3... 2...... .5 8.... 0.2.3.8.... 3...... .. .80.... .8323. 052:5... 0... .58... 05.1.8... 0... 2.8.3:. .200. 35...... d .020: 0001 .0 9.3002005 00 9.00... .0500 000 .6030... 0.0) 000 000050.000 50.. £3... .15.... 2.2.33 2. 9.0... 11...... .n .20... 6.. £8.38??? 80.... a... 901:3... 05.8.8... .80... 8.382.150 .020) 000.089.300.200... 950300980 000 .0003: 50...... >8 as...) 2. 0:0... 2.3.02.2. .. ”0.020020 .515258220230328032 3.8.8.3623... 3: 2. 828. 90.2. 30> 02:! Punk ngguxh<§ USP 80 alga 99:30 83> kg 5 w REFERENCES REFERENCES Airasian, P. W. "Uses and Misuses of Standardized Tests." Measurement in Education 10, 1980. Atkinson, P. e Et 0 a c a t 0 ° tua Co 8 f Egnllgx, Routledge, London, 1990. Beggs, D. G., Mayer, R., & Lewis, E. L. ”The Effects of Various Techniques of Interpreting Test Results on Teacher Perception and Pupil Achievement." easuremen d va u Gu e 5 April, 1972, 290-297. Bhaerman, R. "What Do Teachers Think About Tests and Testing?" Angglnnn Educator, Winter, 1977. Brophy, J. E. & Good T. L. ” Teacher's Communication of Differential Expectations for Children's Classroom Performance: Some Behavioral Data." WWW 61. 1970. 365-374. Citizens Research Council of Michigan. gnunnll_§nmmgn§§. Total Financial Support for K-12 Education (1974-1988), No. 987, February 1990, Detroit. "City Low in Urban Scores," Lansing §§ntg Jougnnl, Sunday, February 21, 1982, p. 18. ‘ Cohen, D. K. & Ball, D. L. ”Policy and Practice: An Overview”, n Po A s, Volume 12, Number 3, Fall 1990, 347-353. Cohen, D. K. & Ball, D. L. ”Relations Between Policy and Practice: A Commentary," Educational Evalunglnn und Enllgx Analysis, Volume 12, Number 3, Fall 1990, 249-256. Connelly, M. & Clandinin, D. "Stories of Experience and Narrative Inquiry,” Educational Researcher, June-July 1990, Volume 19, Number 5, 2-14. Corbett, H. D. ”The Role of Stakes and Pressure in Determining Local Response to State-Mandated Minimum Competency Testing," luu;nnl_n§ figsgnrgh and ngglgnment in Edugaglnn, Vol. 23, Number 4, Summer 1990, 204-215. ' 182 183 Darling-Hammond, L. "Instructional Policy into Practice: ’The Power of the Bottom Over the Top,'" ucationa valuat and Polic Analysis, Volume 12, Number 3, Fall 1990, 233-241. Dusek, J. B. & O'Connell, E. J. "Teacher Expectancy Effects on the Achievement Test Performance of Elementary School Children." gnuznal Qfi Edugagional Psycnglggy 65, 1973, 371-377. Fuhrman, S. & Elmore, R. "Understanding Local Control in the Wake of State Education Reform". MW. Spring 1990, Volume 12, Number 1, 82-96. Guthrie. J. W. (Ed.) MW. The American Educational Research Association, Washington, D.C., Volume 12, Number 3, Fall 1990. Haladyna, T., Nolen, S. & Haas, N. ”Raising Standardized Achievement Test Scores and the Origins of Test Score Pollution.” EQEQQEiQDQl Rgsguxnngz, June-July, 1991, Volume 20, Number 5, 2-7. Krueger, R. A. F c s r ° C d Sage Publications, Newbury Park California, 1988. Lansing School District Memorandum. Chamberlain, R. J., March 12, 1982. Lansing School District Memorandum. Halik, D., March 23, 1982. Madaus, G. "The Influence of Testing on the Curriculum,” Q;i§ignl_l§§ug§ n urriculum 8 h earboo o t o o t e t d o Enunnninn, Ed. Laurel Tanner, Chicago, University of Chicago Press, 1988,88-98. ‘ . Manen. M. W Sensitive Pedngogy, State University of New York Press, 1990. Mathison, 8. "Implementing Curricular Change Through State-Mandated Testing: Ethical IssueS” MW. Spring 1991, Vol. 6, No. 3, 201-212. Morine-Dershimer, G. ”How Teachers See Their Pupils” Euunuuinnnl_fig§gn;nn Quarterly 3, Winter, 1978-79, 43-53. Olejnik, S. F. ”Standardized Achievement Programs Viewed From the Perspective of Non-Measurement Specialist.” Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, April, 1979. O'Regan, M., Airasian, P. & Madaus, G. "The Use of Standardized Test Information by Irish Teachers." Paper presented at the annual meeting of the National Council on Measurement and Education, San Francisco, April, 1979. ' 184 Orenstein, A. C. "National Reform and Instructional Accountability”, Thu flign School Jounnal, October-November 1990, Volume 74, Number 1, The University of North Carolina Press, 51-56. Pedulla, J. J., Airastian, P., Madaus, G. & Kellaghan, T. ”Proportion and Direction of Teacher Rating Changes of Pupils' Progress Attributable to Standardized Test Information.” l2uIngl_2i_EQ§2§£12D§1_2§1£h212§1. 69, 1977, 702-709. Perrone. V. Wain. Bloomington. Indiana: Phi Delta Kappa Educational Foundation, 1977. ° Porter, A. C. ”Creating a System of School Process Indicators,” WWW. Spring. 1991. Volume 13. Number 1, pp. 13-29. Porter, A. ”External Standards and Good Teaching: The Pros and Cons of Telling Teachers What to Do", v Anulyuig, Winter 1989, Volume 11, Number 4, 343-356. Porter, A. C., Schwille, J. R., Floden, R. B., Freeman, D. J., Knappen, L. B., Kuhs, T. M., & Schmidt, W. H. "Teacher Autonomy and the Control of Content Taught." Research Series No. 24, The Institute for Research on Teaching, Michigan State University, December 1979. Rudman, H. 0., Kelly, J. L., Wanous, D. 8., Mehrens, W. A., Clark, C. M. 6 Porter, A. C. "Integrating Assessment With Instruction: A Review (1922-1980),” The Institute For Research on Teaching, Michigan State University, East Lansing, Michigan, 1980. Shavelson, R. J., Cadwell, J., & Izu T. "Teachers' Sensitivity to the Reliability of Information in Making Pedagogical Decisions”, Angrinnn MW 14. Spring. 1977. 83-97. Shulman. L. W Occasional Paper No. 8- The Institute For Research on Teaching, Michigan State University, East Lansing Michigan, 1978. Sizer, T. o ace's Com 0 se' e e he 1, Houghton Mifflin Company, 1984. Smith, M. "Put to the Test: The Effects of External Testing on Teachers", Euugagignul Reseaggner, June-July 1991, Volume 20, Number 5, 8-11. Smith, M. “Meanings of Test Preparation", Anuginun_£§un§§innul_fig§gnyun inuznul, Fall 1991, Volume 28, Number 3, 521-542. Sorotzkin, F., Fleming, E. & Anttonen, R. ”Teacher Knowledge of Standardized Test Information and Its Effects on Pupil I.Q. & Achievement.” Jounnal Qt Ennggimentul Educnginn 43, Fall 1974, 79- 85. 185 StetZ. F. W 1" "' 5. V',‘ 3 ‘ _‘-~ ,,'!C'!1 1: 1‘ ,,. 3.9,. 'i ‘-V'!!' lung; Market Research Report No. 10. New York, The Psychological Corporation, 1977. Stetz, F. P. & Beck, M. D. "A Summary of Opinions Concerning Users of Educational Tests." Paper Presented to the National Council on Measurement in Education, Toronto, 1978. Stetz, F. P. & Beck, M. D. "Comments from the Classroom: Teachers' and Students' Opinions of Achievement Tests.“ Paper presented to National Council on Measurement in Education, San Francisco, 1979. Stevenson, D. L. & Baker, D. D. .“State Control of the Curriculum and Classroom Instruction”, finninlngy_nf_fiuunn;inn, Philip Wexler ed. 1991, Vol. 64, January, 1-10. 11 ‘1 Sykes, G. ”Organizing Policy into Practice: Reactions to the Cases" WWW. Volume 12. Number 3. Fall 1990, 243-247. Wise, A. E. "The Hyper-Rationalization of American Education.” Emmi; February. 1978. 354-361. Yeh, J. P. ”Test Use in Schools." Washington D.C., U.S. Department of Health, Education and Welfare and the National Institute of Education, 1978. "llljlilllflill“