INFORMATION TO USERS This reproduction was made from a copy o f a docum ent sent to us fo r m icrofilm ing. W hile the most advanced technology has been used to photograph and reproduce this docum ent, the quality o f the reproduction is heavily dependent upon the quality o f the m aterial subm itted. The follow ing explanation o f techniques is provided to help clarify markings or notations which may appear 011 this reproduction. 1 .T h e sign or “ target” fo r pages apparently lacking from the document photographed is “ Missing Page(s)” . I f it was possible to obtain the missing page(s) or section, they arc spliced into the film along w ith adjacent pages. This may have necessitated cutting through an image and duplicating adjacent pages to assure com plete co n tin u ity. 2. When an image on the film is obliterated w ith a round black m ark, it is an indication o f eith er blurred copy because o f m ovement during exposure, duplicate copy, or copyrighted materials th at should n ot have been film ed. F o r blurred pages, a good image o f the page can be found in the adjacent frame. I f copyrighted materials were deleted, a target note w ill appear listing the pages in the adjacent fram e. 3. When a map, drawing or chart, etc., is part o f the m aterial being photographed, a definite m ethod o f “ sectioning” the m aterial has been follow ed. It is customary to begin film ing at the upper left hand corner o f a large sheet and to continue from left to right in equal sections w ith small overlaps. I f necessary, sectioning is continued again—beginning below the first row and continuing on u n til com plete. 4 . F or illustrations that cannot be satisfactorily reproduced by xerographic means, photographic prints can be purchased at additional cost and inserted into y o u r xerographic copy. These prints are available upon request from the Dissertations Custom er Services Departm ent. 5. Some pages in any docum ent may have indistinct print. In all cases the best available copy has been film ed. University MicrxSilms Internationa! 300 N. Zeeb Road Ann Arbor, Ml 48106 8303831 Music, Phyllis Hope Weddington A N A N A L Y S IS O F T H E R E L A T IO N S H IP B E T W E E N SELECTED IN S T R U C T IO N A L M A T E R IA L S A N D T H E M IC H IG A N E D U C A T IO N A L ASSESSM ENT P R O G R A M OBJECTIVES FO R T H E P R IM A R Y G R A D E S Ph,D, Michigan S ta te University University Microfilms International 300 N. Zeeb Road, Ann Arbor, M I 48106 1982 AN ANALYSIS OF THE RELATIONSHIP BETWEEN SELECTED INSTRUCTIONAL MATERIALS AND THE MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM OBJECTIVES FOR THE PRIMARY GRADES By Phyllis Weddington Music A DISSERTATION Submitted to Michigan State University in p artial fu lfillm e n t of the requirements fo r the degree of DOCTOR OF PHILOSOPHY Department of Administration and Curriculum 1982 ABSTRACT AN ANALYSIS OF THE RELATIONSHIP BETWEEN SELECTED INSTRUCTIONAL MATERIALS AND THE MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM OBJECTIVES FOR THE PRIMARY GRADES By Phyllis Weddington Music The purpose of this study was fourfold: (1) to determine the relationship between the ski 11-exercise objectives of three reading series and the MEAP reading objectives tested in grade 4; (2) to compare the percentages of reading-series objectives th at matched MEAP objectives, in order to establish congruence with MEAP objectives; (3) to ascertain whether the degree o f congruence between readingseries objectives and MEAP objectives affected MEAP-test results; and (4) to determine whether teachers' methods and strategies might have affected MEAP-test results. For this study, a panel of reading experts was asked to ascertain the purposes of s k ill exercises in the workbooks and teach­ ers' manuals of three selected reading series. Congruency between MEAP objectives and reading-series objectives was determined according to the percentage of ski 11-exercise objectives that matched MEAP objectives; then co lle c tive scores from Classroom Listing Reports were used to ascertain whether members of the treatment group that used the most congruent series had the highest MEAP-test scores. Phyllis Weddington Music F i n a l l y , a questionnaire was administered to primary-school teachers in each d is tric t, and the results were compared to determine whether t h e i r methods and strategies influenced test scores. Although the Houghton-Mifflin series objectives were found to be the most congruent with MEAP objectives, the treatment group using th is series did not perform better than the other two groups on the MEAP te s t. Collective MEAP-test scores revealed that classes in the th re e districts performed equally well on four out of fiv e test c a te g o rie s . On the category of c ritic a l reading, however, the H olt, R in e h a rt, and Winston treatment group performed best. Since the three treatment groups performed equally well on fo u r categories of the MEAP te s t, i t seemed that the methods and s tra te g ie s of teachers did not affect the test scores for those cate­ g o r ie s . I t is possible, however, that the methods and strategies of te a c h e rs did affect the test results for category 4, c ritic a l reading, as in d ic a te d by the higher scores of the Holt, Rinehart, and Winston tre a tm e n t group on this category and by th e ir teachers1 s e lf­ perceptions. To the memory o f my beloved parents, C rit and Ella Weddington, both of whom surely share this accomplishment. ACKNOWLEDGMENTS Sincere appreciation and gratitude are extended to many in d i­ viduals and groups for th e ir encouragement and contributions through­ out this study. Among those I would lik e to acknowledge are: my husband, Gene, fo r his love, encouragement, and assistance each day; my son, Mark Edward, fo r understanding and enduring my preoccupation with my work; my committee chairperson, Dr. Janet Alleman-Brooks, for her friendship, guidance, and invaluable assistance throughout my doctoral program; committee members Dr. Sheila Fitzgerald, Dr. Peggy R eithm iller, and Dr. Richard Featherstone for th e ir willingness to serve; my mother-in-law, Mildred Johnson, fo r her constant fa it h , caring, and assistance; my sisters in Delta Kappa Gamma In te rn a tio n al, who provided both monetary and moral support; administrators and teachers participating in this study; and my family and friends, both old and new, for th e ir companionship and encouragement. TABLE OF CONTENTS vi TABLES INTRODUCTION TO THE STUDY ................................ 1 Statement o f the Need for the Study . . . Statement of the Problem ............................ Hypotheses ........................................................ General Hypothesis I ................................ General Hypothesis I I ................................ Statement o f the Purpose ............................ Importance of the Study ................................ D efinition of Terms ........................................ Limitations and Assumptions ........................ Summary ............................................................... 1 5 6 6 7 9 10 10 13 14 ........................................ 16 Introduction .................................................... Beginning Reading ............................................ Criterion-Referenced Achievement Tests . Michigan's Accountability Model ................ Summary of the Literature Review . . . . 16 17 30 44 53 PROCEDURE AND METHODOLOGY ................................ 54 REVIEW OF LITERATURE Research Questions ........................................ Materials and Sample .................................... Research Design ................................................ Basic Design Elements and Procedure . . Description of Materials and Instruments P ilo t Studies ................................................ Analysis Treatments ........................................ Contingency Tables .................................... Analysis of Variance ................................ Summary ................................................................ 54 55 57 57 58 62 64 64 66 71 DATA ANALYSIS ........................................................ 73 Purpose o f the Study .................................... Research Questions ........................................ 73 74 iv Page V. Findings o f the S tu d y ............................................................ Introduction ........................................................................ Contingency T a b l e s ........................... One-Way Univariate Analyses o f Variance .................... Summary........................................................................................ 80 80 81 83 91 CONCLUSIONS AND NEED FOR FURTHER RESEARCH........................ 94 Summary........................................................................................ Statement of the P ro b le m ................................................ Need fo r the S t u d y ............................................................ Review of the L i t e r a t u r e ............................................... Method and D e s ig n ................................................................ Analysis of the D a t a ...................................■.................. Discussion of Findings .................................................... Conclusions of the Study.................................................... Reflections From the Study................................................ Need fo r Further Research....................... 94 94 95 95 97 99 101 105 107 109 APPENDICES................................................................................................... Ill A. QUESTIONNAIRE D A T A .................................................................... 112 B. MEAP CLASSROOM-LISTING REPORT ................................................ 127 BIBLIOGRAPHY ............................................................................................... v 129 LIST OF TABLES Table 3.1. 3.2. 3.3. 3.4. 4.1. 4.2. 4 .3. 4 .4. 4 .5. 4 .6. 4.7. Page Matrix fo r Determining Correlation of MEAP Objectives With the Skill-E xercise Objectives Found in Workbooks and Teachers' Manuals of Three Reading S e r ie s ............................................................................................... 65 D istribution of Objectives for S k ill Exercises Across Reading Series ................................................................................ 67 Matrix fo r a 3 x 5 Design fo r Comparing Treatment Groups' Mean Scores on Categories of MEAP Objectives . . 68 Matrix fo r 3 x 1 5 Design for Comparing Treatment Groups' Mean Scores on Categories of Questionnaire Responses.................................................. ..... ............................... 70 Correlation of MEAP and "Other" Objectives With Objectives of S k ill Exercises Found inWorkbooks and Teachers' Manuals of Three ReadingSeries .................... 82 Distribution of Objectives for S k ill Exercises Across Reading Series ................................................................................ 83 Analysis of Variance fo r MEAP Test Scores by Category of MEAP Objectives ........................................................................ 84 MEAP Test Mean Scores onObjective Category 4 for the Three Treatment Groups ...................................... 85 Correlation o f MEAP Objective Category 4, C ritic a l Reading, With Number of Matching Reading-Series O b je c tiv e s ........................................................................................ 86 Analysis of Variance fo r Teachers' Scores on Question­ naire Categories in D is tric ts Z, Y,and X ........................... 87 Teachers' Mean Scores on the S ta tis tic a lly S ig nificant Questionnaire Categoriesfo r D is tric ts Z, Y,and X . . . 89 vi CHAPTER I INTRODUCTION TO THE STUDY This chapter contains a statement of the need for the study, a statement of the problem, testable hypotheses, a statement of the purpose, the significance of the study, definition of terms, lim ita ­ tions and assumptions, and summary. Statement of the Need for the Study The need for this study was evidenced by public concern for accountability of educators and by the back-to-basics movement. These issues have given rise to the need for specifying what is being taught and precisely what is being accomplished, i . e . , what students are actu ally learning. Various research studies, national polls, and the news media have indicated that the citizen ry is concerned about the quality and quantity of instruction in the classroom.^ "Accountability" seems to be the watchword; people want to be sure th e ir tax dollars are really providing the services that w ill ensure students have the competen. . cies that are essential for daily liv in g . 2 ^Robert L. Ebel, "The Case for Minimum Competency Testing," Phi Delta Kappan (April 1978): 546. 2 Ned B. Lovell, Rodney P. Riegle, and Clinton R. Bunke, "Minimal Competency Testing: Hopes, Fears, and Fallacies," The Educa­ tional Forum 45 (January 1981): 199-206. 1 2 The need to emphasize fundamental learning has resulted in the back-to-basics movement. This has caused concerned school o f f i ­ cials to implement policies that are designed to raise student achieve­ ment. The curriculum does make a difference in what students learn; educators, therefore, need to address the increasing concerns fo r the quality of education by demonstrating that the basic objectives of education are being accomplished through the use of selected curricu­ la r m aterials. Students are expected to demonstrate, through various types of evaluative programs, what they have learned. Robert L. Ebel, an authority on educational measurement, maintains that the best way to determine how much learning has occurred is to observe, through te s t­ ing, how successfully the student can cope with the tasks that require learning. 3 Roger Farr, an authority on reading, suggests that i f an achievement test is adequate, i t must sample the same behaviors as those developed in the instructional program. 4 Achievement cannot be expected on topics that are tested but not taught. A high correlation between curriculum and te s t content is essential i f the te s t is to measure adequately what has been taught. I f the correlation between 3 Robert L. Ebel, The Uses of Standardized Testing, Fastback 93 (Bloomington, Indiana: Phi Delta Kappa Educational Foundation, 1977), p. 8. 4 Roger Farr, Reading: What Can Be Measured?, An IRA/ENA Knight Research Fund Monograph (Newark',' Del aware": The International Reading Association, In c ., 1969), pp. 7-36. 5 Donald Freeman, Therese Kuhs, Lucy Knappen, and Andrew Porter, A Closer Look at Standardized Tests (East Lansing: In s titu te for Research on Teaching, Michigan State U niversity, November 1978), pp. 1-10. 3 the two is low, the test w ill be insensitive to achievement gains that the curriculum produces. Thus, the test w ill have l i t t l e value in the basic purposes of evaluation—namely, decision making.7 Inconsis­ tencies between these two may result in inaccurate estimations of p student achievement o f the goals of the curriculum. A study of the relationship between reading objectives and the objectives tested in the Michigan Educational Assessment Program Read­ ing Test is important because reading is basic; i t is the communicag tion process on which success in school depends. The basic purpose of reading is comprehension, which is essential fo r learning most other content. include: The components of comprehension id e n tifie d by Farr (1) knowledge of word meanings, (2) a b ility to select appropriate meanings of words or phrases in a contextual se ttin g , (3) a b ility to follow passage organization and to id e n tify antecedents and references in i t , (4) a b ilit y to select the main thought, {5} a b il­ it y to answer questions that are s p e c ific ally stated in a passage, (6) a b ilit y to use other words to answer a question concerning a passage, (7) a b ility to draw inferences from a passage about its ^Kasten G. Tallmadge and Donald P. Horst, "The Use o f D if­ ferent Achievement Tests in the ESEA T itle I Evaluation System" (paper presented at the 62nd Annual Meeting of the American Education Research Association, Toronto, Ontario, Canada, 1978), pp. 4-8. 7Ralph W. Tyler and Sheldon H. White, Testing, Teaching, and Learning (Washington, D.C.: The National In s titu te of Education, U.S. Department of Health, Education, and Welfare, 1979), pp. 4, 10. 81bid. , p. 10. □ Mary Anne H a ll, Jerilyn K. Ribovich, and Chris J. Rameg, Readinq and the Elementary School Child (New York: D. Van Nostrand Co., 1979), p. 4. 4 content, (8) a b ility to recognize lite r a r y devices used in a passage and to determine its mood and in te n t, and (9) a b ility to determine the w rite r's purpose, in te n t, and point of view, i . e . , to draw inferences about a w r it e r .^ The o ffic ia ls in Michigan's State Department o f Education have responded to public concerns regarding basic s k ills by develop­ ing minimal reading-performance standards and by developing a c rite rio n referenced testing program with which to sample the reading behaviors that students should have acquired by a given grade le v e l. For example, a ll fourth-grade students are tested on those objectives they should have achieved in the previous grades, f ir s t through th ird . I f the students have not attained the specified objectives, this should be accomplished as soon as possible. In this way, data that are com­ mensurate with the instructional content and purposes of Michigan's schools can be produced. 11 The State Department of- Education holds the position that a common core of objectives, which transcend local d is tr ic t boundaries, do exist and that schools are responsible fo r helping students a tta in them. 12 The Department has id e n tifie d the common core of objectives through input from representatives of the Michigan Reading Association, the Michigan Council of Teachers o f Mathematics, the Michigan Department 10Farr, PP- 53-54. ^ Michigan Educational Assessment Program, Technical Report, Vol. 1 (Lansing: Michigan State Board of Education, 1980), p. 28. 5 of Education, teachers, curriculum sp e c ia lis ts , school administrators, and other c itize n s . 13 I t can serve, therefore, as a credible guide fo r determining the essential objectives fo r reading. The purpose of the Michigan Educational Assessment Program (MEAP) is to assess va lid ly the outcome of Michigan's instructional programs,^4 These programs are largely determined by the curriculum materials that are purchased and used. 15 I t is important, therefore, to determine the extent to which the Michigan Educational Assessment Program's objectives are included in the instructional reading series and whether the d is t r ic t using the series most congruent with the MEAP objectives w ill produce the best MEAP test results. Statement of the Problem In this study the w rite r compared the 25 MEAP minimal reading objectives with those represented in the grades one through three basal reading exercises o f the three most widely used series in Michi­ gan, as determined by Market Data R e triev a l, I n c . , ^ and Nicholas P. Criscuolo. 17 S p e c ific a lly , the researcher attempted to ascertain the following: 13Ib id . 14Ib id . 15 Marda Woodbury, Selecting Instructional M aterials, Fastback 110 (Bloomington, Indiana: Phi Delta Kappa Educational Foundation, 1978), p. 26. ^Market Data R etrieval, In c ., Reading K-8 Survey, HM Co. Market Research Report No. 17 (New Yorkl Market Data R etriev al, In c ., 1977), p. 97. ^Nicholas P. Criscuolo, Improving Classroom Reading Instruc­ tion (Worthington, Ohio: Charles A. Jones Publishing Co., 1973), p. 23. 6 1. How many objectives of the reading-skill exercises found in workbooks and teachers' manuals match MEAP objectives? 2. What reading series has the greatest percentage of s k ill - exercise objectives that are congruent with MEAP objectives, as deter­ mined by a panel o f reading experts? 3. W ill the degree of congruence between reading-exercise objectives and MEAP objectives a ffe c t MEAP test results? 4. Are methods and strategies used in teaching reading lik e ly to influence the MEAP test results? Hypotheses General Hypothesis I There w ill be no difference between the MEAP reading-test mean scores of classes in the d is tr ic t using the most congruent read­ ing series and the mean scores of the classes in the d is tric ts using the other two reading series on each of the dependent variables. Operational HI a : There w ill be no difference between the MEAP reading-test mean scores of the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores o f the classes in the d is tric ts using the other two reading series on vocabulary meaning. Operational HIb: There w ill be no difference between the MEAP reading-test mean scores of the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two read­ ing series on lit e r a l comprehension. Operational Hlc: There w ill be no difference between the MEAP reading-test mean scores of the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on in fe re n tia l comprehension. 7 Operational Hid: There w ill be no difference between the MEAP reading-test mean scores o f the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on c r it ic a l reading. Operational Hie: There w ill be no difference between the MEAP reading-test mean scores o f the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on related study ski 11s. General Hypothesis I I There w ill be no difference between the methods and strategies of teachers in the d is t r ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on each of the dependent variables, according to mean scores on teacher-questionnaire responses. Operational H2a: There w ill be no difference between the methods and strategies of teachers in the d is t r ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on classroom enrich­ ment, according to mean scores on teacher-questionnaire responses. Operational H2b: There w ill be no difference between the and strategies of teachers in the d is t r ic t using the most reading series and the methods and strategies of teachers d is tric ts using the other two reading series on homework, to mean scores on teacher-questionnaire responses. methods congruent in the according Operational H2c: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on teacher-made and other m aterials, according to mean scores on teacher-questionnaire responses. Operational H2d: There w ill be no difference between the methods and strategies o f teachers in the d is t r ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on parental involve­ ment, according to mean scores on teacher-questionnaire responses. 8 Operational H2e: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on tim e, according to mean scores on teacher-questionnaire responses. Operational H2f: There w ill be no difference between the methods and strategies o f teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on organizational patterns, according to mean scores on teacher-questionnaire responses. Operational H2g: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on independent learn­ ing, according to mean scores on teacher-questionnaire responses. Operational H2h; There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on student in te ra c tio n , according to mean scores on teacher-questionnaire responses. Operational H 2i: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on outside enrichment. according to mean scores on teacher-questionnaire responses. Operational H 2j: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on basal m aterials, according to mean scores on teacher-questionnaire responses. Operational H2k: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on s ta ff support. according to mean scores on teacher-questionnaire responses. Operational H21: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on language develop­ ment, according to mean scores on teacher-questionnaire responses. 9 Operational H2m: There w ill be no difference between the methods and strategies o f teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on w ritten work, according to mean scores on teacher-questionnaire responses. Operational H2n: There w ill be no difference between the methods and strategies o f teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on test uses, accord­ ing to mean scores on teacher-questionnaire responses. Operational H2o: There w ill be no difference between the methods and strategies o f teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on rewards, according to mean scores on teacher-questionnaire responses. Statement of the Purpose A criterion-referenced MEAP testing procedure was designed to re la te student achievement to specific standards of performance accord­ ing to a predetermined c rite rio n or set of c r it e r ia . The MEAP testing instrument was constructed to y ie ld results that would be interpreted in terms of essential minimal reading behaviors. These c r ite r ia had to be based on the reading programs used in the classroom because research has shown that what is taught is determined prim arily by the instructional-program materials used in a school system. 18 This study was conducted to provide information concerning the degree to which primary-grade minimal reading objectives, as determined by the MEAP, are represented in teachers' manuals and workbooks of the three most widely used basal reading series in 18 William A. Mehrens and Robert L. Ebel, Some Comments on Criterion-Referenced and Norm-Referenced Achievement Tests, NCME Measurement in Education, Vol. 10, No. 1 (Washington, D.C.: National Council on Measurement in Education, Winter 1979), pp. 4-5. 10 in Michigan, as determined by Market Data R etriev al, In c ., Nicholas P. Criscuolo. 20 19 and These include Ginn and Co.; H o lt, Rinehart, and Winston Co.; and Houghton-Mifflin Co. The d is tr ic t that used the reading series thatwasmost congruent with the MEAP objectives was expected to show higher achievement scores than were those d is tric ts using the series less closely aligned with MEAP objectives. Importance of the Study The v a lid ity of the Michigan Assessment Test depends on whether the te st samples the same behaviors as those developed through the reading curriculum or course of study. I t is generally agreed that the purpose of criterion-referenced testing is to determine whether students have learned what they have been taught. The extent of the re la tio n ­ ship between instructional materials and the MEAP should, therefore, be valuable to the following: (1) those revising the Michigan Educational Assessment Test, (2) local school o ffic ia ls who are using the test to ascertain whether students are attaining minimal objectives, (3) teach­ ers who are s trivin g to id e n tify and define more clea rly what is essential fo r students to learn , and (4) publishers who need to know what is essential to be included in the planning and designing of reading series. D efin itio n of Terms The following definition s apply throughout this research project. 19 Market Data R e trie va l, In c ., p. 97. 20 Criscuolo, p. 23. 11 Basal reading series is a set o f instructional m aterials, which is provided in elementary schools and which offers systematic guidance in developing basic reading s k ills by carefully planned sequences in each level of the series. Basic reading s k ills are those s k ills that are fundamental for la te r learning and achievement. These include word study, comprehen­ sion, and study habits. Content v a lid ity is the degree to which a test measures what i t purports to measure. I t is sometimes defined as truthfulness. Criterion-referenced te st is a te st in which performance is judged, re la tiv e to some absolute d e fin itio n of mastery or adequacy, on a predetermined c rite rio n or set o f c r it e r ia . I t is interpreted with respect to the proportion of correct responses. The term "objective referenced" is used synonymously. Curriculurn, as defined fo r this study, is the formal course of study used in schools. Evaluation is the systematic collection and interp retatio n of information regarding student progress or lack of progress toward edu­ cational objectives. Instructional v a lid ity refers to the adequacy of the time and the degree of emphasis used in teaching a specified s k ill or concept. Measurement is the quantification o f observedlearner behavior. The results are commonly used to underscore alleged educational d e fi­ ciencies or accomplishments. Minimal competency test is a device used for measuring the achievement o f those s k ills assumed to be necessary fo r liv in g . It 12 is a criterion-referenced instrument by which an individual is assessed re la tiv e to a predetermined standard or c r it e r ia . Homing population refers to an exemplary population th a t is selected as representating the larg er population fo r whom the te s t w ill eventually be used. This group is tested before the release of the te s t, and the scores are used as the standard by which to measure the performance o f other students o f the same age and grade. Primary grades, as defined fo r this study, are grades one through three. Primary years o f schooling may be used synonymously. R e lia b ility refers to the consistency with which equivalent forms of a te s t measure the same s k ills or concepts. S k ill exercise, as defined fo r this study, is a ll subject matter grouped under one t i t l e or heading in workbooks and teachers' manuals. A s k ill exercise covers one or more pages in a workbook; in the teachers1 manuals, each s k ill exercise has a t i t l e and a general set of directions. Standardized achievement tests are measurement devices designed by professional te s t constructors and drawn from a broad sampling of general content to assess pupils' knowledge and s k ills a t a p a rtic u la r time. Standardized reading achievement test measure word recognition, vocabulary, comprehension, related study s k ills , and sometimes the rate of reading. Survey te s ts , as defined fo r this study, are norm-referenced tests that are designed to cover a broad range o f achievement and to designate generally what students have learned as compared to a speci­ fied norming population. 13 Limitations and Assumptions The findings of this research study were lim ited by the following: 1. The Michigan Educational Assessment Program standards fo r identifying minimal competencies may not include a ll essential s k ills for d aily liv in g . 2. Other reading experts may not agree that the objectives o f the s k ill exercises are the same as were determined by the panel of reading experts used in this study. 3. Only basal reading workbooks and teachers' manuals were used to determine whether students were taught the essential MEAP objectives. 4. Disproportionate numbers o f classes and teacher responses in the three d is tric ts provided a greater likelihood o f sampling error in some treatment groups than in others. 5. Variation in types o f students fo r whom co llective scores were reported on the MEAP Classroom Listing Reports may have affected mean scores o f the treatment groups. 6. The qu ality of instru ctio n , as determined by the teachers' methods, may lim it or enhance the degree to which the s k ill exercises help students a tta in the objectives that are tested for on the MEAP te s t. 7. Teachers used a variety of strategies and m aterials; these were considered through the use o f a questionnaire to determine teachers' perceptions o f th e ir own instructional methods and strategies. Instructional v a lid ity was determined by each instructor in d iv id u a lly . 14 But how an instructor perceived his/her own methods may not neces­ s a rily have represented what actually occurred in the classroom, 8. Differences in students' environments, a b ilit ie s , a t t i ­ tudes, economic opportunities, and the d is tr ic t's a b ility to provide educational experiences may p a r tia lly explain the differences in students' achievements on the MEAP te s t. These may not have been influenced by teaching methods and strategies or materials used. 9. Gross reporting of scores may have accounted fo r ind is­ cernible differences in MEAP te s t results. 10. The teachers' manuals and workbooks that were used repre­ sented the state of knowledge at the time they were published. Methods recommended fo r teaching s k ill exercises may not have been closely aligned with current teaching practices. 11. Differences in teacher responses may be attributed to the manner in which the questionnaires were administered. In D is tric t Y, principals preferred to collect responses, but in D is tric ts Z and X, responses were mailed d ire c tly to the researcher. Summary The f i r s t and second research questions are based on the need to include b a s ic -s k ill objectives in the reading programs that are used in most schools. Concern about accountability has given rise to the need for specifying what has been taught and precisely what has been learned. Research has supported the need fo r cle arly defining those objectives that students need to a tta in . 15 The f i r s t hypothesis, based on the th ird research question, has been supported by authorities in both the fie ld of measurement and the fie ld of reading. This hypothesis was founded on the supposition that correlation of curriculum and te s t content is necessary to measure adequately what has been taught. Supportive research indicated that what is taught is largely determined by the reading series that is selected and used fo r instructional purposes. Achievement gains can hardly be expected to be meaningful i f the objectives of a testing program are not congruent with the s k ills and concepts taught. The second hypothesis originated with the fourth research question. This hypothesis was based on the premise that teaching methods and strategies, instructional m aterials, and factors outside the school environment influence learning. Research has not shown, however, whether individual learning depends more on one variable than another. Although this study dealt with only some o f the variables that influence how and what students learn in reading, i t was none­ theless important. Each piece o f data that contributes to a greater understanding of how students become p ro fic ien t readers provides use­ ful information. I t was, therefore, important to examine the re la ­ tionship between teachers' perceptions of what occurs in the classroom and the reading objectives that are tested. CHAPTER I I REVIEW OF LITERATURE Introduction Considerable controversy has been generated over the various means o f testing and over whether these tests are v a lid —whether they actually te s t what is taught in schools. Concern about accountability has given rise to concern about te stin g , content taught, and whether these two are related. One of the major goals for primary students is proficiency in reading because i t is an essential tool fo r furth er learning. Teachers are expected to teach students to become flu en t readers and to prove that this goal has been accomplished. Testing is one measure­ ment device that is commonly used in evaluating student reading per­ formance. The relationship between what is taught and the testing instrument used is important i f achievement is to be adequately assessed. This chapter, therefore, addresses issues related to the following: (1) beginning reading, (2) criterion-referenced te s ts , and (3) Michigan's Accountability Model, which was designed to ascer­ tain whether students were becoming competent readers. 16 17 Beginning Reading H a ll, Ribovich, and Rameg defined reading as a communication process in which w ritten language symbols are used to arrive at a meaningful interp retatio n of the author's intentions, a ttitu d e s , b e lie fs , and/or feelin gs, as well as to arrive at a meaningful in te r ­ pretation of the reader's s k i l l , experience, and in te lle c tu a l pur­ pose.^ Its function is shared with w ritin g , speaking, and lis te n in g , Both Hall 2 and Chall 3 concurred that reading is central to educational achievement because success in reading is essential fo r learning other content. In the same vein, Criscuolo reiterated that a basic reading program is most important because i t develops s k ills needed to func­ tion e ffe c tiv e ly in a ll c u rricu lar areas.4 Chall cited Bloom's studies as evidence that reading plays a dominant role in the la te r success of children's academic achievement. In these studies, Bloom found that fa ilu re to learn to read by the end 5 of grade one is predictive of la te r fa ilu re . This study, however, ^Mary Anne H a ll, Jerilyn K. Ribovich, and Chris J. Rameg, Reading and the Elementary School Child (New York: D. Van Nostrand Co., 1979}, p. 8. 2 I b i d , , p. 4. 3Jeanne Chall, Reading 1967-1977: A Decade o f Change and Promise, Fastback 97 (Bloomington, Indiana: The Phi Delta Kappa Edu­ cational Foundation, 1977), p. 6. 4 . Nicholas P. Criscuolo, Improving Classroom Reading Instruc­ tion (Worthington, Ohio: Charles A. Jones Publ ishinglCo~., 1973J, p. 23. 5 B. S. Bloom, Human Characteristics and School Learning, quoted by Jeanne Chall in Reading 1967-1977: A Decade of Change and Promise, p. 6. 18 lacked substance because i t fa ile d to address the major issue of the original cause of fa ilu re . The philosophy o f teaching reading and theories o f how c h il­ dren learn to read have changed considerably over the past two decades. Vocabulary in basal reading materials is no longer rig id ly controlled, and units of meaning are believed to be of greater value in comprehend­ ing what is read. During the early part of the twentieth century, rig id vocabu­ lary control in children's basal readers was advocated by Thorndike and other pioneer researchers. I t was believed that young children should be given stories and textbooks that contained easier words and a lo t o f re p e titio n ; the purpose of this was to a lle v ia te fru s tratio n in the process o f learning to read. Acceptance of this theory of vocabulary control was evident, as the publishing industry employed Thorndike's techniques in word-frequency counts. Now th at much more is known about the process of learning to read, researchers and theorists question the use of controlled vocabu­ lary and simple syntax in basal readers. I t is now believed that natural-sounding sentences provide contextual cues that help children recognize a wide range of words, not ju s t core vocabulary or frequently encountered sight words. Children may, in fa c t, respond to printed messages with immediate understanding, rather than going through the laborious process of iden tifying individual words. Present-day advo­ cates of this whole-language approach do not consider instant c John Guthrie, "Vocabulary Control," The Reading Teacher 33 (November 1979): 240-42. 19 recognition of core vocabulary relevant to measuring the reading process. 7 Instead, they feel that measuring oral-reading errors in relation to self-correction strategies is more appropriate. English is a positional language. The order of sentences as well as the order of words and phrases within sentences plays an important role in reading comprehension. interaction of sentences. Meaning is derived from the Researchers have found that attempts to make reading easier by using simple sentence structure have only g served to complicate the process. Omission of e x p lic it connective words--such as because, instead, fo r , and while--makes the re la tio n ­ ship between sentences vague or incomprehensible i f the young reader cannot discern im p lic it relationships. I f the reader fa ils to under­ stand the im p lic it relationships integral to the te x t, both recall and comprehension become more d i f f i c u l t . Lundsteen studied the e ffe c t of simple and complex reading materials on the level of thinking of 190 Berkeley elementary stug dents. She sought to determine whether i t was the materials or the level of student maturity that made the difference in th e ir selection of abstract, functional, or concrete word and paragraph meanings. She used three measures having abstract, functional, and concrete choices. These included Choose a Meaning Test, Depth of Meaning ^Joyce Hood, "Sight Words Are Not Going Out of Style," The Reading Teacher 33 (January 1977): 379-82. 8I b i d . , pp. 380-82. g Sara W. Lundsteen, "Levels of Meaning in Reading," The Read­ ing Teacher 28 (December 1974): 268-72. 20 Test, and Creative and C ritic a l Reading (paragraphs) Test. She found that both primary- and upper-grade students made s ig n ific a n tly more abstract choices with paragraph measures than with word measures. She concluded that cognitive-level reactions were more relevant to higher-level thoughts that are encompassed in larger units of meaning, regardless of maturity level. Lundsteen's study suggested that students, both younger and older, need exposure to abstract reading material and to provocative questions about the material. From these ta c tic s , a child can develop a repertoire of various kinds of meanings, which w ill encourage problem­ solving behavior. As early as 1969, Clay studied the oral-reading strategies of children who had been taught to read by the whole-language approach. She demonstrated children's achievements by reporting errors and s e lfcorrection rates for the median children in four a b i l i t y groups.^ The highest group had an accuracy rate of 95 percent in s e lf ­ corrections. She concluded that this group was using other contextual cues without systematic repetition of a core vocabulary. Using the same self-correction system, she found that the lowest group was able to correct only 5 percent of its errors. This lowest group evidently needed a larger core vocabulary to provide contextual clues for the recognition of less-fam iliar words. This suggested that a percentage of children do need a core vocabulary of known words. ^M arie M. Clay, "Reading Errors and Self-Correction Behavior," British Journal of Educational Psychology 39 (1969): 47-56. 21 Biemiller researched the importance of teaching children the basic strategy of risk taking in word pronunciation. 11 He analyzed the oral-reading errors of f i r s t graders and found that the errors could be categorized in three phases. At f i r s t , beginning readers primarily used context clues to guide th e ir guesses. In the second phase, children frequently made no attempt to guess at a l l . In the third stage, the students used both graphic and contextual restraints to aid in word id e n tific a tio n ; in this stage, the number of nonresponse errors decreased. According to this study, children f i r s t approached the read­ ing task confidently but gradually became less w illin g to risk guess­ ing as they became aware of the constraints placed on th e ir guesses. In the third phase, children, though hesitant, used multiple clues to word meanings and made in te llig e n t guesses with few errors. The children needed a great deal of encouragement and training to develop the confidence to take risks necessary to make the educated guesses that successful readers make. The psycholinguistic model of reading stresses the interac­ tion between thought and language. I t is based on units of meaning that are derived from the whole context, rather than emphasizing word meaning and/or pronunciation in isolation. This process combines cog­ nitive psychology and linguistics to analyze and understand the language and thinking process. Psycholinguists contend th a t, since ^ A . Biem iller, "The Development of the Use of Graphic and Contextual Information as Children Learn to Read," Reading Research Quarterly 6 (Fall 1970): 75-96. 22 iden tificatio n of meaning is the goal of reading* the human brain must rely on syntactical rules to bridge the processing system between visual information and the deeper structure of the brain from which meaning is derived. 12 The interaction of words in sentences depends on the syntax of the language. Even though the speech of young children reflects basic syntactic information, there is evidence that children continue to grow throughout elementary-school years in th e ir a b ilit y to understand and use syntactic structures. I t is now believed that the fa ilu re of beginning readers to organize an author's words into meaningful units creates comprehension problems. Psycholinguists, therefore, advocate the use of whole units of meaning rather than learning or using words in isolation. Even though a great deal more is known today about the read­ ing process, Hall stated that basals are s t i l l the most widely used materials for teaching reading in the United States since the McGuffey Readers were introduced in 1840. 13 McGuffey Readers were character­ ized by the use of sequenced materials and controlled vocabulary, according to the grade level for which the book in the series was designed. Woodbury substantiated the fact that textbooks are s t i l l the major means of conveying curricular content. 12 14 They are in flu e n tia l Charles Cooper and Anthony Petrosky, "A Psycholinguistic View of the Fluent Reading Process," Journal of Reading 20 (1976): 184-207. 13 H a ll, Ribovich, and Rameg, p. 8. 14 Marda Woodbury, Selecting Instructional M aterials, Fastback 110 (Bloomington, Indiana: Phi Delta Kappa Educational Foundation, 1978), p. 26. 23 in standardizing the curriculum by grade le v e ls, and they influence the teaching approach that is used. Often they are validated by f ie ld tests in schools before publication. Woodbury further stated that since standardized achievement tests tend to be based on textbook content, scores on achievement tests are correlated with mastery of te xts . ^^ In a survey of 1,300 teachers throughout the United States, Spache found that 95 to 98 percent of a ll primary-grade teachers and 80 percent of the intermediate-grade teachers used basal readers. 1 fi Spache further noted that seven out of ten elementary classes used the basal reading materials as th e ir only major instructional program. Rosecky studied the major instructional tools used by 61 teachers in two d is tric ts and found evidence suggesting th a t, not only are basals the core of the reading in most American elementary schools, but teachers are also expected to rely on the basal guide­ books for daily reading instruction.^7 Of those teachers studied, she found that 90 percent used the basal guidebook suggestions most or a ll of the time. Within that same group, 75 percent omitted a ll or part of the directed reading a c tiv it ie s . The most frequently omitted were such follow-up enrichment a c tiv itie s as dramatization, composing, 15I b i d . , p. 26. 1fi George D. Spache, The Teaching of Reading (Bloomington, Indiana: Phi Delta Kappan, In c ., 1972), pp. 35-36. 17Marion Rosecky, "Are Teachers Selective When Using Basal Guidebooks?" The Reading Teacher 31 (January 1978): 273-74. 24 and reading to students. The reason cited was lack of time for these additional a c tiv itie s , which were considered "extras.11 Coordinated textbooks, workbooks, and teachers' manuals form I Q the core of American reading programs. The overall purpose of these programs is to provide comprehensive sequential reading materials. The textbooks are accompanied by workbooks, which contain important supplementary features and which are designed to strengthen and extend the s k ills presented in textbooks. These s k ills include the following: (1) decoding a c tiv it ie s , (2) syllabication, (3) structural and textual a n a ly s is , (4) measurement s k i l l s , and (5) comprehension s k i l l s . 19 Spache and Spache studied reports of research done in 36 classes of 18 Los Angeles schools; in these reports, comparisons were made between standardized reading test scores when teachers used work­ books and standardized reading test scores when teacher-made materials were used. 20 I t was reported that students using workbooks in the second and third grades had higher vocabulary and comprehension scores, but f i f t h and sixth graders showed no difference in reading t e s t scores, with or without workbooks. The groups that did not use workbooks used teacher-prepared materials. After Spache and Spache studied the reports, they studied teachers' attitudes toward preparing th e ir own teaching materials. 18 George D. Spache and Evelyn B. Spache, Reading in Elemen­ t a r y Schools (Boston: Allyn & Bacon, In c ., 1973), pp. 146-70. 19 Albert J. Harris and Edward R. Si pay, How to Increase Reading A b ility (New York: David McKay Co., 1976^! 20 Spache and Spache, p. 170. 25 They found that teachers preferred not to prepare th e ir own materials but wanted to use workbooks that were closely related to th e ir read­ ing programs. The teachers believed that coordinated workbooks pro­ vided concrete evidence of reading growth in both work habits and mechanical s k ills at primary levels. Spache and Spache concluded that the quality of teacher-made materials might have been affected by teachers' a ttitu d es , thereby affecting students' achievement. The conclusions they reached might have been substantially d iffe r e n t i f a third control group had used a combination of teacher-made materials and workbooks; attitudes as well as test results might have been d i f ­ ferent. Whether materials are made or selected from among various commercial materials that are available, particular attention needs to be given to materials used in the primary grades because research­ ers have found that most basal comprehension s k ills are i n it ia t e d in the primary grades. Jenkins and Pany stated that comprehension s k ills that received the greatest attention during the middle grades were introduced and taught during the f i r s t years of instruction. 21 H a ll, Ribovich, and Rameg also discovered that intermediate- and middle-grade students generally are refining the basic reading strategies th a t were learned at the primary levels. 22 Rosenshine, too, concluded th a t most 21 Joseph R. Jenkins and Darlene Pany, Teaching Reading Comprehension in the Middle Grades, Reading Education Report No. 4 (Urbana-Champaign: University of I I 1in o is , January 1978), p. 1. (ED 151 756) 22 H a ll, Ribovich, and Rameg, p. 186. 26 s k ills practiced in the la te r elementary years were not new but ju s t a continuation of the progress begun e a r lie r . 23 Since the i n i t i a l introduction to desired reading behaviors begins in the primary grades, i t is important to consider not only the appropriate content, but also the proper use of that content. Both are important in producing the desired outcomes. McClutcheon researched the question of how teachers plan a c tiv itie s fo r students. She found that teachers did not f i r s t f o r ­ mulate objectives and then choose the appropriate learning a c t i v i t i e s , organize them, and select evaluative procedures accordingly. 24 Rather, they focused on the a c tiv it ie s suggested or provided for in the teachers' manuals before considering the objectives of learning. McClutcheon cited this as one of the major weaknesses of using basals when trying to provide a balanced program. In fle x ib le use of basals and indiscriminate use of the accom­ panying workbooks were two of the problems that Criscuolo found in or his research to be related to inappropriate use of basals. He surveyed 940 elementary students in 34 classrooms and found that only 9 percent changed reading groups during the school year. Immobility 23 B. Rosenshine, "Skill Hierarchies in Reading Comprehension," in Theoretical Issues in Reading Comprehension, ed. R. J. Spiro, B. C. Bruce, and W. F. Brewer [H ills d a le , N.jT: Erlbaum, 1978). 24 Gail McClutcheon, "How Do Elementary School Teachers Plan? The Nature of Planning and Influences on I t , " The Elementary School Journal 81 (September 1980): 4-10. 25 Criscuolo, pp. 75-77. 27 was diagnosed as resulting from in fle x ib le use of a single set of basal reading materials. He stated that teachers tend to use a single reader for the entire class and to keep students in the same reading group regardless of growth. Some authorities in the f ie l d advocate the use of basals as appropriate reading materials. 2fi Basals continue to be a major factor in reading instruction in the nation's schools because they provide the basic organizing thread to ensure that pupils are exposed to an ordered sequence of reading s k ills . 27 Criscuolo emphasized that i t is advantageous to use basal series because they are adaptable to the theory of continuous child growth and development and because they provide economy of instruction for teachers who need the guidance of a basic reading program. 2R Duffy and Sherman, likewise, advocated the use of basals because there is a defined structure in them that does not depend on teachers' judgments to determine the sequence or content of instruction. 29 Spache and Spache recommended the use of basal reading series because they have many advantages, including the following: (1) sys­ tematic guidance in planned sequential learning, (2) subject matter based on common childhood experiences, (3) a superior program to that 26 Spache and Spache, pp. 160-63. 27 Ohio: Jack Bagford, Instructional Competence in Reading (Columbus, Charles E. M errill Publishing Co., 1975), pp. 51-52. 28 29 Criscuolo, p. 23. Gerald G. Duffy and George B. Sherman, How to Teach Reading Systematically (New York: Harper & Row, 1973), pp. 238-39. 28 which a teacher can create, (4) the development of readiness proce­ dures in sequential steps, (5) core vocabulary, (6) material scaled in d if f ic u l t y , (7) well-rounded selections, and (8) useful accompanying workbooks to practice and test the development of s k i l l s . 30 Representing the opposing point of view, Durkin questioned whether basal reading series should be used. She stated that the problem with basals is that "what is done with comprehension is com­ monly concerned with assessment rather than with instruction. ever the focus, most a c tiv itie s lead to written exercises." What- 31 Huey, too, is s t i l l concerned with comprehension rather than specific vocabulary as he was more than 40 years ago. He said that the rigid vocabulary control in basals is contrary to the natural process of learning to read. 32 Children's reading vocabulary should come from th e ir own environment and should pave the way fo r re fle c tiv e thinking, he contended. He further emphasized that i t is not necessary for c h il­ dren to be able to pronounce words, s p e ll, or write them in order to grasp th e ir meanings in context. H a ll, Ribovich, and Rameg also cited some disadvantages of the basal reading series: They (1) present a limited vocabulary; (2) include s tilte d language for beginners; (3) included, until the 1970s, sexual stereotyping; (4) present a lack of lite r a r y merit; 30 31 (Boston: Spache and Spache, pp. 160-63. Dolores Durkin, Teaching Young Children to Read, 3rd ed. Allyn & Bacon, In c ., 1980), pp. 447-48, 482-83. 32 Edmund Burke Huey, The Psychology and Pedagogy of Reading (Cambridge, Mass.: The M .I.T . Press, 1977), pp. 348-49. 29 (5) are often used for pronunciation rather than reading fo r meaning; (6) result in a routinized reading program; (7) result in continuous a b ilit y grouping; (8) often result in enrichment materials being ignored; and (9) leave l i t t l e motivation for learning, i f used in a 33 routine manner. In summary, reading is essential to success in most educational endeavors because the s k ills that are inherent in reading are the same as those that are essential for learning other content. Textbooks are the major vehicles for conveying curriculum content. Basal reading programs are used by a majority of schools in the United States because they provide a framework whereby to struc­ ture the curriculum. At least 85 percent of the nation's primary teachers use basal reading programs a ll or most of the time. There are both advantages and disadvantages to using basal reading programs. Spache and Spache, Criscuolo, Duffy, and others recommended the use of basals because they f e l t that the advantages fa r outweighed the disadvantages. Durkin, Huey, and others f e l t that the basal approach was too narrow, in that i t limited many aspects of the reading process. Much more is known about the reading process now than even a decade ago. Thus the available research can serve as a guide for teachers in determining the manner in which reading w ill be taught. ^ H a l l , Ribovich, and Rameg, pp. 100-113. 30 Criterion-Referenced Achievement Tests Today there is great concern about educator accountability. This issue has given rise to the need for determining whether the testing instruments used to measure student achievement are adequate and appropriate. Ebel stated* "No school can do a good job and show i t is doing a good job without system atically auditing the results i t is getting." 34 I f te st content and in s tru c tio n a l content are not congruent* however, test results c o u ld re fle c t a distorted picture of the effects of instruction. 35 This suggests that tests should measure what is taught, i f an accurate representation of instruction is the purpose for administering a te st. If a test instrument does not measure achievement in a specific educational program, the growth or lack of growth measured by the te s t may be unrelated to the instruc­ tion that the student received. Only to the extent that the objec­ tives of both the test and the in s tru c tio n a l program coincide is the testing instrument a valid means of measuring the success of a program. 36 34 R. L. Ebel, The Uses o f Standardized Testing, Fastback 93 (Bloomington, Indiana: The Phi Delta Kappa Educational Foundation, 1977), p. 31. 35 Robert E. Floden, Andrew C. Porter, William H. Schmidt, and Donald 0. Freeman, Don't They All Measure the Same Thing? Consequences of Selecting Standardized' Tests (East Lansing: In s titu te for Research on Teaching, Michigan State U n iv e r s ity , July 1978), p. 1. 3G J. Wayne Wrightstone, Thomas P. Hogan, and Muriel M. Abbott, "Accountability in Education and Associated Measurement Problems," in Readings in Measurement and Evaluation in Education and Psychology, ed. William A. Mehrens (New Vork: H o l t , Rinehart, and Winston, 1976), p. 325. 31 For many years, the general public has supported the use of standardized tests for internal and external management purposes. 37 Externally, tests permit the public to keep track of what pupils and schools are doing; in te rn a lly , they can be used for placing children in appropriate instructional programs or groups. Standardized t e s t ­ ing responds to the American society's long-standing need to use human resources e f f ic ie n t ly , to reward talen t regardless of social origins, and to moderate and/or lim it the autonomy of local school d is tric ts . 38 I t is within this framework of American values that precision in meas­ urement has developed. Defining objectives and measuring outcomes of learning are in step with the trend toward greater accountability. has grown the need for criterion-referenced tests. Out of this These tests are f u l f i l l i n g an important and v ita l ro le , so the demands made on test developers to design appropriate tools to f a c i l i t a t e the measurement process have increased. The growing demands to assess the outcome of instructional programs and to hold teachers and administrators respon­ sible for actual gains in student performance have given impetus to criterion-referenced testing. 39 To date, however, test developers have not devised a method of measuring other factors that enter into 37 Lauren B. Resnick, "Introduction: Research to Inform a Debate," Phi Delta Kappan 62 (May 1981): 623-24. 38 Daniel P. Resnick, "Testing in America: A Supportive Envi­ ronment," Phi Delta Kappan 62 (May 1981): 625-28. 39 Stephen P. Klein and Jacquiline Kosecoff, "Issues and Pro­ cedures," in Readings in Measurement and Evaluation in Education and Psychology, ed. William A. Mehrens (New York: Holt, Rinehart, and Winston, 1976), p. 277. 32 the learning process and a ffect student performance--such factors as a ttitu d e, environment, and child development, over which the school has only partial influence. The need fo r specific information relating to the student's mastery of or deficiency in s k ills has caused a major s h ift from normreferenced to criterion-referenced testing. 40 Criterion-referenced procedures are gaining in popularity because they relate student achievement to predetermined objectives, and the results are in te r ­ preted in terms of specific standards of performance. During the te s t- construction process, behavioral objectives are stated f i r s t , and then items are designed to sample student a b i l it y in relationship to the objectives that are presented. Tyler said that i t is valuable to identify and define educational objectives in terms of behavior; this helps teachers recognize more clearly what they expect students to learn. 41 The performance standard set for the mastery of objectives is usually 80 percent. 42 Many people, however, believe that mastery should be determined on the basis of whether the objective is an essential step for further learning. I f accomplishment of an objec­ tiv e is necessary for further learning or i f i t is essential for the next higher step in the learning process, 80 percent is inadequate because 20 percent of the essential component is missing. ^°Bagford, pp. 91-96. 41 Kappan Interview, "The Father of Behavioral Objectives C r i t i ­ cizes Them: An Interview With Ralph Tyler," Phi Delta Kappan 55 (September 1973): 55-56. 42 H a ll, Ribovich, and Rameg, pp. 286-87. 33 C riterion- and norm-referenced testing instruments have many s im ila ritie s as well as differences. Both kinds of tests are used fo r assessing behaviors and making decisions. proficiency is required in both. A demonstration of I t is the interp retatio n, or anchor point, of the two that is d iffe re n t, according to Farr. 43 Norm- referenced tests are centered on the ty p ic a l, or average, scores; performance tends to be spread out to make comparisons possible. In criterion-referenced tests, i t is not the spread of scores that is of interest but whether students perform well enough to master the objec­ tives. Because more items are used in criterion-referenced measures, the tests are more conducive to diagnosing strengths and weaknesses in limited areas of learning, but they require more testing time. Norm-referenced and criterion-referenced tests both have special advantages that depend on the purposes for which the tests are used; but Airasian and Madaus, in response to E. L. Thorndike, concurred that criterion-referenced instruments may prevail i f education follows the course that developed in the physical sciences. 44 They cited four reasons for the growth of criterion-referenced testing: (1) growing criticism of standardized tests of achievement, (2) growing controversy surrounding grades, (3) growth of instructional technology, and (4) continuing b e lie f that a ll or almost a l l students can achieve 43 Roger Farr, "Standardized Reading Tests," in Reading for A l l , ed. Robert Karlin (Newark, Delaware: The International Reading Association, 1972), pp. 200-206. 44 Peter Airasian and George Madaus, Measurement in Education (May 1972), citing E. L. Thorndike, Seventeenth Yearbook of th¥ National Society for the Study of Education (1918). 34 competency in a ll or most subject matter. I t is generally agreed that whether or not competency can be attained by a ll depends on the level or degree of proficiency required fo r further learning. Although the trend toward using criterion-referenced testing instruments is growing, Harris and Si pay specified that one should use norm-referenced tests, as w e ll, in a complementary fashion. 45 It should not be either one or the other; both are useful since they provide different information. Norm-referenced tests help educators discern whether pupils are performing rela tiv e to th e ir grade and age expectancy in general areas of learning, whereas criterion-referenced instruments are used to determine whether students have or have not mastered specific objectives. Since i t is interpretation that makes the objectives meaning­ ful for mastery or comparative information, Canney stated that in te r ­ pretation should always be viewed in relation to other informational sources. 46 To be meaningful for instructional purposes, scores should not only re fle c t what is taught but should be examined c a re fu lly , and decisions should always be tentative. Canney further stated that measuring students' a b ilit ie s always results in some inaccurate in fo r­ mation. Not only is some inaccurate information generated when using criterion-referenced instruments, but many educators have voiced 45 Harris and Si pay, pp. 165-66. 46George Canney, "Organizing and Applying Test Results," in Reading Tests and Teachers: A Practical Guide, ed. Robert Schreiner (Newark, Delaware: The International Reading Association, In c ., 1979), pp. 53-56. 35 concerns that tests cannot measure a ll important objectives of the reading curriculum. 47 Guthrie 48 and Blachowicz 49 views concerning what can be measured adequately. presented opposing Guthrie stated that tests are limited to concrete cognitive s k ill areas because adequate measurement instruments have not been devised for higher-order cogni­ tive achievement or affective and emotional development. He expressed concern for whether this may r e s tr ic t the goals of education to those that are measurable. Blachowicz, on the other hand, acknowledged that i t is d i f f i c u l t , but not impossible, to assess a ffective objec­ tives; she maintained that these can be appropriately measured as cognitions about a ffective events and that they can be stated and ordered. Blachowicz further stated that this process is important in clarify in g goals for developing reading interest and appreciation. To determine exactly what criterion-referenced tests can or cannot do, the tests' contents need to be examined with respect to the intended purposes. Criterion-referenced tests are measurement instruments with which individuals are assessed re la tiv e to a given standard. Criterion-referenced tests permit educators to ascertain whether a student can display a clearly defined set of behaviors. The "criterion" to which the student's score is referenced is the class of 47 "A Position on Minimum Competencies in Reading," Journal of Reading 23 (October 1979): 50-51. 48 John T. Guthrie, "Research Testing: Uses and V is i b i l i t y , " Journal of Reading 23 (March 1980): 542-44. 49 Camille L. Blachowicz, "Reading Objectives: A CompetencyBased Accountability Model," The Reading Teacher 28 (April 1975): 659-61. 36 s k ills that the test is designed to measure. I f the c rite rio n - referenced test is well constructed, i t is possible to reference a student's performance to a clearly described set of behaviors so that i t can be said with certainty whether or not a pupil possesses a particular competency. 50 Criterion-referenced tests focus on specific objectives that are clearly defined in a particular learning program or specific objectives that are independent of any instructional program. The items must flow d ire c tly from the learning specifications, and they must be representative of the class of objectives being assessed. They should provide information on mastery or nonmastery that can be used for diagnosing deficits or ascertaining strengths when measuring an individual's level of proficiency. 51 When selecting and administering criterion-referenced tests, the primary consideration, once the purpose for assessment is established, is whether the instrument is v a lid . 52 There are several d iffere n t kinds of v a lid ity ; the referent type in this instance is content v a lid ity . To the extent that a te s t measures what i t purports to measure, i t is said to have content v a l id i t y . 50 W. James Popham, "Normative Data for Criterion-Referenced Tests?" Phi Delta Kappan 57 (May 1976): 593-94. 51 George Prescott, "Criterion-Referenced Test Interpretation," in Elementary Reading Instruction: Selected M a te ria ls , ed. Althea Beery, Thomas C. B arrett, and William Powell (Boston: Allyn & Bacon, I n c . , 1974), pp. 605-13. 52 William A. Mehrens and Irvin J. Lehmann, Standardized Tests in Education (New York: Holt, Rinehart, and Winston, 1980), p. 42. 37 Kavali stated th a t there are three basic factors th a t should be used to ascertain how well a test measures reading achievement: (1) an agreement about the s k i l l s , knowledge, and understanding that comprise the te st selector's d e fin itio n of reading achievement; (2) an examination o f the te s t to determine the s k i l l s , knowledge, and understanding required fo r i t ; and (3) a determination of whether the te st items correlate with the instructional objectives. Kavali emphasized, "To the extent that the course content is represented in the te s t, the te s t is considered to possess adequate content v a lid it y ." 53 One should examine the te s t instrument c a re fu lly and then determine whether i t re a lly matches what one is trying to assess. Mehrens and Lehmann stressed r e l i a b i l i t y as the second most important qu ality that a measurement device should possess. 54 They defined r e l i a b i l i t y as the degree o f consistency between two measures of the same thing. For example, i f d iffe re n t scorers measure a person's achievement on sim ilar items, one expects the outcomes to be s im ila r. The same theory applies i f one is using equivalent forms of the same te s t. Criterion-referenced tests are based on the premise that whatever is worth teaching is worth teaching to the point of mastery. Prescott suggested that this is the basic weakness of criterion-referenced 53 Kenneth Kavali, "Selecting and Evaluating Reading Tests," in Reading Tests and Teachers: A Practical Guide, ed. Robert Schreiner (Newark, Delaware: The International Reading Association, I n c ., 1979), pp. 9-23. 54 Mehrens and Lehmann, p. 42. 38 Cr te st in te rp re ta tio n , one that lim its its a p p lic a b ility . Many variables—such as pupil a b i l i t y , instructor effectiveness, and relevance of content—condition mastery. Specific lim itatio ns of criterion-referenced tests are based on the following facts: (1) mastery is a reasonable c rite rio n ; (2) each item in the test has inherent worth by design; (3) h ie r ­ archy or sequence of s k ills exists in any content area; and (4) common consensus regarding the objectives selected fo r testing exists. 56 In areas in which there are defined goals that are absolutely essential fo r daily liv in g , these might not be defined as lim ita tio n s . Many researchers have advocated criterion-referenced tests as useful for evaluating the effectiveness of instruction and for making decisions concerning appropriate instructional programs for individuals. For example, Otto and Chester stressed the need for criterion-referenced te sts , although they cautioned against poorly designed instruments. 57 They stressed the need to proceed with great care when (1) w riting objectives fo r a ffe c tiv e q u a litie s , (2) deter­ mining the performance necessary fo r proficiency, (3) specifying the universe of the task, and (4) stressing mastery, rather than reten­ tion and transfer of what is learned. ^ P re s c o tt, pp. 605-13. 5®Kavali, pp. 19-23. 57 Wayne Otto and Robert D. Chester, Objective-Based Reading (Reading, Mass.: Addison-Wesley Publishing Co., 1976), pp. 250-51. 39 Jenkins and Pany recommended the use of criterion-referenced tests as a means of overcoming "curriculum bias." 58 Their study matched the words presented in seven reading series with those pre­ sented in fiv e reading-achievement tests. Curriculum bias was evi­ denced between tests for a single curriculum as well as in a single test that was used for d iffere n t reading curricula. As a result of this study, the need to establish clearly the relationship between test and curricula was emphasized and was advocated as the necessary f i r s t step in interpreting te st results. Tallmadge and Horst studied the relationship between test results and the measurement device used. 59 They wanted to ascer­ tain whether one standardized achievement test was more effective than another in measuring achievement gains. They found that both survey types of reading achievement tests were poorly correlated with three d iffe re n t commercial reading programs and were, therefore, insensitive to achievement gains that the curricula produced. They concluded that the only valid way to assess the effects of an instructional treatment was to use a test in which the items were sampled from the same domain as the instructional exercises. In 1978 a jo in t statement concerning testing was issued from the National Conference on Achievement Tests and Basic go Joseph R. Jenkins and Darlene Pany, "Curriculum Biases in Reading Achievement Tests," Journal of Reading Behavior 10 (Winter 1978): 345-57. nq Kasten G. Tallmadge and Donald P. Horst, "The Use of D if ­ ferent Achievement Tests in the ESEA T it le I Evaluation System" (paper presented at the 62nd Annual Meeting of the American Educational Research Association, Toronto, Ontario, Canada, 1978), pp. 4-5. 40 S k ills .*^ Renowned educators, professional test developers, and legislators agreed that tests should be used to supply meaningful information about individuals, promote quality education, and help people learn. The National Council on Measurement in Education surveyed teachers nationally in 1979 to ascertain th e ir attitudes toward fil standardized testing in general. Teachers were found to be in t e r ­ ested in the knowledge that standardized achievement tests provide. The teachers considered tests of this kind to be helpful, f a i r , and useful; but at the same time, they did not feel that additional te s t­ ing instruments were necessary. Turnbull, 62 Harmer, 63 and others agreed that tests are useful and appropriate measurement devices i f the purposes for testing are determined in advance and i f the results are used as instructional aids. Kavali advised that i t is better not to use any test at a ll i f the purposes for testing have not been clearly established before 6^The National Conference on Achievement Tests and Basic Ski 11s (Washington, D.C.: The National In s titu te of Education, U.S. Department of Health, Education, and Welfare, February 1979), pp. 2529. (ED 171 784) fil Michael Beck and Frank P. Stetz, "Standardized Testing as Viewed by Test Specialists and Users" (paper presented at the National Council on Measurement in Education, Washington, D.C., 1980). 62 William Turnbull, quoted by John Guthrie in "Research Test­ ing: Uses and V i s i b i l i t y , " Journal of Reading 23 (March 1980): 542-44. 63 William R. Harmer, "The Selection and Use of Survey Reading Achievement Tests," in The Evaluation of Children's Reading Achieve­ ment, ed, Thomas C. Barrett (Newark, Delaware: Internationa1 Reading Association, In c ., 1967), pp. 53-64. 41 administering i t . 64 Harmer re ite ra te d , "Testing fo r the sake of testing can not be countenanced." 65 He reasoned that i f they are not used to promote further learning or to diagnose learning d i f f i c u l t i e s , they constitute unnecessary expense and waste time and e f f o r t - - a l l of which are ill-a ffo rd e d by our schools. Perrone pointed out that teachers need to be mindful of the fact that primary students' growth, both mentally and physically, is sporadic and uneven. 66 The s k ills that these students need for success in school are in a rather f lu i d , acquisitional stage. A great deal of fluctuation and inconsistency in learning patterns is inevitable. Perrone advocated the use of many evaluative tools over various time spans. Farr cautioned against reliance on a single testing instru­ ment because performance on any one test is a sample of behavior in 67 a given situation under a single set of conditions. Price con­ curred and emphasized that since standardized tests are only one CQ device for obtaining information, they have l i t t l e value in isolation . 64 Kavali, pp. 9-23. ^Harmer, pp. 58-62. fifi Vito Perrone, The Abuses of Standardized Testing, Fastback 92 (Bloomington, Indiana! The Phi Delta Kappa Educational Foundation, 1977), p. 74. ^Roger Farr, Reading: What Can Be Measured?, An IRA/ENA Knight Research Fund Monograph (Newark, Delaware: The International Reading Association, In c ., 1969), p. 80. fifl Gary Price, "Standardized Achievement Tests for Young Chil­ dren: An Analysis" (paper presented at the Big Ten Symposium on Early Childhood Education, Madison, Wisconsin, October 1978), pp. 1-18. 42 McDonald suggested that since an achievement test is a meas­ ure of performance on a specific t e s t, the score is inadequate to represent overall reading performance. 69 He recommended using four steps to determine a student's level of proficiency: ( ! ) carefully define the variable being measured, (2) test under conditions lik e ly to secure student cooperation, (3) compare raw scores with suitable norms, and (4) consider the possibility that scores may deviate sizably from the true scores. In the same vein, Tyler and White acknowledged that i t is not possible to test a ll educational objectives that are deemed important because no way has yet been devised to measure higher-order cognitive achievement or affective and emotional development. 70 They stressed the need to make legislators and the general public aware that measurement is restricted to concrete cognitive s k i l l s . They feared that i f this is not done, the goals of education may be restricted to those that are measurable. The International Reading Association agreed that not a ll important outcomes of the reading curriculum can be measured.^ They feared that dependence on a single test may dictate a narrowing of school curricula to ju s t test items, thus omitting other important 69 A. S. McDonald, “Using Standardized Tests to Determine Reading Proficiency," Journal of Reading 8 (1964): 58-61. ^Ralph W. Tyler and Sheldon H. White, Testing, Teaching, and Learning, Report of a Conference on Research on Testing (Washington, D.C.: National In s titu te on Education, U.S. Department of Health, Education, and Welfare, August 17-26, 1979). (ED 171 784) 71 "A Position on Minimum Competencies in Reading, pp. 50-51. 43 objectives. They advocated collecting and using various kinds of assessment data that are gathered many times during the year. Two well-known national organizations have opposed the use of standardized tests for any and a ll purposes. The National Educa­ tion Association (NEA) and the National Association for the Advance­ ment of Colored People (NAACP) have both advocated a moratorium on achievement testing because of the variety of ways the results are being used. 72 The NAACP opposes testing, mainly on the grounds that tests are c u ltu ra lly biased. The NEA task force studied various aspects of tests and testing over a period of three years and con­ cluded that tests were not only biased, but were being used too heavily to determine who was excluded or who was included in programs or in the allocation of federal funds. Both organizations observed that the p r i ­ mary purposes of testing were tracking and categorizing. In summary, concern about accountability has given rise to a greater use of test instruments than ever before for auditing pur­ poses. The most widely used kind of test is the norm-referenced survey of achievement. Criterion-referenced testing is becoming more popular, however, because the public is demanding precise, rather than general, knowledge concerning student achievement of predetermined objectives, and the results are interpreted in terms of specific standards of performance. 72 Frances Quinto and B, McKenna, Alternatives to Standardized Testing (Washington, D.C.: National Education Association, 1977), pp. 1-17. 44 Whether one should use norm-referenced or criterion-referenced instruments depends on the information required. Both are useful and serve to substantiate teacher judgment. Many of the questions regarding criterion-referenced testing are related to interpretation and use of the resu lts. Research showed that teachers generally find these tests worthwhile because they provide information that can be used in the decision-making process. They are helpful in determining students' instructional needs and in diagnosing specific strengths and/or weaknesses. Many groups and individuals, how­ ever, are speaking out strongly against reliance on te s t data fo r meas­ uring educational growth and categorizing students fo r instructional purposes. Michigan's Accountability Model Since 1969, Michigan has responded to the public's concern about accountability by developing a six-step accountability model with which to establish and implement an assessment program. 73 The six steps are as follows: 1. Involve persons throughout the state in establishing common goals of educational accountability. 2. Translate these goals into objectives that are a broad base o f important variables fo r assessing needs in Michigan schools. 3. Assess needs, as related to the objectives derived from the goals, to provide information fo r state and local decision makers to help in determining p r io r itie s for a variety of needed change e ffo r ts . ^Ernest R. House, Wendell Rivers, and Daniel L. Stufflebeam, "An Assessment of the Michigan Accountability System," Phi Delta Kappan 55 (June 1974): 663-69. 45 4. Test alternative delivery systems to help d is tric ts develop a research basis for th e ir schools. 5. Foster the development of local evaluation capacity to help schools assess local needs; to help them design, implement, and assess th e ir own innovative e ffo rts ; and to help them evaluate th e ir personnel f a i r l y . 6. Use feedback to guide state and local policy makers in f u l f i l l i n g th e ir leadership roles in education. The assessment program is now based on criterion-referenced testing and is designed to show whether or not individual students have mastered the basic reading and mathematical s k ills iden tified as important by Michigan's educators.^ During the f i r s t four years (1969-1973), the instrument was norm referenced and showed how stu­ dents performed re la tiv e to one another. In the 1973-74 school year, the testing instrument was changed to a criterion-referenced test that could more adequately provide information on basic-skills mastery. The State Board of Education, the Executive O ffice, the legislature, local educators, teachers, students, and parents wanted information on the status and progress of Michigan's basic-skills education. 75 I t was hoped that data produced by the new c rite rio n referenced instrument would be more s p ecifically related to the instructional content and purpose of Michigan's schools. I t was evident that there was a need fo r a measuring device that would ^ Michigan Educational Assessment Program, Technical Report, Vol. 1 (Lansing: Michigan State Board of Education, 1980), p. 2. 751bid. , p. 2 1 . 46 re fle c t specific learning and teaching. This new focus was uniquely adapted to the concept of minimality because the Michigan Educational Assessment Program (MEAP) o f f ic ia ls wanted to focus on those objec­ tives on which a ll students in a given grade should have had instruc­ tion; i f these objectives have not been attained, they should be acquired as soon as possible.7® The Michigan State Department of Education holds the position that a common core of educational objectives does exist; these transcend local d is t r ic t boundaries, and schools are responsible for helping students attain them. In f a c t , these objectives can be iden­ t if i e d through a rational process, and the e ffo rt to do so is con­ sidered worthwhile.77 After these common core objectives were id e n ti­ fied by o f f ic ia ls from the State Department of Education, they were reviewed and approved by a panel of educators, c itize n s , and students and also by the Council on Elementary and Secondary Education. Both the Michigan Council of Teachers of Mathematics and the Michigan Reading Association were issued contracts to review both the objec­ tives and the results of the respective tests and to make recommendations on how to improve the quality of the objectives. 78 In his address to the National Conference on Achievement Tests and Basic S kills in 1978, John W. Porter, the Superintendent of 76Ib id ., pp. 21-22. 77 C. Philip Kearney, David L. Donovan, and Thomas H. Fisher, "In Defense of Michigan's Accountability Program," Phi Delta Kappan 56 (September 1974): 14-19. 78 An Assessment of Michigan Accountability System (Lansing: Michigan Department of Education, May 1974), pp. 6-8. 47 Public Instruction for the state of Michigan at that time, discussed the purposes for which the minimal-competency testing program had been established in Michigan. He said i t was a matter of public policy to identify those youngsters who had special needs, so that those needs could be met. He said, "Proposals for experimental and demonstration programs are required for systems where the test data demonstrates special needs; compensatory funds are made available." 79 Porter stated that norm-referenced standardized tests are no longer a viable alternative. They were adequate as long as (1) the operation of public schools was re la tiv e ly inexpensive; (2) public schools could screen, sort out, and select students; (3) those who paid for the schools were the parents of children in attendance; and (4) the "have nots" did not actively participate. Since these were no longer true, Michigan had an obligation to provide amore spe­ c i f i c a l ly outlined program of instruction and to report the results of those programs to a ll concerned. 80 Some of the advantages of the Michigan program were cited by Porter: (1) the b u ilt - in necessity of studying expectations before beginning instruction; (2) a capacity to report to parents on an item-by-item basis concerning th e ir children's performance; (3) its encouragement of teachers at the classroom level; (4) its automatic 79 John W. Porter, "Achievement Testing: The Interests," in The National Conference on Achievement Tests and Basic S k i l l s , Conference Proceedings (Washington, D.C.: National In s titu te of Educa­ tio n, U.S. Department of Health, Education, and Welfare, March 1-3, 1978), p. 14. 48 check on the purchase and use of materials; and (5) its potential for focusing on needed in-service programs. 81 The MEAP tests are given at the beginning of each school year to fou rth-, seventh-, and tenth-grade pupils to gather specific information on student progress in the basic s k i l l s . These tests are given early in the school year and cover objectives that the students should have attained in the three years before taking the . , 8 2 test. The revised edition of the fourth-grade MEAP reading test comprises 25 objectives, each of which is measured by a set of three 83 items. Answering two out of three of the items correctly is the standard by which mastery is determined. There are four additional "positive response" test objectives that are used to ascertain the extent to which pupils make reading a part of th e ir personal liv e s . These objectives are affective in nature and are neither calculated nor included in the proportions data. One of the major controversial issues surrounding the use of the MEAP is whether the objectives th a t are assessed are rea lly minimal. 84 Lovell stated that minimal-competency testing is a system for measuring the acquisition of those s k ills assumed necessary for 81 Ibid. 82 Questions and Answers About the Michigan Educational Assessment Program (Lansinq: Michigan Department of Education, September 1980), p. 4. Michigan Educational Assessment Program Handbook (Lansing: Michigan Department of Education, 1980), pp. 1-2. 84 House, Rivers, and Stuff!ebeam, pp. 663-69. 49 minimal competency m some area of l i f e or of learning. 85 He did not elaborate further on ju s t how to determine for sure which compe­ tencies are necessary or who should decide this issue. Before the MEAP te st was revised, House, Rivers, and S tu ffle QC beam were c r itic a l of the te s t because i t was norm-referenced. They f e l t that norm-referenced tests had content that discriminated against minorities and that norm-referenced interp retatio n caused unfair com­ parisons between d is tric ts . Responsible state and d is tr ic t personnel contended that i t did not measure what teachers were teaching. These criticism s have been arrested by the change to a criterion-referenced instrument. A third major critic ism charged that the test was biased. The MEAP o ffic ia ls responded to this charge, stating that m inorities should not be given separate tests since a ll children can and should achieve the minimal s k ills that are essential for liv in g in this country. 87 Since the onset of the testing program, there has been increas­ ing concern by local and state o ffic ia ls about how to report the results f a ir ly and adequately to local communities. As soon as the results are in , in January or February each year, the newspapers carry a rtic le s about the d is tric ts ' scores. 85 Reporting of these Ned B. Lovell, Rodney P. Riegle, and Clinton R. Bunke, "Minimal Competency Testing: Hopes, Fears, and F alla cie s," The Educa­ tional Forum 45 (January 1981): 199-206. Rfi House, Rivers, and Stuff!ebeam, pp. 663-69. 87 Edward D. Roeber, David L. Donovan, and Richard F. Cole, "Telling the Statewide Testing Story...And Living to Tell I t Again!" Phi Delta Kappan 62 (December 1980): 273-74. 50 results was at f i r s t comparative in nature, and this was damaging when comparisons were made among races and/or between d is tric ts that d iffered economically. School o ffic ia ls f e l t that such comparisons were unfair since they did not consider individual progress or c h ild growth and development concepts. MEAP o ffic ia ls attempted to a lte r this situation with s tra te ­ gies based on the following assumptions: (1) the public has a rig h t to know, even i f school o ffic ia ls fear to a ir problems; (2) the public has a rig h t to know what action is being taken to remedy the situ a­ tion i f the scores indicate that students have not acquired minimal s k ills ; (3) i t is b etter to provide honest and complete information on the results than to have the negative aspects discovered; (4) to put these scores in the proper perspective, the public should be given information on performance measures related to other school learning, not ju s t to minimal s k ills . 88 School superintendents, other key school personnel, and the news media were briefed by state o ffic ia ls on how to present MEAP scores to the public so that the scores could be interpreted with respect to each school d is tr ic t's accomplishments and not in relation to other d is tric ts . Roeber, Donovan, and Cole maintained that this public-relations campaign helped the news media report scores, based on an understanding of the purpose o f the te s t. They said that media coverage has changed; newspaper a rtic le s now deal with fewer comparisons, and the scores are more favorably interpreted and 51 reported. 89 In view of many recent Michigan newspaper a rtic le s , this claim is questioned by many people who are concerned about educational practices in Michigan. Two other perplexing concerns about the MEAP test program, which have yet to be resolved, relate to testing conditions and cost effectiveness. Both of these issues must be r e a lis tic a lly appraised i f the test is to produce meaningful results and i f the program is to be economically sound. Testing conditions and adm inistrative procedures may not necessarily be uniform among d iffe re n t groups that are tested, even though they are reported as i f they are. Test results can only re fle c t the s k ill level of the student, which was demonstrated on the specified day and time that the te st was administered; they cannot account for adverse conditions or fo r adm inistrative procedures that deviate from the manual instructions. Most educators would agree that eith er one or both o f these factors may influence the test results. In response to issues concerning the cost of the MEAP testing program, MEAP o ffic ia ls stated emphatically that the cost is reasonable compared to the cost of alte rn ative commercial testing programs. 90 They reported th at the cost to Michigan in 1979-80 was $190,000 for the MEAP every-pupil mathematics and reading te sts . mately $1.05 per pupil. 90 This was approxi­ This amount included test development, Questions and Answers About the Michigan Educational Assess­ ment Program, p. 5. 52 p rin tin g , and d is trib u tio n costs but not administering or grading costs. These la s t two costs were borne by the school d is t r ic t . Other hidden costs, such as those fo r student and teacher prepara­ tion time, were not considered in the cost analysis. In summary, Michigan’ s six-step accountability program centers on a criterion-referenced testing instrument that v a lid ly assesses whether students are acquiring minimal competencies in those s k ills deemed essential for d a ily liv in g . This program is considered valid because i t has been endorsed by many educational leaders, and i t measures those objectives that have been defined as minimal. At f i r s t , the MEAP test was norm referenced. Later i t was changed to a criterion-referenced test so that i t could show whether students were mastering the basic reading and mathematical s k ills id e n tifie d as important by Michigan educators and endorsed by the Michigan Reading Association and the Michigan Council of Teachers of • Mathematics. Testing conditions and administrative procedures have not been kept constant from group to group, even though both of these factors may influence test results. No way has been devised to con­ tro l these variables. MEAP o ffic ia ls feel that the cost o f the MEAP te s t is reason­ able, even though hidden costs have not been analyzed. d is tric ts must bear a portion of this cost. Individual 53 Summary of the Literature Review Although there is a need to include minimal s k ills in the instructional programs, l i t t l e has been done to ascertain whether these s k ills are actually included in the instructional materials that are used. Inasmuch as they have been deemed worthy o f a tta in ­ ment, i t is important that students receive instruction in them. Reading is a basic s k ill on which school success depends because reading is essential fo r learning most other content. Teach­ ers use basal reading programs 85 to 90 percent of the time in the primary grades, and they follow the suggested a c tiv itie s in the teachers' manuals. The manuals also supply information on coordi­ nated workbook pages that provide s k ill practice for basic reading objectives. Norm-referenced survey tests o f achievement are the most widely used kinds o f tests. Criterion-referenced tests , however, are gaining in popularity because they provide specific information con­ cerning whether students have mastered predetermined objectives. Criterion-referenced tests are more closely correlated with instruc­ tional programs than are norm-referenced tests. Michigan developed and implemented a statewide c rite rio n referenced test designed to ascertain the state of b a s ic -s k ills educa­ tio n . The objectives were developed on the theory that there is a common core of essential basic s k ills that a l l , or nearly a l l , c h il­ dren can a tta in . These function as a guideline fo r Michigan's schools, to ensure that students have acquired the "basics." CHAPTER I I I PROCEDURE AND METHODOLOGY The procedure and methodology of this study are presented in this chapter. The topics included are research questions, materials and sample, research design, and the analysis treatments. Included in the research design are (1) the method used to select reading series, (2) the means of determining congruency between objec­ tives o f exercises in workbooks and teachers' manuals and the objec­ tives of MEAP, and (3) a description of the instruments. Research Questions The four research questions originated with the statement of the problem, which was based on the study of the relationship between reading series used in primary classrooms and objectives tested in the MEAP at the conclusion of the primary years of schooling. They were: 1. How many objectives of the reading-skill exercises found in workbooks and teachers' manuals match MEAP objectives? 2. What reading series has the greatest percentage of s k i l l - exercise objectives that are congruent with MEAP objectives, as deter­ mined by a panel of reading experts? 3. Will the degree of congruence between reading-exercise objectives and MEAP objectives a ffe c t MEAP test results? 54 55 4. Are methods and strategies used in teaching reading lik e ly to influence the MEAP test results? Materials and Sample The materials used in answering the f ir s t two research ques­ tions were (1) a ll student s k ill exercises in the workbooks and teachers' manuals fo r each of the three basal reading series and (2) the Michigan Educational Assessment Program's 25 objectives fo r reading. (The reading series included Ginn and Co.; H o lt, Rinehart, and Winston Co.; and Houghton-Mifflin Co.) To answer the th ird research question, co lle ctive reading test scores were taken from a ll the MEAP Grade Four Classroom Listing Reports in three Western Michigan school d is tric ts (X, Y, Z ). These d is tric ts had mandated the use of the three selected reading series for a minimum of three consecutive years before administering the 1980-81 MEAP te s t. The three treatment groups, or d is tr ic ts , that were compared included 11 fourth-grade classes in D is tric t X, which used the Houghton-Mifflin series; 12 fourth-grade classes in D is tric t Y, which used the H olt, Rinehart, and Winston series; and 30 fourth-grade classes in D is tric t Z, which used the Ginn series. The three d is tric ts were not comparable in size nor in the state equalized value of property per pupil. These d is tr ic ts , however, had comparable expenditures per pupil fo r instructional programs, and median incomes were approximately the same. According to 1981 data 56 from the Michigan Department of EducationJ D is tric t Z had a state equalized property value of $77,807 fo r each o f its 8,998 pupils, and the per pupil expenditure fo r instructional programs was $1,006. D is tric t Y had a state equalized property value of $48,307 fo r each of its 4,736 pupils, and the per pupil expenditure for instructional programs was $1,103. D is tric t X had a state equalized property value o f $43,509 fo r each of its 3,817 pupils, and the per pupil expenditure fo r instruc­ tional programs was $1,197. According to 1977 data from the D etroit 2 Census Bureau, the median income for D is tric t Z was $5,300. The median incomes fo r D is tric ts Y and X were $5,547 and $5,191, respectively. Test scores fo r classes in D is tric t X included those of 10 special-education and 24 T it le I students. D is tric t Y scores included those of one special-education and one T itle I student. No scores fo r special-education students were reported in the D is tric t Z summaries. No scores fo r bilingual students were included in any o f the three d is t r ic t s ’ summaries. In response to the fourth research question, a questionnaire was used to survey f i r s t - through third-grade teachers in each of the three d is tric ts concerning teaching methods and strategies. Ninety-one percent of the 34 teachers in D is tric t X rep lied. In D is tric t Y, 75 percent of the 32 teachers re p lie d , and in D is tric t Z, 93 percent of the 72 teachers rep lied. Most of the nonrespondents ^Shirley Waldon, Supervisor o f State Aid, Michigan Department of Education, telephone interview, duly 12, 1982, O Timothy dones, Information Services S p e c ia lis t, D etroit Census Bureau, telephone interview, duly 12, 1982. 57 were from D is tric t Y. This was the only group that chose to have questionnaires collected by school administrators rather than m ail­ ing responses d ire c tly to the researcher. Research Design Basic Design Elements and Procedure The basic design consisted of the following elements: (1) MEAP reading objectives, co lle ctive reading test scores from MEAP Grade Four Classroom Listing Reports, and a teacher questionnaire; (2) three treatments (sets of workbooks and teachers' manuals); (3) contingency tables; and (4) the s ta tis tic a l model, univariate analysis o f variance. In response to the research questions, observations were based on (1) a comparison of the objectives inherent in the s k ill exercises of grades one through three in the workbooks and teachers' manuals fo r the basal series, with the MEAP objectives; (2) comparisons among the three series, according to congruence with MEAP objectives; (3) comparisons o f the MEAP te s t results among the three d is tric ts that were using the specified reading series; and (4) an analysis, according to d is t r ic t , of f i r s t - through third-grade teachers' per­ ceptions of th e ir own methods and strategies. Contingency tables were used both fo r comparisons of readingseries objectives with MEAP objectives and fo r comparisons among the objectives of the three reading series. Univariate analysis of v a ri­ ance was used to compare the three treatments (d is tr ic ts ) according 58 to MEAP test results and according to categories o f teacher responses on questionnaires. The procedure used to analyze the reading series, the test res u lts , and the questionnaires was the same fo r a ll treatment groups. The following design sequence was used: 1. Three reading experts analyzed a ll of the s k ill exercises in the teachers' manuals and workbooks fo r the three reading series; they id e n tifie d the objective o f each exercise and determined whether these exercise objectives matched the MEAP objectives. 2. Comparisons were made o f the s k ill exercises in the three series, according to the percentage of matches and mismatches across the MEAP objectives. 3. Comparisons were made of the c o llec tive MEAP reading test scores fo r the fourth-grade classes across the three d is tric ts . 4. Responses to the questionnaire were analyzed by d is tr ic t and were compared to the MEAP test results. Description of Materials and Instruments Two types of materials and two instruments were used to obtain data. The materials used were (1) the s k ill exercises in the sets o f workbooks and teachers' manuals fo r each o f the three reading series and (2) a l i s t of the 25 MEAP fourth-grade reading objectives from the MEAP Handbook.3 The instruments used were ^Michigan Educational Assessment Program Handbook (Lansing: Michigan Department of Education, 1980), p. 48, 59 (1) the c o llective fourth-grade reading test scores from the Classroom Listing Reports fo r 1980-81 and (2) a f i r s t - through third-grade teacher questionnaire. The c r ite r ia for selecting treatment groups were based on whether the students in the d is t r ic t had used one o f the three read­ ing series that were most widely adopted in Michigan and on whether they had used i t for three consecutive years before taking the fou rthgrade MEAP te s t. The most widely adopted and used basal series in 4 Michigan were determined from reports by Nicholas P. Criscuolo and 5 Market Data R e trie v a l, Inc. In 1973, Criscuolo reported that Ginn and Co. was the most widely used basal reading series in Michigan and that the Houghton-Mifflin Co. series and the H o lt, Rinehart, and Winston series were second and th ir d , respectively. In 1977, Market Data R etrieval, In c ., reported th a t Ginn and Co. was s t i l l the most widely used series but that the H o lt, Rinehart, and Winston series had become second and the Houghton-Mifflin series was in th ird place. The unit of analysis in each of the three reading series was the s k ill exercise. The purposes o f the exercises in the workbooks and teachers' manuals were determined independently by a panel of c e rtifie d reading experts. in a ll three series. The experts examined the s k ill exercises There were seven sets of workbooks and teachers' ^Nicholas P. Criscuolo, Improving Classroom Reading Instruc­ tion (Worthington, Ohio: Charles A. Jones Publishing Co., 1973), p. 23. 5 Market Data R e trie va l, In c ., Reading K-8 Survey, HM Co. Market Research Report No. 17 (New YorFl Market Data R e trie va l, In c ., 1977), p. 97. 60 manuals that were graduated in d iffic u lty from Levels 2 through 9 in the Ginn series. Levels 3 through 4 were grouped together in a single manual and workbook; a ll other levels had separate manuals and workbooks. The H olt, Rinehart, and Winston series had seven sets of workbooks and teachers' manuals; Levels 3 through 6 were grouped together in one s e t, whereas Levels 7 through 12 had one workbook and one teachers' manual for each level in the set. The Houghton-Mifflin series had nine sets o f workbooks and manuals that were graduated in d iffic u lty from Levels 3A, 3B, 3C, and 4 through 9. (See Table 3 .1 .) A ll of these sets of early 1970s workbooks and manuals were published for use in primary grades one through three and were not designed fo r one p artic u la r grade. The 25 objectives, which a ll students in a given grade should have attained, were id e n tifie d by the State Department of Education in Michigan. These objectives were categorized in the MEAP Handbook as follows:^ 1. 2. 3. 4. 5. Vocabulary Meaning L ite ra l Comprehension In fe re n tia l Comprehension C ritic a l Reading S k ills Related Study S k ills (6 (5 (9 (1 (4 objectives! objectives) objectives) objective) objectives) A l i s t o f the specific objectives in each category can be found in Table 3.1. The 25 MEAP minimal-performance objectives provide the basis fo r the MEAP testing program. The te s t was designed to assess Michigan Educational Assessment Program Handbook, p. 3. 61 whether individual students have attained the MEAP objectives. The co llective te s t scores fo r grade four from the MEAP Classroom Listing Reports fo r 1980-81 within each d is tr ic t were used to ascertain whether the d is tr ic t that was id e n tifie d as using the materials that were most congruent with the MEAP objectives had the highest MEAP te st scores. The MEAP test is administered during the in it ia l weeks of grade four because the te s t is a measure of minimal s k ills that should have been acquired during grades one through th re e .7 The test items were based on those objectives that selected Michigan teachers, curQ riculum sp e c ia lis ts , and other educators id e n tifie d as important. Each reading objective is assessed by a set of three te s t items. A stu­ dent must answer two out o f three of the items correctly to demon­ strate attainment of the objective, according to the standards set g by MEAP. The Classroom Listing Reports that were used in this study are summaries o f class averages from the MEAP reading te s t. (An example of a Classroom Listing Report for fourth-grade reading is shown in Appendix C.) The second instrument that was used was a questionnaire. To consider a variety of variables that might have had a bearing on student learning, a questionnaire was sent to a ll f i r s t - through ^Michigan Educational Assessment Program, Technical Report, Vol. 1 (Lansing: Michigan State BoardVf Education, 1980), p. 1. O Michigan Educational Assessment Program Handbook, p. 2. g I b i d . , p. 1. 62 third-grade teachers in the three d is tric ts . Part I o f the ques­ tionnaire was made up o f eight background-information questions; these were related to teacher tra in in g , basal series used, and years of experience. The second part of the questionnaire was made up of 55 questions concerning teaching methods and strategies. The categories of these questions were (1) classroom enrichment, (2) homework, (3) teacher-made and other materials th at were not part of the basal series, (4) parental involvement, (5) time, (6) organizational pattern, (7) independent learning, (8) student in terac tio n , (9) outside enrich­ ment, (10) use of basal m aterials, (11) s ta ff support, (12) languagedevelopment a c tiv itie s , (13) w ritten work, (14) test uses, and (15) rewards. (See Appendix A fo r specific questions.) P ilo t Studies Three p ilo t studies were conducted. for the panel of reading experts was held. F ir s t, a trainin g session Second, the teacher ques­ tionnaire was p ilo t-te s te d ; th ird , the questionnaire was revised and pi lo t-te s te d again. The reading experts were recommended by a c e rtifie d reading specialist who was on the faculty o f Central Michigan University and was not a f filia te d with the three treatment groups. Each of the three reading experts held an M.A. degree in reading and was an elementaryschool reading consultant in his/her local school d is tr ic t but not in schools involved in this study. The researcher conducted a training session for the panel of reading experts so that they might gain consistency in the application of rules and procedures fo r matching s k ill-e x e rc is e objectives with 63 MEAP objectives. During this trainin g session, three identical sets of workbooks and teachers* manuals, which were not a part o f the study, were used. The experts were instructed to examine each s k ill exercise independently and then to match i t with the appropriate objec­ tiv e . In a situation in which an exercise contained more than one objective, the experts were instructed to classify the exercise accord­ ing to which objective made up the m ajority of the exercise. (A m ajority of the exercises were defined as simply "more than h a lf." ) Based on this determination, the experts were instructed to l i s t the number of each exercise, according to its t i t l e and the objective i t represented, in the appropriate cell o f a prepared tab le. This process was repeated u n til a ll exercises had been examined. A fter the s k ill exercises were matched with the appropriate objectives, the group reconvened to discuss questions, concerns, and d iffe rin g opinions so that the thinking of the group would be more closely aligned. A rate r r e lia b ilit y o f .84 was established among the three c e rtifie d reading experts. A fter a questionnaire was developed, two p ilo t studies were conducted to examine other variables that might have had a bearing on the reading scores of the classes within each of the three d is tric ts . Input was sought from a c e rtifie d reading s p e c ia lis t to ascertain whether the questionnaire included the variables that were important in beginning reading classes. This sp ec ia list recommended th at four additional questions be added and th at one other question be changed to a continuum form. (See Appendix A fo r recommended changes.) 64 The questionnaire was p ilo t-te s te d on 21 elementary-school teachers who were not included in the study. the group responded, and changes were made. changes that were made.) Eighty-five percent of (See Appendix A for Eight volunteers from the same group were pi lo t-te s te d a second time, and verbal input was sought from each. The fin a l d ra ft of the questionnaire was administered to a ll f i r s t through third-grade teachers in D is tric ts Z, Y, and X. A majority o f the teachers responded. Analysis Treatments Descriptive and s ta tis tic a l treatments were used. Contin­ gency tables were prepared (1) to ascertain how frequently the ski 11exercise objectives matched the MEAP objectives and (2) to calculate the percentage of reading-exercise objectives that were congruent with MEAP objectives. Univariate analysis of variance was used to compare the reading-test scores on the fiv e MEAP categories and also to com­ pare the scores on questionnaire categories in each of the treatment groups. Contingency Tables A contingency table is a joint-frequency d is trib u tio n of cases as defined by the categories o f two or more variables. The chief component of the contingency-table analysis is the display of the distrib u tio n o f cases in the interacting c e lls . Contingency Table 3.1 was prepared to determine how frequently ski 11-exercise objectives matched MEAP objectives. The frequency dis­ tributions provided the raw data fo r calculating what percentages of 65 Table 3.1. Matrix for determining correlation of MEAP objectives with the sklll-exerclse objectives found in workbooks and teachers' manuals of three reading series. CD a\ SO n ■a X > 43 MEAP OBJECTIVES 61 VOCABULARY MEANING Word meanings affected by prefix Word meanings affected by suffix 3. Meaningswhenwordhasmulti-meanings Identifying synonyms 5. Identifying antonyms Determining meaningbasedoncontext LITERAL COMPREHENSION 7, Identifying main Idea of a selection 8. Identifying supportive details Identifying sequence in a selection 10. Identifying cause/effect relationship 11. Identifying stated Hkeness/diff. INFERENTIAL COMPREHENSION 12. Inferring main Idea of selection 13. Inferring cause/effect 14. Predicting probable outcome IS. Inferring details that support main idea 16. Inferring sequence 17. Inferring likenesses/differences 18. Drawing conclusions from given Info. 19. Identifying relationships (analogies) 20. Making Inferences about characters CRITICAL READING 21, Determining author's purpose RELATED STUDY SKILLS 22. Identifying major uses of dictionary/ glossary/contents 23. Locating Information 1n dictionary/ glossary/contents 24, Sumnarlzlng a selection 25. Alphabetizing to second letter OTHER 26, Other 1. 2. 4. 6. 9. 3= a. o' 0 £ 3 s o1 o VI (0 VI c « C* CO 1 2 c O *0 a. (6) 54S3fj 01 ♦ L/> to u > — u 3 c u £ IS e ai X O s I* c nJ £ O mv J wl V If a s z 1 O M Xi V I 01 w c oa w a o C l C □Q zto l/l O J i *j >j .c a3 s a 1 £ •t 31 trt X x cc - 7 O o 00 cn f-v *0 0 ■2 4J o 4 1 4 1 4 1 X L CC .0 a. Oi u i- •o4J C. 61 4 a 0> o > x0 £ O J *(43 5 rJ L> 61 X re u M 4 £ s _6c) 6 >1 u O JQ n. ► - Z1 tO < Signposts (5 Secrets (6) Rewards (71 n Houghton-MiffUn Co. Tigers (3A) Lions (3B) Holt, Rinehart, & Winston &inn & Co. 66 MEAP and "other" objectives were covered in the basal workbooks and teachers' manuals. {"Other" was a convenient term that would permit the inclusion of a ll exercises in the c la ss ifica tio n process.) There were 598 cells in a 23 x 26 design fo r the ski 11-exercise data, as i t interacted with 26 objectives. The 25 MEAP objectives and the "other" objectives were grouped categorically and placed down the l e f t side of the ta b le . The 23 t it le s of basal-series levels were alphabetized by publisher across the top of the table. Reading experts indepen­ dently determined whether the ski 11-exercise objectives in the teachers' manuals and workbooks of each reading series matched the MEAP objec­ tiv e s . Their determinations were classified by number in the cells formed by the intersection between the objectives and the t it le s of m aterials. I f an exercise did not match any of the MEAP objectives, i t was placed in the category labeled "other." Contingency Table 3.2 was prepared to summarize the percent­ ages and proportions of ski 11-exercise objectives th at intersected with three groups o f objectives. These included MEAP objectives, "other" objectives, or objectives not rated in each o f the three read­ ing series. "Other" was the term used fo r any objective o f the reading- s k ill exercise th at was not congruent with a MEAP objective. Objectives "not rated" were ski 11-exercise objectives on which the experts did not reach a consensus. A 3 x 3 design was used. Analysis o f Variance One-way univariate analysis of variance (ANOVA) was used to analyze the MEAP reading test scores by categories o f objectives. 67 Scores on categories o f responses to the questionnaire administered in each d is tr ic t were also compared through univariate analysis of variance. A ll analyses o f variance were tested fo r significance a t an alpha level o f .10. This alpha level was chosen because the three treatments were sim ilar in format and structure; scores were expected to be very close. An alpha level greater than .05 would provide a wider scope from which to discuss differences in the groups. Table 3 .2 .—D istribution of objectives fo r s k ill exercises across reading series. Ginn and Co. Ski 11-Exercise Objectives H o lt, Rinehart, and Winston Ski 11-Exercise Objectives HoughtonM ifflin Ski 11-Exercise Objectives (N = 1 ,094) S k ill Exercises (N = 1 ,399) S k ill Exercises MEAP Objectives Other Objectives Exercises Not Rated (N = 1 ,327) S k ill Exercises One-way analysis o f variance is appropriately used when only one treatment is administered to each group of subjects. Univariate ANOVA is used when there is one dependent variable in each analysis. This la tt e r treatment was used to determine (1) whether the mean scores o f the three d is tric ts d iffered s ig n ific a n tly from one another on the fiv e MEAP objectives (variables) and (2) whether the scores 68 on the categories of the questionnaire responses d iffered s ig n if i­ cantly from one another according to d is t r ic t . Table 3.3 illu s tra te s the layout of data fo r the 3 x 5 design, which was used to compare the mean scores o f the classes in the three d is tr ic ts , or treatment groups, with the fiv e categories of MEAP objectives. Table 3 .3 .—Matrix for 3 x 5 design fo r comparing treatment groups' mean scores on categories of MEAP objectives. Factor A {Main Effect) Independent Variable (Categories of Objectives) Factor B (Main Effect Independent Variable (Treatments) (Z) Ginn (Y) Holt (N=30 classes) (N=12 classes) (X) HoughtonM ifflin ( N=11 classes) 1. Vocabulary Meaning 2. L ite ra l Comprehension 3. In fe re n tia l Comprehension 4. C ritic a l Reading 5. Related Study S k ills Univariate analysis of variance was used to determine whether the mean scores on Factor A (categories of objectives) d iffered sig ­ n ific a n tly from one another in each of the treatment groups (Factor B). The independent variables (categories of objectives) and treatments 69 were regarded as fixed factors since they were drawn from the en tire target population. There was not an equal number of observations in each c e ll because of the gross reporting o f scores. The Ginn variable had 30 observations per c e ll, whereas the Holt variable had 12 obser­ vations per c e ll and the Houghton-Mifflin variable had 11 observations per c e l l . Table 3.4 illu s tra te s the layout of data fo r the 3 x 15 design, which was used to compare the scores on categories of responses to the questionnaire administered in each d is t r ic t . assigned to the following values: Scores were 1 = always, 2 = most o f the time, 3 = sometimes, 4 = seldom, and 5 = never. Univariate analysis of variance was used to determine whether the teachers in the three d is tric ts (treatment groups) d iffered sig ­ n ific a n tly on the strategies and methods they used in teaching read­ ing and to determine whether these differences affected MEAP te st results. Since a ll f i r s t - through third-grade teachers influence how and what th e ir students learn and since the MEAP objectives tested in grade four were supposed to have been attained in grades one through three, the questionnaire was administered to a ll f i r s t - through third-grade teachers in the three d is tr ic ts , even though not a ll teachers had taught these p articu la r primary students. There was not an equal number of observations in each c ell o f the 3 x 1 5 design. The treatment groups varied as to th e ir number of f i r s t - through third-grade classroom teachers. D is tric t Z had 66 responses to the questionnaire, whereas D is tric t Y had 24 and D is tric t X had 31. 70 Table 3 .4 .—Matrix for 3 x 1 5 design fo r comparing treatment groups' mean scores on categories of questionnaire responses. Factor A (Main E ffect) Factor B (Main E ffect) Independent Variable (Treatments) Independent Variables (Questionnaire Categories) 1. Classroom enrichment 2. Homework 3. Teacher-made and o th e r... 4. Parental involvement 5. Time 6. Organizational pattern 7. Independent learning 8. Student interaction 9. Outside enrichment 10. Use of basal materials 11. S ta ff support 12. Language dev. act. 13. Written work 14. Test uses 15. Rewards (Z) Ginn (Y) Holt (N=66 teachers) (N=24 teachers) (X) HoughtonM ifflin (N-31 teachers) 71 Summary The objectives inherent in the s k ill exercises found in work­ books and teachers' manuals o f three basal reading series were examined by reading experts to determine how they compared with MEAP objectives. Then percentages of these objectives were calculated according to the proportion of matches or mismatches across MEAP objectives. Frequency distributions on a contingency table provided raw data for c a lc u la t­ ing the percentages of MEAP objectives covered in each of the reading series workbooks and teachers' manuals. Contingency tables were prepared in a 23 x 26 design fo r the frequency count, and the percentage data were summarized in a 3 x 3 design. The 23 t it le s of levels within the three reading series were alphabetized by publisher across the top o f Table 3.1. The 25 MEAP objectives and the "other" objectives were placed down the l e f t side of the table. S k ill-e xe rc is e objectives were clas sified by number in the cells formed by the intersections between t it le s and objectives. Table 3.2 was designed in a 3 x 3 layout, which was used to summarize percentage data fo r s k ill-e x e rc is e objectives by publisher that in te r ­ sected with MEAP objectives, other objectives, or objectives not rated. Univariate analysis of variance was the method used to compare the mean scores of the fiv e categories of MEAP objectives in each of the three d is tric ts . A 3 x 5 design was used. Univariate analysis of variance was the method used to com­ pare the mean scores of the 15 categories o f questionnaire responses in each of the three d is tric ts . A 3 x 15 design was used. 72 The purpose of the questionnaire was to id e n tify other v a ri­ ables that might have influenced te st scores. The questionnaire was reviewed by a reading s p e c ia lis t, p ilo t-te s te d , revised, and p ilo ttested again. CHAPTER IV DATA ANALYSIS This chapter contains a review of the purpose of the study, descriptive data fo r Research Questions 1 and 2, hypotheses fo r Research Questions 3 and 4, and a report of the findings o f the study. The findings fo r the f i r s t two research questions are reported descrip­ tiv e ly according to frequency counts and percentage data. The findings fo r the la s t two research questions are reported according to rejection or acceptance of the null hypotheses. Purpose o f the Study F ir s t, th is study was conducted to determine the relationship between the s k ill-e x e rc is e objectives of the three most widely used reading series in Michigan and the MEAP reading objectives tested in grade four. Second, this study sought to compare the objectives of the s k ill exercises in the reading series to one another, in order to determine which one of the three had the highest percentage of objec­ tives th at were congruent with the MEAP objectives. Third, this study attempted to ascertain i f the pupils in the school d is tr ic t using the basal series that was most congruent with the MEAP reading objectives had the highest reading scores on the MEAP te s t. Fourth, the study attempted to determine whether the methods and strategies of teachers 73 74 in each of the three d is tric ts might have influenced MEAP te st results. Research Questions A contingency table was used in response to the following research question: How many objectives of the read ing-skil1 exercises found in workbooks and teachers' manuals match MEAP objectives? Frequency counts in contingency Table 3.1 were used when describing the relationship between the objectives inherent in the s k ill exercises found in the workbooks and teachers' manuals of each series and the 25 MEAP objectives and "other" objectives. ( I t was expected that the s k ill-e x e rc is e objectives of the three series would d iffe r from the MEAP objectives.) A contingency table was used in response to the following research question: What reading series has the greatest percentage of s k i l l exercise objectives that are congruent with MEAP objec­ tiv e s , as determined by a panel o f reading experts? Contingency Table 3.2 was used to describe the percentage of reading-series objectives that corresponded with the MEAP objec­ tiv e s . Frequency distributions across a ll 25 MEAP objectives and "other" objectives provided the raw data fo r calculating the percent­ ages of matches and mismatches with s k ill-e x e rc is e objectives from the three basal series. ( I t was anticipated that one of the basal series would be more closely aligned with the MEAP objectives than would the other two.) 75 Hypothesis I was derived from the th ird research question, which is as follows: W ill the degree of congruence between reading-exercise objectives and MEAP objectives a ffe c t MEAP te s t results? The f i r s t testable hypothesis was based on a comparison of the MEAP te s t scores in the three treatment groups (D is tric ts X, Y, and Z) to determine what e ffe c t the congruence of reading-series objectives with MEAP objectives had on the te st scores of the classes within each of the d is tric ts . (Congruency was determined according to the percentages of matches and mismatches o f reading-series objec­ tives across MEAP objectives in Table 3 .2 .) The independent variables were the treatments, or reading series, used by the three d is tric ts and the fiv e categories of objectives tested by the MEAP testing pro­ gram. The dependent variables were the mean scores obtained on the fiv e categories of fourth-grade MEAP reading-test objectives. Analysis of variance was used to compare the mean scores on the fiv e categories of MEAP objectives according to classes within the three d is tric ts using the three treatments. The mean scores on the MEAP reading tests of the classes within the three d is tric ts were expected to d iffe r . The mean scores of the classes within the dis­ t r i c t using the basal series that was most congruent with MEAP objec­ tives were expected to exceed the mean scores o f the other two treatment groups, or d is tric ts . Hypothesis I stated that there would be no difference on each of the dependent variables between the MEAP reading-test mean scores of the classes in the d is tr ic t using the most congruent reading 76 series and the MEAP reading-test mean scores o f the classes in the d is tric ts using the other two reading series. To enable the researcher to compare the mean scores o f the three treatment groups on each of the fiv e categories of MEAP objectives, Hypothesis I was restated as fiv e operational hypotheses: Operational HI a : There w ill be no difference between the MEAP reading-test mean scores o f the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on vocabulary meaning. Operational HIb: There w ill be no difference between the MEAP reading-test mean scores of the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two read­ ing series on lit e r a l comprehension. Operational H lc: There w ill be no difference between the MEAP reading-test mean scores of the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on in fe re n tia l comprehension. Operational Hid: There w ill be no difference between the MEAP reading-test mean scores of the classes in the d is t r ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on c r itic a l reading. Operational Hie: There w ill be no difference between the MEAP reading-test mean scores of the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on related study s k i l l s . For the analysis of variance test of significance, the alpha level was set a t .10. To reje c t a null hypothesis, the te s t required a s ig n ific a n t difference for the dependent variables (mean scores on the fiv e categories o f MEAP objectives). Hypothesis I I was derived from the fourth research question, which is as follows: 77 Are methods and strategies used in teaching reading lik e ly to influence the MEAP te s t results? The second testable hypothesis was based on the relationship between the teachers' methods and strategies and the MEAP test results. I t was anticipated that the teachers in the three d is tric ts would d if f e r according to strategies and methods they used to teach reading and that these differences would influence the te s t scores. The inde­ pendent variables were the 15 categories o f responses to the question­ naire that was administered to the teachers in the three d is tric ts and the fiv e types of possible responses. The dependent variables were the mean scores on the 15 categories o f responses to the ques­ tio nnaire. Analysis o f variance was used to compare the teachers' mean scores on the 15 categories o f responses to the questionnaire accord­ ing to d is tric ts using the three treatments, or reading series. scores were assigned the following values: The 1 = always, 2 = most of the time, 3 = sometimes, 4 = seldom, and 5 = never. Hypothesis I I stated that there would be no difference between the methods and strategies of teachers in the d is t r ic t using the most congruent reading series and the methods and strateg ies o f teachers in the d is tric ts using the other two reading series on each of the dependent variables, according to mean scores on teacher-questionnaire responses. To compare the scores o f the teachers in the three dis­ t r ic t s , or treatment groups, on each o f the 15 categories of question­ naire responses, this hypothesis was restated as 15 operational hypotheses: 78 Operational H2a: There w ill be no difference between the methods and strategies o f teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on classroom enrich­ ment., according to mean scores on teacher-questionnaire responses. Operational H2b: There w ill be no difference between the and s tra te g ie s 'o f teachers in the d is tr ic t using the most reading series and the methods and strategies of teachers d is tric ts using the other two reading series on homework, to mean scores on teacher-questionnaire responses. methods congruent in the according Operational H2c: There w ill be no difference between the methods and strategies o f teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on teacher-made and other m aterials, according to mean scores on teacher-questionnaire responses. 9 Operational H2d: There w ill be no difference between the methods and strategies- of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on parental involve­ ment, according to mean scores on teacher-questionnaire responses. Operational H2e: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on time, according to mean scores on teacher-questionnaire responses. Operational H2f: There w ill be no difference between the methods and strategies" of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on organizational patterns. according to mean scores on teacher-questionnaire responses. Operational H2g: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on independent learn­ ing, according to mean scores on teacher-questionnaire responses. Operational H2h: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the_ d is tric ts using the other two reading series on student in te ra c tio n , according to mean scores on teacher-questionnaire responses. 79 Operational H2i; There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on outside enrichment, according to mean scores on teacher-questionnaire responses. Operational H 2j: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on basal m aterials, according to mean scores on teacher-questionnaire responses. Operational H2k: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on s ta ff support, according to mean scores on teacher-questionnaire responses. Operational H21: There w ill be no difference between the methods ancT strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on language develop­ ment. according to mean scores on teacher-questionnaire responses. Operational H2m: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on w ritten work, according to mean scores on teacher-questionnaire responses. Operational H2n: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on te s t uses, accord­ ing to mean scores on teacher-questionnaire responses. Operational H2o: There w ill be no difference between the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on rewards, according to mean scores on teacher-questionnaire responses. For the analysis o f variance te s t of significance, the alpha level was set at .10. To re je c t a null hypothesis, the te s t required a s ig n ific a n t difference fo r the dependent variables (mean scores on the teachers1 responses to the categories o f questions on the ques­ tionnaire) . 80 Findings of the Study Introduction The findings fo r the f i r s t two research questions were based on (1) a comparison of s k ill-e x e rc is e objectives in the three reading series to the objectives tested by MEAP and (2) a comparison o f the percentages o f reading-series objectives that matched or mismatched the 25 MEAP objectives. The Ginn series, used in D is tric t Z, contained 1,327 s k ill exercises; the H olt, Rinehart, and Winston series, used in D is tric t Y, contained 1,094 s k ill exercises; and the HoughtonM ifflin series, used in D is tric t X, contained 1,399 s k ill exercises. The findings fo r Hypothesis I were based on the mean scores of the MEAP reading test for 53 fourth-grade classes w ithin the three d is tric ts , or treatment groups. There were 30 classes that used the Ginn treatment; 12 classes th at used the H o lt, Rinehart, and Winston treatment; and 11 classes that used the Houghton-Mifflin treatment. Classes were not randomly assigned to treatments but were selected according to the d is tr ic t that had mandated the use o f the specified series, or treatment, fo r a minimum of three consecutive years before administering the 1980-81 MEAP Grade Four Reading Test. The findings fo r Hypothesis I I were based on the mean scores, according to treatment group, of the teacher-questionnaire responses. There were 66 responses from teachers in the Ginn treatment group; 31 responses from teachers in the H o lt, Rinehart, and Winston tre a t­ ment group; and 34 responses from teachers in the Houghton-Mifflin treatment group. 81 Contingency Tables Contingency tables were used to answer the f i r s t two research questions. Shown in contingency Table 4.1 is the specific number of objectives in each level of the reading series that match HEAP or "other" objectives. (Each of these f i r s t - through third-grade levels was designated by t i t l e according to publishing company.) In this table we can see that (1) 430 of the Ginn series s k ill objectives matched HEAP objectives, whereas 830 matched "other" objectives; (2) 418 of the H o lt, Rinehart, and Winston series s k ill objectives matched HEAP objectives, whereas 623 matched "other" objectives; and (3) 828 o f the Houghton-Mifflin series s k ill objectives matched MEAP objectives, whereas 439 matched "other" objectives. Table 4.2 reveals th a t, fo r each reading series, less than 60 percent of a ll the s k ill-e x e rc is e objectives matched the MEAP objectives. Thirty-two percent of the s k ill-e x e rc is e objectives in the Ginn series matched the MEAP objectives; 38 percent of the s k i l l exercise objectives in the H o lt, Rinehart, and Winston series matched the MEAP objectives; and 59 percent of the s k ill-e x e rc is e objectives in the Houghton-Mifflin series matched the MEAP objectives. In the Ginn series, 63 percent of the s k ill-e x e rc is e objectives matched "other" objectives, whereas 57 percent of the s k ill-e x e rc is e objectives in the H olt, Rinehart, and Winston series and 31 percent of the s k i l l exercise objectives in the Houghton-Mifflin series matched "other" objectives. Comparing each of the s k ill-e x e rc is e objectives of the three reading series to the MEAP objectives, the researcher found that the 82 Table 4.1.—Correlation of HEAP and "other" objectives with objectives of skill exercises found In workbooks and teachers' manuals of three reading series. Holt, Rinehart, A Winston Ginn A Co• — . a Qt ID rn 0 0 m o o s— P i/IT • —- — VI C C - — in V V. n in U DDC VI — —— O M a . £ I— c ■Jf. 3 r-* u o 4 0 ■o — J X e r c u J O B J L 15 V 0 u p L. o > > O J % *J fe CL o a/ > 4/ a £ 1/1 H X 3 bl < < CL I—Z v> t£ Houghton-Mifflin Co. ■ MEAP OBJECTIVES VOCABULARY MEANING 1. Word meanings affected by prefix 5 2. Word meanings affected by suffix 1 4 12 4 12 1 3. Meanings whenwordhasmulti-meanings 2 2 3 2 6 1 2 2 4. Identifying synonyms 1 1 7 e 1 5. Identifying antonyms 4 i 2 1 1 B 10 14 14 19 14 14 >1 >6 6, Determining meaning basedoncontext LITERAL COMPREHENSION 7. Identifying main idea of a selection 1 4 2 3 t 1 3 B. Identifying supportive details 3 4 17 10 16 14 4 10 5 9. Identifying sequence in a selection 2 5 7 4 a 5 7 1 10. Identifying cause/effect relationship 1 1 i 2 11. Identifying stated 1ikeness/dlff. 1 1 2 INFERENTIAL COMPREHENSION 12. Inferring main Idea of selection 4 6 3 2 1 1 13. Inferring cause/effect 1 3 4 1 14. Predicting probable outcome 2 4 2 1 1 15. Inferring details that support 1 1 1 2 2 3 main idea 16. Inferring sequence 1 2 1 17. Inferring likenesses/differences 1 IB. Drawing conclusions from given 1nfo. 2 3 7 8 10 7 1 1 19. Identifying relationships (analogies) 1 2 20. Making Inferences about characters 2 5 3 3 CRITICAL READING 21. Determining author’s purpose 2 RELATED STUDY SKILLS 22. Identifying major uses of dictionary/ 1 1 2 glossary/contents 23. Locating information in dictionary/ 1 1 1 1 7 glossary/contents 24. Summarizing a selection 1 1 25. Alphabetizing to second letter 1 1 1 OTHER 26. Other 7 134121 34164111 16 113<>7 13 11 —, — n — ID 03 i/t VI lO r-» or m in L A bJ /c D o V) 1 V0I xs> in L. 4/ a. £ U fa O c D- u £ Cfa a/ _l o VI m ac CL u_ , 4 2 4 1 13 2 4 2 3 3 3 1 T 32 3 3 1 1 3 3 5 47 6 1 5 1 4 2 7 1 1 2 1 1 12 2 15 25 34 36 19 24 24 60 66 64 52 71 47 8 9 5 1 3 7 2 2 1 2 5 11 8 8 1 20 3 1 1 17 13 2 11 3 5 12 6 9 2 3 1 3 3 9 7 94 1 4 3 4 3 2 2 1 1 4 2 2 1 19 2 3 2 2 2 1 2 2 2 1 1 2 1 4 5 1 3 3 1 3 2 4 1 1 5 7 2 1 1 2 4 2 1 5 4 8 5 15 16 8 12 2 3 1 1 1 1T 1 1 1 1 4 1 2 3 3 1 3 2 3 1 3 19 4 4 3 9 04 95 72 11 13 13 15 59 79(?T 51 57 83 match between the Houghton-Mifflin series objectives and the MEAP objectives was 27 percent greater than the one between the Ginn series objectives and the MEAP objectives. The match between the Houghton- M ifflin series objectives and the MEAP objectives was 21 percent greater than the one between the H o lt, Rinehart, and Winston series objectives and the MEAP objectives. The Houghton-Mifflin series objec­ tiv e s , therefore, were most congruent with the MEAP objectives. Table 4 .2 .—D istribution of objectives fo r s k ill exercises across reading series. Objectives Ginn and Co. Skill-E xercise Objectives H o lt, Rinehart, and Winston S kill-E xercise Objectives HoughtonM ifflin Ski 11-Exercise Objectives MEAP objectives 430 (32%) 418 (38%) 828 (59%) "Other" objectives 834 (63%) 623 (57%) 439 (31%) Exercises not rated 63 53 139 (N=l ,327) (N =l,094) (N =l,399) One-Way Univariate Analyses of Variance One-way univariate analyses of variance were used to answer Research Questions 3 and 4. This s ta tis tic a l treatment was used (1) to compare mean reading-test scores on the various categories of MEAP objectives according to treatment groups and (2) to compare mean 84 scores on the various categories of questionnaire responses according to treatment group. As shown in Table 4 .3 , Operational Hypotheses la , lb , lc , and le were accepted. There was no difference between the HEAP reading- te s t mean scores of the classes in the d is tr ic t using the most congru­ ent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on vocabulary mean­ ing, lit e r a l comprehension, in fe re n tia l comprehension, and related study s k ills . Table 4 .3 .—Analysis of variance fo r HEAP test scores by category of MEAP objectives. Category MS Between MS Within F Sig. 1 Vocabulary meaning 17.9361 32.6575 .5492 .58 2 L ite ra l comprehension 12.8852 30.2280 .4263 .66 3 In fe re n tia l comprehension 27.1254 36.5209 .7427 .48 237.1645 86.2719 2.7490 26.7338 65.1603 .4103 4 C ritic a l reading 5 Related study s k ills .07* .67 Between-groups df = 2; within-groups df = 50. As indicated in Table 4 .3 , the only category o f MEAP objec­ tives that was s ta tis tic a lly s ig n ific an t was Category 4, c r itic a l reading. Operational Hypothesis Id was rejected; there was a difference 85 in the MEAP reading-test mean scores o f the classes in the d is tr ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two read­ ing series on c r itic a l reading. The significance level fo r Category 4, c r itic a l reading, was .07. As shown in Table 4 .4 , the Houghton-Mifflin treatment group (of the three, the one most congruent with MEAP) did not have the highest mean score on Category 4, c r itic a l reading. The H o lt, Rinehart, and Winston treatment group had the highest mean score on MEAP Cate­ gory 4; the Ginn treatment group and the Houghton-Mifflin treatment group had the second and th ird highest scores, respectively. Table 4 .4 .—MEAP te s t mean scores on objective Category 4 fo r the three treatment groups. (Z) H o lt, Rinehart, and Winston (Y) HoughtonM ifflin (X) X X X 74.8 77.36 68.75 Ginn and Co. MEAP Category 4 C ritic a l reading Population means: 73.96. To examine the number o f s k ill-e x e rc is e objectives that matched MEAP objectives, the researcher used Contingency Table 4 .5 . This table provided a summary of the number of s k ill-e x e rc is e objectives that matched MEAP Category 4. 86 Table 4 .5 .—Correlation of MEAP objective Category 4, c r itic a l reading, with number o f matching reading-series objectives. MEAP Category 4 C ritic a l reading (Z) H o lt, Rinehart, and Winston (Y) HoughtonM ifflin (X) 0 4 0 Ginn and Co. Correlating the mean scores in Table 4.4 with the number of s k ill-e x e rc is e objectives in the three reading series that matched MEAP Category 4 in Table 4 .5 , the researcher found that only the H olt, Rinehart, and Winston series had s k ill-e x e rc is e objectives that matched the MEAP objective on Category 4. The H o lt, Rinehart, and Winston treatment group was expected to score highest on this cate­ gory of MEAP objectives, but why the Ginn treatment group scored higher than the Houghton-Mifflin treatment group on Category 4 is unclear because neither used reading series with s k ill-e x e rc is e objec­ tives that matched the category o f MEAP objectives that was tested. To enable the researcher to examine variables other than the reading series that might have influenced the MEAP te s t scores, a questionnaire was administered to the f i r s t - through third-grade teachers in the three treatment groups, or d is tric ts . A.) (See Appendix Questionnaire responses ranged in value from 1 to 5, with a response o f 1 representing "always" and a response of 5 representing "never." Lower numbers indicated greater emphasis placed on the subject matter. 87 Table 4 .6 .—Analysis of variance fo r teachers' scores on questionnaire categories in D is tric ts Z, Y, and X. Category MS Between F MS Within Sig. 1 Classroom enrichment .32 .19 1.60 .20 2 Homework .58 .64 .91 .40 3 Teacher-made and other materials .63 .34 1.85 .16 4 Parental involvement .74 .33 2.24 .11 3.01 .37 8.18 .0005* .67 .17 4.02 .02* 7 Independent learning 4.25 .47 9.02 .0002* 8 Student interaction 1.71 .61 2.81 .06* .30 .26 1.16 .32 .1032 .1116 .72 .40 5 Time 6 Organizational pattern 9 Outside enrichment 10 Use of basal materials 49.03 1.27 38.62 .0000* 12 Language development 2.47 .27 9.20 .0002* 13 Written work 1.59 .42 3.78 .026* .78 .59 1.31 .27 4.48 1.33 3.37 .038* 11 S ta ff support 14 Test uses 15 Rewards Between-groups df = 2; w ithin groups df = 118. As shown in Table 4 .6 , 2 i, 2 j , and 2n Operational Hypotheses 2a, 2b, 2c, 2d, were accepted. There was no difference in the methods and strategies of teachers in the d is t r ic t using the most reading series congruent and the methodsand strategies o f teachers in the 88 d is tric ts using the other two reading series on the following: classroom enrichment, homework, teacher-made and other m aterials, parental involvement, outside enrichment, use of basals, and test uses. As shown in Table 4 .6 , eight categories o f questionnaire responses were s ta tis tic a lly s ig n ific a n t: 12, 13, and 15. Categories 5, 6, 7, 8 , 11, Operational Hypotheses 2e, 2 f, 2g, 2h, 2k, 21, 2m, and 2o were rejected. These categories of questionnaire responses were s ta tis tic a lly s ig n ific a n t according to the analysis of variance test of significance. The alpha level was set a t .10. The eight categories of questionnaire responses that were s ta tis tic a lly s ig n ific a n t are lis te d in Table 4 .7 . Discussion of the findings fo r each o f these categories follows. As shown in Table 4 .7 , D is tric t Y teachers devoted more time to reading than did D is tric t Xand Z teachers. This means they spent more time fo r preparation and in s tru ctio n , as well as fo r teaching and discussing basic s k ills with th e ir students. Teachers in D is tric t Zemphasized organizational patterns morethan did teachers in D is tric ts X and Y. Organizational patterns include structured programs, team teaching, and in d iv id u aliza tio n ; they also include grouping by a b ilit y , seating students in rows, and having students remain in the same reading group a ll year. (For teachers to have indicated that they individualized and inmobilized students of the same group was contradictory, so these two questions in the category termed organizational patterns were considered in valid as answered.) 89 Table 4 .7 .—Teachers' mean scores on the s ta tis tic a lly sig n ifican t questionnaire categories fo r D is tric ts Z, Y, and X. Questionnaire Category D is tric t Z D is tric t Y D is tric t X 5 Time 2.23 2.14 2.70 6 Organizational pattern 2.67 2.78 2.92 7 Independent learning 2.28 2.64 2.89 8 Student interaction 3.40 3.75 3.26 11 S ta ff support 2.21 1.58 4.03 12 Language development 2.33 2.35 2,79 13 Written work 2.30 1.94 2.15 15 Rewards 2.76 2.38 2.13 Independent learning, Category 7, was emphasized more by teachers in D is tric t Z than by those in D is tric ts X and Y. This cate­ gory includes learning that is s e lf- in it ia te d , reading in the content areas, using lib ra ry resources in a variety of ways, requesting e x p lic it directions, choosing reading matter, and choosing the assign­ ment to complete f i r s t . Student interaction received greater emphasis in D is tric t X than in the other two d is tric ts . There was more peer tutoring and older-student tutoring in this treatment group than in the others. D is tric t Y had more s ta ff support for both students and teachers. This d is t r ic t provided more aid from s p e c ia lis ts , more help fo r students by the classroom teacher, and more inservice tra in ­ ing programs for teachers when a new reading program was adopted. 90 Teachers in D is tric t Z placed more emphasis on language development than teachers in the other two d is tric ts . Language- development questions referred mainly to te llin g stories and to dis­ cussing information and ideas. (See Question 47-49, p. 126.) Teachers in D is tric t Y more often required th e ir students to examine and correct th e ir own w ritten assignments than did teachers in the other two d is tric ts . Teachers in D is tric t X more often provided rewards, such as treats or prizes, fo r doing good work than did teachers in D is tric ts Z and Y. In summary, the MEAP te st results revealed th at the H o lt, Rinehart, and Winston treatment group had the highest mean score on the only variable that was s ta tis tic a lly s ig n ific a n t: c r itic a l reading. Category 4, This might have been a ttrib u ta b le to two factors. F irs t, this group's treatment, or reading series, was the only treatment that had any s k ill-e x e rc is e objectives that matched the MEAP objective in the c ritic a l-re a d in g category. Second, the teach­ ers from this group perceived themselves as devoting more time to a ll aspects of reading than did the teachers from the other groups; they also perceived themselves as stressing w ritten work more and as having more s ta ff support fo r both teachers and students. The Ginn treatment group had the second highest mean score on Category 4, c r itic a l reading. Their treatment, or reading series, did not contain any ski 11-exercise objectives that correlated with the 91 MEAP objective in Category 4. Their teachers d id , however, stress independent learning, language development, and a more structured organizational pattern than did teachers in the other two groups. The Houghton-Mifflin treatment group had the lowest mean score on Category 4, c r itic a l reading. Their reading series did not contain any ski 11-exercise objectives that correlated with the MEAP objective in Category 4. The teachers in this group stressed only two of the s ta tis tic a lly s ig n ific a n t categories of questionnaire responses. These teachers perceived themselves as emphasizing student in te r ­ action and student rewards more than did the teachers in the other two treatment groups. Summary Contingency tables were used (1) to illu s tr a te the re la tio n ­ ship between the objectives o f the s k ill exercises found in the work­ books and teachers' manuals of the three reading series and the objec­ tives o f the MEAP test and (2) to illu s tr a te what percentage of the s k ill-e x e rc is e objectives of each reading series corresponded with MEAP objectives and to f a c ilit a t e the comparison of these percentages. The data showed that 430 o f the Ginn series s k ill objectives; 418 of the H olt, Rinehart, and Winston series s k ill objectives; and 828 of the Houghton-Mifflin series s k ill objectives matched the MEAP objectives. Thirty-two percent of the s k ill-e x e rc is e objectives in the Ginn series matched the MEAP objectives, whereas 38 percent of the s k ill-e x e rc is e objectives in the H olt, Rinehart, and Winston series and 59 percent of those in the Houghton-Mifflin series matched the 92 MEAP objectives. The Houghton-Mifflin series s k ill-e x e rc is e objectives most nearly matched the MEAP objectives; they were, therefore, most congruent with MEAP objectives. Analysis o f variance tests o f significance were used (1) to compare, by categories o f objectives, mean scores of the treatment groups on the MEAP Grade Four Reading Test and (2) to compare, by treatment group, mean scores on categories o f teacher-questionnaire responses. The alpha level fo r a ll ANOVA tests was set a t .10. Operational Hypotheses la , lb , lc , and le were accepted. There was no difference between the MEAP reading test mean scores of the classes in the d is tr ic t using the most congruent reading series and the MEAP reading te s t mean scores o f the classes in the d is tric ts using the other two reading series on vocabulary meaning, lit e r a l comprehension, in fe re n tia l comprehension, and related study s k ills . Operational Hypothesis Id was rejected. There was a s ig n ifi­ cant difference among the three groups on c r itic a l reading a t the .07 significance le v e l. Operational Hypotheses 2a, 2b, 2c, 2d, 2 i , 2 j, and 2n were accepted. There was no difference in the methods and strategies of teachers in the d is tr ic t using the most congruent reading series and the methods and strategies o f teachers in the d is tric ts using the other two reading series on classroom enrichment, homework, teachermade and other m aterials, parental involvement, outside enrichment, use of basals, and te s t uses. Operational Hypotheses 2e, 2 f, 2g, 2h, 2k, 21, 2m, and 2o were rejected. There was a difference in the methods and strategies 93 of teachers in the d is t r ic t using the most congruent reading series and the methods and strategies of teachers in the d is tric ts using the other two reading series on time, organizational pattern, independent learning, student in te ra c tio n , s ta ff support, language development, w ritten work, and rewards. The mean scores on these categories of questions were s ig n ific a n tly d iffe re n t, according to the analysis of variance te s t of s ta tis tic a l significance. The alpha level was set at .10. I t was anticipated that the Houghton-Mifflin treatment group would have the highest mean scores on the fiv e categories o f MEAP objectives because th e ir treatment, or reading series, was the most congruent with MEAP objectives. However, the H o lt, Rinehart, and Winston treatment group had the highest mean score on the only s ta tis ­ t ic a lly s ig n ifican t category of MEAP objectives; and the H o lt, Rinehart, and Winston series was the only one that had s k ill-e x e rc is e objectives that matched MEAP objectives in Category 4, c r itic a l reading. The teachers from the H o lt, Rinehart, and Winston group perceived them­ selves as devoting more time to a ll aspects of reading than did teach­ ers from the other groups. These teachers also perceived themselves as stressing w ritten work more and as having more s ta ff support than did the teachers from the other two groups. CHAPTER V CONCLUSIONS AND NEED FOR FURTHER RESEARCH This chapter begins with a summary of this study. following topics are presented: Then the Conclusions of the Study, Discussion o f the Findings, and Need fo r Further Research. Summary Statement of the Problem This study compared the 25 MEAP minimal reading objectives with the objectives represented in grades one through three basal reading exercises of the three most widely used series in Michigan, as determined by Market Data R etrieval, In c .J and Nicholas P. ? Criscuolo. The researcher attempted to ascertain the following: 1, How many objectives of the read ing -skill exercises found in workbooks and teachers' manuals match MEAP objectives? 2. What reading series has the greatest percentage o f s k i l l - exercise objectives that are congruent with MEAP objectives, as deter mined by a panel o f reading experts? 1 Market Data R e trie v a l, In c ., Reading K-8 Survey, HM Co. Market Research Report No. 17 (New York: Market Data R e trie v a l, Inc. 1977), p. 97. 2 Nicholas P. Criscuolo, Improving Classroom Reading Instruc­ tion (Worthington, Ohio: Charles A, Jones Publishing Co., 1973), p. 23. 95 3. Will the degree of congruence between reading-exercise objectives and MEAP objectives a ffe c t MEAP te s t results? 4. Are methods and strategies used in teaching reading lik e ly to influence the MEAP te st results? Need fo r the Study The need fo r this study was based on the premise that the curriculum, or course o f study, affects what students learn. Edu­ cators need to address the increasing concern about the qu ality of education by demonstrating that certain s k ills —those that meet the objectives o f the school system as well as o f the community--are being developed through the use o f selected cu rricu lar m aterials. Of course, achievement cannot be expected on topics that are tested but not taught; a high correlation between what is taught and what is tested is essential i f tests are to measure adequately what is taught. I f the correlation between what is taught and what is tested is low, the te s t is insensitive to achievement gains resulting from the cur­ riculum. There was a need, then, to establish the relationship between the MEAP objectives and the objectives o f read ing -skill exercises found in workbooks and teachers' manuals o f the reading series used in teaching minimal concepts and s k ills . Review of the Literature A review o f the lite r a tu re revealed th at the back-to-basics movement and the c all fo r accountability have given rise to concerns about the capacity of testing instruments to assess accurately what is taught in schools. The Michigan State Department of Education 96 established an accountability model that was designed to insure that students receive instruction in basic s k ills . Reading is a basic s k ill on which the success of schools depends because reading is essential fo r learning most other content. Teachers of the primary grades in schools throughout the United States use basal-reading-series textbooks, teachers' manuals, and workbooks 85 to 90 percent of the time. Criterion-referenced tests are gaining in popularity as a tool fo r assessing the achievement of specified objectives of instruc­ tional programs. The criterion-referenced te s t developed by MEAP o ffic ia ls was intended to be consistent with the instructional con­ tent and purposes o f basic education in Michigan schools. This te s t, however, focuses on specific objectives that are independent of any p a rtic u la r reading series that is used. Research has shown that testing and test results are dependent upon the type of measurement instrument used. Some researchers have concluded that the only valid way to assess the effects o f a treatment is to use tests that parallel what is taught. Such tests are c rite rio n referenced and focus on specific objectives that permit educators to ascertain whether a student possesses a p artic u la r competency. For the criterion-referenced te s t to be well constructed, the te s t items must represent the same domain as the instructional exercises. Criterion-referenced tests should measure an ind ividu al's level of proficiency, providing information on mastery or nonmastery that can be used to diagnose d e fic its or ascertain strengths. 97 Method and Design The three reading series used in the three d is tric ts were the Ginn and Company reading series; the H o lt, Rinehart, and Winston reading series; and the Houghton-Mifflin reading series. The objec­ tives of the s k ill exercises found in the workbooks and teachers' manuals o f the reading series were determined by a panel of three reading experts. Percentages of s k ill-e x e rc is e objectives that matched MEAP objectives were examined to determine the congruence o f the reading-series objectives with the MEAP objectives. The sample for the study consisted of 53 fourth-grade classes in three d is tric ts that used three d iffe re n t reading series for primary-level reading. D is tric t Z had 30 fourth-grade classes that used the Ginn reading series; D is tric t Y had 12 fourth-grade classes that used the H olt, Rinehart, and Winston reading series; and Dis­ t r i c t X had 11 fourth-grade classes that used the Houghton-Mifflin reading series. The three instruments used in this study were (1) a l i s t of MEAP objectives, (2) Classroom Listing Reports, and (3) a primaryschool-teachers questionnaire. A 23 x 26 contingency table was designed to determine how frequently s k ill-e x e rc is e objectives from workbooks and teachers' manuals of the three reading series intersected with MEAP and "other" objectives. A second contingency table in a 3 x 3 design was prepared to illu s tr a te the percentage of s k ill-e x e rc is e objectives that in te r ­ sected with MEAP objectives, "other" objectives, or exercises "not rated" for lack of expert consensus. 98 A 3 x 5 design was used fo r comparing the fiv e categories of the MEAP test scores fo r the three treatment groups, or d is tric ts . One-way univariate analysis of variance was used for comparing the three treatment groups' mean scores on the fiv e categories of the MEAP te s t. In each of the univariate analysis of variance te s ts , the alpha level of significance was set a t .10. To determine whether teachers' methods and strategies might have influenced test results in each of the three d is tr ic ts , a ques­ tionnaire was administered. A 3 x 15 design was used fo r comparing teachers' responses in the three d is tric ts according to the mean scores on 15 categories of responses to the teachers' questionnaire. The primary s ta tis tic for comparing the categories of responses was one-way univariate analysis of variance. In each of the univariate analysis of variance te s ts , the alpha level of significance was set at .10. The f i r s t hypothesis was that there would be no difference between the MEAP reading-test mean scores of classes in the d is tr ic t using the most congruent reading series and the mean scores o f the classes in the d is tric ts using the other two reading series on each of the dependent variables (mean scores on categories of objectives). This hypothesis was divided into fiv e operational hypotheses to make comparisons among the three d is tric ts on the fiv e categories of the MEAP te s t. The second hypothesis was that there would be no difference between the methods and strategies o f teachers in the d is tr ic t using the most congruent reading series and the methods and strategies of 99 teachers in the d is tric ts using the other two reading series on each of the dependent variables (mean scores on categories o f teacherquestionnaire responses). This hypothesis was divided into 15 opera­ tional hypotheses to make comparisons among the three d is tric ts on the 15 categories of the questionnaire. Analysis of the Data The contingency tables showed (1) the frequency with which reading-series objectives matched MEAP objectives and (2) the per­ centage of objectives in each reading series that matched MEAP objec­ tiv e s . Out of 1,327 exercises, the Ginn series had 430 s k ill-e x e rc is e objectives that matched MEAP objectives, whereas the H o lt, Rinehart, and Winston series had 418 out of 1,094 exercises and the HoughtonM ifflin series had 828 out of 1,399 exercises. Thirty-two percent of the Ginn series s k ill-e x e rc is e objectives matched MEAP objectives; 38 percent of the H olt, Rinehart, and Winston series s k ill-e x e rc is e objectives matched MEAP objectives; and 59 percent o f the HoughtonM iffl in series s k ill-e x e rc is e objectives matched MEAP objectives. The Houghton-Mifflin series objectives were found to be the most con­ gruent with MEAP objectives. Operational Hypotheses la , lb , 1c, and le were accepted. There was no difference between the MEAP reading-test mean scores of the classes in the d is t r ic t using the most congruent reading series and the MEAP reading-test mean scores of the classes in the d is tric ts using the other two reading series on vocabulary meaning, lit e r a l comprehension, in fe re n tia l comprehension, and related study s k ills . 1 DO Operational Hypothesis Id was rejected. There was a d if f e r ­ ence between the HEAP reading-test mean scores of the classes in the d is tric t using the most congruent reading series and the HEAP readingtest mean scores of the classes in the d is tric ts using the other two reading series on c ritic a l reading. (The significance level for this hypothesis was .07.) Operational Hypotheses 2a, 2b, 2c, 2d, 2i» 2 j, and 2n were accepted. There was no s ig n ific a n t difference between the methods and strategies of teachers in the d is tric t using the most congruent reading series and the methods and strategies of teachers in the dis­ tric ts using the other two reading series on classroom enrichment, homework, teacher-made and other materials, parental involvement, outside enrichment, use of basals, and test uses, according to mean scores on teacher-questionnaire responses. Operational Hypotheses 2e, 2 f, 2g, 2h, 2k, 21, 2m, and 2o were rejected. There was a difference in time a t the .0005 significance le v e l, a difference in organizational pattern at the .02 significance le v e l, a difference in independent learning a t the .002 significance le v e l, a difference in student interaction at the .06 significance le v e l, a difference in s ta ff support at the .0000 significance le v e l, a difference in language development at the .0002 significance le v e l, a difference in w ritten work at the .026 significance le v e l, and a difference in rewards at the .038 significance le v e l. Teachers from the Ginn treatment group in D is tric t Z emphasized organizational pat­ terns, independent learning, and language development more than did teachers in the other d is tric ts . Teachers from the Holt, Rinehart, 101 and Winston treatment group in D is tric t Y were strongest in time, w ritten work, and s ta ff support. Teachers from the Houghton-Mifflin treatment group in D is tric t X stressed student interaction and rewards more than did the teachers in the other two d is tric ts . Discussion o f Findings Less than 60 percent of a ll the s k ill-e x e rc is e objectives in the three basal series correlated with the MEAP objectives. does not mean that the objectives o f eith er are inappropriate. MEAP objectives are minimal, whereas many other are not. This The objectives of reading For example, skimming, rate o f reading, higher-level reason­ ing, and c re a tiv ity are important outcomes of reading that are gen­ e ra lly considered to be broader objectives than those id e n tifie d as essential components for further learning. The Houghton-Mifflin series had the most s k ill exercises and had nearly twice as many s k ill exercises with objectives th at corre­ lated with MEAP objectives as did the other two groups. However, the percentage of reading-series s k ill-e x e rc is e objectives that corre­ lated with MEAP objectives did not positively influence the test results 80 percent of the time, nor did the Houghton-Mifflin treatment group have the highest mean score on any of the fiv e categories of MEAP objectives tested. of fiv e o f the Four out mean scores onthe categories of MEAP objectives did not prove to be s ta tis tic a lly sig ­ n ific a n t. This indicated that the greatest amount of exposure, through the reading series, did not most po sitively a ffe c t test results. 102 The disproportionate size of made a difference in te s t scores: the treatment groups might have there were 11 classes in the Houghton-Mifflin treatment group, 12 classes in the H o lt, Rinehart, and Winston treatment group, and 30 classes in the Ginn treatment group. TheGinn treatment group was groups; the larger the group tested, much larg er than the other two the more lik e ly i t is that te s t­ ing errors w ill be balanced—with some children scoring below th e ir "true" scores and others scoring above. When a greater number of scores is used, this provides a wider range from which to generate mean scores. This may explain why the Ginn treatment group scored higher than did the Houghton-Mifflin treatment group. This, however, does not explain why the H olt, Rinehart, and Winston treatment group scored the highest, since this was the second smallest group of classes. The Primary-School Teachers' Questionnaire was administered to ascertain some of the other variables that might have influenced the MEAP test scores. I t was noted that the Houghton-Mifflin treatment group had the lowest mean score on c r itic a l reading, the only s ta tis ­ t ic a lly s ig n ifican t category of MEAP objectives; in keeping with these findings, the teachers in this group had the weakest scores on six out of eight o f the s ta tis tic a lly sig n ific an t categories of responses to the questionnaire. The teachers in this group indicated that they spent less time planning and teaching reading, had lessstructured organizational patterns, had fewer independent learning a c tiv itie s , had considerably less s ta ff support, and had fewer language-development lessons than did teachers in the other two 103 groups- Also, these teachers did not require students to examine and correct th e ir w ritten assignments as often as did the H o lt, Rinehart, and Winston group's teachers. Inasmuch as these variables did not seem to a ffe c t vocabulary meaning, lit e r a l and in fe re n tia l comprehen­ sion, and related study s k ills , they might have had more e ffe c t on the c r itic a l reading category than on the other four HEAP categories. The two s ta tis tic a lly sig n ifica n t categories o f the question­ naire on which the teachers from the Houghton-Mifflin treatment group scored the strongest were student interaction and rewards. A greater percentage of teachers from this group perceived that th e ir students received more tutoring from peers and other students and were given more treats and prizes when they performed w e ll. The Ginn treatment group had the second highest mean score on c r itic a l reading, although this group did not have any s k ill exercises with objectives that correlated with the MEAP objectives for Category 4, c r it ic a l reading. Their teachers, however, stressed independent learning, language development, and a more structured organizational pattern than did teachers in the other groups. These variables might have been more important than student interaction and rewards in learning to read c r i t ic a lly , thereby affectin g the mean scores more in the c ritic a l-re a d in g category than in the other four categories. The MEAP te s t results revealed th at the H o lt, Rinehart, and Winston treatment group had the highest mean score on the only v a ri­ able that was s ta tis tic a lly s ig n ific a n t: Category 4, c r it ic a l reading. This might have been attrib u ta b le to the reading series used, since 104 this group used the only reading series that had any sk ill-e x e rc is e objectives that matched the MEAP objectives fo r the c ritic a l-re a d in g category. Too, there were three questionnaire variables that might have influenced the test results in this category. The mean scores on the questionnaire showed that teachers from this group perceived that they devoted more time to a ll aspects of reading than did teach­ ers from the other groups. These teachers also perceived that they stressed w ritten work more and had more s ta ff support than did the teachers from the other two treatment groups. Written work, time, and s ta ff support might have had greater e ffe c t on c ritic a l-re a d in g scores than on the scores fo r the other MEAP reading-test categories. In summary, although there was no difference in the MEAP test results on four out of the fiv e categories of MEAP objectives, one reading series d iffered considerably on the number of objectives that matched MEAP objectives. The Houghton-Mifflin series had more s k ill exercises than did the other two series, and i t also had the highest percentage of s k ill-e x e rc is e objectives that matched MEAP objectives, but this treatment group did not have the highest scores on the MEAP te s t. Although the Houghton-Mifflin treatment group's reading-series objectives were the most congruent with MEAP objectives, its mean scores on the MEAP te s t were not the highest of the three groups. It is assumed that eith er the variables th at were considered or other confounding variables affected the te s t resu lts. Some o f the v a ri­ ables that were discussed in this chapter were qu ality of instruction, 105 teachers' methods and strateg ies, students' a b ilit ie s , testing condi­ tions, mechanical errors, and disproportionately sized treatment groups. Also, the gross reporting o f scores might have made d if f e r ­ ences in MEAP te s t results indiscernible. The results o f this study can be generalized only to the three populations or treatment groups. This study suggested that exposure to the objectives o f reading and/or to some methods and strategies of teachers might have influenced the te s t scores fo r c r itic a l reading. There was not enough evidence to determine which variables had the greatest e ffe c t on the overall reading process or on specific aspects of reading such as word meaning, comprehension, and related study s k ills . I f one knew which variables contributed most to beginning reading, i t would be possible to delineate other variables and to ascertain th e ir causes and effects in various student populations. Conclusions of the Study Conclusions were derived from the analysis of the three types of data generated in the study. Contingency tables were used to compare the reading-series objectives found in workbooks and teachers' manuals with the MEAP objectives; they were also used to make compari­ sons among the s k ill-e x e rc is e objectives o f the three reading series according to the proportion o f matches and mismatches across the 25 MEAP objectives. One-way univariate analysis of variance was used to determine (1) the significance levels of mean scores on the MEAP reading te st by categories of MEAP objectives and (2) the sig n ifican t differences among the mean scores of responses to the 15 categories of questions on the teachers' questionnaire. 106 There was a difference between the s k ill-e x e rc is e objectives of each of the three reading series and the MEAP objectives. Although the Ginn and Houghton-Mifflin series had approximately 300 more exer­ cises than the H o lt, Rinehart, and Winston series, the two series had nearly the same number of s k ill exercises with objectives that matched MEAP objectives. The Houghton-Mifflin series had almost twice as many s k ill exercises with objectives that matched MEAP objectives as did the other two series. Four hundred th ir ty out of a to ta l of 1,327 Ginn-series s k ill exercises had objectives that matched MEAP objectives; 418 out o f a total o f 1,094 H o lt, Rinehart, and Winston series s k ill exercises had objectives that matched MEAP objectives; and 828 out o f a to ta l of 1,399 Houghton-Mifflin series s k ill exer­ cises had objectives that matched MEAP objectives. Comparing the three reading series across the MEAP objectives revealed that 59 percent of the Houghton-Mifflin series objectives matched MEAP objectives, but only 32 percent of the Ginn series objec­ tives and 38 percent of the H olt, Rinehart, and Winston series objec­ tives matched MEAP objectives. The Houghton-Mifflin series objectives were the most congruent with MEAP objectives o v e ra ll. However, the H olt, Rinehart, and Winston series was the most congruent with the MEAP objectives in Category 4, c r itic a l reading, which yielded the only s ta tis tic a lly s ig n ific a n t variable in the ANOVA tests. Since the Houghton-Mifflin series objectives were the most congruent with MEAP objectives o v e ra ll, i t was expected that this treatment group would have the highest MEAP reading-test scores. However, this was not the case. The mean scores on the MEAP test 107 revealed a sig n ific a n t difference among the treatment groups on only one category of MEAP objectives—Category 4, c r itic a l reading, on which the H o lt, Rinehart, and Winston treatment group scored highest. The Ginn treatment group scored second highest, and the HoughtonM iffl in treatment group scored th ird . Other variables that might have made a difference in the MEAP results were ascertained from a teachers' questionnaire. The mean scores on the 15 categories of responses to this questionnaire were s ta tis tic a lly s ig n ific a n t on eight of the categories. The teachers from the Ginn treatment group in D is tric t Z had the strongest mean scores on Categories 6, 7, and 12: organizational pattern, inde­ pendent learning, and language development. The teachers from the H olt, Rinehart, and Winston treatment group in D is tric t Y had the strongest mean scores on Categories 5, 11, and 13: support, and w ritten work. tim e, s ta ff The teachers from the Houghton-Mifflin treatment group in D is tric t X had the strongest mean scores on Cate­ gories 8 and 15: student interaction and rewards. The mean scores on the 15 categories o f responses to the questionnaire were not s ta tis tic a lly sig n ifican t in seven of the categories. These categories included classroom enrichment, home­ work, teacher-made and other m aterials, parental involvement, outside enrichment, use of basals, and test uses. Reflections From the Study Although the scope of this study was lim ited to specific reading series and the MEAP te s t, i t generated many questions that have not been answered or that are debatable. Because o f the 108 complexity of the beginning reading process, i t is not an easy fie ld to study. There is a need fo r sorting out and carefully weighing edu­ cational practices that should promote further learning. Based on this study, some of the questions that require careful consideration are as follows: 1. Are criterion-referenced tests the best measures of success in school? 2. What are some ways to adequately assess transfer of learning? 3. How can adequacy of exposure to a learning process be determined? 4. Are there other evaluative procedures that are better measures of learning than are criterion-referenced tests? 5. Should teacher-training programs address ramifications of competency testing? 6. Should objectives that w ill be tested in various commer­ c ia l testing programs be id e n tifie d and/or stressed in basal reading materials? 7. Should teacher accountability be dependent on student test results? 8. What are some of the major effects o f tests on teaching? 9. Is i t possible, through criterion-referenced te stin g , to diagnose specific problems that children have in reading even though these problems have come from a number of d iffe re n t sources? 109 10. Can the effectiveness of a program be determined without measures of the a ffe c tiv e domain? What are the implications and ju s tific a tio n s of testing in the a ffe c tiv e areas? Heed fo r Further Research Analyses of the relationship between reading theories and MEAP objectives were conducted using the fiv e categories of objectives on the MEAP reading test and a ll of the s k ill exercises found in workbooks and teachers' manuals of each o f the reading series fo r grades one through three. Based on the fa ct that congruence between basal reading materials and test content did not a ffec t te st resu lts, other concepts concerning appropriate learning procedures and teaching techniques need to be examined. A number o f p o s s ib ilitie s for further research have grown out of these findings and from reflections on this study. These include: 1. A random-block design could be used for each d is tr ic t to analyze data fo r more homogeneous groups—that is , fo r students with sim ilar a b ilit ie s , in terests, strengths, or weaknesses. These variables might influence how and what students learn. 2. A s tr a tifie d sample would insure that certain subgroups in the population would be included in each sample. In this method, students could be grouped in three levels on the basis o f general in te llig e n ce ; this would permit comparisons among subgroups of sim ilar a b ilit ie s . 3. Randomly selecting a sample from each of the classes within the d is tric ts would insure a balanced sample size that could be small enough to eliminate gross reporting o f scores. 110 4. Demographic data could be gathered to determine the degree of s im ila rity or difference among students in the three dis­ t r ic t s . These data could include the I.Q .; levels of students with high, medium, and low a b ilit ie s ; the number o f special-education students; and/or the number of T it le I students in each group. 5. Teacher interviews could be used to ascertain (1) the number of MEAP objectives taught in classes other than reading, (2) the amount of time spent on MEAP objectives, (3) what evaluative procedures other than criterion-referenced tests are used, (4) teacher and student attitudes toward te stin g , and (5) a ffe c tiv e reading goals of the teacher. 6. This study could be replicated with the use o f more recent lite ra tu re and basal-reading series that would r e fle c t current knowledge concerning the reading process, 7. Classroom observations might be used to determine the e ffe c t o f teacher methods and strategies on the beginning reading process today, inasmuch as great advances have been made in determin­ ing how students learn In conclusion, to read. to assure that evaluation does not become the major concern in teaching, i t is v ita l to focus on a variety of v a ri­ ables that a ffe c t the reading process. Many variables need to be considered, and the most important ones need to be closely delineated so that the effects of instructional programs can more easily be ascertained. Inasmuch as reading authorities do not agree on which variables have the greatest influence on the beginning reading process, i t is important continually to explore more and d iffe re n t p o s s ib ilitie s . APPENDICES APPENDIX A QUESTIONNAIRE DATA 112 113 September 1981 Dear p ilo t-te s t volunteer, To prepare fo r my dissertation survey, I need your response to the enclosed questionnaire. The purpose o f this p ilo t-te s t is not to acquire data, but to enable me c r it ic a lly to evaluate the question­ naire i t s e l f , which w ill be used la te r to gather data reflec tin g various sources that may contribute to student achievement in reading. The questionnaire assumes that you are a primary-school teacher who is responsible fo r teaching reading on a regular basis. Please follow these directions: 1. Time yourself to ascertain how long i t takes you to complete the questionnaire; answer the questions at one s ittin g . 2. Correct any errors in spelling. 3. Comment on confusing directions and/or questions. Comment on the organization of the questionnaire, its wording, e tc.--o n anything that might confuse teachers. 4. Note anything you feel would improve the questionnaire. Your input is c r it ic a l in my determining needed changes in the survey instrument. Please be honest; your evaluation is important! You w ill fin d an additional sheet of paper fo r your comments. Please complete both the questionnaire and the comment pages and return them to me as soon as possible. Thank you fo r your help. I t is greatly appreciated. Sincerely, Phyllis H. Music 114 PRIMARY-TEACHERS’ READING QUESTIONNAIRE GENERAL DIRECTIONS: For research purposes, i t is important that you c irc le one response fo r each question (o f this questionnaire). Mark your choices c le a rly in pencil. Erase completely when necessary. PART I This section is regarding background information. 1 answer for each. C ircle only 1. Grade teaching (1 s t, 2nd, 3rd, combination o f these) 2. Years of teaching (0 -3 , 4 -7 , 8-11, 12-more) 3. Years of teaching at the present grade level (0 -3 , 4 -7 , 8-11, 12-more) Number of grades in the building (K-3, K-4, K-5, K-6, other 5. Basal reading series used in your building through 1979-80 school year) (Houghton-Mifflin, Scott Foresman, H o lt, Ginn, o th e r _______________) 6. Copyright date of series indicated in number 5 4. 7. (1969-70, 1971-2, 1973-4, 1975-6, 1977-8, other __________________ ) My students have (many, few, no) opportunities fo r educational experiences or enrichment a c tiv itie s outside of the school environment. 115 PART I I . Primary Reading Instruction Below are a variety of statements related to various strategies and methods used in teaching reading. Consider your own methods. C ircle one choice fo r each of the following statements that would id e n tify your own p articu lar methods and/or classroom situ atio n . X = Always X = Most t^e t™ 6 2 = Sometimes £ = Seldom £ = Never 1. Instructional b u lle tin boards are used as a part of my teaching method. 2. 12 3 4 5 Reading centers are used to stimulate and motivate students to do reading a c tiv itie s . 12 3 4 5 3. I read aloud to my students. 12 3 4 5 4. I share my own personal stories with children. 12 3 4 5 5. A variety of hands-on a c tiv itie s are provided in my room. 12 3 4 5 6. I use games and simulations to teach reading. 12 3 4 5 7. Students are required to do reading homework for my class. 12 3 4 5 Homework is used as a means fo r students to fin is h assignments that were begun in class. 1 2 3 45 I use teacher-made materials to teach reading. 12 3 4 5 I use a variety of other resources as well as the basal reading program (SRA, Reader's Digest, commercially prepared m aterials, e t c .) . 12 3 4 5 8. 9. 10. 11. 12. 13. 14. 15. I use enrichment a c tiv itie s that are not suggested in the teachers' guides. 12 3 4 5 I work on reading s k ills that are not included in the instructional reading program. 12 3 4 5 The students' parents are w illin g resources fo r my class. 12 3 45 The parents evidence concern fo r th e ir children's progress. 12 3 45 The parents contribute to th e ir children's foundations in basic s k ills . 12 3 45 116 16. At least 50 minutes is a llo tte d fo r each reading instruction period. 12 3 4 5 17. More time is spent on teaching basic s k ills than on the application o f newly acquired s k ills . 12 3 4 5 18. I spend more time discussing ideas with students than instructing students. 12 3 4 5 19. I spend more than 30 minutes d aily in preparation for the teaching of reading. 12 3 4 5 20. Structured programs work better than lessstructured ones fo r me. 12 3 4 5 21. In my room, students are grouped according to a b ilit y fo r instructional purposes. 12 3 4 5 22. The seating arrangement is by rows. 12 3 4 5 23. There are more than 27 students in my group. 12 3 4 5 24. We use team teaching to teach reading. 12 3 4 5 25. My students remain the same reading group fo r the whole school year. 12 3 4 5 26. I use an individualized approach to reading. 27. 12 3 4 5 Older students come into my room to tutor the younger ones. 12 3 4 5 28. Classmates are encouraged to tutor each other. 12 3 4 5 29. My students spend th e ir free time outside of school watching te le v is io n . 12 3 4 5 There is evidence that my students read at home as well as at school. 12 3 4 5 30. 31. My students particip ate in ex tra-c u rric u la r a c tiv itie s such as scouts, after-school sports, music, etc. 12 3 4 5 32. I emphasize and encourage s e lf-in itia te d learning. 12 3 4 5 33. My students read in the content areas during reading class time. 12 3 4 5 My students use the lib ra ry resources fo r a variety o f a c tiv itie s . 12 3 4 5 34. 117 35. My students use the lib ra ry as an informa­ tional resource. 12 3 45 36. My students require detailed directions before they can begin assignments. 12 3 45 37. The students are permitted to choose the books th at they wish to read. 12 3 45 38. Students are permitted to choose the order in which they prefer to cover classroom assignments. 12 3 45 39. I use the basal reading program that has been adopted by my school system. 12 3 45 40. I rely on the teacher's manual fo r instructional ideas and information. 12 3 45 41. Workbooks are used as a part of my instructional program. 12 3 45 42. Workbooks are used with a ll reading groups that I teach. 12 3 45 I use the enrichment a c tiv itie s that are suggested in the teachers' guides. 12 3 45 I provide extra help for students who need i t during breads, a fte r school, a t recess time, etc. 1 2 3 45 45. A reading s p ecialist works with some o f my students. 12 3 45 46. My students are encouraged to share stories that they have read. 1 2 3 45 My students are encouraged to share information that they have read. 1 2 3 45 More time is spent discussing reading ideas and stories than doing w ritten a c tiv itie s . 1 2 3 45 Students are required to examine th e ir w ritten work to find and correct th e ir mistakes. 1 2 3 45 43. 44. 47. 48. 49. 50. I use standardized te s t scores to establish reading groups. 1 2 3 45 51. I use te st information to a lte r my instructional program. 1 2 3 45 52. My students are rewarded fo r doing good work (p rizes, tre a ts , e t c .) . 1 2 3 45 118 COMMENTS Time required to complete the questionnaire: Other comments.. . . Please be sure to complete and return to me as soon as possible. Thank you! Phyllis Music 119 Survey Instrument Changes A reading sp e c ia lis t was consulted fo r recommendations on how to improve the questionnaire. She recommended, fo r Part I , adding Ques­ tion 8 and changing Question 7 to a continuum form; too, she recom­ mended adding Questions 44, 54, and 55 in Part I I . As a resu lt o f the p ilo t-te s t, the following changes were made: 1. "Stories" was changed to "experiences" in Question 4. 2. "Evidence" was changed to "show" in Question 14. 3. "Contribute to" was changed to "help" in Question 15. 4. "Basic" was inserted before the la s t word in Question 17. 5. "Yearly" was added to indicate the usual number, instead of the specific year, in Question 23. 6. The typographical error "breads" was changed to "breaks" in Question 46. 120 Table A1 . —Questions included in each questionnaire category. Questionnaire Categories Question Numbers 1 Classroom enrichment 1-6 2 Homework 7-8 3 Teacher-made and other materials 9-12 4 Parent involvement 13-15 5 Time 16-19 6 Organizational pattern 20-21 7 Independent learning 32-38 8 Student interaction 27-28 9 Outside enrichment 29-31 10 Use of basal materials 39-44, 55 11 S ta ff support 45-46, 54 12 Language development 47-49 13 Written work 50 14 Tests 51-52 15 Rewards 53 Total number of questions = 55. 121 October 5, 1981 Dear Primary-School Teachers, I need your help! I am conducting a study o f primary-school teachers to ascertain what methods, m aterials, and strategies they use to teach reading. The data w ill be used in my dissertation research to determine what variables other than curriculum materials influence how and what students learn in reading. Input is needed from school d is tric ts that use specific basal reading curriculums. I have spoken with . . . and . . . about the reading program used in and they have permitted me to conduct a survey of the teachers in your d is t r ic t . Won't you please take 15 minutes o f your valuable time to complete the enclosed questionnaire? I hope the data gathered w ill provide addi­ tional insights into e ffe c tiv e reading practices and so help a ll of us who s triv e to become better teachers. C o nfid en tiality in research is important. At no time w ill the respondents to questionnaires be id e n tifie d . You w i l l , however, notice a code number on your questionnaire; this number w ill be used only to f a c ilit a t e follow-up procedures and to prevent you from receiving bothersome reminders. Please complete the questionnaire and return i t to your p rin c i­ pal's o ffic e no la te r than October 12. I t w ill be forwarded to me. Thank you in advance fo r your time and cooperation. both greatly needed and appreciated. They are Sincerely, Phyllis H. Music 658 Bay East Drive Traverse C ity , Michigan 49684 122 PRIMARY-TEACHERS1 READING QUESTIONNAIRE GENERAL DIRECTIONS: For research purposes, i t is important that you c irc le one response fo r each question on this questionnaire. Mark your choices clearly in pencil. Erase completely when necessary. PART I This section is regarding background information. answer fo r each. C ircle only 1_ 1. Grade teaching (1 s t, 2nd, 3rd, combination of these) 2. Years o f experience (0 -3 , 4 -7 , 8-11, 12-more) 3. Years o f teaching at the present grade level (0 -3 , 4 -7 , 8-11, 12-more) Number o f grades in the building (K-3, K-4, K-5, K-6, o t h e r __________ ] Basal reading series used in your building through 1979-80 school year (Houghton-Mifflin, Scott Foresman, H olt, Ginn, o th e r ___________________) Copyright date of series indicated in number 5 (1969-70, 1971-2, 1973-4, 1975-6, 1977-8, other ___________________ 4. 5. 6. 7. Where do your students f i t on the continuum below? (Place an £ to indicate the number o f opportunities that your students have for educational and enrichment experiences outside the school environment.) many 8. none I have had (0 -1 , 2 -3, 4 -5 , 6-more) courses in how to teach reading to elementary students. 123 PART I I Primary Reading Instruction Below are a variety of statements related to various strategies and methods used in teaching reading. Consider your own methods. C ircle one choice fo r each of the following statements that would id e n tify your own p articu lar methods and/or classroom s itu a tio n . The choices are: X = Always 1. X = Most o f the time 2_ = SometimesX = Seldom X “ Never Instructional b u lle tin boards are used as a part of rry teaching method. 1 2 3 4 5 Reading centers are used to stimulate and motivate students to do reading a c tiv itie s . 12 3 4 5 3. I read aloud to my students. 12 3 4 5 4. I share my own personal experiences with children. 12 3 45 5. A variety of hands-on a c tiv itie s are provided in my room. 12 3 4 5 6. I use games and simulations to teach reading. 12 3 45 7. Students are required to do reading homework for my class. 12 3 45 Homework is used as a means fo r students to fin ish assignments that were begun in class. 12 3 45 I use teacher-made materials to teach reading. 1 2 3 45 I use a variety of other resources as well as the basal reading program (SRA,Reader's Digest, commercially prepared m aterials, e t c .) . 12 3 45 I use enrichment a c tiv itie s that are not suggested in the teachers' guides. 12 3 45 I work on reading s k ills that are not included in the instructional reading program. 1 2 3 4 5 The students' parents are w illin g resources fo r my class. 12 3 45 The parents show concern for th e ir children's progress. 12 3 45 2. 8. 9. 10. 11. 12. 13. 14. 124 15. The parents help th e ir children to acquire basic reading s k ills . 12 3 4 5 16. At least 50 minutes is a llo tte d for each reading instru ctio n period. 12 3 4 5 17. More time is spent on teaching basic s k ills than on the application o f newly acquired basic s k ills . 12 3 4 5 18. I spend more time discussing ideas with students than instru ctin g students. 12 3 4 5 19. I spend more than 30 minutes daily in preparation fo r the teaching of reading. 12 3 4 5 Structured programs work better than lessstructured ones fo r me. 12 3 4 5 Students are grouped according to a b ility for instructional purposes in my room. 12 3 4 5 22. The seating arrangement is by rows. 12 3 4 5 23. There are more than 27 students assigned to my classroom y e a rly . 12 3 4 5 24. We use team teaching to teach reading. 12 3 4 5 25. My students remain in the same reading group throughout the school year. 12 3 4 5 26. I use an ind ividu alized approach to reading. 12 3 4 5 27. Older students come in to my room to tutor the younger ones. 12 3 4 5 28. Classmates are encouraged to tutor each other. 12 3 4 5 29. My students spend th e ir free time outside of school watching te le v is io n . 12 3 4 5 There is evidence th at my students read at home as well as a t school. 12 3 4 5 20. 21. 30. 31. 32. My students p a rtic ip a te in extra-cu rricular a c tiv itie s such as scouts, after-school sports, music, etc. 1 2 3 45 I emphasize and encourage s e lf-in itia te d learning. 12 3 4 5 125 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. My students read in the content areas during reading class time. 12 3 4 5 My students use the lib ra ry resources fo r a variety o f a c tiv itie s . 12 3 4 5 My students use the lib ra ry as an informational resource. 12 3 4 5 My students require detailed directions before they can begin assignments. 12 3 4 5 The students are permitted to choose the books th at they wish to read during s ile n t reading periods. 12 3 4 5 Students are permitted to choose the order in which they prefer to cover classroom assignments. 12 3 4 5 I use the basal reading program that has been adopted by my school system. 12 3 4 5 I rely on the teacher's manual for instructional ideas and information. 12 3 4 5 Workbooks are used as a part of my instructional program. 12 3 4 5 Workbooks are used with the reading groups that I teach. 12 3 4 5 I use the enrichment a c tiv itie s that are sug­ gested in the teachers' guides. 12 3 4 5 My students must complete a textbook level before going on to the next higher le v e l. 12 3 4 5 I provide extra help fo r students who need i t during breaks, a fte r school, at recess time, etc. 12 3 4 5 A reading sp e c ia lis t works with some of my students. 12 3 4 5 My students are encouraged to share stories that they have read. 12 3 4 5 My students are encouraged to share information th at they have read. 12 3 4 5 More time is spent discussing reading ideas and stories than doing w ritten a c tiv itie s . 12 3 4 5 126 50. 51. 52. 53. 54. 55. Students are required to examine th e ir w ritten work to find and correct th e ir mistakes. 12 3 4 5 I use standardized te s t scores to establish reading groups. 12 3 4 5 I use test information to a lte r my instructional program. 12 3 4 5 My students are rewarded fo r doing good work (p rizes, tre a ts , e t c .) . 12 3 4 5 Inservices are provided when niy school system adopts a new basal reading program. 12 3 4 5 Once a reading textbook is begun, I am expected to have students complete i t . 12 3 4 5 APPENDIX B MEAP CLASSROOM-LISTING REPORT 127 CLASSROOMLISTING REPORT MICHIGAN EDUCATIONAL ASSESSMENT PROGRAM 1980-31 TEACHER: SCHOOL GRADE: SECTION: o DISTRICT; II H 1 Sk i w i b n d r to. SKUt AREA and OSjECllVt CESOIIPHOM R E A D IN G O B J E C T IV E CODE ?£! STUDENT NAME I A 1 B i c I D I E 1 F I I [ I 11 B C E a F Li G a i [It [II III [II III [II III n r A C 0 D G E F K i H H IV A V A V D N H V D V F II N N H H N H H 24 24 M N H M N 11 H M N U U H H H 11 II ri H n N N 11 N H It II H N N N H n H H H N H H N N H M N H H N H H II H II K N 25 10 ?3 25 25 N 11 N 25 21 21 12 25 n II H H N H H B9 * 6 36 36 n 35 A 33 3? n A B9 tw 93 A9 93 A9 PS f il 19 22 17 21 25 25 15 ?5 24 25 u PERCENT OF STUDENTS ATTAINING OBJECTIVEI 21 23 N N H ■3= S-* 24 25 B5 70 ft9 93 74 74 BIBLIOGRAPHY 129 BIBLIOGRAPHY Airasian, P ., and Madaus, G. Measurement in Education, May 1972. Citing E. L. Thorndike. Seventeenth Yearbook of the National Society fo r the Study of Education, 19l8. An Assessment of Michigan Accountability System. Department of Education, May 1974. Lansing: Bagford, Jack. Instructional Competence in Reading. Charles E. M e rrill Publishing Co., 1975. Michigan Columbus, Ohio: Beck, Michael, and S tetz, Frank P. "Standardized Testing as Viewed by Test Specialists and Users." Paper presented a t the National Council on Measurement in Education, Washington, D.C., 1980. Berdie, Douglas, and Anderson, John F. Questionnaires: Design and Use. Metuchen, N .J.: The Scarecrow Press, In c ., 1974. —— B iem iller, A. "The Development of the Use o f Graphic and Contextual Information as Children Learn to Read." Reading Research Quarterly 6 (Fall 1970): 75-96. Blachowicz, Camille L. "Reading Objectives: A Competency-Based Accountability Model." The Reading Teacher 28 (April 1975): 645-61. Bloom, B. S. Human Characteristics and School Learning. Quoted by Jeanne C h all. Reading 1967-1977: A Decade of Change and Promise. Bloomington, Indiana: The Phi Delta Kappa Educational Foundation, 1977. Canney, George. "Organizing and Applying Test Results." In Reading Tests and Teachers: A Practical Guide. Edited by Robert Schreiner. Newark, Delaware: International Reading Association, In c ., 1979. C hall, Jeanne. Reading 1967-1977: A Decade of Change and Promise. Bloomington, Indiana: The Phi Delta Kappa Educational Foundatio n , 1977. Clay, Marie M. "Reading Errors and Self-Correction Behavior." B ritish Journal of Educational Psychology 39 (1969): 47-56. Clymer, Theodore, and B arrett, Thomas C. My Sound and Word Book. Lexington, Mass.: Ginn and Co., 1973. 130 131 Clymer, Theodore, and Jones, V irgin ia W. Mass.: Ginn and Co., 1973. Clymer, Theodore, and Parr, B il l i e . Mass.: Ginn and Co., 1973. Seven Is Magic. ADuck Is aDuck. . Helicopters and Ginqerbread. Co., 1973. Lexington, Lexington, Lexington, Mass.: Ginn and Clymer, Theodore, and Ruddell, Robert B. How I t Is Nowadays. Lexington, Mass.: Ginn and Co., 1973. Clymer, Theodore, and Wolfe, Elaine Vi 1seek.The Dog Next Door and Other S tories. Lexington, Mass.: Ginn and Co., 1973. Clymer, Theodore, and Wolfing, Gretchen. With Skies and Wings. Lexington, Mass.: Ginn and Co., 1973. Clymer, Theodore, and Wyatt, Nita Ginn and Co., 1973. M. May I Come In?Lexington, Mass.: Cooper, Charles, and Petrosky, Anthony. "A Psycholinguistic View of the Fluent Reading Process." Journal of Reading 20 (1976): 184-207. Crawley, Sharon. "SOB: Selection ciation Journal 12,2 (1977), Basal s . 11 HewEngland p. 3. ED155 588. ReadingAsso­ Criscuolo, Nicholas P. Improving Classroom Reading Instru ctio n. Worthington, Ohio: Charles A. Jones Publishing Co., 1973. Duffy, Gerald G. "Maintaining a Balance in Objective-Based Reading Instruction." Reading Teacher 31 (February 1978): 519-23. ________, and Sherman, George B. How to Teach Reading System atically. New York: Harper & Row, 1973. Durkin, Dolores. Teaching Young Children to Read. Bacon, In c ., 1980. Durr, William; LePere, Jean M.; and A lsin, Mary Lou. Boston: Houghton-Mifflin Co., 1971, 1974. ________. Lions. Boston: ________. Rainbows. . Secrets. .• Signposts. Boston: Allyn & Dinosaurs. Houghton-Mifflin Co., 1971 , 1974. Boston: Boston: Boston: Houghton-Mifflin Co., 1971 , 1974. Houghton-Mifflin Co., 1971, 1974. Houghton-Mifflin Co., 1971 , 1974. 132 _ . Tigers. Boston: Houghton-Mifflin Co., 1971 , 1974. Durr, William; LePere, Jean M.; and Brown, Ruth. Houghton-Mifflin Co., 1971, 1974. ________. Panorama. Boston: Fiesta. Boston: Houghton-Mifflin Co., 1971 , 1974. Durr, William; LePere, Jean M,; and Niehaus, Bess. Houghton-Mifflin Co., 1971, 1974. Rewards. Ebel, R. L. "Blind Guessing on Objective Achievement Tests." of Educational Measurement 5 (1968): 321-25. . "The Case fo r Minimum Competency Testing." (April 1978): 546. Boston: Journal Phi Delta Kappan ________. "One of the Troubles With Testing." College o f Education Newsletter (Michigan State University) 3 (S p rin g 198 TJ. ________ . In Readings in Measurement and Evaluation in Education and Psychology. Edited by William A. Mehrens. New York: H olt, Rinehart, and Winston, 1976. ________. The Uses of Standardized Testing. Fastback 93. Bloomington, Indiana! The Phi Delta Kappa Educational Foundation, 1977. Farr, Roger. Reading: What Can Be Measured? IRA/ENA Knight Research Fund Monograph. Newark, Delaware! Reading Association, In c ., 1969. . "Standardized Reading Tests." In Reading fo r A l l . Edited By”Robert K arlin. Newark, Delaware: International Reading Asso­ c ia tio n , 1972. Farr, Roger, and Brown, Virginia L. "Evaluation and Decision Making." In Elementary Reading. Edited by Althea Beery, Thomas C. B a rre tt, and William R. Powell. Boston: Allyn & Bacon, In c ., 1974. FIoden, Robert E .; Porter, Andrew C.; Schmidt, William H.; and Freeman, Donald J. Don't They A ll Measure the Same Thing? Consequences of Selecting Standardized Tests. East Lansing: In s titu te fo r Research on Teaching, Michigan State U niversity, July 1978. Freeman, Donald; Kuhs, Therese; Knappen, Lucy, and Porter, Andrew. A Closer Look at Standardized Tests. East Lansing: In s titu te fo r Research on Teaching, Michigan State U niversity, November 1978. Guthrie, John. "Research Testing: Uses and V is ib ilit y ." Reading 23 (March 1980): 542-44. Journal of 133 ________• "Vocabulary Control." 1979): 240-42. The Reading Teacher 33 (November ---------------- -------------- H a ll, Mary Anne; Ribovich, Jerilyn K .; and Rameg, Christopher J. Reading and the Elementary School C hild. New York: D. Van Nostrand Co., 1979. Harmer, William R. "The Selection and Use o f Survey Reading Achieve­ ment Tests." In The Evaluation of Children's Reading Achievement. Edited by Thomas £. B arrett. Newark, Delaware: International Reading Association, In c ., 1967. Harris, Albert J . , and Si pay, Edward R. How to Increase Reading A b ilit y . New York: David McKay Co., 1976. Hood, Joyce. "Sight Words Are Not Going Out of S tyle." Teacher 33 (January 1977): 379-82. The Reading House, Ernest R.; Rivers, Wendell, and Stufflebeam, Daniel L. "An Assessment o f the Michigan Accountability System." Phi Delta Kappan 55 (June 1974): 663-69. Huey, Edmund Burke. The Psychology and Pedagogy of Reading. Cambridge, Mass.! The M .I.T . Press, 1977. Jenkins, Joseph R ., and Pany, Darlene. "Curriculum Biases in Reading Achievement Tests." Journal of Reading Behavior 10 (Winter 1978): 345-57. ________. Teaching Reading Comprehension in the Middle Grades. Reading Education Report No. 4. Urbana-Champaign: University of I llin o is , January 1978. ED 151 756. Jones, Timothy. Information Services S p e c ia lis t, D etroit Census Bureau. Telephone interview , July 12, 1982. Kappan Interview. "The Father of Behavioral Objectives C ritic ize s Them: An Interview With Ralph T y le r," Phi Delta Kappan 55 (September 1973): 55-56. Kavali, Kenneth. "Selecting and Evaluating Reading Tests." In Reading Tests and Teachers: A Practical Guide. Edited by Robert ScTTreiner. Newark, Delaware: International Reading Association, In c ., 1979, Kearney, C. P h ilip ; Donovan, David L .; and Fisher, Thomas H. "In Defense of Michigan's Accountability Program." Phi Delta Kappan 56 (September 1974): 14-19. 134 Klein, Stephen P ., and Kosecoff, Jacquiline. "Issues and Procedures." In Readings in Measurement and Evaluation in Education and Psy­ chology. Edited by William A. Mehrens. New York: H o lt, Rinehart and Winston, 1976. Lovell, Ned B.; Riegle, Rodney P.; and Bunke, Clinton R. "Minimal Competency Testing: Hopes, Fears, and F a llacies." The Educa­ tional Forum 45 (January 1981): 199-206. Lundsteen, Sara W. "Levels of Meaning in Reading." Teacher 28 (December 1974): 268-72. The Reading Market Data R etrieval, In c ., HM Co. Reading K-8 Survey. Market Research Report. New York: Market Data R e trie va l, In c ., 1977. McClutcheon, G ail. "How Do Elementary School Teachers Plan? The Nature of Planning and Influences on I t . " The Elementary School Journal 81 (September 1980): 4-22. McDonald, A. S. fic ie n c y ." "Using Standardized Tests to Determine Reading Pro­ Journal of Reading 8 (1964): 58-61. Mehrens, William A ., and Ebel, Robert L. "Some Comments on C rite rio n Referenced and Norm-Referenced Achievement Tests." NCWE Measure­ ment in Education, Vol. 10, No. 1. Washington, D.C.: National Council on Measurement in Education, Winter 1979. Mehrens, William A ., and Lehmann, Irv in J. Standardized Tests Education. New York: H olt, Rinehart and Winston, 1980. in Michigan Educational Assessment Program. Technical Report, Vol. 1. Lansing: Michigan State Board o f Education, 1980. Michigan Educational Assessment Program Handbook. Department of Education, 1980. Lansing: Michigan The National Conference on Achievement Tests and Basic S k ills . Washington, D.C.: The National In s titu te of Education, U.S. Department of Health, Education, and Welfare, February 1979. ED 171 784. Otto, Wayne, and Chester, Robert D. Objective-Based Reading. Mass.: Addison-Wesley Publishing Co., 1976. Reading, Perrone, V ito . The Abuses of Standardized Testing. Fastback 92. Bloomington, Indiana: The Phi Delta Kappa Educational Foundation, 1977. ________. "Achievement Testing: An Overview o f the Issues." The National Conference on Achievement Tests and Basic S k ills . Washington, D.C.: The National In s titu te of Education, U.S. Department of Health, Education, and Welfare, February 1979. 135 Pipho, Chris. State A c tiv ity Minimal Competency Testing. Denver, Colorado: Education Commission of the States, October 5, 1978. Popham, James W. "Normative Data fo r C riterion Referenced Tests?" Phi Delta Kappan 57 (May 1976): 593-94. Porter, John W. "Achievement Testing: The In terests." The National Conference on Achievement Tests and Basic S k ills . Conference Proceedings. Washington, D.C.: National In s titu te of Education, U.S. Department of Health, Education, and Welfare, March 1978. "A Position of Minimum Competencies in Reading." 23 (October 1979): 50-51. Journal of Reading Prescott, George. "Criterion-Referenced Test In te rp re ta tio n ." In Elementary Reading Instruction: Selected M a te ria l. Edited by Althea Beery, Thomas' C'.' B arrett,' and William R. Powell. Boston: Allyn & Bacon, In c ., 1974. Price, Gary. "Standardized Achievement Tests fo r Young Children: An Analysis." Paper presented at the Big Ten Symposium on Early Childhood Education, Madison, Wisconsin, October 1978. Questions and Answers About the Michigan Educational Assessment Proqram. La'n'sVnq: Michigan Department of Education, September 1980. Quinto, Frances, and McKenna, B. Alternatives to Standardized Testing. Washington, D.C.: National Education Association, 1977. Resnick, Daniel P. "Testing in America: A Supportive Environment." Phi Delta Kappan 62 (May 1981): 625-28. Resnick, Lauren B. "Achievement Testing: An Overview o f the Issues." The National Conference on Achievement Tests and Basic S k ills , Washington, D.C.: The National In s titu te of Education, U.S. Department of Health, Education, and Welfare, February 1979. ED 171 784. "Introduction: Research to Inform a Debate." Kappan 62 (May 1981): 623-24. Phi Delta Roeber, Edward D.; Donnovan, David L .; and Cole, Richard F. "Telling the Statewide T Sting Story . . . And Living to Tell I t Again!" Phi Delta Kappan 62 (December 1980): 273-74. Rosecky, Marion, "Are Teachers Selective When Using Basal Guidebooks?" Reading Teacher 31 (January 1978): 381-84. Rosenshine, B. "S kill Hierarchies in Reading Comprehension." In Theoretical Issues in Reading Comprehension. Edited by R. J. Spiro, B. C.'Tfruce, and W. F. Brewer. H ills d a le , N .J.: Erlbaum, 1978. 136 Spache, George D. The Teaching o f Reading. Delta Kappan, In c ., 1972. Bloomington, Ind.: Phi ________* and Spache, Evelyn B. Reading in Elementary Schools. Boston: Allyn & Bacon, Inc.TT§737 Tallmadge, Kasten G ., and Horst, Donald P. "The Use o f D ifferent Achievement Tests in the ESEA T it le I Evaluation System." Paper presented at the 62nd Annual Meeting of the American Educational Research Association, Toronto, Ontario, Canada, 1978, Turnbull, William. Quoted by John Guthrie. "Research Testing: Uses and V is ib ilit y ." Journal of Reading 23 (March 1980): 542-44. Tyler, Ralph W., and White, Sheldon H. Testing, Teaching, and Learn­ ing. Washington, D.C.: The National In s titu te of Education, U.S. Department of Health, Education, and Welfare, August 1979. Waldon, Shirley. Supervisor of State Aid, Michigan Department of Education. Telephone interview , July 12, 1982. Weiss, Bernard J .; Evertts, El donna L .; and Stever, Loreli Olson. Never Give Up. New York: H o lt, Rinehart, and Winston, 1977. . People Need People. 1977. . A Place fo r Me. 1977. New York: New York: H o lt, Rinehart, and Winston, H o lt, Rinehart, and Winston, ________. Rhymes and Tales, Books and Games, Pets and People, Can You Imagine? New York: H o lt, Rinehart, and Winston, 1977. . Special Happenings. Winston, 1977. New York: H o lt, Rinehart, and . A Time fo r Friends. 1977. New York: H o lt, Rinehart, and Winston, . The Way of the World. Winston, 1977. New York: H olt, Rinehart, and Woodbury, Marda. Selecting Instructional M aterials. Fastback 110. Bloomington, Indiana: Phi Delta Kappa Educational Foundation, 1978. Wrightstone, J. Wayne; Hogan, Thomas P.; and Abbott, Muriel M. "Accountability in Education and Associated Measurement Problems." In Readings in Measurement and Evaluation in Education and Psy­ chology. Edited by William A. Mehrens. New York: H o lt, Rinehart, and Winston, 1976,