'n'i processes of that test th ‘T‘k items Vthh ‘ use the proc inta. The D lining which used by most Vidual test formance mea 5311mm t the process 7and 9). an from the pro 9. 12 and 13 ich student § ABSTRACT THE DEVELOPMENT OF ITEMS WHICH ASSESS THE PROCESSES 0F CONTROLLING VARIABLES AND INTERPRETING DATA By Richard Wayne Robison With the increasing emphases in instruction in the processes of science. there is a need to develop test items that test these processes in an efficient way. This study is an attempt to develop objective test items which will assess seventh grade students' ability to use the processes of controlling variables and interpreting data. The procedure followed in constructing and deter- mining which items could be used was not the same as that used by most standardized test writers. The success of indi- vidual test items was determined by using independent per- formance measures. The external criteria used in this study were two of the Individual Competency Measures for the process of controlling variables (Controlling Variables 7 and 9). and three of the Individual Competency Measures from the process of interpreting data (Interpreting Data 9. 12 and 13), all from Part 6 of Science--A Process Approach. Each student was tested individually on the selected Compe- tency Measures and was scored as to whether he or she met the behavioral 12117111131 i'. 37fo, a; meesses . “31851850 {efining one {xterest to eerie of 56 362001 in C‘s-1 hie: of ite the ite: res relation to 31$ shaved t 2! controlli :3? the DTOC fiscriminagi level of RCC An . 340?! OOeff . h e .‘1 test and the 2math)” to}? elatIOn . 7.31.1 933D BC ore 3 3.. COntroll 1" k t <;,?)~;ag)13 Richard Wayne Robison behavioral objectives specified in the Measures. After the individual testing. the author. in cooperation with Darrel H. Fyffe. assembled a 79 item group test. ”A Group of Four Processes“. The test covered the processes of controlling variables. interpreting data, formulating hypotheses and defining operationally. The latter two processes were of interest to Fyffe. The group test was administered to a sample of 56 students who had attended the Kinawa Middle School in Okemos. Michigan. during the sixth grade. Item analysis data were obtained which reported the index of item discrimination. index of item difficulty. and the item response pattern. The analysis of items was in relation to the external criterion measure. The item analy- sis showed that 13 of the original 18 items for the process of controlling variables. and 22 of the original 25 items for the process of interpreting data had an index of item discrimination of .20 or greater (this was the predetermined level of acceptance). Another analysis was performed to show the corre- lation coefficients between the processes on the individual test and the processes on the group items with high dis- crimination (.20 or greater). The Pearson Product moment correlation coefficient showed a significant correlation (significant at the .001 level) between the individual and group scores for each of the two processes. For the process of controlling variables, the correlation coefficient between the group items and the individual measure was .70, Pad; - 11 ‘55 .av. v1 4» S‘va ’3 d Richard Wayne Robison while the coefficient for the process of interpreting data was .66. The results of this study indicate that it is pos- sible to develop objective test items which are signifi- cantly correlated with students' ability to perform the processes of controlling variables and interpreting data. THE DEVELOPMENT OF ITEMS WHICH ASSESSTHE PROCESSES OP CONTROLLING VARIABLES AND INTERPRETING DATA By Richard Wayne Robison A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OP PHILOSOPH! Department of Secondary Education and Curriculum 001lege of Education Copyright by RICHARD MAINE ROBISON 1973 ACKNOWLEDGMENTS In a project of this magnitude. there are always many peOple who help. I cannot possibly thank all of them in this short space, but there are several who must be mentioned. First, there is one person without whom this project would never have been possible. I refer to my advisor, Dr. Richard J. McLeod. It was he who first suggested this topic to me and who has been a constant source of guidance through- out the long task. The other members of my committee, Dr. Julian R. Brandon, Dr. Dale Alam and Dr. Norman T. Bell, have also been of invaluable assistance. Their scholarly criticisms have added considerable to the quality of my work. A special note of thanks to Darrel w. Fyffe, my partner in research and two of his committee members, Dr. Glenn C. Berkheimer and Dr. Robert L. Ebel. I must also express my thanks to the administration and staff of the Kinawa Middle School who allowed me to do this study. Finally--my wife and our seven children who had to put up with my weird working hours for years and who truly pitched in to help with all the hard work--it truly must be love. 11 II, TABLE OF CONTENTS ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . LIST OF TABLES . . . . . . . . . . . . . . . . LIST OF FIGURES . . . . . . . . . . . . . . . . LIST OF APPENDICES . . . . . . . . . . . . . . CHAPTER I. THE PROBLEM . . . . . . . . . . . . . . Background . . . . . . . . . . . . . Purpose . . . . . . . . . . . . . . . Design . . . . . . . . . . . . . . . Assumptions . . . . . . . . . . . . . Limitations . . . . . . . . . . . . . Overview of the Dissertation . . . . II. REVIEW OF THE LITERATURE . . . . . . . What are Educational Processes? . . . Why Teach Processes? . . . . . . . . Identifying Processes . . . . . . . . Defining the Two Processes . . . . . Recent Efforts in Evaluating Science ”0008808............. Problems with Evaluating Processes . Summary............... 111 Page ii vii viii 10 10 11 12 1a 16 17 19 21 23 35 37 31.9733 III. [ET II. A}: CHAPTER Page III. METHODS AND PROCEDURES . . . . . . . . . . . . 39 Overview . . . . . . . . . . . . . . . . . . 39 validation Design . . . . . . . . . . . . . 39 Deve10pment of Test Items . . . . . . . . . 41 Testing of Items . . . . . . . . . . . . . . #5 Analysis Procedure . . . . . . . . . . . . . 51 Summary . . . . . . . . . . . . . . . . . . 5h IV. AHALISIS OF RESULTS . . . . . . . . . . . . . 56 Data from ”Individual Competency Measures"................ 56 Data from ”A Group Test of Four Processes“ . . . . . . . . . . . . . . . . 62 Non-Acceptable Items . . . . . . . . . . 65 Acceptable Items . . . . . . . . . . . . 73 'Visual vs Non-Visual Items . . . . . . . . . 79 Correlation Coefficients . . . . . . . . . . 82 Summary . . . . . . . . . . . . . . . . . . 87 V. SUMMARY. CONCLUSIONS AND FUTURE RESEARCH . . . 89 Summary . . . . . . . . . . . . . . . . . . 89 Conclusions................91 Implications for Future Research . . . . . . 91 BIBLIOGRAPHI . . . . . . . . . . . . . . . . . . . . . 98 APPENDICES . . . . . . . . . . . . . . . . . . . . . . 101 iv Table I. 3. 10 11, Table l. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. LIST OF TABLES Frequency Distribution from the “Individual Competency Measures” for the Process of Controlling Variables . . Frequency Distribution from the ”Individual Competency Measures“ for the Process of Interpreting Data . . . . Percentage of Students Who Performed Each Task Correctly--00ntrclling variabIOB e e e c e c c c c e c c e e 0 Percentage of Students Who Performed Each Task Correctly--Interpreting Data 0 c c c e e e c e e e e e e e e e 0 Data from for the Data from for the Frequency Test of “A Group Test of Four Processes“ Process of Controlling Variables "A Group Test of Four Processes“ Process of Interpreting Data . . Distribution from ”A Group Four Process” for the Process of Controlling Variables . . . . . . . . Frequency Test of Distribution from ”A Group Pour Processes" for the Process Of Interpreting Data e e c e e e c e c c Acceptable Items from "A Group Test of Four Processes" for the Process of Controll1ng Variables e e c c c e e e e Acceptable Items from 'A Group Test of Four Processes“ for the Process of Interpreting Data 0 e e e c e c e e e e Comparison of Indices of Item Discrimi- nation and Difficulty for the Items Presented Visually and Non-Visually . . V Page 57 58 60 61 62 63 7h 75 77 78 80 enable 12. 113. 1J3. 1:5. 16’. 't-test' for the Comparison of Indices of Item Discrimination and Difficulty for the Items Presented Visually and NOD’Vlsually c c e e c c c c e e e e c 0 Scores Received on the “Individual Competency Measures” and ”A Group Test of Four Processes“ for the Processes of Controlling Variables and Interpreting Data 0 s e c c e e e e Pearson-Product Moment Correlation Coefficient for the Processes of Controlling variables and Interpreting D‘tl e e e e e c c e c c c e c c e e c 0 Correlation Coefficients of "Individual Competency Measures" scores and Group TOBt Scores 0 e e e e e e e e e e e c 0 values or Pearson-Product Moment Corre- lation Coefficient Required for Specific LOVOIB Of Significance e e e c e e c e 0 vi Page 81 83 8h 86 87 Figure LIST OF FIGURES Figure Page 1. Frequency Distribution from the “Individual Competency Measures” for the Process of Controlling Variables................. 59 2. Frequency Distribution from the “Individual Competency Measures“ for the Process of Interpreting Data................... 59 3. Frequency Distribution from VA Group Test of Four Processes” for the Process of Controlling Variables . . . . . 76 “a Frequency Distribution from "A Group Test of Four Processes“ for the Process of Interpreting Data . . . . . . . 76 vii A. B. LIST OF APPENDICES Appendix Page .A. Script. Slides and Items Used in “A Group Test of Four Processes”. . . . . . 101 .3. Complete Item Analysis Data for the Accepted Items on the Process of c0ntr0111n8 Variables c e e c s s e e e e e 102 C. Complete Item Analysis Data for the Accepted Items on the Process of InterprOthg Data 0 e s e e s s c e s e e e 103 7111 “o'er n. V -. "m# 1 -..s U V V: ‘4 “A.‘ U Q Stile: ti?e 531 i; ‘ ‘ ‘eaa: ani 7 year: a). S U P. a..eS‘ CHAPTER I THE PROBLEM Background The new curriculum materials for elementary school science emphasize the processes of science to an extent never before realized. Yet, there is very little develop- ment work underway on the tools for their evaluation. There is an urgent need for a time efficient, reliable. and valid means to evaluate the process-skills obtained by students. The purpose of this study is to develop objec- tive test items over the processes of controlling variables and interpreting data that will give evidence of criterion- related validityl while still being time efficient, reliable, and valid. The goals of teaching science as listed in the 31st yearbook of the National Society for the Study of Education are similar to those listed in the Society's h6th yearbook. These goals were to develop an understanding of the scien- tific principles and an ability to apply them. Included in these was the development of children's ability to acquire . and use scientific attitudes and skills in problem solving. One would find it difficult to quarrel with these statements Criterion-related validity pertains to a relation- ship between the scores on test items and an independent criterion measure that defines what is being measured. 1 u) m L; ('9' 9 co' 69'4- ‘isr baa... 2 as part of educational objectives. P. Hurd and J. Gallagher, ~in their book, New Directions in Elementary Science Teaching, said that these ”objectives have meaning only as they are recognizable in the changed behavior of children."2 They then point out that "These objectives were not achieved by children. and the way elementary science was taught could only insure that these goals would not be attained.”3 There was a pressing need for our schools to develop students who were adept in those processes about which so much had been written. The concern about science teaching in the elementary school reached a point in 1960 where "something had to be done.”u By 1961 several projects were underway. These projects have included, as part of their goals, the seldom practiced aims that were written in the Blst and 46th NSSE Yearbooks. In general, these projects have included the intellectual skills that would be needed if the student were to discover. organize. and apply the information that he gains from his science experience. W.W. Welch5 points out that when new projects are ZHurd, Paul and James Gallagher, New Directions in Elementary Science Teaching. California: wadsworthfiPfisIIshing Os. nO.. I p9 ' 31b1a.. p. 28. “Iblde g ps 29. 5W.W. Welch, ”Curricular Decisionse How can Evalua- tion Assist Science Teachers?”. The Science Teacher. Vol. 35. Nov. 1968. pp. 22-25. r‘.“.‘ “leave 8" the: 0A- 1903‘} ‘iva- resau: “1' . I .I ' 455 8 I $1119 -L." e a, “ .Om. x v 16791 0 ‘96 . I f o s c 3 ‘h' a. h .9». 3 being developed, the authors need to answer two questions. These questions are: (1) How can these programs be taught effectively? and (2) What are the methods of evaluation? Welch also points out that most of the newer federally supported programs have provided financial assistance in answering the first question. These programs have prepared materials such as teachers' guides. laboratory manuals. resource books and teacher training films. The government has also financially aided the development of special summer institutes in an effort to provide assistance in training teachers to teach the new programs. The answer to the second question. the problem of develOping methods of evaluation. has been another matter for the newer curriculum projects. It has. by no means. been given the attention that it deserves. Welch points out two problems in developing evaluative materials for the newer science curricula. "First. curriculum developers are often hesitant to state their objectives explicity in behav- ioral terms. Second, they are even more reluctant to have these objectives evaluated."6 In the development of behavioral objectives and measures to determine the extent that students meet these ObJectives. one program stands out. This program. Science-- A Process Approach, developed by the Commission on Science Eflucation of the American Association for the Advancement of ‘1 61bid. 4 Science (AAAS). has provided the materials to help teachers teach the processes to be determined. It has provided the teacher with explicitly stated objectives that state. in behavioral terms. what they expect each student to be able to do at the end of each lesson. Science--A Process Approach has also provided a means for the teacher to evaluate whether the student has achieved the objectives. To evaluate a student. the teacher constructs a situation in which the student is given materials euth.which to work. The teacher then asks questions which involve the use of the process being tested and observes the individual as he manipulates the materials in an attempt to answer the questions. Since the teacher must infer from the student's actions. his ability to exhibit the acceptable behaviors. it must be administered by someone who is familiar with the specific process involved. There is a need for an ObJective test that could be administered by anyone. even one‘unfamiliar with the behaviors that define the process tested. Another problem associated with evaluating the pro- cess competency of a student is that it requires the evalua- tor to work with one student at a time. A group test would be more efficient. The amount of equipment that the evaluator must have at hand for evaluation creates still another problem. There 13 a need to develop a test over the processes of science that W111 minimize the amount ofemuipment needed for administration. 5 Although the above points (objectivity, efficiency, requirement for small amount of equipment) are requirements for a good process test, none are sufficient if item valid- ity is not demonstrated. That is to say. an item testing the process of controlling variables is valid when a person who can actually control variables gives a correct response and when one who can't gives an incorrect response. In test construction. validity is frequently appraised in terms of content validity. validity Content validity is related to how well the content of the test samples the subject matter about which infer- ences are to be made. The method of determining this rela- tionship is to obtain the opinions of persons knowledgable in the content area sampled. It has generally been left up to these persons to examine each test item and decide whether the items will elicit the apprOpriate response from the learner. The relationship for each item is then deter- mined by looking at the opinions from each person. If the test items are to measure specific facts, or the recall of specific information, content validity is probably sufficient. If. however. the items are to measure something other than factual retention. such as whether the learner is able to control variables or interpret data, content validity is. in the author's opinion. not sufficient. There is a need for test items that correlate with a student's r- .‘ISIJ ass...‘ of ‘3‘ five ., ‘. I 4‘ va---4 v... 0383:? 6 ability to actually perform the processes in question. The validity that is most desirable for this type of test is criterion-related validity. Criterion-related validity pertains to a relationship between the scores on test items and the behavior exhibited on an independent external measure. A test over processes of science should use the student behavior from a real situation as their external measure. The actual student behavior can be determined by defining the processes being evaluated in behavioral terms and then constructing situations in which the evaluator can observe the student's behavior. The individual student behavior could then be used as the criterion for validating test items on that process. Recent Attempts in Evaluating Processes There have been two major attempts to develop a test over scientific processes since AAAS published their science program Science--A Process Approach. The Test of Science Processes. (Tannenbaum. R.S.. 1968) is an instrument over eight processes which are to be used with junior high school students. The validity repor- ted in this study is content validity obtained by submitting the test items to a jury of science education experts. Even though this report shows considerable evidence of con- tent validity. there is no evidence of criterion-related validity. While Tannenbaum realized the importance of 7 criterion-related validity, he made only a limited attempt to give some evidence of it. His investigation. undertaken to demonstrate criterion-related vality. used the teacher's subjective rating of the student's ability to use the pro- cesses of science as the external criterion measure. The other test. Basic Science Process Test, (Beard, Jean. 1970) was designed to measure the processes of mea- suring and classifying with first. second. and third grade students. Even though the test meets the requirements of a group test, it. too. uses content validity instead of criterion-related validity. The science program Science-~A Process Approach provides two means of evaluating students' attainment of the stated objectives. Each lesson of Science--A Process Ap- proach contains an "Appraisal” (group assessment) and a ”Competency Measure” (individual assessment). The ”Appraisal“ is designed to evaluate. as a group, the attainment of the specific skills for which the lesson Was designed. It requires the student to work either individually or in small groups on a task set up by the 1‘95“3’1'161‘. Even though there is evidence of criterion-related Validity (observing the student either solving or helping *0 8Olvo the task which involves the process) the instrument is deSigned for students familiar with the vocabulary or inv01Ved in Science-«A Process Approach. This makes the 11"Strument usable for only a limited population of students. The ”Appraisal” must also be administered by someone who is 8 familiar with the behaviors that define the processes being tested. The ”Individual Competency Measure" is designed to evaluate science processes in a one-to-one situation. The student is presented with a series of tasks set up by the evaluator which would require the student to use the process in question. The validity of the competency measure is deter- mined by the correspondence between the behavioral objectives and the tasks required by the competency measure. The com- petency measures were developed to give observational evi- dence of a student's attainment of the science process in question. The "Individual Competency Measure" indicates the student can perform the process when they can successfully complete all the tasks of the competency measure. For examples A student can control variables if he successfully handles the set of tasks identified in the ”Individual Com- petency Measure" for the process of controlling variables. One of the problems with the "Individual Competency Measures" is that they were designed to be used with students who are familiar with the vocabulary of Science--A Process Approach. The competency measure also requires the adminis- trator to be familiar with the required behaviors which define the processes. Another problem is that they were not designed for group administration. 9 ‘ Purpose The purpose of this study was to develOp a pool of objective test items for use with seventh grade students to measure the ability to perform two of the integrated pro- cesses. The two processes selected were Controlling Vari— ables and Interpreting Data. These processes have been defined with behavioral objectives and measured by the use of "Competency Measures“ in Science--A Process Approach. The criterion measures for this study were selected from those behavioral objectives and ”Individual Competency Measures”. By comparing the analysis data for each objective test item to the data from the respective ”Individual Competency Mea- sures". the items which adequately test the processes were determined. This study was so designed that the items which were accepted did, in fact. require the students to be able to perform the process. The criterion measure was given and scored. the students in the upper and lower 27% were determined. and from these groups the index of item discrim- ination7 was calculated. Only those items which produced an index of item discrimination of .20 or greater were accepted. This level. .20. was chosen because scores on an external criteria was used in computing discrimination. This is explained in greater detail in Chapter III. In particular. the objectives of this study are to develop objective test items over the processes of controlling 7The ability of a test item to differentiate between persons possessing much of some trait and those possessing little. 10 variables and interpreting data that: 1. will test the same behaviors as the selected ”Individual Competency Measures" from Science-- A Process Approach. 2. will be capable of administration by anyone interested in evaluation of students' acqui- sition of the specified processes. 3. can be used in the development of an instrument for use in a group testing situation. u. can be administered with a minimum of manipula- tive material. 5. uses a vocabulary suitable for use with seventh grade students even if unfamiliar with the vocabulary in Science--A Process Approach. Design To fulfill the purposes of this study a combination of two designs was chosen. The two designs were: One-Shot Case Study and the One-Group Protest-Posttest. The combination of these two designs implies the technique of making two sep- arate observations after the group of students have been ex- posed to some eXperimental treatment. In this study. the prior exposure to the science program Science--A Process Approach will act as the experimental treatment while the two observa- tions will be the administration of the selected ”Individual Competency Measures” and the group test ”A Group Test of Four Processes.” were made: In 1. 3. 11 Assumptions conducting this study. the following assumptions Paper and pencil items can be constructed to adequately measure the selected processes of science. The ”Individual Competency Measures” of Science --A Process Approach are valid measures ofya student's ability to control variables and interpret data. The students who have studied Science--A Process Approach in the school year prior to the one in which this study was conducted will have suffi- cient experience with the vocabulary and manipu- lative skills used in the “Individual Competency Measures” to be suitable subjects. The choice of the index of item discrimination as the basis of evaluating the effectiveness of the test items is apprOpriate for this study. No appreciable amount of learning will take place as a result of the administration of the “Individual Competency Measures“ prior to the student response to the written test items. 12 Limitations In conducting this study, the following limitations are made: 1. No attempt has been made to construct a test. Only a pool of test items that have a high index of item discrimination and that exhibit criterion- related validity have been constructed. 2. This study was limited to one group of 75 seventh grade students in one school in Okemos. Michigan. The students had received instruction in Science--A Process Approach the year before and was limited to the material covered during that year. 3. The “Individual Competency Measures” used and the test items constructed were limited to sub— tasks in which the sample could reasonably be expected to have competency. 4. This study was also limited to an upper middle class, suburban. professional bedroom community. Overview of the Dissertation In this chapter the background. problem. need, pur- pose. assumptions and limitations of this study were presented. Chapter II contains literature related to the measurement of educational processes and some of the current attempts to evaluate the processes of science. 13 Chapter III contains a detailed description of the procedures used in carrying out this study, including the selection of the sample, administration of the "Individual Competency Measures”, development of the test items and the procedures for the analysis of data. Chapter IV includes the results and interpretations of the analysis of data. Chapter V presents the summary. conclusions and implications for further research derived from this study. CHAPTER II REVIEW OF THE LITERATURE Recent science curricula deve10pments illustrate a shift in the recommended emphasis of science instruction from a body of scientific facts to helping the children to discover. acquire and apply the information gained. Much has been said concerning the merits of emphasizing the means by which these facts and concepts are obtained. Sund and Trowbridge point out that science of the future will need to be “dynamic and changeable in order to meet the demands of a rapidly accelerating scientific age."8 The new science curricula are giving our students the ”opportunities for increased laboratory work and application of inquiry methods for learning. They are directed to better understandings of how scientists work and how knowledge is obtained. More attention is given to the processes of science."9 Sund and Trowbmidge also point out that the new curriculum projects ‘ 33. sand and L. Trowbridge, Teaching Science by Inguigy in the Secondary School, Oh 0: ar es . errill OO 8. nee. e 91bid. lu 15 will help students ”gain skills and knowledge needed to handle the problems of this age successfully."10 Hurd and Gallagher stress that teaching the processes of science and its modes of inquiry “is providing young people with tools that are not only applicable in the content of science but to life in general. What is more, if children learn to use these processes of science, they will come closer to understanding the spirit of inquiry that characterizes science."11 Thus, the development of student abilities necessary to carry on the processes of scientific inquiries seems to be a major goal of science instruction. Cole12 states that process education recognizes as its foremost objective. the processes which the learner needs if he is to acquire, organize and utilize in a produc- tive manner the information that he gains. In this chapter the author will present some of the literature concerning the measurement of educational pro- cesses. and particularly the processes of_controlling vari- ables and interpreting data. In addition. some of the cur- rent attempts to evaluate the processes of science will be presented. 1°Ibic1. 11p. Hurd and J. Gallagher. op. cit. 12Henery P. Cole, ”Process Curricula and Creativity iDevelopment “ Journal of Creative Behavior Vol. 3 No. a Fall 1969, pp;‘§fi3;u4. ' ' ' 16 What Are Educational Processes? The educational processes that the learner needs carry various titles. Bruner calls them intellectual habits.13 Piaget refers to them as the logical operations.1u Skinner calls them self management behaviors.15 while Gagne has referred to them as learned capabilities and intellec- tual skills.16 Burns and Brooks17 say that processes are specific mental skills which are any of a set of actions. changes. treatments. or transformations of cognitive or affective entities used in a strategy in a special order to achieve the solution of a problem associated with the learn- ing act. the use of learned products. or the communication of things learned. Gagne states that educational processes are not entities of verbalization. They are not content processes such as the processes of economics. Instead. they are the strategies. the behavioral capabilities. the psychological 13Jerome S. Bruner. The Process of Education. New York: Random House. Inc.. 1960. ll"Jean Piaget. Six Ps chological Studies. New York: Random House. Inc.. 1§57. 153.F. Skinner. The Technolo of Teachin . New York: .A. Century-Crofts. 19 . 16Robert M. Gagne. "Contributions to Human Develop- :ment.“ Psychological Review. 75. (3). 1968. 17aicherd w. Burns and Gary D. Brooks. ”What are Educational Processes?" The Science Teacher. Vol. 37. No. 2, FCbe 1970. pp. 27-28e #— 17 processes used by the individual to deal with the content.18 Gagne's recent writings offer one of the best definitions of processes. "Processes are not something mysterious which the learner acquires. but behavioral tendencies or habits in particular types of situations."19 Processes are merely names applied to collections of behavioral cap- abilities which are observed or inferred in the student. Why Teach Processes? Several justifications for teaching processes have been stated by Bloom.20 First. the world is changing so fast that it is impossible to predict what knowledge and information will be needed. Therefore. it is important to help the student acquire generalized intellectual abilities and skills which will serve him in many new situations. Second. the amount of knowledge is so great that it is impossible to instruct the students in anything but a small portion of what is known. Thus. one of the goals of educa- tion must be to help the students to acquire some of the more relevant and central information combined with those processes which will help him adapt and expand this limited 18Robert M. Gagne. ”Learning Hierarchies.“ Presidential Address. Division 15. American Psychological Association. San Francisco. August. 1968. 19Gagne. "Contributions to Human Development."02. cit. 20Benjamin S. Bloom. Taxoncgf of Educational Obiec- tives Handbook 1: Co itive oma n. ew or : v HOE! comPWo Inc 0 e 1526 I PP 0 32-133 0 18 knowledge that he acquires. Third. the acquisition of essen- tial processes will contribute to an individual's ability to successfully solve problems. Fourth. processes are more transferable than specific knowledge. Bloom says that pro- cesses are more widely applicable than knowledge and if we are concerned with the problems of transfer. we would then select processes over specific knowledge.21 Fifth. intel- lectual skills are retained longer than other types of learning. Bloom also states that ”it would seem reasonable to expect greater permanence of learning for those outcomes of education which can be generalized and applied in a num- ber of different situations throughout the individual's formal educational experience than those outcomes which are so specific that they are likely to be encountered only once or at the most a few times throughout the educational pro- gram."22 There has been some evidence to show that specific information which is learned is subject to rapid extinction. but the intellectual skills are more permanent.23'2u'25 211bid.. pp. 38-42. 22Ibid.. p. #2. 23H. Freud and N.D. Cheronis. ”Retention in the Physi- cal Science Survey Course.” Journal of Chemical Education. 17. l9h0. pp. 289-93. 2hr. Frutchey. “Retention in High School Chemistry.” Journal of Hi her Learnin . 8. 1937. pp. 217-218. 25Ralph W. Tyler. "Permanence of Learning.” Construc- ifiéglfipfichievement Tests. Columbus: Ohio.State University. 19 Gagne uses all of these arguments as justification for the deve10pment of processes and adds at least one more for «consideration. He points out that information can and is usually recorded somewhere. This means that. if needed. information can be readily accessible when needed. 0n the other hand. process skills are not so easily acquired since they must be developed. It is this justification that has helped influence the shift in emphasis of science instructional materials in today' 3 schools . Identifying Processes The assumption that there are basic processes of science is advanced by Gagne in The Psychological Bases of Science--A Process Approach. 1. The scientists' behavior in pursuing science constitute a highly complex set of intellectual activities which are. however. analyzable into simpler activities. 2. These intellectual activities (processes) are. as most scientists would agree. highly general- izable across scientific disciplines. 3. These intellectual activities of scientists may be learned. and it is reasonable to begin with the simplest ones and build the more complex activities out of them. since this seems to be in fact the way they are organized. 20 4. Accordingly. one can construct a reasonable sequence of instruction which aims to have children acquire process skills. beginning with the simplest kinds of observations. and building progressively through classifying. measuring. communicating. quantifying. organizing through space and time. to the making of inferences and predictions. As further building occurs. one finds it possible for the students to learn how to make operational definitions. how to formu- late testable hypotheses. how to carry out ex- periments. and how to interpret data from exper- iments.26 The Commission on Science Education of the American Association for the Advancement of Science has identified e18kt basic processes and five integrated processes for their elenmentary school science program. Science--A Process .EEA roach. The eight basic processes are: observation. clas- 811"leation. recognition and use of space/time relations. r °°Ognition and use of number relations. measurement. com- lmilitant-don. inference. and predictions. The five integrated p1"Decisses are: formulating hypothesis. defining Operationally. \ 26American Association for the Advancement of Science. ‘3Onnlission on Science Education. The Ps cholo ical Bases of Scienc¢--A Process A roach. WashIngton 5.5.: m5. I§53: (m3 Miscellaneous £3351! 7 cation 65-8 0) - 21 controlling variables. interpreting data. and experimenting.” For each of the processes. the Commission has developed many exercises. Each exercise contains a set of behavioral objectives that describe what the child will be able to do at the end of the exercise. a rationale. a vocab- ulary list. a list of needed materials. suggestions of how to originate the problem. a suggested instructional proce- dure. suggestions of how to help generalize experiences. and two evaluative methods called "Appraisals” and ”Competency Measures". Two of the integrated processes. controlling vari- ables and interpreting data. as defined by the Commission on Science Education of the American Association for the Advancement of Science. have been chosen for this study. Defining the Two Processes Controlling Variables variables are the conditions. properties. or factors "‘31: may influence the operation of something in a system. The variables are identified by observing the components and proIberties that are included in the system. To determine "nether a variable influences the Operation of part of a system. all other variables must be held constant while the °ne variable being tested is manipulated. Therefore. if a a‘3lltient can control variables. he will be able to: \. 27American Association for the Advancement of Science. Wash. New York: XEROX Education V 8 on. e 1. 22 IDENTIFY variables which may influence the behavior or the prOperties of a physical or biological system. IDENTIFY variables which are held constant or manipulated in an investigation or an experi- ment. DISTINGUISH between conditions which hold a given variable constant and Conditions which do not hold a variable constant. CONSTRUCT a test to determine the effects of one or more variables on a responding variable. IDENTIFY and NAME variables which were not held constant in the description of an investigation. although they varied in the same way in all treatments or were randomized.28 _Int erpret ing Data The process of interpreting data requires the inte- gration of the skills learned in the basic science processes Plus ingenuity on the part of the interpreter. The ability to use this process is desirable for the child as afuture °1tlzen. since much of his daily activity will be concerned ‘Irtrx 1nterpreting data. If a student can interpret data. he \vill be able to: l. DESCRIBE in a few sentences the information 28American Association for the Advancement of Science. 3‘31ence-n-A Process A roach. Commentary for Teachers. Commis- a5io"n"""on"3"ic enc'e Ed'u' 'c;a%1'on.' American Association for the Alti‘vancement of Science. 3rd Experimental Edition. 1968. 5. 23 shown in a table of data or graph. CONSTRUCT one or more inferences or hypotheses from the information given in a table of data or graph. CONSTRUCT one or more inferences or hypotheses from a comparison of the information in two or more related tables of data or graphs. DESCRIBE certain kinds of data. using the mean. median. range. and frequency distribution: and CONSTRUCT predictions. inferences or hypotheses from this information. CONSTRUCT inferences or hypotheses from pic— torial data. DISTINGUISH between linear and nonlinear rela- tions. APPLY A RULE to find the slope of graphs and linear relations. and DESCRIBE the informa- tion provided by the slape.29 Recent Efforts in Evaluating Science Processes The problems of developing methods of evaluating the newer curriculum projects have not been given the attention théir deserve. Evaluation should be an integral part of cur- r19'ulum development. Primarily it is important that the 1"Bizruction be evaluated by means of instruments which mea- 3“UDe the competencies that are included within the goals 8““1 philosophy of the program. However. evaluation is more \ 291bid. 24 thaxi testing and examining. It is studying and assessing the total growth of the students in relation to their poten- tial and the objectives of the program. In a classroom that is content-oriented. the major goal. is the student's acquisition of scientific facts and principles. Evaluation of this goal is possible through tests such as true-false. multiple choice. short answer and essay questions that test facts but do not necessarily require the processes used in determining those facts. In a curriculum that includes science processes as Dari: of their objectives. the evaluation should reflect not only students' acquisition of facts and principles. but also their ability to use the processes of science. Recently there have been several attempts to assess the student's ability to use the processes of science. They are: l. the ”Competency Measures” of the American Association for the Advancement of Science's program Science-~A Process Approach.30 2. the “Appraisals” of the American Association for the Advancement of Science's program Science--A Process Approach.31 3. Processes of Science Test by the Biological 3°AAA3. 1967. op. cit. 311bid. 4!“- '- 25 Science Curriculum Study group.32 A. W by Jean Beard.33 5- W by Robert Tannen- baum.3u 6. Stanford Achievement Test. 1964 edition.35 7. Metropolitan Achievement Test. Advanced Science Test.36 8. Sequential Test of Educational Progpess: Science (STEP Science).37 'There has also been an attempt to assess the science processes of teachers. Two of the major works in this area are: 1. Science Process Test for Elementapy School Teachers by Evan Sweetzer.38 2. Science Process Measure for Teachers by the 32Biological Science Curriculum Study. Processes of Science Test. New York: The Psychological Corporation. I962. 33Jean Beard. Grou Achievement Tests Develo ed for Two Basic Processes of KENS Sc!ence--I Process I roacfi. EH.D. Thesis. CorvaIIIs. Oregon State UnIversIty. I975. 3“Robert Tannenbaum. The Develo ment of the Test of Science Processes. Ed.D Thesis. Columgla University. I953. 35Oscar K. Buros. (ed.). The Sixth Mental Measurements Yearbook. New Jersey: The Gryp on see. . 361bid. 37Ibid. 38Evan Sweetzer. Science Process Test for Elementary School Teachers. Michigan a e n vers y. 26 American Association for the Advancement of Science.39 @mpetency Measure The Commission on Science Education of the American Asscaciation for the Advancement of Science has developed a process oriented elementary school science program through more than nine years of extensive research. experimentation. testing. try out. revision. and evaluation on a nationwide scale. This program. Science--A Process Approach. under- takes development of the scientific processes. For each of these processes. the commission has devised a list of behav- ioral objectives that describe exactly what the learner will be able to do. One of the evaluative devices of this pro- gram, the ”Competency Measure". will be reviewed now while the crther device. the ”Appraisal”. will be reviewed next. The competency measure consists of tasks intended to assess the achievement of the objec- tives for each exercise. The effectiveness of the measure is directly related to the accuracy with which the behavioral objectives of each exercise are translated into performance tasks that sample the behaviors described as objec- tives for that exercise. The validity of the competency measures rest upon the demonstration of the existence of the one-to-one correspondence between the behavioral objectives of the specific process exercise and the tasks requirgd by the competency measure for that exercise. 0 39American Association for the Advancement of Science. 196?. op. cit. A11 “OAmerican Association for the Advancement of Science. Evaluation Model and Its A lication: Second Re ort. Eollimission on Science 'F'Zucation American Association for he Advancement of Science. 1966 (AAAS Misc. Publication 68-h). 27 This instrument is administered individually and is con- structed around the stated behavioral objectives for the exercise. In this type of test. the teacher. or some other competent person. using the behavioral objectives as a standard. actually observes the student's behavior to de- termine whether the student can perform the process being tested. The competency measure from Exercise 2 under the heading “Classification“ will be used as an example. This exercise deals with constructing and demonstrating the use of different classification systems. It includes the clas- sification of some objects on the basis of whether or not the objects can be used in a given way and then constructing a second classification of the same objects. called reclas- 811’!ing. on the basis of another property. The behavioral Obfleotives for this exercise are clearly stated in terms which prescribe the sorts of behaviors that the students Inst exhibit in order to demonstrate competency in this Phase of classifying. The "Individual Competency Measure“ much is used to measure the student's achievement of these °bJectives involve a group of objects (a ball. a block. a alice of bread. a grape. an orange. a ladle. and an empty t31-h can). These objects are placed in front of the child and his first task is to organize them into two groups QBpending on whether or not they can be eaten. After this the child is told to reclassify the objects on the basis 28 of whether or not he could use the objects to carry water. Finally. the teacher makes an inclined plane and asks the child to reclassify the objects once more on the basis of whether or not they can roll down the inclined plane. This example is taken from Part A of Science-~A Process Approach and is usually presented in kindergarten.“ Even though the "Individual Competency Measure” is 8 Valid test of the objectives of the exercise. there are some disadvantages to this method of evaluation. First. the Oom'petency measure must be administered by someone who is familiar enough with the processes to be able to interpret Whether the student has performed the behaviors specified by the behavioral objectives. Second. it is almost impos- sible for a teacher to spend that much time in evaluating her class because this method of evaluation demands a low Student-teacher ratio. Third. a considerable amount of equipment may be required because the student is frequently required to set up experiments and then carry them out. The evaluator must have on hand any equipment for which the Student may ask . égpraisal The “Appraisal” is designed to evaluate. as a total class. the attainment of the specific skills for which the lesson was designed. They are set up so students work either individually. in small groups or as an entire class on a set ¥ l”From Science--A Process A roach. Part A. New York: xEROX Education Division. I957. 29 of teasks related to objectives of the exercise. Exercise 12 under the heading ”Observation” will be used as an exam- ple. This exercise. Observing Mold Gardens. is designed so each child should be able to do the following: 1. state the color. shape. and size of a few kinds of molds. 2. construct a chart of the class record of the color. shape. and size of one mold colony at intervals during its cycle. 3. state the color. shape. and size of the mold colony at intefigals during its cycle by reading a chart. The "Appraisal" for this exercise says. Show the class a vegetable or fruit which has a mold colony on it. Ask children to de- scribe the growth. After several children have responded and the teacher has a sampling of responses. each child might be asked to draw the object and inclade a careful representation of the mold colony. 3 Even though the appraisals are designed around the 8Decific objectives of the exercise. there are some disad- Vantages to this method of evaluation. First. the "Appraisals” lmflst be administered by someone who is familiar enough with the processes to be able to interpret whether the student this performed the behaviors specified by the behavioral ob- Jectives. Second. with individual members of the class responding to the question. it is almost impossible to deter- mlne the extent to which each child has mastered the objec- tives of the lesson. “ZFrom Science--A Process Approach, Part B. New York: XEROX Education fiivision. i937. “3lbld. 30 Process of Science Test The Biological Sciences Curriculum Study group in developing this test was concerned with the students' knowledge of. and ability to use. the methodology of science. A list of their abilities include judging facts. interpre- 1fixu; data and evaluating experimental designs. The test consists of forty multiple choice items. All of the ques- tions were chosen from the biological sciences. but the autllors indicate that this is incidental to the main pur- Pose of the test. They feel that any knowledge of biology is unnecessary for a high score on the test. Therefore. they state that the test may be useful in courses in biol- ogy and also in other science courses that stress the Processes of science. The items are multiple choice and mitable for group administration. The test seems to be aimed at the tenth grade student. The real disadvantage of this test is the absence or criterion-related validity. It has not been validated aSailrlst the actual behaviors that would indicate the stu- dent's ability to use the processes that are being tested. Basic Science Process Test Jean Beard developed a test over two of the basic Processes as defined by Science--A Process Approach. This test is aimed at the early elementary school child. It is E“lulinistered in a series of 35 mm colored slides which 1llustrates laboratory situations involving basic science 31 processes. The slides are in sequence with a synchronized tape recording which provides oral instruction for the children. states the problem to be considered. and changes the slides. Each child has his own answer sheet which he marks as directed by the recording. This procedure was used as an attempt to assess science process skills of primary pupils who had not yet perfected their reading and writing skills. Even though the test is appropriate for group admin- istration and can be administered by anyone. there is still the question of validity. The validity of this test was determined by a jury who evaluated each item and made a decision concerning the probability of the abstract item eliciting the apprOpriate information from the children --a similar procedure to that used by authors of standard- 128d tests. The items were not evaluated against the ac- tual behaviors of the children. 3&1: of Science Processes Tannenbaum has developed a paper and pencil group 13631: on some of the processes of science. The processes be selected were: observing. comparing. classifying. quantifying. measuring. experimenting. inferring. and pre- dicting. For each process. a list of behaviors that the 3tudent must exhibit in order to demonstrate competency were specified in detail. These statements were submitted to experts for their opinions and validation before final 32 writing. The instrument was then administered to 3.673 stu- dents who were selected to include all ability levels and a wide range of socioeconomic backgrounds. The results of this administration were used to create norms for total score and for each of the eight subscores. The norms are reported by grade. sex. and by urban versus suburban. and for the total sample within each grade. This test reports a high total reliability (0.91) and reliability-coefficients for each of the eight subgroups ranging from 0.30 to about 0.80. It is important to realize that these reliabilities on the subparts of the test (using Kuder-Richardson 20 for- mula) are very low and must be used with caution. The study reports considerable evidence of content and curricular validity of the test. While there is evi- dence of validity. submitting test items to experts for Validation does not guarantee that the items are measuring the student's ability to use the process. Tannenbaum says that the criterion-related validity of this test is very difficult to assess because ”this is the first attempt to me“Burs this age level student's ability to use science ”Deanna“ The criterion-related validity could be attempted if the standard were the student's ability to P°rform the science processes in an actual situation where the student is faced with a science phenomenon. If. in this actual situation. the student's behaviors define the ¥ M‘Tannenbaum. Op. cit. 33 processes being tested, validity could easily be established. Tannenbaum reports a small investigation undertaken to demon- strate the criterion-related validity where the criterion was the teacher's rating of the student‘s ability to use the processes of science. “The results of this investiga- tion were not unequivocal. However. they do give some indi- cation of a degree of criterion-related validity. More con- clusive evidence of this must await much more extensive in- vestigationflu'5 A major question that needs to be answered in order to determine how much confidence can be placed in the "indication of criterion-related validity” stems around the background of the teachers in determining the processes of science. In this case. the validity still relies on someone's interpretation of the processes of science. Stanford Achievement Test H. Byran points out that the authors of the Stan- ford Achievement Test have sought to insure content validity by ”Examining appropriate courses of study and textbooks as a basis for determining the skills. knowledge. understandings. 9“- to be measured.”’+6 The author could find no indication that the test items were based on a performance criterion. Estacopolitan Achievement Test w. Cooley states that the authors of this test ”have “51bid. ”614.11. Byran. in Euros. cp. cit. 31» been far too concerned with the standardization process. while item content and validity appear to have been almost completely ignored.”u7 He goes on to say that the tests are “cpen to very serious criticism regarding the nature of the items and the knowledge and abilities assessed."l‘8 Se uential Tests of Educational Progress: Eogence Elf-E Science} The STEP Science test is designed to measure the student's knowledge of basic scientific concepts and his Pr0blem solving skills. The test items were written by a committee who worked together for several weeks so they could "criticize each other's work. maintain continuity from level to level and 'rub their ideas together'.'“9 Even though no coefficient of validity is reported. the authors indicate they were concerned about content validity. All three of these standardized tests have relied uDori a group of interested people to determine content valid- “=3. It has been up to these people to determine whether 1'-'1‘1e test items measure the skills and concepts used in existing science curricula. Even though a group of interested people have stated that these test items will elicit the appropriate responses. there is no guarantee. To determine whether these test items ¥ “7w.w. Cooley. in Euros. Ibido. p. 877. “81bid. “91bid. 35 do in fact measure the skills and processes they were designed to measure. they must be compared to the actual ability of the student to perform the skills and processes. The Other Instruments There are three other works in the field of science processes. Science Process Test for Elementary School Teachers and Science Process Measure for Teachers are both designed to assess teacher's knowledge of and ability to use Processes. The other instrument. Wisconsin Inventory of 3°1ence Processes. seems to be designed to measure knowledge or. and not necessarily ability to use. the processes of so lance s Problems with Evaluating Processes when one considers the evaluation of processes. one must be cautious of expecting a ready made test which is appropriate for group administration. There are several Problems associated with assessing the processes of science. Ebel gives two reasons why items written to measure the higher mental processes have failed to assess the pro- c’esses. “One is that they may be quite difficult and thus 0311 for more than ordinary examinees are capable of deliv- e1‘i-l’ig....Another is that they may involve fairly complex Biltuaticns. which require many words to describe and may Present the examinee with problems of comprehension and interpretation which may be irrelevant to the main purpose 36 of the examination."50 The study presented here has attempted to deal with these problems by providing both visual and oral information about complex situations and by using vocabulary which students should be expected to know. Another problem associated with the evaluation of science processes is that of time. To determine whether a student can perform a particular process. he must be able to demonstrate the behaviors which define that process. To successfully evaluate science processes. the evaluator needs to work with each child one at a time. There is just not enough time for a teacher. or evaluator. to assess the com- Potency of all children in all the different processes of science this way. If a process test is to be developed. it must be more efficient. The most efficient way would be to develop an instrument which could be administered to a 81‘oup of students at one time. Another problem with assessing the processes of science is the amount of equipment that is needed. Usually. evaluating the student's competency requires situations much permits one to observe the student working with equip- ment. As the student investigates the situation he must “'8 the freedom to work with all and any equipment he feels that he needs to successfully complete the tasks. There must be some way of eliminating the amount of equip- ment which is needed. There needs to be a test which can ¥ ”Robert L. Ebel. Measuri Educational Achievement. “9" Jersey: Prentice Half'fi“. o..2i§53. p. 52. 37 'be administered by anyone. does not take too much time. and requires little or no material. One solution to the above problems is to produce a pencil and paper test that is objective and uses little or no equipment. As mentioned earlier. several process tests are available (Beard. Tannenbaum. Buros. etc.). Associated with the production of such a test. however. is the difficulties experienced in trying to produce test items that are. indeed. valid. To overcome this. a decision was made to use an external criterion as a means of validating the test items. Horrocks and Schoonover in their book geagurement for Teachers51 describe the requirements of a criterion. These requirements are: 1) true outcomes of the construct in question. 2) observable. 3) measurable in some quantita- tive fashion. #) readily definable. and 5) agreed upon by the individuals concerned with establishing the behavior as a criterion. The ”Individual Competency Measures” from Science--A Process Approach appear to meet these conditions as an adequate criterion and will thus be used as the exter- nal criterion measure. Summary The current trends in science education have placed an emphasis on the development of the processes of science. 51John E. Horrocks and Thelma I. Schoonover. Measurement for Teachers. Ohio: Charles E. Merrill Publishing Company. :P- 0 38 There is at least one program that is committed to the instruction of the processes of science. This program. Science--A Process Approach. has also developed a method to evaluate the student's acquisition of these processes. This method. called the ”Competency Measure”. requires administration in a one-to-one situation by someone who is familiar with the processes being tested. Since the evaluation requires a one-to-one situation. it is not really feasible for the teacher to evaluate her entire class. There have been other attempts to evaluate the student's abilities to use the processes of science. All are suitable for group administration by anyone. and four are geared for junior high school students. The real problem is their failure to show a relationship between a score and some recognized standard of performance. EVen though the items have been submitted to experts in elementary and junior high school science education. it is important to remember that this does not guarantee the relationship of a student's test score to his ability to use the actual pro- cesses. This study is an attempt to develop a method of evaluation possessing a direct relationship between students' scores on a group test and their scores on the standard. the ”Individual Competency Measures" from Science--A Process Approach. CHAPTER III METHODS AND PROCEDURES Overview The purpose of this study was to create objectively scored. large-group administered. criterion validated test items that could form the basis of a test to measure stu- dent's performance on the processes of controlling variables and interpreting data. The work in this study is divided into two parts. The first. the develOpment of the test items. including writing. critiquing. and rewriting items. It also includes the development of a series of colored slides to support some of the test items. The second part of this study is the actual testing of the items. This section includes the selection of the population and sample. instrumentation. and the analysis procedure. validation Design for the Items in a Group Test of Four Processes The design for the validation portion of the study involved selecting a sample that had previously experienced Parts h and S‘ggience--A Process Approach. no This experience was considered necessary because the criterion measures which were administered to the sample contained vocabulary which could be unique to this science program. TWO observations were obtained on the sample: 1) the criterion measure. selected "Individual Competency Measures“ from Science--A Process Approach and 2) the test items from the group measure. “A Group Test of Four Processes“. The first observation period was necessary to establish which students could and which students could not perform the processes in question. The second observation period was administered immediately after the first observation period to keep the confounding variable of maturation to a minimum. Detail concerning the administration of the criterion measure and the group measure will be presented later in this chapter. Besides the confounding variable of maturation. a question arises concerning the “test wise“ effect. or the amount of learning which might occur by taking a test. It was felt that if learning did occur during the observa- tion periods. it would more likely occur during the admin- istration of the criterion measure. Since the criterion measure was a situation where students were presented with a problem and allowed to work on the solution until they were satisfied with their results. any learning that did occur while working on the problem would be reflected in their scores on the criterion measure as well as their scores on the group measure. the second observation period. :fisnszl #1 Based. on the assumption. if the criterion measure were given last. learning would not be reflected equally in the two measures. Because of this. it was decided not to split the students into two groups and alternate the administration of the two instruments . Deve10pment of Test Items To develop items for "A Group Test of Four Pro- cesses“ on the processes of controlling variables and inter- preting data. the following procedure was used: 1. 3. The author reviewed all the material in Science--A Process Approach. Parts 5. 6 and 7. Abstract items were then developed around situations which might be within the scope of junior high students. The series of items were submitted to a group of faculty and graduate students at Michigan State University's Science and Mathematics Teaching Center for the purpose of establishing face validity. They were asked to make comments and suggestions as to whether these items would elicit the appropriate responses. The members of this group were: Dr. Richard J. EcLeod. Associate Professor of Science Education: Darrel H. Fyffe. Donald Maxwell. Kenneth Calkins. and Howard Hetzig. 42 graduate assistants in science education: and Dr. Glenn D. Berkheimer. Associate Pro- fessor of Science Education. h. Those items which the group felt were inappro- priate were deleted while the items for which the group made suggestions were revised. The retained items were then reviewed by Dr. Richard J. McLeod and this author to determine the most appropriate means of presenting them to the students. According to Ebel. items written to measure the higher mental processes may fail because they ”...may involve fairly complex situations. which require many words to describe and may present the examinee with problems of comprehension and interpretation which may be irrelevant to the main purpose of the examination."52 With this in mind. it was felt that 25 of the #3 items would be more successful if they were presented with the aid of some visual materials. Because of the nature of the situations to be deveIOped. they seemed to lend themselves to be presented with the aid of colored slides and an oral script. The remaining 18 items were judged sufficiently simple to be presented in the SZRcbert L. Ebel. op. cit.. p. 52. 7. 8. 9. 43 usual paper and pencil form without the aid of a visual presentation. Slides were develOped for those items where it was deemed appropriate. The series of slides. along with accompanying items were then reviewed by McLeod and the author. Suggestions and revisions were made. The author then revised all items and placed them (the items and slides) in a sequence. The same procedure of multiple critique and revision was followed for the non-visual items. All the proposed items. those with a slide sequence and those without. were alternated with the items produced by Fyffe. The total series was then reviewed by McLeod. Berkheimer. Fyffe and the author for the final revisions. The items were revised and ”A Group Test of Four Processes” was assembled. It is most important to note that this represents only an assemblage of test items and not a finished test. At this time. however. the items were judged to possess content validity. The first #2 items of the 79 item test were developed around several different visual situations and recorded on an 35 mm color slides. Of these 42 items. 12 were on the process of controlling variables and 13 were on the process of interpreting data. A script was prepared and read orally as the students progressed through the items. 01" the remaining 37 items (WithOtIt color slides). six were on the process of controlling variables and 12 were on the process of interpreting data. These items were presented in the manner that is most commonly used for standardized tests. presenting the test booklet and letting the students work at their own rate through the items. The entire script. test. and prints of slides are presented in Appendix A. Since this was a power test and not a speed test. the students were allowed sufficient time to finish. All the students had finished in 75 minutes and most finished in 70 minutes. Each item was designed to require the use of one particular science process and was prepared in a style and language that would make it suitable for use in schools where Science--A Process Approach is not used. Testing of Items The testing of the items developed included not 0111! the actual process of testing but the selection of the Population and sample. _P_c, Elation and Sample The criterion measures use terminology that is unique to Science--A Process Approach. Therefore. it was llpcl‘tant to select the sample from a population that had 45 been exposed to that science program to avoid a communica- tion bias in the materials. The population for this study consisted of junior high school students in grade seven who had received instruction in Science--A Process Approach in the sixth grade. The sample used in this study was selected from students who had attended the Kinawa Middle School in Okemos. Michigan. during the sixth grade. These students received. as part of their sixth grade science program. instruction from the science program. Science-—A Process Approach. Parts h and 5. The seventh grade students in this school are grouped into two clusters. The school principal stated that the students are assigned to one of the two clusters without regard to ability or background. The clusters are then divided into three separate classes. These classes. science. social studies. and English. share a three hour block of time which allows the three teachers to arrange the three hours in any manner they wish. Through an 9eSZI‘eement reached with the principal and teachers. one cluster of students was assigned to us to use as our sample (N-59 ). A working agreement was reached with the science. locial studies and English teachers to periodically take one or more students from their classes for testing purposes. The school chosen for this study. the Kinawa fiddle School. is in the upper middle-class community of- Okemos. Michigan. Okemos is a suburban. bedroom community ”r6. trust is located three miles from the campus of Michigan State University. This community contains a large share of the University's faculty and married graduate students. The public school system is well equipped and financially well supported. Instrumentation Two instruments were administered to each student in the sample. They were: 1) selected "Individual Compe- tency Measures“ from Science--A Process Approach. and 2) “A.Group Test of Four Processes". The first instrument. the selected ”Individual Competency Measures". consisted of the following: 1. two of the “Individual Competency Measures" for the process of controlling variables: a) Controlling variables 7 consisting of four tacke.53 and b) Controlling Variables 9 consisting of eight tasks.5u There were a total of twelve tasks for the process of Controlling Variables. 2. three of the "Individual Competency Measures" for the process of interpreting data: a) Interpreting Data 9 consisting of six 53American Association for the Advancement of Science. 3°1ence-a-A Process A roach Part Six. (AAAS Miscellaneous IEBIIGBEIOYI 67-11,: E867: PP- 97-980 5“Ibid.. pp. lac-1&2. , 47 tasks.55 b) Interpreting Data 12 consisting of three tasks56 and. c) Interpreting Data 13 consisting of six tasks.57 For the process of interpreting data. fifteen tasks were pre- sented to the students. These five ”Individual Competency Measures” were. chosen because they: 1) were representative of the activities for the processes in question. 2) contained a variety of tasks which represents the skills the sample could reason- ably be expected to possess based on their previous year's experience. 3) did not limit the student to only one situa- tion where recall could play a big part in success. and it) used materials and equipment with which the students were familiar and which were readily available. These measures were administered individually in an interview type situation according to the guide lines described in Science---A Pracess Approach. One score for 930?! process was recorded which represented the total number 01' tasks each individual completed correctly for each pI‘OCe 88 s The students in this sample were also tested in- dl'idually on two other processes which were part of the 55Ihid.. pp. 82-83. 56Ibid.. pp. 227. 57lbid.. pp. 2&7-2h8. #8 research of Darrel w. F‘yffe.58 The average time for the total individual testing of the four processes was about 90 minutes per student with controlling variables and ‘ interpreting data accounting for an average of 55 minutes of the 90. The second instrument used was "A Group Test of Four Processes“. This test was developed in cooperation with Darrel W. Fyffe whose primary concern was in developing items to test the processes of formulating hypothesis and defining operationally. The test consisted of 79 items of which 18 were on the process of controlling variables. 25 were on the process of interpreting data and the re- maining 36 were on the other two processes. formulating hIPctniesis and defining operationally. The seventy-nine items were either multiple choice or numerical fill-in. A?! crample of a multiple choice item would be item 30 and an example of a numerical fill-in would be item 5. All items can be found in Appendix A. The reader should note that the purpose of this Study was not to develop a test but test items that corre- late with the student's ability to perform the processes of controlling variables and interpreting data as measured by the “Individual Competency Measures” from Science-u-A PI’_ccess Approach. 58Darrel w. Fyffe. The Develo ment of Test Items for ”‘9 Integrated Science Processes: Eormuléafiifi ficthesis an; geflngng fieratlh’fam. unpub s e sser a on. ° gan ta e nlverslty. 1971. 49 Administration of Instruments The Kinawa Middle School is equipped with a faculty hallway that connects the science rooms with storage area. conference rooms and the professional library. The facili- ties that were provided for our individual testing were: one of the conference rooms. and two small table and chair sets in the hallway next to a sink that had hot and cold running water. The equipment used was stored in a small office. Because of the schedule of the cluster of students. all testing was conducted in the morning. The students were available before lunch during the second. third. and fourth class periods. Each class period is 50 minutes with a five minute change period. For the individual testing. each researcher was able to draw one student at a time from the science classes. To facilitate efficiency. each of the researchers had familiarized themselves with all the ”Competency Mea- sures“ prior to testing and enough equipment was on hand so both researchers could be testing the same process at the same time. Generally this was not necessary. Each researcher was able to test our own particular processes. but at times both administered the same tests. In a number of cases the scoring was checked with the researcher of concern to assure that all scoring was uniform. 50 The individual testing was begun on April 13. 1970. and concluded on May 29. 1970. Fyffe was able to test five mornings a week while the author tested four mornings a week. Most students were completely finished with the ”Individual Competency Measures” of all four processes within two periods of testing. Several took ten to fifteen minutes longer while a small number required a third period. The group test was administered on June 3 and h. 1970. On the first day. all three classes reached item 30 and. due to lack of time. the materials were collected. The group testing was completed during the second day. During the group testing. Fyffe handled the directions. controlled the lighting and observed the class for signs to indicate movement to another slide. This author read the script and advanced the slides. The classroom teacher allowed the researchers complete control of the classes during each period. Each student was provided with a pencil. an answer sheet. and a test booklet. The answer sheet was marked as illustrated in the directions. Because of absences. only 56 of the original 59 students who had been tested individually completed the group test. The students. teachers and administration were not informed of the individual or group test results. 51 Analysis Procedure The student responses were transferred from their answer sheets to IBM H9h060 answer sheets. The IBM answer sheet consists of two sections. the tap part has space for 92 five choice items while the other section has no ten choice items. The responses to the 79 items of the group test were recorded on the top part of the answer sheet.- The student's score on the ”Individual Competency Measures” for the four processes was coded on the bottom part of the answer sheet for analysis. Using the computer at Michigan State University's Test Scoring Office. the 56 papers were analyzed to deter- mine the item analysis data based on the criterion measure. the ”Individual Competency Measure". This meant that a separate item analysis was reported for each item for all four processes. Two indices were of importance in this study. the index of item difficulty and the index of item dis- crimination. The index of item difficulty is the percentage of the total group marking a wrong answer or omitting the item. For example. if 64 percent of the students marked the question correctly. then the index of item difficulty would be 100 minus 6h or 36. The index of item discrimin- ation is the difference between the percentage of the upper group marking the right answer and the percentage of the lower group marking the right answer. For example. if for 52 a certain item. 80 percent of the upper group marked the correct response while 34 percent of the lower group marked the correct response. the index of item discrimination would be .80 minus .34 or .h6. The upper group was defined as the 27$ who received the highest scores for each process on the ”Individual Competency Measures” while the lower group was defined as the 27% who received the lowest scores on the "Individual Competency Measures”. Thus. it is impor- tant to note that success in the criterion measure deter- mined the upper and lower groups. The students in the upper and lower groups for the process of controlling var- iables were not necessarily the same students as those in the upper and lower groups for the process of interpreting data. These groups were determined independently for each process in question. To determine which items from ”A Group Test of Four Processes" would be acceptable. a minimum level for the index of item discrimination was established. Usually. 'the index of item discrimination that has a value of .ho or sweater is considered desirable. This must be looked at in the light in which the upper and lower groups are usually determined. The usual practice in determining this index is to define the upper and lower groups by the total score on the test. In this study. the scores on an external measure. the "Individual Competency Measures". were used 53 as the criteria for separating the two groups. Therefore. the choice of the upper and lower groups were not affected by the responses to any test item. It was therefore sug- gested by Dr. Robert L. Ebel.59 an expert in the field of testing. that for this study. any item having an index of item discrimination of .20 or greater should be accepted. The other analysis reported in this study gives the Pearson product moment correlation coefficients for different pairs of the four processes from the "Individual Competency Measures“ and ”A Group Test of Four Processes”. These correlation indices are important as they give further evidence of the ability of pencil and paper tests to measure student's ability to actively perform the processes in question. The correlation coefficients of interest for this study are: l. the score on the “Individual Competency Measures” and those group items with a discrimination index of .20 or greater on the process of controlling variables. 2. the score on the ”Individual Competency Measures” and those group items with a discrimination index of 0.20 or greater on the process of interpreting data. Items exhibiting correlations that are significant at the .05 level or better will be accepted as measuring sub- 59Through a personal interview with Robert L. Ebel. 51$ stantially the same thing as the corresponding competency measures 0 Summary With the goal of this study being the development of objectiviely scored test items that measure the processes of controlling variables and interpreting data. it was im- portant to compare these proposed items against the actual behaviors that define the processes in question. The pro- cedure followed was to set up a criterion measure. in this study the “Individual Competency Measures“ from Science-- A Process Approach. and then construct group test items over the same processes. The test items show evidence of content (face) validity as they were evaluated and revised by science education experts before they were administered. In addi- tion to content validity. these items show evidence of criterion-related validity using the ”Individual Compe- tency Measures” (actual student behavior) as the criterion measure. The difference in content and criterion-related validity ism the standard used for validation. Content validity uses the opinions from a Jury of science educa- t=1<>n experts as the standard while criterion-related valid- ity uses the actual student behavior as the standard. There were.two separate analyses run. First. the indices of item difficulty and discrimination were deter- P-J 55 mined using the scores on the criterion measure as the basis. The second analysis gave the Pearson product- moment correlation coefficient for different pairs of the four processes of the “Individual Competency Measures" and ”A Group Test of Four Processes“. Of particular interest to this study were the coefficients for the in- dividual and group scores on each process as further evidence of the ability of pencil and paper tests to measure students' abilities to actively perform processes of science. [1.11 CHAPTER IV ANALYSIS OF RESULTS The design of this study provides a mean to compare a astudent's performance on certain selected processes to their performance on pencil and paper group test items. Irndices of item discrimination. item difficulty and cor- relation coefficients for the paper and pencil test items Were produced and are reported in the following manner: 1. Data from the ”Individual Competency Measures" for the processes of controlling variables and interpreting data. 2. Data from “A Group Test of Four Processes” for the processes of controlling variables and interpreting data. a. Data on non-acceptable items b. Data on acceptable items 3. Data relative to the items that were presented visually and non-visually. #. Correlation coefficients which were signifi- cant to this study. Individual Competency Measures The following tables and graphs give the reader data concerning the frequency of scores on the "Individual 56 57 Ccunpetency Measures” for the processes of controlling vari- ables and interpreting data. Table 1 Frequency Distribution from the ”Individual Competency Measures" for the Process of Controlling variables (Total possible of 12) Raw Cumulative Percentile Score Frequency Frequency Rank 12 u 56 96 11 7 52 87 10 6 #5 75 9 5 39 65 8 8 3h 5h 7 5 26 n2 6 7 21 31 5 5 10 21 4 2 9 14 3 3 10 l 3 h 0 l l 1 Range 0-12 Mean 7.4 Standard Deviation 3.1 58 Table 2 Frequency Distribution from the ”Individual Competency Measures” for the Process of Interpreting Data (Total possible of 14) Raw Cumulative Percentile Score Frequency Frequency Rank in 5 56 96 13 2 51 89 12 h 49 84 11 6 #5 75 10 12 39 59 9 5 27 44 8 5 22 35 7 4 17 27 6 3 13 21 5 3 10 15 h 2 7 ll 3 3 5 6 1 2 2 2 Range 1-1h Mean 8.8 Standard Deviation 3.3 Tables 1 and 2 show that the sample of students idea's a wide range of abilities to use the processes of con- "=Jrolling variables and interpreting data. The data shown 3L1: these distributions are both slightly positively skewed. 'Irriis type of distribution does in fact enhance this study. It the scores on the ”Individual Competency Heasures' were 59 negatively skewed. a high correlation on the group test items would be meaningless. It would only tell us that students who cannot perform the process also cannot cor- rectly answer the items from the group test on that pro- cess. The following graphs. Figures 1 and 2. may help to visualize the distributions. Frequency Distribution from the “Individual Competency Measures” for the Process of Controlling Variables (A Score of 12 was possible) U) I. E e g '-'1 a V) 6“ 1 1 ,1 g- 9 fl '7 F— E“ D Q 4 § ['13 _ a 2: o 2 a e a no It SCORES Figure 1 Frequency Distribution from the ”Individual Competency Heasures' for the Process of Interpreting Data (A Score of 14 was possible) m to 21m» 3 29* U) “-6.. F'- o J ¢__,__w l-T c:4 4 N In 20 * Fl HHHHH f] 2 e e of 2 ’7‘!“ ”6” B IO I2 I4 SCORES Figure 2 60 EVen after observing that there is a real variance in the student's ability to control variables and interpret data. it could be assumed that not all the tasks from the “Individual Competency Measures” were contributing. Tables 3 and 4 show the percentages of students who responded cor- rectly to each task of the ”Individual Competency Measures" Table 3 Percentage of Students Who Performed Each Task Correctly Controlling variables Competency Task Percent Index of Task Measure Number Correct Difficulty 1 63 37 Controlling Variables 2 50 50 7 3 59 2.1 4 59 41 1 63 37 Controlling 2 80 20 variables 9 3 71 29 4 66 34 5 38 62 6 54 46 7 8? 13 8 54 46 61 Table 4 Percentage of Students Who Performed Each Task Correctly Interpreting Data Competency Task Percent Index of Task Measure number Correct Difficulty 1 48 52 2 71 29 Interpreting 3 43 57 Data 9 4 57 43 5 86 14 6 71 29 Interpreting 2 34 66 Data 12 3 82 18 1 87 13 2 91 9 Interpreting 3 89 11 Data 13 4 43 57 5 41 59 6 38 62 Tables 3 and 4 show that all tasks were contribu- ting with only three falling below 40‘. The range of task difrfiiculty for all the tasks of controlling variables was 13 to 62 while the range of task difficulty for the tasks Of iirterpreting data was 9 to 66. The data presented concerning the "Individual Com- POPOne-y measures" shows that all the individual tasks were 62 contributing. Some tasks were easier than others but none were trivial. There were no tasks that all students got right and no tasks that all students got wrong. The data also shows that the distributions for the processes of controlling variables and interpreting «hats are both slightly positively skewed. .As stated earlier. this does in fact enhance the value of the criterion measure. If there were a large number of students who could not per-1 form the processes in question. then subsequent analyses would be relatively fruitless. Following the administration of the “Individual Competency Measures“. the group measure (“A Group Test of Four Processes“) was administered. Group Measure The following tables. Tables 5 and 6. show the per- cent of correct responses for each item.and gives the cor- responding indices of item difficulty and discrimination. Table 5 Data from “A Group Test of Four Processes” for the Process of Controlling variables Naflzgzr Coigzzgngeggonses Digggziigitizzn Dggffcgititen 24- 79 .14 21 25 38 .40 63 27 71 .34 29 L 23. 23 .oo 76 29 39 .40 60 3o 46 .34 54 Table 5 (cont'd.) 63 31 32 .46 67 32 63 .47 38 33a 16 .14 84 34 41 .47 59 35 64 .40 35 36 55 .40 44 44* 41 .07 negative 59 50* 38 .13 62 63 57 .53 43 64 61 .66 39 65 54 .46 46 78 50 .33 50 rams 48.5 i 3 .33 51.5 * These items were rejected because of insufficient index of item discrimination. Table 6 Data from “A Group Test of Four Processes“ for the Process of Interpreting Data Item Percent of Index of Item Index of Item Number Correct Responses Discrimination Difficulty 3 70 .26 30 4 29 .20 71 5 36 .20 64 6 57 .40 42 11* 61 .14 4o 12* 23 .07 77 13 52 .60 48 23 32 .26 68 Table 6 (cont'd.) 64 26 20 .33 81 37 43, .53 57 38 70 .33 30 39 64 .20 36 40 39 .20 6o 45 59 .33 40 46 64 .60 35 47 71 .46 29 48 36 .60 64 49 43 .20 57 66 52 .27 48 67 63 .26 37 68 64 .60 35 73 79 .53 21 74 84 .47 16 75 64 .40 36 77* 52 .13 48 HEARS 53 .34 47 * These items were rejected because of item discrimination. Tables 5 and 6 show that while all items were meeting some degree of success. there were a few that did not achieve insufficient index of the predetermined index of item discrimination to warrant their retention. With the level of acceptance established at a discrimination index of .20 or greater. there were eight items that needed to be rejected. Five of these items were for the process of controlling variables and three 65 items were for the process of interpreting data. These are indicated by an asterisk after the item number. Non-acceptable Items The item analyses data for the rejected items will be presented along with an attempt to analyze the reason for the item's failure. No attempt was made to retest the suggested revision because of the large number of accep- table items retained. Item 24 Item 24 asks the student to analyze an investiga- tion to determine which of the variables were changed. The original item read as follows: 24. Which of the following variables were changed in this investigation? 1. amount of salt in the solution. 2. temperature of the freezer. 3. time required for the mixture to freeze. 4. size of the container. The response pattern for this item was: Foil Number Group 1 2 3 4 Omitted Upper 271 7% 87% 0% 7f 05‘ Hiddle 46;: 4x 77$ 12x 01 8;: Lower 27$ 7% 73¢ 71 7X 71 Total 5% 79$ 7;: 4% 5% Index of item discrimination .14 Index of item difficulty 21 The correct response is foil 2. 66 The foils for this item indicates that only one variable was changed during the investigation. Answering this question could then be a process of elimination. I would suggest that by changing the foils to include the possibility of more than one variable being changed. this item might be more acceptable. In doing this. I suggest revising the foils so the item reads: 24. Which of the following variables were changed in this investigation? 1. temperature r—m_.‘*_-_==q 2. temperature and time 3. temperature and amount of salt 4. temperature. time and size of container Item.28 Item 28 asks the students to look at a slide which shows three small beakers filled to the same level with a cloudy solution. one large bag of salt. and one large beaker with a cloudy solution. It is not stated that the three smaller beakers have been filled with the salt solu- tion from the larger beaker. Assuming that the beakers did not have the same salt solution. there is no one cor- rect answer since foils l. 4. and 5 all become possibilities. The item response pattern shows the following: Group 1 Upper 27$ 47% Middle 465 311 Lower 27$ 47$ Total 39% 67 Foil Number 2 3 4 5 0% oz 27% 27% 4% 12$ 19% 27% 0% 7% 27% 7% 2% 7f 23% 21% Index of item discrimination .00 Index of item difficulty 76 The correct response is foil 4. Omitted 0% 8% 131 73 I feel it would be best to change the stem of the item so it includes a statement which indicates that the three smaller beakers were. in fact. filled with the same salt solution. The item might read: 28. By filling each of the three smaller beakers from the larger beaker. the students could investigate the effect of different: 1. 2. 3. 4. 5. Item 0 amounts of salt kinds of containers sizes of containers freezer temperature none of the above This item asks the students to identify the vari- able or variables which need to be held constant to test a certain hypothesis. The original item read: 50. Which of the following variables would you held constant to test the hypothesis that the number of tablets dissolved in the water has no effect on the time needed to dissolve an- other tablet? 68 1. amount of water 2. number of saccharin tablets 3. temperature of the water 4. both amount and temperature of the water The item response pattern is: Foil Number Group 1 2 3 4 Omitted Upper 271 20$ 331 13% 33% 0$ Riddle 46x 8 z 15% 27;! 50$ 01 Lower 271 201 331 7% 20$ 20; Total 14% 25$ 18% 38$ 5% Index of item discrimination .13 Index of item difficulty 62 The correct response is foil 4. A look at the response pattern indicates there is no one foil which the majority of students preferred. The pattern also shows a similarity in the way the upper and lower groups responded. All of this leads the author to feel that the item was confusing. It might help to change the stem of this item by including some extra words in an attempt to clarify the problem. The stem should specify that the tablets. which are mentioned. have already been dissolved in the water. Changing the stem in this way also suggests a change in the second foil to include the ‘words ”already dissolved“. The item should read: 50. Which of the following variables would you hold constant to test the hypothesis: The number of tablets already dissolved in the water has no effect on the time to dissolve another tablet? 69 1. amount of water 2. number of tablets already dissolved 3. temperature of the water 4. both 1 and 3 There were five items for which this author was unable to suggest revisions which might make the items acceptable. Item 32 Item 33 shows two rods: A. which is metal. two centimeters long and solid: and G which is plastic. eight centimeters long and solid. It then asks the students to determine which variable(s) you could test by rolling them down the same inclined plane. The item reads: 33. By rolling rods. A and G. down the same in- cline. you could test the effect of the vari- able: 1. solid or hollow 2. length of rod 3. material of the rod 4. angle of the incline 5. none of the above The item response pattern is: Foil Number Group 1 2 3 4 5 Omitted Upper 271 13$ 13$ 40% 7% 27$ 0% middle 46$ 27x 15; 275 12;: 4;: 0;: Lower 271 331 71 27% 0i 131 20$ Total 25% 135 301 95 16$ ?$ >1 by. 70 Index of item discrimination .14 Index of item difficulty 84 The correct response is foil 5. The student is asked to recognize the fact that there is more than one variable which could affect the time that it took the rods to roll down the incline and therefore. foil 5. none of the above. is the correct response. This apparently is too abstract for this age student. Item 44 This item deals with information which is pre- sented in a data table as well as in graphic form. The information represents the results of an experiment where students recorded the time that it took saccharin tablets to dissolve in water of different temperatures. The item read: 44. Did this group of students change the vari- able of time? 1. yes 2. no The response pattern for this item was: Foil Number Group 1 2 Omitted Upper 27$ 60;: 401 0;: V Middle 46$ 62$ 38$ 0$ Lower 27$ 33$ 47$ 20$ Total 54;: 41;: 55¢ Index of item discrimination .07 negative Index of item difficulty 59 The correct response is foil 2. 71 The author is unable to explain the reason for the failure of this item. There were only three items in the process of interpreting data that were rejected and all three dealt with graphs. They were items 77. 11 and 12. Item 22 Item 77 presents four graphs of supposedly the same data presented in a Table and asks the students to determine which is the best graph of that data. This item should be omitted as the author is unable to explain or suggest reasons for its failure. The item response pattern was: Foil Number Group 1 2 3 4 Omitted Upper 27% 27% 13% 0% 53% 7% Middle 46$ 19$ 12$ 8$ 58$ 4$ Lower 27$ 20$ 13$ 13$ 40$ 13;: Total 21% 13% 7% 52% 7% Index of item discrimination .13 Index of item difficulty 48 The correct response is foil 4. Items 11 and 12 Both of these items refer to a graph which showed a relationship between the amount of weight hung from the end of a stick and the height the end of the stick is from the floor. To answer these questions. the students were required to write in a numerical answer which represents the weight of the object causing the particular amount of 72 bend. The students' responses. hereafter referred to as R. were then transcribed by this author to the machine scoring answer sheets in accordance with the following code. Foils for question 11 were: 1. BS 25 grams 2. 25 grams ‘i 8 I2. )6 FORCE APPuED (BOOKS) O (a 0 VOL. OF AIR (ms) 9 B 8 v e 12 I6 FORCE APPLIED (BOOKS) O FORCE APPLIED As books were added to the top of the plunger. the Below are the VOLUME OF AIR (books) (milliliters) 2 26 ‘4 21 6 18 8 16 12 13 16 ll U Q VOL. OF MR "(5) v e :2 16 FORCE APPLIED (BOOKS) O at? 0' N O m 0 v01. OF AIR (ms) 4 8 12 I6 FORCE APPLIED (Bocxs) O A-33 78. In this experiment. which of the following were kept constant in the experiment? 1. volume of air 2. number of books 3. the syringe 4. none of the above APPENDIX B COMPLETE ITEM ANALYSIS DATA FOR THE ACCEPTED ITEMS ON THE PROCESS OF CONTROLLING VARIABLES 102 APPENDIX B COMPLETE ITEM ANALYSIS DATA FOR THE ACCEPTED ITEMS ON THE PROCESS OF CONTROLLING VARIABLES Item 25 The correct option is 4 ITEM RESPONSE PATTERN l 2 3 4* 5 Omit Upper 27% 3 2 0 8 2 O 203 13% 01 531 13% 0% Middle 461 6 2 1 11 3 3 2 31 8 z 4% 42¢ 12 x 12 3 Lower 27% 6 2 2 2 2 1 40$ 13% 132‘ 13$ 13$ 7% TOTAL 15 6 3 21 7 4 275 11% 55 333 131 75 Index of difficulty .63 Index of discrimination .40 Item 27 The correct option is 4 ITEM RESPONSE PATTERN l 2 3 4* Omit Upper 27% l 1 0 13 0 7% 7% 0% 87% 0% Middle 46% l 3 1 19 2 4% 12% 4% 73% 8% Lower 27% 2 3 l 8 l 13% 20% 7% 53% 7% TOTAL 4 7 2 40 3 7% 13% 4% 71% 5% Index of difficulty .29 Index of discrimination .34 Item 29 The correct option is 3 ITEM RESPONSE PATTERN l 2 3* 4 Omit Upper 27% 1 5 8 1 0 7% 33% 53% 7% 0% Middle 46% ' 3 4 12 4 3 12% 15% 46% 15% 12% Lower 27% ‘ 0 8 2 3 2 0% 53% 13% 20% 13% TOTAL 4 17 22 8 5 7% 30% 39% 14% 9% Index of item difficulty .60 Index of item discrimination .40 Item 30 The correct option is 4 ITEM RESPONSE PATTERN l 2 3 4* Omit Upper 27% 2 2 1 10 O . 13% 13% 7% 67% 0% Middle 46% 6 3’ 4 11 2 23% 12% 15% 42% 8% Lower 27% l l 6 5 2 7% 7% 40% 33% 13% TOTAL 9 6 ll 26 4 16% 11% 20% 46% 7% Index of item difficulty .54 Index of item idscrimination .34 Item 31 The correct option is 3 ITEM RESPONSE PATTERN 1 2 ' 3* 4 5 Omit Upper 27% 3 1 ' 8 1 2 o . 20% 7% 53% 7% 13% 0% Middle 46% 6 1 9 3 6 1 23% 4% 35% 12% 23% 4% Lower 27% 5 1 1 s - o 3 33% 7% 7% 33% 0% 20% TOTAL 14 3 18 9 8 4 (25% 5% 32% -16% 14% 7% Index of item difficulty .67 Index of item discrimination .46 Item 32 The correct option is 2 Upper 27% Middle 46% Lower 27% TOTAL ITEM RESPONSE 1 0 0% 1 4% 1 7% 2 4% 2* 12' 80% 18 69% 5 33% 35 63% Index of item difficulty Index of item discrimination Item 34 The correct option is 3 Upper 27% Middle 46% Lower 27% TOTAL PATTERN 3 4 1 1 7% 7% 2 3 8% 12% 1 1 7% 7% 4 5 7% 9% .38 .47 ITEM RESPONSE PATTERN 1 1 7% 11% 2 4 27% 8 31% 3 20% 15 27% Index of item difficulty Index of item discriminatir I .3 .. .. e 3* 10 67% 10 38% 3 20% 23 41% .59 .... 4 0 0% 5 19% 4 27% ' 9 16% S Omit 1 0 7% 0% 1 1 4% 4% 3 4 20% 27% 5 5 9% 9% Omit O 0% 0 . 0% 3 20% 3 5% B-S Item 35 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* 4 5 Omit Upper 27% 2 0 13 0 0 O 13% 0% 87% 0% 0% 0% Middle 46% 2 0 16 4 3 1 8% 0% 26% 15% 12% 4% Lower 27% 4 1 7 O 0 3 27% 7% 47% 0% 0% 20% TOTAL 8 ‘ 1 36 4 3 4 14% 2% 64% 7% 5% 7% Index of item difficulty .35 Index of item discrimination .40 Item 36 The correct option is 1 ITEM RESPONSE PATTERN 1* . 2 3 4 5 Omit Upper 27% 10 1 1 2 l 0 67% 7% 7% 13% 7% 0% Middle 46% 17 0 2 6 1 - 0 65% 0% 8% 23% 4% 0% Lower 27% 4 3 1 4 . 0 3 27% 20% 7% .27% 0% 20% TOTAL '31 4 4 12 2 3 55% 7% 7% 21% 4% 5% Index of item difficulty .44 Index of item discrimination .40 Item 63 The correct option is 4 ITEM RESPONSE PATTERN 1 2 3 Upper 27% 0 2 1 0% 13% 7% Middle 46% 5 0 5 19% 0% 19% Lower 27% 2 2 4 13% 13% 27% TOTAL 7 4 10 13% 7% 18% Index of item difficulty .43 Index of item discrimination .53 Item 64 The correct option is 3 4* 12 80% 16 62% 4 27% 23 57% ITEM RESPONSE PATTERN 1 2 3* Upper 27% 1 0 14 7% 0% 93% Middle 46% 5 4 16 19% 15% 62% Lower 27% 3 4 4 20% 27% 27% TOTAL 9 8 34 16% 14% 61% Index of item difficulty .39 Index of item discrimination .66 4 0 0% 1 4% 4% 7% Omit 0% 0% 20% 5% Omit 0 0% 0% 20% 5% Item 65 The correct option is 1 ITEM RESPONSE PATTERN 1* 2 3 4 Omit Upper 27%' 11 3 1 0 0 73% 20% 7% 0% 0% Middle 46% 15 5 4 2 0 58% 19% 15% 8% 0% Lower 27% 4 - 2 4 2 3 27% 13% 27% 13% 20% TOTAL 30 10 9 4 3 54% 18% 16% 7% 5% Index of item difficulty .46 Index of item discrimination .46 Item 78 The correct option is 3 ITEM RESPONSE PATTERN. 1 2 3* 4 Omit Upper 27% 4 l 9 1 0 27% 7% 60% 7% 0% Middle 46% 6 2 ’ 15 2 . 1 - 23% 8% 58% 8% 4% Lower 27% 3 2 4 3 ~ 3 20% 13% 27% .20% 20% TOTAL 13 5 28 6 4 23% 9% 50% 11% 7% Index of item difficulty .50 Ihidex of item discrimination .33 APPENDIX C COMPLETE ITEM ANALYSIS DATA FOR THE ACCEPTED ITEMS ON THE PROCESS OF INTERPRETING DATA 103 Item 3 APPENDIX C COMPLETE ITEM ANALYSIS DATA FOR THE ACCEPTED ITEMS ON THE PROCESS OF INTERPRETING DATA The correct option is 3 Upper 273‘ Middle 46% Lower 27% TOTAL Index of item difficulty ITEM RESPONSE PATTERN 1 o 0% o 0% 1 7% 1 2% 2 2 13% 5 191 5 331 12 211 .30 Index of item discrimination 3* 11 735 21 811 7 1471 39 70% .26 u 1 7% o 01 o 0% 1 2% 0% 0% 0% 01 Omit 7X 0% 2 13% 5% Item 4 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* 4 Upper 27% 1 5 5 3 7% 33% 33% 20% Middle 46% 7 3 9 7 27% 12% 35% 27% Lower 27% 5 l 2 5 33% 7% 13% 33% TOTAL 13 9 16 15 23% 16% 29% 27% Index of item difficulty .71 Index of item discrimination .20 'Item 5 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* 4 Upper 27% 0 3 6 5 0% 20% 40% 33% Middle 46% 3 4 11 7 12% 15% 42% 27% Lower 27% 3 1 3 5 20% 7% 20% .33% TOTAL 6 8 20 17 11% 14% 36% 30% Index of item difficulty .64 Index of item discrimination .20 0 0% 0% 0% 0% 0% 4% 7% 4'2; Omit 1 7% 0% 13% 5% Omit 7% 0% 13% 5% Item 6 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* 4 s Omit Upper 27% 0 0" 11 3 0 1 0% 0% 73% 20% 0% 7% Middle 46% 0 3 16 7 0 0 0% 12% 62% 27% 0% 0% Lower 27% 3 0 5 4 0 3 20% 0% 33% 27% 0% 20% TOTAL 3 3 32 14 o 4 5% 5% 57% 25% 0% 7% Index of item difficulty .42 Index of item discrimination .40 Item 13 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* 4 5 Omit Upper 27% 0 2 11 1 0 l 0% 13% 73% 7% 0% 7% Middle 46% 2 2 16 3 1 . 2 8% 8% 62% 12% 4% 8% Lower 27% 2 1 2 4 . 3 3 13% 7% 13% 27% 20% 20% TOTAL 4 5 29 8 4 6 7% 9% 52% 14% 7% 11% Index of item difficulty .48 Index of item discrimination .60 Item 23 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* 4 Upper 27% 2 3 t 5 4 13% 20% 33% 27% Middle 46% 6 2 12 6 23% 8% 46% 23% Lower 27% 7 2 1 2 47% 13% 7% 13% .TOTAL 15 7 18 12 27% 13% 32% 21% Index of item difficulty .68 Index of item discrimination .26 Item 26 The correct option is 3 ITEM RESPONSE PATTERN l 2 3* 4 Upper 27% 0 3 S 3 0% 20% 33% 20% Middle 46% 1 11 6 4 4% 42% 23% 15% Lower 27 % 1 5 0 3 7% 33% 0% 20%' TOTAL 2 19 11 10 4% 34% 20% 18% Index of item difficulty .81 Index of item discrimination .33 Omit 1 7% 0 0% 3 20% 4 7% 5 Omit 2 2 13% 13% 3 . 1 12% 4% 3 3 20% 20% 8 6 14% 11% Item 37 The correct Option is 2 ITEM RESPONSE PATTERN 1 2* 3 Omit Upper 27% 4 11 0 0 27% 73% 0% 0% Middle 46% 9 10 6 1 35% 38% 23% 4% Lower 27% 8 3 2 2 53% 20% 13% 13% TOTAL 21 24 8 3 ‘ 38% 43% 14% 5% Index of item difficulty .57 Index of item discrimination .53 Item 38 The correct option is 1 ITEM RESPONSE PATTERN 1* 2 3 . Omit Upper 27% 14 1 0 0 , 93% 7% 0% 0% . Middle 46% 16 7 2 1 62% 27% 8% 4% Lower 27% 9 3 1 2 60% 20% 7% . 13% TOTAL 39 11 3 3 70% 20% % 5% Index of item difficulty .30 Index of item discrimination .33 ‘Item 39 The correct option is 2 ITEM RESPONSE PATTERN l 2* 3 Omit Upper 27% 3 10 2 0 20% 67% 13% 0% Middle 46% 4 19 2 l 15% 73% 8% 4% Lower 27% 4 7 2 2 27% 47% 13% 13% TOTAL 11 36 6 3 20% 64% 11% 5% Index of item difficulty .36 Index of item discrimination .20 Item 40 The correct cption is 3 ITEM RESPONSE PATTERN l 2 3* _ Omit Upper 27% 7 0 8 0 47% 0% 53% 0% Middle 46% 14 2 9 1 54% 8% 35% 4% Lower 27% 5 3 5 2 33% 20% 33% 13 TOTAL 26 5 22 3 46% 9% 39% 5% Index of item difficulty .60 Index of item discrimination .20 Item 45 The correct option is 2 Upper 27% Middle 46% Lower 27% TOTAL ITEM RESPONSE PATTERN 1 4 27% 6 23% 4 27% 14 25% 2* 11 73% 16 62% 6 40% 33 59% Index of item difficulty Index of item discrimination Item 46 The correct option is 2 Upper 27% Middle 46% Lower 27% TOTAL ITEM RESPONSE 1 0 0% 0 0% 0 0% 0 0% 2* 13 87% 19 73% 4 27% 36 64% Index of item difficulty Index of item discrimination .60 3 4 0 0 0% 0% 2 1 8% 4% 1 2 7% 13% 3 3 5% 5% .40 .33 PATTERN 3 4 1 1 7% 7% 4 2 15% 8% 4 5 27% 33% 9 8 16% 14% .35 Omit 0% 4% 13% 5% Omit 0% 4% 13% 5% Item 47 The correct cption is 3 Upper 27% Middle 46% Lower 27% TOTAL ITEM RESPONSE PATTERN 1 0 0% 3 12% 1 7% 4 7% 2. 0 0% 1 4% 1 7% 2 4% Index of item difficulty Index of item discrimination Item 48 The correct option is 1 Upper 27% Middle 46% Lower 27% TOTAL 3* 14 93% 19 73% 7 47% 40 71% .29 .46 4 1 7% 8% 27% 13% ITEM RESPONSE PATTERN 1* 10 67% 9 35% 1 7% 20 36% 2 4 27% 11 42% 5 33% 20 36% Index of item difficulty Index of item discrimination 3 . 0 0% 4 15% 6 40% 10 18% .64 .60 4 1 7% 1 4% 7% 5% Omit 0 0% 4% 13% 5% Omit 0% 4% 13% 5% Item 49 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* 4 Omit Upper 27% 4 1 7 2 1 27% 7% 47% 13% 7% Middle 46% 5 2 13 5 1 19% 8% 50% 19% 4% Lower 27% 1 3 4 5 2 7% 20% 27% 33% 13% TOTAL 10 6 24 12 4 18% 11% 43% 21% 7% Index of item difficulty .57 Index of item discrimination .20 Item 66 The correct option is 4 ITEM RESPONSE PATTERN l 2 3 , 4* Omit Upper 27% l 3 2 9 0 7% 20% 13% 60% 0% Middle 46% 3 4 3 15 1 12% 15% 12% 58% 4% Lower 27% 1 3 4 5 ' 2 7% 20% 27% 33% 13% TOTAL 5 10 9 29 3 9% 18% 16% 52% 5% Index of item difficulty .48 Index of item discrimination .27 0J0 Item 67 The correct option is 1 ITEM RESPONSE PATTERN 1* 2 3 4 Omit Upper 27% 11 3 l 0 0 73% '20% 7% 0% 0% Middle 46% 17 0 6 2 1 65% 0% 23% 8% 4% Lower 27% 7 4 1 1 2 47% 27% 7% 7% 13% TOTAL 35 7 .8 3 3 63% 13% 14% 5% 5% Index of item difficulty .37 Index of item discrimination .26 Item 68 The correct option is 2 ITEM RESPONSE PATTERN l 2* 3 , 4 Omit Upper 27% 0 14 1 0 0 0% 93% 7% 0% 0% Middle 46% 3 17 4 1 1 12% 65% 15% 4% 4% Lower 27% 0 5 6 2 ' 2 0% 33% 40% 13% 13% TOTAL 3 36 ll 3 3 5% 64% '20% 5% 5% Index of item difficulty .35 Index of item discrimination .60 C-ll Item 73 The correct option is 4 ITEM RESPONSE PATTERN l 2. 3 4* Omit Upper 27% 0 0 0 15 0 0% 0% 0% 100% 0% Middle 46% 1 0 2 22 1 4% 0% 8% 85% 4% Lower 27% 3 2 1 7 2 20% 13% 7% 47% 13% TOTAL 4 2 3 44 3 7% 4% 5% 79% 5% Index of item difficulty .21 Index of item discrimination .53 Item 74 The correct option is 3 ITEM RESPONSE PATTERN 1 2 3* - 4 Omit Upper 27% 0 0 15 0 0 0% 0% 100% 0% 0% Middle 46% 0 1 24 0 1 0% 4% 92% 0% 4% Lower 27% 2 3 8 0 ' 2 13% 20% 53% 0% 13% TOTAL 2 4 47 0 3 4% 7% 84% 0% 5% Index of item difficulty .16 Index of item discrimination .47 C—12 Item 75 The correct option is 4 ITEM RESPONSE PATTERN 1 2 3 4 Upper 27% 0 0 0 13 0% 0% 0% 87% Middle 46% 1 1 4 ' 16 4% 4% 15% 62% Lower 27% 0 0 2 7 0% 0% 13% 47% TOTAL 1 1 6 36 2% 2% 11% 64% Index of item difficulty .36 Index of item discrimination .40 5 Omit 1 1 7% 7% 3 1 12% 4% 4 2 27% 13% 8 4 14% 7% _. __ '1 I II .. g. - 1, FEB 74 ’ 1f. MANCHESTER IND] "1111101m