91m: A 14‘. . ' Q‘ 25¢ per day per item ‘ ‘l Eflamx‘ IA RETURNIW LIBRARY MTERIALS: ‘ at? Place in book return to remove \< r "x ‘ charge from circulation records ~33. , THE DEVELOPMENT AND ASSESSMENT OF PROCEDURES TO DERIVE REPRESENTATIONS OF STUDENTS' PROPOSITIONAL KNOWLEDGE FROM MULTIPLE CHOICE TEST RESPONSES By John Addison Caldwell A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Secondary Education and Curriculum 1980 (:) Cepyrig ht by JOHN ADDISON CALDWELL 1980 ABSTRACT THE DEVELOPMENT AND ASSESSMENT OF PROCEDURES TO DERIVE REPRESENTATIONS OF STUDENTS' PROPOSITIONAL KNOWLEDGE FROM MULTIPLE CHOICE TEST RESPONSES BY John Addison Caldwell A question raised by educational researchers studying the cognitive organization of knowledge before and after instruction is, "How does the knowledge a person has previously acquired interact with organized instructional content to produce a new knowledge structure in the person's mind?" (Posner et al. 1977). To answer the question of knowledge interaction requires assessment. The assessment of an individual's knowledge and its organization presents a difficult methodological problem. Traditional assessment procedures position an individual on a continuum.and compare individuals with each other (Magnusson 1967) or a criterion (Hambleton et a1. 1978). Although traditional assessment items test knowledge of propositions (Ebel 1965), the scores received from the analysis indicate a proportion of acquired knowledge rather than the specific acquired knowledge. This study focused on one kind of knowledge to be acquired-—propositional knowledge. The purpose of the study ‘was to develop and apply techniques for the analysis of student knowledge as demonstrated by the propositions the student implicitly asserts on multiple choice items. The study was conducted during the summer of 1978 at Michigan State University. Twenty-seven Elementary Education majors enrolled in Biological Science 202 (BS 202) provided the sample for the study. Students' responses to an essay question and a set of multiple choice questions developed for a unit on terrestrial succession were analyzed. The objectives of the study were: 1. Develop procedures to organize the propositional know- ledge tested in a test and apply them to a test on terrestrial succession 2. Develop procedures to analyze the propositional know- ledge implicitly asserted and interpret student item responses in terms of the underlying set of propositions presented in the test 3. Assess the validity of the item analysis by comparing the students' set of propositions received from the multiple choice analysis to those propositions derived using techniques for assessing propositional knowledge. The analysis involved the following steps: 1. Analyze the multiple choice test and derive the inher- ent propositions 2. Organize the propositions and apply the student responses to that organization 3. Following Pines (1977), derive propositions from the essay responses and calculate the inter- and intra-judge reliability of the essay analysis 4. Calculate the percentage agreement between the students' essay propositions and multiple choice propositions to estimate the validity of obtained information The developed procedures from this study provide a method to succinctly organize relatively large sets of propositions within a domain of knowledge, large sets of propositions within a test, and a lengthy list of proposi- tions students assert as true in a traditional assessment instrument. These procedures have implications for those researchers and instructors interested in cognitive science, instructional improvement, instructional interactions, and the assessment of knowledge. References Ebel, Robert L. Measuring Educational Achievement. Engle- wood Cliffs, New Jersey: Prentice Hall,7l965. Hambleton, R. K.; Swaminathan, H.; Algina, J.; and Coulson, D. B. "Criterion-Referenced Testing and Measure- ment: A Review of Technical Issues and Develop- ments." Review of Educational Research 48 (Winter 1978): 1-47. Magnusson, David. Test Theor . Reading, Massachusetts: Addison-Wesley PuEIisHing Company, 1967. Pines, A. L. "Scientific Concept Learning in Children: The Effect of Prior Knowledge on Resulting Cognitive Structure Subsequent to A-T Instruction." Ph.D. dissertation, Cornell University, 1977. Posner, George S., and the Cognitive Structure Group. "The Assessment of Cognitive Structure." Research Report No. 5. Department of Education. Cornell University, 1977. To Sandy, Mbm and Dad, and my girls, Stacy and Nicole, with love and appreciation iii ACKNOWLEDGMENTS To express appreciation to all individuals who have contributed to the achievement of this goal would require enumerable pages. I would like, however, to specifically recognize and thank a small minority. First, I extend gratitude to my committee: Dr. Edward L. Smith, Dr. N. Jean Enochs, Dr. Robert G. Bridgham, and Dr. Joe L. Byers. Dr. Smith, as my teacher, committee chairman, and dissertation director, has provided enormous time and energy toward the completion of my studies and this dissertation. I hope I can do at least half as much for my students. I want to thank Dr. Enochs for her assistance with the essay analysis and allowing me to conduct this study as part of her course improvement project. To Dr. Bridgham, I deeply thank-you for your encouragement to develop these procedures and your practical and thoughtful suggestions. To Dr. Byers, much appreciation for your suggestions and questions which allowed me to consider the alternative facets of my thoughts. I would also like to thank Dr. James Stewart, the course instructor during the summer of 1978, for altering the schedule so I would have a population of students for iv my study. In addition to Dr. Stewart, I also thank John Fogl and Chuck Elzinga for teaching the instructional unit of interest. A special note of appreciation is extended to Dr. Mohan Wali at the University of North Dakota for his editorial assistance on the final copy and to my typists, Tena French and Sharon Hillestad for their long hard work and valuable assistance. A special note of gratitude is extended to my wife, Sandy, my parents, and my daughters for their love, encouragement, and sacrifice over the years of study and writing. As Hemingway and Newton have indicated, one does not stand or triumph alone. And, regretfully, I am unable to mention everyone; but you are in my thoughts. TABLE OF CONTENTS Chapter I. STATEMENT OF THE PROBLEM . General Problem Purpose . . Problem in Context . . . Limitations of the Study . Objectives of the Study Overview of the Study REVIEW OF THE LITERATURE . Overview . Measurement Propositional Knowledge . Knowledge Assessment Techniques from Cognitive Science Summary . Background of the Study PROCEDURES FOR TEST ANALYSIS . Organizing the Propositional Knowledge within Multiple Choice Items . Derivation of Propositions . Recording and Scoring Student Responses Proposition Score Derivation . . . Guidelines for Interpreting and Analyzing Student Responses . . Procedures to Assess the Validity of the Information Received from.the Multiple Choice Items Procedures for Propositional Analysis of . Student written Responses . . Establishing Intra- -judge Reliability . Establishing Inter- -judge Reliability . Assessment of the Validity of Information Received from the Multiple Choice Analysis . Sample and Data Collection . Summary vi IV. RESULTS Overview . . . Multiple Choice Analysis . Discussion . . . Essay Analysis . . . Establishing Intra- -judge Reliability . Establishing Inter- -judge Reliability . Summary . Validation of Multiple Choice Propositions Summary . . V. DISCUSSION OF THE RESULTS AND CONCLUSIONS Overview . . Overview of the Study Discussion . . . . Objective 1 Objective 2 Objective 3 Summary . . Extensions of. the Developed Method . Test Propositional Content Analysis Multiple Choice Analysis . . Diagnosis of an Individual Student' s and the Group' 5 Knowledge of the Subject Matter . . . . . . Models of Student Knowledge Summary . Implications for Research and Practice . Research . Practice . Conclusions APPENDIX A. SEMANTIC NETWORK FOR THE TERRESTRIAL SUCCESSION INSTRUCTIONAL UNIT . APPENDIX B. INSTRUCTED PROPOSITIONS FOR EACH TASK . APPENDIX C. TEST B APPENDIX D. ESSAY ANALYSIS PROCEDURES . APPENDIX E. FREQUENCY A PAIR OF CONCEPTS WERE ADDRESSED BY STUDENTS' ITEM RESPONSES . LIST OF REFERENCES . vii 72 72 100 101 101 104 107 108 113 116 116 116 119 119 122 127 130 133 138 140 150 153 153 155 157 159 161 163 167 174 176 O‘UI-l-‘w 10. 11. 12. 13. 14. LIST OF TABLES Proposition Matrix for Terrestrial Succession with Tested Propositions Indicated . Test-Item Proposition Matrix . Table appears on . Proposition Summary Matrix for the Test Proposition Frequency Matrix . Proposition Selection Matrix . List of Propositions Derived from Twelve-Item Terrestrial Succession Multiple Choice Test Proposition Matrix for Terrestrial Succession Unit with Tested Propositions Indicated Table appears on . Test—Item Proposition Matrix with Types of Relations and Tasks Indicated . Table appears on . Test-Item Proposition Matrix with Student A Responses Coded Table appears on . Test-Item Proposition Matrix with Student B Responses Coded . . . . . Table appears on . Proposition List for Student A Derived from the Test-Item Proposition Matrix . Proposition List for Student B Derived from the Test-Item Proposition Matrix . Proposition Frequency Matrix for Student A . Proposition Frequency Matrix for Student B . viii 49 51 51A 56 57 67 73 77 77A 78 78A 81 81A 82 82A 83 84 86 87 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. Test-Item Proposition Matrix for BS 202 Students . . . . . . . . . . . . Table appears on . Group Proposition Scores Proposition List for Group Derived from Test-Item Proposition Matrix . Analysis of Student B's Essay Response . Results of Assessing Intra-judge Reliability . Results of Assessing Inter-judge Reliability . Proposition Selection Matrix for a Student's Multiple Choice Responses and Essay Responses Table appears on . Results of Multiple Choice and Essay Response Comparison for the Student Sample Table appears on . Percentage Agreement Between Comparable Instances of Multiple Choice Propositions and Essay Propositions . . . Proposition Matrix for Test Development Table appears on . Proposition Frequency Matrix for a Test on Terrestrial Succession . Test-Item Proposition Matrix for Student B . Table appears on . Test-Item Proposition Matrix for Student C . Table appears on . . . . . Test-Item Proposition Matrix for Group Errors Table appears on . ix 94 94A 95 97 102 103 104 111 111A 112 112A 113 134 134A 136 142 142A 144 144A 147 147A LIST OF FIGURES Figure 1. An Example Semantic Network about Terrestrial Succession . . . . . . . . . . . . . . . . . 4 2. Representation of a Portion of the Succession Semantic Network . . . . . . . . . . . . . . 37 3. Chart Representing Translation of Relation into Intended Learning Situation (a Task) . 41 Figure appears on . . . . . . . . . . . . 41A 4. Task-Concept Matrix for an Instructional Unit on Terrestrial Succession . . . . . . . . . 43 5. Code Definitions and Explanations . . . . . . 52 6. Formation of a Row in the Test-Item Proposition Matrix . . . . . . . . . . . . . 53 7. Row within Test—Item.Proposition Matrix with Example Student Responses Recorded . . . . . 54 8. Form to Record Frequency of Comparable and Noncomparable Propositions from Two Sources . . . . . . . . . . . . . . . . . . 69 Figure appears on . . . . . . . . . . . . 69A 9. Pairs of Concepts Addressed by Multiple Choice Items and Expected Pairs Addressed by Essay Item . . . . . . . . . . . . . . . 109 10. Concept Map of a Student's Conception of Niche Specialization . . . . . . . . . . . . 145 11. Concept Map of Group's Conception of Niche Specialization . . . . . . . . . . . . . . . 149 CHAPTER I STATEMENT OF THE PROBLEM General Problem Martin (1972) states, "one of the traditional goals of science education has been the acquisition of knowledge" (p. 133). As a goal, "acquisition of knowledge" is not new to science education or education in general. Typically, knowledge has been considered in terms of subject matter topics. That is, the topic represents the knowledge to be acquired. For example, if the topic was ecological succession, then the student was to acquire knowledge about ecological succession. The instructor organized lessons that dealt with the topic, taught, and gave a test. If the student passed the test, his behavior is assumed to be evidence of his knowledge of the topic. If the student did not pass the test, then his behavior indicated he did not know the topic. The test score was interpreted as reflecting the amount of knowledge possessed for the topic. Treatment of knowledge in this way does not develop much information about the knowledge acquired (knowledge being a result that allows a behavior to be exhibited). 2 It does not provide answers to questions like: What spe- cifically was the knowledge a student was to acquire? What knowledge about the topic does the student now possess that he did not have before? If the student failed to pass the test, what did he learn or not learn about the topic? Recently, cognitive scientists have begun to investigate the kinds of knowledge that can be acquired. One kind of knowledge that can be acquired is propositional knowledge (Martin 19721; Posner 1978). Propositional knowledge is defined as knowledge of concepts and their relations to other concepts. An example of a proposition is, "Early succession has only a few species." In this proposition a relation is formed between two concepts-- succession and species. Posner (1978) suggests instructional and curricu- lum developers consider the potential cognitive science technology offers. Specifically, Posner addresses the use of semantic networks, a diagram of the interrelated propositions representing the knowledge to be acquired. Using the above proposition as an example, the representation would be: 1Martin (1972) is a philosopher not a cognitive scientist and has provided an interesting analysis of the knowledge level in Bloom's Taxonomy of Educational Objec- tives. ® ® (type .lue) (quant1 y value) number of species stage 0 succession property The proposition states a property of early succes- sion. Following Posner (1978), other lines representing properties could be drawn. Attached to these lines would be other concepts--stratification, amount of light, etc.-- which pertain to early succession. Furthermore, "early" could be considered as a value of a variable succession and other values could be attached. The same could be done for species and other relevant concepts, resulting in the diagram in Figure l. Succession is an ongoing process and all values of the represented propositions are simultaneous. A change in one variable value can change the value of other variables within the system represented by the network. Therefore, the represented concept relations are true or false depend- ing on the value of the variable concept at a particular point in time. The specificity of a network depends on the degree of resolution desired. For curriculum development, resolu- tion would be coarse; for instructional development, reso- lution would be fine. @ High E Pioneer Stage Middle Stage Climax Stage Low i Medium 9 o High Medium Ecosystem L_____._—- Figure 1. An example semantic network about ter- restrial succession Posner (1978) suggests a number of instructional applications of propositional networks: 1. Comparison of the effects of different content struc- tgggg. The network specifies what is intended to be taught and learned--the curriculum” 2. Explication of what was taught. A comparison of what was actually taught to what was intended to be taught can be made. Omissions can be detected and the amount of time devoted to certain propositions determined. 3. Integration of previous topics the student has encoun- tered using the same concepts orgpropositions. If the student has studied the concepts and propositions before 5 but in a different context, they can be used as orga— nizers for subsequent instruction or the application of those concepts to other instances can be indicated. 4. Generation of questions to determine what a student believes. With the network, the instructor has a representation of what is intended to be learned. Therefore, the instructor can use this to form dis- cussion questions or test questions or both. 5. Evaluation of student learning. The student responses to questions can be compared to the instructional and curricular networks to determine student learning as a result of instruction. The potential applications indicate an important role in education for representation of propositional know- ledge. With the application of cognitive science technology, it should be possible to describe what is intended to be taught and learned (curriculum), what is actually taught (instruction), and what is actually learned (results of instruction). The technology appears to make explicit what is intended and what occurred during instruction, but how are the results of instruction assessed? The question becomes: What methods are available for assessing students' propositional knowledge? Recent investigators (Preece 1976; Shavelson 1974; Posner et a1. 1977) have developed techniques to assess propositional knowledge. These techniques include Word Association, Conceptual Mapping or Conceptual Tree 6 Construction, Propositional Analysis of Clinical Interviews, Generating Propositions, and Problem-Solving Tasks (Posner et a1. 1977). Using these techniques, models of student knowledge have been built and compared to models of the knowledge to be acquired. Of these techniques, Generating Propositions and Problem-Solving Tasks are most amenable to regular class- room assessment situations. The format is similar to essay questions. Although the techniques have some construct validity, data analysis is tedious, because the techniques require an individual to transform the subject's written material into a set of numbers representing the number of lines between concepts. Then the numbers are entered into a computer for analysis. The raw data cannot be entered directly for analysis as can data from standardized tests.2 Therefore, the techniques will probably not be incorporated into regular classroom assessment programs on a regular basis, at least not in the near future. But some form of assessment is needed. Knowing what was taught is valuable information, but knowing what was learned indicates the effects of what was taught. Instructors have devised their own assessment mea- sures. These measures usually take the form of multiple choice, true/false, fill—in-the-blank, matching, or essay 2A‘more complete description of the propositional analysis technique appears in Chapter II and in Posner and thehCognitive Structure Group, 1977, listed in the bibliog- rap y. 7 items with multiple choice items most commonly used. These assessment forms are easily administered and scored. These items or tests measure student knowledge of the subject matter. Part of the subject matter knowledge is proposi- tional knowledge. Therefore, test results should help to indicate the propositional knowledge the student possesses. Present methods of item and test analysis provide the following information: item discrimination, item difficulty, item intercorrelations, frequency of selected answers, reliability coefficients, scores, and score dis- tributions. This is valuable information with respect to the test and items. But the information does not indicate the propositional knowledge students possess. In current test analyses, there is no indication of the conceptions or misconceptions students have with respect to the propositions inherent in the subject matter. If propositions to be tested can be detailed, then repre- sentation of student propositional knowledge is also possible, and present methods of assessment could provide a basis for that representation. Multiple choice items can be viewed as constituting a set of alternative propositions. In responding, an indi- vidual can be viewed as asserting one or more propositions and denying others. This set of asserted and denied propositions can provide a basis for inference concerning the individual's propositional knowledge. Purpose The purpose of this study is to develop, apply, and evaluate a procedure for the analysis of student knowledge as demonstrated by the propositions implicitly asserted on multiple choice items. Problem in Context The following section describes the manifestation of the specific problem, that is, the particular situation where student knowledge will be assessed. Next, the limitations of the study are briefly discussed followed by the objectives of the study. At Michigan State University elementary education majors enroll in a science course entitled Biological Science for Elementary Teachers (BS 202). BS 202 is a general biology course based on the study of organisms and ecosystems. One of the goals of BS 202 is for students to learn the "facts, concepts, and conceptual relations of biology" (Course Syllabus 1978, p. 1). As part of an instructional improvement program, efforts have been initiated to organize the facts, concepts, and conceptual relations that pertain to ecosystems into propositions and the corresponding semantic networks. The networks form a model of what is to be taught and learned. Depending on the resolution desired, the model illustrates which areas of content are to be addressed for different levels of course improvement. It can also form 9 a monitoring system for instruction to indicate when a concept or proposition is to be introduced or reintroduced. In addition, the model can be used as a basis for student feedback, a tool to use for generating questions to detect student conceptions and misconceptions of the subject matter and begin remedial measures. Maximum utilization of the instructional model is dependent on having available a representation of the stu- dent's knowledge. It is then possible to compare his knowledge to the content model. The problem becomes: With of student knowledge £9 relate £9 the content model? The previously mentioned techniques of proposition generation and problem-solving tasks with clinical inter- views (Posner et a1. 1977) provide one possible answer. These techniques yield an enormous amount of data. If course revision was static and resources available, the tedious analysis demanded by these techniques would be worth the time expended. The analysis of one group of stu- dents would indicate the areas of the course needing improvement. Once improved the course revision would then hold for all future students. However, course revision is not totally static. Each term a new p0pulation of students enrolls in BS 202 and brings with them different propositional knowledge bases than did the previous students. Even if the propo- sitional knowledge to be learned does not change, the 10 introduction/reintroduction path followed through the content model may need to be changed to reflect the dif- ferent student knowledge bases. This dynamic aspect of instructional revision requires continual assessment of student knowledge. The availability of required resources prohibits large-scale continual usage of the previously developed techniques. Furthermore, continual assessment is needed to provide students with feedback on their learning. Therefore, a technique is needed that canAquickly generate a representation of student propositional knowledge, not only for course revision but for student feedback. Items have been written to test the BS 202 students' propositional knowledge of biology. These items are multiple choice items. At this time, What is obtained from the analysis of these items are test scores indicating "the student got 80 percent of the questions correct," or "the student does not know the correct answer to this ques- tion." Present analysis techniques do not provide an answer to the question, "What knowledge about biology does the student now possess?" or more specifically, "Given these item responses, what propositional knowledge of eco- logical succession can the student be inferred to possess?" For BS 202 a detailed representation of the subject matter to be learned is available. What is lacking is an expeditious assessment of propositional knowledge. Pre- viously developed techniques are costly in terms of time and present analysis of developed items does not provide 11 the information needed. Limitations of the Study Before any technique or method is applied on a large scale, it requires testing on a small scale. There- fore, this study addresses items from a subset of BS 202, specifically instruction about terrestrial succession. In the development of these items, it was realized that the propositional knowledge of succession does not stand alone. Attention was given to the use to be made of the propositional knowledge guided by a mode of task descrip- tions developed by Smith (1973). Each task description indicated what the student was to do with the propositional knowledge. The task description stated what information (input) was provided to the student and what form the response (output) was to take. The task did not specify the propositional knowledge to be used in an item, but did specify the type of proposition and the use to be made of that knowledge. Therefore, the information regarding stu- dent propositional knowledge was shaped by the task descriptions used as a basis for the items. A consideration for any assessment of student know- ledge is how does the student organize the information to generate an answer (Posner 1978). Although this is an important question, this study does not attempt to answer it. 12 The study will focus on the analysis of proposi- tional knowledge inherent in multiple choice items and responses. A.method for assessing such knowledge would be an important contribution to the investigation of how knowledge is processed in specified task environments. Objectives of the Study 1. Develop procedures to organize the propositional know- ledge tested in a test and apply them to a test on terrestrial succession 2. Develop procedures to analyze the propositional know- ledge implicitly asserted and interpret student item responses in terms of the underlying set of propositions presented in the test 3. Compare the information received from a subset of stu- dents' multiple choice item responses to that developed using the proposition generation technique for assess- ing propositional knowledge in order to assess the validity of the item analysis technique This study thus represents an attempt to develop a technique that answers the question, "What propositional knowledge does a student possess as indicated by his response to multiple choice items?" Overview of the Study The second chapter begins with a discussion of traditional knowledge measurement. Propositional knowledge assessment techniques are then reviewed followed by a 13 discussion of the instructional unit on which this study was based. Chapter three presents the procedures for organiz- ing the propositional knowledge tested in the items, analyzing the propositional knowledge implicitly asserted, and assessing the validity of the item analysis technique. Chapter four presents the results of applying the procedures following the order outlined above for chapter three. Chapter five provides a review and discussion of the results followed by a discussion of implications and recommendations for future research and practice. CHAPTER 11 REVIEW OF THE LITERATURE Overview This chapter provides the theoretical framework and background information for the study. The first sec- tion reviews knowledge measurement in the traditional sense. The second section considers propositional knowledge fol- lowed by a discussion of knowledge assessment techniques from cognitive science. After summarizing the theoretical framework, the context of the study is described in the final section. Measurement When a test of knowledge is administered and scored, the data received indicate the relation of one individual‘s knowledge to another individual's knowledge. The scores obtained for individuals are relative scores. The indi- viduals are placed on a continuum» Magnusson (1967) summarizes the measurement of psychological variables with: The position of an individual on a continuum is not given as an absolute score but as a relative score. . We can only compare individuals with each other. . we use the variation between individuals as the unit of measure (p. 18). 14 _—- 15 The previous statements about norm-referenced tests also apply to tests of knowledge classified as criterion- referenced tests. A criterion-referenced test compares individuals to some objectively defined behavior. Magnusson's summary can be paraphrased to state: With a criterion-referenced test, "we can only compare individuals with [the criterion] . . . we use the variation between individuals [and the criterion] as the unit of measure." A criterion—referenced test is used to relate an individ- ual's status to a well-defined behavior domain (Hambleton et a1. 1978). If that behavior domain is knowledge, then the individual's score is relative to that domain of know- ledge, usually defined as subject matter topics. In both norm-referenced and criterion-referenced tests, a score is received that relates the individual's knowledge to another individual or some objectively defined behavior. The scores are relative scores. Therefore, a score on a test of knowledge indicates a degree of know- ledge. Present test analysis provides information regard- ing the reliability of the instrument, how the items differentiate between "knowers" and "non-knowers," the difficulty of the items, and the mean score of individuals responding to the set of items. Further analysis provides information regarding the range of scores, median score, and various correlation coefficients indicating the rela— tion of the items to the total test continuum. From these 16 data, inferences are made about the knowledge an individual or group possesses. The information received from the analysis of a test is useful and has been discussed in several texts (Ebel 1965; Magnusson 1967; Butcher 1968; Nedelsky 1965). The point is: the inferences made from the data are invariably inferences of the relative amount of knowledge possessed. The inferences are not about the specific knowledge possessed. For example, a student responds to a set of test items and receives a score of 80 percent. The inference can be made that the student knows 80 percent of the knowledge tested by the test. The score actually indi- cates the student correctly answered 80 percent of the items. What is the 80 percent the student knows? What is the 20 percent he does not know? Does the missed 20 percent indicate a lack of knowledge or does the student have mistaken or alternate conceptions of the subject matter? Does the student know twice as much as a student who received 40 percent? The score does not provide an answer, nor does the item analysis. The item analysis indicates the number of respon- dents selecting the correct response and the number select- ing the incorrect response. How easy or difficult the items are for a particular group of students is also indicated (Ebel 1965). Recent investigators have offered test analysis techniques that begin to address the specific knowledge 17 being tested or the knowledge a student indicates by his response to test items. Klopfer (1971) suggests a tech- nique to organize information presented in science programs and tests. The technique is based on the programs' content, expressed as topics, and Bloom's description of desired stu- dent behaviors. He organizes the information with a matrix constructed by listing the desired student behaviors above the matrix columns and the topics along the matrix rows. With this matrix, an individual can mark a cell that repre- sents the occurrence of a particular topic and the student behavior addressed by a particular program. Klopfer further refined this matrix for analysis of a particular test and described a diagram.which identifies the particular teaching objective and subject matter element addressed by a set of questions comprising a test. KlOpfer (1971) claims this diagram reflects "as closely as possible the content hierarchy of its unit . . ." and is "a useful tool for diagnosing a student's difficulty" (p. 630). Schmidt et a1. (1978) have developed a classifica- tion system for various curricular materials in the field of mathematics. The classification system is based on three factors: (1) mode of presentation (how information is pre- sented), (2) nature of materials (mathematical terms used or type of numbers), and (3) operations (required cognitive processes). Porter (1978) states, "The intersection of these three dimensions results in 468 categories; each category represents a topic that a teacher may elect to 18 teach or not to teach" (p. 5). This system has been applied to four standardized fourth-grade mathematics tests. Both of the preceding alternatives to traditional test analysis for science or mathematics education can be used to describe the coverage of a test beyond the usual results of test analysis. In both techniques a clearer picture of the content of a test is obtained. Klopfer's technique indicates the subject matter element addressed by the test item and the behavior exhibited as a result of knowledge of that element. The classification system for mathematics considers the processes involved in producing a particular behavior, the manner by which the question was presented, and the nature of the presented materials. Neither technique provides, as a product, an explicit indi- cation of the knowledge being tested. The product is an indication of the topic addressed by the items in a test and does not provide answers to questions regarding the knowledge comprising the topic. Of the two, Klopfer's technique does address student responses, but it is done to test whether the student was correct. An item on a test, however, is an opportunity for a student to provide evidence of understanding a particular bit of knowledge. As Ebel (1965) points out, "What we test, beyond the student's ability to understand the lan- guage used in the test item, is his knowledge of the proposition that makes one answer correct and others incor— rect" (p. 43). If items test "knowledge of propositions," 19 then it ought to be possible to analyze items in order to summarize the propositions a student selects. Present analysis only indicates the proportion of agreement between the respondent's choice of propositions and those in a model of knowledge or, at best, an indication of the topics addressed; it does not indicate what knowledge the respon- dent is prepared to assert. Propositional Knowledge Propositional knowledge is defined as knowledge of concepts and their relations to other concepts. Posner and Rudnitsky (1978) defined a proposition as "a statement that asserts something. The assertion contains two or more con- cepts and some specified relationship between them" (p. 165). Cohen and Nagel (1934) define a proposition "as anything which can be said to be true or false" (p. 27). Therefore, a proposition also has truth value. Cohen and Nagel (1934) have provided an expanded definition of propositions (assertions with a specified relationship and a truth value). They qualify their defi- nition of a proposition by describing what a proposition is not. The following description of a proposition is based on Cohen and Nagel (1934). A proposition is not the same as the sentence that states it, but that which is referred or symbolized. Sen- tences contain a group of words (symbols) which are physical objects, distinct from what they represent. Sentences are 20 on the surface and are not true or false. The truth value of a sentence is predicated by the proposition signified. Although symbols are required to express a proposi- tion, not every combination of symbols expresses a proposi- " does not represent tion. "Niche specialization food webs a proposition; the combination of symbols is only words. "As niche specialization increases, the food web complexity increases" does represent a proposition. The symbols refer to a relation that has a truth value and communicates a conclusion about characteristics of terrestrial succession. Wishes, questions, and commands imply propositions; they do not assert propositions. Questioning the truth value of a wish, question, or command is meaningless. Declarations are propositions as long as they have truth value and the truth value is dependent on its quali- fication. For example, "During succession, as stratifica- tion increases, the amount of light at fifteen feet decreases" is a declaration which implies a proposition which can be said to be true or false. As stated, both values of truth might be correct because it is not known if the referent is forest succession or prairie succession. In forest succession, the statement is true; in prairie succession, the statement is false. Therefore, the truth value is dependent on the qualification, i.e., the type of succession. In general, our statements do not contain all qualifiers. The statement may be completed so that the 21 proposition is sometimes true and sometimes false. Some qualifiers are understood and others are not thought of. Therefore, a proposition may not be universally true. In the preceding example, if an individual was discussing succession and all previous examples referred to forest succession, the assumption would be made that the individ- ual was referring to forest succession making the qualifier unnecessary. If the referent was something else, then a qualifier is required. Although a proposition has truth value, that does not imply knowledge of the value. For example, "A heart attack is preventable" is a proposition, but it is not known if the statement is true. It is assumed to be true if certain procedures are followed, but it is an assumption and truth is not actually known. A problem arises when discussing the knowledge of truth and statements, such as "That meter is one hundred centimeters." This is a statement of resolution. Although it can be said the statement is true, the statement is true only because it has been agreed that a meter is one hundred centimeters by definition. Definitions are not propositions. They are statements that have been resolved to be true. In the statement, it is not known if the meter actually refers to a unit of measure or an instrument found in a parking lot. The assumption is made that a definition is being offered, and there is no method to establish the actual truth value of the statement. 22 Propositions are not the same as the mental acts required to think them. They are as distinguishable from the acts of the mind as they are distinguishable from the sentences that state them. Propositions are not to be identified with concrete objects, events, or things. Cohen and Nagel (1934) state: For propositions are at most only the abstract and selected relations between things. When we affirm or deny the proposition The moon is nearer to the earth than the sun, neither—Ehe moon alone, nor the earth, nor the sun, nor the spatial distance between them is the proposition. The proposition is the relation asserted to hold between them. The relations which are the objects of our thought are elements or aspects of actual, concrete situations. These aspects, while not spatially and temporally separable from other characters in the situation, are distinguishable in meaning. That is why sense experience never yields knowledge without a reflective analysis of what it is we are experiencing. For knowledge is of propositions. And propositions can be known only by di§criminating within some situations relations between abstract features found therein (p. 29). With these qualifications, based on Cohen and Nagel, propositions are defined as assertions containing concepts (symbols or referents) and their relations to other concepts and the relationships have truth value (true or false). Given what a proposition is and that items test knowledge of propositions, propositions ought to constitute a good source of test items. However, texts and reference works do not lend themselves to the generation of proposi- tions for item usage. As Ebel (1965) points out, many sources provide indefinite and tentative sentences. Much of what is written is heavily dependent on context for 23 meaning. The propositions in texts are rarely stated succinctly.1 However, a proposition or set of pr0positions underlie the words that one writes or speaks. When a person engages in discourse, there must be some type of relationship being conveyed. The difficulty arises when one uses several sentences in discourse to imply a proposi- tion. Recently, cognitive psychologists, linguists, and people working in artificial intelligence have combined efforts to study the organization of knowledge for storage, retrieval, and utilization. The apparent high degree of organization has led these investigators (cognitive scien- tists) to capitalize on the Kantian notion of "schema" (Posner 1978). To the cognitive scientist, schemata are structures of interrelated concepts (propositions). To specify the concepts without their interrelationships is not a profitable experience. For example, in reviewing the work of Odum (1971) the following proposition was derived: As species diver- sity increases, the food web complexity increases. If this proposition was represented without the relationship, it would appear as: species diversity food web complexity 1This problem has been analyzed from a different perspective by Strike and Posner in "Epistemological Per- spectives on Conceptions of Curriculum Organization and Learning," Review of Research in Education 4, ed. L. S. Shulman (Itasca, Illinois: F. E. Peacock, 1976). 24 The representation with the relation specified be- comes: increase species diversity (——-) food web complexity Here the labeled arrow indicates that as one increases so does the other. As Cohen and Nagel (1934) indicated, prop- ositions are relations between concepts, not just the concepts. By specifying the relation of concepts, meaning is obtained from the representation of the proposition. Posner (1978) has suggested that the tools of cog- nitive science be employed for representing the knowledge embedded in subject matter sources. One particular tool is a semantic network. The network specifies the set of propo- sitions that represent the knowledge of a particular field. The networks can be employed as instructional tools, orga- nizing the information to be learned during instruction. Knowledge Assessment Techniques from Cagnitive Science Recent investigators (Preece 1976; Shavelson 1974; Posner et a1. 1977) have developed techniques to assess the knowledge structures of an individual or group. These tech- niques include word association, tree construction, graph construction, free recall, proposition generation, proposi- tional analysis of clinical interviews, and problem-solving tasks. The techniques are used to construct and compare models of student knowledge to subject matter models. 25 Word association is perhaps the oldest technique. A person is given a word and asked to respond with the first word that comes to mind or all the words he can recall (Shavelson 1974). The words recalled first are assumed to be located in close proximity to the stimulus word. This technique is called controlled word association by Preece (1976). Tree construction involves providing subjects with a list of concepts. The subjects are directed to select the two most closely related concepts and write them in the middle of a page, connect them with a line, and number the line. Then the subject returns to the list and selects another concept closely related to the original pair, writes it down, and connects it with a numbered line to the origi- nal pair. If concepts remain that are more closely related to each other than to the original pair, a new tree may be started. The subjects continue until all concepts have been used (Posner et a1. 1977). Graph construction is similar to tree construction. The subject is given an alphabetical list of words on gummed labels. The subject selects a pair of closely related words and attaches them on a piece of paper close together. The subject returns to the list, selects another word, and attaches it either close to the original pair forming a cluster or starts a new set. The procedure is repeated until all labels have been used, then lines are drawn connecting related labels. The spatial arrangement 26 is taken to suggest how the subject sees the relation of concepts (Shavelson 1974; Preece 1976). Free recall (Preece 1976) is a variation of the word association. In this situation, the stimulus word is listed on a page ten times and given to the subject. The subject is to look at the word, write down the word that comes to mind and repeat the procedure ten times in one minute. As with the word association, the order of response indicates the proximity of concepts in the subject's cogni- tive structure. In the techniques described thus far to assess know- ledge structures, the relationship between connected words or concepts is not specified or required. The product of the described technique is a knowledge structure built on the association of concepts. If "knowledge is of proposi- tions," (Cohen and Nagel 1934, p. 29) then the assessment of knowledge should result in propositions which comprise the knowledge structure, not just a group of concepts. Therefore, these techniques do not assess propositions. There is no relation specified; therefore, a truth value cannot be assigned. As was pointed out in the previous discussion of propositions and will be in the discussion of other knowledge assessment techniques regarding propo- sitions, the specification of the relation provides a clearer interpretation of the conceptual relations an indi- vidual or group asserts. Strike and Posner (1976) have questioned the use of associative techniques by stating: 27 there is a problem in using associationistic or quasiassociationistic techniques to study cognitive structure, especially when cognitive structure 13 supposed to result from learning content structure. This problem is suggested by the cryptic remark of Deese (1962) that associative meaning is not logical, for the structure of a discipline, if it is anything, is logical (p. 125). Stewart (1978) has suggested a modification to the tree and graph construction techniques. The lines between the concepts can be labeled or numbered and a statement about each line written. This modification does generate relations and a set of propositions that can be evaluated. Stewart's suggestion is an important and welcome addition to the assessment of knowledge prepositions. Three other techniques have also been used to gen- erate propositions. The first provides the subject with a pair of concepts and directs him to write all he knows about the two. The second requires the subject to write one sentence about the relation of the two concepts. The third provides a statement of a proposition to the subject and directs him to indicate the truth value of the proposi- tion and write an explanation of the indicated truth value (Posner et a1. 1977). These techniques are similar to short essay or true-false items used in classroom testing. The modification of the tree and graph construc- tions suggested by Stewart (1978) and the proposition generation techniques described by Posner et al. (1977) provide the investigator with a set of sentences. The sentences contain related concepts rather than associated 28 concepts and imply an underlying proposition. For example, by associative techniques one could receive the following from two students: a) niche specialization food web complexity, b) niche specialization food web complexity. The students know the relation between the two concepts. But do they? With the relations specified, as suggested, one could receive: is the same as a) niche specialization food web complexity, increases along with b) niche specialization food web complexity. The conclusion being student "a" is incorrect and does not know the relation of the two concepts. To "a" the con- cepts are equivalent, which is a misconception of the underlying proposition. Student "b" asserts the correct relation of the two concepts. Without the relation speci- fied, the investigator would not be aware of "a's" misconception. Posner et a1. (1977) have described the procedures used to analyze the data from the described associative and relational techniques. The data obtained are trans— formed into distance matrices. This device is a record of the number of lines between the various concepts and is to indicate how far apart the concepts are within the cogni- tive structure of the subject. Data obtained by association are readily transformed into distance matrices. Data con- taining relations require the investigator to complete 29 incomplete sentences, diagram the sentences, and convert the diagrams into distance matrices. The distance matrices from each technique are transformed and compared to each other or the content model by either multidimensional scaling, Waerns Technique, or hierarchial clustering (Posner et a1. 1977). The comparison is done to establish the validity of assessment (Posner et a1. 1977). But often the compari- son is between a relational technique and an already suspect association technique. Therefore, Posner et al. (1977) appear to compare data they consider to be highly questionable. They have quantified the data and lost the rich relational information. This also occurs when the relational data are compared to other relational data described thus far using the indicated quantitative methods. If propositions are being assessed, then the assessment and comparisons need to include the relation. If the relation is omitted or not readily available, then by definition a proposition is not being assessed. Propositional analysis of clinical interviews is a method to analyze discourse. The collection of data is a modification of Piaget's clinical interview method. The subject is asked to describe something. With different probes, the investigator questions the subject for further explanation of the statements given. The data analysis involves transforming a transcrip- tion of the interview tapes into a propositional format 30 (each proposition formed is an independent unit of dis- course). The propositions are numbered and arranged for comparison of pre- and post-instructional interviews. Irrelevant propositions are discarded. The data are trans- formed into a conceptual map (Pines 1977). Propositional analysis produces a large amount of data. The analysis product is qualitative rather than quantitative and is highly desirable, as it represents a detailed model of the students' propositional knowledge by retaining the asserted relations. It enables the investigator to compare individuals among themselves and others. However, with the large amount of data obtained, the technique does not lend itself to studying group data (Posner et a1. 1977). Atkin (1977) has designed problem-solving tasks for use after instruction. The tasks can be administered by a pencil-and-paper test or by the clinical interview method. The problems may require application (near trans- fer) or interpretation (far transfer). Success on near transfer problems (applications) is taken to indicate con— gruence between the subject's cognitive structure and the content structure (Atkin 1977). This technique is similar to the classroom testing presently used. (High success indicates high knowledge.) Although Atkin's problem-solving tasks are similar to normal classroom testing, her analysis is based on an information-processing model for the items designed to 31 "test use of knowledge following instruction" (Posner et a1. 1977, p. 29) rather than assessing the knowledge being used. The technique describes what is happening to exist- ing structures as opposed to what the existing structures are. Greeno (1976) refers to this as assessing cognitive processes as opposed to cognitive structures. The focus of the study is propositional knowledge and the assessment of propositional knowledge. The nature of an individual's propositional knowledge can only be determined when a situation arises requiring statements based on that knowledge be used. When that situation occurs, then the subject's formation of the statement (cog- nitive processes) or the formed statement (cognitive structures) can be analyzed. The former analysis is to determine how the subject organized the underlying propo- sitions to form the statement as Atkin (1977) did. The latter is to determine what underlying proposition is the basis for the statement the subject formed as Pines (1977) did. Both analyses are valuable; however, this study does not address the process of organizing propositions. The study focuses on the underlying pr0positions indicated by the statement. Knowing "what" was the basis that formed the statement seems prerequisite to "how" the statement was formed. 32 Summary Thus far, it has been indicated that usual test analysis does not provide sufficient information regarding the specific propositional knowledge held by students even though, as Ebel (1965) pointed out, tests of knowledge test knowledge of a set of propositions whose truth value is judged by the subjects. On the other hand, the literature of propositional knowledge assessment techniques indicates more adequate descriptions of propositional knowledge are possible. However, the present methods for analyzing propositional knowledge are time consuming, tedious, and not amenable to large-group data. The probability of their being employed with large groups on a regular basis is very low. Test theory indicates traditional tests are testing knowledge of propositions. Therefore, the major aim of this study is to develop and test an economical and effi- cient method to analyze an existing test and generate a set of propositions which represent the students' propositional knowledge at a point in time. Background of the Study The propositions of interest come from the study of terrestrial succession. As indicated in chapter one, an instructional improvement program has been initiated in Biological Science for Elementary Teachers (BS 202) to orga- nize ecological facts, concepts, and conceptual relations into propositions and the corresponding semantic networks. 33 The networks form a model of the intended instructional content and learning outcomes. Part of the instructional improvement program focused on the succession instructional unit (the unit of interest for this study). The following discussion describes the development of the succession unit and provides the context for the focus of this study-- propositional knowledge assessment. A proposition has been defined as concepts and their relation to other concepts, and the relationship has a truth value (true or false). The concepts and their relations for this unit were derived from an analysis of literature about succession (Whittaker 1974; Horn 1974; Drury and Nisbet 1973; Marks and Borman 1972; Odum 1971; Loucks 1970; Kormondy 1969; Odum 1969; Harper 1967). As the literature was reviewed, it became apparent that the process of succession was not as simple or as definite as presented in the texts for high school or intro- ductory college biology courses (Otto and Towle 1977; Keeton 1973). Several questions were raised regarding the process, the geographic extent of the process, and if and when the process was completed. The answers provided for these questions were indicative of the authors' discipline (mathematics, zoology, botany, or geology) and method of inquiry. The institutional constraints, the amount of material to be covered in an introductory biology course, and the course goals preclude including all these issues. 34 Therefore, the necessary knowledge to understand the dynamics of an ecosystem and the interactions that are exemplified by the process of succession became the focal point of the analysis. In other words, what concepts and relations is it necessary for an individual to possess in order to understand the dynamics and the interactions of the process of succession? When this consideration was perused in the litera- ture, it was found that although the authors were studying different types of succession or debating theoretical explanations or both, their discussions were based on the same concepts and relations. The following concepts2 were derived and used for the instructional unit in this study: (1) food web complexity, (2) niche specialization, (3) stability, (4) species diversity, (5) stratification, (6) amount of light, (7) soil texture, (8) soil organic content, (9) amount of litter, (10) succession, (ll) pioneer stage, (12) middle stage, and (13) climax stage. Succession is a natural process which occurs over a period of time. Concepts 1-9 change as the process of succession proceeds; therefore, they are considered to be variables whose qualitative value (low, medium, high) 2(The concepts' production, biotic potential, life cycle, organism size, species equitability were also derived. These concepts were discarded after piloting the unit. It was found that a large amount of time to develop their relations to succession was required and the complex- ities of the concept seemed to be beyond the grasp of the BS 202 population in general.) 35 changes during the process of succession. Concepts 1-4 are studied regardless of the type of succession being investigated (aquatic, bog, sand dune, old field, etc.). Concepts 1-3 are considered as inferential concepts, i.e., the value is not easily observable or quantifiable. Con- cepts 4-9 are considered directly observable and quantifi- able, i.e., an individual can use their senses or easily measure instances to quantify and obtain corresponding values of the concepts. Concepts 5-9 are studied when the example of succession is terrestrial, which is the case for this unit. Concepts 11-13 represent stages or periods in time during the process of succession. Each stage can be characterized by the values of the variable in question. Each stage also represents a value of succession which is a variable when time is considered. The types of variable relations in the literature were those which focused on the value of a variable during a stage of succession, the change of a variable during succession, and the relations between variables or their values during natural succession or after some perturba- tion. Therefore, when the concepts were organized to form a semantic network for this unit, the relations of concepts (propositions) were classified in four categories: variable-stage relations (the value of a variable during a stage); variable covariation (the change in a variable as the process occurred); variable correlation (the change in a variable related to the change in another variable); and 36 value—value correspondence (the correspondence of the value of a variable and the value of another variable at a period of time). Part of the resulting semantic network is dis- played on the following page. (The total network is displayed in Appendix A.) Using the network displayed on the following page, the following propositions can be read to provide examples of the four relation categories: Variable-stage relation. Climax3 stage of succes- sion has3 high3 species diversity. For this type of relation to be true, the numbers in superscript need to be the same. Variable covariation. As succession proceeds, the niche specialization increases. Variable correlation. As species diversity increases (during succession), the niche specialization increases. This relation states there is a direct correla- tion between niche specialization and species diversity during the process of succession. In the diagram, -1 indi- cates that the reciprocal or inverse relation is also true, i.e., if x increases, y increases; and if x decreases, y decreases. (This does not necessarily mean 1:1 correspon- dence or causality, which is the subject of much debate.) Value-value correspondence. Lowl species diversity occurs with low1 niche specialization. Again, the numbers in superscript must be the same. The relation is stating when one value of a variable occurs, there will be a 37 Low . 2 1 ' pecialization Ami—i e‘ "\ ”'9 3 - <3” 9 Pioneer Stage 63 4a e 29 < c 0 Middle Stage Succession Climax Stage Low pecies Diversity m 3 ‘HQ 3 Figure 2. Representation of a portion of the suc- cession semantic network Key : -1 indicates reciprocal relation is true proceeds and increase = As concept X does this, increase increase concept Y does this enclose an observable, measurable, or inferential value of a variable concept enclose concepts has X indicates the value of a concept during a corre- 6 ’ sponding stage of succession « >emelose stage designations which are values of the succession variable Numbers within boxes indicate variable value which correspond to other variable values. (This is done to simplify the diagram.) 38 corresponding value for a related variable. It should be noted that the relations formed with the variable ”amount of light" are the only relations whose direction of change is different. During succession the amount of light decreases at certain levels. The value of the amount of light was determined at the levels one foot, five feet, and fifteen feet and assigned values of low, medium, and high. Relational types one and four are both value-value relations, and types two and three are both correlational relations. The difference is whether the relationship addresses a stage of succession or succession itself. Also, relational types three and four are correlational depending on whether a value is to be correlated or a variable is to be correlated. After the semantic network of propositions was formed, the delivery of instruction became more important. How can the propositions be organized for instruction? What will the students do to learn the propositions? How can the instructional inputs and outcomes be described and .organized to facilitate delivery of instruction, monitor instruction, and later assessment of the developed unit? A possible answer could have come from the use of behavioral objectives, which were not used. If written to specify each proposition contained in a unit of thirteen concepts and four types of relations, the number of behavioral objectives would be large and cumbersome. If 39 the objectives were written to specify only the relations as classified, then the objectives would be too general to express instructional intentions. Behavioral objectives only indicate the result of knowledge. The interest here was to have additional information regarding the knowledge that generated the result. To describe the situation that was to generate the knowledge (propositions) the framework described by Smith (1973) was used. Smith expresses instructional intentions (what is done to learn something or what one should be able to do as a result of learning) as tasks which indicate the inputs and outputs. The students are provided information (givens) and are to output information (requireds). The descriptions he uses are content-free but reflect the nature of content to which they apply. He has described sets of tasks for three types of concept networks: variable-value, class-member, and part-whole. The task descriptions are then crossed on a matrix with the concepts which are to be learned as a result of performing the tasks. This framework is advantageous°because it succinctly specifies the information (concepts) involved, the infor- mation provided to the student, and the information the student is to provide. For example, a task can be described at a generic "analytic" level: Given: A correlational rule (a relation between variables) and specification of a change in one variable; 40 Required: A prediction of the results of the modification on the other variable within the correlational rule. The concepts involved in this task are variables. At a systemic level, the preceding task and con- cepts could be used in physics instruction as: Given: Newton's Second Law, F = ma, and a speci- fied change in F, m, or a; Required: To predict the consequence of the speci- fied modification on another one of the variables in Newton's Second Law. The concepts are force, mass, and acceleration. Since the concepts of interest for this unit are characterized as variables, Smith's variable-value task descriptions were used to describe the situation for intended learning. Not only can the task descriptions be used to describe the intended situation for learning con- cepts, but the descriptions can also be viewed as describing the intended situation for learning relations of concepts, i.e., propositions. In the preceding example, the students are to learn not only about force, mass, and acceleration but the proposition which entails the relation of the three concepts--Newton's Second Law. The relations of interest are variable-stage, variable covariation, variable correlation, and value-value correspondence. Task descriptions were generated for the intended learning activities for each type relation similar to those used by Smith. Figure 3 lists the type of 41 Figure 3. Chart representing translation of rela- tion into intended learning situation (a task). Stages of succession are considered separately; characteristic con- cepts are considered variables; values are relative observa- tions of variables obtained directly or inferred to be present. Relation of Interest "A Analytic Task Description Systemic Task Description Example Input/Output Variable-stage Variable covariation Variable correlation Value—value correspondence Variable correlation Given: Stage value, variable name, and observation/measurement procedure Required: The value of the variable correspond— ing to the given stage value Given: The values of variables across ele- ments of a named variable Required: A rule relating the change in variables across the named variable Given: A set of vari- ables that change across a named variable Required: A correlation rule for the variables Given: The value of a variable at a point in time and a variable name Required: A rule relating the values of the variables Given: A correlational rule Required: A prediction of the results of modi- fication of a variable within the correlational rule Task 1 Given: A definition and description of succession, a list of defined characteris— tics (stages and vari- ables) and observation/ measurement procedures Required: A description of general characteris- tics during designated stages Task 2 Given: Tabular descrip- tion of stages and suc- cession and characteris— tics Required: A rule relat- ing changes in charac- teristics to stages of succession Task 31 Given: A set of charac- teristics which change during succession Required: A rule relat- ing the characteristics to each other Given: A set of charac- teristics and their values during succession Required: A rule relat- ing the values of the characteristics to each other Task 42 Given: Rule describing relation of character- istics to other charac- teristics in a set across stages of succession Required: Prediction of consequences of the modi- fication of a particular characteristic to a set of characteristics and successional sequence 1: Count the number of species and char- acterize species diversity in the climax stage 0: Climax stage of succession has high species diversity I: From the table, what is the relation between succession and niche specializa- tion? 0: As succession proceeds, the niche specialization increase I: Characterize the relation of niche specialization and species diversity during succession 0: As species diver- sity increases, the niche specialization increases I: If species diver- sity is high, what is the value of niche specialization? 0: High species diversity occurs with high niche specializa- tion I: If you decrease species diversity, what happens to niche specialization? 0: As species diver— sity decreases niche specialization decreases and succes— sion reverses 1Tasks three and fo correlations and value-val ur can apply to both because both require a cor , variable as correspondence relations relational rule as output. 2 Task four represents a prediction task. 42 relations to be formed in a proposition, an analytic state- ment of the intended instructional task, the systemic statement of the task, and an example input-output for the task situation. Figure 3 represents the task descriptions used to characterize the types of relations and the situation for a particular proposition to be generated. With the tasks describing the type of relation, the concepts within the proposition could be identified from the semantic network and organized in a concept-task framework as described by Smith (see Figure 4). The concepts to be related are indicated across the top of Figure 4. The type of relations to be formed with the concepts are indicated by their respective task descriptions listed along the left side of Figure 4. As defined, each of the task descriptions indicates a relation between concepts. Completion of the tasks requires decla- rations of the appropriate relationships between the concepts. In other words, each task requires production of a proposition from the student. With the concepts and relations organized, instruc- tional activities were developed for the students to carry out the tasks intended for their learning of the concepts and relations. The activities revolved around completion of worksheets used to record observations or measurements of the values of variables, writing or illustrating relations or both, and using the worksheets to make 43 of Light Amount of Light Amount of Light 15' ()hsurva t inn Specialization Stratification Snil Organic Amount of Light All Factors Content All Stages Nee-d More Fund Web Nltiht‘ Species Diversity Amount uf Litter Soil Texture Pioneer Middle Climax Stability Succession Amount I l i i I I i I I l i Task 1: Given: a definition and description of succession, a list of defined charac- teristics and types of organisms to observe Required: a description of organisms' characteris- tics and changes in gen— eral characteristics 32:91:: _d_e_:isn.-I_t_c_d_st_xl_ss:z _ _ Zfd-S—k 22 --'———-fi—~——--*--—~-~‘—-~-—‘— Given: tabular description of stages of succession and characteristics Required: a rule relating changes in characteristics t0_stages of succession 47 Task 3: Given: a set of characteris- tics which change during succession Required: rule describing relation of characteris- tics to other character- istics in a set across stages of sutccssion Task 4: Given: rule describing rela- tion of charActeristics to other characteristics in a set across stages of suc- cession Required: prediction of the consequences of modifica- tion of particular charac- teristics to set of characteristics and succes- sional sequences . Figure 4. Task-concept matrix for an instructional un1t on terrestrial succession. The concepts to be used in formation of propositions are listed across the top. The form of the relation within the proposition is provided by the task descriptions listed at the left. 44 predictions of the consequences of variable modification. The propositions to be taught and learned for each task are listed in appendix B. After the activities had been developed, the focus was turned to assessing what the students learned and writing assessment items. To write assessment items the task descriptions were used. The descriptions indicate what information the question would provide to the student and what information would constitute an answer. The con- cepts and relations provided what was to be used as infor- mation. Three multiple choice items per task were used in this study, together with an essay item based on tasks three and four. The items used are reproduced in appendix C. The items are viewed as requiring the student to select from an array of statements based on underlying propositions. The student indicates which of the array of statements he believes to be true with respect to the problem (task) presented. Therefore, if the student is providing an indication of his knowledge and it is known what he was to learn, then the analysis of his responses ought to indicate the underlying propositional knowledge the student is indicating he believes to be true. Therefore, this study focused on the need for a procedure to organize the potential alternative assertions provided by the test and the information provided by the student responses to develop a representation of the 45 student's propositional knowledge. To assess the validity of the representation derived from the test, a cross—check was necessary. The validity assessment was carried out using the propositional analysis procedures described by Pines (1977). Validity is supported if equivalent information is received from the analysis of multiple choice items developed for this study and analysis of an essay item following the procedures described by Pines (1977). CHAPTER III PROCEDURES FOR TEST ANALYSIS Overview In chapter two, describing the background of the study, a set of propositions were derived from an analysis of written material about terrestrial succession. A set of tasks were selected which described the use to be made of the propositions. These tasks also described the frame- work of assessment items. This chapter describes the procedures used to address the objectives of the study. Sections in this chapter describe procedures to identify and organize the propositional knowledge tested by the items, organize stu- dent responses, interpret and analyze the responses in terms of propositional knowledge asserted, and validate the information received from multiple choice responses using Pines's propositional analysis technique for assessing propositional knowledge. 46 47 Or anizing the Propositional Enowledge within Multiple .Choice Items Derivation of Propositions Chapter two has provided a lengthy definition of a proposition. With the stated qualifications in mind, a proposition is an assertion containing concepts and their relations to other concepts and the relationships have truth value. A multiple choice item.can be viewed as offering a set of propositions to the student. From this set of propositions the student is required to choose a subset (often one). Thus, the student can be viewed as asserting a proposition which indicates his belief at a particular point in time. Propositions were derived from multiple choice items by combining the stem concept and its relation with the relevant concept(s) of each option. For example: Item: In terrestrial succession, the pioneer stage has 1. high food web complexity 2. low species diversity 3. low amount of light at one foot 4. high niche specialization 5. high stability Propositions: (T = True, F = False) 48 T The pioneer stage of terrestrial succession has high food web complexity. T The pioneer stage of terrestrial succession has low species diversity. F The pioneer stage of terrestrial succession has low amount of light at one foot. F The pioneer stage of terrestrial succession has high niche specialization. F The pioneer stage of terrestrial succession has high stability. Treatment of each item in this manner generates a list of propositions for the test. By listing the proposi- tions the student selects, evidence of a student's beliefs at a point in time is obtained. In principle, generating a list is not difficult. However, from a practical standpoint the length of such a list makes it difficult to handle. A matrix has been found to be a useful tool for representing the proposition found in a test in a fairly concise way. Table 1 represents the proposition matrix used in this study. The matrix cells represent all possible propo- sitions within the unit. Empty cells represent propositions which could be used as a basis for further item development or pairs of concepts for which there are no important relationships defined. The numbers within the cells indi- cate the items and foils where the tested proposition occurred. Asterisks indicate the correct response. The 49 TABLE 1 PROPOSITION MATRIX FOR TERRESTRIAL SUCCESSION WITH TESTED PROPOSITIONS INDICATED 7 7 cz7ue>turoc & S 7 5 7 .mLer ”7.7-7. n 7A n t I 77.».vzm -< 4.. . _ _ 1* «Luzum :< . 7 7. . I .7 I- I . I. I I I II I II I- I+ I- “zu74 A 7 Tc firs}. U 7 7 . 7 I’D...) C.7.7..Ia7. If. ._ _ MIN-W 3:75;.“ 7 ‘ 7 7 7 . _ . 7 III Ila III IO.I I II I .0 .0 I.I 4 nI.I| IHI I] . I III I I 7 126.77 H 7. _ 7 I I. I . o ..- II. I. i III. I I II I. I II. III;..yII . I .I. 7 7 7 7 7 2).). -777 _ I. 7 2 7 7 _ . . .v—Tfl‘, I.“ 7. 7 7 7 IIII V I? I... II II I4 II . III .I v I I 7 u 7 t 7 7 3 7 .J 7 _ 7 . . 7 7 , . I. 7 1.. . _;:::_a 1 7 u 7 7 ,I I I -I- I I I .. -I 7.I|I- I II kI II II7 I I I - 7 7 .r_ “32“— «C uCS. w}. 7 _ 7 7I,:- : --I .7 I. - I 7 I III- I. I 7 7 _ .7... 2%.; 7 . 7 .__7 3:...Ea7 7 7 I . I II. 7 I .II II 7 .II- I 7 . 7 I I 7 7 a I P _ 7 .7 3.37 97 n7... 2.7 _ 7 “a ccz7e< 7 I l I IF III I II II .I III . III I---.,II III, .I 7 7 7 4 7 2 7 7 I}. _ _ surging. _m;v. U _ III. v ...II. I I II . I .¢ . I .:.I.II III. . I. III III II I 7 7 7 7 7 5 u:;.—::7 .J . 7 7 7“ 7 “7:73;: Chm. 9 9 7 . U .IIIII IQII .II II . III III I I7I IIIIF 6 I I 7 7 t;.z_7 7 7 J.J 7 v? aging... 7 _ .H H” 7 I a If. I I- I II II- II III .i I w 5 7 7 7 v/. I _7 _ 7 _ I )6 11 I a . 7:72.77, 72.25 7. 7. 7. m. 8 7 .5... 7. ,II I a” .II III III III II I 11. A . 7 . A 1., .. I t >u—...L>_S 1235 u I. .1) a. r) 2 V — n o p I ‘II \I 1. Ir. iv. “W ¢.v_ .5... I. 11 1! ~I ./ 7 I I (7 I. (\ 7 , I- _I I , ,II.IIII.II I s 2 I _ I I I 7-- I772: . _ . .l 9 _ _ . . . . I .l M . cc.us~_~adc;_v 7 I 2 3 k 7 L _ 7 27.7... 7.7.7.... 7.. 7 7 . 31315.6 7 . J J 1... £33 v.35..— _ «IV. m 8 7 II- 4 II+I II. - II. ,III I I II. I _ 7 V. J 7 f. ‘ _ Y. . l.- f '- b 1 .1 r. I n 0 0| 0 v e I I. a .1 f U C t e I r) H t a m c 5 .1 «A t I I n r. I l l e I r I r 7 t t n e n e u 7 n t 7 n t d m: h c I C c a a u t l I. t I t u h u :u 0 r. C c l 7 P. u. r c U L I B n 4.. x a on 0 K o.u I P P.I L M 1 a r o n e m I .,1 F ( N S S D _ S L S 0 C S T A L M L I I I I | l I Observation; stages Amount of LIght 15' Pioneer Hiddlc Climax Stability qucceuslon ARV-UH: . LIgnt All ‘ All FALtors Need More 50 items used as a basis for the matrix are listed in appendix C. To focus on the propositions tested, a matrix was formed to indicate only the test propositions (Table 2). ,In this matrix each row corresponds to one item. The con- cepts at the beginning of each row represent the concept found within the stem of the item. The item numbers are indicated on the left margin. The concepts above the columns represent those which could be included in an item foil. The last three columns designate distractor phrases which do not form a proposition with the stem concept. The concepts used in item foils are indicated by foil numbers within the cell. The cells are divided by truth value hori- zontally (upper is true, lower is false) and divided by information source vertically (right is the test-item foil, left is for recording student response). Letter codes are included to represent the relations within the tested propositions. The code definitions are listed on Table 2 and explained in Figure 5. Figure 6 is an example of the formation of a row in the test-item proposition matrix. Item: In an early successional stage, we would find 1. low amount of light at one foot herb layer 2. simple food webs 3. high species diversity 4. low amount of light at the fifteen foot level 51 TABLE 2 TEST-ITEM PROPOSITION MATRIX Key: Relation Symbols h = has d = decrease hi = high n = narrow s = simple I = increase 1 = low - = negation of relation term li = little I. = therefore pr = proceeds r = reverse mo = moderate du = during stage 5M .s 73. N ’E 2‘ c. 7 ; L :2 7 ‘7’ ‘ -;» ‘5 ‘ . T . 7 '~ : : ”7 ' 7 7 i. , I u .' ~ .4 . .4 , - - 7 ~ , - 3 '~ . '- ‘7‘ r 3 I “ t : '- .- < 7 - ~ 2 :- 7f? f: I: r : 2.; —~: 7 2‘ 'i - : f. — ‘7 2': ~ . I'. '— ’ ~ I: — .‘ < . 7 7' - 7. — . r ‘.:-I I I I II.I I I l-l‘lonvcr “I ‘ I7 7L I 7‘ I ‘ I I ' I I I I .I . .o I ' I I I I ‘II,I I ' II.I 7 _ , 777777- 7 7.7 . .7 -7 77 _ 7 _777 7 .777777777 7 I7. r 7 7 7 I .I“ I . I I ‘ Z-StrmtI- I ,II,I|I ‘ I .. I 7 b . . +77-—_ ‘ .17 . I In Ill-m _‘ .I A 1 II.‘ II,I II,I _. . 7 77+ . 7777 7. . . . . . , . . . - .77 77777 . ___7 7 .- 7 I 7“ 'JvIIlIrrIAx . ‘ I ‘1 I -»—-9—-b o»-——- . 7, I_77. 77 . I .II' I ] I In“: MI, ‘ II,‘ 7 77- 7 -7 7777 7 . . . 7 . _ _ . 7 . _ ..7 _7777 .7 . . ..._-._- I ,It 8-S-u'Le-s— . 7. v‘pf‘n 777—7. ‘ sion .1 I 2 . ' prps 1 "Yul: :Y-I 7 . 7‘ — ~~ 7 “Y - . . c q . . 7- 77- 7 777.7 7 - . 777.7777. 7 ~ 7 I , H l I I I I I 3—SUII eu- ‘ ‘ ‘ I III 'II I y—~»+ a a A c . 7-74 3.1m I I I .I ' I 3 j .u I ‘ I I I I rrII I rrIII rr.dI ‘ . I - —- —~—¢-————-q ———— —~l '3' u . I o . . 77 .7 7 . ~ 4 . ‘v i 0‘7 .7 77.7.7 - 7 - .0 7 ,4. .I, a, I . 'i 'fl I . h-SLIL‘LJ‘M- I ' ) I )1 I I I I".[ ....-n ' I “1544 IF" .7 I I - . I I . O I I I I ‘ I 1 I I" .I .f—z—vl ‘ ‘ 77 7. A —— .7 _ _. 7 ‘ . 777 '. .. 77 7. 777 7.7 7 .‘ . . ‘7. .7 _ 0 777,7 —— ' 7 j.7II I.” .II I ' I I I I 7-Fnod Hub -.~3'I‘ . II : .II - I tIvmplrx- 'Ll’l . . -‘7‘7I77""7'7I7 7 .- 4 I . I , I J,’ d ‘ I . Id 7 1 .2d | 3d '5517 7 777 7 7 'I-"nrv . 2i: ‘ Iva-«In! z"f_ .Il'f” I.It'.rr In "-”’ "—'_"“ . ,. - 7 7. A [hm R, l‘ ‘ ‘I’I‘II 7‘ . 47 I 7 v 77 -7 7 7 . . . ‘ 77 - . . . 7 7 . 7 7 - 7 7 7 - .7 - . l ‘ [o 9-IuI-ld Heb I i . I j ,,.. . _ I . . Vii—+<— .. I N I (‘7 .hltx 'L_-»_7;———. : y“ , . ‘ ‘ t7 ' .3I ' .'II | -‘ : -d I I.dI I I.rII _7_ —*-‘r}— +7 ____7_ __ ._ . - 77 .. . 777777 777. . - 7 _ 77 .777 77 - 777777 I.) ‘ 2d I 1 .w 1,. I d.1 I I d.r IO—Rz . - ¥.3d _ rItlfI +7 I r. ——I»e—I math”: I ‘. I d T V I - I .3- I . . Iw I I I . I I I ' I . I lI-Spl-cleu : I idJ.‘ I ' ‘ i ##1sz I I I I l ‘ . I DIvcruItv {IMI I “r— r w- ‘ ‘ I 7 f“ I 7 'I 7 .I , ' . s I . . , . .7 <43”: ‘ ; I 4.4.. .I..II. -.=.I~ . .s . -—————— -7---_..*77 ‘ 9- ~-~ o 7-— —-. . —————.——————-— - -~.— ———~~ — - —o—— v—— ~ -~ o —— I ”I 7' I .. IZ-SLYIIIIII- I d_I -7777771‘d .I‘ d I . , I . cIIH-m' I ' ' ‘ r " I— I —I 4.1 . 0 .7 ’—- 4- — - ‘ —— I , .I . ‘ »—- . .. d,II I d.I ‘I 77.— ...7 _. 7.7.. - - .. .777 ... . -77 7 7._7, _ 77- 777. 7777 . 7 777777 .7 777-777 . 77 . ’ 77777.7 -77 I . I 52 h = has: used to indicate a property or quality of the concept (Pioneer has . . .) hi = high: value of a concept variable (high amount of litter) s = simple: value of a concept variable (simple food web complexity) l = low: value of a concept variable (low species diversity) li = little: value of a concept variable (little stratification) pr = proceeds: for proposition of covariation (As succession roceeds, x . . .) d = ecrease: for proposition of correlation (As x decreases, y decreases) _ n = narrow: descriptive of niche (narrow niche phrase is reciprocal of high niche specialization) I = increase: for proposition of correlation (As x Increases, y Increases) - negation of relation term (I = no increase) .2 therefore (Item eight states, "A has more litter than B, .u (therefore) A has . r = reverse: change in direction of successional trend (decrease stratification reverses succession process) mo = moderate: value of a concept variable, some- where between high and low (moderate stability) Figure 5. Code definitions and explanations Propositions forming the basis of the item: F 1.1 The pioneer stage of terrestrial succession has low amount of light at one foot. T 1.2 The pioneer stage of terrestrial succession has low food web complexity. F 1.3 The pioneer stage of terrestrial succession has high species diversity. F 1.4 The pioneer stage of terrestrial succession has low amount of light at fifteen feet. Recording and Scoring Student Responses Student responses are to be recorded in the right- hand cell segments in Table 2. The recording makes visible 53 >. u4 H4— fifi m3 0- on 3% 0-H r-l .—I m a m u u 'UF4 0:4 a4J p44 09. mm 23.-G $3.1: c>a CL> oem ocm ho m'l" E’H 3'14 0 >4 Q N x >4 N n—l x o—l >4 Stem Concept .2* Pioneer h,l 3 .l 4 h, h, h, hi 1 l Figure 6. Formation of a row in the test-item propo- sition matrix Codes: h = has; 1 = low; hi = high; * = correct option what the student implicitly asserts and what the student chose not to assert. The recording process can easily be computerized. The logic behind the process is presented in the following paragraph. For each item the student responses were recorded in the right-hand segments of the cells corresponding to the option chosen. The location of the marks indicated the truth value of the corresponding proposition (those in the upper half of the cells are true; those in the lower half are false). When a student had an opportunity to select a foil and did not, a zero was assigned to the appropriate 54 location. If there was no Opportunity to select, the seg- ment remained blank indicating there was no information relative to that proposition. Figure 7 illustrates this procedure for an item. >. ma tH- $3 «.3 o- om 3N 0-H H H 0) Ha: u 4.) 13.—a OH :44 C44 on. cum 54: SJ: °a 8‘5 °.S.° °.EP F148 N O N N N N EA :4 EA Stem Concept .2* Pioneertn]. l 3 .l 4 1L 0 In]. 0 h, 0 hi 1 Figure 7. ROW'Within test-item proposition matrix with example student responses recorded Codes: h = has; 1 = low; hi = high; * = correct option Proposition Score Derivation The codes within the right half of the cells for a row of the test proposition matrix represent the scores (1 = asserted, 0 = not asserted) for propositions within a particular item. A proposition score is considered to reflect how willing the student is to assert a proposition. More than 55 one item may offer an opportunity to assert similar propo- sitions. (Both items four and five, Table 2, test a proposition about succession and the amount of light at one foot.) To obtain more meaningful proposition scores, the test proposition matrix is collapsed across rows combining cells which test for a proposition containing the same concepts. This forms a proposition summary matrix for the test (Table 3 displayed on the following page). Table 4 summarizes the frequency of propositions containing the same concepts. The numbers in the upper and lower cell halves indicate the frequency a pair of concepts occur in different test items forming a true or false propo- sition, respectively. In some cases, a foil represents more than one proposition. The propositions within the foil may differ in truth value (see item twelve, Table 2). The administra- tion of the multiple choice requires the students to select one and only one option. Therefore, the student could not indicate in the item response all the true propositions which the student would be willing to assert as true. If the student selects the correct foil and there are other true propositions in other item foils, the student receives credit for those other true propositions within the item. The assigned credit (a score of l) for other true proposi- tions within the item is awarded under the assumption that the student realizes the false proposition within the foil is incorrect and the other proposition is true, but the 56 u. 311 ‘L ‘l ‘L 7 7 ll 7 l .| 7 7 \VI .L! a 7 A [J >‘ \ . .o.‘ ‘ HmMH mmH mom NHMB . 7 7 7 7 7 7 7 7 7 7 7 7 mid; __ 7 7 l x I 7 7 7 7 7 7 ..{1‘ ‘Tll « ||§ \ 4.‘!\ ..x. 7 7 7 \t 7 L 7 7.97 7 7 w, .\n\‘. U \ ol‘ 7 7 7 r ‘ Y A 0 w l 7 7 4.m_ 7 “.4 l ‘0: \T A 7 7 7 7 . M — tw I A7 ‘ ‘ ] 1’ 7 7 7 V t . l u. .x i .. w: u - 7H - .... u 7. w. u . U f T... s . 7 _._ 1.. l . I . I ,\ .H. h. m u. ,. .... .7 7. + - - - ¢ - .. ,... m n m z u .I ,. U . C m mqm<9 7 7 7 .1!» .11 I . 7 7 7 I! l 7 7 7 7 7 7 .l- 7 75. . 7 : . I [I] 7 .37....7573 _ 7 .c _¢ .. . .3;- 7 r..~ .au 1|. .7 . t3 :7 E. Esta}. ¥ }\1 . >ufl19~QFfl§U N .v. :27... with I1.K11|Fr..... . , it’ll I- ,7 :7 7.“ -. 7 1 n a w] I. .7 o.. _ a. alltcw t .IIIIII .. .‘i. 94129.43}. I: 7: a 4.: 3.: 1T KEL elf. . . Tl--. 7 , 7 7 3 7 p y 7 7 . . 7 7 w \w .‘ Ii. 1:17 1327.73 7 7 7 7 7 _ 7 - -11- L7 711- _ _ 7 7 7 7 . _ 7 _ m _ 7 m... 7 n..,«_ 7 77 m.... 7 #.ra_ _ “4:7 7 3— «.71: _ 9.7.7:... ‘7- . y. llfll . x . .- . 7.7 7 «.7, r. Lttgum .c.a7_. H “.777 ~ ~57 .1: «.1: 512:4; i 4. "l a M ._. ... c _. U. L7 , 7, K. ,) J . . r .7 1 I 1 I. I m H. I l. .4 v u x l l 1 . v - . p. J P y. .. H .I A ‘ I, u T .1. H X .1 .. 1. M... \I la \l. . .1 .U .3 .: I. x. .l 1 vi I .6, L \A U .l I A l .4 1: H u ZOHHHmOMOMm TABLE 4 PROPOSITION FREQUENCY MATRIX uoinenlasqo 010“ paaN $103981 [IV sofiujg {IV Jqfigq ;o nunomv uogssaonng Knlquvls wi;13 Olpplw Joauogd .91 Jufitw Jo nunomv .9 3u311 30 lunumv .I Jufi11 Jo lunnmv ajnjxnl Igug Juaqug oguvfilo 1103 133311 ;o Junowv uugzungggnving KJISJQAKG sagaadg uo;3v2;1213adg rN-‘lN Anyxatdmog an P003 57 u-n 0-1 0—. --¢ —1 y—l N —a —-a .—o ---l #0 —-4 r-d -—4 v—c .— N I-l N .—o —a F. I—d —d o—‘ N —-t .0 0. 0H r—d N N u-d —1 N —1 —a o—t u— -—4 p-a u-a -—‘ —‘ —-4 -< N —‘ o— N N -—¢ L—d __ -7l777i7777777 77 ’ c x I l O U U—a >~ *v-fi I *4 ,0 '9-4 0 U h ma ‘ w m X m M G) H C X ’1) 3 Q) U L. Q) '1) Q) H O l m a) -——4 C Q) *4 L: C m- I E U U 0- D U U w 0 h H d U 0 E O U m > w u m | H D C O E % Q~4 m m U . U m m D < A m D 58 student does not select that foil with the distracting false proposition when an alternative foil has all true propositions. If the foil selected is false and contains a true pr0position, no credit is assigned for other true propositions in other item foils. The student's score for propositions is equal to the ratio of the indicated assertions of the particular proposition compared to and no greater than the frequency the proposition occurred per item. The case of assigned credit described inflates some proposition scores (the stu- dent did not select the particular propoSition in a particular item, but receives credit). This provides a constant denominator for all student scores and is con- sidered due to test construction, not a student's lack of willingness to assert a proposition. Not assigning credit as described would reduce the student's proposition score, penalize the student for not selecting a false option, and could lead to false interpretations of students' beliefs. Guidelines for Interpreting and Analyzing Student Responses Once the student responses were recorded on the test proposition matrix (Table 2) and the proposition fre- quency matrix (Table 4), the following questions were employed to interpret and analyze the information received: 1. What list of asserted propositions can be generated for students? 2. What are the proposition scores? 59 3. What consistencies are reflected in the proposition score? (Does the student always select the same propo- sition? Does the student form contradictions? If the student is consistent, does that consistency lead him to select incorrect options? For multiple proposition occurrence within items, were the assumptions for assigning credit to nonselected propositions supported by other responses?) 4. What are the types of relations (based on the task descriptions in chapter two) within propositions that seem troublesome? (Does the relation reflect a charac- teristic of a stage--variable stage relation, a change during succession--variable covariation, the relation- ship of characteristics during succession or the modifi- cation of a characteristic--variable correlation, or correspondence of characteristics' values--value-value correspondence?) 5. What final list of propositions can be generated for the student? The procedures used to answer these questions involve a comparison of two student matrices (Table 2 and Table 4 with responses and scores recorded). For example, Table 4 is searched to find low-true proposition scores or high-false proposition scores. These indicate little belief or false beliefs, respectively. Scores in both halves of a cell for a proposition would either indicate inconsistency in belief or selection due to another proposition in an item 6O option. Table 2 is employed to determine which is the case. Table 2 makes explicit the proposition in question, i.e., the relations and concepts in each option are speci- fied. As one proceeds through the list of questions, more consideration is given to Table 2. Table 4 provides a starting point; Table 2 provides more substantive informa- tion. The process is repeated for each individual in the sample. The following is a summary of the procedures devel- oped to analyze the propositions a student asserts on a multiple choice test: 1. Derive propositions from multiple choice items by com- bining the stem concept and relation with the relevant concept(s) of each option. This provides a succinctly stated list of tested propositions. 2. {gym a proposition matrix by listing all concepts from the instructional unit along rows and repeat listing along columns of matrix. Record item/foil numbers in the cells representing tested propositions from step one. This provides an efficient representation of the coverage of the test for the instructional unit. 3. 3239.3 test pr0position matrix by listing the stem con- cepts along the rows and all unit concepts along the columns. Within each cell representing a tested propo- sition, record the foil number and coded relation by truth value. Provide space to record student item 61 selections. This provides an economical representation of the tested propositions. 4. Record student responses pp test proposition matrix with l for assertion and O for no assertion. This provides an efficient representation of student assertions and scores for propositions per item. 5. Form a test pr0position frequency matrix for individuals by collapsing the test proposition matrix across rows for repeated stem concepts. Record by truth value the frequency of proposition occurrence. This provides an efficient representation of the frequency a proposition occurred in the test. 6. Record student's propositions scores 93 test proposition frequency matrix by summing student scores for proposi- tions from test proposition matrix and record in respec- tive cells of proposition frequency matrix. This provides an efficient representation of the frequency a student asserted a proposition represented by the cell. 7. Interpret and analyze the test proposition matrix and proposition frequency matrix following the described guidelines. This provides a characterization of the student with respect to the propositions tested. Procedures to Assess the Validity of the Information Received from the Multiple Choice Items In the previous sections, procedures to organize information from a multiple choice test have been described. 62 The following section describes procedures used to assess the validity of the information received from the multiple choice analysis. Evidence of validity will be assumed if the students consistently assert propositions in response to an essay question equivalent to those asserted propositions on the multiple choice test. This section is divided into subsections describing procedures for propositional analy- sis, determination of intra- and inter-judge reliability for that analysis, and assessing the validity of the proposi- tions obtained from the multiple choice items. Procedures for Propositional Analysis of Student Written Responses Following Pines (1977), a procedure called proposi- tional analysis was employed. The analysis procedures involve rewriting the students' statements in the form of simple propositions containing two concepts and a relation. If a student's statement provides a series of concepts and their relation(s), then that single statement yields a series of propositions. All pronouns are replaced with their referents. (The analysis procedures and rules described below are included in appendix D.) The students' propositions were rewritten in a list. As the propositions were rewritten, they were translated into a standard format. For example, if the response con- tained "During succession the number of different organisms gets larger . . .," then the translated response became "As 63 succession proceeds, species diversity increases." The standardization of propositions allowed comparison of the written response to the multiple choice response and extended Pines's procedures. (Pines's work compared stu- dents' statements before and after instruction and did not necessitate this form of translation.) In addition to the propositions made explicit by the student, certain kinds of implied propositions were also identified. One kind was an unstated, surpressed premise in a transitive argument. For example, the response could be "During succession, the food webs become more complex because the number of species becomes greater." The state- ment contains two explicit propositions and one implied proposition indicated by "because": (1) As succession pro- ceeds, food web complexity increases; (2) As succession proceeds, species diversity increases; and (3) Food web complexity increases as species diversity increases. The transitive arguments were usually signaled by the words "because," "thus," "and," or "therefore." If the transitive statement included three concepts, they were completed with the initial listing. Transitive arguments with more than three concepts were completed after the initial listing and listing of translated propositions. After the original propositions were listed and translated into standard form, the translated propositions were listed next to the original list. Then the lists of original and translated pr0positions were reviewed to 64 determine if the lists were complete. Incomplete proposi- tions were those propositions required to complete a multiple transitive argument or those the student might have asserted as straight-forward inferences from those listed, e.g., if A is related to B and B is related to C, then he would probably say C is related to A. This is not consid- ered a transitive proposition because there is not a transitive phrase in the statement. If there were proposi- tions in these categories, they were posted in an additional list. Therefore, from.the students' responses three columns of listed propositions were obtained--the original and translated, the translated, and the inferred. Establishing Intra-judge Reliability To determine intra-judge reliability of the essay propositional analysis, eight responses were randomly selected from the original sample by an independent party. The eight responses were duplicated. Half of the selected responses were reintroduced into the original sample and the other half were placed at the end of the original sample. No selected responses followed its duplicate. The selected responses provided the data used to establish intra-judge reliability--the consistency of the judge's judgments. After the total sample (originals and duplicates) had been analyzed, the selected responses were removed. The original and duplicate responses analyses were then compared. The comparison was based on the equivalence of the two 65 analyses. In the comparison, equivalence was established by answering the question "Did the judge have the identical proposition (concepts and relations) in both cases and dur- ing the same list for the analysis?" The comparisons were performed by an independent party who recorded the number of equivalent/nonequivalent propositions. The number of equivalent/nonequivalent propositions was totaled and the percentage of agreement calculated with the following formu- la. The percentage of agreement indicated the reliability of the separate observations. x equivalent X EQUivalent + y nonequivalent x 1004 = A agreement Establishing Inter-judge Reliability Inter-judge reliability was also examined to further assess the procedures used to analyze the essay responses. The underlying question to be answered is, "To what extent can someone follow the essay analysis procedures and obtain the same results?" To determine inter-judge reliability of the essay propositional analysis, another sample of eight responses was randomly selected by the investigator. The sample responses and analysis procedures were given to another judge familiar with the course and the propositional know- ledge of interest. The second judge received a brief training session. During the training session the proce- dures and rules were discussed. Two examples of essay responses from a different exam were used for training. 66 These responses were based on similar propositional know- ledge, but the exam question specified the formation of different propositions. Once the training session was completed, the second judge received the sample to be ana— lyzed. When the second judge completed the analysis, the sample was returned to the investigator. The investigator compared the two sets of data fol- lowing the procedures outlined for establishment of intra- judge reliability. In addition, stylistic (the breakdown of the original written responses following Pines) and inter- pretive (translation--did the judges agree on what the student said) differences were noted to record variations that might exist between the two judges. Assessment of the Validity of Information Received from the Multiple Choice Analysis To assess the validity of the information received from the multiple choice analysis, the propositions derived from the multiple choice items were compared to those from- the essay containing the same pairs of concepts. For each individual the essay and multiple choice propositions were recorded on a matrix similar to the stu- dent's individual multiple choice proposition frequency matrix (Table 5). Table 5 is a propositional selection matrix which includes selected relations rather than fre- quency of selections as Table 4 has. It also contains all concepts for the unit. The first seven rows are for stem 67 TABLE 5 PROPOSITION SELECTION MATRIX coag¢>cawfie acct vLLI mucuccm H~<7 wtueum __( szw7a .n S u : . :uht. Ca; 1.7.21.3“ xsr__nua¢t~>.a mumuvnz |l|l-vl _ _,_______ ceauwu«~:_c:;m .1717. 7__..0_7 xquuH332o L33 Cirvh Pioneer Stratifi- cation Climax Succession Food Heb Complexity Amount of Litter Species Diversity Niche Special- tzatlon 5011 Organic Content Texture Amount 0! Soil Light 1' Amount of Light 5' Amount of Light 15' Middle 7 77,77 7 Stability Amount of Light 68 concepts of the multiple choice items. Once recorded, the number of equivalent and non- equivalent proposition instances was counted. Equivalent propositions stated an identical or a reciprocal relation- ship.1 Nonequivalent propositions stated relations with non-identical concepts or differed in truth value. Figure 8 was used to record the proposition counts. The propositions were divided by the type of relation described in chapter two (variable-stage, value-value, variable covariation, or variable correlation) and whether they were directly translated or inferred from the essay response. Relations not comparable were also counted. Once the data were recorded, the percentage of agreement was calculated: x equivalent x equivalent + y nonequivalent x 100% = Z agreement Sample and Data Collection The sample for all analyses was selected from forty- four individuals enrolled in Biological Science for Elementary Teachers (BS 202) Summer Term 1978, at Michigan State University. During the third week of the session, all students received instruction on terrestrial succession. The instruction was based on the instructional unit described in chapter two. The intended instruction was to develop the 1Reciprocal relations state the opposite relation, e.g., in one proposition both concepts would increase; in the reciprocal both would decrease. 69 Figure 8. Form to record frequency of comparable and noncomparable propositions from two sources Key: VS = Variable/stage VV = Value-value correspondence VCR = Variable correlation VCO Variable covariation MS Match same MR = Match reciprocal MM Mismatch 614 mu> 00> >> w> somwuwmaou ummH oz mo> 00> >> m> zwmmm comfiumosoo oz UOU u: uou 22 m2 m2 wouuowaa zz\z Hmuoe 22 m2 m2 woumamcmuu zz\z Hmuoe mo> 00> >> m> vacuum: uoc ponesz mu> 00> >> m> Hmoouafiomm possum: umnssz mo> 00> >> m> HmofiucwnH empoumz nonenz .oz ucmpsum 70 students' propositional knowledge of terrestrial succession. The tasks, based on Smith's (1973) variable-value network, were used to describe what was to take place during the instructional activities. At the conclusion of instructional activities, a twelve-item multiple choice test and an essay item were given to the students. The items were based on the task descriptions and the propositional knowledge addressed. The students were instructed that the test represented a prac- tice set of questions which were similar to those they would find in an upcoming examination. During a subsequent review session they were provided the answers to the questions. Of the forty-four students enrolled in the course, twenty-seven completed both the multiple choice items and the essay. The twenty-seven were selected for analysis because they provided the necessary data (essay and multiple choice response) for this study. The seventeen not included indicated they did not have time to complete the essay or did not feel it was necessary to practice taking a test. For the data analysis, all the student response sheets were coded and references to individuals were removed by an independent party. The investigator analyzed the essay responses before conducting the multiple choice analy- sis. This was done to prevent bias in the essay analysis which could influence the assessment of validity for the multiple choice procedures. 71 Summary The procedures developed to organize the proposi- tional knowledge tested for in items representing a multiple choice test on terrestrial succession were presented in this chapter. Procedures also have been described for applying and analyzing the results of that organization and to assess the validity of the results. Chapter four presents the results of the developed and applied procedures. CHAPTER IV RESULTS Overview Chapter four begins with a description of the results of the multiple choice analysis. Section two describes the results of the essay analysis and assessment of intra-judge and inter-judge reliability. Section three describes the results of assessing the validity of the multiple choice propositions using the results of the essay analysis. Each section concludes with a brief discussion of the results presented. The chapter concludes with a summary which ties the sections together. Multiple Choice Analysis Following the procedures described in chapter three, a list of propositions from a twelve-item multiple choice test for the unit on terrestrial succession was derived. The proposition list is presented in Table 6. Each proposi- tion per item is listed in column one. Column two lists a reference to the task (described in chapter two) used as a basis for the item. Column three lists the type of relation involved in the proposition. 72 73 TLABIJS 6 LIST OF PROPOSITIONS DERIVED FROM TWELVE-ITEM TERRESTRIAL SUCCESSION MULTIPLE CHOICE TEST Propositions Task Relation 1.1 Pioneer stage of succession has low Task 1 Variable-stage amount of light at l' *1.2 Pioneer stage of succession has simple food webs 1.3 Pioneer stage of succession has high species diversity 1.4 Pioneer stage of succession has low amount of light at 15' 2.1 Low stratification is found in the Task 1 Variable-stage climax stage of succession 2.2 Low stratification is found in the middle stage of succession *2.3 Low stratification is found in the pioneer stage of succession 2.4 Low stratification is found in all stages of succession 3.1 Climax stage of succession has low Task 1 Variable-stage soil organic content *3.2 Climax stage of succession has low amount of light at 15' 3.3 Climax stage of succession has low niche specialization 3.4 Climax stage of succession has low food web complexity 4.1 As succession proceeds, food web Task 2 Variable complexity decreases Covariation 4.2 As succession proceeds, species diversity decreases *4.3 As succession proceeds, niches become narrower 4.4 As succession proceeds, the amount of light at 1' increases 5.1 As succession proceeds, stratifica- Task 2 Variable tion does not increase Covariation 5.2 As succession proceeds, soil organic content does not increase *5.3 As succession proceeds, stability does not increase 5.4 As succession proceeds, amount of light at 1' does not decrease 6.1 6.2 6.3 *6.4 *7.1 7.2 *7.3 7.4 *7.5 7.6 8.1 *8.2 8.3 8.4 8.5 *9.1 9.2 9.3 9.4 9.5 10.1 10.2 74 TABLE 6—-Continued Propositions As succession proceeds, species diversity increases As succession proceeds, stratifi- cation increases As succession proceeds, stability increases (1 and 2 correct, but: As succes- sion proceeds, stability does not increase As food web complexity increases, species diversity increases As food web complexity increases, niche specialization decreases As food web complexity increases, stratification increases As food web complexity increases, species diversity decreases As food web complexity increases, niche specialization increases As food web complexity increases, stratification decreases A has A has A has A has A has B has A has B has N more litter than B, therefore low stratification more litter than B, therefore higher species diversity more litter than B, therefore higher food web complexity more litter than B, therefore higher stability eed more observations As food web complexity increases, amount of light at 1' decreases As food web complexity increases, niche specialization decreases As food web complexity increases, stability decreases As food web complexity increases, soil organic content decreases As food web complexity increases, soil organic content decreases As stratification decreases, strati— fication increases As stratification decreases, succes- sional process increases Task Relation Task 2 Variable Covariation Task 3 Variable Correlation Task 3 Value-value Correspondence Task 3 Variable Correlation Task 4 Variable Correlation *10.3 10.4 10.5 *10.6 10.7 10.8 *10.9 11.1 11.2 *11.3 11.4 75 TABLE 6--Continued Propositions As stratification decreases, species diversity decreases As stratification decreases, the amount of light at 1' increases As stratification decreases, soil texture increases As stratification decreases, suc- cession reverses As stratification decreases, food web complexity increases As stratification decreases, species diversity increases As stratification decreases, food web complexity decreases As species diversity decreases, the greatest impact is on climax stage with high niche specialization As species diversity decreases, the greatest impact is on middle stage with moderate stability As species diversity decreases, the greatest impact is on pioneer stage with low species diversity As species diversity decreases, the greatest impact cannot be told from information given As stratification decreases, the amount of light at 1' increases As stratification decreases, the amount of litter increases As stratification decreases, the organic content of soil decreases As stratification decreases, species diversity (herbaceous) increases As stratification decreases, all factors increase Need more observations As stratification decreases, the amount of light increases Task Task 4 Task 4 Task 4 Relation Variable Correlation Value-value Correspondence Variable Correlation Note. * = true propositions options contained in correct 76 As indicated in chapter three, a list of this length. was difficult to handle; therefore, a matrix was used to organize the propositions. The first matrix employed was the unit proposition matrix (Table 7). All the concepts for the unit were listed along the rows and columns. The last three columns referred to the item foils which did not fit the basic pattern. The item foil numbers were recorded in the cells of the matrix representing a tested proposition between a row and column concept. The empty cells represent untested propositions. Table 7 provides a representation of the amount of the intended instruction encompassed by the test. Empty cells represent propositions which could be used for further item development. The numbers in the cell indicate that a proposition was tested. The numbers per cell indicate the emphasis given to the relation between a pair of concepts on the developed test. The total number in the row and column reflect the emphasis on a given concept. For exam- ple, the row and column for stratification indicate that concept was part of several propositions found in the test. Of the twelve items, stratification was used in seven of the items. The amount of light at five feet was never used in an item. This test emphasis and item development will be discussed further in chapter five. To focus on the test itself, a test-item proposition matrix was formed and is represented in Table 8. To facili- tate interpretation of student responses, references to task 77 TABLE 7 PROPOSITION MATRIX FOR TERRESTRIAL SUCCESSION UNIT WITH TESTED PROPOSITIONS INDICATED Code: * = correct option 77A :3— a..>u....L: .7. z} 7.7.... 77 [ll .. I 5.. ..u..r 77¢“ III 7 III. II a. I 7 m ...... M 7 7 . 7 7 I II - III 7 0.7.3.— 7 4. 7 .c 7.77.1.7. 7 7 7 P. 7 7 7 7 7 7 7 71 I I II% I IIIII . 7 rI I I II I I Ia III 7 IIAII. II . IIII A I 7 III .. 7 ..II. ..I.. W...“ 7 7 7 _ 7 7 7 7.. z... .2; . ..J. H IIH .III... 7 7 7 J - 7 .377??? 7. II III-Ii 7H . .. I . IIII II.I II .I. III _. III ,7IIII 7 7 7 - I. 7 7 7 1.3;“; 7 7 7 .II” ... 7 W 7 7 7 7 I IIIIII v I II I . .I IIII I7I I +III I 7 III . I II W II III .IIII _ 7 7 7 _ 7 7 7 I 2 7 7 7 7 7 77%.:7 7 H I. 7 7 7 7 7 7 7III-III. .7 . III. IlII . . .1_ III . II II. III . I . III 4 - 7 II . II 7. IIIIII 7 . _ 7 7 7 _ H. In) 7 7 _ 7 7 7 - . 7. 7 .~.v...:,.7L. _ U 77‘ 7 7 7 7 7 7 .I II .II . III. II . I . I I III . II. . 7 I. II _I II .. 7 _ 77 fl _ Q ...7 5.73.7 7 7 .7... an: it... I. . I . . I .7 I I 7 ‘ .... «:2. ”II... 7 .7. 5.77:7 . 7 7 7 7 7 7 . 7 7 IIII..7 7 _ ... ....IIII.. I . .I7. ...III 7IIII 7 IIIIIII... . hIIII . . 7 II7 I.I . .. .7 7 _ 7 _ I 7 . l ...... . ;.i:_m m; :7 _ 7 7 7 I 7 7 7., :7... ...... “I 77. 7 I I I _ 7 7 7 I 7 7 7 II. -. I17 I. 7. . I. I . I I77III II 7 II 7 7 II 7 I.. III . 7 7 7 7m 7 7 7 7 7 7 7 7 7 . 7 7 I I.IIII 7:4...15 4. ...” 7 77 7 .. . 7 7 7 7 7 I” 7 ..7_:7...c: :1» 7 Q. 9 7 _ .Il. . 7 7 7 - 1 _ .II I r IIIII I +II « . II . III. I 7I . IIII I I7 III... I IIIII7 7 7 _ 7 7 7 7 7 7 7 7 .c7a7”_ 7 7 7 J J 7 7 7 7 7 . 7.. 27.-sh... 7 7 7 ”ml. 7 .. 77 7 R ? 7 7II...f. u7xz7a5207 7 7 S J l 7 7 7 7 2 7 a 7 . \l 0 ~ - ......7 7.6-:7 I-. I 8 _ 7 7 7 I J 7 _ I III—II . III. II, II III I I III I III h II IL. . IIT I IIIi .7 III- III v a _ _ 7 l 2 V. . 7 f. (I. I. I 7 _ _ f A“ n 7 In mm 7 o - .I e 7 o- .n .3 7 I u 7 D ‘ . . I7 \ l I ll .1 In.” .. .I.. m .c s. _ I ..l _ I f «I n f . f. _ I 7 t e 0 X 1 .H .I _ m c 7h I... U 9|. H. _ M e 7 n 8 W n ... n t n I M I. 7 I. I. I . I . .. ,I _ I I1 I. l I. . U .n u I . u ..n d _ -...- XI u. .....I ”am I. _ I. m... , .... .- I. m 7 ... .t k . rr I. _ 7.1 .\ r I A. 0 .1 II ... I. I. l I F C X S <.D S A L S O C S I 7 M L _ A L 7 M L P . H U M --..__-__- «7. - I 777- .7 .77 Sucafiusiun Light «11 Stages All Facturu Need More Observation 78 TABLE 8 TEST-ITEM PROPOSITION MATRIX WITH TYPES OF RELATIONS AND TASKS INDICATED Key: Relation Symbols h = has d = decrease hi = high n = narrow s = simple I = increase 1 = low - = negation of relation term ii = little 3. = therefore pr = proceeds r = reverse mo = moderate du = during stage A r. _ Ha . _ 7 7 7 . 7 7 7 a 7 7 7 l . llJllllI vllwl- . _ _ . _H.fl 7 _ 7 _l.. 7 . 7 7 ll. 7 9 !!'ll . IIAI 4., ii 1 . 4. 7 1pm.? .31.$ 3“4.“; 77 7 4. 7 . 7 . r 7 _ _ . . . ..o 7 l a In _ l. 7 7 , 7 7 1 fl 7 7 7 M 71.. 7 -!|!!!- l!!!!!! . - - A 4-1!- 7 »-lll . - . ll. . 7.c, , 7 7 7 7 7 7, 7 7 .. . 7 7 4 7 _ 7 7 7 7 7 .! ! ‘vi“‘ ! 1 -l l»: a . ...-loll]!!! . l!!! U m 7 l o 7 7 l l . - ! l ! .l l - 2?. ll . l l . 7 1.. r. . 9 . 7 7 7 7 IllI.l l ill-1 Ol'lll! 1 o.; 7 7 _...7_ 7 _ 7 -. . 7 7.737 7 7 .— uc . ~16. 7 . 7 _. 7 ,- 7 7 7 7 7 7 7 o c c 7 fl..7 n7 , .7 7 7 . 7 [o|. 7 7 7 _ 7 .- -.l - - ---ll. 1 7- - u .I l 1 I H ..4- .! ll7Y ll. lll. 7 7 7 7 7 .,! lo . _ v.~ 7 . 7c— fill! 111:.» .. -.- ._ [ l JUN" u. 77 v7 7 7. 7 7 7 ... Tli+|l|L_ 7 «5.4.:3 4-!!- 7 _~.uv _ .-7777177m-~_ _ .m- 7 7 uu> «an» . -. l l,- 7 - -- 7 77:.v7 7 _ 7. 7 7 _ II.I _ -I!- 7»..:.u>72 7 7A._u 7 77 ...-4_.4.~_qu_. . _l.h!7-.. -.. 7!: :1! , 2.? 7 ....v _ ..v _. TI;.|-!. _ m. , 7 . tum. . h 7 :nau.r. 7 7 7 3.. 7 a 12:35.17: 7 7 P... a? .3, 2.77. t . lll-llllllu..l rt .ll-l l . a _ 7 v.7 7p 7 7m. ’77 , _ l;- -_ 7 . .M- 4 uxufluuau 7 _ 7 _ 7 7 gym w :7-o 7 . - . l 1 l ..7 l : - .u> 77.“ 1,7 7 7. .7 a _ .7. 7 7.7. .u A < alll+ll T.l!!!!. .l.!h!- :7 .977fi7 7 7 75.; 7 7 M: uzaae> u.uH . Avlllolllp .h 0: \ {Ill um. um. 7 u. 7 7 v7. 7 an. u_.7 >77 7 4.3.7 7!“? :7 7 -7319; _ n. _ HM.7 ~Cq.. 7 £33 Uovmuu 7 ~_. 7 7~.7 7 .n.. _ .u> "flak 7 7 _ 7 7.- 17.x!- 3 _ 2:. am. 7 ... _ 7 Iwauusnuc m. WW . 7 cu> ”any 7. 7.... -l‘lll .i 7 _ _ 7 7 7 _. 7 ca“: llll. _ 7 ll-u-U-m—_W‘C 7 _ _ 7 Gu> "NIH -!!l . ll7l! IL l-i .--i! 7v._a _ 73.7; 7 _ 7 u. 7 _ _. :2_m . :9 I t u . .A 7 :._n . lcuuuzmu: . . . _v> u¢n - . !----.- .--.Ll-Lm. 7.7-: ..M -- w w , 7 77.; 777.; 7 7 7 7 . M 7 7n. 7 7 .7 7 . 7 7 ‘1 u_-H . . ~£a£ ll 7I|7vllltl . I _ M. _ n _ ..17 la- rl-7 7 h25:... 7 7 _ 74% >9 ”_|H .I _ 7 _ I. w ... W” .-, -- w.” .7 n “m. .-- .. M .- .--, u. l. m .-_ 79 description and type of relation involved in the item have been included with the row concepts. Table 8, the test-item proposition matrix, provides a representation of tested propositions indicated by a foil number within a cell, a code for the relation within the proposition, the truth value of the proposition which is indicated by the location of the foil number and code in the cell (upper is true; lower is false), and the task and rela- tion type which is indicated above the row concept represent- ing the stem concept of an item. The cells are divided into right and left halves. The left half is for information from the test; the right half is for recording student responses. By scanning Table 8, it becomes possible to deter— mine not only what propositions are being tested but also the type of proposition tested and the distribution of con- cept usage by the type of proposition and task. For example, the amount of light at fifteen feet is only tested in propositions based on variable-stage relations described by task one. The test does not provide information about propositions containing this concept in other types of relations. If propositions about the amount of light at fifteen feet are viewed necessary, then the type of propo- sitions for which test items have yet to be developed is known. Therefore, Table 8 is more useful for item develop- ment than Table 7 as the test information presented in Table 8 is more explicit. 80 Table 8 is also used to record student responses. Two of the twenty-seven analyses have been selected to demonstrate the application of the procedures. Table 9 and Table 10 represent the test proposition matrices for stu- dents A and B, respectively. From these matrices, lists of propositions can be generated for each student (see Table 11 and Table 12). Table 13 and Table 14 present the frequency which the students asserted a proposition compared to the proposi- tion frequency in the twelve items as presented in Table 4. The frequencies summarize the list of propositions selected, and the tables provide a record of how often the student selected a proposition, the frequency of the proposition, and the truth value of the proposition. They also yield proposition scores expressed as a ratio. As defined in chapter three, a proposition score is equal to the number of times the student selected a given proposition compared to the total number of times that proposition (or a concept pair in the case of false propositions) occurred on the test. As indicated in chapter three, in computing the proposition score, credit for true propositions contained in incorrect foils was given to students who selected the correct foil. This was done rather than counting them as failing to assert the true proposition or having to use dif- ferent proposition frequencies for students depending on the options chosen. In the examples given, both students 81 TABLE 9 TEST-ITEM PROPOSITION HATRIX WITH STUDENT A RESPONSES CODED Key: Relation Symbols h = has d = decrease hi = high n = narrow s = simple I = increase 1 = low - = negation of relation term 1i = little .1 = therefore pr = proceeds r = reverse mo = moderate du = during stage _ ._ fill-.llll . :o«u~u«u H DE-fiumnz n.. m._ . m _ _~u> “a-“ . _ _ _ . i _ .4 .1a _ ¥ _ i c u . _ o . 1 _ Ml1lrl-lxllp1m1lg 1nu¢~9>ua _ 1 111‘ 1 .1 . 1| h no«ooawv~_ h , O H . O i . 111-1111 . 111 1. A . 1 ..uew. 11L1:11 couunouu . 1 1- i _ An 0 .1.“. _ W _ o.. 3.. .___ v -3131: 111 11- |- . -1 1 o1. .... - 11 11 .11,1o1_..|11 -1 1.1.itm.-.1w>mq-~1 a; . . . .o.~. . "a.“ c . . _ o 7. H _ o ... u s: . 1.1 fl . .. . O m... .1- 111i. .onufuflfiu a; _ 1 n _ so: 32... A ... , _ .1...) ante _ .1111 1. 1 11 11 111 - 1. ,..1~.111 . 1 1 1 . . 1 1 11111.- 4 . . .1 .1 1 - a o . . . ...h... . .1 . o .7 1 . .c . m A < . .1 . _ .1141.- .-1: 1 . . 312:: h "if. F0 vanifiéum 1 1 11- 11 .1 11- 1 - .1 1 c >>um1H 11 . 1111 111111111 - . 11.. .- .. .1 . 1 . 1, A1111 1 o . , c 2. o _ , .11 ...: _ 11 . 1.11.11- 1 . . L335 . O i _ H “0 its :.,o a... 9.: “sf; _ ,_ .A .7; am...— , fl . i. w 1 . _ _ _ , i: 1. 7 vfi-MQ , — . I, | . -| fi—Wq..- “ i M In” _ 1.... )4 :4 i A ., . _ c a. a . d cu> flunk 111111111 1- 1 1 1 K. .. .. - 1 1.11 1 11. 1 - 1 . . 1 .e r ..c _11. - 11- 1 11 v i} . . . .. me i. r. c N E C . cm 1. .1 . . .11r1v1 1.1 _ . :7 ~fi.._._& . . . -..; :7 . u.. . j _ . Cb) ”NI-H 1.11 111111. .11.. . 11 ... 11 111111111 1 .. -1111 1-1- . .1111 -11111fi - .....- 1 1 111111 24> ”Nuk A A c i: o .....r o 2.; . ~—. , m. . V , ..l1 .1 1., .11 >1 1 , v . A ., 7 _ .r, x-YM441m , . _ l i. _ . _ 1 . . _ #31. W i 1 >2 ..L. 11 11. . I . 1. 1 It. 11 . . c .— I c o 1 11 1 1. 1r o 1 . cl. c ...; : ...: _ 1 V H i W W V 1 . . _ . i i 11 ,. _ . . . ._ H . _ H * ‘ ...-4.5.114,“ ”ni;._~. , _ W -..»...evuu _ i .1121. W >4. J-.— 11. 1.111 1 1. -I .11. -. . 1 w--11111 . . -1 1 1. 1 f +. 11.1.1.1 .. . . -1 c A L 0 .fi . o A. m . y, 7 , E _ i I . . o . 110 . v1.9 1 nu...‘C..flLl— ,_ , Ti: y n _ .. >2 u_-u A m H .1. ..._1 .1 1 1 w I 11 .1. 1 - .. ..- x... u w -1. .. ..1 .1. 1.. u. 1. .1 w . 1 x - 1. .1. .11.“ 1 11. n n a. m .11- .1. 1 ... H 1 .... fl r q. 1 1. - ,.. 1 -. 11 - ... . 11 1-. ..1 . ... 1. I. .. x .1 l 1. .1.. u. 1. H 11 .. 1 I... 1 . U I I. 11 1. In - . u u n n Key: hi li pr mo 82 TABLE 10 TEST-ITEM PROPOSITION MATRIX WITH STUDENT B RESPONSES CODED Relation Symbols has high simple low little proceeds moderate IH'JCL H-° du decrease narrow increase negation of relation term therefore reverse during stage A . _ ‘ i .. . 1 111 1111 1 - 1 .11111 1 _ c . fl is . 1 1 11 .11 . .1. 11111 .. . . 1.111 1 . c .. _ i i. 11 11 . . q 1 1 _ . 1 1.1 a. . L: 11 . 1 c I .. _ . T .11 ..3 . 1 1. . 1 1 1 111 1 .1. i c . i . i . . . 1111111111 11,. 1 11 . 11- 1-1 ... . O i C i. 1 - 1 - .1 11 . c . o . 2 H. _ . .. c .- c.. , 2 7 H 1. 1 _ 1 4 ~..” . ”111 11 1. 1 . 1s. . .AH - _ W . ....1 . C _.1H ...”H’»; u i». 11 u 1 . . 3 M. a C -.T,» 111 1-7» . 111 1 - 1 11 _ i n C ,.; C ..fi _, ..W . . c a : ... c1- .1 , . . 1 . _. _ ... i . . . : ....; c , a i . . 11 . _ i 7 i .111 _ . . a: 5.; ... ..— -“... cumlmfi .3. ”duh b.“ a .. 37:; ...; ...-in. «q >>n41_ 9:22.: L 25513. .....> ”41H tau“ 15.59.. .i n91. $1.. 1. .... . Tr LL. 3 .2 A < r: 73:; we .51....cxfx .... :1.— 83 TABLE 11 PROPOSITION LIST FOR STUDENT A DERIVED FROM THE TEST-ITEM PROPOSITION MATRIX 6a. 6b. 6c. 73. 7b. 7c. 10a. 10b. 10c. 10d. 11. 12a. 12b. 12c. 12d. Pioneer stage of succession has simple food webs. Pioneer stage of succession has low stratification. Climax stage of succession has low amount of light at 15'. As As As As As As As As succession proceeds, niches become narrow. succession proceeds, stability does not increase. succession proceeds, species diversity increases. succession proceeds, stratification increases. succession proceeds, stability increases. food web complexity increases, niche specialization increases. food web complexity increases, species diversity increases. food web complexity increases, stratification increases. High amount of litter goes with high species diversity. As food web complexity increases, the amount of light at 1' decreases. As stratification decreases, food web complexity decreases. As stratification decreases, species diversity decreases. As stratification decreases, succession reverses. As stratification decreases, the amount of light at 1' increases. A decrease in species diversity has greatest impact on pioneer stage of succession with low species diversity. As stratification decreases, the amount of light at 1' increases. As stratification decreases, (herbaceous) species diversity increases. As As stratification decreases, the soil organic content decreases. stratification decreases, the amount of light increases. 84 TABLE 12 PROPOSITION LIST FOR STUDENT B DERIVED FROM THE TEST-ITEM PROPOSITION MATRIX 6b. 6c. 73. 7b. 7c. lOa. 10b. 10c. 10d. 11. 12a. 12b. Pioneer stage of succession has simple food webs. Pioneer stage of succession has low stratification. Climax stage of succession has low soil organic content. As succession proceeds, niches become narrow. As succession proceeds, stability does not increase. As succession proceeds, species diversity increases. As succession proceeds, stratification increases. As succession proceeds, stability increases. As food web complexity increases, niche specialization decreases. As food web complexity increases, species diversity increases. As food web complexity increases, stratification increases. High amount of litter requires more information before an infer- ence about a field can be made. As food web complexity increases, the amount of light at 1' decreases. As stratification decreases, succession reverses. As stratification decreases, species diversity decreases. As stratification decreases, food web complexity decreases. As stratification decreases, the amount of light at 1' increases. A decrease in species diversity requires more information to determine greatest impact. As stratification decreases, the amount of light at 1' increases. As stratification decreases, (herbaceous) species diversity increases. 85 TABLE 12--Continued 12c. As stratification decreases, the soil organic content decreases. 12d. As stratification decreases, the amount of light increases. TABLE 13 PROPOSITION FREQUENCY MATRIX FOR STUDENT A 86 F. a-i UOIJBAJBSqo 5 5 ozou paaN 5103395 IIV h__. _ 533225 IIV ° ,\ nu?! 1 ;o lunouv :‘, uoxssaaang ~ ~ g 6 o o XIIIrqvns — é xem;13 —' u-G EKPPIN ° 0 Jeauoyd _ ~ .51 zufixq 5 3 go lunomv .s iufiti Jo Junoulv A 3' _ N ~ 3 5 a o 5 S )0 )unnmv V é aanxal 1105 ;~ nuaiuoo '71 T: .. 31013810 nus C O N —. 191311 5 5 ;o Junomv ‘ d L 6 ” u0139313139115 —-4 -¢ N --l —-4 —-4 Anus/1m '3 5 (a 5 5 .. 5 sayaads - u0112211913ads " n 1 c KnixaIdmog : a u : an P003 '" N O a a a o u u “ * A ~ 0 n n H M4 a g 0 fl 0 vi a a: In :I u h a h o u o a a r u u u u a a a u v 3 u o > o u u a u o u n-u « u a H = o g H m n m m u o m m r A TABLE 14 PROPOSITION FREQUENCY MATRIX FOR STUDENT B uoIJBAJaqu 0,111“ poof" Sionavd [1v snfivus 11v 3H311 J0 Junumv 1101.55.13 3115 Slzttqvls wi113 OlPPBH Jaauoid .91 1H311 Jo Junomv C 1118?] n J Jo nunomv .I infilw Jr) JUIHMUV ainnxal 1105 3u31Uo3 DIUUEJO 1108 133311 )0 nunouv uoincogngeins Alysisniq sagaadg u0332211u130dg oqngN £11xa1dm03 qan P001 87 -‘ a—n .— o Q ~ u—i C 0—! O H H _..-___1__.--_ N -4 —t .— ,_ OI O O O O -——-—---~-1 O -1 u— 5 o a-‘ 0—! ad 0 O A 1 .— N -< N -—-n '° N c o —- 1 I’ 1 -_—._-__.+____.___—--_ _ . _ 1..-- 1 -‘ 1 C A 0—1 H —I —l :4. o—d O O V N —1 O O .— -¢ N N L: N c u—I N v-d ... —o —l u—d u-d u-l r—l .—a -—< A N u-n 5 L: c: — c u—‘ N iLN -—¢ o-d " a o 6 ~Li_-”_ii_1 1 C.‘ >~. I 1 O U u >‘ 1a 1 ~~ ,n w o u H W I U: Q) x U) "4 w w G 1 x m 3 o u h w m w u o 1 m 0 F4 c m w u C N 'H . E U '0 D- 23 U U U o u u - -H u o E O U m > 'r" U R! I H D O O 5 “H O-‘H 0- m U 1 U V) IL. U .. ..J m G 88 received credit for a proposition about stratification and soil organic content, but from their test matrices (Table 9 and Table 10) neither selected the proposition, the assump- tion being they knew that only the false portion of the foil was incorrect. And the assignment of credit keeps proposi- tion scores constant across students when students are forced by the item to select only one option. With the student responses recorded and scored, the interpretation of the results can take place. The questions that serve as guidelines for interpretation of student responses are repeated from chapter three. Rather than list the questions and answers for interpreting the student responses to the propositions presented, the interpretation for each student will refer to the question number. Student A is presented first. Questions for interpretation of stu- dent responses are as follows: 1. What list of asserted propositions can be generated for the student? 2. What are the proposition scores? 3. What consistencies are reflected in the proposition scores? 4. What are the types of relations within the propositions that seem troublesome? 5. What final list of propositions can be generated for the student? Student A l. The propositions are listed in Table 11. 89 The proposition scores are represented in Table 13. This student has indicated no inconsistencies as re— flected by the proposition scores (Table 13) and for the test selected all correct responses as indicated in Table 9. By selecting all correct responses, the student does not appear to have difficulty with any of the relations being tested. The following is a final list of propositions the stu- dent would assert based on the information obtained. Student A did not directly assert some of the proposi- tions in Table 11; therefore, until further evidence is obtained regarding those propositions, they are deleted from the final list. (The -1 following the statement indicates the reciprocal is true; e.g., if X increases then Y increases, and if X decreases then Y decreases.) 1. Pioneer stage of succession has simple food webs. 2. Pioneer stage of succession has low stratifica- tion. 3. Climax stage of succession has low amount of light at fifteen feet. 4. As succession proceeds, niches become narrow. 5. As succession proceeds, stability does not increase. 6. As succession proceeds, species diversity increases. 90 7. As succession proceeds, stratification increases. 8. As succession proceeds, stability increases. 9. As food web complexity increases, niche special- ization increases. 10. As food web complexity increases, species diversity increases. 11. As food web complexity increases, stratification increases (-l). 12. High amount of litter goes with high species diversity. 13. As food web complexity increases, the amount of light at one foot decreases. 14. As stratification decreases, species diversity decreases. 15. As stratification decreases, succession reverses. 16. A decrease in species diversity has greatest impact on pioneer stage of succession with low species diversity. 17. As stratification decreases, the amount of light at one foot increases. 18. As stratification decreases, (herbaceous) species diversity increases. Student B The propositions are listed in Table 12. The proposition scores are represented in Table 14. The student selected a foil containing a false proposi- tion between food web complexity and niche 91 specialization. But when the student had another oppor- tunity to select an option with the same false proposi- tion, the student chose to select another option (a true proposition relating food web complexity to another variable). To establish the student's belief about food web complexity and niche specialization requires further information. At this point, the situation is indetermi— nate. However, student B was consistent in nine of the ten opportunities to select or reject the same proposi- tions that occurred more than once. The student's responses indicated consistent trouble with value-value correspondence relations. In both items that address this type of relation, the student indicated the need for further information. A possible inference is that the choice not to assert a proposition in item eight about the relation of values for the amount of litter and another variable is related to the incorrect assertion about the relation of the value of soil organic content and the climax stage. (The amounts of litter and soil organic content are very closely related.) The following is a final list of propositions the stu- dent would assert based on the information obtained. As with student A, the propositions not directly asserted, but given credit for, are deleted from student B's final list. Given the above characteriza- tion and the information obtained, providing credit for 92 non-asserted propositions is questionable. (The -1 indicates the reciprocal is also true and selected.) 1. Pioneer stage of succession has low food web complexity. 2. Pioneer stage of succession has low stratifica- tion. 3. Climax stage of succession has low soil organic content (a false proposition). 4. As succession proceeds, niches become narrow. 5. As succession proceeds, stability does not decrease. 6. As succession proceeds, species diversity increases. 7. As succession proceeds, stratification increases. 8. As succession proceeds, stability increases. 9. As food web complexity increases, species diversity increases. ‘10. As food web complexity increases, stratifica- tion increases (-l). 11. As food web complexity increases, the amount of light at one foot decreases. 12. As stratification decreases, species diversity decreases. 13. As stratification decreases, succession reverses. 14. As stratification decreases, the amount of light at one foot increases. 93 The preceding student examples indicate that much information can be obtained from the analysis of the multiple choice test using the developed procedures. Student A indi- cates no misconceptions as the student selected all the correct options. Student B, on the other hand, indicates a misconception not apparent using traditional analysis tech- niques, and the student also indicates a lack of knowledge of a type of relation the instruction was intended to teach. Both students' responses are representative of the informa- tion obtained when the procedures were applied to the sample population for the study and the entire population of BS 202. The test proposition matrix for the entire popula- tion of BS 202 during the study is provided in Table 15 to demonstrate the application of the developed procedures to group data. In the right-hand quarters of the cells is the number of students selecting an option thus indicating an assertion of the proposition represented by the cell. The proposition scores for the group are recorded in Table 16. The proposition scores for the group are equal to the ratio of the number of students selecting the proposition and the frequency the proposition occurred. For computing group scores, the rule for assigning credit for true propositions in incorrect foils is not employed. The questions for analysis of individual student responses also can be applied to group data. Again, the questions are referred to by number. 94 TABLE 15 TEST-ITEM PROPOSITION MATRIX FOR BS 202 STUDENTS Key: Relation Symbols h = has d = decrease hi = high n = narrow s = simple I = increase 1 = low - = negation of relation term 1i = little 3. = therefore pr = proceeds r = reverse mo = moderate du = during stage 7"”iv , .cc_u4s - . H U l_.m.,_u_.._u.f.l»a ho> ”suk I .... 1.11.. . 1 . i ... a _ .. ...-...;Q .W.v L , rt_uhana‘~ m. . , . , >> uglk a... ... c:..a. . 9 v2. . u -.Lcu_tum-:. _ x n¢ .n., .3) walk i.“ a. .1. ...u. 11 , -....TEQ V 5...} ”17....‘I? , eu> mpnw . m ... i z A < .1111 ,.—. :11Hr: yawni— ..L.F ,un. uCfiOEtm .. c. 1,4 P -.._xaz. u. . NM Hus. . Ln): «.3..1.._Im m. c ii. tu> “m-h 94A '“Eni 0111} 1". 1| |I|||l dfi.c; “:.ML . q n_ —- 11 11 1111; . T. ... _ W . P ”A AM. u .11 1 111 1 v 1 "..fl a#.£ o ,M. __ .. i _ I‘erafi. VIM . _ _ >. ”.1H H 5.14.4 H.;..fiw Iagfuvmnm i >. u_1c .;.A a. _ 11 _ .1.. , . . ....vn—Canla . . ... . L. >1 u_IH 0m in. TABLE 16 GROUP PROPOSITION SCORES “01 IL’AJOSQO 0.10” p()i-)N SJUJJvJ [IV saflu15 [IV lqfiyj )0 Junomv Ungssonnng finttxqvns wiyIn alppsw Joauoid .91 3H311 go nunowv .9 luflri J() )UIunuv .I Ju311 J0 Junomv ainqxaL {iog auazuoa DIUEBJO {103 181111 ;o Junomv uoxiwniglnslng KJISJJAIQ saisadg u01382119193dg aqagN Anyxatdmog 93M P003 95 -—u v-l u—‘ N 00 -¢ m 1——— —-._4F—— p—fi in ———~ — — ——~—TV—— - >———- -————-.——4—1 — -——-— -«H———-—————4r~- — —-‘-—————_ >———-- —.— k —1 Ln '4 .— u—i 0-1 m N o N m rx 'II‘ ,_‘ u— d 0. -—¢ -—1 O x? M u—d ---I -—-0 N N N m f") co m ('5 -—¢ N —I v-l H —l M F! C N v—l .n A —4 --4 N N v-G H \D N § N H v-d N v-fi I—l 1-1 0-1 0-1 a-q v-‘l —1 O N N N O v-1 ('5 v-i N IN Q \1 . M N ,_. o—c \O m N m M M --4 u-c N —I 0-4 0 M ‘1 --d N ~D l‘ . C % w 1 ---4 .o «A O u L- tu U! U K In 'H w H c 1 x m 3 0 u u w m 0) U 0 w a.» r-4 :2 0 -v-4 La £3 «I H E U 'U D- D U U 0 o 3.. u H U 0 E O U 9’ > v.4 u to v-i D 0 O E "'4 Q74 m m U ‘ U m m U —J m D 96 The list of propositions students assert is found in Table 17. The group proposition scores are recorded in Table 16. Thirty-two individuals consistently asserted true propo- sitions about the relation of food web complexity, niche specialization, species diversity, and stratification. One to nine individuals asserted a false proposition about the relation of the concepts; The majority of the false assertions was about species diversity, food web complexity, and niche specialization with most of the false assertions involving niche specialization. In addition, a total of eight individuals (including stu- dent B) made an incorrect assertion about niche specialization and food web complexity, but only four were consistent in making the false assertion. Also, in item seven, none of the individuals made a false assertion about food web complexity and stratification, but in item ten, four individuals made the false asser- tion about food web complexity and stratification when the relation was in the reciprocal form. The student responses indicate difficulty with value- value correspondence and variable correlation relation- ships. Their responses indicate knowledge of the value of a variable during a stage and knowledge of the covariation of a variable and succession. Based on the data received, however, knowledge of the propositions appears to be the source of difficulty rather than the 97 TDABIJZ l7 PROPOSITION LIST FOR GROUP DERIVED FROM TEST-ITEM PROPOSITION MATRIX 10. ll. 12. 13. 14. 15. l6. l7. 18. 19. 20. 21. 22. Pioneer stage of succession has low amount of light at 1'. Pioneer stage of succession has simple food webs. Pioneer stage of succession has high species diversity. Low stratification is found in the middle stage of succession. Low stratification is found in the pioneer stage of succession. Climax stage of succession has low soil organic content. Climax stage of succession has low amount of light at 15'. Climax stage of succession has low niche specialization. Climax stage of succession has low food web complexity. As As As As As As (1 succession proceeds, succession proceeds, succession proceeds, succession proceeds, succession proceeds, succession proceeds, and 2 correct, but: not increase). As As As As As food web complexity food web complexity food web complexity food web complexity food web complexity A has more litter than food web complexity decreases. species diversity decreases. niches become narrower. stratification does not increase. stability does not decrease. amount of light at 1' does not decrease. As succession proceeds, stability does increases, species diversity increases. increases, niche specialization decreases. increases, stratification increases. increases, species diversity decreases. increases, niche specialization increases. B; therefore, A has low stratification. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. A has more litter diversity. A has more litter complexity. A has more litter A has more litter As As As As As As As As As As As As As stage with As stage with As stage with As food web complexity food web complexity food web complexity stratification stratification stratification stratification stratification stratification stratification stratification 98 TABLE l7--Continued than B; therefore, A has higher species than B; therefore, B has higher food web than B; therefore, B has higher stability . than B; therefore, need more observations. decreases, decreases, decreases, decreases, decreases, decreases, decreases, decreases, increases, amount of light at 1' decreases. increases, niche specialization decreases. increases, stability decreases. stratification increases. successional process increases. species diversity decreases. the amount of light at 1' increases. soil texture increases. succession reverses. food web complexity increases. species diversity increases. stratification decreases, food web complexity decreases. species diversity decreases, the greatest impact is on climax high niche specialization. species diversity decreases, the greatest impact is on middle moderate stability. species diversity decreases, the greatest impact is on pioneer low species diversity. species diversity decreases, the greatest impact cannot be told from information given. As stratification decreases, the amount of light at 1' increases. As stratification decreases, the amount of litter increases. 99 TABLE l7--Continued 45. As stratification decreases, the organic content of soil decreases. 46. As stratification decreases, species diversity (herbaceous) increases. 47. As stratification decreases, all factors increase. 48. As stratification decreases, need more observations. 49. As succession proceeds, species diversity increases. 50. As succession proceeds, stability increases. 51. As succession proceeds, stratification increases. type of relation. 5. The final list of propositions for 85 percent of the group is as follows: 1. Pioneer stage of succession has low food web complexity. 2. Pioneer stage of succession has little strati- fication. 3. As succession proceeds, niches become narrower. 4. As succession proceeds, stability does not decrease (it increases). 5. As succession proceeds, species diversity increases. 6. As succession proceeds, stratification increases. 7. As food web complexity increases, the amount of light at one foot decreases. 100 While performing this analysis of group data, more emphasis was placed on the test proposition matrix than the proposition score matrix. The proposition score matrix was used to indicate proposition scores which were high or low. The test proposition matrix provided a richer description of those propositions in question. The 85 percent cutoff point for listing the proposi- tions that students would assert is an arbitrary point. It is selected on the basis of confidence that a sizeable majority would assert the listed true propositions. A dif- ficulty in using the group data is that the composition of the group represented by the number in the cell can change. The utility of applying these procedures to group data will be explained further in chapter five. Discussion The developed procedures do organize the proposi- tional knowledge tested by items representing a test on terrestrial succession. Student item responses can be interpreted in terms of a set of propositions presented in the test and the implicit student assertions can be ana- lyzed. The developed procedures allow one to ask and answer, "What propositional knowledge does a student indi- cate by his responses to multiple choice items?" The answer is that list of propositions which can be generated from the described student matrices. From these matrices and the subsequent lists, conceptions of the propositional 101 knowledge tested can be pinpointed. Data concerning the validity of the information obtained by employing the devel- oped procedures will be presented in the next section. Essay’Analysis The modified propositional analysis techniques (de- scribed in chapter three) were applied to the twenty-seven essay responses. For illustration, Table 18 represents the analysis of student B's essay response. The tasks, which were the basis for the question, and the questions are restated prior to student B's response. Establishing Intra-judge Reliability As indicated in chapter three, eight analyses were repeated to assess intra-judge reliability. Intra-judge reliability was determined by calculating the percentage of agreement between two separate analyses performed by the investigator of eight selected responses. The percentage agreement was established by dividing the number of equiva- lent propositions (m) by the total number of propositions-- equivalent (m) and nonequivalent (mm)--and multiplying by 100. The percentage agreement was calculated for each list- ing of propositions (translated and inferred): §gf¥_Ymm x 100% = % agreement Table 19 displays the data. Column one represents the equivalence of the first set of translated propositions; column two represents the 102 TABLE 18 ANALYSIS OF STUDENT B'S ESSAY RESPONSE Column 1 The more species that there are (as species diversity i55reases) the more complex the food webs will be (food web complexity increases) If you modify the amount of light then you will decrease the amount of plants that will row (modify the amount of light, species diversity decreases) (Decrease the amount of plants that will grow therefore kill- ing some of the Species which IiVe off those plants (species diversity decreases, food web complexity decreases) The food web will become less com’lEX’Tfood web complexity decreases) Because you are eliminating some of the plants (as species diver~ sity decreases) then the content of the soil is about to change (the organic content of the soil changes) Column 2 Column 3 As species diversity increases, food web complexity increases (Increase/decrease) the amount of light, the species diversity decreases (Increase/decrease the amount of light, the food web complexity decreases) As species diversity decreases, the food web complexity decreases As species diversity decreases, the organic content of the soil changes Essay Question: On the back of your answer sheet, describe how niche specialization and food webs are related during ecological succession. Explain your answer. Describe how you could modify one of these characteristics. Indicate the stage in which your modification took place. Explain the effect of your modification on the other char- acteristic. Explain how that might effect the stability of the area. Task Descriptions: Task 3--Given: a set of charac- teristics which change during succession; Required: rule describing relation of characteristics to other charac- teristics in a set across stages of succession. Task 4-- Given: rule describing relation of characteristics to other characteristics in a set across stages of succession; Required: prediction of the consequences of modification of particular characteristics to set of characteristics and successional sequences 103 TABLE 19 RESULTS OF ASSESSING INTRA-JUDGE RELIABILITY % agreement Column 1 Column 2 Total per student Student m mm m mm m mm 1 10 0 1 5 ll 5 69 2 17 l 4 2 21 3 88 3 6 O 4 l 10 l 90 4 5 O l 0 6 O 100 5 5 3 2 3 7 6 54 6 4 O 2 2 6 2 75 7 l3 0 3 4 l6 4 80 8 7 2 l 3 8 5 62 Total 67 6 18 20 85 26 % agree by column 92% 47% 76% 77%* *Rounded percentage agreement/student equivalence of inferred or multiple transitive propositions; column three represents the totals for each student; and column four represents the percentage agreement for each student. The percentage agreement for each column is pro- vided at the bottom. The data for the analysis performed on the same stu- dent responses by one judge presented in Table 19 indicates 104 consistent translation of the student responses (column one). Column two indicates the consistency of judgment decreased for the inferred propositions. This indicates caution is required when one is attempting to go beyond what is actually stated to what might have been meant. Establishing Inter-judge Reliability Eight responses were randomly selected from the twenty-seven analyzed essay responses. After a brief train- ing session, a second judge analyzed the responses using the procedures described for propositional analysis of written responses. These were compared to the investigator's analy- sis of the eight responses. Table 20 displays the results of the comparison. TABLE 20 RESULTS OF ASSESSING INTER-JUDGE RELIABILITY Column 1 Column 2 Column 3 Column 4 Student Judge 1 Judge 2 m mm Judge 1 Judge 2 m mm Judge 1 Judge 2 m mm m mm 23 7 5 2 8 6 8 4 6 6 3 1 7 12 11 12 2 1 O 3 2 3 2 1 l 0 0 1 3 3 24 9 6 3 9 8 7 5 5 11 7 2 14 10 28 Totals 56 37 19 55 49 44 31 31 ll 28 6 47 65 115 2 Agree 41? 67? 202 511 105 The columns represent the parts of the analysis. Column one is the initial proposition breakdown and trans- lation of the students' responses. Column two represents the translated propositions derived by each judge. Column three represents the inferred propositions deriVed by each judge. Each column is divided to indicate the number of propositions derived by the judges (the first judge = l; the second judge = 2), the number of equivalent propositions (m), and the number of nonequivalent propositions (mm). The total for each column is listed at the bottom of the table. Column four represents a summary of the total data analysis per student. The figures in column four do not equal the sum of the figures in the three previous columns, as some propositions were placed in a column by one judge and some propositions were placed in another column by another judge. The data are summarized at the bottom of Table 20. The question to be answered by this comparison is, "To what extent can someone follow the essay analysis procedures and obtain the same results?" The question stated more quanti— tatively becomes, "On the average, how often will two judges agree in the analysis of the essay responses?" Therefore, the formula used for columns one through three is: Number of propositions matched 100 Average number of propositions derived by the two judges % agree = Column four's percentage is based on the number of total matches divided by the total number of matches and total number of matches and mismatches present: 106 Total number of propositions matched a . . x 100% Total number of propOSitions present % agree = These numbers are obtained by counting the number of equiva- lent propositions in the total analyses and the number of nonequivalent propositions. For columns one through three the percentage agree- ment means: given the average number of propositions derived, the judges agree 20 through 67 percent of the time. Column four percentage agreement means: of all the propo- sitions derived, the judges agree 53 percent of the time. In general, the overall percentage agreement (53 percent) indicates the judges agreed slightly more often than by chance if there were only two possibilities. However, given the range of possible propositions, 53 percent is not con- sidered a chance occurrence. The change in percentage agreement across the various listings of propositions indi- cates some inconsistency in the analysis. However, column two indicates fairly high agreement in listing the translated forms of the student responses. This is interpreted as meaning the judges are in close agreement as to what the student has written. Also, the rules for translation and recording are able to be followed by a judge other than the investigator. Reviewing results of the analyses and discussing the results with the second judge, explanations for columns one and three were obtained. The major difference in the results for column one was the breakdown of the student 107 response. One judge formed the students' propositions with only two concepts and a relation. The other judge wrote the statements as the student had, including several con- cepts. This contributes to the difference in number of propositions derived and therefore to the number of possible equivalent matches. In column three, which was used for listing inferred or completing transitive arguments, a greater discrepancy occurs. One judge inferred more general statements (vari- able correlation); whereas, the other judge was more inclined to infer specific statements (variable-stage relations). In addition, several differences in the con- cepts inferred from the students' statements were recorded. These factors and the results further indicate caution is advised when extending what is stated to what might have been meant. Summary The applied essay analysis procedures have provided a list of propositions from the student's response to an essay question. The essay propositions can be compared to the multiple choice propositions. The intra-judge reliability estimates indicate that: given the data to analyze, on the average 76 percent of the analyses will be equivalent. Providing the rules and procedures used in essay analyses to another judge indicates that the same data will 108 be obtained 53 percent of the time and a higher agreement can be expected for the translated propositions. Therefore, the results of these procedures indicate the list of derived propositions are in a form that can be used to assess the validity of the multiple choice propositions. Validation of Multiple Choice Propositions The first section of this chapter described the results of the propositional analysis of students' multiple choice item responses. The second section described the results of propositional analysis of students' essay responses. This section describes the results of comparing the two sets of data. The marked cells of the matrix in Figure 9 represent the pairs of concepts addressed by the multiple choice items (me) and expected to be addressed by the student's response to the essay question (e). No one student is expected to indicate, in response to the items, all of the indicated pairs of concepts, but for the group of students these pairs of concepts could potentially be addressed. In addi- tion, several other pairs of concepts could be addressed in response to the essay item. Of forty-five pairs of concepts addressed by the multiple choice items and the fifteen that could be used for a response to the essay question, eleven pairs of concepts are common to both. The comparison was based on judgment of the equiva- lence of assertions concerning the same pair of concepts 109 Pairs of concepts addressed by multiple Figure 9. choice items and expected pairs addressed by essay item 110 obtained from each information source. Equivalence was defined as an identical relationship or a reciprocal rela- tionship. If a student asserted in one case that one characteristic would increase as another increased and in the other case said the same characteristic would decrease as the other decreased, and the truth value of the two assertions was the same, the assertions were considered reciprocal and therefore equivalent. Comparisons were pos- sible only when the student made assertions about the same pair of concepts in both sources. Table 21 represents the matrix used to record data from each source for a student. The example student is student B. The multiple choice propositions are recorded in the left-hand segments of the cell and the essay proposi- tions are recorded in the right-hand segments. Codes in the cells indicate the asserted relations between the concepts. Table 22 presents the results of the response com- parison‘for each student in the sample. The comparisons represented in Table 22 have been divided to indicate type of relation compared, source of essay proposition being com- pared (translated or inferred), total propositions present in each source, and the number and type of propositions not compared from each source. The percentage agreement for item comparisons was determined by dividing the number of matches (equivalent) by the total number of comparable propositions (matches and mismatches). Table 23 presents the results for the 111 TABLE 21 PROPOSITION SELECTION MATRIX FOR A STUDENT'S MULTIPLE CHOICE RESPONSES AND ESSAY RESPONSES K_ez: I = increase d = decrease I,d = As X increases, Y decreases H H reverse I;I = two assertions of the same pair of concepts 5‘ H has INA nuouutn so.-:-uu:m s..._n-.m s—vgz totem... .a. .zn.u ac :5! .n “An..— uc :35 .3 use“; we :55 ...—gunk :3 52:5 nus-n5 _ “ow Eluauczrfiuw 3.3.43: :31; Ely-«:53: it; $133.59 A2; ‘00; Stratifi- nun Diversity Spec“- V. t x e W C Food "lb organic MICE! Sal]. future Alum: 0! Lil!!! 15' Subfllty Sell 112 TABLE 22 RESULTS OF MULTIPLE CHOICE AND ESSAY RESPONSE COMPARISON FOR THE STUDENT SAMPLE 152$ * = indicates inferred propositions which were ascribed to the individual and not match a = indicates inferred proposition did not match b = indicates student term not translated VS = Variable/stage VV = Value-value correspondence VCO Variable covariation VCR = Variable correlation MS = Match same § Match reciprocal MM = Mismatch 111A I? Number Number Total Total 1E MC No 1350 Student Hitched 1.‘1.'1tc.2'1ed ifCumbcr not 31./15111 71/131 1U” C015 COWPJYiN’m ITOSC No. Identical ' Reciprocal 11-1atc‘ned translated Iinferred Essay 1Comparison VS W VCO VCR VS \’\’ VCO VCR‘N’S W \’(20 VCR 2133-13 MN .115 MR MM VS W VCO VCR‘VS ‘.’\' VCO VCR 6 1 ' 1 1 1 1 1 1 2 20 1 3 '1 1 5 1 9 “L 1 | ‘1‘ 1 1 1 1 .11 1 5201.2 ‘13115111 17 3 2 12 1 o 1 1 0 12 20 3*/43 11516 15 13 2 1822 10'1 18120 2 5*/531 5 5 12 g 2 2 0 0 3 . 20 1* 3 1 4 10 ———- . . — ~——~:~ + 24 _212 . :13 111 19'20 32125*/231 3 8 23 1 1 1 2 1a 2 '2 1 11 20 2 1*24/13 1 4 7 1 1 ' g_rgu_p of def. __ -- ;_ 1 . -_.__1. 19 3 l 5 10 18 4 g 2 .' 4 1 1, 11 20 1 1.1/33 1 5 5 1 ..... l . r . 8 , 4 1 . 2 2 1 ' 10 20 3 1 5 6 19 ,’ 11 ‘2 1 12: 1 1 1819 3211/11 3.31 2 9 : —4>——.—-—-+—-—- ' 1 ..---.11- l 1 1 I 20 1' 2 3 1 3! 1 3 1 221917. 124/52'1 3 5 -- . 1 fr“ ' 1 fl 4 . 1 1 1 1 ; 1 8 20 4 2 3 1 5 10 4 — —--: ..-+—~—«- % ——0— 21 :2 2 1 '2 1 1 ’ 2 6 20 1 3 1 5 5 ‘ «>- L ...... __ .. 22 . 1 ' i 1 5 20 ‘1 . 3 1 ' 5 10 : ; ' 5—1— ; + . -+——1 1o 13 i 11 131 1 11020 1143114 8 _-~—~- % 1 --~-— ‘ ; i ' i 26 1 : 2 ' 1 1 2 1 1 6 20 3 3 1 5 9 ! . 11 1 1 I l 1 1 2 l 0 . 12.17 111/3 1 4 2 4 8 27 4 1 3 1 1 1 3 - 4 . . :1 ”13420 1 4 /9 3 1 5 . 6 14 ' 1 1 ' 1 1 1 1 : | 5 18 3 3 5 1 8 13 12 12 112‘ 2:1 1020 4314 9 16 1 2 12 2 1 11 11 17 10*/2§ 3 7 7 7-b 1 13191 3 818 1 313.1 4 8 5 2 2 ‘ 2 2 0 15 201211 85311 5 7 1 _L I 1 I I I 3 . 1 .3 11 5 7 1 . e— 1 i 2 3 1 2 . 1 3 2 0 22 17 1 3 13 3 11 2 4 1 J ' __» | I 1 I TOTALS 2 15 43 13 28 : 11 42 24 7 18 7 4 2721468 25 14 16 112 76123 113 198 f ; . — ‘ 1 1 — J. ; 1 1 1 l J - l 1 1 1 1 I 1 I l 1 I 1 5 . 1 I 1 ° - ' l g I i 1 1 f 41 I 1 : 1 ; : - -1 i 1 : p 1 1- 1 ' % 1 ‘ | 1 1 1 1 1 I 113 TABLE 23 PERCENTAGE AGREEMENT BETWEEN COMPARABLE INSTANCES OF MULTIPLE CHOICE PROPOSITIONS AND ESSAY PROPOSITIONS Translated Inferred Total Comparison Comparison Comparison 90% 86% 89% comparison of translated propositions, inferred propositions, and all comparable propositions. The percentage of agreement between propositions selected from the multiple choice items and comparable propositions from the essay response indicates a close cor- respondence of assertions. In 89 percent of the cases, where comparisons were possible, the students asserted equivalent propositions in both their multiple choice and essay responses. Of the 468 propositions from.the multiple choice test, 102 were comparable to propositions for the same pairs of concepts from the essay test. The frequency that a pair of concepts was addressed by the students' multiple choice responses and essay responses is displayed in appendix E. Summary The results of application of the developed proce- dures for this study have been presented in chapter four. The sample for analysis was twenty-seven students enrolled 114 in Biological Science 202 at Michigan State University. Application of the multiple choice analysis produced twenty- seven matrices organizing propositions students selected on a twelve-item multiple choice test. This information was compared to the propositions derived from analysis of the students' essay question responses. The essay prepositions were derived using a modified version of an established technique for propositional analysis. The results of the essay analysis were tested to establish the reliability of judgments made by the investi— gator (overall percentage agreement being 76 percent) and to determine if the analysis procedures could be used by a person other than the investigator (overall inter-judge agreement being 53 percent). These results also indicated that when a judge was making inferences about what the student might assert, there was a decline in agreement between cases (47 percent for intra-judge and 20 percent for the inter-judge). The comparison of the two sources of information indicated the propositions the student asserts on the multi- ple choice items were equivalent to those propositions (containing the same concepts and relations) asserted in the essay response (89 percent agreement). A slight decline in agreement of inferred propositions was detected (86 per- cent). The procedures developed organize a conceptually simple, but lengthy, list of propositions contained within 115 a set of multiple choice items. Once the propositions are displayed on the matrices, they provide an indication of the extent propositions are tested. When the student responses are recorded on these matrices, an indication of the students' conceptions and misconceptions is obtained. These matrices then provide a basis for inferences regarding a student's (or groups of students) beliefs about the subject matter. Chapter five will discuss these results and the implications in more detail. CHAPTER V DISCUSSION OF THE RESULTS AND CONCLUSIONS Overview This chapter begins with an overview of the entire study. Next, the results and conclusions of the study are discussed. A more general discussion follows which indi- cates possible extensions of the developed methods and potential problems. The chapter concludes with a discus- sion of implications for practice and further research. Overview of the Study The purpose of this study was to develop, apply, and evaluate techniques for the analysis of student knowledge as demonstrated by the propositions the student implicitly asserts on multiple choice items. The techniques were applied to multiple choice items from a part of a course for which a systematic, detailed representation of the subject matter was available. The subject matter representation was in the form of a semantic network. What was lacking was an expeditious assessment of the students' propositional know- ledge. Tests of knowledge are designed to assess students' knowledge of propositions (Ebel 1965). Conventional 116 117 analysis of multiple choice items did not provide the set of propositions the student believes to be correct. Pre- viously developed techniques of propositional knowledge assessment were considered too costly in terms of time needed for large numbers of students. The objectives of the study were to: 1. Develop procedures to organize the propositional know- ledge tested in a test and apply them to a test on terrestrial succession 2. Develop procedures to analyze the propositional know- ledge implicitly asserted and interpret student item responses in terms of the underlying set of propositions presented in the test 3. Compare the information received from a subset of stu- dents' multiple choice item responses to that developed using the proposition generation technique for assessing propositional knowledge in order to assess the validity of the item analysis technique Chapter two provided an expanded discussion of con- ventional and propositional knowledge assessment. Proposi- tional knowledge was defined as knowledge of concepts and their relation to other concepts and the truth value (true or false) of the relationship. The set of propositions to be used was derived from an analysis of literature about the process of terrestrial succession. Knowledge of these propositions can be determined when a situation arises requiring that knowledge to be used. To describe these 118 situations, a concept—task framework developed by Smith (1973) was employed. The task descriptions were selected from Smith's variable—value network, since most of the con- cepts in the propositions represented variables that characterized the process of succession. The task indicated the information provided to the student and the information required of the student--an asserted proposition. The resulting task descriptions were used as a basis to con- struct each of the items used to assess the propositional knowledge of the students. Chapter three described the procedures used in this study. The propositions included in the test were organized in a test-item proposition matrix that was used later to record student item responses. The test-item matrix was collapsed across rows to record the frequency of proposi- tion occurrence and to calculate proposition scores for the student. A series of questions were described to ana- lyze the recorded student item responses. The questions provided guidelines to characterize the propositional know- ledge of a student at the time the test questions were answered. To assess the validity of the information received from the multiple choice analysis, Pines's (1977) proposi- tional analysis procedures were expanded and employed to analyze students' essay responses. The propositional analysis procedures required reliable judgments for each analysis. Therefore, the judgments of the judge were tested 119 by comparing the analysis of eight duplicate responses. To determine if the procedures for the essay analysis were reliable, another eight analyses were performed by two judges and compared for agreement. Finally, procedures to establish the equivalence of propositions between the essay and multiple choice tests were described. Chapter four presented the results of the study. From the test-item proposition and proposition score matrices for each student a list of propositions was gener- ated. The questions for analysis of student response generated a brief description of the student's beliefs. The application of the procedures to group data provided a characterization of the group's beliefs. The essay analysis provided a set of propositions to compare to the multiple choice propositions. The agreement of judgment for intra- judge reliability was 76 percent; the judgments of two judges agreed 53 percent. The agreement of comparable mul- tiple choice propositions and essay propositions was 89 percent. Discussion The discussion of results is organized according to the three objectives of the study. Objective 1 The first objective for this study was to develop procedures to organize the propositional knowledge tested in a test and apply them to a test on terrestrial succession. 120 As described in chapter three, procedures were developed for succinctly representing relatively large sets of propositions. The procedures employed a matrix formed by listing each of the concepts from the instructional unit along the rows and columns. The cells of the matrix repre- sented sets of propositions about the corresponding row and column concepts. In total there were 225 cells represent- ing propositions. Procedures also were developed to derive the propo- sitions tested by the multiple choice items. The item.and foil numbers were recorded on the proposition matrix for the unit. This provided a representation of the sample of propositions tested relative to the potential propositions that could be tested. Some of the potential propositions that could be tested were considered irrelevant. Also, the matrix with recorded item foil numbers did not provide information about the relation within the tested propositions. As described in chapter two, four types of relations could be found within a proposition represented by a cell; therefore, procedures were developed to organize the propositions tested by the test and make the relations explicit. This organization was the test-item proposition matrix. The test-item proposition matrix was formed by list- ing the stem concepts along the rows and the option concepts along the columns. The cells of the tested propositions were divided into segments indicating the truth value, the 121 relation, and the foil number for the tested proposition. This matrix provided an organization for the conceptually simple, but cumbersome, list of propositions found in the twelve multiple choice items. In total, sixty-two proposi- tions were derived from the twelve multiple choice items. In addition, more than 166 propositions not tested in the twelve items were illustrated by the empty matrix cells. The test-item proposition matrix represents 228 possible propositions. This number can be doubled when the truth value of a proposition is considered. The number can also be increased by a factor of four when the type of rela- tion to be found in a proposition is considered. By using the developed procedures, it is possible to represent suc- cinctly a large set of propositions that could be tested by the multiple choice items for this unit. Several of the sixty-two propositions derived from the test and recorded on the test-item matrix were repeated propositions. The same proposition was tested by more than one item. To obtain a representation of the frequency a proposition occurred, the proposition frequency matrix was developed. This matrix was formed by listing the stem con- cepts once along the rows and option concepts along the columns. The cells of the matrix were divided into two segments representing truth value. The frequency a propo— sition occurred was recorded by truth value in the cell representing that proposition. This procedure provided a summary of proposition occurrence by truth value. The 122 matrix did not contain relations but did indicate how often a proposition was tested. The procedures developed to meet objective one provided an explicit representation of the propositions being tested and their frequencies. To summarize, these procedures accomplish the fol- lowing: l.l a representation of relatively large sets of propositions, 1.2.1 the derivation of propositions tested by multiple choice items, 1.2.2 a representation of the sample of propositions tested relative to the potential propositions that could be tested, 1.3 an organization of the propositions tested by the test with relations specified, and 1.4 a summary of proposition occurrence by truth value. Objective 2 The second objective for this study was to develop procedures to analyze the propositional knowledge implicitly asserted and interpret student item responses in terms of the underlying set of propositions presented in the test. Procedures were developed for efficiently generating sets of propositions consistently asserted and denied by students in their responses to multiple choice test items as described in chapter three and applied in chapter four. Recording the 123 student item responses on the test-item proposition matrix provided an organization for the list of propositions which the student could be viewed as asserting to be true. In addition, student choices among all propositions contained in an item represented on the test-item proposition matrix provided an indication of what the student was asserting to be untrue. From this matrix, the list of the student's assertions was generated. After the student responses were recorded on the test-item proposition matrix, the proposition scores (sum of proposition assertions compared to the frequency of proposition occurrence) were computed and recorded on the proposition frequency matrix. This matrix provided an indication of how often the student could be viewed as asserting a proposition. Once the test-item proposition matrix and the proposition frequency matrix were formed, the recording and scoring of students' foil selections were routine. Follow- ing the guidelines for interpretation of the recorded selections and scores, the student's responses were analyzed and a characterization of the student's propositional know- ledge was obtained by following the described guidelines. Dividing the proposition scores by truth value indicated contradictory responses which were made explicit by the test-item proposition matrix. When propositions occurred more than once, further evidence of an individual's propositional knowledge was 124 available. If a proposition was true and always selected by the student, then the consistency of responses was con- sidered as evidence the student believes the proposition to be true. When a false proposition was selected consistently by the student, a misconception was indicated. In general, if selected propositions contradicted, the inference was made that the student was confused and the situation was considered indeterminate (what the student actually believed or knew could not be determined). Following the guidelines for interpretation of recorded responses also provided information regarding the student's understanding of the types of relations presented in the propositions. By recording the relations on the test-item proposition matrix with the student responses, it became possible to make inferences about the student's understanding of the relation. It is possible the relation §Ed_the conception of the proposition caused the student's confusion. The final product of the interpretation procedures was a characterization of the student's propositional know- ledge and the conceptions the student had of the proposi- tions in the test. This characterization was represented by the answers to the guideline questions and the final list of propositions. The foregoing generalizations about the analysis of student knowledge need to be tempered. When the proposition scores were constructed, the student received credit for all 125 true propositions in the item if the correct option was selected. The credit was assigned to simplify the computa- tion of proposition scores. This arbitrary rule kept the total number of proposition occurrences constant across individuals. Otherwise, the scores across individuals would not be consistent. When the final list of proposi- tions for the student was written, these credited proposi- tions were deleted. By following the developed procedures, it has been demonstrated in chapter four that a list of propositions can be generated characterizing the assertions of each student or group of students. The list can be generated verbally or written from the test-item proposition matrix. That list can be organized and compared to the total set of proposi- tions made explicit within a test. Propositions represented by cells of the test-item proposition matrix with empty segments or totally empty indicate propositions not assessed. A student's response pattern could be determined if an individual compares the student answer sheet to the test questions item by item. The point to be made is that the techniques used to analyze multiple choice item responses in this study provide a succinct representation of the test, the items, and the responses. With the developed organiza- tion, the investigator can quickly and efficiently analyze the responses and explicitly discuss the assertions of the student or groups of students. 126 In addition to providing feedback to the student or groups of students, the investigator can employ these proce- dures and obtain feedback regarding instruction. With this organization of test responses and following the guidelines for interpretation, the investigator can detect areas of the propositional knowledge being tested that students indi- cate a misconception. Then the investigator can make course revisions where necessary or compare instructional sequences addressing the misconceptions. Before expanding on the' preceding conclusions, the validity of the obtained repre- sentation of student beliefs provided by these procedures requires discussion. In summary, these procedures accomplish the follow- ing: 2.1 an organization for the list of propositions a student implicitly asserts in response to a set of test items, 2.2 the frequency which a student asserts a propo— sition, 2.3 a set of assertions that can be attributed to a student or a group of students, 2.4 an indication of the students' conceptions and consistency of those conceptions, and 2.5 information to provide feedback to students and to enhance course revision. 127 Objective 3 The third objective of this study was to assess the validity of the item analysis by comparing the students' set of propositions received from the multiple choice analy- sis to those propositions developed using techniques for assessing prepositional knowledge. Of the 468 assertions obtained from the multiple choice analysis, 102 comparable propositions were obtained from the essay analysis, of which 91 were judged equiva- lent. The percentage agreement between the two lists was 86 to 90 percent. The agreement of 86 to 90 percent between the list of propositions obtained from the multiple choice test and the list obtained from the essay question supports a claim of validity for the information from the multiple choice analysis; that is, the propositions implicit in the students' choices on the multiple choice exam were congruent with those stated in response to the essay question. This supports the claim that the list of propositions derived from the multiple choice test represents portions of the students' propositional knowledge. However, each technique generated propositions for which comparisons were not possible. Therefore, the evidence for agreement is restricted to that set of propo- sitions elicited by both techniques. The following diagram may clarify the statement. 128 ’7 Total Set of Propositions Possible Propositions Multiple Essay elicited Choice Propositions by Propositions both techniques The diagram illustrates the set of propositions that could be tested, the set of propositions possible for the essay, and the set of propositions contained in the multiple choice items. As previously indicated, the cells of the test proposition matrix (Table 1) represent over two hundred sets of propositions. Each cell is formed by crossing the row and column concepts. The propositions within the set represented by the cell are dependent on the type of rela- tion contained within the proposition. This matrix provides an organization for all propositions within the unit on terrestrial succession. Several propositions within the matrix are not interesting for purposes of investigating, instructing, or testing about terrestrial succession. Of the over two hun- dred cells, approximately ninety propositions are interest- ing. These ninety interesting propositions represent the domain of knowledge to be addressed in a study of terres- trial succession, and they are indicated in the diagram as the total set of propositions. 129 The multiple choice items address thirty-nine pairs of concepts within this domain of ninety propositions. Potentially, the response to the essay question could address any of fifteen pairs of concepts within this domain of ninety. Of the fifteen pairs of concepts to be addressed by the essay, eleven pairs of concepts are addressed also by the multiple choice items. The frequency which pairs of concepts were addressed by the students is displayed in appendix E. The agreement of the information developed by the two techniques is based on the area of overlap between the essay and multiple choice techniques. The overlap is esti- mated to be about 12 percent of the total set.1 The overlap contains only those propositions that contained equivalent concepts. The area covered by the multiple choice is esti- mated at 43 percent of the total set; the area covered by the essay is estimated at 16 percent of the total set.2 While the area of overlap is estimated to be small, the percentage of agreement in that area is very high. The overlap could be increased by increasing the "coverage" of 1Estimate based on ninety propositions of interest and potentially eleven of those are found in the overlap. 2Data for this study indicate that 38 percent of the essay propositions matched 22 percent of the multiple choice propositions. The figures presented here are based on the total set of propositions of interest and not all those that could be found. There is a possible set of over 200 propo- sitions in the unit on terrestrial succession; several are unreasonable. 130 the multiple choice test (adding more items) about proposi- tions covered by the essay. Alternatively, the essay could be broadened in scope to increase the number of propositions elicited in answers. The results from the comparisons of analysis high- lighted a key feature of propositional analysis. The match of inferred propositions in the essay answers to multiple choice propositions was lower. The match of propositions for inter-judge reliability was low before the listing of propositions in standardized language was required. The match of inferred propositions for both the inter- and intra-judge comparisons was lower. The findings suggest that considerable caution be taken when attributing state- ments to a person and going beyond what actually is said to what appears to be implied. This is especially true when the opportunity for follow-up questioning is not avail- able. Summary Procedures have been developed for succinctly repre- senting relatively large sets of propositions tested for in items of a multiple choice test. These procedures not only allow the representation of the set of propositions addressed by a given multiple choice test, but also a larger set of which those tested are a sample. Procedures have also been developed for efficiently generating sets of propositions consistently asserted or denied by students responding to 131 multiple choice test items. Recording student responses on the test-item.proposition matrix and the proposition frequency matrix provides the investigators with a succinct representation of the assertions being made by a student or group of students. With this representation, it is possible to obtain an indication of the student's propositional know- ledge with respect to the item and the test and provide explicit feedback regarding the student's conception of the subject matter. Also, the developed organization provides the investigator information for instructional improvement. Evidence has been obtained supporting the validity of the proposition lists representing portions of the students' propositional knowledge. Therefore, the primary conclusions of this study are that lists of propositions can be gener- ated from multiple choice items and that the obtained information is valid. The purpose of the study was to develop, apply, and evaluate techniques for the analysis of student knowledge. The study has achieved that purpose. A procedure has been developed that provides an economic method for determining the propositional knowledge of a stu- dent at a partiCular point in time. This procedure is considered to be an extension of existing methods of know- ledge assessment. The procedure provides a means for analysis of students' propositional knowledge in courses with high enrollment. The extent of these findings is limited by the fairly small sample of propositions that can be compared. 132 In addition, the sample of propositions is not representa- tive of all types of propositions. Also, some problems appear in the essay analysis as indicated by low percent agreement of portions of essay analysis. But, given the amount of information obtained and the high percentage agreement, these limitations are not viewed as major prob- lems. The development of the matrices used in this study was straightforward because the set of propositions had been derived by a systematic analysis of the subject matter to be instructed and tested. The type of relations to be tested was known and described by the selected task descrip- tions before the items were written. Once the items were written, it was a routine task to organize the information with the matrix and to record student responses for analy- sis. Therefore, use of the developed methods may be restricted to those instances where the intended outcomes of testing are known in detail, a priori. If the prior comprehensive analysis of the subject matter for instruction and testing has occurred, the efforts to employ the devel- oped methods are definitely worthwhile. It is possible that the effort will have a positive effect on clarifying what the test developer wishes to assess. Whether the developed procedures can be used for post hoc test analysis is a question that requires an answer. This study has demon— strated the test analysis can be conducted following systematic organization of subject matter. Further studies 133 are needed to determine if the procedures can be applied to post hoc test analysis. Extensions of the Developed Method The previous section indicated that propositions can be derived following the procedures. With the preceding summary statements in mind, the following section outlines some possible benefits the developed methods offer to test analysis, item analysis, diagnosis of student conceptions/ misconceptions, and models of student knowledge. Test Propositional Content Analysis In chapter three, the knowledge domain for this study was organized with a proposition matrix. Table 1 was formed by listing the concepts from a unit on terrestrial succession along the rows and columns of the matrix. Each cell represented a set of propositions about terrestrial succession. The proposition matrix displayed in Table 24 will be used to illustrate the use of a prOposition matrix in constructing a test. Table 24 is similar to Table l, but does not have the items indicated within the cells. Table 24 has the concepts from the instructional unit listed along the rows and columns of the matrix. Each cell represents a set of propositions for the unit. Certain cells represent propositions of little or no interest. These cells are indicated by an "x." Other cells represent propositions of high interest, i.e., they represent 134 TABLE 24 PROPOSITION MATRIX FOR TEST DEVELOPMENT IS'M c7-_:_..>I_.....£Q. 7 7 _ . -d 7 7 7 7 7 chit «9...... 7vI . T 7II II II . , -I- III” IIII7I. I I7! 7. . - I- _ 7 I 7 - _ . 7 - - 7 7 l I 7 7 7 7 7 .-I II _ 7 7 7 7 accuzg :< 7 7 7 7 7 71I| - - - -III . I I I? II +_II II I a II-II I - I . 7 VII I, II I 7 _ . 7 7 7 7 7 . 7 7 7 1.41....“ _—< _ 7 7 III IIII+ IIII . ..... . I _. III I I I . .7 I. . II II II I ..... 1 I I 7 7 7 7 H III II II I I 7 7 7 7 . 7 _ 7 7 fix: a _ n I a 7 a b 7 x x I I 7 b 7 b b 7 n a I .c unrrfiwa. 7 7 7 7 7 | I a r II A 7 II III IIIII I II II II I II la 1 . . 1 I III II III I I $7IIII . I I . I I I4 II I 7 7 _ _ . . 7 . a I I 7 I A b b 7 b 7 b b I x I 7 I 7 I 7 7 C 71.1.7.3...“ 7 _ 7 7 . IIIII I VIII I . r II I III _I IIIIII . I III I p III IIII 7I I I 7 .7. _ 7 _ 7 7 7 a 7 7 7 I I _ _ 7 7 7 a I I a a b 7 b 7 b 7 b 7 b I 7 a a x 7 7 7 _ 7.;7:L..7.7 7 7 77III II . . IIIIII I .II II I v I I II v I 7II II I I I + I . IIII‘ II 7 7 7 7 7 _ 7 7 . a a a 7 a a b . b b 7 b b I x 7 x 7 7 7 aft—“7 7 _ 7 7 7 I I I . v I I 7 a _+ 7 Ir 7 I . I . a II I .- 77 _ 7 . _ 7 a 7 a a _ a a b b b 7 b b I x 7 I _ 7 _ ..7 7.7.. 7 7 7 7 7 I I. I. . v I I VI II . . «I II I n . 7 o .7 I. a 7 7 7 7 _ n n 7 .- n a b b b b b x 7 7.715;?— 7 7 7 I II I II II I 7‘ I I74IIIIII4 I I I 7I IIII 7‘ . . . I I I III.T 7 I... I I .7 7 7 7 7 7 1 I b b b b I 7 _ 7 7 7 ...7 777.37 b 7 _ 7 x 7 x x 7 7 .7. “c7733. 7 7 7 _ 7 I II IIIa- I-q _ . . I IIII . . -7 1:7 7 7 _ _ ... Ex: .0 7 b b b 7 b 7 b 7 x x x 7 7 . T. 7:77.57... IT . . 77 7 7 7 77 .7 I- . - 7. - 7 l -- 7. 7 7 7. II- I- I I T . II 7 7 . b b b b b b x 7 7 7 . .7 Eu: 7 7 n 7.. “5:77.777. 7 I- 1.,- . 7 - ,,,,,,, . . - - . 7 _ 7 I . I I - 7 7 7 7 7 7 7 h 77 b b b b b x 7 7 . 7 ..ZLZ..._, HT}... 7? 7 - IIIIII7I I 7 II . IIII-II I III II I I II_I III I . . I . I I . III . I7oI 7 . 7 7 .ur..,;7,“.7.7 b .D b .D 7 b I 7 7 7 , 7 7 77:77.7: ”Tm 7 7 7 7 7 7 7 ..I . 7. I I. v . 7. IIIIIII 7- IIIII I . I . .I. 7 7 7 I n _ a a . u x 7 7 53:7 _ 7 7 .... Erik... 7 . 7 _ IIII- III a I 4 a 7 7 1:- 7 7 . I a 7 A X 7 7 22:. 7:377:77 . 7 7 7 7 I IIIIIV II . o I I I I III _ . 7 7 7 ~ 4 4 1 III III IIIIIII _ _ 7 7 7 7 7 7 7 93:72:: a a 7 x 7 7 7.7.. .1171 7 7 7 7 7 I» II ..o I r . I I« 4 I v 7. . I I o I I . I + 7 7 . 7 7 7 7 7 :o.:.~::o..:7m I I 7 7 7 7 7 7 7 7 7 IIIIIII+ v I I .:7 -_ 7 ..n 7 7 7 7 I 7 AF... Yach— ..7.:xu~afizu .- 7 _ | k _ 7 7 - III . - 7 I I: I:-. I I- - . - --7 - . n. 7 _ 7 v, _ t .2 f _ f. 7 v, 7 n 7 a an 7 u 7 In I o 7 7 o. o- .5 7 I 7 m I 7 e . _ .l 7 C! e l 5 l r 7 1 a 0 7 B r. 7 We an 0.0 1n I! In 7 r. t t L I Q 7 l I; I t u U 7 mm 7 l elo 1... to no. no 7 u n... n... nt 0 I. 7 .1 e. ... v _ d”. hCl 7 cc 3.... ut 1a.... 71! uh uh uh n d m b nu" S O r O Ctr. 9V rt Or. icon 1x 08 8 7 8 o d .1 a a .m. 7 _ r. Men _ 00 41p“ Pi. ‘3 M1 OTC 00 I. m4. m1 1. I. I. t u (A 7 u _ 1C (DOM FC NS: SD 5... L SOC ST ML L L P H C S S L A 7 M.” hm“. 135 propositions the students should understand, if they are to understand succession. These cells are indicated by an a. Some cells are important to the understanding of terrestrial succession. These cells are indicated by a "b." With this representation, the test developer has a visual aid to construct items and options. In Table 24, there are forty-six sets of propositions indicated as being important for understanding succession. Therefore, when writing an item, the developer can select from these cells the propositions that are to be included. By checking off the cells as the items are written, the test developer can track the cells that are covered and obtain an idea of the coverage of the test and whether the test is adequately testing the domain of knowledge. In addition, each option used in an item not only will provide information of stu- dents' understanding, but will provide information regarding their conceptions of the material being tested. A proposition frequency matrix will be used to illustrate the use of the proposition matrix to analyze a test. Table 25 represents a proposition frequency matrix for the entire test. The cells have been divided according to truth value and the numbers represent the frequency the corresponding proposition occurred. The first seven row concepts are those found as stem concepts in the existing test. The last nine row concepts are potential stem con- cepts. All option concepts are listed above the columns. 136 TABLE 25 PROPOSITION FREQUENCY MATRIX FOR A TEST ON TERRESTRIAL SUCCESSION The empty cells and half cells represent propositions that are untested. By scanning the matrix, it can be noted that most of the propositions tested concern stratification, species diversity, and succession. They have the highest frequency of cells marked. If the instructor wanted to improve items by replacing nonfunctional options within the test or increase the test length, the matrix indicates which proposition sets could be considered. For example, niche specialization appears in five different propositions (going down the column) with four other concepts. If niche specialization as related to other concepts needed to be tested, then the vacant cells offer potential propositions that could be 137 used. This matrix indicates the propositions that can be considered. The propositions could be used to form new items or to replace nonfunctional foils. A comparison of empty cells in Table 25 to the interesting sets of propositions in Table 24 reveals that the test used for this study did not address twenty of the interesting sets of propositions. These twenty sets of propositions can provide alternative options to the cells tested in the last three columns of Table 25. This would provide more information regarding the students' under- standing of the unit than allowing the students to respond to these particular non-informative options. Also, there are several proposition sets that tested only once and in the false sense. These sets could be used, as the above sets could be, to replace options that are nonfunctional. Only eight of the interesting sets of propositions were tested in both the true and false form of the propositions. The comparison of the frequency matrix to the matrix for test development offers valuable information for improving the test used in this study. As indicated in chapter two, Klopfer (1971) and Schmidt et al. (1978) have developed methods for analyzing tests. Klopfer's organization compares the subject matter topics to behavioral objectives which are addressed by a test. Schmidt's method compares mode of presentation, operations, and the content of the items found in a test. The present study has focused on the propositions found 138 within the test item and student responses to those proposi- tions. In doing so, the developed procedures offer a method to further define the subject matter context addressed by the techniques of Klopfer and Schmidt. The developed procedures succinctly delineate the knowledge being addressed by the item. As indicated earlier, this matrix represents over two hundred sets of propositions. The actual number can be doubled by truth value or quadrupled by type of relation included within the proposition; therefore, this matrix offers the test writer a multitude of propositions that could be used to form items for several test batteries. It also provides the test writer a means to ensure the test is testing the important propositions in a domain of knowledge. Multiple Choice Analysis Item analysis provides basic information regarding an item's difficulty and discrimination and the percentage of individuals responding to various options. By viewing an item as a set of propositions, additional information can be obtained depicting students' assertions of truth and for item.improvement. For example, the analysis of item three of the test provides the following information: Item 3. During late succession, which of the following would be observed? 1. low organic content of the soil 2. low amount of light at fifteen feet 139 3. low niche specialization 4. little complexity of food webs Item 3 analysis. (Correct option, 2; percentages are given, N = 42) Option 1 2 3 4 5 Omit Upper 27% O 100 O O O 0 Middle 46% O 90 5 5 O 0 Lower 27% 9 45 45 O 0 0 Total 2 81 14 2 0 0 Index of difficulty = 19 Index of discrimination = 55 Item 3 propositional analysis. U C1 C1 0 m 0H U — u a In >5 :6 o .4 U N U 'H .Dw4 -H 0&4 0% H OH (I! 3:m m «444 u r-l 0-H Go cu “OD. .110 mm :34: 08 .23. 2°... 0.30 £8 zm oo 5.: Climax T i 817 T = True ° F = False L = has low F .4 .3 .l L 2% L 14% I. 2% The propositional analysis of the test item indi- cates 2 percent of the students asserted climax stage has low organic content of the soil (false); 81 percent of the students asserted climax stage has low amount of light at fifteen feet (true); 14 percent of the students asserted climax stage has low niche specialization (false); and 2 140 percent of the students asserted climax stage has little food web complexity (false). A significant minority of the students tested have false assertions about the character- istics of climax, especially the value of niche specializa- tion for the climax stage. If an instructor wanted to decrease the misconcep- tions made apparent by this analysis, he could then focus on those misconceptions and organize the instruction accord- ingly. If the instructor wanted to improve the item, he could develop Options that would reflect likely misconcep- tions rather than simply considering the distracting power of options. Which alternative conceptions and also which relations to address can be determined by the pattern of responses obtained from the propositional analysis of the entire test. Diagnosis of an Individual Student's and the Group's Knowledge of the Subject Matter In chapter four the guidelines for interpreting student responses were employed. Using these guidelines, a characterization of the student's propositional knowledge was obtained. This characterization of the student was based on a preliminary list of propositions, proposition scores, an indication of consistency of response to concept pairs, a consideration of response to types of relations, and a final list of propositions. These characterizations can be expanded by considering conceptions within the 141 propositions. To make these inferences, the test-item matrix is used. The focus of this discussion is the con- cepts and task involved. Table 26 represents the student's test-item proposition matrix. This student had a test score of ten out of twelve. By searching the matrix, the answers to the ques- tions can be determined. As noted before, the student had a misconception about the relation of stratification and soil texture; however, the specific relation appears only once. But if item twelve is considered, it can be noted the student again has trouble with stratification. Except with the latter, the relation is formed with the "amount of light" and then "all factors." The question becomes, "Is the problem with stratification or perhaps the task?" In previous items the student correctly asserted or denied propositions about stratification. For task four the stu- dent was correct for only one out of three times. Also, for other tasks, the student had no apparent problem; therefore, it can be inferred that the student has trouble with the task which involves modifying characteristics for succession. Given his score it would be assumed the student knew the subject; but given the information above, the student has difficulty with identifying the consequences of a pertubation (task four). In the section on item analysis, it was noted that a significant minority of students had difficulty with the relation of niche specialization and climax stage. The 142 TABLE TEST-ITEM PROPOSITION Key: Relation Symbols h = has hi = high 5 = simple 1 = low 1i = little pr = proceeds mo = moderate d = decrease 26 MATRIX FOR STUDENT B narrow increase negation of relation ternl therefore reverse during stage correct option HIA Hu_u ac ~tvwa:.£.« «u_“ a< ~-m~«;n»- apniflu;: Hu_n u< aonuwaox ulm:nmeau whi: HINn coo cum:r(awl "v.03 auan ’Ll t 1 i .H u . a i . Tii. i 7.7 .i, .i . ii 7 7 _ 0 ii 7. 7 . 7 i T 7 . 7 7 7 7 7 2 3 i7 _ _ .7 _ H .i'Ll r , t I i i V - 7 7 7. In .13 7 i . . _ _ . _ , .... 3 . 3 7. ~7.7 . .7 v . i v .i ii. i riii iiii . 7 . . [ ii? i . . . i i v 7 7 7 .7 7 _7 7 _ 7 v~.u a. Illir 1|i.|07..w .9. ill? ..I . 7 ....7M 7 7 7 7.4—.77 7 7 7 vi...iiviii¢ 7 7 Ca.“ _ _ 31.175 iili ii i 7 7 7 7 7 7 _ i 3|» 1. __ 7 _ . . 7 . iiii .iliiill7.7m . . i,iiiiii,i i. . _ 7 7 u 7 7 a . 7 .7 .77 7 _.7 oi .I i I.) n..|o.|'1| i .. L.H 7 i ii... viii. iii. « 7 7 i. i fluvi . . Q a 7 “7.1.7 . .ii i .ilii : i. 7 .x 77 7 _ _ . 7 7 7 7n.7 7 7 I i. ll [tifll .7» C _ u.~ . i i. . i . i\¢ii i‘i . :iL| i . 7,7 _ _ _ ... _ hi .4 .. I ilo _ w 7 145 Item 3 false value narrows CIUQ h ‘ / Climax as 13 Niche Specialization - Y property ’9 v.» (\ 9.0 O 1.)!) 73:8 hi8 mu 0‘90 903 NEH: use «up 00"! E CU O‘Chr-‘i oxa won U E aw HU.>- UX'U <‘ u Hwy \‘t Food Web Complexity - X ‘1: 'J U: Q '1) 3 u M Ix“ '1) '19 U a c: a o E.— 00 F4 0'0 2,: u 3 u.4 U UN [5 0" H34 an 0! 6'“ w a,” H H On . < >‘ m ‘ < Item 10 s X decreases Stratification \Species Dlvcrqity - X Y decreases true Figure 10. Concept map of a student's conception of niche specialization 146 decreases, food web complexity increases." This analysis of the student could continue. These misconceptions were illustrated further by his response to the essay. The student provided a group of definitions (student nine, Table 23) which did not answer the questions. The student does appear to know about the charac- teristics of the pioneer stage and the change in variables as succession proceeds. The student's score on the entire test was six correct out of twelve. The score alone indi- cates the student has difficulty with the subject matter tested. The analysis used illustrates that it is possible to be specific about the student's assertions; an instructor can detect contradictions in the student's assertions; and the instructor has an indication of where to begin ques- tioning the student. This particular student has difficulty with the niche specialization and food web complexity relationships and their relationships to other characteris- tics. The matrices and analysis procedure can also be used for group data. Table 28 lists the number of students selecting an option. This group data has been discussed in chapter four. In chapter four a list of true propositions was generated for 85 percent of the students. Those propo- sitions represented a set which a large majority of students were willing to assert. A list can also be gener- ated that is false and represents a set of propositions students are incorrectly willing to assert. The list that 147 TABLE 28 TEST-ITEM PROPOSITION MATRIX FOR GROUP ERRORS Key: Relation Symbols h = has n = narrow hi = high I = increase 3 = simple - = negation of relation terni l = low I. = therefore li = little r = reverse pr = proceeds du = during stage mo = moderate * = correct option d = decrease l‘l'lA Fond Hob Funpluxity N‘- ht" Spi'r in i I r Itinn Diversity Stratlilultiun 5pm I v91 {If Anuunt Litter Soil Organic Cunlont Soil Texture of leht l' Ahuunt of Light 5' nf IS' Annun ( Am-mnl light Pioneer M l dd I r‘ Climax stability Surress Inn Sngcs All Fortnrs All Nl‘t‘d Mn Wu Observatiun "I 7.1.7... ' .- D a a|_n -< u 0 7vmwo=ao~ a1_u ac _ Numnnan»- 7 n»nun.c= Hn—n u< mun—”max .v 7.7» Hn~u ~7 7. «nun aocsn on r7nnn~ A: 27 ....37 77...... : ML. 73.. Olmccm (an. noavumn- ~n< Hip" .—._ 148 15 percent (six or more) of the students are willing to assert as true when they are actually false is as follows: 1. Climax has low niche specialization. 2. As food web complexity increases, niche specialization decreases. 3. If a field has more litter than another field, then need more observations to determine other characteris- tics of the field. 4. A decrease in species diversity has greatest impact on climax stage. 5. A decrease in species diversity has greatest impact with high niche specialization. 6. A decrease in species diversity requires more observa- tion to determine the impact. To illustrate the misconception, a concept map (Figure ll) for niche specialization will be used. The numbers indicate the number of students asserting the indi- cated relation for the indicated item. Niche specialization and food web complexity appear present in most of the students' incorrect selections. The number of students falsely asserting these relationships indicates instruction needs to better address these concepts and their prepositions. Also, as the test progresses across the task dimension, student errors increase. The students begin to make contradictory asser- tions for the variable correlation propositions and indicate the item does not provide enough information for a judgment 149 Low 8 High 0 Niche f u v 1 o Uldth e w m H or: a m 7 7 g g 2 0'3: “DH 0 min. 2 C a u m ‘ G3 5 —* > u c g 2 H a“: L- H E maul- @ c H m "" < As S proceeds Climax has as Niche Specialization - Y Succession property Y increases True '1) Ga"! .0 f0 Q'J'I 0“?“ 032:: u “ ° ’ r . glib—4 "3'_‘ CB“ C "‘ A. E E a u : C 3 C x r “ 9 H... m 2:: < VIC. E0 Nam H Food Heb Complexity . X ,5 g E F‘Q 4.: EC L4.) uh Hdi 20$ 9 u x qmnm o 4"19 '1 - $0: 4 O'CLL-lq 05 u—oA-l'doa: h UCQL‘. U 50"!) 0 on u o x r V t—nm Stratification - Y Species Diversity - X Figure 11. Concept map of group's conception of niche specialization To determine if these students are the same requires the student's individual analysis. However, if the data were computerized, it would not be difficult to select out those responding to a particular set of options. 150 to be made, in particular the value-value relations. This indicates more instruction is necessary on these tasks and the relations they signify. Models of Student Knowledge' .Application of the developed procedures provides a set of propositions the student asserts or denies. This set of propositions can be in the form of a list of a matrix representation. The list and matrix represent models of the students' beliefs at a particular point in time. An 'alternative model can also be generated--a concept map. As indicated in chapter two, concept maps have been used in research to compare students' knowledge of subject matter content. These maps can also be formed using the data obtained from this analysis. To form the map, the concepts are listed and circled with the relations indicated on the lines.. Following are examples of the three models that can be made from an item. The models can be expanded to include the information obtained from the entire set of questions. The models are based on item ten; the student selected option two. The models include all propositions presented in the item. A comparison to what the student asserted and denied can be made with each of the models. 151 Proposition Asserted +As stratification decreases, succession reverses +As stratification decreases, species diversity decreases -As stratification decreases, soil texture increases List Denied -As stratification decreases, stratifica- tion increases -As stratification decreases, succession increases +As stratification decreases, food web complexity decreases -As stratification decreases, food web complexity increases -As stratification decreases, the amount of light at one foot decreases O Am0unt of Light ‘ - fl - - ‘ — ...—— \ / / o I a \ \ / l \ ‘i+l ngzllgffiy \1' \ \\ ‘\\ \ 0 \ \ I / / / //7/_ 152 G o -H m u g >. :4 ‘3 8 B u u. p. .H -H x .Dw4 044 u '-H a) a) an: cu raw-1 'r'i co [-4 30 H man u m r4 GJJ -H H m 0 r4 vocm as: o o H o -H 08 Obi) 03> L3 D o c>o Era cud co a: a: FnL) 7; car: 2 () .2 1 .3 0 HT .5 1 .5* o .4 o 5 0 Stratifi- d,r d d,I .d cation ' F .l (J .1 () .2 1. 3 O d,I d,I d,I d,I Key for Map and Matrix: /\ d = decrease Ldy= selected concepts in map I = increase r = reverse 0 = selected relations in map The models provide a representation of a student's conceptions and misconceptions. The map and the matrix are preferred, as they provide a picture of a student's beliefs about a decrease in stratification and the effects on species diversity, soil texture, and the process of succes- sion. While two of the propositions the student selected were true, the proposition dealing with soil texture was false. To determine the extent of the student's false beliefs, analysis of other items is needed. This analysis can be done by either expanding the maps or the matrix. In addition, these models can be used for group data following the same format used for an individual. 153 Summary The previous section has indicated some potential uses of the developed procedures--test analysis, item analysis, diagnosis of student assertions, and concept mapping. These uses may be applied to individuals for groups. The following section provides recommendations for further research and practice. Implications for Research and Practice Research The previous sections have indicated that a more explicit definition of what an item or test is assessing is possible. In the present study, the propositional know- ledge that was to be learned by the students had been carefully specified before the test was developed. This characteristic of the present study was definitely an asset. Although it limits the generalizability of the results, the prior analysis facilitated the entire process of developing the procedures. It is recommended that the application of the pro- cedures to other types of propositions be examined in courses like BS 202, where the propositional knowledge to be learned has been specified. If the efforts have already been made to specify the propositions to be learned, the employment of these procedures requires a comparatively small effort, and the information received provides 154 productive instructional implications. Were this unit taught again, more emphasis would be given to niche specialization and food web complexity. Further research should evaluate application of the procedures in analyzing existing tests not designed from the explicit representation of propositional knowledge. The procedures have the potential of explicating the coverage of a test. And given tests are extensively used to classify people, it would be interesting to know specifically what the test says people know and do not know. These suggested types of analyses could provide information that would further enrich the findings of research focusing on aptitudes of students and the effects of instruction and studies of the ecological framework of classrooms. Coupled with their findings, questions regarding when and for whom did changes occur, what changes occurred, and what was the situation that brought about these changes could be answered in greater detail. Another area of research that can be facilitated by applying the developed procedures is the assessment of cognitive structure. Coupling these procedures with the techniques of clinical interviews and propositional analy- sis of written responses provides more information with regard to the propositional knowledge possessed by students. The developed procedures provide a quick and effi- cient indication of knowledge. Using that information, 155 interview or essay questions could be generated. The procedure would indicate the foci for interview or essay questions. These additional questions could validate or add to the information received from the multiple choice procedure. Also, the information would greatly improve the representations of knowledge obtained by any of the preceding procedures and the developed procedures can be used with large data sources. In addition to the struc- ture of student knowledge obtained by the developed procedures, the interview questions could also focus on the processing of information with that structure. Practice The suggestions for research have importance and application to instruction. Given a background in tech- niques available, an instructor can trace a student's changing propositional knowledge base during instruction. For example, a pretest could be administered and analyzed. This would provide baseline data regarding the students' entry level propositional knowledge. The instructor would then be able to modify instruction accordingly. As the students complete instructional activities, quizzes could be given and analyzed for the propositional content. This information would allow the instructor to monitor the modifications during the instructional unit. At the end of instruction, the posttest could be administered and analyzed also. A comparison of the sets of data could 156 indicate when a misconception has occurred or been remedied. These analyses could then be used to modify existing instruction, look at the instructional effects on certain individuals, and design subsequent instructional activities. In addition, the information received will enable the instructor to provide students explicit feedback regarding their knowledge. Rather than stating that the student does not understand the material, the instructor can inform the students of what it is they do not under- stand. It is not suggested an individual instructor per- form all the analyses as have been done for this study. It is suggested that the preceding recommendation be conducted by a team of instructors and researchers. Once the developed procedures are computerized, they can be used in courses with large enrollments. The concepts and relations can be coded and the codes could be part of the regular output of test analysis. This would be of benefit for those courses where propositional know- ledge obtained is of great importance. An instructor would be able to modify the focus of lectures, discussions, and other activities to address the propositions that appear to be difficult. The multiple choice analysis would benefit a single instructor. The analysis is simple to follow and the information received suggests instructional emphasis. With slight modification, these procedures can be employed with 157 a true-false test and fill-in-the-blank test. These sug- gestions are not meant to indicate that essay questions should be omitted. The procedures can facilitate analysis of achievement tests presently used. It is recommended that attention be given to essay tests and the procedures to analyze the responses. Caution is advised when ascribing a proposition to a student during essay analysis. It appears that when student statements are extended beyond what is actually written, there is a tendency to make weak assumptions about the assertions of the student. This problem could be rectified by a closer adherence to the rules provided or omitting the extension of what is written. Conclusions The purpose of this study was to develop procedures that would organize the propositional knowledge tested and selected by students within a set of multiple choice items. The purpose has been achieved. It is possible to charac- terize the students' beliefs using the procedures developed in this study. The achievement of the objectives allows questions to be asked about the propositional knowledge of students who have responded to a traditional type of know- ledge assessment instrument. The importance of prior comprehensive analysis of the subject matter to be learned cannot be denied. Without that analysis, the objectives and purpose of the study may not have been possible. The 158 benefits of that analysis coupled with the described tech- niques for organizing the information in a test and student item selections are very rewarding. The benefits are worth the time and effort, especially for important educational goals. They provide succinct representations of the relation of the knowledge domain, the test, and the student assertions. With propositional knowledge being one of the main interests of education, more attention should be devoted to the analysis of subject matter. In addition, with assessment instruments being extensively used to evaluate learning or potential to learn, more attention needs to be addressed to the analysis of the assessment instruments being used. This study has developed procedures to analyze the multiple choice test. Although the test for this study is based on a systematic analysis of subject matter, the developed procedures definitely imply a method to analyze an existing test. APPENDICES APPENDIX A SEMANTIC NETWORK FOR THE TERRESTRIAL SUCCESSION INSTRUCTIONAL UNIT 159 Key for Appendix A: As X :.‘Y = as one variable changes there is a corresponding change in the other variables indicated along the periphery As proceeds, X increases = as succession proceeds, a variable will increase X decreases, succession r = a decrease in a variable X results in a reversal of successional trend has X = during designated stage the value of a variable will be As X .a T'= a directional change in one variable will result in a change of the amount of light in the opposite direction APPENDIX B INSTRUCTED PROPOSITIONS FOR EACH TASK 161 APPENDIX B Instructed Propositions for Each Task Task I Propositions Pioneer has high amount of light at 1 foot poor soil texture low low low low low organic content of amount of litter species diversity stratification species diversity, specialization low species diversity, webs low species diversity, webs, low stability Middle stage of succession has 5 feet 15 feet soil low stratification, therefore, low niche low niche specialization, low niche specialization, low amount of light at 1' moderate amount of light at 5' high amount of light at 15' loose soil texture richer organic soil content greater amount of litter increase in species diversity increase in stratification increase species diversity, increase stratification, increase niche specialization simple food simple food increase species diversity, increase niche specialization, increase food web complexity increase species diversity, increase niche specialization, increase food web complexity, increase stability Climax stage of succession has low amount of light at 1' low amount of light at 5' low amount of light at 15' loose, fine soil texture rich organic soil content high amount of litter high stratification high species diversity 162 complex food webs high stability Task II Propositions As succession proceeds, the amount of light at 1 foot decreases the the the the the the the the soil texture increases soil organic content increases amount of litter increases stability increases stratification increases species diversity increases niche specialization increases food web complexity increases Task III Propositions at 5 feet decreases at 15 feet decreases As species diversity increases, stratification increases As the amount of light decreases, soil characteristics increase As niche specialization increases, species diversity increases As niche specialization increases, food web complexity increases As amount of litter increases, soil characteristics increase Task IV Propositions As food web complexity decreases, species diversity decreases As stratification decreases, the amount of light at one foot increases APPENDIX C TEST B 163 APPENDIX C Test B In an early successional stage we would find 1. low amount of light at herb layer. 2. simple food webs. 3. high species diversity. 4. low amount of light at fifteen foot level. Little stratification would be characteristic of 1. late stages of succession. 2. middle stages of succession. 3. early stages of succession. 4. all stages of succession. During late succession which of the following would be observed 1. low organic content of the soil. 2. low amount of light at 15 feet. 3. low niche specialization. 4. little complexity of food webs. As succession takes place 1. food relationships become simple. 2. species diversity decreases. 3. niches become narrower. 4 . the amount of light reaching the herb layer increases. Which of the following indicated trends is incorrect for succession? 1. Hi 2. Hi strati— organic fication matter in soil _w Lo Lo Early middle late Early middle late 164 Hi 4. Hi Stability Amount of Light Lo at herb Early Middle Late layer L0 L A Early Middle Late 1 Early Middle Late In the above graph, the x on the vertical axis could represent? LBJ-\wNH species diversity. stratification. stability. a and b, but not c. a, b, and c. In succession, if feeding relationships become more complex, then one will find 1. 2. species diversity increasing, niche specialization decreasing, and stratification increasing. species diversity decreasing, niche specialization decreasing, and stratification increasing. species diversity increasing, niche specialization increasing, and stratification decreasing. species diversity increasing, niche specialization increasing, and stratification increasing. species diversity decreasing, niche specialization increasing, and stratification decreasing. A student studying succession observed two fields (A and B). He reported that field A had soil with more litter than field B. We might also infer U'l-l-‘UONH field A supports a lower degree of stratification. field A supports a larger number of populations. field B supports a more complex food web. field B supports a very stable community. we need more observation before we can make inferences about the field. 165 9. complex feeding rela- tions simple early middle later If the above trend is true for a given woodlot, we also should expect 1. decreasing amount of light at the surface. 2. decreasing niche specialization. 3. decreasing stability. 4. decreasing organic matter in the soil. 5. decreasing soil organic. 10. Due to the energy crisis all the trees that were medium size and short—lived were harvested from the old fields on campus. Which of the following would occur after a few (3-4) years? 1. There would be an increase in stratification resulting in a speeding up of the successional trends. 2. There would be a decrease in species diversity and an increase in soil texture thus reversing successional trends. 3. There would be an increase in food web complexity caused by the decrease in species diversity. 4. There would be a decrease in the amount of light at one foot and therefore an increase in species diversity. 5. The trend of succession would reverse as indicated by the decrease in species diversity and the presence of less complex food webs. 11. A chemical was developed which destroys herbaceous plants in twenty- four hours. This chemical was released in an area of Tennessee with a lot of fields, young woodlots, and old woodlots. Which of the following areas and reasons are correct for describing its greatest impact? 1. the old woodlot because of its high niche specialization. 2. the young woodlot because of its moderate stability. 3. the field because of its low species diversity. 4. cannot tell from the information given. 12. During old field succession, as the organic content of soils increase, we also observe an increase in the amount of litter, the amount of stratification, and a decrease in the amount of light at the herbaceous layer. If during the middle stages of succession we reduced the amount of stratification, we might also expect 166 1. increased amount of light at the herb layer and increased amount of litter. 2. increased amount of litter and decreased amount of organic content. 3. increased amount of light at the herb layer and an increase in the herbaceous species of the previous stages. 4. the only consequence would be related to light; all other factors would continue to increase. 5. need more information to properly determine the consequences. On the back of your answer sheet, describe how niche specialization and food webs are related during ecological succession. Explain your answer. Describe how you could modify one of these characteristics. Indicate what stage your modification took place. Explain the effect of your modification on the other characteristic. Explain how that might effect the stability of the area. APPENDIX D ESSAY ANALYSIS PROCEDURES 167 APPENDIX D Essay Analysis Procedures I. Extracting the basic proposition From the text of the student's answer, copy the student's propo- sition-relation of concept A and concept B. Substitute pronouns with their referents, and make phrases complete sentences. II. Translating to a standard form As the student's propositions are listed, substitute standard concepts and relations for the student's concepts and relations. Underline the word or words being substituted and place the substitu- tion in brackets. III. Listing translated propositions List the propositions in standard form in order of occurrence. IV. Adding implied propositions Scan the list; list other propositions the student implies given the propositions he did state. Instructions There are four types of propositions being sought. These are listed in order of importance. 1. Variable correlation: As X increases, Y increases 2. Variable covariation: As X proceeds, Y increases 3. Value of variable at a point in time: Stage X has high Y 4. Variable value correspondence: High X goes with high Y. The first three correspond to tasks describing what information the student is given and what is required of the student. The fourth 168 can be described in terms of a task, but is not an expected proposition. The first type of proposition is the type requested by the question the student is answering, and it is expected to be the most frequent propo- sition in the text. Step I: Copy the student's answer forming a numbered list of propositions. The propositions are to contain only two concepts and their relation and must be independent statements. When the student forms a compound relation, the first concept listed will form the first concept in the set of propositions that are extracted. Example 1: As N occurs, X, Y, and Z increase. N is related to X, Y, and Z forming three separate propositions. 1. As N occurs, X increases 2. (As N occurs), Y increases 3. (As N occurs), Z increases Note the parenthesis; whenever a substitution is made, it is surrounded by parentheses--single indicate the student's words, double indicates the recorder's words. Double parentheses will be used when a pronoun is supplied with its referent and when a phrase is made into a complete sentence. When the student forms a transitive argument involving three concepts in a sentence, form all three relations. If there are more than three concepts, form the initial three relations and list the rest to complete the multiple transitive argument in step four. When listing transitive arguments, place the translation term between the initial proposition and the explanatory proposition: 169 Example 2: Transitive argument As A increases, B increases because C increases. 1. As A increases, B increases . . . because . . . 2. As C increases, A increases 3. As C increases, B increases Example 3: Multiple transitive argument As X then Y because Z which means A thus B. 1. As X, Y . . . because . 2. As Z, X 3. As Z, Y 4. Z which means A 5. A thus B Step II: Translation involves underlining a word or words and inserting a bracketed concept. Example 4: As N occurs [proceeds], you have more X [X increases] Below is a list of concepts to be used as substitutions and examples of concepts they replace: Concepts Food web complexity Niche specialization Species diversity Stratification Amount of light Amount of litter Soil texture Soil organic content Pioneer Middle Climax Succession Stability Increase Examples complex food webs feeding relations specialized niches broader/ narrower lots of organisms different of diverse species levels, layers brightness, darkness debris, leaves, dead things loose soil, moist soil, packed soil rich soil, poor soil early, first, young stage middle stage late, last, old stage stable more, higher, modify '170 Decrease less, remove, reduce, modify (modify dependent on text. What other variables do when modify occurs) Proceeds continues, occurs Has there is, have or - Step III: List all translations as standardized propositions (As X, Y) and indicate what X and Y are doing to obtain a complete proposi- Example 5: Standard form by type of proposition 1. As X increases, Y increases 2. As X proceeds, Y increases 3. X has high Y 4. High X goes with high Y Step IV: From the list of propositions in Step III, list other propositions that the student implies. Indicate with a bracketed G (general proposition relating two variables) or S (specific proposition involving values of variables). Example 6: Step III Step IV As X, Y [C] As X, Z As Y, Z Also complete multiple transitive arguments from step I. Step I and Step II Step III Step IV As X, Z As X, Z As Z, Y As Z, Y As X, Y As X, Y Z which means A As Z, A As X, A A thus B As A, B As X, B 171 Step IV As X, A As Y, B As Z, B Format for Recording: Divide paper into three columns. In column one, perform Steps I and II. In column two, perform Step III. In column three, perform Step IV. Rules: Step I: Step II: Basic proposition contain two concepts and a relation. Transitive arguments containing three concepts in a sentence are completed. Transitive arguments containing more than three con- cepts or more than one sentence are listed but completed in step four. Compound propositions are listed by initial concept then subsequent related concepts. Some concepts do not indicate variability; to stan- dardize concepts, rewrite as a variable. Example: food web becomes food web complexity; niche becomes niche specialization. At times, responses may not specifically indicate the concepts intended. When this occurs look at other mentioned concepts to determine the context, then supply the standard concept for unspecified concept. Example: Climax has lots of organisms [high species diversity], several organisms to feed on [high food Step III: Step IV: Diagnosis: 1. 172 web complexity]. The context may be found in the same sentence or in prior or subsequent sentences. This condition also occurs with "modify." 3. When the student says "all factors," list only those explicitly mentioned. 1. The order of concepts within a proposition does not matter. List in any order that is comfortable to say. 1. The propositions listed must be based on previously listed proposition. Normally these take the form found in the example. These are listed because if the student were confronted with the proposition, the student would agree to its truth value based on another concept previously mentioned. 2. List only those propositions the student has provided concepts. 3. If a misconception of a concept or proposition occurs, then implied propositions involving those concepts are excluded. Did the student provide: 1. 2. a relation of niche specialization and food web complexity an explanation of the relation an indication of how to modify niche specialization, food web complexity or another variable an indication of when modification took place 173 5. the effect of the modification on niche specialization or food web complexity 6. an effect of the modification on stability? Which type of propositions does the student depend on for explanations? What type of propositions may trouble students? What concepts appear difficult or are avoided? What definitions or views of concepts are provided? What problems may the student have? APPENDIX E FREQUENCY A PAIR OF CONCEPTS WERE ADDRESSED BY STUDENTS' ITEM RESPONSES 11‘] —11 I I I _ I _ I I I .1111 , .1 ..... 11 - 1 11. t.1 . . I 111 I I I I I I I I N I I I I I I 1‘ 11‘. e ‘ 11 1II I I I I I _ I I I I _ I I I I I I I I I I I _ ,I I I I _ I _ - 11IT 1:11. .1 ,. . I .I 11 _ I I I . I _ I I ~ _ I , I _ _ I I . , I I I I I 1 [I’v1‘ll 1 i ‘ 4 11 v11 1 o 1 I I I ‘1‘ 1‘ 1‘ |||II b “ _ . I a a. r I _ I 1 I I I I _ _ I . I I I I _ I I I _ I _ , I 111111: - . I 11 1.1111111e1111111h 1. 1111 m 1 1 - I I I I I . a a I n I I I I I I I . I I , I I I I I I I . I I q I : I I .I I I I I N _ I I I I 11! . 11111I111111I ....... b11111111111111L11.1 1 1111 I11 1. w T1111 1 .1 1 1 I I I I I I s I N I I I I II II I I I.-11 I. . . I I I I I I a I I I , I «N I _ I I I I I I I _ _ o 1 1101 A ‘m‘ 1% ‘1 . o 11‘ 1 cm . I I I N I I I . . ‘ 1I.‘1+: ‘I1 1 I I .I m I I - I I 1 o 1 [‘111 I 1111 ‘110I II 1110.11 151 I I I - _ I I I I I I I 11.4 111111 J1 I I. ‘1 11 11 u 1 1 1 111 11LI1I #I11111 I .I e I I I I I I _ I I I I I I I a . I I I I I a I I _ I I1 1 ---1I1 I111 _ 11 1 m x w n n x n 1 m m .... In n m m. I q I 1 3.1 . 1 1 w , ~-~ a . up m 1 A s u w I w u H m 1 w r u r m m u a n . n x a m 1 z 1.: 1 h. n. L m u 1 3 (. u . u 1 I ... u a 1 1 A 1.. J U J .\ U U N i ! .—_.-...- —— no 1 1.. 1 1.... l1“ 71”§ 111? lllv‘IIlu '. 1 .\1-1]‘.\[ Jluvfiln 131111 II__I,I 3(1111 luv ‘ fl. U Ha31r3:11)r115 ‘ _ luck «0 couuchuoo vanquua> passes: m I ao«u«uuaaoo I . uasaaz I ucuucou I o«:_tco ~_:m coda neuaaaauuam I ozouz _ Auditsbua nodowam I hang“; no uc306< uuuxogaeou 993 mean m~ . codaaouusw 1 1 1 uoy)w711rxw\dg c nwlfiau coduau 1Icau