DlFFERENCES [N LEARNING AND ATTIATUDES OBTAINED 7 FROM TWO METHODS FOR FEEDBACK OF MID - TERM f EXAMINATION RESULTS Thesis for the Degree of PhD. MICHIGAN STATE UNWERSHY JOHN B. BALCH ' 1969 11’1“}. A__ u ——-—A- ._4 — --—'- IJBRARY Michigan State University “*7— This is to certify that the thesis entitled Dififerences in Learning and Attitudes Obtained from Two Methods for Feedback of Mid-Term Examination Results presented by JOhn.B. Balch has been accepted towards fulfillment of the requirements for Ph.D. degree in Education Date' April 17, 1969 0-169 Dfirn;:' -1971 . J 5 fi‘ "1 .. R54 , zub’l—O'W‘Vn-w Zwr , A; 1"," ACKNOWLEDGMENTS The author wishes to eXpress his appreciation to the members of his guidance committee, Dr. Walter F. Johnson, Chairman; Dr. Harold w. Grant; Dr. William E. Sweetland; Dr. Donald L. Grummon; and Dr. Donald J. Freeman. Particular reSpect and admiration is eXpressed for Dr. Walter F. Johnson, whose continued support, interest and friendship throughout the entire program of doctoral studies and related research proved to be a consistent source of motivation. . To my wife and parents this study is dedicated, without whose love and kind understanding it might never had been brought to completion. ii ABSTRACT DIFFERENCES IN LEARNING AND ATTITUDES OBTAINED FROM TWO METHODS FOR FEEDBACK OF MID-TEEN EXAMINATION RESULTS by John B. Balch This study had three purposes. First, using as a control group students who listened to a televised explanation of test results, it was to deter- mine whether students who discussed their answers to an objective test (a mid-term) with each other in small groups scored better on a subsequent objective test (a final examin- ation) than the television group. In the first two parts of the study, student scores on repeated test items and their scores on the total final examination were investigated. The second purpose of the study was to determine whether there was any positive relationship between students' final exam scores and their tested tendency to participate in discussion. Third, it was to determine whether there were any reSponse differences on an exPloratory attitude questionnaire relevant to the purposes and content of exam feedback when administered to different students assigned to the discussion or television group. Design and Procedure of Study The learning eXperiment was conducted with 166 students in the Summer Term and the attitude questionnaire was adminis- tered to 787 students in the Fall Term of 1967. Analyses of covariance were conducted to evaluate the eXperimental treatment effects of students' discussing or watch- ing TV. Scores on students' tested tendency to discuss were classified as above or below the mean. The interaction of these levels of discussion tendency were analyzed with the treatment effect of having or not having the opportunity to discuss mid- term test items. Students indicated their attitude reSponses on a five point scale of agreement. The Mann-Whitney U statistic was used in analyzing students' reSponses to each of the thirty- one questionnaire items. The problem of objectively classi- fying the questionnaire items was met by factor analyzing the attitude scale. Findings Although the mid-term exam scores were equated in analyses of covariance, no significant differences were found on final exam scores and their sub-parts between groups of students discussing mid-term exam results and students watch- ing a televised presentation of these results. Of the thirty- one items eXploring attitudes about examination feedback listed in the questionnaire, eighteen items revealed signifi- cant differences between the groups. Twelve of the items favored the instructional effectiveness of the discussion treatment and six of the items favored the television treat- ment with respect to student agreement on the questionnaire. When the questionnaire was factor analyzed, three of the items which had significantly favored the discussion group loaded on one factor. The discussion group agreed that (1) the feedback session stimulated them to interact with other students, (2) they felt personally involved, and (3) the session was not formal. This factor seemed relevant to the rationale of this study, namely that the discussion group more actively participated and was more personally involved. DIFFERENCES IN LEARNING AND ATTITUDES OBTAINED FROM TWO METHODS FOR FEEDBACK OF MID—TEEN EXAMINATION RESULTS By \ John B. Balch , \ ‘ A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Administration and Higher Education 1969 TABLE OF CONTENTS Chapter I II III THE PROBLEM . . . . . . . Introduction . . . . . . Purpose of the Study . . Statement of the Problem I“? 8 ed 0 O O O O O O O Assumptions . . . . Hypotheses . . . . . . . Definition of Terms Used Limitations of the Study Organization of Study . . REVIEW OF LITERATURE . . Classroom Communication Importance of Testing for Students Research Related to Student Discus- sion of Examination Results Identification of Possible Inter- actions Between Students and Method of Feedback Student Attitudes to Televis Versus Conventional Instruction Summary . . . . . . . . . METHODOLOGY . . . . . . . Population and Sample . . Instrumentation . . . . . Reliability Coefficients in Validity Procedures . . . Factor Analysis Procedures iii This Study ion Page N \O CD'Q O\-{:'\Jo 11 12 13 15 16 20 21 21 22 — 23 24 26 Chapter IV Statistical Hypotheses . . . . . . . . . . . Hypotheses About Achievement . . . . . Attitude Hypotheses . . . . . . . . . . Data Collection . . . . . . . . . . . . . . Analysis of the Data . . . . . . . . . . . . Pre-EXperiment Analyses of Variance . . Covariance Statistical Methodology of EXperiment . . . . . . . . . . . Covariance Assumptions . . . . . . . . Attitude Analysis . . . . . . . . . . . Summary of Kethodology . . . . . . . . . . . ADEALYSIS OF RESULTS 0 O O O O O O O O O O O EXperiment I: The Effect of Method of Presenting Feedback and Student Discussion Tendency on Final Exam Performance . . . . . Assumption of Homogeneity of Regression Uncontroled Variables . . . . . . . . . Hypotheses About Achievement . . . . . Final Exam Means . . . . . . . . . . . Analyses of Covariance Result . . . . EXperiment II: Student Attitudes Toward Different Methods of Presenting Feedback . . Significant Attitude Items . . . . . . Non-Significant Attitude Items . . . . Factor Analysis . . . . . . . . . . . . summary 0 O O O O 0 O O O O O 0 O O O O O O 0 FINDINGS, CONCLUSION, AND SUhhARY . . . . . The PrOblem O O O O O O O O O O O O O O O 0 Design and Procedures of the Study . . Results of Learning EXperiment . . . . . . . Results of Attitude Experiment . . . . . . . Attitude Questionnaire Factor Analyzed: Consistent Results . . . . . . . . . . . . . iv Page 26 26 30 30 32 34 35 37 38 39 39 39 42 43 47 50 53 56 57 6O 62 62 62 66 66 67 Chapter Page Other Significant Differences Within FaCtOrS O O O O O O 0 I O O O O I O O O O O O 69 Conclusions . . . . . . . . . . . . . . . . . 73 Implications for Further Research . . . . . . 74 BIBLIOGRAP hTY O O O O O O 0 O C C O O C O O O 78 APPENTDICES O O O O O O O O O O O O O O O O O 80 LIST OF TABLES Table Page 2.1 Summary of the Differences Between Televi- sion vs. Conventional Teaching . . . . . . . 17 4.1 Analyses of Variance for Departure from Homogeneous Regression Using Final Examin- ation as the Dependent Variable . . . . . . 40 4.2 Mean Scores on the Final Examination of Television and Discussion Groups, Students Above and Below the Average on Tested Ten- dency to Discuss, and Their Interactions* . 46 4.3 Analysis of Covariance Using the Total Final Exam as the Dependent Variable . . . . . . . 47 4.4 Analysis of Covariance Using the Final Exam Items Repeated from the Mid-Term Exam as the Dependent Variable . . . . . . . . . . . . . 48 4.5 Analysis of Covariance Using the Final Exam Items Modified from the Hid-Term Exam as the Dependent Variable . . . . . . . . . . . . . 49 4.6 Analysis of Covariance Using the Final Exam Items Repeated and Modified from the Mid- Term Exam as the Dependent Variable . . . . 49 4.7 Differences in Student.Attitude Toward Dif- ferent Methods of Presenting Feedback on the Hid-Term Exam . . . . . . . . . . . . . . . 51 vi Appendix A B C D E F LIST OF APPENDICES Influence of Evaluating Procedures . . . . 80 Instructions to Teaching Assistants . . . 81 Instructions to Subjects . . . . . . . . . 83 T318 O O O O O O O O O O O O O O O O O O O 85 Course Questionnaire . . . . . . . . . . . 87 Factors, Factor Loadings, Mann-Whitney U Values, and Frequency of Agreement on Questionnaire ReSponses of TV and Discus- sion Groups . . . . . . . . . . . . . . . 9O Rotated Factor Loadings . . . . . . . . . 106 vii CHAPTER I THE PROBLEM Introduction In almost every college course, one obvious aim is to provide students with information. To improve the students' graSp of such information, some educators and psychologists recommend that instructors use test results for more than merely evaluating the students' current performance. That this is desirable is unquestionable, since it is relevant to the whole learning process. The feasibility of using tests to teach as well as evaluate is suggested by two studies. Curtiss and Wood (3) and Sassenrath and Gaverick (20) found that when students discussed the results of their objective classroom examina- tions they learned more than if they merely viewed an author- ity's explanation of these answers. In these two studies neither test scores alone, nor referral to the chalkboard and textbook was as effective an aid to learning as students' discussion of the correct answers with their teacher. This investigation follows these two studies closely, though it employs a different rationale, different variables, and in addition, eXplores students' attitudes about how they have received their examination results. 1 2 Purpose of the Study The general purpose of this pilot study was to improve instructional effectiveness from the way students receive results of their performance on objective examinations. This study had three Specific purposes. First, using as a control group students who listened to a televised explanation of test results, it was to determine whether students who discussed their answers to an objective test (a mid-term) with each other in small groups scored better on a subsequent objective test (a final examination) than the television group. In the first two parts of the study, student scores on repeated test items and their scores on the total final examination were investigated. The second purpose of the study was to determine whether there was any positive relationship between students' final exam scores and their tested tendency to participate in discussion. Third, to determine whether there were any reSponse differ- ences on an eXploratory attitude questionnaire relevant to the purposes and content of exam feedback when administered to different students assigned to the discussion or televi- sion group. Statement of the Problem The problem of this investigation was to offer three sets of eXperimental data relevant to the three purposes of the study and to interpret these data with respect to improving 3 instructional effectiveness. A subsidnxm problem of the study was to eXplore why classroom discussion of examination results might improve instructional effectiveness in comparison to televising these results. It is to be remembered from page one that Curtiss and Wood (3) and Sassenrath and Gaverick (20) found that instruc- tion was more effective when students discussed their objec- tive classroom examinations than if they merely viewed an authority's explanation of these answers. But in this study the effect of small group, student-led discussion of examina- tion answers is compared to the effect of students viewing authorities' eXplanations of these answers via closed-circuit television. Need During the school year when this investigation was con- ducted, students in the educational psychology course for Teacher Education majors at Michigan State University had experienced televised presentations of the correct answers to their mid-term examinations. Some of the students and their instructors felt that students appeared to be so passive dur- ing these televised presentations that little learning was taking place. Hence this contention seemed to merit investi- gation by comparing the effects on learning and attitudes by students who received explanations to their mid-term examina- tion by means of television with students who discussed their mid-term examination results among themselves. Assumptions Results stemming from this study may make some contri- bution to theory construction. The rationale upon which the study is based was to explore the feasibility of the idea that active involvement and participation in small groups might eXplain why classroom discussion could be an effective instructional aid. A number of eXperiments have demonstrated that active learning is more effective than passive learning (McGeoch and Irion, 11). Both Bloom (2) and Krauskopf (9) have presented evidence that when discussions are effective, they promote active thinking about what went on in class and these discus- sions correlate with achievement measures relevant to these thoughts. Korn and Black (10) and Stern et. a1. (18) have found a positive relationship between the amount of classroom participation and academic achievement. It seems logical to assume that students who merely watch TV are less actively involved mentally when they receive examination results than students who listen to exam results followed by a discussion of these results. One way to eXplore the feasibility of whether there was active participation as a result of the feedback session involved determining whether students' attitudinal responses showed they were stimulated to interact with each other and were personally involved when discussing examination answers among themselves. In contrast the feasibility of the idea (that student discussion involves active participation as present in this study) will be 5 supported if the attitudes of students passively listening to TV indicated that they were not personally involved and stimulated to interact with each other. By the rationale of this study, classroom discussion would seem to be a more effective instructional medium than televised instruction, especially for students whose test scores indicate they have a tendency to be active discus- sants in the classroom. In a review of television instruction, McKeachie (12: 42) has summarized the relevant research: In a majority of experiments in which there were adequate controls, greater learning occurred in the 'live' classes than in those taught by television. Host of these differ- ences were not statistically significant by themselves but their consistency is statisti- cally significant. One can thus conclude that at the college level television is generally not as effective as face-to-face instruction. In a different summary of research and by the same rationale, McKeachie (Gage, 7:1149) concludes that college students in television classes tend to rate courses and instructional television less favorably than students in conventionally taught classes. McKeachie, then, has proposed that televised teaching will not eliminate the need for live classroom participation. He (12:22-25) also offers research results supporting the advantages for student-led discussion over teacher-led instruction with respect to both student achievement and how students rate courses. 6 However, from this investigator's (1) review of the literature, there does not seem to be any research evidence specifically evaluating TV techniques as a method of examin- ation feedback. Nor has any eXperimentor compared TV and discussion conditions with individuals whose test scores indi- cate they have different tendencies for active discussion in the classroom. It is, then, significant for this study that Giddan et. al. (8) have emphasized that different individuals have different discussion tendencies. Hence it seemed useful to try to determine whether students would benefit differen- tially from different methods of test feedback. Hypotheses The two hypotheses to be tested stem from the research reviewed in Chapter I and Chapter II and from the investiga- tor's (1) previous research presented in Appendix A. The basis for these hypotheses, then, is that learning is affected by both the external elements of the situation, including the importance of testing to students and their actiVity during the discussion of their examination results, and by internal factors characteristic of the individual learner's tendency to participate actively in classroom discussions. The two hypotheses to be tested in this study are: 1. There is a significant difference in achievement between students discussing mid-term examination scores in small, student-led groups and students listening to a closed-circuit TV presentation of 7 these results as measured on (a) final examination items presented previously on the mid-term examin- ation and during a feedback session and (b) the total final examination. 2. There are significant interactions among individuals' student discussion tendencies, students' final examination scores and their sub-parts, and whether students have experienced discussion versus TV feedback of their mid-term examination scores. Based on the results of the research of Davis and Johnson (4) and McKeachie's (12:43) review of research discussed in the next two chapters, it is also tentatively hypothesized that students' attitudinal responses will indicate that they prefer discussing their examination results to watching a televised presentation of these results. However as this is a pilot study, the purpose of analyzing Specific attitudinal prefer- ences is exploratory: it is possible that more definitive statements as to the operation of these variables can be offered subsequent to the examination of the data. Definition of Terms Used To avoid semantic difficulties, the following terms were defined. These definitions as used in the context of this study were as follows: Discussion Group. Students who discussed the results of their performance on their objective mid-term examination in small student-led groups. Television Groups. Students who received results of their objective mid-term examination by television. Feedback. Receiving results on objective mid-term examInaEion. 8 Active Participation. Feedback by discussion. Student Discussion Tendency. Student score on a scale employed to evaluate discussion tendency. This scale is labeled the Research Form B of the Academic Behavior Inventory, but it is also referred to in this study by the author's abbreviation TSIS. See Giddan et. al. (8). Learning and/or Achievement. Students' scores on the final examination and sub-parts of the final examina- tion which consisted of repeated and modified items from the mid-term examination. Attitudes. Students' reactions to a questionnaire designed to assess their reactions to feedback. Limitations of the Study The students in this study were enrolled in the Summer and Fall Terms of Education 200, a course in educational psychology for Teacher Education majors at Michigan State University. Since the learning or achievement of these students must take precedence over other considerations, the investigators were not permitted to conduct experimental studies which would in any way interfere with normal class- room procedures such as drastically varying the method of television feedback or altering the number and kind of tests and test items students were to eXperience. In accordance with an agreement that experimental testing would not take too much classroom time, it was agreed that students would receive discussion tendency tests only during the Summer Term and attitudinal questionnaires once in the Fall Term. These tests were eXploratory instruments and may not sample adequately the population of possible items 9 with reSpect to student discussion tendency and student atti- tudes about feedback. Further there was no attempt to con- trol variables between the Summer and Fall Term courses. It was assumed that students were eXposed to the same course and that eXperimental techniques from one term could be applied in the next term to eXplore their effect. As the application of theory in this study was also exploratative, it is assumed that the results may make some contributions to theory construction. The limitation of this applied study, then, is in the particular way evidence will be accumulated on the conceptual variable of active personal involvement and in the more important conceptual matter itself. Organization of Study Chapter I presents an introduction of the study and to the purposes of the study, a statement of the problem, defini- tion of terms, limitations of the study, and the hypotheses. Chapter II reviews the literature on classroom part1- cipation and television instruction, the importance of dis- cussion during feedback of examination results, and the advan- tages to students who are more active during classroom dis- cussions. Chapter II also reviews the scales which evaluate individual differences in a) student discussion tendency and b) attitudes towards televised versus conventional instruc- tion. 10 Chapter III describes the selection of the eXperimen- tal groups that discussed their answers to tests during feed- back and the selection of those students who listened to a televised explanation of test results. This chapter presents the procedures for testing the students' tendency to partici- pate in discussion in small groups, the method for selecting the items from the mid-term feedback session that could reappear on the final examination, and the statistical methods used to analyze the data. The criteria of learning and students' attitudes are also described. Chapter IV presents the analyses of (a) final examina- tion scores obtained from comparing the scores of students who actively participated during discussion of test results to the scores of students who listened passively to a tele- vision presentation of test results and (b) how these eXperi- mental conditions of activity and passivity relate to students! tested tendency to discuss and (c) attitudinal differences between experimental and control groups. Chapter V summarizes the problem, methodology, results, and implications for further study. CHAPTER II REVIEW OF LITERATURE The nature of this study necessitated a general review of research pertinent to classroom communication, the rela- tionship of classroom testing to learning and discussion, and the relevance of the scales used in this study. Classroom Communication The importance of classroom communication can be empha- sized in several ways. A number of experiments have demon- strated that active learning is more effective than passive learning (McGeoch and Irion, 11). Korn and Black (10) and Stern, et. al. (18) have found a positive relationship between the amount of participation and several criteria of academic achievement, such as the development of critical thinking and logical eXpression of ideas. Opportunity for discussion also leads to the achievement of other higher educational goals, such as active thinking (Bloom, 2), problem-solving and a scientific attitude (McKeachie, 13), and increased academic motivation (Thistlethwaite, 19). Nevertheless many students do not participate in class discussions and many institutions have no active student dis- cussion in their classrooms. Pace (17:82) has stated: "In almost none of the colleges and universities was it said by 11 12 the bulk of the students that class discussions were typically vigorous and intense." Likewise, Korn and Black (10) reported that one-third of the students they studied participated in class below the minimal expected amount. Importance of Testing_for Students It is the rationale of this study that students are apt to participate in discussions about topics concerning them. McKeachie (15) has suggested that students are more concerned about how they are to be examined than about any other a8pect of their instruction. In a previous study, the present investigator (1) sur- veyed the literature and gathered strong support for the assump- tion that classroom testing procedures influence the nature and consequences of student learning (see Appendix A). McKeachie (Gage, 7:1154) analyzed studies related to feedback from examinations and summarized them as follows: While we usually think of testing procedures in terms of their validity as measures of student achievement, their function as instruments prompting learning may be even more important. After dismal recitals of nonsignificant differ- ences between differing teaching methods, it is refreshing to find positive results from varia- tions in testing procedures. These conclusions suggest that the activity or passiv- ity of students during a discussion of examination results may be educationally significant factors and hence support the need for conducting the present eXperimental study. 13 Research Related to Student Discussion of Examination Results The following studies support the effectiveness of group discussion as a method of active participation during feedback of examination results. Curtiss and Wood (3) assessed the effect of having students actively check the incorrect items on their papers during a discussion of objective test items. This active student involvement resulted in greater pupil learning than having the teacher correct and return the papers without having student discussion. Sassenrath and Garverick (20) reported that students who discussed examination results learned more than (a) students who reread sections of their textbook after checking their answers and the appropriate page references from a correct list on the chalkboard or (b) students who had no opportunity to see their examinations but merely were given their total scores on the examination and their letter grades. In these two studies on the effects of discussion during feedback, the critical factor appears to be verbal interaction with others. It has been said that it is a principle of learning that active learning is more effective than passive learning, and Bloom (2) found that students in discussion classes tend to Spend more time in problem-solving types of thought than students in lecture classes. Both Bloom (2) and Karuskopf (9) have presented evidence that when discussions are effective, they promote active thinking about What went on in class and these discussions correlate with achievement measures relevant 14 to these thoughts. There is also evidence that active writing of reactions to test materials promotes learning. McKeachie (14) found that students who were permitted to write whatever comments they wanted about their test items as they took their objec- tive test did better on the second half of that test. These results suggest that allowing students to discuss their answers orally after taking a test might also improve their perfor- mance on a succeeding test. Another advantage of discussing test results derives from social and clinical studies of attitudinal and person- ality change. Failure of some students to achieve goals of learning may not be due to deficiencies in the materials presented but rather to emotional barriers in the learner. Social and clinical psychological theory can be interpreted to suggest that if a student eXpresses his attitudes in a non-threatening situation it may help "unfreeze" the attitude. Small, student-led group discussions may provide such oppor- tunities for expression as well as give opportunities to eXpress other attitudes about testing which may be instrumental to meeting students' instructional needs. Consequently, the social influence of the small group may facilitate change during discussion of test results. McKeachie (12:19) may support this view when he interprets classic experiments on small group decision-making to show that it is sometimes easier for a group meeting as a group than for an individual, who may not have the 15 feeling he is in a group to change attitudes. McKeachie (12:24) also interprets the many eXperiments on student-led discussions within this framework. For example, he states that Webb and Grib report that students report as a major advantage of student-led discussions their feeling of freedom to ask questions and eXpress their own opinions. McKeachie (12:22—23) quotes Leuba of Antioch College to say: A student is likely to understand a concept, principle, or other idea as he questions him- self about it, looks for its implications and applications, puts it in his own words, and integrates it with previously acquired knowledge . . . Leuba reports satisfying results (from instructorless student discussion groups) in psychology courses and Webb and Grib report superiority of this technique in achievement in statistics and philosophy courses. . . HcKeachie (12:22-23) interprets the experiments in educational psychology, and general psychology by Gruber and Weitman to have established that students taught in small discussion groups without a teacher do at least as well on a final examination as students who heard the teacher lecture, but they were also superior in curiosity (as measured by question-asking behavior) and in interest in educational psychology. Identification of Possible Interactions Between Students and Method of Feedback As it might relate to learning in this study, there was an attempt to identify the student characteristics that inter- acted with the two feedback variables of the eXperiment: Passively listening to a TV eXplanation as opposed to small 16 group discussion of examination results. The Research Form B of the Academic Behavior Inventory or the TSIS, a scale developed by Giddan et. al. (8) has twenty-four true-false items which predict the interaction between students and their discussion leaders in discussion sections. These investigators state that: "Where the effects of different teaching methods are being studied [as in the present study] this scale could assess the role of discus- sion tendencies on reSponse to differing methods of instruc- tion." Hence in the present study it was thought that students who scored higher on this student interaction scale would have the potential to score higher on repeated test items and the whole final examination, under conditions of discussion feedback, as compared to similar students eXperiz encing TV feedback. Student Attitudes to Television Versus Conventional Instruction The eXploratory question raised in this study is whether the attitudes of students receiving examination results by closed-circuit television are better than the attitudes of students who discuss these results. Student attitudes toward closed-circuit television in general tend to be influenced by the interaction of a number of variables. All other things being equal, college students in television classes tend to rate courses and instructional television less favorably than students in conventionally taught classes: Support for this view is presented in Table 2.1 adopted from McKeachie's (12:43-44) review of research 17 Table 2.1. Summary of the Differences Between Television vs. Conventional Teaching (Reproduced from McKeachie's Table 4 in New Developments in Teaching). I Reference Criteria Factual Retention Attitude Knowledge Higher Motivation Level Personality Cognitive Social Science Carpenter and Greenhill Psych. C,C C C Sociol. C Macomber and Siegel Psych. TV,C,C*,C,C,C C C Sociol. TV Econ. C,C* C*,C*,C LePore and Wilson (Psych) TV TV C Humanities Carpenter and Greenhill (Music Apprec.) C LePore and Wilson (English) 7 C Klapper (English) C,TV C,C Seibert (English) C* C Natural Science and Engineering Carpenter and Greenhill Chemistry C,TV Meteorology C Macomber and Siegel Physiol. TV,C,C TV Biol. TV* Zoology C Table 2.1. (Continued) 18 Reference Criteria Factual Retention Attitude Knowledge Higher Motivation Level Personality Cognitive Seibert Chem. C C Mech. Engin. C Martin et. al. Chem. TV* Graphics TV* LePore and Wilson (Science) C,TV TV,C C,C Miscellaneous Macomber and Siegel (Air Science) TV,C,TV,C*C TV,C,C Seibert Math. TV Calculus C Kasten and Seibert (Military Sci.) C Grossman et. al. (Dentistry) TV,C,C McDaniel and Filiatreau (Educ.) C* * Significant at beyond the .05 level of confidence TV = TV superior C = Conventional class superior Differences are simply the actual direction of results of the experiment: when more than two measures were used, the Table reports the direction of the majority of the measures. 19 with college students. Davis and Johnson (4) had administered a questionnaire to Michigan State students similar in construction to the one used in this study. They had factor analyzed their question- naire to validate their a priori grouping of seventeen ques- tionnaire items. Their final grouping was based on a priori judgment and factor analysis of the relevant attitudinal factors of students at Michigan State viewing television ver- sus conventional classroom instruction. The final groupings were: 1. Stimulation of the students 2. Student-(Teacher) interaction 3. Clarity of presentation 4. Attention Their questionnaire also contained items concerning optimum viewing conditions and the opportunities for additional viewing from which they were able to draw overall conclusions about the effectiveness of television instruction. With the exception of these questions, the kinds of problems Davis and Johnson (4) were trying to solve covered four aSpects of instruction at Michigan State University (interest and stimu- lation, student-(teacher) interaction, clarity of presentation, and attention). ReSponses of students viewing television were compared with students viewing more conventionally taught courses. Differences between the two groups were tested and nine statistically significant results were obtained to the seventeen factor analyzed questions. Only three of these 20 differences favored their television groups. Summary Leaders in the field of education are concerned about the lack of discussion in classrooms. The advantages of dis- cussing test results were reviewed in this chapter. There was also a review of scales which evaluate individual differ- ences in a) student discussion tendency and b) student atti- tudes toward television versus conventional instruction. CHAPTER III METHODOLOGY Population and Sample This investigation was conducted at Michigan State University in the discussion sections of beginning educa— tional psychology students. host of these stulents were sophomores and juniors who planned to teach in elementary and secondary education after graduation. One hundred and sixty students were tested in the summer learning eXperi- ment: Eighty-one students composed the discussion groups and seventy-nine students the TV groups. Students from MSU Education 200 sections seemed to be appropriate subjects as they are accustomed to interacting together and to view- ing the closed-circuit TV sets which are left throughout the term in their discusSion sections. During Summer Term, each of the eight sections of students in the course was randomly assigned to one or the other of the eXperimental treatment conditions. In Fall Term, an eXploratory attitude questionnaire was administered to seven hundred eighty-seven students in thirty of the thirty-five sections of beginning educational psychology. Each section was asSigned randomly to the eXper- imental treatment conditions. 21 22 Instrumentation A differential student discussion effect was investi- gated with the Research Form B of the Academic Behavior Inventory or, as it is also referred to, the Teacher-Student Interaction Scale (TSIS). Employing the Kudar-Richardson Formula 20, Lovell and Giddan found that it produced a reli- ability coefficient of .77 (Giddan et. al., 8). Across their eight samples of undergraduate discussion sections, the Research Form B of the Academic Behavior Inventory had been found to correlate with instructor ratings for quality and quantity of student discussion. Giddan, et. al. state "all but two (of sixteen) validity coefficients reached acceptable levels of statistical significance, with a majority of them having a p «(.001. The median correlation for quantity of discussion was .61 and for quality of dis- cussion .41." Across their eight samples of undergraduate sections, Giddan, et. al. (8) demonstrated that the T813 did correlate with instructor ratings for quality and quantity of student discussion. (The two four-point rating scales were developed and used originally by Stern, et. al., 18:169). Giddan et. al. (8) concluded that "where the effects of different teach- ing methods are being studied [as in the present study] this scale could assess the role of discusSion tendencies on re8ponse to differing methods of instruction." 23 Reliability Coefficients in This Study In this study the Kudar-Richardson formula 20 reli- ability coefficients were 1) for the TSIS,.79, 2) for the mid-term examination,.70, 3) for the final examination..82, 4) for the fourteen items on the final examination repeated from the mid-term examination in modified form,.54, 5) for the ten items on the final examination repeated in the exact form as they appeared on the mid-term examination, .55, 6) for the combined total of the twenty-four items repeated on the final examination from the mid-term examin- ation,.69. Reliability coefficients were not computed for the exPloratative attitude questionnaire used in this study because members of the educational psychology department believed that they would not be valid. Davis and Johnson (4) did not report reliability coefficients to their simi- lar questionnaire but felt that such five point Likert Scales were reliable when testing large numbers of students. For the purposes of eXploring attitudes in the present study, reliability was assumed because 787 students were tested on a five point agreement scale. In this study it was possible to compare the impact of students' discussing exam results to the impact of tele- vising them rather than having students only evaluate tele- Vision and discussion against some idealized standard. This Procedure avoided invalid student judgments based on entirely 24 different standards. Validity Procedures Validity was established with the help of Drs. William E. Sweetland and Donald T. Freeman, eXperts in the content and the purposes of the course being scaled. They approved each five point Likert item used in the questionnaire in terms of the content and purposes of the course being scaled. Dr. Robert R. Davis advised on the technical construction of each scale item. Like Davis and Johnson (4), this study used students from Michigan State University in order to permit more mean- ingful interpretation of both sets of similar questions. The limitations of interpretation should be noted: Different populations of students at MSU, the method of instruction and the eXposure time to method of instruction, and the necessary rewarding of questions created some real differences in the two studies. But it was thought that these very differences could help eXplain obtained results better than a set of ques- tions which had not been administered to Michigan State students and which had not been related to television versus more conventional methods of instruction. There is some evidence that the questions asked by Davis and Johnson (4) would elicit some comparable replies in the present study. They had concluded that most students (eighty-seven per cent) would not object to taking television 25 instruction under certain conditions. To a Winter Term exploratory opinion questionnaire eighty-seven per cent of the students in the course presently being studied had similarly indicated that television feedback had been "personally profitable." Hence it was thought that the results of the present study would be more interpretable by constructing a similar questionnaire to that of Davis and Johnson (4). The rationale of the present study was not to follow the rationale of Davis and Johnson (4) but to modify their questions when they fitted the purposes of this study. Davis and Johnson (4) emphasized four aspects of instruction in seventeen of their questions, but this study had over half- again as many additional questions designed to assess other attitudes relevant to this course. For example, it was the opinion of a course evalua- tion committee consisting of teaching assistants that students watching television as compared to students discuss- ing examination answers would respond as if the hour were relatively more "boring," "formal," and "oversimplified" and that they would not become as personally involved as they would when they are stimulated by student interaction. All questionnaire items (e.g., items related to personal involvement and stimulation of students to inter- act with one another) were factor analyzed to objectively classify the items on the five point attitude scale. 26 Factor Analysis Procedures The procedures for factor analyzing the students' reSponses to the thirty-one items on the attitude question- naire were: 1. A thirty-one by thirty-one response matrix was built and the intercorrelations among the items were obtained. 2. The principal axis solution was employed to factorize the matrix. 3. The factors were rotated by the varimax method of factor rotation. StatisticalgHypotheses Hypotheses About Achievement The research hypotheses of this study were stated in Chapter I. To facilitate statistical testing, these state- ments were transformed into null or operational form. The two major null hypotheses of this study were as follows: I. II. No differences will be found between groups of six stu- dents Spending a a classroom hour discussing right and wrong answers of their objective mid-term examination and students who listen to a closed circuit TV presen- tation of these results as measured on a) final exam- ination items presented previously on the mid-term examination and during a feedback session and b) the total final examination. No interactith' will be found among individual students' discussion tendencies as measured on the Research Form B of the Academic Behavior Inventory, students' final examination scores and their sub-parts, and whether students have eXperienced discussion versus TV feedback of their mid-term examination scores. 27 Null Hypotheses I: Symbolically: H : M1 = M2 Legend: M1 = Group final examination means of students discussing their initial examination results M2 = Group final examination means of students SXperiencing televised eXplanations of their initial examination results Both sets of means are adjusted to equate students' initial examination results. Alternative Hypothesis I: The discussion group mean scores will exceed the television group mean scores on the final exam and their sub-parts. Symbolically: Hla: M1 > M2 Legend: Group final examination means of students discussing their initial examination results M2 = Group final examination means of students eXperiencing televised eXplanationS of their initial examination results Both sets of means are adjusted in order to equate students' initial examination results. Alternative Hypothesis II: There will be significant inter- actions in the means of final exam scores among (1) the two eXperimental treatments and (2) the two levels of discussion tendency test scores. Alternative Hypothesis II (a): Within the discussion treat- ment, the group scoring above the mean on the discussion tendency test will have a higher final exam mean than the group scoring below the mean on the discussion ten- dency test. 28 Alternative Hypotheses II (b): Within the TV treatment, there will be no difference on final exam scores between the group scoring above the mean (on the discussion ten- dency test) and the group scoring below the mean (on the discussion tendency test). The final exam means are adjusted to equate students' initial exam results. FEEDBACK CONDITION TV DISCUSSION SCORES SCORES above DISCUSSION mean TENDENCY SCORES below mean An assumption of hypothesis II, then, is that the eXploratory discussion tendency test is a valid and reliable measuring instrument. Supportive data for this assumption have been presented. There are no hypothesized differences between the students in the discussion treatment who score lower on the discussion tendency test and students in the television treat- ment regardless of their discussion tendency test score. Support for statistical assumptions used in analyzing the data for both hypotheses will be presented. In inter- preting the statistical data, the .05 level of confidence will be used to assess statistical significance. The above alternative hypotheses are based on the con- ceptual assumption of the influence of active participation on student learning. 29 The first alternative hypothesis is based on the assumption that if students are given the Opportunity to participate actively by interacting with each other during small group discussions their scores can be interpreted as if they have learned more than if they are not given this opportunity but instead watch television. The second alternative hypothesis is based on the assumption that scores of students with a tested tendency to participate actively in discussion can be interpreted as if they have done so when given the opportunity to discuss mid- term exam results. Scores of students with less of this tested tendency to discuss are assumed to be interpretable as if they have not actively participated in discussion. It is assumed that students with a below the mean dis- cussion tendency score will have lower final exam scores if asked to participate in discussions than those with above the mean discussion tendency scores. It is presumed that a low discussion tendency score is related to the lack of skill in the ability to profit from discussion. AS the television groups were not given the opportun- ity to participate in classroom discussions during feedback, their tested tendency Undo so is presumed less relevant to their learning. Nevertheless, the scores of students watch- ing television are assumed to serve as a control or compari- son condition between two extremes: Specifically, the com- parison is between the hypothesized highest final exam scores of students assumed able to participate actively in discussion 30 (because of their high discussion tendency scores) and the hypothesized lowest final exam scores of students presumed not able to participate actively in discussions (because of their low discussion tendency scores). Attitude Hypotheses An investigation of students' specific attitudinal preferences towards either discussion or televised presenta- tion of examination results was conducted. Although based on the research of Davis and Johnson (4) and McKeachie (22) previously discussed, these attitude tests are of an eXplan- atory nature and are therefore two-tailed. In null hypotheses form to facilitate statistical testing: no differences will be found in attitudinal reSponses between students Spending a class- room hour discussing mid-term exam scores and students listening to a closed-circuit TV presentation of these results as measured on each of thirty-one questionnaire items rele- vant to the purpose and content of examina- tion feedback. Data Collection During the Summer and Fall Terms, students in differ- ent discussion sections of beginning Education 200 at MSU were presented correct answers to their mid-term examination in one of the two randomly assigned conditions: 1) In sections presented with correct test alterna- tives over closed-circuit TV with a description of why the correct alternatives were correct together with a blown-up picture of the mid-term 31 examination. These students watched a televised team of two teachers on one tape. 2) In discussion sections of small groups of six students, with one graduate assistant giving each group one copy of the correct answers to the test during the feedback session and asking them as a group to participate in discussing why the cor- rect answers were correct. Graduate assistants assessed the adequacy of the TV presentation and the discussion coverage of test items dur- ing the feedback sessions. These raters judged both of the above conditions to be adequate along the dimensions referred to in Appendix B. During the feedback session, each student in this eXperiment was given his answer sheet and informed of the correct mid-term answers. In the discussion groups one mem- ber of the six-man discussion group was given the correct mid-term answers and asked to present them to his group for discussion. The groups were instructed to discuss each test question. (The instructions are presented in Appendices B and C.) (Pre-tests indicated students had sufficient knowl- edge of course materials to discuss test questions effectively.) To facilitate discussion of pertinent course materials, each student in the discussion groups was also given an unmarked examination to be returned at the end of the feedback hour: one student in each discussion group was given the cor- rect answers to the mid-term and the other members of each dis- cussion group had to refer actively to him (see Appendices B and C). 32 During the Fall Term, attitudes of students who received examination results by television were compared with attitudes of students who discussed these results. The method used was an exPloratory student attitude questionnaire. The questionnaire was administered five days after the examination feedback session to 787 students in thirty sec- tions of educational psychology. Each teacher of a different section in this course had one of his sections of students watch the presentation over television and had the other sec- tion of students discuss their examination results (see Appendices B and C). Sections of students were assigned randomly to these eXperimental conditions so there were equivalent numbers of sections assigned to each condition at each hour the sections were meeting. Generally it took stu- dents about ten minutes to fill out the questionnaires and no more than fifteen minutes to administer them. Analysis of the Data Evaluation of all Summer sections consisted of deter- mining the relative effectiveness of the discussion and tele- vision groups on the final examination. For separate statis- tical analysis and comparison the final exam included fourteen items from the mid-term examination which were modified and ten items which were in the exact form as they appeared on the mid-termtest.1 These repeated items, selected by a panel of .— 1The modified items measured the same concept or prin- ciple as the correSponding mid-term items, but were presented 'under different stimulus conditions on the final examination. 33 judges after the feedback sessions, served to assess whether students had learned the content of these items (and/or the concept related to these items) as a result of one of the two feedback conditions of the eXperiments. There were four final examination test score criteria or dependent variables in the summer eXperiment: 1) Number of modified recurring test items answered correctly. (These modified recurring items were chosen at random from adequate items of the item pool in advance of the mid-term. Some of these items had equivalent statistical results from previous test administrations.) 2) Number of recurring items answered correctly. (These items were again chosen at random from the adequate items of the item pool in advance of the mid-term.) 3) The above modified and recurring test scores combined. 4) Total scores on the final exam. Independent variables included: 1) The two treatment conditions outlined above. 2) Student discussion tendency scores as assessed on the TSIS. (The TSIS is also referred to as the Research Form.B of the Academic Behavior Inventory.) 3) Test scores on the mid-term examination. 4) Breakdown of test scores on the mid-term examina- tion into sex differences. As my hypothesis has indicated, small group discussion during feedback of examination results was thought to improve learning as measured by an evaluative test: in the TV groups student discussion was not possible: in the Six man discus- Sion groups, discussion was possible and encouraged but not 34 controlled. It was, therefore, necessary to investigate for a differential discussion effect among the students eXperienc- ing television and among those eXperiencing discussion feed- back. Pre-EXperiment Analyses of Variance A one-way analysis of variance: TSIS by classroom sections was initially conducted to investigate whether there were any initial differences between sections on the TSIS discussion tendency test. Similarly the mid-term scores by classroom sections were analyzed in a one-way analysis of variance to investigate whether there were any initial sec- tion differences in the students' scores on the mid-term examination. Further one-way analyses of variance were con- ducted to insure that there were no initial differences between those four sections of students drawn at random to eXperience TV during feedback, using student TSIS scores as the criteria,and few initial differences,using mid-term exam- ination achievement scores as the criteria. These four analyses of variance were conducted to investigate whether there were any initially significant differences among sec- tions or between sections that had eXperienced discussion feedback versus those sections that had experienced TV feed- back.2 2There were no significant differences among the eight sections of students or between the two treatment groupings of students on either the discussion tendency test or the mid-term examination. The section means on the mid-term examination were 35 Covariance Statistical Methodology_9f Experiment McNemar (16:373) states that an analysis of covariance is more sensitive to the potential significant differences of of dependent variables than more ordinary techniques. There- fore, in this study, initial student mid-term examination scores were adjusted in analyses of covariance using the final examination scores as the dependent variable. Analyses of covariance were conducted to evaluate the eXperimental treatment effects of students discussing or watching TV on their final examination scores. Scores on the discussion tendency scale (the test is found in Appendix D) were classified as above or below the mean. The interactions of the levels of discussion tendency were analyzed with the treatment effects as they together might interact and influence final exam scores. Because two of the dependent variables were short tests of ten and fourteen items, these tests also were 32.00. 33.05. 33.50. 33.30. 35.85. 33.85. 32.50 and 32-43- The treatment means on this examination were 33.15 and 33.47. The F-ratio for section differences was 1.17 which was within the .32 level of confidence. The F-ratio for treatment means was .17 which was within the .69 level of confidence. The section means on the discussion tendency test were 14.10, 14.90, 16.60, 14.15, 13.65, 12.55, 15.61, and 13.62. The treatment means on this test were 14.37 and 14.39. The F-ratio for section differences was 1.52 which was within the .17 level of confidence. The F-ratio for treatment means was .001 which was within the .98 level of confidence. (At the .02 level of confidence it can be concluded, therefore, that there were no initial differences between treatment groups as to discussion tendency scores.) 36 then run under the Special consideration of combining these two final examination sub-parts of repeated items from the mid-term examination to increase criterion reliability. McNemar (16:372) also states that if uncontrolled multiple variables correlate near zero with dependent vari- ables, they need not be controlled eXperimentally or statis- tically. Simple correlations between the uncontrolled stu- dent mid-term examination scores and final examination scores were expected to exceed a .6 correlation as has happened on previous Education 200 tests. However, breakdowns of the students on the mid-term examination as to sex will be investigated to establish correlation near zero with final examination scores and their sub-parts. Following McNemar's (16:372) reasoning, the sex variable can be eliminated from covariance analyses in this study if it contributes very little to the prediction of final examination scores and their sub-parts. Covariance Assumptions Edwards (6:292) emphasized that "it is important to stress that the application of the analysis of covariance does assume that the regression lines for the various treat- ment groups all can be assumed to have a common slope." Consequently, analyses for homogeneity of regression were conducted by assessing the extent that the regression lines for the two eXperimental treatments and two levels of student discussion were all four parallel. More Specifically. 37 different analyses for homogeneity of regression were con- ducted with the students classified by (a) experimental treat- ment and (b) above or below the mean on tested level of dis- cussion tendency. These analyses were repeated for the depen- dent variables of this experiment. If the assumption of homogeneous regression for these treatments and levels is tenable for the dependent variables, then following Edwards (6:292), analyses of covariance can be performed. The uncontrolled multiple variables in this study were the mid-term examination scores of the students and the break down of these scores into sex differences. By using the highly sensitive covariance analyses statistic, the uncon- trolled multiple variables were to be controlled statistically if there was evidence they might influence final examination scores and their sub-parts. Attitude Analysis The eXplorative attitude questionnaire was analyzed with the Mann-Whitney U statistic. Each of the thirty-one questions was compared with reSpect to student responses grouped by whether they had listened to television or dis- cussed examination answers. Ties between groups on each point of agreement on the scale were adjusted in the statis- tical analysis by the method recommended by Siegel (22:124- 125). Siegel (22:15?) recommends the use of the Mann-Whitney U statistic because it is the most powerful test "of location" appropriate to large samples meeting the assumption of ordinal data. 38 Summary of Methodolggy Analyses of covariance were conducted to evaluate the eXperimental treatment effects of students discussing or watching TV on their final examination scores. Scores on the discussion tendency scale (the test is found in Appendix D) were classified as above or below the mean. The interactions of the levels of discussion tendency were analyzed with the treatment effects as they together might interact and influence final examination scores. By using the highly sensitive covariance analyses statistic, the uncontrolled multiple variables were to be controlled statistically if there was evidence they might influence final examination scores and their sub-parts. Mann-Whitney U tests for independence of experimental and control groups were computed for each Likert scale ques- tionnaire item. The attitude scale is found in Appendix E. A factor analysis of the whole scale assessed the groupings of items. CHAPTER IV ANALYSIS OF RESULTS In this chapter a report of the analysis of data is presented in two parts. The results of the learning eXperi- ment are presented in the first part. The results of the attitude exPeriment are presented in the second part. EXperiment I: The Effect of Method of Presenting Feedback and Student Discussion Tendency on Final Exam Performance The analysis of covariance statistic used in testing EXperiment I, assumes (1) homogeneity of regression and (2) all significant variables have been controlled. It was therefore necessary to test for these assumptions prior to completing the final analyses of covariance. Assumption of Homogeneity of Regression As a test of homogeneity of regression, an analysis of variance for departure from homogeneity of regression was completed for each dependent variable. The results of these tests are presented in Table 4.1. 39 4O pesoacaewdm p02: om.mmm Hmpoa mm.H ma.mmm mma nouns am.mm m cosmmoawoa Haves *ma. mm. oo.H m kudosomoaos Scam chaphdaon ou.mm d :oammohwom m .m.E .m.m .m.d mohfiom seam sameness map sown Seam Henna esp :0 emdcaeoz mampH peeoauaewam pozs om.mwwsa Hence mm.:m mm.m:mm NmH Honnm mm.mssw n coammoawon H6008 #mH.H mm.mo mn.ama m avaoaowoaos achm chaphdaom ma.ammw : :oammoawom m .m.z .m.m .m.d mohdom soapd:H8de Huaam Hence oafldahd> pamusmaom on» ma noapmaassxm Heads wean: aoammoawom msoosomosom Scam madpnsaom you ooamans> mo mommassd .H.e manna 41 pesosedewam pozs pemoaeaemdm poz* om.mmma Hence sm.muo Hence mm.w mm.onm NmH Honhm Hm.m nu.m:e mma gonna mm.mme A cosmmmnwmn H~.emm A eodmmmuwmn Haves Hence *H:. um.h m hpaoaowoaon emu. SN.N ow.w m hpdoaowoaos Song Song cadphsaon manphcaom Hm.mam : aoammonwom H:.mmm : Scammonwom m .m.: .m.m .m.p condom m .m.z .m.m .m.o condom scam Headm 0:» no anom poamddoz no pocwm ad scam Susanna: Incoaa4_swwm aaoauedz aonm maopH nonspeco on» Sony swam Humam on» so doaaaeoz maopH .eoeeapeoo .H.e capes 42 Since none of the observed F-ratios approached a signi- ficant level (when alpha was conservatively set at .20), it is reasonable to conclude that the regression lines are parallel and that the assumption of homogeneity of regression is there- fore tenable. Uncontrolled Variables From Chapter III, it can be recalled that the uncon- trolled variables in this study were the mid-term examination scores of the students and the break-downs of these scores by sex differences. If sex differences correlate near zero with the final examination and its sub-parts, then it need not be controlled experimentally or statistically. Sex differences did not contribute demonstrably to the over-all variance. Multiple correlation coefficients using sex differences correlated .04 with the final examination, .07 with the ten items repeated on the final examination in exact form as on the mid-term examination, and .04 with the fourteen items modified on the final examination from the mid-term examination. Covariance analyses were, therefore, performed without using sex differ- ences. From Chapter III, it can be recalled that mid-term exam scores were to be statistically controlled in covariance analyses. This decision was reached because mid-term exam scores had correlated .60 with final examination scores in a pilot study prior to this investigation. During the term 43 when this eXperiment was conducted, mid-term examination scores correlated .65 with final examination scores. Mid-term examination Scores correlated .62 with the com- bined twenty-four items modified and repeated in exact form on the final examination from the mid-term examination. Mid-term examination scores correlated .57 with the fourteen items modified on the final examination from the mid-term examination. However mid-term exam scores only correlated .50 with the ten items repeated in identical form on the final exam from the mid-term. Hence, corrollary null hypotheses 1b and 2b (to be presented next) are to be viewed within this limitation (mentioned in Chapter I). Hypotheses About Achievement (EXperiment I) The research hypotheses of EXperiment I were stated in Chapter I. The null hypotheses of EXperiment I and their alternatives were also Specified in Chapter III. The research hypotheses, and their corrollary null hypotheses, are stated as follows: gyppthesis I: The performance on the final exam of students who receive televised feedback will differ from the corresponding performance of students who receive discussion feedback. The corollary hypotheses stated in null form are as follows: Corollary La: The mean total final exam score of students receiving TV feedback will not differ from the correSponding mean for students receiving discus- sion feedback. 44 Corollary Lb: The mean score on repeated items for students receiving TV feedback will not differ from the corresponding mean for students receiving discus- sion feedback. Corollary Ic: The mean score on modified items for students receiving TV feedback will not differ from the corresponding mean for students receiving discus- sion feedback. Corollary Id: The mean combined score (modified and repeated items) for students receiving TV feedback will not dif- fer from the corresponding mean for students receiving discussion feed- back. Hypothesis II: There will be a Significant interac- tion between the method of presenting feedback and student discussion ten- dency in terms of their effect on final exam performance.1 The corollary hypotheses stated in null form are as follows: Corollary IIa: There will not be a significant interaction between the method of presenting feedback and student discussion tendency in terms of the effect of these two variables on total scores on the final exam. Corollary IIb: There will not be a significant interaction between method of presenting feedback and student discussion tendency in terms of the effect of these two variables on repeated items on the final exam. Corollary IIc: There will not be a significant interaction between method of presenting feedback and student discussion tendency in terms of the effect of these two variables on modified items on the final exam. 1Giddan's (8) Research Form B of the Academic Behavior Inventory was used to test students' discussion tendency. 45 Corollary IId: There will not be a significant interaction between method of presenting feedback and student discussion tendency in terms of the effect of these two variables on combined items (modified and repeated) on the final exam. Final Exam Means The means on the final examination of (1) the television and discussion groups, (2) students above and below the aver- age on tested tendency to discuss, and (3) their interactions are presented in Table 4.2. The means are adjusted to equate students mid-term examination scores. The means are presented in Table 4.2 according to students' average scores on the: 1. total final exam, 2. ten items repeated in identical form on the final exam from the mid-term exam, 3. fourteen items modified on the final exam from the mid-term exam, and 4. combined total of twenty-four items from the mid- term exam appearing in identical or modified form from the final exam. There appear to be two consistent relationships and a third nearly consistent relationship among the means presented in the four groupings of Table 4.2. As the total final exam is made up in part by identical and modified items from the mid-term exam, the dependent variables were not independent of one another: These relationships are: 1. Students with an above-average discussion tendency test score consistently have higher mean scores on the final exam and its sub-parts than do students with a below-average discussion tendency test score. 46 .moaoom ammo anoplpaa .wpsoBSpm opdswo op am.ma mo.ma mm.wa om.ma HH.mH mwdhobw zoaom mm.ma mm.ma mo-ma owSHobs obond aoammSOmam >9 seam Assam can :0 Show poamaeoz no HmoapsooH ma madamoaad aswm ascendaz can Scam maopH adomthpnoze dosapsoo ho made: om.m mm.m mm.m me.m mm.m mweno>e zoamm Hw.m mw.w nm.m mwdaobc o>094 Gadonsomdn. >8 scam sampled: Song scam Hedda on» no snow poswm ad popcoaom maopH Goa Mo wanes eopmsfips one mance auxo Heady chops can mo HH<* N¢.oa 35.0H 0:.0H No.0H mm.oa owwnmbm 30H0m Wm.oa Hw.oa om.oa owdhobm mbofld scammdomaa >9 swam sameness can aonm scandadaswm Assam can so poamapoz maopH zoopHSOm we made: mm.om 00.00 mw.mm mm.mm Mo.oo owsao>s SOHom mm.aw NN.N© wH.Hw omenobs o>094 aoammSOmam >9 seam Hanam Hdpoa map so made: Hausa can .mmfiomam 0p hoaoeaoe pounce so *msoaposnoan owsnobd esp zoaom use chops mpaoeSpm .mazonc scammzomda can aoamaboaoe mo soapsaassxm Hazam can no monoom new: .N.e oHDdB 47 2. Students with an above-average discussion tendency test score who discussed their mid-term exam answers consistently have the highest mean scores on the final exam and its sub-parts. 3. A less consistent relationship was that students with a below-average discussion tendency test score who discuss their mid-term examinations have the lowest mean scores on the final exam and its sub- parts than students who did not. The one exception was on the ten items repeated in the identical form on the final examination as they were on the mid- term exam. Although none of the predicted difference and/or inter- actions were extensive, an analysis of covariance was neverthe- less completed to determine whether or not they were statistic- ally significant for any of the dependent variables. This analysis seemed reasonable in view of the large sample size. Hpgiyses of Covariance Results The analysis of covariance test was used as a test of Hypotheses I and II. The results of this test was repeated for each of the dependent variables. The results, when the total final exam served as the dependent variable, are pre- sented in Table 4.3. Table 4.3. Analysis of Covariance Using the Total Final Exam as the Dependent Variable Source d.F. S.S. M.S. F Treatment 1 3.10 3.10 .06* Level of Discussion 1 145.84 145.84 2.68* Interaction 1 24.45 24.45 .45* Error 155 8434.71 54.42 *No Significant Differences 48 The F value needed for (hypothesized) significance (P ( .05) is 2.67. Since none of the obtained ratios exceeded. this value, it can be concluded that the means did not differ significantly. The results of the analysis of covariance for the test- ing of Null Hypotheses I and II, using the ten items repeated on the final exam in the identical form as they appeared on the mid-term exam as the dependent variable, are presented in Table 4.4. Table 4.4. Analysis of Covariance Using the Final Exam Items Repeated from the Mid-Term Exam as the Dependent Variable Source d.F. S.S. M.S. F Treatment 1 1 o 9 7 1 o 97 1 o 03* Level of Discussion 1 .66 .66 .34* Interaction 1 .61 .61 .32* Error 155 296.80 1.91 *No Significant Differences The F value needed for Significance (P ( .05) is 2.67. Since none of the obtained ratios exceeded this value, it can be concluded that the means did not differ significantly. The results of the analysis of covariance for the testing of Null Hypotheses I and II. using the fourteen items modified from the mid-term exam for the final exam as the dependent variable are presented in Table 4.5. 49 Table 4.5. Analysis of Covariance Using the Final Exam Items Modified from the Mid-Term Exam as the Dependent Variable source d.F. SoSo I'IoSo F Treatment 1 4.04 4.04 1.39* Level of Discussion 1 4.93 4.93 1.70* Interaction 1 7.68 7.68 2.65* Error 155 449.53 2.90 *No Significant Differences The F value needed for significance (P < .05) is 2.67. Since none of the obtained ratios exceeded this value, it can be concluded that the means did not differ significantly. The results of the analysis of covariance for the test- ing of Null Hypotheses I and II, using the combined repeated and modified items from the mid-term exam on the final exam as the dependent variable are presented in Table 4.6. Table 4.6. Analysis of Covariance Using the Final Exam Items Repeated and Modified from the Mid-Term Exam as the Dependent Variable Source d.F. S.S. M.S. F Treatment 1 .37 .37 .06* Level of Discussion 1 9.19 9,19 1,46% Interaction 1 12.63 12.63 2.00* Error 155 978.19 6.31 *No Significant Differences 50 The F value needed for significance (P <’.O5) is 2.67. Since none of the obtained ratios in this study exceeded this value, it can be concluded that the means did not differ sig- nificantly. EXperiment II: Student Attitudes Toward Different Methods of Presenting Feedback A summary of student reSponses to the exploratory atti- tude questionnaire are presented in this section of Chapter IV. The null hypothesis tested for each of the thirty-one questions was: There are no attitude response differences between students who Spend a classroom hour discussing right and wrong answers of their objective mid-term examinations and students who listen to a closed-circuit TV presentation of these results. The Mann-Whitney U statistic was employed in determin- ing independence of relationship because of the reasons presented in Chapter III. (Siegel (22:15?) recommends the use of the Mann-Whitney U statistic because it is the most powerful and valid test "of location" appropriate to large samples meeting the assumption of ordinal data.) The problem of objectively classifying questionnaire items was met by factor analyzing the attitude scale. All items will be reported with reSpect to the factor in which they had their highest loadings in Table 4.7. 51 smo. *Hmwse >9: Ram Ram oodcaaaadoawopo .am saw sow soopoooe oaooaog .em *mo. *dnmow >9: fimm Rom mafiaon 902 .mm ***Hoo. *semmamm >9+ Rum mow asdaopds psdboaohad Sosa oop poz .HN .m.z mm. m mms mam mmm noses: so: use eoapeoppd .eH *mo. *mommo >9: ans ewe eoapeoppo same maodnopoe .o *eeaoo. ***ooaso >9+ mwm wmw soapaoppd ado: op stosm .m Soapsoppd oaom uHH Hopomm .m.z om. memos Rum Rmm paopsoo omnsoo pneumaooab .mm seefioo. *eemsmmo >e+ so: Rm: nooao poe omHSoo no moooaasm .Am *mo. smammo >9+ Rm: Ram maozmad poonaoo mo Samphoo 902 .mm .m.z ma. Hommw Rum Rm: honodop omoahsa osdpmnooas poz .om .m.z ma. Somme Rm: Rmm omhdoo unannom mandadmoo poz .wH *mo. *mswwo >9: mom mm: madness» ohms H ma hangoaomwao on .ma .m.z Hm. mamma sum sea monoznouao pd poops seas» pom use .mH ***Hoo. **#wmmmm >9! Raw Rm: 0969 £30 #6 dswpmhmdflfi no: GHQ .NH .m.z on. omams ma: mm: Home pooaw o enema so: ode .OH .m.z HH. enema mom mew ammo Hesse no means Hoppom .e ***Hoo. ***mmsmm >e+ son saw omHSoo maseoeoapoaowwopea eooponoeeo .m .m.z Ne. mmema go: as: ammo Hogan you sense mama .N enscoo 0:» mo waaBSSpmHooap “H Honodm Ho>oq D soap >9 sodmmsomaa .oz ahaaancpoam Amanda: noonan m napsm obdpamom aoapaaaomoa Soundbonnnd aopH lanes 90 paooaom seam aao9aoaz on» so xosnooom wadpsmmoam Mo moospoz pacemakan ons309 messapp4 p soapm ad moosohommam .m.: oHQS9 52 m Naosoaad mom .mammawad Honosm on» no mpHSmoh op wadohooom vendoaw soon obs: msopH peooaedewao no: .m.z Hoo . V m*** Ho. v m** mo. v 9* m .m.z so. Swamp mom was soonoooc seduce mooeonw5pmao zom .AH ***Hoo. ***womd© >9I Raw Rmm no wfiaom ma pas: Ham: mhdkdd .QH .m.z Ho. onmms Row Row hopes noeooop ooo Boo .n SoaposnpmaH Spas oohowhopaH omaoz .> nopowm ***Hoo. ***moomm >a+ emu ems meoswanao aoom nownofl o: meospmosa .mm ***Hoo. ***mwmao >9- awe smm eooooe mpeoaopoaaad Hopo>om .mH .m.z AH. msemm new new noeoeop spa: same on amuse; .HH ***Hoo. ***mmooe >9: ems moo maoanopoa swoops on mass eoeooz .m .m.z HH. enema Roe smm meospmose Ame op copes: .e ***Hoo. *esesmmm >9: awn Rae maednopoa wefioom ABHSoHecHe oz .a SOHSSpaomonm Macao .>H Hoposm ***Hoo. ***mmmmm >9- ans mos empaoped aaaaeomnoa poz .om sssfioo. ***momms >9: Ann new season oop poz .mw ***Hoo. sssmflmsm >c: sea ANN monoeSpm soapoonoped opoHSSHpm p.2edo .mm oopodaopsH pad dobaobsH .HHH Hoposm Hoboq D Soap >9 soameOmaQ spaadnoponm gonads: nomads owedpam o>dpsmom .oz asses uo peoowom eoapadnomon eopoaponpnd aopH ml oosaapsoo .m.: oHnw9 53 To facilitate the interpretation of the meaning of the experimental results, some positively worded items have been changed to negative statements. In addition some negatively worded items have been changed to positive statements. Fur- ther all items will be reported in percentages of students' acceptance by either pooling "Strongly agree" and "Agree" responses or "Strongly disagree" and "Disagree" reSponses.3 These percentages are depicted in columns three and four of Table 4.7. In view of the magnitude of these differ- ences, a Mann-Whitney U test was computed for each item on the questionnaire. The results of these tests are depicted in columns five and six of Table 4.7. Significant Attitude Items Of the thirty-one items covering attitudes about examination feedback listed in the questionnaire, eighteen items revealed statistically significant differences between groups (P < .05). In Table 4.7 the results are summarized for those items that were statistically significant. The percentage of student acceptance of the item is given and whether the acceptance was or was not in favor of the instructional effectiveness of TV (see footnote 3). As indicated in Table 4.7 there were 12 significant items in favor of the discussion treatment and six 3In Appendix F items are rgported with respect to exact distribution of reSponses on all five points of the attitude scale. Items are also reported with their exact wording as they appear in Appendix E. 54 Significant items in favor of the television treatment with respect to student agreement. These items can be summarized as follows: Thirty-nine per cent of the students watching televi- sion agreed that the feedback session helped them understand interrelationships among various parts of the course, but only 21% of the students discussing their answers responded as if it had helped them: 31% of the television group and 54% of the discussion group said they were not helped (item 3). However 61% of the students in the discussion treatment reported that the feedback hour helped them to understand course materials at their own rate of comprehension and only 43% of the television group acknowledged this help (item 12). One half of the students who saw TV said they would not give exam results this way if they were teaching the course: 43% of the discussion group agreed that they would not have students discuss exam results if they were teaching (item 15). Nevertheless as a result of feedback more students who discussed their exam results than those who saw them on TV still felt they were uncertain of why the correct answers were correct (51% and 45%, respectively, for item 26). Forty- eight per cent of the discussion group and 40% of the TV group agreed that the purposes of the course were not more clear as a result of the feedback hour (item 27). Seventy-eight per cent of the TV group and 65% of the discussion group replied that there often was not enough going on to hold their attention when they received their exam 55 results (item 8). In contrast 68% of the TV students and 63% of the discussion students did reply that the materials they saw during feedback held their attention (item 9). But 77% of the students watching TV and 66% of the students discuss- ing exam answers disagreed with the statement that there was too much irrelevant material covered during the eXperimental session (item 21). Sixty per cent of the discussion group and 52% of the TV group did not think the way they received examination feedback was boring (item 23). Whereas 52% of the students in small discussion groups thought the presenta- tion over simplified, 59% in TV groups gave this response (item 31). There was a consistency in reSponse to items related to whether students said they were stimulated by other stu- dents, personally involved, and whether the feedback session seemed too formal. Students watching TV said they were not stimulated to interact with other students (74%) and not personally involved (71%) whereas only 22% and 16% of the students discussing exam answers gave these replies (items 22 and 30, respectively). Similarly 86% of the students discussing answers but only 53% of students watching TV disagreed with item 25 related to whether the feedback hour was too formal. Thirty-four per cent of the TV group and 20% of the discussion group replied they had difficulty seeing materials; 52% of the TV students and 60% of the discussion students reported no difficulty (item 1). Three quarters of the TV 56 group and 60% of the discussion group said they needed more time to absorb and assimilate materials (item 5). Sixty- eight per cent of the TV students and 55% of the discussion students reported that several improvements needed to be made in the method by which they received their examination results (item 19). However 55% of the TV group and 72% of the discussion group stated that many exam questions still seemed ambiguous after feedback (item 29). Seventy-eight per cent of the students listening to TV and 60% of the students in small group discussions reported that they could always hear what was going on during the feedback hour (item 16). Non-Significant Attitude Items 0f the thirty-one items listed in the attitude ques- tionnaire, the reSponse to thirteen items revealed no statis- tically significant differences between groups and conse- quently the null hypothesis was accepted. In Table 4.7 the results are summarized for those items that were not statis- tically significant. An examination of Table 4.7 shows that seven of the items related to learning and understanding the course were not significant. Further insPection of the six remaining non-significant items reveals no differences between the TV and discussion students in termS' of items having their high- est loadings in three different factors. 5? Factor Analysis The problem of objectively classifying questionnaire items was met by factor analyzing the attitude scale. All items have been reported with respect to the factor in which they had their highest loadings. The loadings of each item within its factor are reported in Appendix F. Three of the items which had significantly favored the discussion group loaded on one factor. The discuSSion group agreed that (1) the feedback session stimulated them to interact with other students, (2) they felt personally involved, and (3) the session was not formal. Items 22 and 30 both loaded highest on this factor related to whether there was interaction and involvement as a result of feed- back. Item 25 related to whether students agreed that the exam presentation was too formal. In Table 4.7 the significant differences within this factor were reported along with the percentage endorsement of students indicating that the direction of Significance favored the instructional effectiveness of discussing exam answers. Discussion students agreed more than TV students with statements about whether they were stimulated to inter- act with each other and were personally involved during what the TV students agreed was too formal an examination feed- back hour. On item 22, 74% of the TV students disagreed and 62% of the discussion students agreed. 0n item 25, 53% of the TV students and 86% of the discussion students agreed. On item 30, 71% of the TV students disagreed and 64% of the 58 discussion students agreed. Significant differences also occured on items involv- ing what the students felt they had learned and understood. Three of the items with loadings on this factor suggested that students who discussed exam answers were more apt to agree that they were p23 helped to understand the course more than students who received the TV feedback. These items involved understanding: interrelationships and purposes of the course and why the correct answers were correct. How- ever the reverse was true for two other items in this group. Students who discussed the exam were more apt to agree that they were understanding at their own rate of comprehension and that they would give students exam results the same way they received them if they were teaching the course. 0f the seven items related to the factor of holding students' attention, five were also significant. Three of these items favored the discussion group: Sixty-eight per cent of the discussion group agreed that the materials given them to look at during the feedback hour held their atten- tion while only 63% of the TV group agreed. Sixty per cent of the discussion group disagreed with the statement asking them whether the feedback session was boring while only 52% of the TV group agreed. Fifty-nine per cent of the TV group agreed that feedback was oversimplified, while only 52% in the discussion group agreed. 0n the other hand two items in this group favored the TV group. Over three-fourths of this group agreed that during the feedback hour there was: 59 enough going on to hold their attention and that not too much irrelevant material was covered during the feedback hour. The corresponding percentages for the discussion group were 65% and 66%. The only other item favoring TV loaded on a different factor: namely, clarity of presentation. Seventy-two per cent of the discussion group but only 55% of the TV group agreed that even after feedback, many questions were ambiguous. But 61% of the discussion group and only 52% of the TV group agreed that they had no difficulty seeing materials. In addi- tion 75% of the TV group and only 60% of the discussion group said they needed time to absorb and assimilate materials. Sixty-eight per cent of the TV group and only 55% of the dis- cussion group thought several improvements needed to be made in the method by which they received their examination results. 0f the Six items in this factor, four were significant and three favored discussion feedback. The fifth factor which was noise interfering with instruction had one significant item favoring discussion feed- back: Seventy-eight per cent of the discussion students and 69% of the TV students agreed that they were always able to hear what was going on. The two other items loading in this group were not significant. It should be pointed out that to facilitate the inter- pretation of meaning in this study, all items have been reported in percentages of students' acceptance by either pooling "Strongly agree" and'Agree" reSponses or "Strongly 60 disagree" and "Disagree" reSponses. In addition some posi- tively worded items have been changed to negative statements. Some negatively worded items have been changed to positive statements. It is to be remembered that the purpose of the attitude Questionnaire was eXploratative and hypothesis gen- erating. In.Appendix F all items listed in the questionnaire are summarized: 1. by factor loadings, 2. Mann-Whitney U values, 3. level of probable difference and direction of significance, 4. in percentages as well as raw scores for student agreement to each of the five alternatives on the questionnaire and 5. by the content of the item as listed in Appendix E. Summary An analysis of the data was presented in this chapter. Although mid-term examination scores were statistically equated through use of the analysis of covariance procedure, no significant differences were found on final exam scores and their sub-parts between groups of students discussing mid-term exam results and students watching a TV presenta- tion of these results. 0f the thirty-one items eXploring attitudes about examination feedback listed in the question- naire, eighteen items revealed significant differences 61 between the groups. Twelve of the items favored the instruc- tional effectiveness of the discussion treatment and Six of the items favored the television treatment with respect to student agreement on the questionnaire. A discussion of each significant item was presented. When the questionnaire was factor analyzed, three of the items which had Significantly favored the discussion group loaded on one factor. The dis- cussion group agreed that (1) the feedback session stimulated them to interact with other students, (2) they felt personally involved, and (3) the session was not formal. CHAPTER V FINDINGS, CONCLUSION, AND SUMMARY The Problem The problem of this pilot study was to offer two sets of eXperimental data relevant to the instructional effectiVe- ness of presenting examination results. Two eXperiments were conducted to compare the effects of discussion and television feedback. The first eXperiment conpared these effects on students' learning. The second SXperiment compared these effects on students' attitudes. For statistical purposes and to assess instructional effectiveness it was hypothesized that students watching television would not differ from those discussing examination results. Design and Procedures of the Study The learning experiment was conducted with 166 students in the Summer Term and an attitude questionnaire was adminis- tered to 787 students in the Fall Term of 1967. During the Summer and Fall Terms, students in different discussion sections of beginning Education 200 at MSU were presented correct answers to their mid-term examination in one of the two randomly assigned conditions: 62 63 1) Students were presented with correct test alterna- tives over closed-circuit TV with a description of why the correct alternatives were correct together with a blown-up picture of the mid-term examina- tion. These students watched a televised team of two teachers on one tape. This was also the method of feedback used during the previous three terms in Education 200. 2) Students were assigned to discussion sections con- sisting of small groups of six students. A gradu- ate assistant gave each group a copy of the correct answers to the test during the feedback session and asked them as a group to participate in discuss- ing why the correct answers were correct. Evaluation of all sections during the summer learning eXperiment consisted of determining the relative effectiveness of these conditions on the final examination. For separate statistical analysis and comparison, the final exam included 14 items from the mid-term examination which were modified and 10 items which were in the exact form as they appeared on the mid-term test.1 These repeated items, selected by a panel of judges after the feedback sessions, served to assess whether students had learned the content of these items (and/or the concept related to these items) as a result of one of the two feed- back conditions of the eXperiment. There were four final examination performance criteria or dependent variables in this experiment: 1. Number of modified recurring items answered cor- rectly. (These items were adequate items of the item pool in advance of the mid-term. Some of these items had equivalent statistical results from previous test administrations.) 1The modified items tested the same concept or prin- ciple as the corresponding mid-term items but were presented under different stimulus conditions on the final examination. 64 2. Number of recurring items answered correctly. (These items were again chosen at random from the mid-term.) 3. The above modified and recurring items combined. 4. The total scores on the final examination. Independent variables included: 1. The two treatment conditions outlined above. 2. Student discussion tendency as assessed on the TSIS. 3. TeSt scores on the mid-term examination. The treatment groups were found to have very similar scores as to their tested tendency to be active in classroom discussions as measured on the TSIS. Final examination scores of individual students within treatment groups were, therefore, classified as above or below the mean of students tested ten- dency to discuss. These levels of discussion tendency could, then, be analyzed for an interaction effect with the treatment effect of having or not having the opportunity to discuss mid- term test items. FINAL EXAM SCORES Grouped by Discussion TV Group Group Above Grouped by Average TSIS SCORES Below Average 65 Analyses of covariance were, then, conducted to evalu- ate the possible interaction of tested tendency to discuss with the experimental treatment effects of students' discuss- ing or watching TV, as these effects might interact and influ- ence performance on the final examination. Analyses of covariance were also conducted to evaluate the experimental treatment effects of students! discussing or watching TV on their final examination scores. By using the highly sensitive covariance analyses statistic, mid-term exam scores were to be controlled statistically as they might attribute to the variance of final examination scores. During the Fall Quarter an eXploratory thirty-one item attitude questionnaire was constructed. This instrument was based on the M.S.U. research of Davis and Johnson (4). It was designed to meet the same objectives of the Davis and Johnson instrument as well as provide answers to applied ques- tions raised by the faculty of Educational Psychology 200 course and this investigator. During the Fall Quarter, 787 students in this course completed the questionnaire. Reaponses to each item were indi- cated on a five point scale of agreement. The Mann-Whitney U statistic was used to determine whether or not observed differ- ences between students in the TV and discussion groups were statistically significant. The problem of objectively clas- sifying the questionnaire items was met by factor analyzing the attitude scale. 66 Results of Learning Engriment Although mid-term examination scores were statis- tically equated through use of the analysis of covariance procedure, no significant interactions were found between the eXperimental treatments and levels of student discus- sion tendency on the final exams and their subscores. In addition no significant differences were found on final exam scores and their subscores between groups of students discussing mid-term exam results and students watching a TV presentation of these results. Given the obtained consistent but non significant differences reported in the last chapter, it cannot be concluded that no differences in learning occur when students receive exam results by the different methods, but can only be concluded that no significant differences were found in this investigation by the methodology used in this study. Results of Attitude EXperiment Of the thirty-one items eXploring attitudes about examination feedback listed in the questionnaire, the Mann-Whitney U statistic revealed eighteen of these items to significantly differentiate the two treatment groups. Twelve of the items favored the instructional effective- ness of the discussion treatment and six of the items favored the television treatment with reSpect to student 67 agreement on the questionnaire. The greater number of Significant differences in favor of the attitudes of the discussion group over the television group implies: 1. Students might prefer discussing exam answers to having them televised, or 2. the choice of questionnaire items may not be representative of all possible relevant attitudes about exam feedback. Attitude Questionpgire FactOr Analyzed: Consistent Results Personal Involvement and Interaction When the question- naire was factor analyzed, three of the items which had significantly favored the discussion group loaded on one factor. This factor related to the discussion students agreeing more than the TV students that they were stimu- lated to interact and to be personally involved as a result of what they agreed was not too formal a feedback hour. This factor seemed relevant to the rationale of this study, namely that the discussion group was more actively involved as a result of feedback than the TV group. All the items were significant within the factor; however,the meaningful- ness of this fact is offset by a number of considerations. These differences in reported active involvement may be an attribute of the wording of the questions. Students might have been helined to reSpond to questions about discussion as if it had stimulated them to personal involvement even if they had not been eXposed to the dis- cussion method used in this study. The statistical dif- 68 ferences may be less inherent in the method than in super- ficial reSponse sets to formal televised presentations and to what students felt they Should have been reporting about student-led discussion. On the other hand, Davis and Johnson (4) concluded: One of the most significant differences between the lecture hall and the TV classroom lies in the fact that the lecture hall allows two- way communication whereas TV does not . . [nevertheless] questions dealing with interest and stimulation revealed no significant differ- ences between those students who saw the lectures live and those who viewed the lectures in a television classroom. It is a most tentative conclusion to suggest that students in the present study were personally involved and stimulated to interact moreso as a result of discuss- ing exam answers than by watching a more formal presenta- tion in a televised classroom. There was no attempt to assess attitudes towards exam feedback before exposure to the methods and to compare these differences after eXpO- sure. These were eXploratory studies without the intent or methodological controls to eXperiment Specifically with the factor of active student involvement. The research reviewed in earlier chapters and the factorized results of this eXperiment, then, only tend to support the conclusion that students can reSpond as if they are more stimulated to active personal involvement when discussing exam results among themselves then when viewing them over the more formal TV media. 69 Other Significant Differences Within Factors For the practical purposes of this study there were four other sets of questionnaire items factor analyzed. Individual questionnaire items loading within factors will be identified with reSpect to significant differences found between treatment groups. Learning and Understanding Course Of the twelve items loading within the factor of learning and understanding the course, five were statistically significant. Three of these items favored the television group, two the discus- sion group: Favoring TV were 3 items: "understanding interrelationships of the course, its pur- poses, and why the correct answers were correct." Favoring discussion were 2 items: "under- standing at students own rate of compre- hension" and "would not offer students exam answers differently if I were teaching course." Differences in what students learned and how they would teach the course did not reflect any clear advantage for either method used in this study. These results con- cur with the results of the learning eXperiment which reflected no statistical differences between methods. Beyond these conclusions, there is a nagging inconsistency with the simplified generalization that the obtained attitudinal results relate solely to the personal involvement factor. All of the attitude reSponses simply do not correlate with this factor. There 70 is another educational implication found weaving through other factors. Television has the advantage of bringing together interrelationships of the course and holding students attention as it does so. Discussion allows students to absorb materials at their own rate of comprehension. It could be that for some students, discussion has the greater advantage but for other students, TV is more advantageous. In this study there was no measurement of those variables that might eXplain such heterogeneous mixtures of student personality types and cognitive needs as is possibly found within both eXperimental groups. Furthermore, such a mixture might possibly account for not obtaining statistical differences in the learning eXperiment. Holding Attention Of the seven items having their highest loadings within the factor of holding students' attention, five were also statistically significant. But three of these items favored the discussion treatment and two the TV treatment: Favoring TV were 2 items: there was "not too much irrelevant material covered during feed- back"; there was "enough going on to hold students' attention." Favoring discussion were 3 items: "the feed- back session was not boring," nor was it "over-Simplified";'the feedback materials held students' attention." Apart from these obtained differences, there were 71 again no over-all advantages indicated from eXposure to either method. Davis and Johnson (4) concluded the "phenomenon [of attention] requires further study." Beyond this obvious conclusion, influences in the present experiment should be considered. It is understandable students would be attentive to those aSpects of feedback informing them how they have been and will be tested. Hence, students reported they were relatively more atten— tive to the materials given them by the test makers in the discussion condition of the eXperiment. In compari- son the TV group reported relatively greater attention during the whole of the TV presentation when they could attend directly to what the test makers were presenting rather than to their hand-out materials. Further, students did not feel there was as much irrelevant material covered by the test makers over television as during student-led discussions. However, this differ- ence as well as the contradictory complaints students had that TV was both more boring and over-simplified than discussion might have been inherent in the nature of student prediSpositions. Claripy of Presentation and Noise Interfering with Instruction Of the nine items related to the two factors of clarity of presentation and noise interfer- ing with instruction, five were again statistically Significant. Only one of these items favored the TV 72 treatment: Clarity of presentation was composed of six items: Favoring TV was 1 item: "as a result of feedback, exam questions no longer seemed ambiguous." Favoring discussion were 3 items: "students reported no difficulty seeing materials," that not so many students needed time to absorb materials," nor did they agree that many "improvements were needed in the feed- back session." Noise interfering with instruction had 1 of 3 items favoring discussion: students reported "relatively less trouble hearing what was going on during feedback." It is difficult to generalize from the statistical significance to the educational significance of items related to clarity of presentation. The advantage of TV may be in the clarification of ambiguous test questions. However, students viewing television said they needed more time to absorb the Significance of these same ques- tions. Perhaps the request for improvements in the presentation of test answers suggests there are inherent visual and audial problems in the TV media or at least that is what students are prone to report. Davis and Johnson (4) also mentioned: students in television classrooms had difficulty seeing lecture materials . . . in some TV sections the disturbances created by inattentive students dis- tracted others. Only 12% of the students in both experimental con- ditions of the present study reported "disturbances 73 interfering with instruction." However, when the question was reworded, the TV group reported relatively more "trouble hearing what was going on during feedback." These results offer no clear generalization about how the television media should be improved. Basing tentative conCIusions on all the factorized results, it may be the need for improvements in exam feedback relates as much to feelings about active participation and personal involvement as to actual noise interfering with instruc- tion. Conclusions The following conclusions were therefore reached in this study: 1. Students discussing exam results in small student-led groups reSponded to a question- naire as if they had more actively partici- pated as a result of feedback than did students watching a televised presentation of these results. Specifically, discussion students agreed more than TV students that they had been stimulated to interact with each other and were personally involved as a result of what they agreed was not too formal a feedback hour. (This is an admittedly very tentative conclusion in view of obvious 74 methodological problems.) 2. No significant differences on final exam scores and various subscores of final exam performance were found between groups of students discussing mid-term exam results among themselves and students watching a TV presentation of these results. 3. No Significant interactions were found among individual student discussion tendencies as measured on the Research Form B of the Academic Behavior Inventory, students' final exam scores and their subscores, and whether students SXperienced discussion or television feedback of their objective mid-term exam SCOI‘GS o Implications for Further Research The present study is SXploratory; consequently, an issue raised by the findings is the need to carry out additional studies to verify the present results. For example, in addition to further research on the present attitude scale, there is a need to assess the effects of adding and deleting attitude items. Further, the attitude scale should be studied more fully under dif- ferent conditions. It is to be remembered from page one that Sassen- 75 rath and Gaverick (20) (and Curtiss and Wood, 3) found that learning was more effective when students discussed several objective classroom examinations with their teacher than if they merely viewed an authority's eXplanation of these answers. But in this study the effect of small group, student-led discussion of one objective examination was compared to the effect of students viewing authorities' explanation of their answers via closed-circuit television. Specific questions needing further research relate to the methodological differences of these studies: It would seem that a longer time to discuss examination results might be helpful in demonstrating the effective- ness of discussion. In addition choosing more items and more difficult items to repeat from one test to another increases the reliability and the chance for obtained differences to be significant. It may be that when the constructors of course tests televise exam answers, they also cue students to what to look for in subsequent course exams. This effect may offset any advantages students might feel about dis- cussing answers among themselves. This might eSpecially be true when students have the advantage of discussing course materials during half their class hours as in the present study. Certainly there are many local influences in any SXperiment. What students learn from exam feedback and 76 student attitudes about exam feedback will only be clari- fied by a number of research studies under more controlled conditions. It is to be emphasized that the purpose of analyz— ing the attitude questionnaire in this study was eXplora- tory. The following observations might contain the basis for further study, possible extension, and/or replication of this study: When students watch television as compared to discussing examination answers, they reSpond as if they feel: 1. more bored with a "formal" and "over simpli- fied" TV presentation, 2. less "personally involved" and that they are not stimulated to interact with other students, 3. they had difficulty seeing and hearing what is going on during feedback, 4. they are not understanding course materials at their rate of comprehension, and need more time to absorb and assimilate the course materials. 5. improvements need to be made in the way they received their examination results and that they would give students examination results differently if they were teaching the course, and 6. they were not attentive to materials presented them. The following observations might also be the basis for further study of the positive benefits students will feel from listening to TV feedback: They, more than students who discuss these results, will feel as if: 77 7. They are now more certain of why the correct answers are correct and not feel that so many test questions are ambiguous, 8. they better understand the interrelationships of the course; that not too much irrelevant material had been presented; and that the pur- poses of the course are clearer, 9. enough was going on over TV to hold their attention. Determination of cause and effect relationships on both the attitude questionnaire and the Research Form B of the Academic Behavior Inventopy used in this study would prove helpful. Regular use of these scales over a period of years or periodically would be helpful in at least three ways. First, it would provide evidence of the effectiveness of varying the method of feedback. Second, it would provide an index of reliability. And third, the effect of course content, teaching staff, and student differences on a number of variables could be determined. For these instruments to be as useful as possible, continued efforts should be made to refine the items and categories used. New items peculiar to new problems may be added to facilitate evaluation and representativeness of the population of instructional concerns. In general, use of these scales in other academic institutions and courses both similar to and different from those in the present sample would be useful further research. l. 3. 7. 8. 9. 10. 11. BIBLIOGRAPHY Balch, John. "The Influence of the Evaluating Instru- ment on Students' Learning." Am. Ed. Res. J., Vol I (3), May 1964, pp. 169-182. Bloom, B. S. "Thought Processes in Lectures and Discus- sion," Journal of General Education, 1953, 2, pp. 160- 1 9. Curtiss, F. D.. and Wood, G. G. "A Study of the Relative Teaching Value of Four Common Practices in Correcting Examination Papers," School Review, 1929, 32, pp. 615- 23. Davis, Robert H., and Johnson, F. Craig. Evaluation of Regular Classroom Lectures Distributed by CCTV to Campus Classrooms. Project Report No. 202, Educa- tional Development Program, May 1966. Dennis, L. E., and Kauffman, J. F. (eds.). The Colle e and the Student. Washington, D. 0.: American Council onfiEducation, 1966, pp. 76-100. Edwards, Allen L. E_perimental Desig_ in Psychological Research. New York-Chicago-San Francisco-Toronto- London: Holt, Rinehart and Winston, July 1963. Gage, Nathaniel L. (ed.). Handbook of Research on Teaching. Chicago: Rand McNally and Co., 1963. Giddan, Norman 8., Lovell, Victor R., Haimson, Alvin 1.. and Hatton, John M. A Scale to Measure Teacher- Student Interaction. Stanford University Counseling Center, Unpublished paper. Krauskopf, C. J. The Use of Written Responses in the Stimulated Recall Method. Unpublished doctoral dis- sertation, Ohio State University, 1960. (Dissertation Abstract, 1960, 21, 1953). Korn, H. A., and Black, J. D. Factors Influencing Student Participation in Classroom Discussion. Unpublished paper. Stanford Counseling Center, 1962. McGeoch, J. A., and Irion, A. L. The PS cholo of Human Learning. New York: Longmans, Green, 1952, pp. 46. #78 .A'l' 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 79 McKeachie, Wilbert J. New Developments in Teaching. Unpublished paper. Prepared for Higher Education Literature Search Project, U. S. Office of Education under Contract to Duke University. McKeachie, W. J. Procedures and Techniques of Teaching: A Survey of EXperimental Studies. In N. Sanford (ed.) The American Collegp. New York: Wiley and Sons, 1962. McKeachie, W. J. "Anxiety in the College Classroom," Journal of Educational Research, 1951, 55, pp. 153- 160. McKeachie, W. J. Teaching Tipp. Ann Arbor, Michigan: The George Wahr Publishing Company, 1960. McNemar, Quinn. Psychological Statistigg. New York- Lcndon-Sydney: John Wiley and Sons, Inc., 1962, pp. 451. Pace, C. R. PerSpectiveS on the Student and his College. In L. E. Dennis and J. F. Kauffman, The College and the Student, Washington, D. 0.: American Council on Education, 1966, pp. 76-100. Stern, G., Stein, M., and Bloom, B. Methods in Personality Assessment. Glencoe, Illinois: Free Press, 1956. Thistlethwaite, D. L. College Press and Changes in Study Plans of Talented Students. Evanston, Ill.: National Merit Scholarship Corp., 1960. Sassenrath, Julius M., and Garverick, Charles M. Effects of Differential Feedback from Examinations on Retention and Transfer. J. of Educational Psy., 1965, 6, 5, pp. 259-263. Sanford, Nevitt, (ed.). The American College. New York: Wiley and Sons, 1962. Seigel, Sidney. Non-Parametric Statistics. New York- Toronto-London: McGraw-Hill Book Company, Inc., 1956. APPENDIX A The strong influence of the evaluating procedures on students' learning can be summarized from this investigator's (1) 1964 research with respect to the following groups of findings: 1) The influence of the evaluating instrument (on students' learning). 2) The influence of essay, multiple choice, and other types of tests (on students' learning). 3) The influence of the complexity of the evaluating instrument. 4) Frequency of testing. 5) Time between studying and testing. 6) Knowledge of results. 7) Interaction of teaching methods and student char- acteristics. The consensus of the studies over the past Sixty years sup- ports the conclusion that the evaluating procedures do influence students' learning. Learning is affected both by the external elements of the situation, including the amount, organization, complexity, and meaningfulness of the material to be learned, and by internal factors characteristic of the learner. 80 APPENDIX B Instructions to Teaching Assistants Pre-Tested on a Trial Mid-Term This Summer TV Feedback Condition Please hand out Review Formats and Answer Sheets to all students. Please read aloud: The test feedback will be on closed-circuit TV. You have been given a Review Format with instructions not to take notes. You may take the Review Format home with you, but please return all other materials at the end of the TV feedback. (Coincidentally, please do not review course materials preced- ing the mid-term after this date. Please take notes during the feedback session as to your estimate of a) percentage of students attentive to the feedback; b) how attentive you think they are. Please do not permit your biases about method of feedback or those of your students to be discussed in your classroom.) Discussion Condition Please "randomly" assign students to six-man groups, hand out Review Formats, Tests, and Answer Sheets. Please give one group member in each group a keyed copy of the mid- term, with instructions as follows: 81 82 You are to work as a group discussing alterna- tives to test items. These answers will be con- firmed by the group member with the keyed test. You may take the Review Format home but please return all other materials at the end of the feedback session. (Coincidentally, please do not review course materials proced- ing the mid-term after this date. Please do not permit your biases or those of your student to be discussed in your class- room.. Please take notes during the feedback session as to your estimate of a) percentage of students discussing; b) being attentive; and c) how attentive these students are dur- ing feedback.) APPENDIX C Instructions to Subjects TV Feedback Condition "The purpose of test feedback is to help you to clarify in your own mind the correct alternatives to test items. Some of the items on the mid-term will reappear on the final." "You have been given a memeoed Review Format which you may take home to aid your recognition of test items. Please do not take any notes during feedback." "The test feedback will be presented on closed-circuit TV, and will cover all items on the mid-term. PLEASE DO NOT MARK YOUR IBM ANSWER SHEET." Discussion Condition (Pre-tested on a trial mid-term this summer) "The purpose of test feedback is to help you to clarify in your own mind the correct alternatives to test items. Some of the items on the mid-term will reappear on the final." "You will form six-man groups in your sections to dis- cuss items. Each group member will be given a Review Format which you may take home to aid your recognition of test items. Epel free to comment about the test to each other in your group but do not take any notes this hour. You are being asked to systematically discuss each question permitting each 83 84 member of your group the opportunity to participate actively in the learning process. You will want to read each question on the test given to you before asking the member of your group with the correct answer to read it to you. Please do not mark your IBM answer sheet." APPENDIX D TSIS Answer either 1 (true) or 2 (false) on the answer sheet. 1. 2. 3. 4. 5. 9. 10. 11. 12. 13. I often talk to the teacher after a class. I ask a lot of questions in class. I discuss my test results personally with an instruc- tor. I always raise my hand in class when I know the answer to a question. I am able to communicate more easily with a teacher in the privacy of his office than in the classroom. I don't like to be called on by a teacher in class unless I have raised my hand or otherwise indicated I wish to Speak. I like competitive intellectual discussions with my teachers. If a teacher critically evaluates something I've said in class, I tend to clam up. When a teacher criticizes something I've said, it stimulates me to defend my ideas or logical position. I much prefer classes where a teacher lectures and only infrequently encourages questions from the students. - I seldom talk in class. I usually assume that teachers know their field of endeavor and hence seldom discuss or argue with them about interpretations, test results, etc. I'd rather read more about something that interests me than have an intellectual discussion with a teacher. 85 11+. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 86 I talk in class only when I'm absolutely sure of my facts and ideas. I have a great deal of difficulty talking in classes where I know the instructor is carefully evaluating what I have to say in order to "grade" my discussion participation. I like to mingle with teachers and other faculty at receptions, teas, etc. I like to get invited to a professor's house so that I can get to know him better. I would never think of inviting a teacher or pro- fessor to a party at my house. When something I don't understand arises in class, I'd rather discuss it with a fellow student than the teacher. When my classroom participation is not being graded, I talk much less. I don't enjoy classes where the teacher has little time for conversation with the students. I would like to be a college professor. I much prefer classes where the teacher really talks with students, not just 23 them. I tend to shy away from talking in classes where the teacher stirs up heated and intense discussions. APPENDIX E Course Questionnaire Fall Quarter 1967 By answering this questionnaire you will be helping the University in a very important way to evaluate different methods of instruction. This evaluation will not be made until after your instructor has turned in your final grade, and your answers to these questions will not affect your grade. Instructions Fill in the top three lines on the answer sheet sup- plied. Print the information with the soft lead pencil given to you. In the box at the right top of your answer sheet, fill in the correct blanks for your student number. Respond to the following statements by blackening the Space on the answer sheet which, according to the key below, best describes your reaction to the statement. KEY: Strongly agree Agree Uncertain Disagree 1 2 3 L; 5 Strongly disagree 87 1. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 88 I had no difficulty seeing what was presented to me during the hour I received my examination results. As a result of the feedback session I will be able to study more effectively for the final examination. The feedback session helped me understand the interrela- tionships among various parts of the course. I felt the need to ask questions which were not answered during the hour when I received my examination results. I frequently needed more time to absorb or assimilate material presented during the feedback session. As a result of the feedback session, I feel I can achieve a higher grade on the final examination than I did on the mid-term examination. I feel perfectly free to discuss questions stemming from the mid-term examination with my instructor. Often there wasn't enough going on during the classroom hour when I received my examination results to hold my attentiOno The materials given to me to look at during this feedback hour held my attention. I learned a great deal during this feedback hour. I would have understood the material better if it had been possible to discuss test questions with my instructor dur- ing the feedback hour. This feedback hour helped me to understand course materials at my own rate of comprehension. The feedback session was so stimulating that I thought about the subject matter which was covered a great deal after the session was over. I found my attention wandering frequently during the feed- back hour. If I were teaching this course I would give students exam- ination results the same way I received them. I could always hear what was going on during the feedback hour. It was frequently difficult to follow the feedback presen- tation because of disturbances in the room. 89 18. A desirable feature of this course is the way examination results are given to students. 19. Several improvements need to be made in the method I received my examination feedback. , 20. As a result of the feedback session I better understand what my instructor was trying to accomplish in my dis- cussion section. 21. Too much irrelevant material was covered during the examination feedback session. 22. The feedback session stimulated my interaction with other students. 23. The way I received examination feedback was boring. 24. I welcome the opportunity to have examination feedback. 25. The feedback session seemed too formal. 26. As a result of feedback I felt certain of why the correct answers were correct. 27. The purposes of the course were more clear to me as a result of the feedback session. 28. I understand the content of the course better as a result of the examination feedback. 29. Even after feedback, I felt that many items on the mid- term examination were ambiguous (more than one correct answer). \ 30. I felt personally involved during the feedback session. 31. The examination feedback presentation was over simplified. 9O Hopoa see. no peooacsemam mmsmm : ooa mom m mm m.mm oma M.Sm :NH n.0m mm H.H : scammSOmam ooH mo: o.o mm N.sm mm m.mm mma mm med «.3 AH >9 n z a z a z a z a z a z ooHMSmHQ ooamd Hmpo9 mammoapm oonwmmam campuses: ooawd hawsoapm oadom .omHSOO was ac mpama msodnmb macaw wadsmsodpmaohaopaa one pampmaopad ca poaaom soammom Momnpoom o£9 >9+ cosmoamaawam mo SOHpooHHQ mmw.+ wnaomoa Hopcsm ”m aopH omndoo exp 90 wQHBQMDmHopQD «H Houodm manoao scammSomHQ one >9 mo momaoamom opacssoapmoSG so paosooawd ho hososvonm pad .mosam> D hoapH£3tsamz .mwsapmoq Hopodm .mhopomm m NHQZW994 91 Ho>oH mo. ho phooahhawhm mambo D ooa mmm ma em 0.9N mm mm Sch :m mm m.: 0H achmeoth ooh so: m.mh as h.hm smh H.mm aha m.mh No m.m em >h m z m z m z is, z m z m z ooh omhm ooh 4 Hopo9 HHwhohpm oohwomhn shophoosD oohwd HHwhohpm oHoom H as: osom one mpHsmoh hohpohhsoxo .Soap oobhoooh mpaoespm obhw oHsos H othoo was» whasooop ohoz H hH >9: oohoohhhcwhm ho achpoohhm mmd.+ DGHSSOH hopoom «ma aopH Ho>oh hoo. no peoohhhewhm mmsmm p ooh hon e.o mm m.om hmh h.:m am o.om mos o.m mh eohmoSomho ooh so: «.mh No m.os mmh m.hm mm «.mh mo s.h A >9 m z m z m z m z m z m z ooh omhm ooh 4 Hopo9 HHmSthm oohwomhm shohhoosD oohwd HHwDthm oHoom .hohm :hoSohaaoo ho opoh £30 98 pm mHohhopoa othOO whopmhooas on os doaHo: hso: xooncooh mH£9 >9: ooeoohhhemho ho eohpoohho mm:.+ mahoo0H hopoom ”NH aopH 92 HoboH Hoo. no panchmahwhm mammw D ooh hon m.oh mm e.sm mma e.em emh sh he h m eohmmSomho och mos h.s mm m.mm :mh m.hm mmh s.om och m w >9 m z m z m z m z m z m z ooh omhm ooh 4 Hopo9 aHmDthm oohwoohm shophooaD oohwd hHDhthm oHoom .QOHmmom Moopooom osp mo pHDmoh o no on on hooHo ohoa ohoz othoo on» mo momoahsm o£9 >9+ ooaoohwhamhm ho hoapoohhm mH>.+ mhheo0H honoom ”um aopH HoboH no. no panchhhhwam mammm D och hem m.hm >9 9.9m ooh h.hm on o.mm mm N.: ma eohmmSomhm och . mos m.oh mo o.mm shh m.hm mm o.am mhh e.m mm >9 m z M 2 m z m z R z & z ooh oth ooh 4 Hopo9 hHwDthm oohwomam ChophoosD oohw¢ hHwQthm oHoom .poohhoo ohoz mhoSoao poohhoo on» 9:3 mo Chophoo pHoh H Mooneoow ho pHSmoh d o4 >9+ oesooHHHQme mo achpoohhm mom.+ msheooH honoom “mm aopH hh. hphhhnopohm 9o Ho>oh mohos a och mom 0.0 em m.om :9 m.ma 99h 9.9a m9 9.: mh eohmmSomhm och Oh: h.m hm m.mh mm 9.9: mmh m.mm mm o.m mm >9 9 z a z m z is z m z m z oohmomam ooh d Hopo9 HHDDthm oohmomhm shophooaD oohw< HHwSthm oHoom 93 .QOHposHsowo Shop:ehs och so She H can» achpohhaoho Hohhm one so oemhw hommhn o o>oh£oo soc H Hooh H .sOHmmom Moondoow on» 90 pHSmoh o md om>.+ wsheo0H hopomm no aopH mo. hphhhpenohm 9o Ho>oq mmoms p 009 men 9.9 mm 9.0m o9 o.om hhh m.mm mmh N.m oh eohmmSomhm OOH Has m.a on m.hh as mnmm was h.mm and s.s mm >9 9 z m z a z 19, z m z m z ooh omHD ooh 4 Hopo9 HHDQthm oohwomhm :HophooaD oohwd HHwhohpm oHoom .QOprHhHEMNm Hohhm one hOh >Ho>Hpoo99o ohos Heapm on oHQo on HHHS H achmooo soonpooh on» no pHsooh o md :H>.+ mahooOH hopoom “N HopH ’1 94 hm. hphhhnopohm 9o Ho>oq mhmms D och mom mm mm h.es has m.eh mm 9.0h mm h m eohmmsomhm 00H mos 9.0N 00H >.Hm HHN n.0H N: H.0H H: N.H m >9 9 z m z m z 19 z m z m z ooh omhm oo ¢ Hopo9 HHmQthm oohwomhm shophooaD oohwd HHDQthm oHoom .hobo mos scammom oSp hopwo Hood poohw o dohoboo mos sohnz hounds poonnsm ogp phone psmsoap H posh mhhpoHsshpm om mo: achmmom Moonooom o£9 mwm.+ wshem0H houoom ”MH aopH o9. hphhhnonohm 9o Ho>oq omhms o ooh mom h.m mm e.mm hmh e.mm hmh o.hm ma m.m m eohmmsomhc ooh mos m.hh a: o.mm hmh 9.mm mmh m.om mm 9.: om >9 m z m z m z m z z m z ooh mmHQ ooh ¢ Hopo9 HHwhohpm oohwomhn shophoohD oohwd hHmQthm oHoom .hsos Moopoooh was» whhhso Hoop poohw o oohhooH H mma.+ meheooh hopooh ”oh aoph 95 _ oh. hphhhnonohm 9o ho>oq hosmo a och hon h.:h hm m.mm has m.mm has o.oh oo N.m m eohomsoohn och mo: s.Ah 99 mm nah h.om mmh h.oh oo m.h o >9 9 z z 9 z .9 z m z :m:::m¢m ooh omhm ooh Hopo9 HHDQthm oohwomhm shophooaD oohws hHwQthm oHoom .hohpoom Schoohooho Hadhh noHHasoooo op wahahp mos hopedhpmsh Ha poms Shopahoohs hoppon H achmmom Moonoooh one 90 pHsmoh o o4 mmm.+ weheooh hopooh ”om aoph mh. hphhhpopohm 9o Ho>oq somoo o ooh oom m.mh ma o.mm mm on och 9.9m soh h.m hh eohmmeooho ooh mos m.mh am o.om hmh h.mm mhh m.mm mm N.e 9h >9 m z m z m z m z m z z ooh moan ooh Hopo9 HHwQthm oohmomhm shophoohD oohwd hst0hpm oHoom .mphodhpm op sobhw ohm mpHdmoh achpohaaoxo 9oz o£p mH omhdoo mth no ohhpoow oHDohHmoe 4 mmm.+ waHoooH hepoom «mH aopH ho>oh 900. he peoohhhewhm 909:0 > och mom 9.mh 9m hm mmh e.mh me e.mh on m.m om eohmmSooha och oh: om mm m.wm 9mm m.hh 9: 9.9 mm e.m ch >9 9 z m z m 2 us 2 m z m z ooh oohm ooh 4 Hopo9 Hstohpm oohwomhm shophoohD oohwd hHwaohpm oHoom .sOHpaoppo as oHoD op mpHsooh aohpoahaoxo as oobhoooh H hogs hacs EOOhmmoHo oap whhhso so whHow swsoso p.2mo3 ohoSp hopmo >9+ oohdoa9hcwhm no QOHpoohHQ m:9.: whdoooH hopoom ”m aopH Godpzoppd dHom ”HH hopoom /O 0/ om. hphhhnonohm 9o hopoq mo909 a OOH own m.w om m.mm :oH m.9m 00H :.:m :NH H N scammsomhm och mos e.o om m.om :mh 9.mm moh em 99h 9.9 ma >9 9 z m z z 19 z m a m z ooh swam .LWIIIII: ooh Hopo9 HHwQthm oohwoohm shophooaD oohws Hstohpm oHoom .Moonoooh achpoahaoho on» 90 pHsmoh m on hoppon omhdoo och Ho pnophoo oSp esopohoohs H m:n.+ wQHHSOH hopoom “mm aopH 97 hoped Hoo. pm pzoohohsohm m:hmo a och mom m.hh m: m.:m mmh :.uh mo n.0h mm m.m ma :ohmmsomho och mo: H.5h on ow mim m.mh mm 3.: ma a n >8 m 2 R z m 2 1m 2 m z m z ooh oohn ooh Hopoe hHwSthm oohwomhm shophoosb oohw4 hHwQthm oHoom .COhmooo Moopoooh soaponhaoxo on» mnhhzd oohoboo ooz Hohhopoa pnoboflohhd £058 009 >9+ ooQoOHhhswhm ho Schpoohhm omm.u wghwoOH hopomm ”hm amph ho>oh mo. pm psoOHMHsmhm momwo : ooh mom N.N m eh m: m.ma m: :.:m nah m.mh om schooneoan ooh mo: h.m hm o.mh om NH m: o.mm mam m.oa :: >9 m z m z m 2 [R z m z m z ooh oohm ooh Hopoa hHmQthm oohmooha shophoomp oohwo haw20hpm oHoom .EOszoppo ha daos hzos Moopdooh oh£p wnhhsc no Mooa op oa op mobhw maohhopoa one >8: ooQoOHhhswhm ho sohpoohhn mmm.+ wnhdooa hopoom ”m aopH 98 Ho>oa no. no pnooh9hswhm Hmm:m D 009 mmm :.m 99 m.o¢ :oh m.9m mm 99 co 9.m m9 nohmmsomhm och mo: m.m 9m m.m: mmh m.mm om 9.mh mm 9.m m9 >9 m z a 2 J 2 IN 2 m z m z ooh omhm ooh Hopoa hawSthm oohwoohm chophooha oohw4 hHwQthm oaoom .oohhhamahm hobo mo: :ohpopsooohm Moondooh soaponhaoxo one >9: ooQoOhmhcwhm ho Soapoohhn m::.| mnhoooa hopoom ”hm aopH ho>mh mo. no psoOhohsohm :mmoo o ooh mom m.w mm 9.9m mmh m9 :m m.om mm m oh :ohmmsomhn och mo: :.m mm «.9: mmh m.oh mo mm om o.m mm >9 m z a z m z m z m z m z oohmoohm ooh Hopoa hanthm oohmoohm Chophoons oohw¢ hammohpm oaoom >9: ooQoOHMAQwhm ho soapoohhn .whhhop mos Moonoooh soapozhaoxo dobhoooh H no: one omo.u wnhumoh nopoom “mm amph 99 mi. hphhhpoQOhm 9o ho>oq ommhu D OOH hon m.H m m.m :9 m.> mm :.o: 01H m.m¢ mwh GehmmSOmhm och mod H m m.m m9 m.m mm 9.3: omh m.m: duh >3 m z o z lmr z iR z m z m z ooh oohm _ ooh Hopes hanthm oohwomdm chophoosb oohwd hHwSthm oaoom .xoondooh Schpochaoxo oboz op hphnzphommo oSp oaooaoz H hmm.+ ochdo09 hopomm ”3m amph mm. hththQOhm ho Ho>oq mommn D OOH mom H.m mm 0.9: @9H m.mH w: N.mm :m m.m Hm sedmmsomdm och mo: n.0h m: m.m: wmh a.oh a: m.:m 009 0.0 mm >5 o z 9 z m z m z m z z ooh omha ooh Hopoa hanthm oohmoohm Chophoosa oohw< hHwQthm oHoom onu.n wshdooa hopoom .hdon Moondooh oz» whhhzo hapzosvohh wshhodqoz Soapnoppo >8 @2509 H ”:H amph "‘3' 100 HoboH H00. no pQQOHMHQme a 00mm: D och mom w.mh mo 9.9m mam h.m mm m.m w m.m m scammsomhm ooh mo: m.: 99 m.o¢ ooh n.9m no n.9m no n.: ma >9 m z m z m z m z m z z ooh omhm ooh Hopoa kHwQthm oohwomhm shophoocb oohw4 >Hw20hpm oaoom ‘|ll| .HdfihOh oop doaooo achoooo Moopdooh osa >9- monoohohsmhm 90 schpomhha hwm.u mahomoh hopomm ”mm aoph hm>mh hoo. pm poochohsmhm mammm a QOH mom :.9 m 9.0m mm :.99 no H.0m ooh m.09 mm achmmsomha OOH mo: N.9H on m.om «mm :.mh no o.m mm 9 : >9 m z m z m, z m z m z ooh omhm ooh Hopoe hanthm oohwomhm shophoosb oohw4 hawSthm oHoom .ophodSpw hozpo :phz Schpoohopzh ha dopoasahpo schoooo Moondoom one >9: monochhhzwhm ho :ohpoohhm >m>.+ mnhuooa hopoom ”mm aopH oopoohopsH duo cobaoan ”HHH honoom 101 Ho>oH hoo. no psooh9hswhm wommm D 009 com o.n m9 o.mh on m.mh mo :.:: ooh h.oh mm nohmmsomho 009 no: o.mh m: w.mm mm m.ma 2m h.mm mmh o.ma mm >9 m z m 2 M 2 1m 2 m 2 IR 2 ooh oohm oohwd Hopo9 hanthm oohwomam shophoonb oohw4 hawsohpm oaoom :ozhaoxo ha .opazmoh Soap oo>hoooh H hSOS oSp wchhzo o8 op oopzomohm mo: nos: wshoom hpafioammad on do: H >9: ooQoOhMHQwhm ho zohpoohhn Hmm.+ mahoooa hopoom ”a aopH :ohpopsooohm hooao ~>H hopoom hm>mh Hoo. pm pcaohohowho mmmmm > 009 Hon :.h n ma 3m :.mh 09 h.mm ooh N.Nh :: :ohmmsomho 009 no: 9.:N mm N.>: Nah n.39 mm m.ma :m N.H m >9 m z m z m z m z m z m z ooh oohm ooh Hopo9 hamQthm oohmoohm chophoozb oohw4 hawsohpm oaoom >9: oozooHMHQwhm mo Qohpoohhm .nohoooo xoonooou on» wzhhso cobaobna AHHoQoohog paom H 999.+ oohoooh hopoom «on aoph 102 Ho>oH 900. no pnoohwhswhm mmew D 009 com m.m w om m9 m.mm 9w 9.9: mz9 «.39 9m :o9mmsom9o 009 mo: m.9 o :.m9 no :.m9 no m.:: Nm9 «.mm mm >9 m z m z m z m z m z m z ooh oohm ooh Hopo9 >HwQthm oohwoohm shophooCD oohw4 >szohpm oHoom .xoonooo9 Sohposaaoxo >8 oo>9oooh H ooSpoa on» SH oooa on on doom mpzoaobohgah Hohobom >9: oonoohmhswhw mo SOHpoohHQ N>:.: mahoooH hopoom ”m9 aopH 9o>o9 900. no pzoo999ow9m mmooo o 009 :om o.m 99 m.9m 909 o.m mm 9.9m mm9 m.mm mm o09mmsom9o 009 09: N w m.m9 no m.> om N.o: mmH 0.:m ~39 >9 m z m 2 U z :& z m z m z ooh ooHQ ooh Hopo9 >Hw:0hpm oohmoohm nHophooSD oohwd >Hw:0hpm oHoom .GOHoooo xoonooom on» mzhhso oopcooohm Hohhopoa opoHHEHooo ho Dhomno op oahp ohoa ooooos >Hpmosvohm H >9: oosoohghnwhm ho Schpoohdm 9m:.: wQHooOH hopoom 9m aopH 103 B}. (A. : in. 99. 99999909099 90 90>09 00:00 0 009 000 0.0 99 0.00 099 9.99 00 :0 009 0.09 00 2090000099 009 090 9.0 99 9.90 999 0.0 00 0.00 009 0.00 00 >9 m z m z m 2 l9, 2 m z m z ooh omhm ooh 4 Hono9 9stohnm oohwoohm shonhoosD oohw4 hHwnohnm oHoom .oanmoh zohnonhsoxo he oo>9oooh H Doss hzon onn mqhhso oohozmzo no: ohoz £0923 oDOHnmosw Moo on doom onn nHo9 H Non.: mahoooH honoom u: aonH HoboH 900. no nsoohmhzwhm moowm D 009 900 0.0 0 m9 :0 0.09 00 0.00 009 0.90 mm 0090020099 009 00: 0.: 09 0.00 00 0.09 90 9.90 909 0.00 90 >9 m z m z m z R z m z m z ooh oo9m ooh Hono9 meQOhnm oohwoohm nhonhoocD oohw¢ >HwQOhnm oHoom >9+ oosooHMHQme ho mohnoohHQ .Ahoxoso noohhoo ono nosn ohoav ososwhnao ohoz schnochaoxo Shon:oha osn no maonh hzoa nosn nHom H .Moopooom honho sobm 39m.: wnhooOH honoom .00 0009 104 90>09 900. 00 00009990090 00000 0 009 900 0.0 0 9.09 00 9.0 00 0.90 000 0.00 09 009000009o 009 900 0.0 09 0.00 00 0.0 00 0.00 090 09 00 >9 2 0 z m z :9 z m z m z oohmomHD oohw« Hono9 9Hmnohnm oohwooHD DHonhoozD oohw¢ mesohnm oHoom .hsos xoonoooh osn wzhhso Go 09900 mo: nosz hoos omozHo oHsoo H >9: oosoohmhcmhm 90 DOHnoohHD m99.+ wDHUoOH honoom 909 EonH nohnoshnomH £n93 oohowhonnH ooHoz u> honomm 99. 90999000099 00 90>09 00000 0 009 000 0.9 0 0.09 00 0.09 00 9.00 909 0.00 00 2090000090 009 000 0.9 0 0.09 00 9.99 09 0.00 909 0.90 009 >9 m z m 2 R z m z m -2 0 z ooh om9D ooh 4 Hono9 mHmQOhnm oohwoo9m chonhoozD oohm¢ meEOhnm oHoom .hson xoopooom osn wchhso honoshnmsh ma £n93 ozohn :mosw noon mosooho on oHnHmoom soon 00: n9 99 honnon Hohhonoa onn ooonohoocs o>o£ oHDoz H 009.: 0090009 900009 “99 0009 105 00. 90999000099 90 90000 00009 0 009 000 0.09 00 9.00 000 0.09 00 0.09 90 0.9 9 0090000090 009 000 0.09 90 9.00 000 9.0 00 0.99 00 9.9 9 >9 m z m z m z m z 0 z m z ooh 009D ooh Hono9 >H000hnm oohwoo9m QHonhoozD oohw< hHwnohnm onom .aooh osn :9 moozonhsn090 mo omsooop Sohnonsooohm xoondoo9 osn SOHHom on nH5099990 >anoswoh9 mos nH 009.: 0090009 900009 “99 0009 90. 90999000090 90 90>09 00009 0 009 000 0.0 0 0.0 09 0.09 00 0.00 009 90 099 0090000090 009 990 0.9 0 9.0 00 0.09 00 0.00 909 0.00 009 >9 m z z m z m z z m z ooh 009m ooh < Hono9 >Hw00hnm oohwomhm SHonhooQD oohw¢ >Hm90hnm oHoom .honodhnoQH 08 anS schnomhaoxo Shon:oha onn Echw mQHanno ouchnoosw monomHo on oohm >Hnoo9hog Hoow H mm:.+ wsadood honoom .9 0009 APPENDIX G Rotated Factor Loadings Item Factor Factor Factor Factor Factor No. No. 1 No. 2 No. 3 No. 4 No. 5 1 0.2572 -O.721 0.2366 0.3209 0.3077 2 0.7139 0.1642 0.0630 0.1282 -0.0044 3 0.6582 0.1129 -0.1196 0.1605 0.1759 4 -0.0815 -0.0753 -0.0511 0.7024 -0.0282 5 -O.1077 0.2576 ~0.1858 -O.4808 -O.2686 6 0.7191 0.0434 0.0676 0.0053 0.0105 7 0.1399 0.1511 0.0541 ~0.0172 0.4531 8 -0.0979 -0.7480 0.0929 0.0411 -O.178O 9 0.2745 0.5930 0.2205 -0.0066 0.0069 10 0.7245 0.2934 0.1366 0.1318 0.0512 11 -0.0804 -0.0479 -0.1064 -O.7377 0.0900 12 0.4579 -0.0063 0.4076 0.3361 0.1148 13 0.5618 0.3979 0.1890 -0.0502 -0.0073 14 -0.1804 -O.7704 -0.1084 -0.0317 -0.0454 15 0.4564 0.2026 0.3125 0.4061 0.0939 16 0.0682 0.0541 0.1627 0.0867 0.7146 17 -0.0240 -0.1516 0.0288 -0.0445 -O.7452 18 0.5329 0.2124 0.3084 0.3035 0.0975 19 -0.3605 -0.2129 -0.3018 -0.4723 -0.1332 106 '.‘. - 'Jl'.".tTJ if“ 107 Item Factor Factor Factor Factor Factor No. No. 1 No. 2 No. 3 No. 4 No. 5 20 0.5847 0.0844 0.1771 -0.0279 0.1971 21 -0.1107 -0.5560 0.1502 -0.2537 -O.1858 22 0.1566 0.0628 0.7865 -0.0675 0.0289 23 -0.2714 -0.6199 -0.3164 -0.1886 -0.1095 24 0.l373 0.3909 0.1426 -0.0365 0.0094 25 0.0405 -0.1689 -0.5809 -0.2694 -0.1847 26 0.5052 0.1184 0.0188 0.5009 0.0489 27 0.7187 0.1568 -0.0221 0.1291 0.0411 28 0.7480 0.1989 0.0385 0.1734 0.0727 29 -0.1027 -0.2361 0.2721 -0.5138 -0.0416 30 0.1286 0.1632 0.7971 0.0470 0.0470 31 -0.0890 -0.4452 -0.0445 -0.2503 -0.0540 ' A "#2 .-_nl‘l...y ‘Hi"