THESlS “‘4‘..— -—--uv-A tum’ ‘ , LIBRAR y a“; Michigan Sam I University This is to certify that the thesis entitled The Construction and Use of Papercases to Observe the Diagnostic Problem Solving Behavior of Reading Clinicians presented by C. Jay Colello Stratoudakis has been accepted towards fulfillment of the requirements for Ph. D . degree in Education ; a Major professor Date F 21 l 80 0-7639 V *1. 31 1136i; ‘ ; .‘ ~24.wi l I W OVERDUE FINES: ~25¢ per day per item RETURNING LIBRARY MATERIALS Place in book return to rem charge from circulation rec THE CONSTRUCTION AND USE OF PAPERCASES TO OBSERVE THE DIAGNOSTIC PROBLEM SOLVING BEHAVIOR OF READING CLINICIANS BY C. Jay Colello Stratoudakis A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Elementary and Special Education 1979 ABSTRACT THE CONSTRUCTION AND USE OF PAPERCASES TO OBSERVE THE DIAGNOSTIC PROBLEM SOLVING BEHAVIOR OF READING CLINICIANS BY C. Jay Colello Stratoudakis Rationale Until quite recently, research studies on the process of diagnosing reading disabilities have been nonexistent (Spache, 1968). Since its founding in 1976, the Clinical Studies Project within the Institute for Research on Teaching at Michigan State University, has addressed the question: "How do experienced reading clinicians approach the diagnosis of children with reading problems?" This study was an extension of the research initiated by the Clinical Studies Project. It sought additional information regarding how exper- ienced reading clinicians think about reading problems and pursued the answer to another question: "What is the most efficient and effective instrument to use for descriptive observation and eventual training of reading specialists in the process of diagnosing reading dis- abilities? C. Jay Colello Stratoudakis Research to date on the diagnostic problem- solving behavior of reading specialists has been con- ducted exclusively utilizing simulated cases called SIMCASES which were developed from diagnostic records of actual children with reading problems. The SIMCASE observational instrument allowed for direct observation of the reading specialist's behavior but required indi- vidual administration. No observational studies have been conducted utilizing a simulated case of reading disability in a form which allowed for group adminis- tration and observation. Purpose The purpose of this study was to construct and to test the usefulness of a group-administered simulated case of reading disability termed a PAPERCASE as an alternative to the SIMCASE observational instrument. ‘Whether the PAPERCASE could be substituted for the SIMCASE in collecting data on diagnostic problem-solving performance depended upon the consistency of a clinician‘s performance on these two instruments. Methodology Twelve certified classroom teachers who earned a superior grade in a graduate-level course in reading diagnosis at Michigan State University were hired as C. Jay Colello Stratoudakis consultants to the Clinical Studies Project and served as subjects in this study. The twelve subjects were randomly assigned to one of three different cases of reading disability. Four subjects were assigned to Case I, Four to Case II, and four to Case III. Each subject participated in two observational sessions with a one week interval between observations. With twelve subjects observed twice, there were a total of twenty- four observational sessions divided equally among the three different cases of reading disability represented in the two observational instruments, PAPERCASE and SIMCASE. Analysis The diagnostic problem-solving performance of the subjects on the SIMCASES and PAPERCASES was analyzed in terms of four measures of "agreement statistics" developed by the Clinical Studies Project: Proportional Agreement, Commonality, Inter-Clinician Agreement, and Intra-Clinician Agreement (Vinsonhaler, 1979). The agreement data was processed through the product analysis division of the Observational Studies Data Analysis System (Clinical Project Research Team, 1978). C. Jay Colello Stratoudakis Results The analysis of the "statistical agreement" data, indicated no meaningful difference in the diagnostic problem-solving performance of clinicians on SIMCASES or PAPERCASES. The PAPERCASES proved to be a more efficient and equally as sensitive an instrument as SIMCASES for observational study of reading diagnosis. Implication The major implication of this study was that PAPERCASES have the potential to be used as a creative, portable, inexpensive simulation instrument for observ- ing, training, and evaluating reading specialists in the process of diagnosing reading disabilities. Furthermore, this study made explicit the need for a practical model of the diagnostic process. Recur- ring behavioral patterns noted among the subjects for this study suggested that the diagnostic strategy employed by these subjects proceeded in a haphazard or random manner. Accordingly, individuals performing as reading diagnosticians need to be provided with a general frame- work or set of principles from which to proceed in order to increase their consistency and accuracy of diagnosis. @ Copyright by C. JAY COLELLO STRATOUDAKIS 1980 In memory of my father Michael Colello who was so proud of his daughter the school teacher ii ACKNOWLEDGMENTS For their indispensible help, I am indebted to my doctoral committee whose individual and collective efforts have made the completion of this dissertation possible. My appreciation is extended to Dr. Byron H. Van Roekel who guided my professional development through a course of study which strongly supported my own growth and development in the fields of reading and educational psychology. I am especially grateful to Dr. George B. Sherman who taught me how to diagnose reading disabilities, how to teach future teachers about the reading process, and who was my mentor and my friend. I am also grateful to Dr. John F. Vinsonhaler who gave me the opportunity to work with the Clinical Studies Project through which I learned more about the research process than I ever could have learned writing a dissertation on my own. A special word of thanks to Dr. Glen 0. Cooper who reminded me of the real world of public school edu- cation when I was emersed in coursework and challenged me to integrate theory with practice. iii In addition to my committee members, I wish to extend my appreciation to the Clinical Studies Project staff for supportive assistance, especially Ruth Polin, my computer consultant and Annette Weinshank, my research cohort. I remain forever grateful to a group of friends whose inspiration and concern I will always treasure: Lynda Sorenson, James Mullin, Lupi and Richard Hamersma, Rose and Sam Hayden, Lynn and Kaela Reynolds, John and Sharon Aycock and Donna Edison. The friendship of Elaine Stephens, Luthene Chappell, and Joel VanRoekel, my fellow doctural stu- dents, as well as many professors, in particular Dr. Gerald Duffy, and secretaries Jacqueline Babcock and Virginia Wiseman, helped make my stay at Michigan State University enjoyable. Also, I thank my parents, Michael and Phyllis Colello, who saw the wisdom of education for their daughter and sacrificed greatly that I might have it. Most of all, I thank my husband, Jim, for so much love and understanding throughout the seemingly endless process of finishing this degree. iv TABLE OF CONTENTS LIST OF TABLES . LIST OF FIGURES LIST OF APPENDICES CHAPTER I. INTRODUCTION . Purpose of the Study . Rationale for the Study The Research Objective . . Theoretical Basis of the Study Assumptions of the Inquiry Theory II. REVIEW OF THE LITERATURE . Introduction The Nature of Diagnosis . The Purpose of Diagnosis in Reading Levels of Diagnosis in Reading . Diagnosis as a Prerequisite to Remedia- tion . . Measures of Teacher Knowledge of. Reading . . Measures of the Diagnostic Ability of Teachers . Observations of the Diagnostic Skill of Reading Clinicians . . III. DESIGN OF THE STUDY Introduction . Development of the SIMCASES Development of PAPERCASES . Subjects . . Sample Selection . Page vii xi \OGJNGLD l-‘ Chapter Presentation of the SIMCASES AND PAPERCASES Procedure for SIMCASE Data Collection : Procedures for PAPERCASE Data Collection Observational Studies Data Analysis The Independent Variables . The Dependent Variables Summary . . . . IV. ANALYSIS OF THE DATA Proportional Agreement . Commonality Score . Inter- Clinician Correlational Agree- ment Intra- Correlational Agreement V. CONCLUSIONS AND IMPLICATIONS Conclusions Implications APPENDICES REFERENCES vi Page 33 36 38 4O 41 41 50 52 54 77 82 97 104 105 113 118 153 LIST OF TABLES Table Page 1. Presentation of Cases and Instruments . . 34 2. The Proportional Agreement Statistic: Case I Most Frequently Occurring Diag- nostic Categories in 8 Sessions Among 4 Clinicians . . . . . . . . . 55 3. The Proportional Agreement Statistic: Case II Mbst Frequently Occurring Diag— nostic Categories in 8 Sessions Among 4 Clinicians . . . . . . . . . 58 4. The Proportional Agreement Statistic: Case III Most Frequently Occurring Diag- nostic Categories in 8 Sessions Among 4 Clinicians . . . . . . . . . 61 5. The Proportional Agreement Statistic: Diagnostic Categories of Highest Fre- quency Across Cases: I, II, and III . 64 6. The Proportional Agreement Statistic: Average Proportional Agreement on Diag- nostic Categories Across A11 Sessions per Case . . . . . . . . . . . 66 7. The Proportional Agreement Statistic: Case I Most Frequently Occurring Cue Categories in 8 Sessions among 4 Clinicians . . . . . . . . . . 67 8. The Proportional Agreement Statistic: Case II Most Frequently Occurring Cue Categories in 8 Sessions among 4 Clinicians . . . . . . . . . . 7O 9. The Proportional Agreement Statistic: Case III Most Frequently Occurring Cue Categories in 8 Sessions Among 4 Clinicians . . . . . . . . . . 72 vii Table 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. The Proportional Agreement Statistic: Average Proportional Agreement on Cue Categories Across All Sessions Per Case . The Commonality Statistic: Cases I, II and III Individual vs. Group Agreement on Diagnostic Categories in 8 Sessions among 4 Clinicians The Commonality Statistic: Case I, II, and III Individual vs. Group Agreement on Cue Categories in 8 Sessions among 4 Clinicians Inter- and Intra-Agreement Statistic: Case I Correlation Matrix on Diagnostic Categories Inter- and Intra-Agreement Statistic: Case II Correlation Matrix on Diagnostic Categories . . . . . Inter- and Intra-Agreement Statistic: Case III Correlation Matrix on Diagnostic Categories Inter- and Intra-Agreement Statistics: Case I Correlation Matrix on Cue Cate- gories Inter- and Intra-Agreement Statistics: Case II Correlation Matrix on Cue Cate- gories Inter- and Intra-Agreement Statistics: Case III Correlation Matrix on Cue Cate- gories The Inter-Agreement Statistic: Inter-mean Phi Coefficients on Diagnostic Categories Cases I, II, and III . . . . The Inter-Agreement Statistic: Inter-Mean Coefficients on Cue Categories Cases I, II, and III . . . . . . . viii Page 75 78 80 83 84 86 87 88 9O 91 94 Table Page 21. The Intra-Agreement Statistic: Intra-mean Phi Coefficients on Diagnostic Categories Cases I, II, and III . . . . . . . 98 22. The Intra-Agreement Statistic: Intra-mean Coefficients on Cue Categories Cases I, II, and III . . . . . . . . . . 101 ix Figure LIST OF FIGURES The Clinical Encounter Clinical Problem Solving Form of the Two by Two Contingency Table on Inter-Clinician Correlation . Form of the Two by Two Contingency Table on Intra-Clinician Correlation . Page 11 12 45 48 Appendix A. B C D. E LIST OF APPENDICES Instructions for SIMCASE Sessions Cue Lists for SIMCASE Sessions . Diagnostic Checklist for All Sessions Instructions for PAPERCASE Sessions Cue Lists for PAPERCASE Sessions xi Page 119 132 139 144 149 CHAPTER I INTRODUCTION The International Reading Association (1979) recently specified the minimum standards for the train- ing of four types of specialists in reading: the special teacher of reading, the reading clinician, the reading consultant, and the reading supervisor. In all four role classifications, skill in diagnosing reading disabilities was stipulated as essential. Currently, according to Guthrie (1976), 252 colleges and universities in the United States offer graduate degrees in reading instruc- tion. Without exception, these institutions include, as a requirement for the degree, coursework in the diagnosis of reading difficulties and usually, in addition, a clinical field experience. In public education, classroom teachers refer millions of children every school year to the many thousands of reading specialists who staff federally funded reading programs such as Title I. Referrals are made on the assumption that the reading specialist will, through the application of a variety of measures, identify the components of the reading process which are causing the child's reading problem. On the basis of the reading specialist's diagnosis, decisions are made as to who is to be placed in the supplementary remedial reading class, who is to be tutored by a teacher's-aide, who is to practice reading with the parent volunteer, who is to receive more intensive instruction in a specified reading skill, and, in general, who is to receive extra attention and time on the task of reading. Because diagnosis provides the starting point for remediation, teacher-educators as well as practitioners in the field of reading con— sider expertise in diagnosis to be a basic skill requirement of reading specialists. While there is a consensus in theory and in practice that diagnosis is a core concept in the field of reading, what constitutes a diagnosis is open to divergent points of view and what the optimum conditions and procedures for collecting and interpreting data are is largely unknown. Until quite recently, research studies on the diagnostic process have been nonexistent (Spache, 1968). As Shulman and Elstein (1973) have observed: "Research typically slights the problem of how teachers think about their pupils and instructional problems; it concentrated instead on how teachers act or perform in the classroom” (p. 3). Since its founding in April of 1976, the Insti- tute for Research on Teaching (IRT), a research center funded by the National Institute of Education and located at Michigan State University, has been studying the decision-making patterns of experienced teachers. In particular, the Clinical Studies Project within the IRT has addressed the question: "How do experienced reading clinicians approach the diagnosis of children with reading problems?" The study described here--an extension of the research previously initiated by the Clinical Studies Project—-was designed to seek another way to obtain information of value in answering the previous question as well as to pursue another question, namely: "What is the more efficient way to initially collect information about the diagnostic problem solving behavior of reading clinicians and to eventually train reading specialists in the process of diagnosing reading disabilities?" Purpose of the Study Research to date, within the Clinical Studies Project, on how experienced reading clinicians approach the diagnosis of children with reading problems has been conducted exclusively through the utilization of the SIMCASE--an observational instrument for data collection. A SIMCASE is a device which provides an operating model or replicia of real world processes. It represents a child with a commonly occurring reading problem and attempts to replicate the clinician-client or reading specialist-student interaction which occurs during the process of diagnosing a reading problem. These simula- tions allow for observation of diagnostic problem-solving behavior outside of the field setting in which this process is typically performed. The SIMCASE is contained in a file box and consists of six different categories of information about a case of reading disability: (1) test scores, (2) test booklets, (3) test directions, (4) test description, (5) audio-recordings, and (6) examiner's comments. Information for clinical problem solving is retrieved from the SIMCASE by the administrator or observer upon the request of the subject or reading clinician. That the SIMCASE should prove to be a reasonably effective device for observing clinical problem solving behavior within the Clinical Studies Project was not unexpected because other research has repeatedly demon- strated the appropriateness of simulations for observa- tional studies (Elstein, Shulman, and Sprafka, 1978). However, the tedium inherent in the individual adminis- tration of the instrument is potentially a significant limitation in the conduct of research and the training of reading clinicians, especially when larger numbers of subjects or trainees are involved. The purpose of this study was to develop and to test the usefulness of an adapted format of the SIMCASE, the intent of which was to mitigate the temporal demands of the individually administered observational instrument and yet retain the sensitivity of the SIMCASE in the observation of diagnostic problem solving performances. Three factors which influence the efficacy of simulations as observational instruments were carefully examined: (1) cost in terms of materials and remuneration to sub- jects, (2) size in terms of pages, and (3) administrative time per subject or trainee. Careful consideration of these effecting factors revealed that cost and conven- ience were not amenable to major adaption without signif- icant losses in range, scope, and representation. The remaining option, then, was an alteration which would reduce the administration time requirement per subject or trainee. Accordingly, the SIMCASE was adapted to allow for group-administration, and the resulting instru- ment was named PAPERCASE. An observer can administer a PAPERCASE to a large number of subjects within a two hour period as opposed to administering a single SIMCASE to one subject in the same time period. Furthermore, the PAPERCASE is independent of the observer; it is limited to a subject retrieving information from the pages of a booklet. The PAPERCASE included four categories of information about a case of reading disability: (1) test scores, (2) test booklets, (3) test directions, and (4) test description with the pages of examiner's comments and the audio-recordings excluded from the PAPERCASE as a minor adaptation in the size of this observational instrument. All audio-recordings and examiner's comments were judged to be nonessential to diagnosing the reading problem and were omitted from the PAPERCASE. This omission represents from 14 to 35 pieces of information or cues depending on the particu- lar case of reading difficulty. Thus, the PAPERCASE contains roughly 20 percent fewer cues than the SIMCASE format. Rationale for the Study In comparison with the SIMCASE simulation, the PAPERCASE may prove to be a more efficient but equally sensitive instrument for use in observing diagnostic problem-solving performance. Furthermore, in addition to using PAPERCASES as an observational instrument directed toward understanding the process of diagnosis, the PAPERCASES have high potential as training exercises intended to teach the process of diagnosis. PAPERCASES may offer a creative, very portable, and inexpensive way of vitalizing graduate and in-service teacher education by fusing theory and practice into efficient learning of diagnostic skills. Hence, individuals training to become reading specialists may learn about a number of reading problems without having to resort to clinical experience to rein- force what they have learned. Moreover, even when clini- cal experience is available and the opportunity is there to actually diagnose a real child's reading problem, staffing problems often pose severe limitations on the ability of the clinical faculty to observe a student's performance and provide the student with the feedback that is necessary to maximize the development of skill in diagnosis (Van Roekel, personal communication). The PAPERCASE is an alternative to the direct observation of an individual diagnosing a reading problem. With the addition of a feedback component, the PAPERCASES could provide students with the opportunity to practice solving a range of reading problems which closely approximate, in breadth and complexity, the problems which they will encounter as reading specialists in the real world. The Research Objective In designing the study it was recognized that there are a number of researchable questions of potential interest with respect to the proposed new instrument. The decision as to which questions would be asked, and which would be deferred for subsequent investigation, was largely decided by this author's conception of the research strategy which would make the most immediate, direct contribution to the ongoing Clinical Studies Project. This strategy suggested the following research objective: Evaluate the usefulness of PAPERCASES as an observational instrument by comparing the diagnostic performance of clinicians on PAPERCASES with performance on SIMCASES. This research objective questions: can the PAPERCASE be substituted for the less efficient SIMCASE as an instrument which is equally effective in collecting data on diagnostic problem-solving performance? The answer to this question depended upon the consistency of the clinicians' performance on the two instruments. Theoretical Basis of the Study When one considers the process of reading diag- nosis, one is faced with describing the very complex and cognitive behavior of problem solving. No intelligible description of phenomena as abstract as problem solving is possible without a sound theoretically based frame- work from which to study the nature of this behavior. The Inquiry Theory of Clinical Problem Solving formed the theoretical basis for this study. Similarities drawn by Elstein, Shulman, Vinsonhaler, and others (1977) between Clinical Problem Solving which was initially developed in a medical mode and the behavior of reading clinicians which is in an educational mode, lends strong credibility to the application of the Inquiry Theory to Reading. Feature by feature, the authors illustrate the correspondence in process between the medical-clinical task of diagnosis and the reading-clinical task of diag- nosis. In summary terms, both clinicians are concened with alleviating problems presented to them by individuals who may be identified as the patient, the client, or the student in difficulty. More importantly, both clinicians as practitioners are considered to informally proceed through a cycle of data collection--hypothesis genera- tion--cue interpretation--hypothesis verification in the process of problem solving. Assumptions of the Inquiry Theory Exploring the nature of Clinical Problem Solving in Reading through observational studies, Vinsonhaler, Wagner, and Elstein (1977) have made explicit the behav- ioral domain addressed by the Inquiry Theory. The behav- ioral domain of this theory is known as the Clinical Encounter which encompasses the events occurring when a Clinician (e.g., physician, reading specialist) interacts with a Case (e.g., a patient, a student) in order to reach 10 a diagnostic and/or treatment decision about the present- .ing problem (e.g., child has a temperature, child cannot read his textbooks). Thus, the Clinical Encounter includes: (1) the Clinical Case, (2) the Clinician, and (3) the Clinical Interaction. These three components in the Clinical Encounter are illustrated in Figure l and the direction of the interaction is indicated by arrows. The Clinical Encounter is the basis of the first assumption of the Inquiry Theory which states: "The behavioral domain addressed by the theory involves a clinician, a case or patient, and an interaction which yields a decision on the Diagnosis (the state of the case) and the Therapy (how this state can be improved)" (Vinsonhaler, 1979). The second assumption relates to the clinical case, and states: "Important problem solv- ing behaviors of children can be elicited by simulated cases" (Vinsonhaler, 1979). The use of simulated cases is based on the educational principal that problem solv- ing skills can best be "learned by doing" (Dewey, 1963; Bruner, 1966; Gagne, 1971). The third assumption, presented in Figure 2, focuses on the Clinician and describes how Clinical Problem Solving occurs: Clinical Problem Solving is determined probabilistically by the interaction of (1) Clinical Memory, (2) Clinical Strategy, and (3) the Case (Vinsonhaler, 1979). ll moo-.0... &. g§ \‘c .“5 ‘ o h} I THE CASE A C M a! 'Oc-O-II O. A“ O “I a! (no In" C) 0 In 0! (an "In. {\I O in a! lump-Qua to at! raccoctal Tract-«nu l L. l :3, CASE 0,?— :A‘x CLINICAL INTERACTION Figure 1.--The Clinical Encounter. 12 . m— ”mi CLINICAL memosv 5mm EGY I In OI host".- 0‘) Iowa." 0! III «Unla- Pct-custar- I to! at he v-xw- (f) x- uv----. I» mun: ’__J I Iczulm DICJU I "Irv-«In. Cyan-tun 0 an “Hem“ o In of tun-nun) I tam». Inaugu- I :l Jon-urn I S“ of Incline I In Incl-«lo. o 00,»)! I 1,1 a: l ICLINICIAN‘ CLINICAL . INTERACTION ‘_ Figure 2.--Clinical Problem Solving. 13 Clinical Memory consists of sets: (1) prob- .lems, (2) cues, and (3) treatments and the relation- ships among them. An example of Clinical Memory in the context of reading diagnosis is: 1. Problem - Child does not attempt to answer inference questions. 2. Cue '- Iowa Tests of Basic Skills (ITBS) Reading Subtest, all inference questions unanswered. 3. Relating problem and cue - Item analysis of errors on the ITBS-—Reading Subtest suggest the likelihood that a comprehen- sion problem is high. Clinical Strategy consists of the mental tasks performed by the Clinician which translate memory into action. These tasks mainly involve information-gathering and information-processing as the clinician makes decisions about diagnosis and remediation (Vinsonhaler, Wagner, and Elstein, 1977). The Research on the Inquiry Theory of Clinical Problem Solving initially conducted by the Medical Inquiry Project form 1969 through 1973 at Michigan State University in the context of medical education has been continued over the past three years by the Clinical Studies Project in the context of teacher education. The Clinical Studies Project is applying the original concepts developed in medical diagnosis and treatment to the field of reading diagnosis and remediation. CHAPTER II REVIEW OF THE LITERATURE Introduction "Concepts are properties of organismic exper- ience-~more particularly, they are the abstracted and often cognitively structured classes of 'mental' exper- ience learned by organisms in the course of their life histories. . . . Within a given community there will be a high degree of commonality in the concepts recognized and attained, in the sense that there will be relatively high agreement among people as to the attributes that are criterial for a given concept" (J. B. Carroll, 1964, pp. 180 and 185). The purpose in this chapter is to mark off the boundaries of the concept of diagnosis in the field of reading-education by attending to selected attributes of diagnosis described in the literature. The Nature of Diagnosis What generic observations have been made about diagnosis in the literature? Description of the nature of diagnosis is very limited. To begin with, the Random House Dictionary of the English Language generally 14 15 defines diagnosis as an answer or solution to a proble- matic situation. According to Della-Piana (1968), "to diagnose is to determine the nature of a process by examining in some detail the differences between the functioning of various parts of the process" (p. 3). Dechant (1968) stated that the heart of diagnosis is an intelligent interpretation of facts, it is an infer- ence from performance. The Institute for Research on Teaching (IRT) has been studying teaching as diagnosis. The IRT has broadly defined diagnosis as a decision-making or problem-solving process in which a clinician interprets information about individual students or a class as a whole. The clinician combines information about stu- dents and classes, as well as information from the edu- cational research literature, with his/her own expecta- tions, attitudes, beliefs, and purposes. Based upon all this processing of information, the clinician then responds, renders decisions, and regroups to begin again (Cruickshank and Kennedy, 1977). The description of diagnosis most closely related to the IRT's point of view comes from H. L. J. Carter (1970) who specified the following four acts as integral to the process of diagnosis: 16 l. Identif the problem and possible causal factors. 2. Assume and reject hunch after hunch until one can be accepted tentatively. 3. Discover possible determinants and explain consequential relationships. 4. Predict that with treatment the disability wIII 5e overcome (p. 20). Similarly, Spache (1976) collaborating with R. W. Prouty described diagnosis as "a continuous process of proposing hypotheses, testing them by teaching strate- gies and referring or discarding them. . . . As we test and as we begin instruction that seems relevant in terms of our first impressions, we must constantly observe the pupil's behavior response to the approach we are using and its apparent impact upon his develop- ment" (p. 9). While theoretically, diagnosis has been gener- ically described as a problem-solving process it is practically described in the literature in terms of its purposes. The Purpose of Diagnosis in Reading In 1935 Bruecker discussed diagnosis as tech- niques by which one discovers and evaluates the strengths and weaknesses of an individual. Much later, Smith and Dechant (1962) elaborated on this purpose: 17 Diagnostic procedure begins with a study of the child's instructional needs based on the expec- tancies of his chronological age, mental age, and grade placement. We seek to discover why he reads as he does, what he can read, and what he does read successfully. We need to know if he is hav- ing problems in reading and, if so, what they are and what are their causes. We wish to know his general abilities and his reading potentiality and we must identify causal factors that have retarded his reading development. In short, we must know his strengths and weaknesses (p. 408). Durkin (1970) has written about the positive and negative overtones of reading diagnosis in contrast with medical diagnosis. She noted that medical diagnosis has a negative overtone since medical personnel are usually trying to learn what is wrong with a patient but reading diagnosis has both a positive and a negative dimension. It is just as concerned about what a child knows and can do as it is about what he does not know and cannot do (p. 402). Focusing on the negative dimension, H. L. J. Carter (1970) stated diagnosis is an explanation of an individual's maladjustment in reading. In the study of a disabled reader, the teacher and clinician are con- cerned with cause which precedes an event called the effect. The purpose is to determine why the individual is disabled, what went wrong (p. 17). Specifying the purpose of diagnosis as the iden- tification of weaknesses and strengths is consistent with what Strang (1969) identified as the first level 18 of diagnosis. On the first or surface level, an effort is made to describe reading performance--strengths and weaknesses in vocabulary, work recognition, sentence and paragraph comprehension, and related abilities. Going beyond the surface, other authorities in the field have set forth additional purposes for reading diagnosis. Levels of Diagnosis in Reading Diagnosis of reading disabilities may be made on different levels of comprehensiveness, psychological depth, and competence (Strang, 1964, p. 4). Monroe (1937) explained that the diagnosis of reading disabil- ities should contain two types of analysis: (1) des- criptive and (2) causative. In the descriptive analysis the examiner details the nature of the child's reading disability based upon subjective observation and objec- tive test data. In the causative analysis the examiner investigates five areas: (1) constitutional, (2) intel- lectual, (3) emotional, (4) educational, and (5) environ- mental, which may be contributing to the problem (p. 14). Extending Monroe's (1937) two levels of analysis to four, H. L. J. Carter (1970) stated diagnosis proceeds from (1) identification of difficulty, to (2) classifi- cation of disability, to (3) determination of specific needs, and at the highest level (4) detection of causal factors underlying the individual's disability (p. 18). 19 Bond and Tinker (1973) discussed diagnosis in terms of successive screenings in which the more complex and subtle cases of reading disability are retained for further analysis at more in-depth levels. Referring to Brueckner and Bond (1955), Bond and Tinker (1973) believed some cases of reading disability must be carried through three levels: (1) general diagnosis, (2) analytical diag- nosis, and (3) case-study diagnosis. General diagnosis is made by studying test results in order to locate gen— eral areas of weakness. Analytical diagnosis identifies specific strengths and weaknesses and indicates skills and abilities wherein the child's weakness lies. The case-study diagnosis analyzes information on mental capacity, vision, hearing, physical characteristics, adjustment to reading, and environmental factors bearing upon the child. Again, the level of diagnosis reached depends upon the characteristics of the case (pp. 168- 171). Rutherford (1972), addressing the classroom teacher, asserted that teachers accept a range of four explanations of a reading disability which proceed from general to specific. At level one difficulties are clas- sified in broad categories of causes: level two applies special terms to the problem; level three describes cer- tain types of general, overt student behavior; and level 20 four is prescriptive in terms of what the child needs to learn (pp. 51-53). Strang (1968) further analyzed the breadth of possible difficulty in reading and has ordered seven levels of reading diagnosis. On the first level infor- mation is obtained on student performance. On the second level the student's reading behavior is observed. On the third level an attempt is made to analyze the student's reading process as opposed to the student's performance. On the fourth level mental capability is investigated. The fifth level involves clinical analysis of personality traits and values. The sixth level involves an examina- tion of possible pathological conditions, such as brain damage. The approach at the seventh level is to ask the reader to describe his reading process, termed intro- spective reports (pp. 4-6). In sum, the numerous levels of diagnosis suggest what may constitute this problem-solving behavior of diagnosis and the possible level of specificity that may be necessary in a diagnosis depending on the complexity of the case. Of course, the reading specialist's prob- lem is to determine just what is, and is not, necessary. But, necessary for what? The widely acknowledged answer is--for remediation. According to such authorities as Bond and Tinker (1976), Ekwall (1976), and Spache (1976), 21 reading specialists should gather only enough initial diagnostic information to begin a program of remediation. Diagnosis as a Prerequisite tolRemediation Several well-known authors of textbooks on the subject of diagnosis and remediation repeatedly position diagnosis as a prerequisite to remediation. To quote Monroe (1973), "To be effective, remedial instruction in reading must be preceded by careful diagnosis" (p. 359). Writing as far back as 1922, Gray stated: "After diag- nosis has shown the kind of instruction that is needed, the remedial program should be carefully planned" (p. 374). In the words of Dechant (1968), "Diagnosis is complete only when remediation occurs" (p. 6). As Della-Piana (1968) puts it, "The major purpose of diag- nosis is to gather information that may be helpful in making treatment decisions" (p. 3). In Spache's (1976) terms, "Diagnosis is pragmatic and directly related to remedial practice" (p. 9). Other authors publishing in the field have reiterated the same position. E. C. Kennedy (1971) stated: "The real purpose of educational diagnosis is to secure specific information about a pupil which will enable teachers to plan for direct and appropriate instruction" (p. 97). Also, R. M. Wilson (1977) 22 believed that "Regardless of the educator's professional position, diagnosis is essential in formulating a remed- ial program which will be both effective and efficient" (p. 15). A final reiteration from Karlsen (1976), "Diagnosis is not simply analysis of basic causes of reading problems. It is oriented toward the future and is most effective when it helps the teacher arrange meaningful and efficient learning experiences that will help each student become a skillful reader" (PP. 147- 148). Measures of Teacher Knowledgg of Reading The agreement among authorities in the field of reading-education that diagnosis precedes remediation is based on the assumption that reading clinicians and classroom teachers know how to diagnose and remediate reading difficulties. However, little research evidence exists to confirm this assumption (Gil, Vinsonhaler, and Sherman, 1979). Research studies have been limited to measuring clinician knowledge of reading. A brief review of these studies focusing on the instruments developed to measure teacher knowledge of reading and the performance of teachers on the instruments follows. Schubert (1959) was interested in finding out elementary and secondary teacher's knowledge of structural 23 and phonetic analysis. He administered an informal ten question quiz to 80 elementary teachers and 42 secondary teachers and reported that a substantial number of them did not possess knowledge of certain principles of word analysis. A sixty-item multiple-choice test on phonetic- generalizations was developed by Aaron (1960) and admin- istered to 104 persons with one or more years of teaching experience and 189 persons with no teaching experience. Results indicated that very few subjects were well- grounded in phonics principles. Spache and Baggett (1965) used a modified version of Aaron's test with graduate students and inservice teachers pursuing graduate work and found that they were generally weak in the areas of phonics and syllabication. Ramsey (1962) and Browman (1962) both developed tests to determine the extent to which teachers possessed knowledge of basic skills in reading. Durkin's (1964) test called the Phonics Test for Teachers was designed for use in reading methods courses to help students iden- tify what they know and what they do not know about phonics. An instrument which covered rather broad areas of reading was developed by Wade (1960) to measure such skills as "diagnosing and correcting phonic and syllabi- cation errors." His instrument included an audio-tape 24 and paper-and-pencil questions. He administered his test to students, teachers, and reading specialists and found, as expected, the students achieved the lowest and the reading specialists achieved the highest. The Inventory of Teacher Knowledge of Reading was recently developed by Artley and Hardin (1975). This test contains 95 multiple-choice items and covers the following areas: the reading act, preparation for read- ing, word identification, comprehension and critical reading, reading in the content areas, reading interests and tastes, and corrective procedures. However, factor analysis indicated that the seven areas from which the items were drawn were not identifiable as discrete areas. Kingston and his associates (1975) also analyzed the results of the Inventory of Teacher Knowledge of Reading. Kingston's factor analysis also failed to reveal seven components of this Inventory. The measures of teacher knowledge of reading by Schubert (1959), Aaron (1960), Spache and Baggett (1965), Ramsey (1962), Browman (1962), Durkin (1964), Wade (1960), and Artley and Hardin (1975) are all possible instruments for estimating some of the categories of information a clinician brings to the process of diagnostic-problem solving. In this respect, these measures relate to "clinical memory,‘ a component of the Inquiry Theory of Clinical Problem Solving which forms the theoretical 25 basis of this study. As initially presented in Figure 2 of Chapter I, the third assumption of this theory is that clinical problem-solving is determined probabilistically by the interaction of (1) clinical memory, (2) clinical strategy, and (3) the case (Vinsonhaler, 1979). Clinical memory includes the background of experience and informa- tion a clinician brings to a case with which to make decisions about the case and these measures of teacher knowledge may partially describe a clinician's clinical memory. Measures of the Diagnostic AEility of Teachers Burnett (1961) developed a test to measure the diagnostic ability of teachers. His test consisted of problems to measure five levels of operation in diagnosis. The first level problems required the examiner to pick critical information from a pool of data; the second level problems required selecting a means of securing additional data; the third level required the interpre- tation of data; the fourth level required recommendations for improving instruction; and at the fifth level, the examiner was required to re-evaluate his fourth level recommendations. Burnett found that reading specialists significantly outscored experienced teachers and exper- ienced teachers outscored the undergraduate student. 26 The mean differences were significant beyond the .01 and .05 levels. Emans (1965) assessed the diagnostic ability of teachers enrolled in a clinical practicum in reading remediation. For an hour a day over five weeks, 20 teachers provided individualized reading instruction to two children experiencing some difficulty in reading. At the end of the five week period the teachers were asked to rank 15 reading skills in the order in which their children needed help on them. The teacher's rank- ings were compared with the rankings of the skills as indicated on the Reading Diagnostic Test by Arthur Gates. The correlations between the teachers' judgments and the scores on the test were very low. Emans suggested that individualized reading programs were doomed from the start if teachers were unable to determine the children's reading needs. Observations of the Diagnostic ’ SkiII'ofiReading Clinicians The Clinical Studies Project has developed simu- lation instruments rather than tests to observe and objectively describe performance of clinicians resulting from the interaction of clinical memory, clinical strat- egy, and a case. The research approach of this project has been to study the "wisdom of the practitioner" with' individual teachers recognized as having expertise in 27 reading diagnosis not only participating as clinical research subjects but also as sources of insight regard- ing their own behavior and functioning when presented with simulated cases of reading difficulty. To date, the Clinical Studies Project has conducted a series of six observational studies as part of a systematic program of research. All of the observational studies have shared a common theoretic and methodological base which included the utilization of SIMCASES as the instrument in data collection. In very global terms, the findings of the first observational study, 0377.1, were that performance on the SIMCASES was inconsistent. That is, a problem diag- nosed one way by one reading clinician was diagnosed differently by another. In addition to not agreeing with each other when diagnosing and remediating the same case, the reading clinicians disagreed with themselves when diagnosing different versions of the same problem (orig- inal SIMCASE and replicate SIMCASE). However, the com- bined diagnosis of several clinicians or group diagnosis appeared to be more consistent than the diagnosis by an individual reading specialist. Considering the additional observational studies, all indications are that a second observational study, 0878.3, being conducted by L. Hoffmeyer and termed a "conceptual replication," will verify the findings of 28 the first observational study, 0877.1. A third study, 0878.2, focusing on the diagnostic performance of class- room teachers as opposed to reading specialists, was conducted by D. Gill. His investigation centered on the identification of the classroom teacher as a diagnosti- cian who is in the position to observe children's read- ing behavior on a continuous basis and diagnose and treat reading problems at their inception, long before the problem becomes severe and is brought to the attention of the reading specialist. Gil's outstanding general results were very similar to the findings of the first observational study, 0877.1. Classroom teachers were inconsistent with each other when diagnosing and remed- iating the same case. "The most frequently mentioned diagnostic judgments differed from case to case. On the average, teachers in this study showed very limited agreement with each other on diagnostic judgments and cues collected for a given case" (Gil, 1979, pp. 104 and 105). A fourth study, 0878.1, now in progress by J. Van Roekel is investigating the problem solving behavior of both teachers trained in the field of Reading and teachers trained in the field of Learning Disabilities. According to Van Roekel, his study was designed with the following purposes in mind: 29 First, to identify the types of measures typically utilized by clinicians representing the fields of reading and learning disabilities when they inter- act with identical cases of education difficulty. Second, to determine if there are systematic dif- ferences between these two groups of clinicians with regard to the type of data collected, the amount of data collected, the degree ot which that information is used in a diagnostic summary, and the diagnostic conclusions drawn by clinicians representing the two fields. Third, to compare problems solving processes and diagnostic products of reading clinicians and learning disabilities resource teachers employed in public schools with a group of senior clinicians. Finally, to explore the application of the Inquiry Theory of Clinical Problem Solving to the dia nosis of reading and learning problems (Gil, HoIfmeyer, Van Roekel, Vinsonhaler, and Weinshank, 1979, p. 32). A fifth study, 0879.2, raised the question: "What is the relationship between diagnosis and remedi- ation?" A. Weinshank is systematically investigating a corollary to the Inquiry Theory of Clinical Problem Solving which states: Problems and treatments are asso- ciated in probabilistic fashion such that given a problem some treatments would be more likely to be selected than others." This corollary, termed problem-treatment asso- ciation (PTA), was previously tested by Weinshank (1978) in a pilot study. The pilot study analyzed the therapeu- tic decisions contained in the remediation plans written by the reading diagnosticians who were the subjects in the first observational study, 0877.1. Her major find- ings were than diagnostic statements outnumbered treatment statements by a margin of from 2:1 to 7:1; the relation- ship between problems stated in the diagnosis and 30 treatments proposed in the remediation was moderate; clinicians who agreed with themselves more on the diag- nosis of a case also agreed with themselves more on the remediation for that case; and clinicians agreed on the use of a core subset of treatments across almost all of the cases. This present study is a sixth observational study, 0879.1, the methodology and results of which are discussed in the following chapters. CHAPTER III DESIGN OF THE STUDY Introduction The construction of a PAPERCASE included a developmental phase and an experimental phase. The development of the PAPERCASE will be outlined first followed by a description of the agreement statistics used in the experiment to compare the diagnostic per- formance of clinicians on SIMCASES with performance on PAPERCASES. Development of the SIMCASES The SIMCASES are a product of previous research in the Clinical Studies Project of the IRT. Each SIMCASE is presented in a metal file box and represents actual cases of male students in second through tenth grade who have a reading disability. Each SIMCASE includes six categories of information on the case: (1) test scores, (2) test booklets, (3) test direction, (4) test descrip- tion, (5) audio-recordings, and (6) examiner's comments. Three original SIMCASES representing actual cases of reading disability were used as observational instruments 31 32 and three replicate forms of these self same original SIMCASES were used in this study. Development of PAPERCASES Three PAPERCASES were constructed by this researcher from thereplicate forms of the three origi- nal SIMCASES. Each PAPERCASE is contained in a booklet and includes four categories of information on each case: (1) test scores, (2) test booklets, (3) test directions, and (4) test descriptions. Subjects The subjects of this study were 12 graduate students who studied Diagnosis of Reading Difficulty, Education 830E, at Michigan State University under Dr. George Sherman, within three past academic terms (Winter, Spring, and Fall, 1978) and earned a final grade of 4.0 thereby demonstrating to Dr. Sherman initial mastery of the requisite skills for diagnosing reading problems. A further requirement of the sub- jects for participation in the study was that they be certified classroom teachers. This sample was chosen because: 1. This group of graduate students had similar amounts of knowledge about diagnosis and similar practi— cal experience in diagnosing reading problems and writing 33 diagnostic case studies through the course requirements of Education 830E. 2. This group of graduate students was gener- ally familiar with the diagnostic testing materials included in the observational instruments for this study. Many of the same testing materials were used in Education 830E to collect information on a child with a reading problem. 3. This group of graduate students was judged to be more accessible than a group of expert reading diagnosticians. Previous observational studies nearly exhausted the local population of experts in the field. Sample Selection The graduate students were contacted by telephone and asked to participate at a rate of $9.00 per hour as subjects in a study of diagnostic problem solving. Sub- jects were contacted in random order and randomly assigned to the three cases of reading disability and the two observational instruments. Presentation of the SIMCASES and PAPERCASES Table 1 illustrates the manner in which SIMCASES and PAPERCASES were presented to the subjects. Each of the twelve subjects were randomly assigned to a case and a replicate form of the same case. Three original 34 Assamese - mm< owumwumum unoEmohw< Hmoowunoeoum one a sense 67 Table 7 The Proportional Agreement Statistic: Case I Most Frequently Occurring Cue Categories in 8 Sessions among Clinicians Case: I Cue ngggory Sesfions Seggiogg_ 1 Background: Parent Form - Record Booklet .875 7 2 Background: Teacher Form - Record Booklet .625 5 3 Background: School Record - Record Booklet .625 5 4 Background: School Information - Record Booklet .500 4 5 Durrell Oral Reading: Examiner's Comments .500 4 6 Durrell Oral Reading: Test Booklet .875 7 7 Durrell Silent Reading: Test Booklet .500 4 8 Durrell Word Recognition & Word Analysis: Test .500 4 Booklet 9 Durrell Hearing Sounds in Words-Primary: Test .500 4 Booklet 10 Durrell Phonetic Spelling of Werds: Test Booklet .500 4 ll Ekwall Phonics Survey Modified: Examiner's Cements 500 4 12 Ekwall Phonics Survey Modified: Test Booklet .750 6 13 Gates McRillop Auditory Blending: Test Booklet .500 4 l4 Attitude: Sentence Completion - Test Booklet .500 4 15 Slosson Oral Reading Test: Test Scores .625 5 16 Slosson Oral Reading Test: Test Booklet .625 5 17 WISC Verbal Scale: Test Scores .750 6 18 WISC Verbal Scale: Examiner's Comments .500 4 l9 WISC Full Scale: Test Scores .625 5 20 Audiometric Record: Examiner's Comments .625 5 21 Audiometric Record: Test Booklet .500 4 22 Vision Test: Examiner's Comments .625 23 Vision Test: Test Booklet .500 4 68 chose to examine information in the form of a record or test booklets which described student behavior as opposed to referring to other forms of information such as test scores and examiner's comments. To the very limited degree of 15 cues selected 50 percent of the time, clini- cians relied more upon a qualitative description of read- ing behavior rather than a quantitative score or the post hoc comments provided by the examiner who administered the texts. The first four (1, 2, 3, 4) of the 23 most fre- quent cues selected on Case I referred to background information, four other cues (5, 6, 15, 16) referred to oral reading information,and at least an additional six cues (8, 9, 10, ll, 12, 13) referred to information on word analysis. Cue selection seemed to cluster in three areas of information: background, oral reading, and word analysis. In all, the 23 cues selected in 50 percent of the sessions suggested a low concentration of Propor- tional Cue Agreement for Case I which exceeded the Proportional Diagnostic Agreement previously reported for Case I. Only five diagnostic categories were agreed upon in 50 percent of the sessions while 23 cue cate- gories were agreed upon in 50 percent of the sessions. The cues collected were more frequently the same than 69 the diagnostic judgments recorded about Case I. How- ever, although agreement was relatively higher for cues than for diagnostic categories, the overall degree of Proportional Agreement was quite low, a 50-50 chance occurrence of a few of the same categories. The first column in Table 8 lists the cues selected by clinicians in at least 50 percent of the sessions on Case II. The second column lists the fre- quency of cue selection in terms of percentages and the third column translates the percentages into number of sessions. Nineteen cues were most frequently collected from a cue inventory averaging 73 cues (see Appendix B for the SIMCASE Cue List and Appendix E for the PAPERCASE Cue List on Case II). As previously observed in Case I, the type of information most frequently referred to during the ses- sions conducted on Case II was the record or test booklet. The group of clinicians diagnosing Case II most frequently chose to examine the record or test booklet which des- cribed student behavior as opposed to referring to other forms of information such as test scores and examiner's comments. It seems reasonable to suggest again, to the limited degree of 11 cues selected 50 percent of the time, this group also appeared to be relying somewhat more upon a qualitative description of reading behavior rather than 70 Tab 1e 8 The Proportional Agreement Statistic: Case II Most Frequently Occurri Cue Categories in 8 Sessions among Clinicians Z’ No. Case: II Cue Categories Sessions Sessions 1 Background: Biographical Data - Record Booklet .625 5 2 Background: Physical/Health - Record Booklet .875 7 3 Background: Home/Family - Record Booklet .750 6 4 Background: Classroom Information - Record .870 7 Booklet 5 Dolch Word List: Test Booklet .750 6 6 Durrell Oral Reading: Test Booklet .750 6 7 Durrell Silent Reading: Test Scores .500 4 8 Durrell Silent Reading: Examiner's Comments .500 4 9 Durrell Silent Reading: Test Booklet .500 4 10 Durrell Word Recognition & Werd Analysis: Test .625 5 Booklet 11 Durrell Visual Memory of Words-Primary: Test .625 5 Booklet 12 Gates-MacGinitie Comprehension: TestScores .500 4 13 Gates McKillop Recognizing & Blending Common .500 4 Word Parts: Test Scores 14 Gates McKillop Recognizing & Blending Common .750 6 Word Parts: Test Booklet 15 Informal Oral Reading: Examiner's Comments .500 4 l6 Slosson Oral Reading: Examiner's Comments .500 4 l7 Slosson Oral Reading: Test Booklet .750 6 18 WISC Full Scale: Test Scores .500 4 l9 WISC Verbal Scale: Test Scores .500 4 71 a quantitative score or the post hoc comments provided by the examiner who administered the tests. Also, resembling Case I,the cues selected on Case II seemed to cluster in two areas. The first four (1, 2, 3, 4) of the 19 most frequently collected cues referred to background information and four other cues (6, 15, l6, l7) referred to oral reading information. The 19 cues selected in 50 percent of the ses- sions suggested a low concentration of Proportional Cue Agreement for Case II which exceeded the Proportional Diagnostic Agreement previously reported for Case II. Only six diagnostic categories were agreed upon in 50 percent of the sessions while 19 cue categories were agreed upon in 50 percent of the sessions. The same information was collected more often than the same diagnostic statements were written about Case II. In general, there was somewhat more agreement within the group on what type of information to collect than on how to describe the reading disability presented in the case. Although Proportional Agreement was relatively higher for cues than for diagnostic categories, the amount of Proportional Agreement remains very low. The first column in Table 9 lists the cues selected by clinicians in at least 50 percent of the sessions on Case III. The second column lists the 72 e con. uoaxoom puma "amok anemone Huuo newscam ma a com. nouoom uses “anew woameom Heuo acaeon we a con. uoaxoom ouch "mono: encouooz goddamn: mouse HA uonoom uses "munch whom a ANS. season meaeaoam a meeuenmouuu aoeeaxuz mouse on nouoom amok "mouse one: s can. 606800 wcumcoam a wcwuaowooum moaawxoz amuse m e com. nououm uuoa : muohsuo< a mason owuaoeuoz noueo w n nmo. mouoom amok ":oaooenoumaoo moaoouoaq Haouuaa u 0 one. uonoon uses .weauaom auto antenna a a con. nouoom unoa unseeded Mono Haouuaa n uuoeaaoo a con. e.uonwawxu : uuomom aeoumoum demon "monouwxuem e e con. uuooaaou e.uoo«awxm : mucosa Hoonum ”mooouwxoom m n One. nuooaaoo a.uoowaoxm : sham nonueoa ”monoumxuem N s new. nuooasoo u.uooaanxm n such unseen "monouwxoem H utOmMMem nooaMuom aowuowouuo 05v HHH "sumo «codewowao wcwooae uncaauem a mu nouuoweuou moo Huunuuo adunoavoum use: HHH ouoo "ouuaausum uooBoon< Heooauuoaoum any a candy 73 frequency of one selection in terms of percentages and the third column translates the percentages into number of sessions. Thirteen cues were most frequently col- 1ected from the cue inventory averaging 91 cues (see Appendix B for the SIMCASE Cue List and Appendix E for the PAPERCASE Cue List on Case 111). Contrary to Case I and Case II, the group of clinicians diagnosing Case III did not choose to examine record of test booklets more often than other forms of information. Instead, cues were almost equally distri- buted among three forms of information: test booklets, test scores, and examiner's comments. However, like Case I and Case II, four (1, 2, 3, 4) of the 13 most frequently collected cues referred to background infor- mation and four other cues (5, 6, 12, 13) referred to oral reading information. No other clusters of cues were observed. The 13 cues selected in 50 percent of the ses- sions indicated very low Proportional Cue Agreement for Case III. The amount of cue agreement observed in Case III exceeded Proportional Diagnostic Agreement by only seven categories. Thirteen cue categories were agreed upon in 50 percent of the sessions and six diag- nostic categories were agreed upon in 50 percent of the sessions. This observation was contrary to Case I and ' 74 Case II in which the number of most frequent cue cate- gories exceeded the number of most frequent diagnostic categories by a wider margin. Table 10 reports the average amount of Propor- tional Agreement on cue categories for all three cases. The mean proportion of cues agreed upon by the group across all three cases was approximately .30. The average frequency with which the same cues were selected on a case was only slightly higher than the average fre- quency with which the same diagnostic judgments were recorded on a case, approximately .20 (see Table 6). Summary of Proportional Agreement Results The case by case examination of the diagnostic categories most frequently agreed upon by the groups uncovered percentages of group agreement which were constantly low. The average amount of Proportional Agreement per case was approximately 20 percent. For each case, only five of six of the same diagnostic cate- gories appeared in 50 percent of the sessions. Half of the time the clinicians included five or six of the same judgments in their diagnosis and half of the time these same judgments did not appear in their diagnosis. For example, in four sessions the clinicians identified "Werd Analysis" as a weakness in a case and in four other sessions they did not mention "Word Analysis" as a 75 mwm. oh w HHH ammo Hon. mm m HH ammo mmm. Nu m H ammo uaofioouw< mofihowoumo x0 mcowmmom ammo HmoowuquOHm one: .02 Hmuoe .oz ommu Mom mcowmmom HH< mmouo< homeomouwo moo no ufiofimouw< HonowunoQOHm owmhm>< owumwumuw ufiofimmuw< Hooowuhoeoum may OH manna 76 weakness. There was only a 50-50 chance agreement that "Word Analysis" was a weakness, far from.a group con- sensus on a possible area of difficulty in the case. The five or six most frequently mentioned diagnostic categories were mostly general statements of weakness which were loosely connected with the actual areas of reading difficulty in each case. The case by case examination of the cue cate- gories most frequently agreed upon by the group uncovered percentages of group agreement on cues to collect which slightly exceeded group agreement on diagnostic judg- ments. The average Proportional Agreement per case on cues was approximately .30. For each case, from 13 to 23 cue categories, as opposed to only five or six diag- nostic categories, appeared in 50 percent of the sessions. In Case I and Case II clinicians most frequently collected information in the form of record or test booklets. In all three cases, there was a low concentration of cues in the areas of background information and oral reading behavior. Four different cues on background information and four different cues on oral reading were collected in 50 percent of the sessions per case. I In sum, the very low Proportional Agreement on diagnostic and cue categories in this study bears a strikingly close resemblance to the very low Proportional 77 Agreement on diagnostic and cue categories found in the original Clinical Studies Project observational study, 0877.1. Commonality78core Table 11 reports the average Commonality Scores on diagnostic categories for each case. This statistic described the degree to which any individual session on a case included the diagnostic categories most frequently recorded during all other sessions on a case. The mean Commonality Scores displayed in the second column were derived by comparing the diagnostic categories within each individual session on a case with the most frequent diagnostic categories in all other sessions on a case, irrespective of the observational instrument. The mean Commonality Scores in column three were derived by com- paring the diagnostic categories within sessions utiliz- ing the SIMCASE instrumentation with the most frequent categories in all other sessions on a case. The mean Commonality Scores in column four were derived by com- paring the diagnostic categories within sessions utiliz- ing the PAPERCASE instrumentation with the most frequent categories in all other sessions on a case. The second column of Table 11 indicates that, on the average, any individual session included approxi- mately 57 percent or half of the diagnostic judgments 78 gums. n M Noam. n M Namm. u M HHH Name. u M xmmm. u M Nenm. u M HH Nmmm. u M gems. u M Numm. u M H maowmmom moowmmom mcowmmom HH< ammo mm Hmnmw>wmoH HHH mam HH .H momma "owumwumum huwamcoeaoo may Ha maan 79 which were most frequently recorded for the same case during all other sessions. In other terms, each session on a case had 57 percent of the most frequently recorded diagnostic judgments in common with all other sessions on the same case. It is interesting to observe that the Com- monality Scores for sessions exclusively utilizing the PAPERCASES were slightly above the mean Commonality Score for sessions exclusively utilizing the SIMCASES. On the average, individual sessions on PAPERCASES included from 63 to 78 percent of the most frequently recorded diag- nostic judgments in common with all other sessions on a case while individual sessions on SIMCASES included from 49 to 55 percent of the most frequently recorded diagnos- tic judgments in common with all other sessions on a case. Table 12 reports the average Commonality Scores on cue categories for each case. This statistic des- cribed the degree to which any individual session on a case included the cue categories most frequently recorded during all other sessions on a case. The mean Commonality Scores displayed in the second column were derived by comparing the cue categories within each individual ses- sion on a case with the most frequent cue categories in all other sessions on a case, irrespective of the obser- vational instrument. The mean Commonality Scores in column three were derived by comparing the cue categories 80 Home. u m Noam. u m Home. u m HHH HHmH. u M ammo. u M HHmo. u m HH NHme. u M NQOH. u M News. u m H HHco meonmmm mHno mtonmmm amen mmm Hmnmw>wmca "owumwumum muwamooafioo one HHH mom mm