ill /////l W I/ /I/////I////Il/////I///I/I///I///I////l////I 3 1293 10511 3934 RETURNING MATERIALS: ‘IV1ESI_J Place in book drop to LJBRARJES remove this checkout from ‘ .‘lllzg-IIL. your record. FINES will be charged if book is returned after the date stamped be10w. .351 1; 3:1 ::::1:'. :m- V i I I ‘§ hinP‘ i 5,3.q. . ~ A J V" v“- I! f1 LEARNING FROM EXPERIENCE: APPLYING THE CLINICAL AND EPIDEMIOLOGICAL RESEARCH PARADIGMS To THE STUDY OF b’iquggis AND TREATMENT IN READING BY Christian Carl Wagner A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY D . . epartment of Counseling, Educational Psychology. and Special Education 1982 __ ' — Copyright by Christian Carl Wagner 1982 ABSTRACT LEARNING FROM EXPERIENCE: APPLYING THE CLINICAL AND EPIDEMIOLOGICAL RESEARCH PARADIGNS To THE STUDY or DIAGNOSIS AND TREATMENT IN READING BY Christian Carl Wagner This research study examined the problem of how clinicians in any fieHIlearn from their experience. The traditional method for learning is through the informal aggregation of experience with its inherent problems of poor, biased human memory and inadequate variety. Alternate, formal methods, embodied in the clinical and epidemiological research paradigms. are universally accepted in the medical field as the basis for the continued improvement of medical care. These paradigms were. here, generalized to the diagnosis and treatment of problem readers. This study in reading was not intended ‘to judge the ultimate adequacy of the“ Paradigms for reading. This was made impossible by the limited Vmfiety C” reading problems, diagnoses, and treatments examined. In the study all Problems were quite severe, one cause was assumed and one treatment applied. Rather, the study was intended (l)to explicate the requirements for applying the research paradigms in reading. (2)t° suSlgest refinements for future use of these methods. and (3) to document QM‘ - o . pIrIcal relatIonshIps among children with reading problems, their 122‘3585. I'EEZY‘Q‘ES. "I 51.3.: .as I :m'n :‘ an: 13; I: :Ot'I. :‘ea: R; :' "“ ”31‘! use". De 3‘: 'vu “'13‘353' az. “'1 it xsz-t'eazn ‘: as: .. 0 utSC'Ifli ca . 51:42-15; ' 31.3: F. A Np P u a: v .r‘ \ 'eSea' 431-. l" 1“ =e='- .‘ "‘ o 9 a, .‘ Christian Carl Wagner diagnoses, treatments, and outcomes. The study was conducted for one school year in the seventh grade classroom of one reading specialist who examined AZ and treated 30 children. Reading problems were of such a serious nature that even keeping the student performance from deteriorating further was to be considered a major accomplishment. Performance data was collected pre-, mid-, and post-treatment; the treatment itself was intensively coded by in-class observation. The data collected was used to generate reliable computer-assisted diagnoses and careful descriptions of treatment. Careful diagnosis was important since in the clinical and epidemiological research paradigms reliable diagnosis and differential treatment are the bedrock upon which valid learning from experience must belnnlt. The data was analyzed in the prospective, retrospective. and uwms-sectional time frames from both the clinical and epidemiological Perspectives. The major results of the study included: examples of computerized decision rules with which reliable and valid diagnoses could be generated: 2° Prevalence rates for diagnostic profiles among severely deficit seventh grade readers; 3. descriptions of specific treatment plans and their differential effectiveness across various problem profiles; he . . ' ' base lIne assocIatIons among students' personal, environmental. dIBQROStIC. treatment, and outcome characteristics; and S. U' u . . . o o g IdelInes and suggestnons for the contInued InvestIgatIon of the an. . 0 a A‘ 0-; 2" pi»: e no 4-. ea. ,. mm :‘ 1‘ese 'e :ueotl'. i ' eg‘e:: as: " :a' “C 23 CC' 5" E is ”3‘s ‘3. Christian Carl Wagner application of the clinical and epidemiological research paradigms in reading. Across all of these results, it appears that reliable diagnoses and differentially effective treatments can be found in reading, and that the clinical and epidemiological research techniques should be further emamined as means for learning from clinical experience in reading. ; 'Q'e . A V G: 3:21: a 331;, 7! '3‘ 1132. C3. “”3 “it as: a I €52.23, .V‘ k u «2.: M :‘a'man . I, u 2! 'es: 3‘ IV c i‘ 9%. I .‘ '35..e's‘. e 1": ”Y 0.5!. 32.3%, H Ii :2" Acknowledgements If I were a poet, I would write a poem. If I were a composer, I would create a song. If I were an artist, I would paint a picture. Being none of these, I leave it to my friends and loved ones to know how much I acknowledge and appreciate their help - for words alone are inadequate. My family - Debbie, Brian, Andy, my parents, and so many more My chairman - Dr. John Vinsonhaler The rest of my committee - Drs. Arthur Elstein, Lee Shulman, and Lawrence VonTersch And my other colleagues - Dr. Annette Weinshank, Dr. George Sherman. Ms. Ruth Polin, Mr. Ray Smith, Ms. Linda Vavrus and Ms. Debie Salters Chris Wagner TABLE OF CONTENTS LIST OF TABLES . ...... . ......... . ..... . ............................... vi CHAPTER I Problem Statement .. .................. . ............... . ...... l Informal Aggregation of Experience ................................. l Formal Aggregation of Experience ................................... h Applying Formal Methods for Aggregating Experience to Reading ...... 9 CHAPTER 2 Related Research ........ ................. . ..... . ...... .....l2 The Clinical Problem Solving Model .. ...................... . ....... IZ Informal Aggregation of Experience . ............. ..... ............. 17 Formal Aggregation of Experience: the Clinical and Epidemiological Research Paradigms ........... .............. ...22 Research Designs ................. .................. . ........ .26 Epidemiological Research Examples ............................3l Clinical Research Examples ............... ..... ........ ...... .3h The Clinical and Epidemiological Paradigms and Improved C‘inica] care 00.000000000000000.0......0.00.00.00.00000039 Diagnosis, Treatment, and Outcomes in Reading . .................... AZ The Nature of Diagnosis and Treatment in Reading ............. #2 The Current State of Diagnosis and Treatment in Reading ...... #3 Reading Instruction in the Classroom . ............ . ......... ..h6 Factors Affecting Reading Instruction ..................... ...h8 Measuring Reading Instruction ................................ h9 Models of the Reading Process ................................ SI The Model of Reading and Learning to Read (MORAL) ............ 52 Sununary of Related Research .................... . .................. 56 CHAPTER 3 Methods and Procedures ..................................... 57 ObjeCtives .‘OOCOIOOOOOOOOOOOOOOOC ..... O OOOOOOOOOOOOOOOOOOOOOOOOOOO 57 ‘- Procedures for Defining a Model of Process .................. ...59 General Procedure ............................................ 60 Procedures in Reading . ....................................... 6i “- Measurement Device Creation Procedures ...................... ..6l General Procedure .......................... . .............. ...62 Procedures in Reading ........................................ 62 Ee'E'a' 3' l ”Hesse: i'azewes Sew: =' - fl 0 "33!... ES .. "2:23:25 1‘; ;' :eaces'ees I 3e~e':' =v; =’\_P.‘. .es Uhuvfl :fle'v ""0eu. EVE:- ' 3:3. is a...” ‘. “F ' Uh. v 1:: ‘33:; :3. Swan. 1 :91'=S:e:: :.. vs:e:: I I'll e :mapy ‘ p ' I O “‘~3.a : Q"‘.. .. I. . - E A.' . 5v ‘ S h .I‘ a y e‘.’e b “A .e: l‘.’ .- ‘§el:.s. : e I ‘6‘ ‘ ‘ q 'I I.I“.I ’t W I L . wif- . ~ 'I F. t . 5‘U3flu s I .23.- ‘I‘X P L' t . v '3”: . 5 I::"‘n‘ I3. .. I L g A ' ~ ‘35-. i e) :. . ‘c ‘R ‘n III. Procedures for Reliable Diagnosis ............................ 7] General Procedure ......... ..... ......... ..... ....... ..... ....7l Procedures in Reading ........................................72 IV. Procedures for Treatment Observation .......................... 75 General Procedure ..................... ............. ..........76 Procedures in Reading ..................................... ...77 V. Procedures for Outcome Measurement .. ........... ..... ...... .....78 VI. Procedures for Executing the Study ............................79 General Procedure ............... ......... . ................... 79 Procedures in Reading ..... ...... .............................8l Summary ....................... ............ ........ ................ 95 CHAPTER A Results ......................... ..... . ..... .......... ...... 96 Introduction .............. ..... ....... ......................... ...96 Epidemiological Data Analysis: P,E -> D .... ...... . ...... . ......... 97 Characteristics of Diagnosis and Cross-sectional Analysis ....98 Retrospective Analysis ............ ............ .. ............ 108 Prospective Analysis ........................................ll2 smary ...OOOOOOOIOOOOOO... OOOOOOOOOOOOOOOOOOOOOOOO O 00000000 1]“ Clinical Data Analysis: D -> T -> O .............. ...... ..........ll6 Characteristics of Treatments and Outcomes ..................ll6 Retrospective Analysis ..................... ................ .l23 Prospective Analysis ........................................l28 Summary ................. ................ . ..... . ............ .l33 Other Analyses ................... .............................. ..l3h Grouping Decisions ............ ...... .... ........ ............l3h Equivalence of Alternate Forms ............................. .l36 Reliability Of Data 0......OOOOIOOOOOOOOOOOOOOOI.0... ...... 0.1“] CHAPTER 5 Conclusions and Implications lbs List csf References . ....... ....................................... ...159 References: Reading Tests .............. .. ........ ..l65 Appendix A: Test Battery Stimulus Materials ..... ..l66 A“’9“de B: Student Protocol Notebook ........... .2.l0 AWendi): C: Sample Simulated Clinician Diagnostic Workup ............ 28l APpendix D: Clinician Diagnostic Workup ..... .......... .............. 29A APpendix E: Equation of Clinician and Computer Diagnoses ............305 'I h Appendix F: Instruction Observation Form ............................ 3l0 Index ....... ..... ................................................... 3l3 - C h E'. .. ~. ‘ =F.‘l . 513 e. I. . VI . A lo..e- ‘ e § _ u 42‘ ”'1... e s Uh. 3‘ sl ‘- . :‘S'EQ. ..l 2e I " 0. ‘ G}... a 12g... $ e:. 5‘ ‘s llive’ah O as it. '- e I U :‘ 9": :- \I ‘ A 5 ~ _ ' It,“ Q’g. a. “5‘1- -; N U-‘te . e §a.~ e. an ~ 3:! .1. o. N a. s e ‘1 -. e; ‘ HI “e I u E35 ‘3' see... LIST gg TABLES Table l The Classic Epidemiological Study ............................. 7 Table 2 A Typical Clinical Research Study ........ . .................... 8 Table 3 Results of Unreliable Diagnosis .............................. 20 Table 11 Association of Smoking and Chronic Bronchitis ................ 32 Table 5 Association of Myocardial Infarction and use of Oral Contraceptives ......... 33 Table 6 Association of EBV Virus and Incidence of Mononucleosis ...'...3lI Table 7 Association of Mortality and Surgical Procedure .............. 35 Table 8 Association of Mortality and Surgical Procedure Stratified by Liver Function 000.00.00.00. ......... 0.0.000.0000.0000000000000035 Table 9 Association of Drug Treatment and Mortality for Hypertension .36 Table IO Test Booklet Characteristics ......... ....70 Table ll Criteria for Decision Rule Diagnoses ................... 711 Table 12 Available Data by Student ..... ...................... 8h Table I3 Prevalence of Critical Performance Problems ................ 100 Table III Agreement Among Diagnostic Sources: The Statistic Definition .0... ......... 000.00.00.00... 000000 0.0.0 0000000 102 Table 15 Agreement Among Diagnostic Sources: Critical Performances ..l02 Table 16 Agreement Among Diagnostic Sources: Profiles ............... IOI-I Table I7 Average Agreement on Total Diagnosis Across Cases .......... l06 Table l8 Differences in Self Perception Based on Diagnosis .......... IO9 Table l9 Differences in Self Perception Based on Diagnosis by Sex ...lll Table 20 Differences in Rate of Problem Diagnosis Based on Ratings °f Language Ability ............... ....... . ....... . ....... . ....... 113 Tab; alaglr—ferences in Rate of Problem Diagnosis Based on Ratings ength of Utterance ........... . .......................... llh vi 'c ‘- 'z . I '[3’0 I. ya - Ltaks ”.... 222'“: 2a.: :13 Pratt 22- ”ex: 2': "U‘ 0 3 II. e..- I e S" I r 4 ' e UU. e51“??? 91' lssx'a: : 3:51:33: a: ; . ‘: :'.. ‘ . .1 \l. 2.:q o “ e ‘ F I 0 e . Q,- U 3": C- a 4' :G:S:e: A» . I‘ I st. 3‘. ‘e A. " Jo '1‘..a:\ I- e 4' :nap’v A Table 22 Mean Days of Instruction by Treatment Group ................ 117 Table 23 Characteristics of Instructional Time by Treatment Group ...119 Table 2% Prevalence Rates Before and After Treatment Across Treatment Groups .00....0000 ..... 00.000 00000000000000 12] Table 25 Distributions of Gains Across Treatment Groups .. ........... l22 Table 26 Differences in Rate of Poor Attenders Based on Relative Outcomes ..... .0000.0.0.0.000...0.0.0.00000000...]2h Table 27 Association of Gain and Class Attendance ....... . ........... 125 Table 28 Associations of Outcome and Time on Class Activities ....... 127 Table 29 Prevalence Rates Across Treatment Groups ....... I. ........... 129 Table 30 Performance Gains by Treatment Group .................... 131 Table 31 Decision: Choosing Students to Teach ...................... .1311 Table 32 Distributions of Alternate Forms .................. 136 Table 33 Differences in Equivalent Forms: Silent Reading ........... .139 Table 3‘: Differences in Equivalent Forms: Listening ........ ......ll+l Table 35 Reliability of Critical Performance Measures ........... 1A2 Table 36 Sumnary of Data Analysis ................ . .................. lh8 vi I CHAPTER 1 Problem Statement 3 Informal Aggregation 2: Experience Learning from experience is a human ability often taken for granted. An individual encounters a problem, perceives certain characteristics of the situation, makes a decision, takes an action and observes the result. Somehow this experience is encoded into memory and serves to guide future decisions and actions. The learning is informal and very natural. Reading specialists learn from their experience in this way. They are constantly confronted with problem readers. The specialists must decide what information to collect on the reader's environment and reading ability, what diagnostic classifications to use, what treatments to recommend and what outcomes to observe. The specialists learn by "YIN to perceive patterns of problems, treatments, and outcomes that confirm existing practices or suggest alternate ones. Each experience "It“ a good reader or a poor reader can serve to inform and improve the specialist's diagnostic and remedial abilities.“ :IEEhWork reported herein is sponsored by the Institute for Research on for R'DQ. College of Education, Michigan State University. The Institute the Nesearch on Teaching is funded primarily by the Teaching Division of Edmailzlonal Institute of Education, United States Department of . VEfleclton. The opinions expressed in this publ Ication do not necessarily of Edu “.13 POSItIon, policy, or endorsement of the National Institute M Alt:atlon. (Contract No. 1:00-76-0073) . and treWgh the research described here is concerned with the diagnOSIs equal] atment of readIng disabilities, the general concepts should apply Y as well to any clinical problem solving field. VTvfi—W ...n. .10. e. 0e. 212' 2'3! '5 :' u 1- ES 3" 'J'a" '» r 2‘: :a':':. 2:. :2 Iaeex 3' :5” s ‘:3: Q'- e, . s K. ::xs 0 :- U . . ~~!.:. e o ‘.. . ' b~ HQSQ In" ' .A, . e~ 9 !A:Q’ I e: C H “ IQ: .h‘ N 0 I 3 ‘19:. e.- ‘ . I I :“:$.. - 5 A” ' In. Unfortunately, this informal method for learning from clinical experience is problematic (Einhorn, 1980, Einhorn 5 Hogarth, 1978). It relies on human memory, which may be quite faulty and biased: it relies on the particular set of cases the diagnostician encounters, which may be quite narrow and non-representative. The result is learning that, at beest, is incomplete and does not make use of all information potentially a\/ailable in experience. Far worse, the learning may actually be in «di rect opposition to what full and systematic use of experience would indicate. Given these problems with informal methods of learning from experience, it is surprising that no alternate methods for learning from cl inical experience seem to be in use in reading. Reading clinicians Sti 11 learn primarily from their memory of the cases they see. As a resentg the wealth of information available to an individual clinician through experience is not being effectively used to guide future diagnostic and remedial decisions. Furthermore, clinical experiences aCross clinicians are not being aggregated to guide the prevailing views 0f the field as a whole. Although the field of reading diagnosis and treatment has not yet 'developed better techniques for learning from experience, another clinical problem solving field, medicine, does offer an alternative. After more than a century of development, it offers the formal clinical and epidemiological research paradigms for the aggregation and IHtegrathon of clinical experience. These paradigms take into account the nature of human decisions and thinking and yield more reliable and Valid learning from experience than do informal techniques. The ; .3... v V. In: I) 3?. :‘E' 42'! C '2533‘52 I: 5" Brett. "cw ,.‘: . 5 . 2:. re ;_ Nee .. . I- n. .‘ ..' a 3 E 3,, ' '1. . 15:23: :1. .. 3.:!~: “ 0. u ‘ e . '23:.‘P I" - D ' . ‘ an :3?!" . “:9.“ .5 .‘ A‘ significance of these paradigms for the field of reading diagnosis is that they were developed for use in a similar decision making setting in response to similar problems Stemming from informal learning from experience. Therefore, it may be argued that these paradigms might serve as. effective guides for improving reading diagnosis and treatment tfsrough the aggregation of clinical experience. The fields of medicine and reading are, indeed, quite similar with reespect to the decision making settings of their clinicians. Surely, the <:c>ntent of the physician's knowledge is vastly different from that of a reading clinician. However, the types of decisions made by these two cii fferent experts are the same. Each clinician is concerned with the l1eealth of a patient or case. The case has some problem which draws upon true clinician's area of expertise. The clinician is expected to collect irtformation about the case and the case's environment, to make a diiagnosis, to suggest a treatment, and to monitor outcomes. The goal for tJIe physician and the reading clinician is to keep the case as "healthy” asipossible. Since physicians encounter the same decision making tasks as reading clinicians, they have also been confronted with the problematic nature of informal methods for learning from experience. Literature. on the history of medicine is filled with examples of improper medical techniques that were followed because of reliance on these informal umthods. Blood-letting as a medical treatment existed from at least the llth century but was discontinued as a result of findings from formal research conducted by Louis (Garrison, 1929: Lilienfeld 5 Lilienfeld, I930. Fh3h). The treatment of choice for gunshot wounds was “0". ’ e &-e ban: ' ‘ '32:: my 2: a e 0. en t d C, v 1. . t. l o 1:: 11:1 :1 f‘S-as ..‘In'se I “,0. I .. CE 13$ 13" it n:.ne::. el- “:9: : nine : tess rs ga -. 3'! X“ n‘ ' d e 3’ 0.... " 9:: ’e I :~al e ‘E P “'2"- ‘ u ‘a. :3... I ‘I'. 3 a»: ..- - .g~ "52:" 'u ‘. cauterization with boiling oil until Pare found a better treatment by direct comparison of outcomes (Benger, 1961). More recently, the accepted preventive measure for swine flu included a special flu vaccination until careful collection of empirical data indicated that this was unwise (Neustadt 8 Fineberg, 1978). In each case, the improper pr-actice was continued as long as informal means were used to aggregate arad integrate experience. Formal methods provided information that ctsanged clinical practice, information that was directly contrary to the innpressions gained from informal reviews of experience. As these examples illustrate, even in a ”scientific” field like Ineedicine, informal aggregation of experience is problematic. It is F>r*ob1ematic whether the physician is highly motivated to give the best caere possible or not. It has been problematic no matter how correct or irtcorrect the prevailing model of the human body was. Clearly, other fearmal techniques had to be developed for learning from experience. Formal Aggregation g: Exgerience The formal clinical and epidemiological research paradigms for learning from experience are the medical field's response to the problems with Informal techniques. Both the clinical— and epidemiological research Paradigms attempt to objectively and scientifically use the data from Past individual experiences in clinical problem solving to improve future medical care. The paradigms direct the careful, systematic recording and comparison of characteristics across many similar and contrasting cases. They do not rely on human memory and continually our ‘23 3' 1‘: 911' rats. "9- I - - DI... 'AI.'US - D. Q 0" I G ' "es: 1»:- 'e 333 El sc-v a; . la. 0 rm: arts: fl las :Q'S:- D‘ 0e I - , ‘0 . . ' D.- . ' .. 1 : 3'95: :11: '3 a: e a‘c "’a. rd." 0".“ ”'5: focus on the problem of nonrepresentative or narrow clinical experiences. They have contributed greatly to the quality of medical care (Roberts, 1977). These two research paradigms are intimately tied into the clinical gsroblem solving model of medical diagnosis and the individual ciifferences across patients. Medical care is concerned with a patient \wfwo has personal characteristics (denoted P), who lives in .an ewavironment with characteristics (E). The physician collects information cars these characteristics in order to reach a diagnosis (D) that is a Ireeliable and valid description of_ the true medical problems. The Earaysician must then use this diagnosis to choose a treatment (T), laciminister it to the patient and monitor the outcome (0). The clinical and epidemiological research paradigms, then, attempt t<> use the data (P,E,D,T,O) from individual clinical experiences to llnprove medical care. Both paradigms assume that the distributions of 'these various data in the medical setting are related and not random. 'The distribution of these characteristics across all cases can assist in determining the overall effectiveness of clinical problem solving Practices. Even more important for improving clinical problem solving is the comparison of these characteristics across various groups. Such Comparisons can indicate the differential effectiveness of alternate Clinical problem solving procedures. The classic division that exists between clinical and wfidemiological research is defined by the data used and the data lWedicted. Epidemiology, is concerned with explanation, with defining I? I. O .' 1212; r ‘Ou‘ - v- a." e .‘kl e .. 9 J~e es a“: 7 "9-51: :13 h .- e". . BIS as -£ l‘ . _ . Q L' 3 3” : s ‘ ‘ :a . e 321:5: n‘. ..3. ”‘32: M c. I. ~ ‘ b a |‘.\-\ ~=:~;. sea..- e.~ 5-: s g. .- v“‘: ‘- O\.\ “-1.9 . 5 ~~|e .‘I '1: I. ..' \ . ‘CE :\ I “a O! ‘ 'E "\- : . lax ‘ e 2325 ‘1. s. n:. i 3.2: :‘E a. . . . 4 ___._M different diagnostic categorizations and the factors affecting their presence or absence. For the practicing physician, this is the first part of the clinical problem solving process, the part that maps the personal and environmental characteristics of the case into a diagnosis (PQE -> D). Epidemiology searches for relationships among the personal and environmental characteristics of the patient (P,E) and the patient's Ineedical diagnosis (D). Since one diagnostic possibility must always be "reo problem," epidemiological research examines both patients and tween-patients and focuses on the “causes” of medical problems. This type (sf: research can, then, be used to assist in the diagnosis of patients by latsysicians as well as in attempts to prevent medical problems by public i1eealth officials. Clinical research, on the other hand, is concerned primarily with acztion and outcome, with defining different treatment alternatives and trte factors affecting their success. For the practicing physician, this Is the second part of the clinical problem solving process, the part that maps the characteristics of the case and its medical diagnosis onto a treatment that produces the best outcome (D -> T -> 0). Clinical research searches for relationships among the treatment (T) and the Patient's outcomes (0) for a given diagnosis (D) in an attempt to discover the most effective treatment for a given profile of patient data. Since this research is concerned with a specific kind of outcome, i.e., the outcome of a treatment applied to a particular problem, Clinical research examines only patients and focuses on the treatment of nmdical problems. This type of research, then, can be used to assist in the choice of treatments for particular patient profiles and in the titer: :" a I: :e : :g :a 111': :5. 3‘9 ‘36 “:12: .212? “9'9 I112! . .O " «...aq a... . U :5 :‘e e .a:'..' \ . . 3 ". ‘S:‘ 5‘ . '03:. In. 5 b s a“. \- 2 A" '- ' .215”: ‘1'!" - ' 11...... s. . . , ‘e T ' ‘: ”I ‘0 I V“ .a. A by. it..- 5 I? be ..a. k E ‘Q 0 k. .. ‘\ S SEN/I. .‘._. S s .Q ,9 ‘e A .3152? it. I "S A: VJI e. s e ‘\ .“-:. e .G ' 3" e statement of a prognosis. The differences between clinical and epidemiological research can be clearly seen in the following two examples. The prototypic epidemiological study was conducted by John Snow in 185h (Lilienfeld 8 Lilienfeld, 1980). During a cholera epidemic in 18h9, Sanow had noted that cholera rates seemed particularly high in areas of Leandon where water was supplied by two specific water companies, the .Scauthwark and Vauxhall Company and the Lambeth Company. These suppliers cit-ew their water from a very polluted section of the Thames downstream ft-om London. Snow suspected that this contaminated water was implicated ire the transmission of cholera, possibly through some intestinal germ. 'Tfeen in 185h, another cholera epidemic began. Snow decided to test his Staspicions and obtained the data summarized in Table 1. Table l The Classic Epidemiological Study EJwvironmental Number of Diagnosis: Deaths per Characteristic: Water Houses Deaths from 10,000 SUpplier Cholera Southwark 5 Vauxhall Co. h0,0A6 l,263 315 Lambeth Co. ' 26,107 98 37 Other 256,h23 l,h22 59 Note that the death rate from cholera was 5 to 10 times higher in the houses serviced by the Southwark and Vauxhall Company than elsewhere. That company still drew its water from some of the most polluted sections of the Thames. The Lambeth company had since changed its water source to another part of the Thames upstream of London where the other 21:11: «'2 31-9? at e*-. 1' "0 ‘ 322'» 1 3 t O . . :' .aee' semi“! : D I.’ :‘: 9" s A. 8:: 113' ‘I S. ts I!!! e. n companies were also situated. Snow had shown such a strong relationship between an environmental characteristic (water quality) and a diagnostic category (cholera) that the existence of a ”cholera poison” transmitted by water became an accepted fact. This new understanding about the cause cfl cholera suggested a rather obvious, preventive measure. Within two years, laws were passed requiring all water companies in London to filter their water. Note that this research did not address the proper care of an individual who had contracted cholera but was concerned, rather. INith understanding cholera and the factors affecting its presence or absence. A typical clinical research study was conducted by Fisher and colleagues (Burdette 8 Gehan, 1970, p.39). Fifty-nine women with anfiirmed breast cancer treated with radical mastectomy were followed-up for 5 years. During this time 23 of the 59 received adjuvant chemotherapy with the drug thio-TEPA and the remaining 36 received a placebo. The resulting data are given in Table 2. Table 2 A Typical Clinical Research Study Recurrence of No recurrence of cancer within 5 cancer within 5 years - years Thio-TEPA adjuvant therapy 65% (15 of 23) 35% (8 of 23) “3“” 86: (31 of 36) 1A: (5 of 36) The data clearly suggest the therapuetic possibilities of the Ch'e'I'C’tI‘I’I'aPy. After analysis, the differences in recurrence rates were statistically significant. Note that this research addressed the problem o. :0“ 0‘ One ‘7 D. U a“ 5'0 Eve's 2 ~- "SII". .A ... ‘ in " 5U . ..l. Iva-u ‘ ' "in. a’fig‘lgs. '., “3 395 it. : {'3‘ It. , ~ 0 ' ‘- n‘ :. I.. K E' . I . H! :‘a .e .. 5 53‘ .; I}? i"§“a\ ~ V \.I V . I‘ ‘ 5 :12... ..‘E. I o .2 a: a. u, t. of treating patients with cancer rather than understanding the factors related to the presence or absence of such cancer and trying to prevent it in the first place. Generalizing the medical paradigms of clinical and epidemiological research to reading diagnosis seems straightforward. The concepts of clinician, problematic case, diagnosis, treatment, etc., have direct analogues. The generalized clinical and epidemiological research paradigms would direct the aggregation of empirical data within and across individual clinicians and readers. The data would be used to relate empirically the particular characteristics of problematic and healthy readers, available diagnostic and treatment possibilties, clinician decision making, and changes in case performance following these decisions. These empirical associations could then be used to determine and improve the effectiveness and efficiency of clinical problem solving performance in reading. M 2% Methods for Aggregating Exgerience .t_o Reading The successes of the clinical and epidemiological research paradigms in medicine and the apparent similarity between the fields of reading and medicine suggested that the generalization of these research paradigms to reading could serve as an effective alternative to the current informal methods of learning from experience: they could lead ‘0 improved care for problem readers. Therefore, the research reported here attempted to generalize the clinical and epidemiological research Paradigms from medicine to the field of reading diagnosis. This was u . + 1‘. w- 569: :~ on C .v- p ' "J: ‘3. > I .- "13‘33; ‘tht's. ‘e v T9" 39":"3*:: 32¢ 13": Tu "5 12:12: s: 3132' Sf}: :. : :e atr— 31.31%“ ....‘.. O ..- “ :.': S5 1‘! '55» :s '2’: .2: .33 h ' is g the c. c‘.“ ~ ‘s- I'es. ’5e .l'l..‘ - . "e: by .‘ ' s ‘etgleks Q13,» in ' “ -. ht...“ ‘0 s 85:. !‘ ;QV cg ‘ 3.: "‘9. e 3:. u. . ' 1 Ce; lO accomplished by conducting a clinical and epidemiological research study in reading. The study was run in a setting in which one reading clinician diagnosed, treated, and followed up a number of problem readers. The readers examined had such severe problems that preventing their performance from falling further behind their grade placement would be considered a major success. Many characteristics of the setting were recorded so that the relationships among personal and environmental characteristics (P,E), diagnoses (0), treatments (T), and outcomes (0) could be determined and compared across naturally occuring groups. Then, co-occurrences of case, clinician. treatment, and outcome characteristics were counted, compared across groups, and interpreted as a means for identifying the more reasonable decision alternatives. Since this study was apparently the first attempt to generalize the clinical and epidemiological research paradigms in the field of reading, the'results were not expected to rule on the ultimate adequacy of the paradigms for reading. Clearly, no single study could be so persuasive. ‘5 is true for clinical and epidemiological research studies in medicine, the outcomes of this study were attenuated by limitations on the number and variety of cases examined and by the chosen research techniClues. The generalizability of the results of this study was restrained by the limited variety of reading problems, diagnoses, “d treatments examined; the reading problems were all of a severe nature f°r Which one common cause was assumed and one baSiC treatment prescribed. It was only after a century of medical studies limited in this way that the medical field came to universally accept the tenets 0f the Clinical and epidemiological research paradigms as a foundation for ii 23": “be: ”I 19:39:13 .:’ 'M-‘t’. :- s “L'M'is :‘ Z‘t 1: Ite‘. I‘m '23- .53.:‘ ”'2 the 2'3: 7"" '7" :e c 2:331; 3‘ avert: . s “43". 1 :39"?! a :"~'1 s...- l “5 SR" ii U "S; ll the continued improvement of medical care. The field of reading could hardly come to such acceptance on the basis of one study. However, this initial study did provide many lessons on the precise requirements of the research paradigms and the reading field's ability to meet these requirements. Possibly the most important results of this research were the more explicit requirements for the continued examination of the research paradigms in reading. Whether or not the paradigms will be ultimately acceptable cannot be speculated, but the very common sense notions of learning from experience by systematic recording of events deserve further scrutiny. This study, then, had two purposes. The first was to determine how to perform a clinical and epidemiological research study in any clinical problem solving field in general, and to carry out such a study in the field of reading diagnosis in particular. The second purpose was to evaluate the results of applying the clinical and epidemiological research paradigms in reading with respect to further understanding of reading diagnosis and treatment and with respect to further understanding of the task of generalizing the paradigms outside of medicine. Specific topics were: l)baseline likelihooods of significant student characteristics, 2) associations between personal 0" environmental characteristics and diagnoses, 3)aSSOCiati°"5 between diagnoses, treatments, and outcomes, and lo) problems and concerns with the use of the two research paradigms in reading. eln" ‘ 'Afl‘ ' ' at!" 0:" 39;.2;31.3. 7?! a salt"; :.a : ' ‘1‘ 3": cc 3:.- tl2-'en:e' a“: ’5’“: r": : ave P e-. 31:255. 31:3! fan. ‘ J. a O. ‘ L A.“ ug~'e p '\~ I. - En. Ea in... “ s a y a? :2 t. \Se a'ea CHAPTER ; Related Research The purpose of this chapter is to document (l)the characteristics of the clinical problem solving task, (2) the problems inherent in using informal aggregation of experience to determine and improve clinical problem solving quality, (3) the methods and past successes of the formal clinical and epidemiological research paradigms for aggregating clinical experience, and (h) the current state of clinical problem solving in reading with particular emphasis on models of the reader and reading process . Ills. M Problem Solving Model The clinical problem solving model is appropriate for many different content areas (Lusted, I968. Preface). It is used in fields concerned with the operation, maintenance and improvement of complex systems. For example, medical clinicians work to solve the problems of their Patients; reading specialists - the problems of their readers; electronic trouble shooters - their circuits; auto mechanics - their aUtomobiles. in each case there is an expert responsible for solving and preventing any performance problems of systems in his/her respective area. These areas may differ vastly in the characteristics of their ”Stems. but the thinking processes of the experts are remarkably Similar. 12 ".2 my "3 2211‘s 3'! :‘ea'. :3: ex 5151!! :a e. e l . “3'35: {a 33': :~ on; s 1') :35! s Re‘ 3".a":g 13 The major cognitive characteristics of the clinical problem solving setting are clear. There are two participants. The first is some type of complex system called a ”case." The case can be viewed as a set of parts which interact through some internal process and which are directed toward achieving some goal. It is assumed that the "health" of the case can be inferred from its ability to demonstrate competence on a certain set of Critical Performances. The case's perceived competence depends upon. the internal process by which the various case parts interact. The second participant is an expert clinical problem solver, a clinician. The clinician has the task of maintaining and improving the case's performance. The clinician's ability to do this depends upon the set of actions that-can be initiated by the clinician to improve case performance and upon the cl inician's own model of how the various case Parts interact. The interaction between clinician and case is usually initiated by a schedule for routine and preventive maintenance or by a problem with case Performance (DeGowin 8 OeGowin, l976, Chapters l-Z) - Once the pa"thiiPants are together, clinical problem solving begins. The first thing the clinician must do is collect information on the characteristics of the case and its environment in order to detect exiStinfi or potential problems. The clinician then selects a procedure believed to alleviate or prevent the problems. Finally, the clinician executes the procedure and monitors the case's response in ”d” t° evaluate improvement, These actions taken by the clinician have been °r93m2ed around the terms "diagnosis," ”treatment," and “followup." For a: r': 39.. :‘e '. 7:531: :ase :‘ :se'ez'. ”:- b 1". vs: raft s Q A ‘ a L. 33':. I Q \I-e§ ., I "C. e. ‘ h n d 'I e 45! uni ‘3199 : r. t, o; , 5 . C3. P. 5 :w. I i"“ not! 9 . 5 ‘5 9:- e“‘ :‘l _ ‘fi‘ ’ u - s- u. \ " - ‘s 5 \al I ‘3}. . he" ‘5 I“. U C“ lh a clinician, the term "diagnosis'I is defined as the statement of any problematic case performances and their causes. This statement is used to help select treatments and is based upon all information known about the problematic case (in medicine - Wulff, l976, Chapter 7; Barnoon 8 Wolfe, l972, Chapter 2; in epidemiology - Friedman, l980, Chapter l3; in readilwg - Spache, 1976, Chapter l; in system design - Hare, 1967). "Treatment" is the specification of a set of actions initiated by the clini<:ian to improve or sustain case performance (Roberts, 1977, Chapter ‘0: Spache, l976, Chapter l). "Followup" is the monitoring of changes in case performance that occur during and after treatment and the adjustment of diagnostic and treatment decisions based upon these changes (Small 8 Krause, l972). The clinical problem solving task, then, is directed by the diagnostic. treatment, and followup decisions of the clinician. But the situation is more complex than these simple definitions imPly. At least two different levels of diagnosis are often used, (Vinsonhaler, Weinshank, Wagner, 5 Polin, 1982). The first level is SimPly a statement of problematic functioning. The clinician examines all Critical Performances* of the case and compares the case's actual Performance to the case's expected or optimal performance. This first ‘eVei of diagnosis is often no more than a statement of £335 performances are problematic . The second level of diagnosis is a statement of the cause of the Case's Problematic Critical Performances. These causes may include other -~ --- ----------—-- k . . esfrétfca] Performances are a set of criterial evaluations used to a llstx the state of case performance. :‘zx'efi'. :‘a'az'. mam m er. 3:"2'92225 re :2 "is same it"?! tan 1": : tx2'1:".':-a, ,_ 1‘ :zse :'::ess as I h I'la n" w 0 Speed . 3 ._ It . Eemeq: ‘ a 5 ‘ss la ’ I ‘ Ht. A ‘ vi‘ pe’e I| :‘ 5.“ .3 Fr- ‘ b I? P . 2 ‘- "- a: . I ~33 'lg ‘. 'i H :-. ‘u .g ..‘I‘n - .e s a. aft, I. h 5, es . ‘3 ’. 0 1;: ‘. . :5, n Slect ”a“ a“ "'e a. E).- ‘Lc '- ”Er t k 5-... “it.“ . S . e We 5“ p ‘. Q :‘~ U concurrent characteristics of the case as well as the case's history of performance and experience. This second level of diagnosis describes why performances are problematic . 'This second level of diagnosis is generally more difficult to perform than the first level. The second level diagnosis requires the exper”t clinician to have and use a model of the case process. This model of case process must relate the functioning of internal case parts and the I1istory of case performance and experience to the case's Critical Performances. in some fields like automobile repair, the model can be very well elaborated and valid. The various internal parts (coolant SYStem, electrical system, drivetrain, etc., and the subcomponents of each of these) and history (miles driven, mainenance provided, etc.) can be reliably and validly related to the Critical Performances (steering ability, speed, acceleration, etc.). Other fields, like the diagnosis and treatment of problem readers, may have very fuzzy, unreliable and invalid models. Various internal aspects of the case (language exPerience, home environment, attitude, etc.) and its history (ihStruction received, etc.) are only unreliably and probabilistically related to Critical Performances (silent reading rate and comprehension, Oral reading fluency, etc.). assuming, of course, that the field can a9"ee on a definition for the important Critical Performances! Given these two levels of diagnoses, two types of treatments can be distihguished. One type of treatment would be aimed at alleviating the pr'°blefliatic Critical Performances directly " a 50 called symptomatic t”atment. The second type of treatment would be directed at the cause of the Pr<>blem. Whatever the type of diagnosis or treatment, the goal is z: i; :52 at: 22- "(5e 1:- :':s:r‘:e:. 3 Is emcte: Wee-ts 13v t 3'35 5:"; 1; 3': ”4‘ "I” :-.e :ase ,H. ' ~.". 3 9-9-33: ‘: swa' 2e. (m- m hose aw... leg is 353‘: Z“! l6 to improve the case's problematic Critical Performances. To this end, during the administration of treatment, the clinician uses his/her expertise to compare the actual treatment received to the treatment prescribed. S/he must further compare the case's actual improvement to its expected and optimal improvement. Any large discrepancies in these improvements may lead to a repeat of the entire diagnostic and remedial twoblem solving procedure with one difference - the diagnosis would then deal*with the case's ability to change with treatment as well as its mfility to demonstrate its Critical Performances. To summarize, the clinician is a problem solver. S/he has an internalized model of case functioning organized around decisions and actions about the following: 1. determining the significant personal characteristics of the case (P). 2. determining the significant environmental characteristics of the case (E), 3- determining the diagnosis of the case (0), both as to what Critical Performances are problematic and whY they are problematic. choosing the treatment for the case (T). and 5- monitoring actual case change and comparing it to expected change to determine outcome (0)- lhe basic chain of reasoning is P,E -> D -> T -> 0. The clinician uses “fluent area knowledge to make decisions concerning the measurement .P , e- e. 2 yr"... .5 H: as 0 t’ A' 3"... 2!... U o. 0 his 3.111925 :3 El 393'. - :a-‘s goal ‘_ , . A”. ...fi :- fig 35 on . "We ' .. . a l..v!:a ‘ : i" “rather” ‘- . H ‘3 Elbe. 10. I. 2.5“ . ‘ ~ I _h ‘ '3 e § '7"... ‘ U. ‘ \ 'Ql 'Y ‘ E 2.. lE la 5“ ' DOCQ' 'f 1.2- '\ net 5 a, a“ 1 .h 3%, a 5A ,g .' L- ‘h‘1 c" . ' WC ‘9‘ “.t.‘ CE:~ES l7 procedures to use next; the diagnostic classifications to consider, reject, or accept as case descriptors; the treatments to administer; the outcomes to expect; etc. (Elstein, Shulman, 5 Sprafka, l978). The clinician's goal is to keep the cases in his/her area of expertise functioning as optimally as possible within cost/benefit constraints. Informal Aggregation of Experience Determining and improving the quality of the clinical problem solving just described is a difficult task because the quality of these decisions can only be evaluated by comparison of outcomes across other similar decisions. Only when such relative outcomes are available is it possible to define better and worse clinical decisions. But relative outcome data can only be obtained through the aggregation of clinical experience. Therefore, any attempt to improve clinical decisions must be based upon data from clinical experience. Existing levels of clinical competence must be “learned" from clinical experience, as must methods for improvement. By far the most common method for gathering and using clinical experience is the informal aggregation performed naturally by any practicing clinician. TYPically..a clinician only diagnoses, treats, and follows-up the Particular population of cases that make up his practice. 5Pe¢ific Clinical decisions are only informally connected with case performance Changes, and these connections are only informally aggregated across cases and clinicians. As a result there are no empirical, objective eStimateS of decision making reliability or validity. 0f case rhea-:2 3m: 3.3m. "re a'. ;:-.:s 2‘ : - - ~ ~ :3:C;C' : 2;“:5 '23. :s. 5e:a.se may: 3‘ c' AL". . . 'v n ‘ ‘er-a-l :‘eai . l ’3 ~ ‘3' trza' ‘fa .. ‘C h {-3.- S ratify '- 0f Ca 3- Q t, H .‘l "e.“ N. U I e'.‘ D J ‘ ‘R. . IHI‘ JU' J 3‘: " ‘ e {W\, "or 0". U ‘a, ‘I L e ' 1". ‘ “Oh ‘~ 1“ I ,‘ I‘Qafi u‘. i performance outcomes, or of the associations of decisions and case outcomes. There are no estimates for individual clinicians or across groups of clinicians. Treatments of unspecified effect are applied to diagnostic categories with unknown reliability to give uncertain results. Because of the fuzziness in relating decisions to outcomes, improvement of clinical problem solving performance is extremely difficult. Alternative decisions and their respective outcomes cannot be meaningfully compared. These problems with informal aggregation of experience are due in part to the structure of typical clinical practice. A given clinician has limited time .and can only interact with a finite number of cases. Therefore, many circumstances may exist for which the clinician's training and experience are not broad enough to make valid and reliable decisions. But, even when clinical experience is broad enough, a given clinician will only interact with the particular, and not necessarily representative, set of cases that comprise his/her practice. Therefore. the clinician's experience may not be representative of the true Population of cases leading, again, to unreliable or invalid clinical decisions (Roberts. l977)- But even broad and representative experience may prove inadequate. Clinicians make decisions that lead to actions and then observe results (Einhorn, i930). So called "false positives," reaching decisions that are inappropriate, are observable and can be instructive. Unfortunately, no information is available for particular alternative decisions that were not made. The "false negative." not reaching a particular alternate dMission that is appropriate, is not observable (Einhorn 5 Hogarth. e. I I p 0 LN)? {'22. 3".123.'!S I“ 3'. 21m: :25. .; I‘e‘:::aa. ‘ean use: rc :e: s 3. :e :3 5:. “gt“: a. I m:- ’t2'21 1‘ a L?!" :ee 5... oe.:‘$! :‘ :h 93 ”2... ea: E. E .h ": i3" saws ‘( 'l ’i 9‘9. l'. "-1.: . o‘ses. ‘e a...“ ":9 .n \V b 0‘“ I -‘.: b” . ": ‘v': lA‘ I J“‘ 'e i‘- . is!” ‘ e 3' 52.. El“ 5. a IY ‘ t .- e e 'b s hie. ' e l 325.. l9 l978). Experience, then, is generally limited to refining existing procedures that seem to work and does not promote the testing of new alternatives. Typical clinical practice can, therefore, lead to less than «optimal learning from experience. The requirements for the types of cases and decisions that must be experienced for ideal learning are quite stringent and are rarely met informally. Broad, representative experience on all possible decisions is still not a guarantee that informal learning from experience will be valid because of the biased ways in which clinicians might interpret their experiemce. Einhorn (l980) reports research indicating that human decision makers ignore false positives and false negatives in learning from experience - even when they are available. From another Perspective, Tversky and Kahneman (l97h) describe a large number of biases that arise in human thinking that could improperly alter the use Of experience. For example, human problem solvers tend to make Propability judgements on the availability of instances in memory. As a Clinician considers a particular decision, s/he might well overestimate the efficacy of a procedure due to the high availability of his/her HSuccesses“ in memory. Phrased differently, if the clinician expects a Procedure to be effective, s/he will tend to remember only those cases Which confirm the expectation - the phenomenon of illusory correlation (SlOVic, 197A). Furthermore, a good match between a particular situation and shnilar successfully completed situations can lead to an illusion of Validity, that is, unwarranted confidence in results. BUt there is still one more problem. Assuming that all of the co . . . . . rmerhs unth informal aggregation of experience described above have assesses. s i‘t:"":'a' s :e 7:: sz'ti-t 3”“ Wei}: 2 i '51" turn $1)? 9‘32. fig :' ‘9 'BS.':5 3: .mp2‘ “.2. :al =‘ less (as? .-.o ”I O‘V' r w (”*1 3‘3. “"5! ‘ . o. 9% u :‘a! :1}. “an ”a this . manage: :5: .m e , . D 33'» 331': ‘ :3 e s of the .eec“’ ‘\ 20 been addressed, such informal aggregation may still be a poor teacher if the clinician's decisions are unreliable. Research in many clinical problem solving fields has demonstrated that human decision making can be very unreliable (Yerushalmy, 1972; Vinsonhaler et a1, 1982). As the clinician alters his/her procedures for decision making based on experience, the clinician assumes that his/her decisions are reliable. The results of assuming reliability when. in fact, the decisions are unreliable, can be seen in the following example. Assume that an unreliable clinician has treated the set of cases described in Table 3. Table 3 Results of Unreliable Diagnosis Case Number True Clinician's Selected Outcome Diagnosis Unreliable Treatment Diagnosis 1 A A X Outcome-l 2 B A X Outcome-2 3 C A X Outcome-3 h A A X Outcome-1 5 A A X Outcome-l 6 B A X Outcome-2 7 D A X Outcome-h 8 A A x Outcome-1 Because of the unreliable decisions, the clinician incorrectly diagnoses an cases as having problem A and uniformly prescribes treatment X. As the Clinician watches the outcomes of applying treatment X to these cases supposedly with problem A, s/he does not take into account that the trUe problem of many of the cases was not A. The clinician's e . stlmate of the effect of treatment X on problem A is inaccurate because th e effects of treatment x on problems 8, C, and D confound it. Clearly. _ :x 'M' :‘e '2 "92m: 2‘: 1: 95112:“; :‘e .se l3 1? ‘s e l‘:e:a: an that , “:9, CT! 8.3! at .‘ . "‘ Pa :3: 3: Which. c; 391; '- .- 1 a‘ DP. Or '21... ‘e v ‘ ‘ . e e..‘_‘c chi. V i m‘ “33'. ‘. 2:. "What .133‘1-1' ' iv 0 ‘mCQCQ ‘e .‘II: : SYSe h‘ ‘e 5:“, h. :‘R~. U5":a] it. re 3;: 21 the lower the reliability of decision making (and, therefore, the more frequent the misdiagnosis), the greater the potential for error in estimating the usefulness of treatment X for problem A. As this example suggests, even the biased and incomplete information that can be had from the informal aggregation of clinical experience may be of limited validity given the unreliable way in which clinicians make decisions. The implication of these characteristics of clinical problem solving and human thinking is that existing informal procedures for the aggregation of experience do not lend themselves to the valid detennination or improvement of decision making quality. Neither relative efficiency nor relative effectiveness can be meaningfully estimated. Alternative decision making strategies cannot, therefore, be reasonably compared. When this is true in a field (e.g., medical or reading diagnosis), even training programs will be affected. Students le be taught prevailing views of good clinical practice that may not fleas accurate as possible, starting the new clinicians out with less 'umn Optimal clinical problem solving expertise. What is needed, then, is a formal system for the aggregation of umfirical clinical experience, a formal system that would guarantee decishon making reliability and eliminate, as much as possible, the limitations of human thinking and individual clinical experience. One such formal system is available through the techniques of clinical and Epidemicilogical research. According to Roberts U973. P- ii). these te ' . . - ° ' ChmClues provade ”possubly, the only methodological framework Withln I‘ 2‘ S: “i! A“: :IlIal l.lv..a. A, ' ' ‘I oyv. ~ :ltggoeh ai'al- ..e been . ' ~ - m :Hn :3: a;-e;a:?or. of U: $33993 s.:-‘q ..Aeo' , no :Ues :a'n ' 1 r ‘ .. .23". 3?“. 501‘ :l-gnc: ... ‘9 DQ'SCRa 3t» -- a . filth-33 5 N . ‘5.” n, . gm he i'i'ar. .ttlits:. :S «‘h “nefstlts er! I the §~‘.~ u' , DU. A ':A | bSe I, '6 Hi S '.:E:u I I} be CA 2; ‘Sth'ctgnv q " I . 'hii he: be ‘ 1h :E‘n \ 22 which science and rationalism can be integrated into the clinical management of individual patients.“ Formal Aggregation pi Experience: tpp Clinical ppp Epidemiological Research Paradigms The clinical and epidemiological research paradigms address the problem of determining and improving medical care through the formal aggregation of experience.* Although these paradigms are generally considered sub-fields of medicine and public health, their concepts and techniques can, in fact, be applied to any clinical problem -solving content area. Both paradigms have evolved in the context of a clinician afllecting personal and environmental characteristics (P,E) of a hcase, nuking a diagnosis (D), selecting and administering a treatment (T). and observing outcomes (0) - Epidemiology is defined by Roberts (1977, p. 16) as ”the study of cfisease occurence in human populations.” Its primary focus is the first Fart of the clinical problem solving task - determining the diagnosis W) from the personal characteristics (P) and environmental characteristics (E) of the case. This focus can be represented graPhically as P,E -> D. It requires the examination of patients and non Patients ' whose various patterns of personal and environmental Characteristics are compared across diagnostic groups. * pr:°r.the purposes of the research reported here, only accepted clinical mwft|ce will be considered. Development of new diagh05tlc and remedial cedures through model building, simulation, laboratory experiments. et 0 . e e a o ofc '}“ll not be examined. The concern is, rather, with the optimal use ex'st'hg techniques. g El 1323' 5: "rates 99:; :‘. ti :ca'. :a: :53. :.‘:2,. "’- 33! 3' s.::ess‘. :5 ‘Xv! :He 5 1‘2. - 3‘:.u:s. , ' ~~ 5 el '1; 'A a ““4 :‘g {1' I ~I “.. ~P’b.’ . 5 t i - earl. a e ' e ‘ O ' ‘ith‘. . ‘ Pr. - 52' ‘J \ \' I ‘6 3:. u‘a U. A H .‘ Iflfi‘ e o‘er 23 By comparison, clinical investigation "is primarily concerned with inferences pertaining to health which can be drawn from the examination of individual patients" (Roberts, 1977, p.21). According to Friedman (1980. p.160), ”The ultimate goal of clinical studies is to learn how to cure or successfully treat disease." Indeed, clinical research has as its° focus the second part of the clinical problem solving task - maximizing outcome (0) by selecting a treatment (T) for a given diagnosis (D). This focus can be represented graphically as D -> T -> D. It requires the examination and comparison across outcome groups of patients with various patterns of personal, environmental, diagnostic, and treatment characteristics. The basic tenet of the clinical and epidemiological research Paradigms is that, to as great an extent as possible, clinical decisions shouhfl be supported by scientific, objective evidence based on the 'finmal aggregation of clinical experience (Feinstein, 1977). Experience is aggregated in terms of counts of clinical problem solving characteristics and functions of these counts across various naturally cumuring groups of cases. The careful recording of characteristics and the comparison of characteristics across groups is the key to both cHnical and epidemiological research. This recording and comparing is simple. empirical, and does not rely on faulty and biased human memory. 't can be aggregated across clinicians and cases to avoid un - rePresentative or narrow samples of cases. The precise coding of experience depends upon the characteristics of the data under investigation (Friedman, 1980, Chapter 2). Qualitative Char ' e e . e o aCteristics such as particular diagnostic categories tend to be 22m: x. 3'34: 33,! :35“ 'g .le ~'. ' 1" Q ‘ 3" . u.&: e 0“:- ‘ 1.. 5 .fi: a P! 3 \ U - ‘ sp a? Fe \. :Et. .‘x‘g .‘ Jae 5 Ir. Q h‘. ‘N {a I 2:: I .. i 'l W. :- \ up“: \ 2h compared in terms of proportions and functions of proportions (Ipsen, 1978). Some commonly used proportions include prevalence rates (the number .of cases in a group with a characteristic divided by the total number of cases in the group) and incidence rates (the number of cases in a group developing a characteristic divided by the total number of cases in the group per unit of time). An example of prevalence rates might be the number of people who have cancer in a particular area divided by the total number of people in the same area. An example of an incidence rate might be the number of people who were newly diagnosed as. cancer patients in a particular area divided by the total number of people in the same area divided by the amount of time over which the new cases of cancer were discovered. These rates alone are useful to nuividual clinicians because they provide empirical data against which u>compare their subjective impressions. Since subjective impressions are the basis of many decisions, confirming or denying their validity cmnd substantially change an individual's decision making practice. In addition to the incidence and prevalence rates, comparisons of rates across groups may provide useful information on the relative Si9nifiicance of alternate decisions (Friedman, 1980, chapter 2). This co"'Pef'ison is done through differences and ratios of proportions. Athfibutable risk is the difference of two incidence rates for two different groups; it describes the differing probabilities of a partlcular characteristic depending upon group membership. For example. a . . . ssume that two different inCidence rates for cancer had been e e stablished: a rate of 5.h cases per 10,000 women per year and a rate of 7'8 cases Per 10,000 men per year. Clearly men are at a greater risk. ‘t 312' 3.132 e ' If!" not: ti'. ie'az've .' 5 (la' . . I , " °$' 3.: he “I a: . “ce'fia SVQS a ‘ e," a, 3‘ t .ite’ 3.“: 'e 43 25 The attributable risk of belonging to the group ”men” compared to ”women” would then be 7.8-5.A a 2.A additional cases per 10,000 per year. Relative risk is another means of comparing rates that is the ratio of two such incidence rates. Quantitative characteristics, such as a particular test result, tend to be compared in terms of the characteristics of their distributions. These characteristics include the usual measures of mean, standard deviation, range, median, etc. Again, the characteristics of a single distribution are useful as a check against subjective impressions. Relative significance can be determined by comparison of distributions across groups. Of course, coding data is not enough. As for any research model, mam design must exist to guide the collection of experience. Feinstein U977. p.18) describes the basic structure of all biomedical research as ”atmmporal sequence in which a preparation is exposed to a maneuver, am! undergoes a response. The preparation is described according to its Hfitial state, and the response is determined by noting the subsequent state either alone or in comparison to the initial state." Of course. fix the clinical and epidemiological research paradigms, the I'i’reParation" is a person. This person is diseased or healthy in the lhitial state and the subsequent state. The maneuver may be controlled by nature, the experimenter or the subject. intern ' V' ‘ ‘e: :Aec 3‘ a“... (1“. . .. I233; 'iz':s:e::' it ‘,g ri'ts Rat's: ' 8 'm’. .l a' y. i'uB.a ‘.., -'.-,' ‘ ..S. De._ .3 L:.:'S \‘ze's: l's . tides. 12‘3. ‘ . :!'-e:s 1‘! :as N ' eh» .;._ “°.‘e e)‘ 26 Research Designs. The most common dimensions of research design mentioned by authorities in the field are setting (laboratory versus field), method (observational versus experimental), data collection (retrospective versus cross-sectional versus prospective), and data analysis (retrospective versus prospective) (Fox, Hall, 8 Elveback, 1970)* Unfortunately, there appears to be as many different ways to describe clinical and epidemiological research designs as there are authors (Alderson, 1976; Friedman, 1980: Holland, 1970; Holland 8 Kahausen, 1978; Lilienfeld 8 Lilienfeld, 1980; Hausner 8 Bahn, l97A). Perhaps the easiest way to examine the design parameters is to do so in the light of the clinical decision making model described earlier: P,E -> D -> T -> 0. Epidemiological research centers around the first part of the decishon making process, P,E -> D. Immediately three forms of research Cmi be described (Lilienfeld 8 Lilienfeld, 1980). First, the characteristics of the case and its environment and the diagnoses at the same instant in time may all be determined. This type of study is called a Cross-sectional study (or, sometimes, a prevalence study or a case-control study). Second, the characteristics of the case and its undronment may be determined at a particular time and then followed forward to a diagnosis at some future time. This tYPe 0f StUdY is called prosPective (or, sometimes, a cohort study or an incidence study, or a 1328::EUdinal study, or a forward-looking study or a followup studyl). :étE::::::iof the-multiple use of the term prospective and . . as......gifig.”‘.’§2.§?22”ii211.2”53:2”23‘E30;3:33.33353.i2.1:?L.2"“.. these terms and the invention of others (Feinstein, 1977). However,9 e . . . . Zetse the terms do describe a useful temporal distinction and because erms are still in common use, they will continue to be used here. r 0. 0:. I?! 38;“. 1‘2“ ‘: one: : arm-1:" :-. U 3'. $35!: 3Q; . :‘3 3.25 3:- he. " I‘.'In . 39. 8 pet-e 3‘ 29131;; 3.: e :‘ .::a‘ De: 1‘! sen-it 33.. : :a;.:s:c ales xxx“ Or a; :e ..l}. . .. gm: 3‘0 Kim is V6.2 O. . -. II 3‘ . e'ln. \ ¥es C‘a- . I ayte' s.- e :3 at: ‘ t. ‘p ode 2'... 3 ha 1;... ‘ ’Ju‘ 27 Third, the diagnoses of the cases may be made at a particular time and then followed backwards through existing records to previous case and environmental characteristics. These studies are called retrospective (or, sometimes, a case-control study or a case-history study). For all the confusion here, we see just one dimension, namely the temporal difference between the determination of diagnoSis and the determination of personal and environmental characteristics. Clinical research has an analogous design parameter centered around the second part of the decision making process, D -> T -> O. The diagnostic and treatment characteristics could be measured before outcomes or after outcomes (Feinstein, 1977, chapter A). Note that treatment and outcome cannot be measured simultaneously because the mnxome is viewed as a result of treatment. The other dimension that is used to describe both clinical and ”fidemiological research is apparent from a closer look at the two parts 01‘ the model, P,E ;> D and D -> T -> O. In either case, the research is ehmYs concerned with particular characteristics of the situation. The goal of the research is to relate differences in these characteristics U) differences in other data. The original differences in Characteristics can be defined in one of two ways: by nature, which r”Mite in an observational study, and by the researcher, which results '" an experimental study. The second design dimension, then, deals with how the original differences in grouping are defined, naturally or exPerimentally. e‘a" a 0 i u .0 IA :1“;:tai. 1.2.. Efi' ’.i c‘ :1 - .159;Cmming 3‘: (a ‘3 i) (D It) (i a. one ~v~$S-se: 9")“ 'C‘rg “he Dc]- 28 Unfortunately, the two dimensions just described are not orthogonal, i.e., not all combinations are possible. Actually, the lnajority of clinical and epidemiological research studies can be placed in the following categories: i. epidemiological studies cross-sectional collection of observational data - personal, environmental, and diagnostic characteristics that exist at the same point in time for the selected cases are determined retrospective collection of observational data - the cases under study are grouped on the basis of diagnosis and then data are collected from existing records on the preceding personal and environmental characteristics of interest prospective collection of observational data - the cases under study are grouped on the basis of personal and environmental characteristics of significance and then the cases are followed up until some time in the future when diagnoses are determined prospective collection of experimental data - the cases are randomly assigned to some level of personal or environmental characteristic of importance and then followed up until some future point when the diagnoses are completed clinical studies retrospective collection of observational data - the cases under study are grouped on the basis of outcomes and then data are D... ’0‘. \U i§u~ t'eat'e“ fires: in 3'35321'. SZJCv 3'! Tc"av: PA “en. U u “ L ‘- v‘ \‘ ...Qs C.‘ .r K '- . u ‘ ”ape” ' 4‘.‘ \‘er h ‘:"3a-‘- . .I (”an :‘V'fl \ ‘. ~h , es. s! I l “a. U ‘A “‘ ‘ litI .,. , ..:2 VJ, 35, 4 I ' 1 ‘Q‘f . .K '%We 29 collected from existing records on the preceding diagnostic. treatment, personal and environmental characteristics of interest b. prospective collection of observational data - the cases under study are grouped by naturally occurring treatments and followed forward until some future time when outcomes are determined c. prospective collection of experimental data - the cases matched on relevant characteristics are randomly assigned to some level of treatment and then followed-up until some future point when the outcomes can be determined 3. combination studies can be run that simultaneously address many of these issues Although the two design dimensions just described account for most clinical and epidemiological research studies, another dimension has been added in recent years. With the expanded emphasis on the thinking Process of physicians, the effect of the analyzed data on the thinking processes of the practicing physician is a third design parameter. This third parameter leads to two different approaches to research. The first aDproach ignores the thinking process of the clinician. It is the traditional approach represented by research from Graunt in 1662 and Snow in 1853 to Collen in 1973 and the Center for Disease Control in l977 (Lilienfeld 8 Lilienfeld, 1977: Collen, 1973). It consists of U)counting the occurrences of personal and environmental I’E'éfi!’ in: . . O ‘ ‘ Is'vv:s a he 3, .::':Oe:'q; :35? SC” 514:. 51.0 lSSJ-‘QS I“ on; cam on :~:‘ ‘ 'ete": y it. hm pee "\ no 0'. “I; :. :‘3'39‘ ' . . 5.! | t . CS .QSQS . ’e “is. 7:, :‘v‘ "3 S". ‘g . The ‘Se the v .p e.‘ of N we .2! seccnc "eel: ? 30 characteristics, diagnoses, treatments, and outcomes across different groups of cases, (2)comparing these counts across groups, (3)interpreting the results, and (h)publishing the results of the comparison study in appropriate professional journals. This type of study assumes that conscientious practicioners will adapt their decision making based on the relative merit of decision alternatives (Roberts, 1978). In recent years the second approach that has arisen is concerned saith how the thinking processes of the clinician are affected by the iiatroduction of the analyzed data. This second type of study is represented by research like Wilson et a1 (1975). It consists of (l)counting -the occurrences of personal and environmental characteristics, diagnoses, treatments and outcomes across groups of icases, (2)comparing these counts, (3)interpreting the comparisons. (hlfeeding the interpreted results back to individual practicing Clinicians, (5)observing the change in clinician decision making, and (6)publishing the results. These two approaches are clearly very similar. Both are primarily ctcroncerned with how differences in case and treatment are related to differences in diagnosis and outcome. The major difference between the PNo types of studies is the added emphasis on the decision making Process of the physician as a systematically observed dependent variable in the second application. The aggregated integrated empirical data are fed back to the practicing clinician and changes in his/her decision a -.; 3’: my .a I.“ "l R‘\ ' 8.“ ‘1! g In t... ‘ I 31 making are observed and included as part of the research.* The distinction between clinical and epidemiological research should be clarified with the following examples. Epidemiological Research Examples. Epidemiological research examines the relationships among personal and environmental cfimaracteristics and diagnostic categorizations (P,E -> 0). Three types of’ studies are common: cross-sectional, retrospective, and prospective. iiirst, let us turn to the simplest study, the cross-sectional research. CROSS-SECTIONAL EPIDEMIOLOGICAL RESEARCH EXAMPLE In 1961, Ferris and Anderson carried out a cross-sectional epidemiological study of chronic respiratory disease in relation to Seigarette smoking and air pollution in Berlin, New Hampshire (Friedman, l980, p.93). Data was collected from about 1200 subjects through ihterview and physical exam. Some of the results of the study are sUmmarized in Table A. ‘ ‘---------------- * Reviews of recent epidemiological and clinical research journals show that the majority of studies are of the former variety. This probably remains true because it is assumed that if a reasonable decision can be identified among a set of alternatives, the conscientious clinician will Choose it. ¥ 312i ‘; .la'zicte wm'fifilec News "~C';a'e::es : H: ilfl'etns 3'“ i ia'etzes . 3' '5" 5 93': 3‘ e e. 5‘ S stuav ‘ ‘ ax’ssha‘i fir noise the s” :25: t1: . \. 't'M'LI's ~~ec s s. _ ..a.‘ A o g. I b 9.. e-ec ‘. ”on ‘si : r i .. 32 Table h Association of Smoking and Chronic Bronchitis Smoking Characteristic Prevalence of Chronic Bronchitis never smoked 15.02 ex-smokers 18.92 l-lO cigarettes/day 29.8: 11-20 cigarettes/day 3h.22 21-30 cigarettes/day £2.32 ‘31-h0 cigarettes/day 61.1% lil or more cigarettes/day 75.32 AJ‘though the data are not related by time, i.e., particular smokers were rust followed forwards to bronchitis and bronchitis sufferers were not fc>llowed backwards to smoking habits, the data seem to indicate a strong association between smoking and incidence of chronic bronchitis. Note tflwat this study was epidemiological because it examined the relationship ‘Df a personal or environmental characteristic (smoking) and a diagnostic Categorization (chronic bronchitis). The study was cross-sectional leecause the smoking and bronchitis data were all characteristic of the ctases at the same instant in time. RETROSPECTIVE EPIDEMIOLOGICAL RESEARCH EXAMPLE After the cross-sectional form of epidemiologic study, the next Cc>rl'imon design is retrospective. A retrospective, epidemiological study ‘"as conducted from 1968 - 1972 concerning the relationship between 'hYocardial infarction and use of oral contraceptives (Lilienfeld 8 Lilienfeld, 1980). Data on the previous use of oral contraceptives were COIlected from a group of 58 women treated for myocardial infarction and a similar group of 166 women without such a diagnosis. The resulting data are given in Table 5. Inst 3'. :s : e a” B 0' (.3 3.‘ men I :‘tvm': a' .r :‘a' I .1 "' e at“ I ”is: c.‘ in: eiav:e.i C“ am. the 3, ‘e Q - . \! ’.°'D u. t I .- , ‘ . ".53.: a l I‘i it: 3.5: It Clay ""etie'ist' : 33 Table 5 Association of Myocardial Infarction and use of Oral Contraceptives Used oral Never used oral contraceptives contraceptives Group of women with diagnosis A02 (23 of 58) 602 ( 35 of 58) of myocardial infarction Group of women with no 20% (3A of 166) 802 (132 of 166) diagnosis of myocardial infarction (Zlearly, the percentage of oral contraceptive users was much higher for the group with myocardial infarction (AOZ) than for the group without myocardial infarction (202). Note that this study was epidemiological because it examined the relationship between a personal or environmental icharacteristic (use of contraceptives) and a diagnosis (myocardial infarction). The study was retrospective because the women were grouped ‘an existing diagnoses of myocardial infarction or not and followed backwards in time to a previous use of oral contraceptives. PROSPECTIVE EPIDEMIOLOGICAL RESEARCH EXAMPLE The last common form of epidemiological research is prospective. One prospective epidemiological study conducted at Yale University was designed to examine the relationship between Epstein-Barr Virus (EBV) antibodies in the blood and the occurrence of infectious mononucleosis (Lilienfeld 8 Lilienfeld, 1980, p.236). Data on the presence or absence 0f EBV antibodies in the blood were collected on a group of 362 incoming freshman. They were then followed for four years by researchers who were watching for subsequent development of infectious mononucleosis. The resulting data are given in Table 6. 135:: :51 in": ares:— ‘5" ""98 use“. I V' . ~8 ' ‘he 5.3m: -. l ,V ‘8‘. a; :u" A. ’UU::es i“. t“! . l l I, f) m m in, r e ate ’20.. ‘25: a 3A Table 6 Association of EBV Virus and Incidence of Mononucleosis Developed mononucleosis Did not develop within A years mononucleosis within A years EBV Virus present 02 ( O of 9A) 100% ( 9A of 9A) EBV Virus absent 152 (A0 of 268) 852 (228 of 268) (:learly, the absence of EBV antibodies was associated with susceptibility, the presence of EBV with immunity. Note that this study was epidemiological because it examined the relationship between a personal or environmental characteristic (presence or not of the EBV antibodies in the blood) and a diagnostic categorization (infectious inononucleosis). The study was prospective because the freshmen were grouped on the personal characteristic of the presence or absence of EBV (antibodies in the blood and followed forward to the future diagnosis of Mononucleosis or not mononucleosis. Clinical Research Examples. Clinical research examines the r‘elationship between treatment and outcome (T -> 0). Two types of 'studies are used: the retrospective and prospective. Consider first an ‘example of retrospective clinical research. RETROSPECTIVE CLINICAL RESEARCH EXAMPLE A retrospective clinical study comes from work by Childs comparing mortality rates of two surgical ,procedures for alleviating portal hYpertension (Burdette 8 Gehan, 1970, p.1A). Two types of portacaval sun‘s a,! $5 :in :c‘ ' :ztec - ‘c stat'st': ' lss: I! \- LPe , _ v ‘3 v‘ 3“ ‘Y h e Il‘:;e ‘ e u 5 e the“ H . \V‘S I ll!- ‘ 5 “ate. 3. a. e.‘ s. S S; s ~‘EEI 35 shunts were possible, one being side-to-side, the other end-to-side. The data collected relating mortality and treatment indicated that there was no statistical difference in outcomes between the two procedures (Table 7). Table 7 Association of Mortality and Surgical Procedure End-to-side shunt Side-to-side shunt [lied 22 7 Lived l07 30 Mortality 17% 192 chmever, on further analysis, the data seemed to indicate that the tsurgical procedure of choice depended upon how well the patient's liver \Nas functioning. Table 8 describes the relative mortality of the two Promedures for differing levels of patient liver functioning. Table 8 Association of Mortality and Surgical Procedure Stratified by Liver Function End-to-side Side-to-side shunt shunt “Ortality with minimal liver damage W 9% “Ortality with advanced liver 53% 36% damage The end-to-side procedure was preferred when there was only minimal 1iver damage; the side-to-side procedure was preferred otherwise. Note that this study was clinical because it examined the relationship between treatment (two methods of surgery for portal hypertension) and L suns-'1': . twang 131-. 3’ .3. S‘s-“t. ...-".. ”a V. \I'. “ "v- .. ‘ .‘e 5 pa... .0 . "a a. ‘ IS; _. I: c C 53. ”-5.“ '~ 3 ... .. a. g" . V ‘22: s. e:.e" ‘ v. ‘ss:>:'aI .I‘e .: .9 ‘W I ’H G U ..:I . “‘9: u. :5 F _I \ w u D . . \ a:;\'pe :, ' qt .5 "‘ E: 1 0 5‘3 3: “ I: I m :t. . . “CPe ulfl'.e‘ m bake § \- i :. ‘:EDCII a. it: ‘ be: e,” 36 outcome (mortality). The study was retrospective because it grouped the subjects with known outcomes of life or death and followed them backwards to the treatment they received of end-to-side or side-to-side surgical shunt. PROSPECTIVE CLINICAL RESEARCH EXAMPLE The second common form of clinical research is prospective. Prwospective clinical research is demonstrated by a study of the relationship between different treatments for moderate hypertension and patient mortality (Roberts, l977) . Veterans with diastolic blood pressure between 90 and llh were randomly allocated to a treatment group land a control group. They were followed for approximately 3.3 years. The data collected at that point are presented in Table 9. Table 9 Association of Drug Treatment and Mortality for Hypertension Dead within 3.3 years Alive after 3.3 years Tl“eated with drugs h.33 ( 8 of 186) 95.72 (178 of l86) Treated with placebo 9.82 (19 of 191.) 90.22 (475 of 191+) 111e relative risk of no treatment to treatment was just 95.7% to 90.2% °C l.06 to l. That does not seem large. However, the attributable risk ‘Nas 95.7% minus 90.2% or 5.5%. For every lOO patients, the treatment kept 5.5 more of them alive after 3.3 years. Note that this study was clinical because it examined the relationship between a treatment (drug 0r placebo) and outcome (mortality). The study was prospective because the patients were grouped based on treatment for moderate hypertension 0* drug therapy or placebo and then followed forward through time to the 2.2m 'es.'.'. ‘ :fia'azte' ‘sz' :5 21-22: in 'e )3“ Nil-rag . ..F,‘ 3‘ "PS 493'" eh i“ 0a, a: 2:53.? . - A. Vsio _’ 13' 3‘3 hazy a 37 outcome resulting from treatment, death or life. These examples typify the differences among clinical and epidemiological studies. Clinical studies focus on treatment and outcome; epidemiological studies on personal and environmental characteristics and diagnoses. They further demonstrate the difference between the retrospective and prospective approach. The retrospective looks backwards from the results of clinical decisions toward the factors upon which they were based; the prospective looks forward. One last characteristic of the clinical and epidemiological research paradigms deserves comment. Both areas rely heavily on the patho-physiological model of the human body which specifies how the var ious body parts interact and defines the most important observations / to be made. Although the research paradigms focus on the personal and environmental characteristics (P,E), the diagnosis (D). treatment (T), and outcomes (0) . an unlimited amount of information is available on any One area. It is the medical model of the body that specifies what are 1 ikely or important effectors of diagnosis and outcome. If not for this u"Kridel of process," the research paradigms might collect totally 3"relevant data. The adequacy of the model in representing the true h“than body is of prime importance. The model of the human body is also important for another reason, I‘amely. the transmission of any knowledge gained from research to other F3racticing physicians. Fortunately for medicine, most physicians agree with most parts of one model. Therefore, research that is based on the prevailing model of the human body can have great impact since the - '3'?! "are-cu "fit-CW. 22¢; 36; ‘t ‘0- 1.31" '3"! " a;;'e;a= *9 3‘ 152 cata £39.73 r in Plants. .a 3’! 33.. . ~ ~~ ar e ‘0 k». I. :a 38 general framework for the information is already set. Without such a framework, integration of research would be much more problematic. Imagine how much more difficult it would be to convince physicians that formally aggregated experience should alter clinical practice if a great deal of the data they believed relevant was not collected. When viewed from the clinical and epidemiological research perspective, it is easy to see why informal learning from experience is d i fficult. The learning that occurs is essentially the identification of Patterns in the data collected over large numbers of clinical encounters. In medicine, the amount of data that could be collected in one particular clinical problem solving setting is large; across Settings the amount of data is almost limitless. Even in reading diagnosis, the number of environmental, demographic and reading performance data is quite large. In either case, it is well beyond the capacity of the human mind's processing capacity unless the effect is eXtremely powerful and obvious (indeed, informal means of learning from QXperience are adequate for effects that are very large). Regardless of the research design, the clinical and epidemiological l" esearch paradigms prov i de_ as noted previously "possibly the only Methodological framework within which ...(science and rationalism in Medicine)... can be applied to the clinical management of individual patients" (Roberts, l978, p.l7h). '30.. we : a :- »l 21%: ‘cr 'el 15:65: I: I“: : 1‘: ".eg'at *; hin.' " . ‘ i 1’ ' a. a. ' 3 ' 54 39 lbs Clinical and Epidemiological Paradigms and Improved Clinical £355. The clinical and epidemiological research paradigms are formal inethods for learning from experience. They are the medical field's response to the problematic nature of informal methods for accumulating and integrating experience. They have proven to be quite effective in improving clinical problem solving in medicine. The success of formally aggregated and integrated empirical data in improving clinical care has been demonstrated often in the field of undeciicine (in both the research areas of epidemiology and clinical research. This success has been measured against the natural history of L3r11:reated problems and the previously defined treatments of choice. 'r11eese successes are documented in the following research reports. Historical studies abound. In 1760, Bernoulli supported the €3f“ficacy of smallpox vaccinations; in l835, Pierre-Charles Louis dEunonstrated the inadvisability of blood-letting as a means of tLl‘eatment. Budd conducted a classic study implicating contaminated water 3'1 the spread of typhoid fever in l873. All of this research was based <3rithe typical application of the clinical and epidemiological research Flaradigm - counting, comparing across groups, interpreting results, and "eporting. The research paradigm continues to be as useful today. According to Lillienfeld and Lillienfeld (l977), “the basic reasoning processes and logic, as well as many of the methods used today, were developed well over 100 years ago. Today's epidemiological problems and situations are not very different from those encountered in the last century prior to 0“ :ap:e' a‘ An ebl Ur- W "e we - Cub. w 533 x1 (3:: “ l‘ct‘e' . In ‘I 5 2! we 4‘. 3" 2212:: ‘ 3r 3‘ .c ‘8‘"! ..ac 3' ‘ e I b .‘n ' ‘ V “VS. ’3' 1‘9 fig. ’ 5" 05:;‘" I. I . a 3:. § 9831):; I". ' U .3-cs s a“ u :3..’a ‘ v :.e. .. u _ ...Q 33.. QM qu;.~ I! I 65.» Bet! 'S. _ h I CI k: ~. \U we i '5 ‘ 0 y :3: ‘Q ‘. ‘. H ‘I. by Ge a 3":32“ hO the Bacteriological Era.” Indeed, in 1977 the Center for Disease Control used the same methods to justify measles innoculations as was used with smallpox vaccinations 125 years earlier. Another recent research series based on the clinical and eepfidemiological paradigm is centered in the Kaiser - Permanente Medical Care Program (Collen, (ed.) , 1978). This program has been in operation since 196‘; and provides a Multiphasic Health Testing Service for early (Seetection of medical problems. An average of over 50,000 people a year raaave had multiphasic checkups. This extensive data base has been used in nnaany ways. For example, it has provided base line data for estimating tzrre co-occurrences of many patient, diagnostic, and outcome <:r{aracteristics. This base line data has been used in computer aided cai'agnosis and in the identification of high risk patients (Siegelaub, Fl'iedman, Collen, 8 Kodlin, 1978). The results of using the system have generally been a great reduction in cost with no reduction in benefits to the patient . Turning to the clinical area, another large data base of clinical Sixperience is available at the Mayo Clinic (Kurland 5 Molgaard, 1981). JSjnce 1885 the Mayo Clinic has kept careful records of patients and ‘their personal, environmental, diagnostic, treatment, and outcome 'characteristics. This formally aggregated empirical data base has been but to many good uses throughout its history. Most recently it was used to provide a base rate of the Guillain-Barre syndrome so that the supposed effects of the swine flu vaccinations could be determined. 2"» :a‘ rese s:.:‘es ae"c".'ec u‘ . 3‘. 735 392:" ” :'a;*cs's c‘ mew ; llt'lgg'egatec ac' :ecfsfon may. 512.3? 3'39: 0‘ a; 5333:? of 9‘ .‘1 3‘? matter-a1“: 35!”... Hor'OCK s . :tt‘l'ans Mn 4 ”‘f ‘. _ Jr “a. pa n! C1 it y he "'05: Se hl Clinical research has been quite successful elsewhere. One set of studies performed at the University of Leeds, Leeds, England (deOombal, 197A), has been concerned with the use of empirical data in the diagnosis of many problems in internal medicine. Clinical experience has been aggregated across cases and made available through a computerized decision support system. In certain areas, like acute abdominal pain. computer aided diagnoses based on empirical data has achieved diagnostic accuracy of 91.12. This is in comparison to unaided clinicians (79.7%) and computer-aided diagnosis based on clinicians' subjective estimates of associations of patient characteristics and diagnoses (82.2%) (Leaper, Horrocks, Staniland, 8 DeOombal, 1972) . For some specific dec 3 3 ions such as discriminating between appendicitis and non-specific abdc>minal pain, computer-aided diagnosis reduced decision error from 13% bY the most senior unaided clinician to 22 with computer support. Not o“ ‘ 7 did the error rate decline when computer support was used. but the Qua} ity of decision making by the clinicians using the system was ‘33 stinctly improved. During the time in which they used the computer, the decision errors in the clinicians' diagnoses without computer 333 i stance were lessened by 34% across all patients. In addition, the pr°bortion of appendices that perforated was reduced from 362 to 1.2. nfOrtunately, when the computer aid was removed physician performance deg! ined back to the original levels (DeDombal, Leaper, Horrocks. tahiland, 5 McCann, l97h). Clearly, the computer support system based on empirical relationships in the data was able to identify reasonable dgcision alternatives more effectively than the human clinician could “8 thout the formally aggregated data. “.ese $1.: war has :3- - . c .31. .‘l 3:1. :0 :rei'IER: V: .e 5.3%: 5'.” : 152215 93; a' M Q. . o r. :efit'ry '5: fitrtathee. . 0‘ 3‘: : ul :USS‘O,‘ 5?. I. :3fis' ‘ Y » LO re (:4. . lugs: f. 3C4 t , “4% ’9 , Ipi tn. ..‘ (:5 Is hZ These studies demonstrate that clinical and epidemiological research has contributed significantly to the improvement of medical ' care. In fact, for the world population as a whole, it is the knowledge of preventive medicine gained from epidemiology and the knowledge of improved surgical and pharmacuetical techniques gained from clinical research that are the main sources of increased life expectancy in the 20th century (Coleman, 1977). Based on these data, it appears that the formal aggregation of esnpi rical data through the application of the clinical and epidemiological research paradigms can be used to determine and improve the effectiveness of clinical problem solving according to a particular MOdel of clinical) reasoning. Now let us review the clinical problem Solvi ng field to which the clinical and epidemiological paradigm will be app} ied, the field of reading diagnosis. W... Mam—eat. mi. ____0utcomes 1.2 ___Reading The Nature o_f Diagnosis 931 Treatment i_Q Reading. If the diagnosis a nu treatment of reading problems were totally clinical in nature, then t h§ discussion here could be quite brief. Most of the concepts from the e(iical model (which is itself primarily clinical in nature) would apply qi"ectly to reading. Unfortunately, reading diagnosis and treatment QCCur most frequently in group settings. In addition, reading is unlike rueClicine in that there is no naturally occurring condition called 'I healthy reading.‘I A problem reader is generally not someone who has gm‘m irea :' as: wash; a test any! 1.: :e‘:'e ‘90, n‘ ‘ $8.. ”I; or ..He “*9 CNN: ”312? :m' :9 235:. 3.“. - a, b ""1 'Omhtal c (3;.‘05‘5. 5-4; ’. . Icfgev tC .m' u “If a '1‘ '1” tne D! E.’ {0,7531} .a. fig (WEN ,. \ =’e ., . brmarliy 351: .t , O. the g ‘55.?“ ”a U1 CE a d6. '3’!" Oh . .Q C] ii I" .u 1.3 gone from "healthy“ adult reading to ”poor“ reading, but, rather, is one whose reading ability has not progressed as fast as "expected." Both of these characteristics complicate the reading situation significantly. But before looking at the complexity, let us first consider an idealized setting for the clinical diagnosis and treatment of reading problems. The clinical problem solving model described earlier in this chapter detailed an interation between a single clinician and a single case. During this interaction the clinician collects personal ahd env i ronmental characteristics and performances in order to make a diagnosis, suggest and carry out treatment, and observe outcomes. As outcomes are observed, it is possible to recycle through these decisions in order to improve the diagnosis, treatment, and outcome. The Current State o_f_ Diagnosis _an_d Treatment fl". Reading. In reading, the personal and environmental characteristics are collected ei ther formally with standardized tests or informally with an informal Peading inventory based on school texts (Spache, 1976). The tradeoffs arg primarily the (reported precision and reliability of standardized ests for the greater validity of the tasks one can setup in informal In §33ures. Once a data base of case information is available from which to WQ"k, the clinician can formalize his/her diagnostic impressions by w" i ting a diagnosis (Ekwall, 1976; note that the question of when diagnostic categorizations are made is ignored here). Ideally, as :es:":e: ea" .-'e;-:s's 5-3." \ xss':’e. : v 9! ‘ .3. be c :2: sets 32:2 :seri'a: . :ha' a" ‘. . ' {3“3: HA :iei‘CSes. Eve- n“ , ' 3 :h' :e' 53:23:32... e at: 0‘ {...n I v.5.a‘fsts. F \ sine re”- 5 .‘I ‘ 5 ‘ n 'gi . 0" III I 3:. s. 'Qa ‘ C'r- dEH .. ‘ ‘ ~31 .6 ‘U H's . eb‘:$‘~. . ‘i!a I. .ly i‘r. ‘- ‘ Sci, ‘5?“ - A,' ‘u la“ ‘1 I O #3 . s: ah‘o 3" S; k .. 'u. l .. g. hh described earlier, the purpose of diagnosis is to determine cause. Diagnosis should be reliable so that valid aggregation of experience is possible. For the diagnosis of reading problems, reliability of diagnostic decisions appears to be quite low. The results from a series of seven Observational studies run over five years indicate that different reading clinicians diagnosing the same child arrive at totally unrelated Ciiagnoses. Even more disturbing, the same clinician diagnosing the same <=hild twice gives very different diagnoses on the two occasions (Vinsonhaler et al, 1982: Gil, 1979; Hoffmeyer, 1980; Stratoudakis, 'l980; VanRoekel, I982; Weinshank, 1980). These results held across many samples of educators who dealt with problem readers (reading Specialists, classroom teachers, and learning disabilities specialists). From the review of the clinical and epidemiological paradigms given earlier in this chapter, it becomes clear that such unreliability in defining the state and cause of reading abilities makes valid learning from experience very difficult. These results on reliability are distressing considering that, in fact, reading specialists are asked to perform tasks essentially identical to those posed in the studies above. The one argument that is (occasionally put forth to belie these data is that the medical clinical loroblem solving model should not be applied to reading; it is inappropriate there. In a typical medical situation, a physician makes the diagnostic and treatment decisions at one point in time. The diagnosis leads directly and differentially to treatment. In reading, a clhfician interacts with the case over an extended period of time, 323‘. :‘I hang a '1‘":- u‘a‘. tag“: ‘3: state: my. 'BSII; :1! the :3 J l.‘.nv v‘ ' 1": LOP DE 5'95 "Ia: bl I‘ee ”It”? '9” .m1 fi:x':.qa:-b :.e-m"lcn . a 0. 5.3.. ..at :Etlrt': .. he Habit} v .- " “”3 of \ 'f ' ”‘53,“..- ' '3 er a“ ‘8... “ . al to 15353.. “5 possibly making a traditional "medicine-like" diagnosis unnecessary. Even what diagnoses there are do not lead anywhere. Spache (1976, p. h36) stated that: "Despite refinements in diagnosis and remediation in reading in the last twenty or so years, there is still a widespread lack cof' integration between these two processes .... In many instances, it seems that these two processes are carried on by different persons IDIstween whom there is a distinct lack of communication .... the irncoordination between diagnosis and remediation is a relatively common F>i1enomenon, as we shall attempt to show.” These differences are meant to Show that the diagnostic perspective is inappropriate for reading and ‘tlne unreliability of clinicians as they make such decisions should not be of concern. However, this argument is in direct opposition to the Opinions of virtually all the experts in the field of reading. \Iinsonhaler and colleagues (1982) cite study after study that clearly indicates that the cardinal principle of reading literature and practice is that remediation must be based on diagnosis. Reliable diagnosis is Icentral to reading instruction. Therefore, the unreliability demonstrated by the practitioners in reading is a major concern. Once the diagnosis is complete, treatments are chosen and iadministered. The treatments for reading problems are almost limitless 7 n their variation (Spache, 1976). A few of the dimensions along which tzhey vary include (l)the assumed sequence of skills needed for reading, (2)the different forms of technology applied, (3)the methods for shaping sstudent behavior, (h)material content, (5)individual or group carientation, and so on. Any treatment, then, combines various elements on all these dimensions and more in a way that yields the wide variety :‘ mac‘s; rut-- We 'as’. :a': 3151813110g- 3‘ .. 1132'. v“ lttc': zl _ i. 3:. “ ‘| Jan 3 1353"“! c' . :e' geacfin |.‘ N ‘ ‘ :He' . . Na" D'Oben Wt. h6 c>f reading treatment possibilities. The last part in the clinical problem solving process is the determination of outcome or change that occurs after the introduction of tJ'eatment. According to Spache (1976, p. 273), variation (Spache, 1976). "F’ractically all reading programs, methods of reading, and experiments Latilize gain on reading tests as their criteria of success." These are rfieasonable criteria. The student performances on some tasks were the taasis for the original diagnosis. As the student's problems are f‘esolved, the performance on these same measures should change to f‘eflect this.* Reading Instrgction lg _55 Classroom. Now, into this manageable clinical problem solving environment, the realities are introduced (Duffy, l98l; Jackson, 1968). The fact of the matter is that little professional reading instruction is conducted in the idealized clinical problem solving setting. Instead, most reading diagnosis and treatment occurs in a group setting. Besides the clinician and the particular student being helped, many other students are also in the room. These iother students add a degree of complexity to the situation that may substantially alter the characteristics of the instructional setting. The primary difference between the ideal clinical setting and the r-ealities of classroom settings is the number of interpersonal * Many of the measurement tasks for reading include connected text that a student might easily remember. Therefore, alternate forms of measurement devices may often be required to determine outcome (Borg 8 Gall.1979: Calfee 5 Drum, 1979). "te'act'rs :‘a m 's a owe-3' :5: met. :~ E’v‘er-mawy ~‘1‘: :73! 49 85,” iag‘cs's. mm m .._ O ’ .e°.s:“:e' '2“?! H If! at ..f. '3‘“ 1." Ev e 113119;; I‘a' 3:2,.3: ‘CE IQ] I . 1' UUa lze: "“3"”. c 9‘35». v, "- 3N .' t’Fs ‘y lh 1 a. :ch . I ::- _ C. 1c.” “311 the A‘ 1 \la'i 9.“. 'h \- «C_g' [" ' I \ be p 5’s... .\Ia‘. . ‘- “7 ir1teractions that can be taking place. In the ideal clinical setting, timere is a one-on-one interaction. The clinician can directly respond to tJae student, the student to the clinician. The classroom setting has a many-on-many interaction. At the first level the clinician must somehow ciivide up his/her energies across many students, each reguiring a (diagnosis, treatment and followup. Just as important, however, are the Student-to-student interactions. There inter-student relations can i nterfere with other clinician-student interactions or facilitate them. Elecause they exist, the inter-student interactions change the range of “treatments that the clinician can administer to the individual students. In general, as the number of students in a class increases, the amount caf a clinician's energies that must be spent on management and control c>f 'inter-student interactions increases and the amount spent on individualized treatment decreases. At some point, little of the clinician's resources are spent on any type of individualized treatment and a standardized, non-discriminatory treatment with accompanying management techniques is the rule. Clearly, the range of treatments that a clinician can' apply are greatly attenuated by classroom size: one-on-one gives total individuality in treatment: one-on-many gives uniformity in treatment across differing problems; one-on-few may be a good real-life compromise. At this time, research reported by Duffy (l98l) indicates that most teachers generally take students through <:ommercially prepared materials providing only corrective and affective ‘feedback. ”It is difficult for them to remember that they are supposed to be cognitive information processors who make individualized differential instructional decisions on the basis of rationally-developed hypotheses (p. 11).” 9.1 " :‘I S '23” 1‘ a: .2139'1': b u u :‘m. the ::* reams ci‘f'c. ere';v. and s:..:« It! fnc’vimi Tze genera'?y ccgaii'v t: ":9"?! I11? .-r es over g' f‘g l‘ g . _' ‘n. {amfifits tea ”N Suntan" if tfiese co: as“ . _ ’ ' 5Q~I:\gs ’te' ' It '71 the . . 1‘8» 35.: ' in E [sz s e . y ||. s}!&'. I Ieh I f} i r‘s ‘ ‘elfitawgi . 3' In; 1 .Eco.‘ \Ps .,.- '5 ‘9 i- Q‘ \I T‘s; I 3::- , I . \ r 4 1‘): “En " U S.\ Ila” A8 This reality of reading instruction makes applying the clinical and epidemiological paradigms to reading more problematic. In the first place, the constraints of classroom activity make differential treatments difficult. The clinician has a limited amount of time, energy, and student cooperation and usually, therefore, provides less tJ1an individualized attention. In addition, since reading treatments are generally cognitive in nature, the social demands of other students tend t1: interfere with what treatment there is. Finally, as classroom size \haries over different clinicians and different settings, the total range C>f treatments that are applied increases, requiring more data collection to make substantive comment. If these complications are not enough, the added complexities of group settings make it difficult to define what treatment the students receive in the first place (Jackson, 1968; Good 8 Brophy, 1973). Clinical and epidemiological research require reliable diagnosis and differential treatments. But the factors that make treatments different are extremely interrelated and difficult to assess. Fortunately. research in instruction and reading have provided several models for <=isentangling this complexity. [Ectors Affecting Reading Instruction. Several models exist for i nterpreting the complexities of the classroom and reading instruction (Bloom, l976; Carroll, l963; Gagne, I977; Fisher, Filby, Marliave, Cahen, Dishaw, Moore, 5 Berliner, 1978). One model of individual learning that suggests the significant aspects of treatment and has had l gvea‘. :ea‘: c‘ “-9 1m" mm R'CVI'Q 333". lSIJCEfi'. to '53.: ‘3' IISI'uct'ch : Eider: 3C!.a"y 339mm ”c it“ :‘Q at” I| | pa ‘ 3' ‘3' In. '. 'F'hre: I '1 h“ in I “9 a great deal of influence in education is the model by Carroll (1963). The Carroll model defines five factors that affect treatment aimed at inuaroving a particular task. These are the amount of time it would take a student to improve under ideal conditions (aptitude), the time allowed f<>r instruction by the clinician, the part of the allowed time that the Student actually uses (perseverance), the student's ability to IJtmderstand the treatment tasks in which s/he is to engage, and the quality of instruction compared to some theoretical ideal. Variance in these factors is variance in treatment. The complexities of the classroom are described by other heuristic Models. One by Dunkin and Biddle (1971.) details the factors that affect 1:reatment in group settings. These factors include everything from <2haracteristics of the clinician and the group environment to entering pupil abilities and classroom activities and materials. So many, many factors can supposedly affect the treatment that it raises questions about the ability to define differentially effective treatments in reading. But, of course, the heuristic models not only provide a means ‘for interpreting the complexities of classroom treatment, they also provide a framework for observing and recording the significant aspects (of actual instruction. Measuring Reading Instruction. With the great complexities of treatment, any clinical research in reading must be very careful to measure the most appropriate factors in treatment. Coding forms for the measurement of group instruction abound. Simon and Boyer (I967. 1972) lave cents: a I‘HCZE'IS of 3 a':e"a:'ves :e xasre 3153;; "34:3! 0‘ :‘2'5. He :8! I i l 3- 5:835 ‘ z! “55"“. It: I 7"" “We"; v ".3557 this 50 have developed a catalog of observational instruments containing dozens arud dozens of different alternatives. Unfortunately, the dozens of .al‘ternatives tend to be merely variations on a theme. Some instruments measure instruction in terms of clinician leadership style, enthusiasm, ciifficulty of questioning, etc. (Brophy 8 Evertson, 1976, p.17h). ()thers, like the Brophy 8 Good Dyadic Observation Scales (Good 8 Brophy, '1973) emphasize the quality of the teacher-student interaction in so rnany ways, yet fail to examine how the content of instruction is related 'to the student's instructional deficits. The realities of a classroom Iaave apparently focussed most of the attention on non-content concerns. F’ossibly this is most appropriate because content does not vary significanly within one instructional group. However, as Peterson and \Malberg point out (1979, p.183) ”Research on teaching appears to be at the threshold of fundamental conceptual reorganization .... First, there has been intensified interest in student variables .... Second, there has been vigorous study of cognitive and individual differences within an information-processing framework." This is precisely what clinical and epidemiological research would suggest. The individual differences among students, i.e., differences in diagnosis, should lead to differences in treatment and outcome. This change in emphasis is spurred on by some of the same factors that originally turned the research focus to teachers. Studies by Coleman (1966) and others suggested that teachers made no difference \when they used “school" as the unit of analysis. Clearly, ”school" lumps together more and less effective teachers to give uncertain results (Brophy 5 Evertson, 1976, p. 9). The clinical and epidemiological was 5.;35’ ‘aszer an: {My msres cf yea... -.". "er-3:1" 95a .2: as W :hI ’I 5. :SQ'V atign '88 £31251 ’. the N. “c 0:- ‘Ps ‘4‘. s , B I‘: ..r,' We}? V 51 paradigms suggest that looking at individual teacher and using "class" as the unit of analysis is also too gross a level because it combines faster and slower students to give mixed results, i.e., inadequate measures of treatment effectiveness. All these concerns combine together to make treatment observation 3 Potentially difficult task. Observation must attend simultaneously to 'tlwe concerns of the Carroll model, Dunkin and Biddle's heuristic, and )fet draw on a relatively homogeneous and narrow set of observational i nstruments. Thus, clinical research in reading may be much more F>roblematic than epidemiological research. The mechanism by which these IDroblems with treatment observation are alleviated is the model of case l>rocess. As Stallings (1977, p.19) points out, "Before choosing an «observation instrument, you must answer the question: What do I want to learn about children, myself, or other adults participating in the classroom?" The model of case process provides the answer. Models 21 355 Reading Process. As the activities of diagnosis and treatment in reading have been explored, one issue has been ignored so far, namely, the model of case process. Clinical problem solving iactivity is centered around a model of how the case works. This model sspecifies the data that is significant and the expected relationships among the data. Such a model of the reading process is needed. lJnfortunately, locating a model that is widely accepted could well be difficult. “If ‘t'ai'or of 1'51 '31 to tear I) the ans-e. ' H T, 0:“ Dhys C . a'WI ‘1\\ fig mes! “I?“ ' s! ”Ouic 3“”. 8 ' quail n: '93:» ..5:.~:' s ' sit :. 11 If . e u i ‘N: 22:2: Vie. . s. .4 \‘J 52 Chall (1967) highlighted the problems resulting from the proliferation of various reading models with her comment "What is the best way to teach a young child to read? No two people. it seems, agree «or: the answer.” (p. l). Apparently there are many competitors in the f ield of creating models of reading (Singer 5 Ruddell, 1976: Davis, 1971). Yet the cornerstone upon which the successes of clinical and epidemiological research in medicine are built has been that of an accepted and stable model of the human body. The anatomy and physiology <>f the human body is a widely accepted model that identifies significant fliedical data and their interactions and that serves as a basis for caissemination of information to practitioners. Without it, results across physicians would be largely noncomparable. In reading, with <:ompeting models that can be substantially different, such guidance for research and practice has not been possible. One would hope that some compromise would be possible and an all encompassing model would evolve. However, quoting Chall again, ”The overriding impression was one of strong emotional involvement on the part of authors, reading Specialists, teachers, administrators, and, unfortunately, even researchers. Their language (defending reading models) was often more cmaracteristic of religion and politics than of science and learning." (9- 7). The Model 21 Reading and Learning Lg Read (MORAg). However «:ontroversial the various models of the reading process are, some model must, nevertheless, be chosen to guide decisions during clinical and .I Q” I Ap‘oa‘ stir-'1, - m ‘s t“: :a: ma v.3", : .z 3 “‘I P. a "I s 9.. “F. :‘(.q . .‘ we: {a a4 . I '25:? 3!. I a" ..l! to Q.- ‘ ..ab #3 ‘ ‘4‘ I 1323‘, A synce. lg“.1 .5“ N'fa ~.‘~. ‘0; ‘. emat '1" " In. 51‘}. I“ up p 1:. “ch . 9 Is ., 1‘s. 53 epidemiological research. At an initial stage, the true key to such a model is the data that it deems significant to collect. When a study is actually run, the only point at which the model is operationalized is in data collection. Therefore, a model was desired that had high face validity' in the depth and range of reading related behaviors that it valued. The model chosen here was the Model of Reading and Learning to Read (MORAL) developed by Dr. George B. Sherman and expanded by the Clinical Studies project of the Institute for Research on Teaching (Sherman, in progress: Vinsonhaler, Weinshank, Cureton, 5 Blatt, 1980). This model includes the definition of processes, factors that affect them, and instruments for measuring them: the requirements for a good information-processing model (Calfee, I981) . The original model defined by Sherman was the result of many years as a reading diagnostician and college professor of reading. The model, designed to assist with the diagnosis and treatment'of reading «problems, describes a series of Critical Performances that a "healthy" reader must be able to demonstrate together with the concurrent cognitive skills, personal and environmental factors, learning history and learning skills that would effect the state of the Critical Performances. Rather than be Overly concerned with the way learning to read took place, Sherman defined what a good reader should be able to do and then watched for factors that seemed to affect these abilities. One result of this Orientation is that the face validity of the Critical Performances for reading is very high. Research that collects these data will, at the very least, be collecting much important performance data. he :‘f's; 'esea'fl' in :‘e :e:**':.es .‘:' "eC"n.’:a' <1 :‘Kessa' 3‘ IC'RSZJi 32cc Iratesses :- "35mm :czr. I Wt’ "MI-'5 as h: '8'" "5 cast ‘lv. OSIOCIs 3' by.“ a-"rc s O u 5' reac; :‘Qg '. “310$; D ,4‘ ‘ a"mic. s. \ 5h The Clinical Studies Project expanded the model with relevant r'esearch in the field of reading and with descriptions of measurement 1:echniques for many of the significant data highlighted by the model. The Clinical Studies version of the MORAL conceptualized the reader as a parocessor of information. Any information processor can be cronceptualized as an entity that l)receives input from the environment, 2)processes this input in conjuction with its own memory of past events and 3) Produces an output that impacts its memory, the environment, or both. A particular reader attempting a particular reading task, then, treceives as input the requirements of the task. This input, together saith his past knowledge of reading and language, are processed in some vmay. Outputs are produced: some are not observable (e.g., changes in memory) and some of which are observable (e.g., performance on the reading task as measured in a particular way). Six reading and language performances are considered critical to ‘the health of a reader. To the degree that these performances are inadequate, some tasks in the range of reading situations will be deficient. The six performances are: l . Instant word recognition performance: defined as the ability to recognize a certain set of words instantly. (Recognize means to associate the visual stimuli of a word with its internal prototype in the mind.) 2. Decoded word recognition performance: defined as the the ability to recognize a set of words using various association strategies (e.g., sound symbol association). This recognition is not immediate and tales 942'! W‘c'aru l \ s 3- i3: 3'. 3'1: " 3’33 8.63:! N!!! ' be cc: Y ‘ O-. " 3' 2,“: Re 3‘5"? sac " ~'5:Qfil“e list-er s- 55 takes more effort on the part of the reader than does instant word performance. 33. Vocabulary performance: defined as the ability to give word meanings Ls . Oral Reading Performance: defined as the ability to read text aloud correctly with appropriate phrasing, fluency and intonation. 5. Silent Reading Comprehension Performance: defined as the ability to answer specific questions on text read silently. 6». Listening Comprehension Performance: defined as the ability to answer specific questions on text read aloud by someone else. The MORAL goes much further than just the specification of the Critical Performances. For each Critical Performance the MORAL specifies 'the associated effecting factors, that is, the student and environmental ‘Factors that affect the performance. For each performance, in turn, a Set of learning or change processes is defined. These processes describe tflwe significant aspects of learning history in reading that affect the \Iarious reading performances. The very pertinent concerns of attention and motivation are viewed as effectors of the Critical Performances. The MORAL, then, is the model of the reader that was used in the research reported here to define the significant data that needed to be collected 1“I"om the subjects in the study. ".1330 J {M‘- an: F :“ecfive a". inc-1pc: 333 ° may?“ v .eapeseh‘ eve been U! 56 Summary _o_f_ Related Research Clinical problem solving is a general mode of human thinking that i s widely used in many content areas. Unfortunately, the generally available method for determining and improving the effectiveness and efficiency of clinical problem solving, namely informal aggregation of experience, is inadequate. The structure of the clinical decision making tasks and the biases of the human problem solver do not lend themselves to valid informal learning from experience. The formal methods of the clinical and epidemiological research paradigms seem to be a potentially effective- alternative. They direct the explicit recording of clinical experience apart from the human mind's biases and faults; they emphasize the comparison of such records across cases and clinicians to avoid non-representative or narrow experience. Where these research techniques have been used in medicine, they have proven quite effective. Adapting them to the field of reading diagnosis and treatment seems to be a Straightforward generalization. This field has been shown to suffer from some poor clinical problem solving performance on the part of its Practitioners. With a recent model of the reader available, it becomes an excellent area in which to examine how well the clinical and epidemiological research paradigm does generalize to other fields. 1’" als'ega: i c. Iii! Dim/en to 2 CHAPTER 3 Methods and Procedures Objectives The purpose of this thesis was to apply the clinical and epidemiological research paradigms to the field of reading diagnosis and treatment. Tflwese research paradigms were developed in the field of medical ciiagnosis and treatment and have served there as effective guides for tlae aggregation and integration of clinical experience. These techniques Iaave proven to be conspicuously better than the traditional informal tneans for learning from experience. It was hoped that generalizing these IDaradigms to reading might point the way toward more effective use of <:linical experience in reading to improve the quality of care received by problem readers . The specific objectives were as follows: 1 . to determine how to apply the medical clinical and epidemiological research paradigms to any other clinical problem solving field in general, and to carry out such a study in the field of reading diagnosis and treatment in particular. to evaluate the results of the clinical and epidemiological research study in reading with respect to further understanding of reading diagnosis and treatment and the use of the clinical and epidemiological paradigms 57 R assc: Char. 31 355: C“ar afia‘ “l Drcc Cha- t° ex c“B'ac: H the I}, 2} as: HI 3; as wi M D f) 58 to examine the epidemiological data of personal, environmental, ‘ and diagnostic characteristics for l) the nature of reading diagnosis and baseline likelihoods of various characteristics with cross-sectional data analysis; 2) associations between diagnoses and personal or environmental characteristics with retrospective data analysis: 3) associations between personal or environmental characteristics and diagnoses 'with prospective data analysis; h) problems with the definition or measurement of the various characteristics or the use of the epidemiological paradigm. to examine the clinical data of treatment and outcome characteristics for l) the nature of reading treatment and outcome and baseline likelihoods of various characteristics; 2) associations between outcomes and treatment characteristics with retrospective data analysis; 3) associations between treatment characteristics and outcomes with prospective data analysis; h) problems with the definition or measurement of the various characteristics or the use of the clinical paradigm. V We 'm ": ‘ps: o:.e::!i_,e Eciacsgca. :'355 so V an 3 smug-ca- “: H ”I. '5: :39 Cfln'u; The remainder of this chapter describes the achievement of the first objective, that is, the determination of how the clinical and Tepidemiological research paradigms are generalized to any clinical problem solving field in general and how the particular clinical and epidemiological research study in reading was performed. The purpose of this division is to provide a guide for other fields that may wish to IJse the clinical and epidemiological paradigms for learning from experience. The description of general procedures details the techniques I~equired by the paradigm and their importance. The specific procedures thed in reading provide a precise history of one attempt to use the paradigm outside of medicine. Clinical and epidemiological research evolved from the clinical problem solving setting, and many of its requirements are dictated by the decisions that must be made by clinicians in such settings. One of the first prerequisites is a knowledge of what is significant in the health of the case and what is not; what is foreground and what is background. This knowledge should be as widely accepted by the practitioners and researchers as possible. It can be conceptualized as a model of case process. 1; Procedures for Defining a Model 2: Process Clinical and epidemiological research require a model of case process, i-e.. an ”anatomy and physiology" for the field to be studied. For any particular clinical problem solving field, the following procedure will assist in the creation of such a model. :52 senator-l mate 3!“; occue‘cas‘i ‘ I o. ' at. 3.! :. 2- 32‘ine ‘. . ‘WC: CH ' Gr 56:: .\ “3 am. V. General Procedure. The purpose of this procedure is to assist with the development of an operational model of case process. The more accurate and precise the description, the better. But a useful model may be obtainable even from general information. Complete the steps below: l. Define the type of system to be diagnosed, treated, and followed up. 2. Define the set of performances that are critical to case functioning. 3. For each Critical Performance, define any concurrent case characteristics that could affect the performance. Estimate the likelihood. direction, and severity of each effect. h. For each Critical Performance, define any past personal or environmental characteristics that could affect performance. Estimate the likelihood, direction, and severity of each effect. \ 5. For each Critical Performance and concurrent case characteristic. define sets of actions that can be initiated by a clinician that will alter the performance or characteristics. Estimate the expected change for each set of actions. 6. For each action that can affect Critical Performances or case characteristics, list all case, clinician, and environmental factors that can attenuate the effect. Estimate the likelihood, direction, and severity of each effect. 7- For any of the effectors of Critical Performance or case characteristic, describe the cause - effect reasoning of the '. 3e’t‘le e 0" mm 3. Ire'aze stao‘e. R 9"...” vvvv In: . . . ..y. ~l C: e: <4'Eag :‘Y‘Q ‘ \u s .e mg( 5553' E talk; ' e ~19 h .5 (t u “ S.q:e H b~'s‘ "lar- 6l association, if possible. 8. Define expectations of Critical Performances for different classes of systems. 9. lterate through the previous steps until the descriptions become stable. Procedures 13 Reading. For the research in reading diagnosis and treatment described here, the Model of Reading and Learning to Read (MORAL) created by Dr. George B. Sherman and expanded by the Clinical Studies project of the Institute fothesearch on Teaching was used. This model, as described in the related research, was created over many years of careful clinical experience in reading. It includes all of the required components, i.e., the Critical Performances and their effectors, the improvement factors and their effectors. ll. Measurement Device Creation Procedures Once the model of case process has been defined, the next requirement is a series of measurment devices for all of the Critical Performances of the model and as many concurrent and historical case characteristics as possible within cost constraints. These measures will be used to deduce the state of the case's Critical Performances and changes in these performances after treatment. I. in £5113,an ‘A ' O Vr i’.' Ital '2}. fifth 62 General Procedure. The purpose of this procedure is to assist with the development of measurment devices for the Critical Performances as defined by the model of case process. Estimate the maximum amount of time and other resources available for information collection for each case. For each Critical Performance in the model of case process, define tasks to be presented to the case that will elicit the level of case performance. If the measurement task itself can change the behavior of the case, alternate forms of each measurement should be available for repeated case measurement. Estimate the generalized cost and the reliability and validity of each measurement procedure. Similarly, for each concurrent or historical case characteristic, define measurement procedures and estimate cost, reliability and validity. Within the constraints of available resources, choose a set of measurement procedures that measure all Critical Performances and as many concurrent and historical effectors as possible. During actual usage of the measurement procedures, estimate actual reliability if possible. Procedures in Reading. Many person-hours were spent in the creation of a testing battery that was true to the concepts in the MORAL. Given the MORAL, measurements were needed for each of the six Critical =e":'ma':es: a'::aa.“a'v. c :m've's ' 3' :ec's'cn ~35 :e:‘slor~ 37'; 15.22:: in 5::ea'ec m- ii‘l sets 3‘ tm'gnea532 ~-=§Q ‘A HL up . fl . 3 3‘21 I re- 1'3“ -3 see: b .5! 6-3 Performances: instant word recognition, decoded word recognition, vocabulary, oral reading, silent reading comprehension, and listening comprehension. Given the difficulty of creating reading tests, the decision was made to use existing standardized tests. While this decision allowed us to start up relatively quickly and inexpensively, it resulted in less than ideal adherence to the MORAL. The single area that appeared most problematic was the reading of connected text. Although many sets of graded selections could be found, none used consistent comprehension frames as defined by the MORAL. Although many selections could be found with consistent comprehension frames, none of these were graded. For the determination of grade level reading, two tests with graded selections were chosen. This meant, however, that different types of comprehension were not independently measured. The MORAL Critical Performances were measured, then, with a specially constructed test battery based on existing tests of reading ability. The components of this special battery were as follows: (note that a sample of the test stimulus materials is included in appendix A and a sample booklet for recording the student's performance is included as appendix B) 1. Instant Word Recognition : the Slosson Oral Reading Test (SORT). The SORT is an individual test of word recognition ability composed of ten lists of twenty words each. Each successive list is at a higher level of difficulty than the one preceding it. The difficulty of the words ranges from primer to high school. The standard procedures for the SORT give the student five seconds to recognize a word before the examiner moves the student on. The student is started at a list in which s/he can recognize all 20 words and is to continue until is": o‘ ' Pen! 9.. ..s 9' it?! a: ilaf‘neg. ’ch' : r '9'! s: “"C'Ear A. *‘uf‘. 6h none of the 20 words in a list is called correctly. The examiner records only right or wrong responses. In the test battery constructed here, non standard procedures were employed. First, since a pre-, mid-, and post-test were planned, three variations of the SORT were created by randomly reordering the words in each of the ten lists. Second, all students were started in the first list because of anticipated poor performances. Third, the students were stOpped when they correctly read only five or fewer words correct in a given list. The reading specialist felt that going further could well introduce affect problems in ths student's reaction to the test battery. Fourth, the examiner recorded word calls in three categories: instant (immediately called correctly), mediated (not immediate, but ultimately called correctly), and miscalled (not called correctly). The miscalls were recorded phonetically. This allowed a finer gradation of student performance than with the normal procedures. Finally, the student was asked to define the twenty words in the last list that s/he attempted. This was an attempt to see if language experience was interfering with word recognition. The special test battery, then, measured instant word recognition using the SORT with non-standard procedures. Decoded Word Recognition : the Slosson Oral Reading Test (SORT). The SORT was also used as a measure of overall decoding ability. As described earlier, the examiner identified which words, if any, the student did not recognize Iinstantly but ultimately did call I flflvv.po .. ‘V 55. V I n s 1h 6 533; C s ‘A HQ 5 ‘SE ‘ I { Cr 65 correctly. The miscalled words were phonetically transcribed on the test form. Further data on overall decoding ability was available from miscues on the oral reading tasks to be described below. In addition to these measures of overall decoding ability, some of the specific subskills of decoding were checked with the Gates-McKillop Recognizing and Blending Common Word Parts test. This test presents three lists (23 consonant blends, 23 phonograms, and 23 nonsense words) for the student to call aloud. In this test battery, each student was asked to call all 69 words or word parts. The examiner categorized the response as correct or incorrect. Again, because of the pre-, mid-, post-test plan, three variations of the test were created by randomly reordering each list. Vocabulary : selected words from the Slosson Oral Reading Test and the Spache and Durrell paragraphs. Two vocabulary measures were used in the test battery. The first measure was based on the SORT. The student was asked to give the definition of all words on the last SORT list that s/he had attempted. The words were pronounced aloud by the examiner and the student's response was categorized as correct, incorrect, or no attempt to respond. The second measure was based on the silent reading comprehension selections and the listening comprehension selections included in the student's test battery (described below). During test construction, words were chosen by the research staff from these selections. Again. the student was asked to give a definition for these words pronounced aloud by the examiner. Again, the (QSDC'SQ \ kc a ne‘lwec Da'amete" CWC'QRp a0. nectar H 66 response was categorized as correct, incorrect, or no attempt. No alternate forms were used here because the words to be defined for a particular student depended wholly upon other design parameters and his/her reading performance (i.e., what form of comprehension measures were used and what was the last SORT list attempted). The logic of choosing these sets of words for vocabulary measures stems from the MORAL, which views word meanings as an effector of the major reading skills of word recognition and silent reading comprehension. Oral Reading : the Gray Oral Reading Test and the Sullivan Placement Test. The Gray presents a series of paragraphs to be read aloud. As the paragraphs proceed, their difficulty level rises very dramatically. The student's task was to read the paragraphs aloud. Reading time, miscues and other oral reading errors were recorded and used to compute grade equivalents. The student was stopped from going further when his/her reading time for any paragraph exceeded two minutes. Four forms of the test were available from the publisher (A, B, C, and D). In our test batteries, versions A, B, and D were used as the alternate forms (form C was not available in the MSU reading clinic files). The Sullivan also presented short paragraphs to be read aloud, but these were substantially shorter, simpler, and were keyed directly to placement into the reading materials of the treatment Program available at the subjects' school. Only one form was I a:”a:‘e (I 1253155W '- 3 En: Rea: “C tile 5.: ; She ‘Gr can...“ I‘ll {mtar 1"3cgf 0‘ 3‘18'1ate Cat; , 3‘655‘ Sinc, 9‘ sac... the D'e-m f: m: iOr . hWe‘ve. 'Og'c 5% D ‘OWeu 5a).”? Ehc ’err ' made . ’n we fid’re ’- ‘Ieat tile q nes} .gh. ‘ all t. be I 67 available but since no measure of retention was extracted from the test, this was not viewed as a problem. 5. Silent Reading Comprehension : the Ourrell Silent Reading Paragraphs and the Spache paragraphs. Two comprehension measures were required, one for silent reading comprehension and one for listening comprehension. The MORAL describes these two kinds of comprehension in similar ways. They both tap into the same semantic net, but through different means. Therefore, it was desirable to have alternate forms of the same test to measure these two different comprehensions. Since these tests are presented as connected text, the chance caf student recall of previous testing materials seemed great given tflwe pre-mid-post test design. This would require three alternate fcarms for each of the listening and silent comprehension measures. T11ewefore, ideally, six equivalent forms of comprehension selections would be provided . However, no silent reading test using graded paragraphs and haxrilng enough alternate forms could be located. Therefore, after reviewing several standardized tests for comprehension, the decision has made to use the Spache comprehension paragraphs (two forms) and the Durrell silent reading paragraphs (two forms). It was decided to treat these tests as if they were equivalent forms. The planned des 5 9n, then, called for one form of the Spache and one form of the D“"'¢Bl l to be used for each test session. One of these forms would be Used for silent reading comprehension and one would be used for “.0 yank 3! “as e ‘0? 6. 68 listening comprehension. For specifics of the design, see the test booklet design table below. Whatever the selections used, the task for the student was to read a paragraph silently, then to answer three or four specific questions from the examiner. The questions were created by the research staff in an attempt to measure separation of main idea and detail, inference, and sequence. The examiner recorded the student's reading time, his/her response to the main idea and inference questions, and the sequence of free recall of the contents of each selection. The students started with the second grade paragraph and proceeded upward through consecutively higher selections. During pretest, a stopping criteria was defined based on the number of sequential memories and on reading time. After pretest, this was cflwanged to ignore reading time and use number of memories only. This cflwange was implemented when it became clear that the time criterion was turning what was intended to be a power test into a speed test. Listening Comprehension : the Durrell Silent Reading Parapraphs and the Spache paragraphs. The same procedures were used as for the Si lent reading comprehension except that the examiner read the selection aloud to the student rather than having the student read it Si? lently. Attef‘ltion/Motivation : a most critical effecting factor. No direct measUre was available. Rather, through interview questions and exam; her judgement during test administration, the past and present "‘0t3vation of the student was estimated. ,- M r ' A. ‘l 'U n i ‘ o' 'I’ "F I! b V '3 565, "at Trf'a' 382"e“1; a" T». .. _h E H: I .3- . -§\ ‘3‘. \ s.- :5: 69 In addition to these specific measures of the MORAL Critical Performances, information on all of these areas and more were obtained from initial structured interviews with the student and from examiner debriefing after the test session. The test was conducted using a set of stimulus materials for the student and a protocol notebook on which the examiner recorded student performance. Depending on the student, it took between one hour and one hour and AS minutes to administer the battery. The student's stimulus materials for the measurement task were kept in plastic sheets in a three ring binder. The student protocol notebook in which the experimenter entered the student's responses was formatted like a script. At each step, the next action to be taken by the examiner was spelled out. The protocol notebooks for recording student performance were generated by computer for each student for each testing session. This guaranteed proper choice of test versions for the student. All test materials were prepared on a text processing system. A sample of the stimulus materials and test protocol notebooks are included as appendices A and B, respectively. The particular characteristics of a test were defined by one of the descriptions in Table lO. Three different combinations of isolated word tasks were defined; eight different combinations of connected text were defined. Combining these tasks into one test gave 2h variations of the test battery. The students in the study were randomly allocated to these 2h versions of the total test at pretest time in such a way that the number of students taking any version of the total test battery or any version of an individual subtest were approximately equal. .g'ury dist. v511-u Q l-l- 5.11 hi i , I n I a ii i i 4‘“ it 4‘4“ 1.! ah! i I ..h ,9." ...a Q" q i {Ease 70 Table lO Test Booklet Characteristics Battery SORT Gates Gray Silent Listening number version version version version version 601 1 602 603 '60h 605 606 607 608 609 610 6ll 6l2 613 61k 615 616 6l7 618 6l9 620 621 622 623 62k WN-‘WN-‘WN-‘WNHWN-HWNHWN—‘WN-d WN~WN—I\NN—-WN-dWN—dWN—WN—'\NN WNdwN—ikAAN-HWN-“WN—‘WN—‘WN—‘WN—O PrWWNN—‘errWWNN—‘firF'WWNN-d-fl Nam—rwrwwuw-at-wrwN-aN-urwrw These five parameters for test booklet generation were combined with some nonvarying tasks to yield a test battery that consisted of l)an introduction to the student, 2)a structured interview with the student, 3)the SORT, h)the SORT vocabulary, 5)the Gates-McKillop, 6)the Gray, 7)the Sulliven, 8)a break for the student, 9)the Silent reading comprehension, lO)the Listening comprehension, ll)the comprehension vocabularies, and l2)the examiner debriefing. '1 neat-.4 ixer :he 5‘;"."’:a*: i'ag'cstlc l"?- 'eang iii-'21,}: m. 1” ir'tte- l‘ctner is ‘3' hirer, “Cher ,‘5 5m fiany 5 “3.53011 3' ‘2” cw wanna 125533.14 rt: 215;: ‘Cr Q “311.. 71 Ill. Procedures Lg; Reliable Diagnosis Given the model of case process and devices for measuring its significant constructs, the next requirement is reliability in diagnostic categorization. The results of previous research in medicine and reading indicated that reliabile diagnostic categorization can be achieved through the use of external memory aids. One type of memory aid is a written form to guide a clinician through the diagnostic task. Another is a simple set of computerized decision rules that would make low inference diagnoses directly from raw student performance. Still another is a complex computer system that tries to combine information from many sources to give a finer grained diagnosis. All three forms of decision aid were investigated in this study. General Procedure. The first decision aid mentioned was a written form. If the model definition procedures were followed, a series of Critical Performances and their concurrent and historical effecting factors have been defined. One simple decision aid would be merely a written checklist of all Critical Performances and their effectors to guarantee that the clinician 'at least considered all of the possibilities. Other additions might include questions concerning the basis for decisions and suggestions for remedial practice. One such form would be completed for every diagnosed case. The second decision aid would be a set of simple computerized deC'SiOO rules. Given the data from the measurement devices created in the p'e\'ious procedures, these rules would give diagnostic M- 'q t wit a .‘ I 'cv u b! P hrAI 'e 0' d- i '1' f) l' l" *9 S 1" .. 72 classifications. A threshold for performance on each measurment task could be established. Any case that scored better than the threshold would be diagnosed as "not problematicz" a case that scored worse would be "problematic.“ Finally, a complex computer simulated clinician could be used to guarantee reliable diagnoses. All of the measurements collected in the content area would have to be related to diagnostic categorization through complex and probabilistic computer programs. These relationships could then be used to define the memory of a simulated clinician. The simulated clinician would use this memory to give diagnostic decisions about specific problem readers. Procedures jg Reading. Given a source for raw data on significant student reading characteristics, three types of diagnoses were considered - those given by the subject teacher using a written decision aid, those generated by simple computerized decision rules, and those given by a complex Computerized ”clinician.” The diagnoses generated by the reading teacher were generated by a highly structured procedure. The diagnoses were conducted as follows: 1. The subject teacher obtained a copy of the diagnostic and remedial record form used by the Clinical Studies Project of the IRT in their l980 study (Vinsonhaler, et al, in progress). This record form is O"gar'lized around the MORAL. It specifically requests a decision on the adequacy of all reading abilties and the information used to male I? 1: rec. 0 .Per ;' l- ’3: 5.: AU . iaCZOr Mi 3... 1 73 make the decisions. Then, for each problematic Critical Performance, it requests a list of factors that might have caused the problem and then procedures that can alleviate the problem. 2. The subject teacher took one such record form and created a "scoring key." For this one record, she wrote down her perception of all factors that would cause problems with each Critical Performance and, then, wrote out remedial recommendations (see appendix D). 3. In forming the diagnosis for actual children, the teacher worked directly from the unidentified test battery protocol record. A diagnostic and remedial record form was filled out for the student by examining the student's performance and deciding which subset of the scoring key was relevant for this student. This, obviously, eliminated all problems with translating different student diagnoses into a standardized vocabulary. The diagnoses performed in this manner were very time consuming.* As a result, only 20 diagnostic reports were written by the teacher. All twenty reports were based on student pre-test performance. Throughout this report, these diagnoses will be called “clinician diagnoses.” The second class of diagnoses were generated by a simple set of computerized decision rules for each of the MORAL Critical Performances. Diagnostic categorization on these factors was determined by a threshold. If the student scored below the threshold, the Critical Performance was declared ”problematic;" if above the threshold, the k The tune consuming part was the scoring of the test battery by the clinician. Given a scored test battery, the diagnosis would take only a few minutes. :ec'a'a: :es:":e O - a "l. ' I- . b 'fistaf' . QE"5F> « hJ’Ios ‘/ 1’ "J 4 L‘Sterf r) I ll". 7h declaration was "not problematic.” The specific criteria used are described in Table II. Table ll Criteria for Decision Rule Diagnoses Critical Performance Measurement Device Threshold Instant Word SORT Subtest Number of words Recognition recognized instantly - 118 Decoded Word SORT Subtest Number of mediated Recognition words - 18 Meaning Vocabulary Vocabulary subtest Number of words defined - lO Oral Reading Gray subtest Number of selections with 8 or fewer miscalls - 6 Silent Reading Ourrell or Spache Number of selections subtest with hot or more sequential recall - 6 Listening Comprehension Ourrell or Spache Number of selections subtest with hot or more sequential recall - 6 In this study the diagnostic decisions were performed by computer analysis of the data. Throughout this report, these diagnoses will be called "rule diagnoses." The last set of diagnoses generated were done by a computerized clinician designed to collect and analyze data from the test battery and give diagnostic categorizations based on the MORAL. The computerized clinician was run on the Multiple Object Simulated Encounter System (MOSES) developed by the Institute for Research on Teaching and running on the University of Michigan's MTS system. The computer system uses a :a’. base fewest: a :‘eestarv i sase. These ”an' I! a.) a sc‘ c31""11135-3— Ifeatnenz s ..l: Ciec'se ~SE‘C ix C lag-1:5“ ll 1! 33:11}: 19353,“! S The fill M5“ gas: ”it“ a: 75 data base of diagnosis, cue, and treatment information to make diagnostic decisions in reading. The computer system has a set of elementary information processing tasks that can be applied to this data base. These elementary tasks are based on the Inquiry model of clinical problem solving and include such elements as hypothesis generation, confirmation and rejection; cue collection; diagnostic termination; treatment selection, and so on (Elstein, Vinsonhaler, 8 Wagner, I977). The precise behavior of the computer decision aid is demonstrated in appendix C. The computer decision aid was used to generate pretest diagnoses for the same twenty children diagnosed by the subject teacher. In addition, it was used to generate midtest and posttest diagnoses for this same set of students. The different procedures for determining the diagnoses were used to investigate the effectiveness and reliability of each diagnostic method in reading. Reliability of diagnostic categorization is central to both clinical and epidemiological research. IV. Procedures for Treatment Observation The next requirement for a clinical and epidemiological research study is, essentially, measures of treatment. Treatment may consist of a large number of actions initiated by the clinician and supported by the case, the clinician or others. Therefore, wide variation may, in fact, exist in treatments that are nominally the same. At this point, the model of case process again becomes critical as it highlights the significant effectors of change in case performance. Some measurement instruments must be created for the important effectors of case change. :se :reat' 222259: 35:15am ~ Tricfiae be c? in: 'Eseafzre 76 General Procedure. The purpose of this procedure is to assist with the creation of a treatment observation form to document the actual (as opposed to the prescribed) treatment that a case receives. Any discrepancies between intention and action might be observed by the clinician, the case, or a researcher. In any first attempt to generalize the clinical and epidemiological research paradigms to a content area, researcher observation will be assumed. The procedure described here can only be suggestive in nature. Wide variation will exist in each field since the concern, now, is observing the treatment received by real cases in a real clinical problem solving situation. I. It is assumed that a treatment has been selected. 2. From the model of case process, pull out the Critical Performances and their most significant effectors. 3. For each of these case performances and effectors, pull out the most significant effectors of the treatment that will be applied (as defined by the model of case process). A. Create an observation form of these significant effectors of treatment. A possible organization would be a two dimensional grid with one axis the effectors, and the other time. Then at each time increment, count or otherwise measure the qualities of each effector. 5. Use the form throughout the course of treatment to determine what actions were taken and what the status of significant effectors was. D-'::e‘ :2 :oci‘y ' he yes: :35 Cree-‘3. Ittfnlcgcs the :fiarac .55.; 9e 3'3““ a '3‘. an: :ESI'UC’JI W11“) a! “vine. W. 1;. "3319 We the 77 Procedures lg Reading. The last measurement device required was one to codify the actual instruction received by the students (as opposed to the prescribed or intended instruction). A treatment observation form was created that was based on the MORAL, research on observational techniques described in the related research section of this thesis, and the characteristics of general remedial reading classroom instruction. Using the Carroll model as a guide, the purpose of the form was to provide a means for recording the types of instructional tasks used, the amount of time spent in each task, the amount of activity during this time, and the corrective and affective feedback provided by the instructors. Because of the limitations “imposed on the study by the particular setting, it was felt that this information would adequately characterize the instruction and that the other constructs from the model, like the instructional quality, were essentially constant. The reading clinician assumed initially that all student reading problems were the result of instructional deficits. The instruction planned for each student consisted of the same activities and differed primarily in terms of pace. A complete description of instructional activities is given later in the section entitled ”Procedures for Executing the Study." The treatment observation form evolved during the first month of the study as it was used daily to record instruction characteristics. It consisted of two 8.5xlh inch pages taped together. The pages had sections for each student in the class. Within each student's section were locations for recording how much time was spent on each type of task with what types of feedback. One form was used for each class. The ass: «“3: sate"a's. ms or pa 3: the se Ll! Ills-1'“: ‘34” Wis 33W! a": 51.:ef1, ‘rselzab: 133". in “F 31on ‘5 al- a! . I.“ . ‘9' Uh: 78 most reliable information collected included type of task, time on task, materials, and instructor. Less reliable was the estimate of number of words or pages read in a task. Least reliable were measures of feedback on the second page, consisting of type and frequency of feedback from the instructor to the student. Each entry in the form was divided into four parts, one for each of four possible tasks in one class session. Rarely did more than four settings occur in one class session for one student. A sample instruction observation form is included in appendix F. Reliability of classroom observation is described later in this report in the section covering the actual procedures used in executing the study in reading. As one last check on the adequacy of the coding form for collecting the type of data described, an experienced classroom observational researcher (Linda Anderson) examined the forms and suggested changes. After understanding the data collection needs of this study, she felt that coding schemes that she had used in previous studies (Anderson, Evertson, 8 Brophy, l979) were inadequate for the current requirements and the forms that had been created empirically for the study would suffice. V. Procedures for Outcome Measurement No new procedures for measurement of outcome are needed. One simply performs a second diagnostic assessment and measures outcome as the difference between performance measures or diagnoses before and after treatment. This is an entirely reasonable definition of outcome since the diagnostic process can distinguish problematic and non-problematic case net‘s L- Q Huang, I V. a :‘n “ .la :y' ‘ 333360 a: the Clin to a of i l " the sze: I'Qp- Se‘( 1 “ the '95 5:” g‘ in. 79 case performances no matter when the case is seen. VI. Procedures for Executing the Study Finally, with all of the groundwork laid, the actual research study was planned and executed. General Procedure. The clinical and epidemiological research paradigms provide a formal means for the aggregation and integration of clinical experience. The steps necessary to complete a clinical and epidemiological research study (Feinstein, I977) can be organized around the determination of the following: I. the objective of the research: the individual clinician, the clinical researcher, or the epidemiological researcher must be able to define what information is desired. This is usually done in terms of initial state, maneuvers, and subsequent state. 2. the ”intake“ (sampling) of subjects: the series of self selecting steps between an unidentified patient and a research subject may be long. The researcher must specify how the potential subjects will be selected. 3. the maintenance plan: the research study takes place over time. The researcher must have means to keep the selected patients in the study. A. the initial identification: the general concepts of the research \I. o \l‘) a f" ('1) atje 0.’ @285 sec? ”at; 5:0: IO. 80 objective must be operationalized into a measurement procedure for the significant characteristics of the case, diagnosis, treatment, or outcome. This can be particularly difficult since many measurements may be the results of subjective, unreliable clinical decision making. the subsequent identification: again a measurement procedure is required. This time it must measure the significant characteristics of the case, diagnosis, treatment or outcome after the maneuver. Similar problems can occur. the allocation of maneuvers: the various maneuvers conducted by nature or man must be allocated to the subjects. For observational studies this consists of determining and classifying what happened. For experimental studies, the maneuver is assigned directly. the handling of intrusions: the maneuver of the study might be interrupted by nature, the experimenter, the clinician, or the subject. The adherance of the study to the desired maneuver must be determined and, potentially, adjusted. the measurement of transition: the outcome of the maneuver must be determined. The transition that occurred from the initial state to the subsequent state must be measured. the analytic procedures: the statistical analysis of the results should be planned. the extrapolation: the final evaluation of the finished experiment should be planned. O .‘C£"€3. II‘ . Page e'l ' ell-e :ategsr I e itse've. :ese . Ce 8l Underneath all of these steps lies the model of the case process. At every point, decisions have to be made about what personal or environmental characteristics of the case to measure, what diagnostic categories to use, what treatments to choose, and what outcomes to observe. The basis upon which all of these decisions are made is that case model. Procedures lg Reading. In this study a junior high school remedial reading teacher was observed while she examined h2 seventh grade problem readers and instructed 30 of these h2 over one school year. The specific data collected included (l)pre-, mid-, and post-test data on the students' reading abilities, (2)teacher and computer diagnoses on selected students, and (3)coded observation of the actual instruction received by the students. Objective: The objective of the combination clinical and epidemiological research study in reading was l)to determine baseline frequencies for various personal, environmental, diagnostic, treatment, and outcome characteristics, 2)to search for associations among diagnosis and personal or environmental characteristics and between outcome and treatment characteristics using retrospective analysis. 3)to test certain hypotheses about the relationship between personal or environmental characteristics and diagnoses and between treatment and outcome, and h)to detect any problems with the definition or measurement of the various reading characteristics. 1'13“ 3' 7.3, a .ass‘ 13 leaner ic' labor; State UP. 3 5909'. I 82 Intake and Maintenance of Subjects: This study was based upon the observation of one reading teacher in a Lansing area junior high school and the AZ seventh grade students this teacher saw during the school year. The reading teacher was a Teacher Collaborator with the Institute for Research on Teaching at Michigan State University and a reading specialist for a Lansing area junior high school. She worked for the Lansing School District each morning and the Institute each afternoon. She was an experienced researcher in the area of clinical decision making (with a Ph.D. in educational psychologY) who assisted in the design and procedures of the study as well as the teaching of the children. Two reading programs were available to problem readers in the subject's school. The first was a skills lab designed to improve the reading of children with an initial reading level of hth grade or higher. The second, for children with the poorest reading levels, was the remedial reading program conducted by the subject teacher. The students in the study, then, were to be those with the most severe reading deficiences. An initial group of hl seventh grade students was selected on the basis of Stanford Achievement Test total reading scores. * A cutoff point of h.O grade equivalence was used. Those students from the entire 7th grade population who scored h.0 or lower were referred to the subject teacher. These bl students were pretested. * Although only hi students were initially examined, one more student with similar background was referred to the teacher later in the year making the total h2. BOG L... - 4 _J I. 83 Because of resource constraints, not all bl students could be placed in the subject teacher's remedial reading program. Thirty students were selected for instruction from the bl as follows: I. Two students were rejected because they were repeating seventh grade and had received instruction from the subject teacher in the previous year with little success. 2. Two students were rejected because other programs were available for their special needs - one was bilingual, one was learning disabled. 3. The remaining 37 students were ranked on the basis of pretest performance on the Slosson Oral Reading Test grade equivalence score and the highest graded paragraph attempted on the silent reading comprehension task. The lowest' 30 were registered to receive instruction from the subject teacher.* This instruction was to continue at least until midtest when performance measures were to be re-taken. At the midtest, several students were dropped from the subject teacher's program. One student was dropped because he had moved from the district: one because he was dropped from school; one because he never came to class. Four students were dropped because they performed well enough in class and on the midtest to not warrant further instruction. One further wrinkle concerning tested students was that due to a special request from the subject teacher's building principal, one extra student * Although choosing the lowest 30 would result in regression toward the mean. some defensible criterion had to be used to satisfy external political constraints. as {we user 0‘ .. .‘na 19:27:: 17 the f: o ssvcent I 8h was given the pretest battery at this midtest time bringing the total number of students examined to AZ. Finally, before posttest at the end of the school year. three more students were lost due to injury, pregnancy, and chronic absence. A summary of the students and the data available on each is given in the following table. The numbers in the description column refer to student record numbers. Table l2 Available Data by Student Description Pre Mid Post Prinicipal's student I l l (1&8) . LD student I O O (107) Bilingual student I O 0 (11h) Students previously taught by instructor 2 2 2 (1113.11.60 Students scored too high to teach 7 7 3 (llS. l25. 127. I30. 137. th. 1h?) Students taught - better by midtest h h h (IO9, I36, lh2. lhh) Students taught the entire year 26 25 2h (the rest) Totals #2 39 3h One characteristic of the students should be emphasized, namely, the severity of their reading deficits. All students examined had extreme reading problems and many had corresponding attitudinal or personal problems. Such problems tend to compound and worsen with the passage of time. Throughout the description of the results of this study, one must keep in mind how poor the reading ability really was, .how limited, 'therefore, were the range of cases, and how the expected outcome in reas"; I .nassistei islt‘a‘. "I The aCtl‘Jltje ‘1' 3CM1.’ Eng ' 3’ aeve ihte ' abSe \II a final DErf 35 reading would be a decline relative to grade placement for the unassisted student. Initial Measures, Subsequent Measures, 5 Allocation of Maneuvers: The procedures for executing the study centered around the major activities described earlier. I. administering the measurement devices to children with reading problems at three points in the year (pre-, mid-, and post-instruction). 2. diagnosing the children's reading problems both by human clinician and by computer. 3. developing and using observational forms to record precise student interaction with actual instruction A. observing and coding of the subject teacher activities while she worked with her students 5. analyzing the relationships among the various aspects of student performance, student gain, and actual instruction. The precise chronology of events is as follows: I. 9/h/80-9/l9/80 Clinical Test Battery generation. Based on the MORAL description of the reading process, the test battery described earlier was constructed. It was designed to measure all of the Critical Performances described by the MORAL and as many potentially relevant effecting factors as possible. cata the Sim} Cass ire; anc 86 9/22/80-lO/l/80 Pretesting and Selection of Students. The test battery was administered to AZ seventh grade children who had been referred to the reading teacher on the basis of SAT test scores the previous spring. Due to limited resources, only 30 of the students could be retained and serviced. Placement into the class, then, was based on a small subpart of the pretest, namely the SORT word recognition performance and the comprehension scores. Note that significant limitations were placed on the study here by the clinician. Although very willing to collect the needed diagnostic data about the children, the clinician used only a small subset of the data for diagnostic and treatment decisions. Having taught similar children in previous years, the clinician had a single causal diagnosis that she presumed to be true for all children initially, namely, that their reading difficulties were the result of some previous instructional deficit. The clinician conducted no differential diagnosis;‘ the clinician prescribed the identical treatment for all children, the only difference being starting point and pace. This restriction on the study can be viewed in one of two ways. First, since the data was collected on all Critical Performances and many effectors, whether or not the clinician was correct could be determined from the empirical data. On the other hand, if she were correct, then this would again limit the range of results that could be gained from the study because the range of students was small. lO/lh/80-l/29/8l Instruction and Instruction Observation. The actual instruction took place during this period. Each student was pulled gm miss 'es: the st.c 'licr . f \H‘ 80‘s.. the? p \O" ‘6: HI” abse 37 out of another class for the reading activities. The timing was generally set up so that one-half of a regular school period was missed to receive the special reading instruction. The student was responsible for making up the missed work and the other teachers in the building were aware of the situation and supportive of it. The students were scheduled to come four days a week, for 25 minutes a day. There were between three and six students in each of five groups. The reading teacher had one teaching assistant and often had the additional assistance of a graduate teaching intern from Michigan State University. The instruction ratio was often two students per adult and only rarely went as high as five students per adult. These students met in a room that measured about twelve feet by twenty feet. Inside this small room were the teacher's desk, a couple of bookcases, chairs and three tables each seating four students, and a couple of chairs for the researcher observers. The students would arrive on the half hour, work for 25 minutes, and leave. The characteristics of this instruction and its interaction with the students were monitored almost every day by a researcher observer. Actual Instruction: The treatment provided to the students in this study was not under the control of the research study but, rather, was normal treatment used by the clinician. The instruction was very structured. The clinician made one basic assumption for all of her st.:¢ in '1 38-1; ‘Pstf BOG Char . ER fly. Sun 52?? $3.5 Desi GHQ 1‘3?! un: aSs: 88 students, namely, that any problems with the Critical Performances in reading were caused by previous instructional deficits. This being the case, the specialist had established a sequence of instructional activities designed to increase a student's instant and decoded word recognition, oral reading, and silent comprehension. The major differences in treatment across students consisted of the point of entry into the instructional sequence and the pace. Note that such a restricted range of treatment characteristics placed a severe limitation on the outcome of the study itself. The sequence of instructiona activities was centered around the Sulliven reading series (Sullivan Comprehension Readers, I980). This series consists of short readers setup in twenty major levels. The first level is written with words requiring the use of only the most basic and clear phonetic regularities. The second level introduces one or two more phonetic redundancies. Each level introduces even more of the phonetic system, at progressively more difficult levels. until the twentieth level contains most of the sound-symbol association rules of the language. For students that progressed past the Sullivan series, grade level basal texts were available for continued reading practice. These readers and basal texts, then, provided the contextual reading material for instruction. From each source of contextual reading material, the reading specialist drew up lists of words that characterized the text. In the Sullivan series the chosen words emphasized the new phonetic rules being introduced. Words from basal texts also reflected any 89 new phonetic regularities as well as other word mediation strategies such as use of affixes. These word lists served as the materials for isolated word practice. The isolated word lists and the contextual reading material were used by the reading specialist in both silent ( or group) activities and oral (or individual) acitivies. The major activities of the class consisted of the following: a. Oral reading of isolated words: A group of students were all given a copy of a typed word list. The teacher would then go in round-robin style asking each student in the group to read in succession a certain number of words. The round-robin would continue until all the words were read. For example, the students might be working with a four-page list of 200 words. The instructor would ask the first student to read the first five words aloud. While the first student read these five words, the other students were variously instructed to listen or look ahead to their words. When the first student had completed the five words, the second student would be asked to read the next five. Then the third student would read the next five and so on. As the students tried to read each word, the instructor would provide feedback on the correctness of the attempt. Correct performance was often signaled with no feedback or some one word positive comment. Incorrect performance was indicated by feedback attempting to focus the student's attention, recall a known decoding rule, etc. 90 Silent reading of isolated words: At times when class management. constraints would not allow the individual attention required for oral word list use, a student or set of students would be asked to read a given word list silently and mark all the words that “gave them'trouble.” When the students were done, they were to inform the instructor. At this point the instructor would assist the students with all marked words as well as spotchecking some of the unmarked words. Oral reading of contextual materials: A group of students were all given the same or occasionally different contextual material. The materials generally consisted of stories with l-h paragraphs per page, but this varied widely over‘ materials of different difficulty levels. The teacher would again go round-robin through the group of students asking each one to read l-2 pages of text at a time. The reading would continue or stop at the teacher's discretion. She had some criteria for determining adequace of oral reading performance and would try to keep the group reading consecutive pages until it was reached. While one student was reading the other students were variously asked to follow along or to silently read ahead the pages for their turn. As the students read the selections aloud, the instructor provided corrective feedback on a variety of reading performances. The major concerns were correctness of word recognition, inflection, fluency, and speed/ Incorrect performance was generally indicated by global comments and cues 9] to help the student correct the performance unaided. Silent reading of contextual materials: Students singly or in a group were asked to read through selections of connected text. When the students wer; done they were to inform the teacher. Often students were asked to complete a comprehension form that required the students to identify the main characters, the setting, the sequence of events, the major problem in the story and its resolution. Group instruction: Only rarely did the teacher attempt to provide direct instruction to the group. At such times the topics tended to be directed at word recognition. Homework: During the first part of the year, homework was assigned and monitored. However, as the term progressed. it became apparent that few of the students completed such assignments and their importance faded. Within the materials and activities just described, the instructor used only one sequence of instruction, specifically, practice of the word list until mastery was demonstrated orally and, then, practice of the associated connected text until mastery was demonstrated orally. When the materials at one level were complete the next level would begin. Oral and silent acitivies varied with the number of people in class and the number of assistants that the clinician might have; oral reading seemed to be preferred, silent reading was the sometimes necessary alternative. In general the instructor would try to 92 keep the students in one class on the same materials. As can be seen, across all instructed students the quality of instruction and opportunity for instruction was essentially the same. During classroom observation the ability of the students to understand the instruction did not seem to differ greatly. The two major differences seemed to be in the students' aptitudes for reading and their perseverance in the instructional tasks. Observation The observation of classroom treatment was conducted by one of two observers seated in the classroom in full view of the students. The observation was conducted primarily during the first part of the year from pre-test time to mid-test time. The observer sat at a desk in one corner of the rectangular room. From this corner the observer watched three tables of students. One table was directly ahead of the observer, one directly to the right and one diagonally in between. Each table was within five feet of the observer. The observers were introduced to the class at their first observational session but were from then on essentially ignored. Observers never spoke with the students on a typical day. As far as could be determined, the existence of an observer did not seem to change student behavior. For example, if a particular student was supposed to be working silently but was instead 93 staring off into space, this time wasting behavior rarely changed if the student saw the observer watching. Many different observation forms were tested before the final one given in appendix F was completed. As the instruction progressed it was apparent that the amount of time provided for each student's instruction, as well as the quality of feedback were essentially constant across students. Aptitude for instruction was not under control of the instructor. Hence, the major instructional determiner seemed to be under the control of the student, namely, perseverance. The form used to collect the data reflects this perspective keying in primarily. on engaged time. As the obervation continued, it became clear that, again, the range of activities for the students was quite structured and essentially constant. To make sure that nothing critical was missing from the observation device, an experienced classroom researcher (Linda Anderson) examined the forms, suggested some changes, and stated that for the types of data desired here, the coding schemes that she had used in the past were inappropriate (Anderson, Evertson, and Brophy, I979). Attempts were made to estimate the ‘reliability of the coders by having both observers collect data on the same classroom session. Because of the extremely cramped quarters available, it was discovered that the two observers themselves interfered with each other. One observer could not help but notice as the other observer redirected attention from one group to another; the reliability measures obtained, therefore, are 9h undoubtedly inflated. However, looking at the low inference nature of the tasks being recorded and the systematic way in which it was performed, possibly the correlation among observed events of 0.85 was not too far from the correct amounts. 2/2/8l-2/l3/8l Midtesting. The test battery was administered again to all available children. Three students had been lost. Alternate forms were used for all subtests. 2/l6/8l-5/l9/8l Instruction and Instruction Observation. Instruction proceeded essentially as before. Observation was greatly reduced. Some students were dropped from further participation because their reading had improved substantially. 5/20/8l-6/5/8l Posttesting. The final test battery was administered to all children again. addition to this schedule of events, at times throughout the year, following took place: Written diagnosis of one-half of the pretest performances by the reading specialist Computer diagnosis of pre and post test performances ”3"- Is .. Silt: '505 P v irea let i 95 Summar The procedures described above detailed some of the more important aspects of generalizing the clinical and epidemiological research paradigms to any field as well as the specific procedures used in this study of reading diagnosis and treatment. More than anything else, the central concern of these procedures is the long-term aggregation of specific, reliable, model-directed measures of clinical experience. As these descriptors of empirical experience accumulate, a data base for decision making is formed. The data base can assist in the diagnostic, treatment, and follow-up decisions of a clinician or in the policy decisions of an administrator by providing access to past experience. Let us turn, now, to the beginning of such a data base in reading. 't'sc' “ I”! re: :1 "51 Clinic (U CHAPTER 5 Results Introduction The results of achieving the first objective, that of determining how a clinical and epidemiological research study could be completed in any clinical problem solving field in general and carrying out such a stUdy in reading, were described in Chapter Three. This chapter centers on the second objective, that of evaluating the results of the reading study for its comments on the field of reading and on attempting to generalize the research paradigms outside of medicine. The data that were available for analysis included raw performance data on the pre-, mid-, and post-tests. the decision-rule diagnoses for all children on all tests, the subject teacher's diagnosis of 20 children's pre-test performance, the computer's diagnosis of the same 20 children on the pre-. mid-, and post-tests, and the coded observation record of actual instruction. Before the data were analyzed, however, two checks had to be made. The first area of concern was the equivalence of different forms of the reading subtests in the battery. Because of the possibility that learning could occur from pre- to mid- to post-test, various forms of different stimulus materials were prepared. The first analysis that was performed, then, was to test the assumption that the different forms of 96 lacs sucte heicatec I tfiemeasm usesucst. ’31 sco'e :ma'able leecec m 97 each subtest were, in fact, equivalent. The results of this analysis indicated that all word recognition measures were equivalent but that the measures of silent reading comprehension and listening comprehension were substantially different. Therefore, adjustments were made to the raw scores for silent reading and listening to make the results more comparable. A complete description of the equivalence of forms and needed transformations is given at the end of this chapter. Once the equivalence of alternate forms was established and adjusted, the second major concern was the reliability of the data. One of the critical requirements for the clinical and epidemiological research paradigms is reliability in the determination of personal, environmental, diagnostic, treatment, and outcome characteristics. The reliability of diagnostic categorization of Critical Performances can apparently be quite high because the rule diagnosis is limited only by the reliability of the subtests themselves. The reliability concerns are elaborated at the end of this chapter. For now, let us turn to the analysis of the clinical and epidemiological data in reading. Egidemiological Data Analysis: fig; :3 Q The clinical and epidemiological research paradigms are formal models for improving future clinical decisions based on experience with past clinical decisions. It is altogether appropriate, then, to organize any research guided by these paradigms around the major decisions made in the clinical setting. 3:1! 13556 me” see: 5240! act-C: I'ea i658 3.- 98 The specific decisions that a reading specialist makes are many. They include choosing which children to examine for problems, diagnosing those children, selecting from the diagnosed children the ones to be retained for instruction, grouping the children for instruction, selecting and administering and adapting treatment, determining when a student is "cured" or "beyond help,” and so on. However, all of these decisions center around (l)the ability to determine the state of a child's reading and (2)the determination of the most appropriate treatment plan for a problem reader. They focus on the epidemiological research concerns of relating personal and environmental characteristics of the student to a diagnostic categorization (P,E -> D) and around the clinical research concerns of relating characteristics of the student, diagnosis, and treatment to outcomes (P,E -> D -> T -> 0). Taking the natural chronological order, a clinician determines the diagnosis first and then chooses a treatment. Hence, let us begin with the epidemiological analysis of diagnostic categorization. Characteristics 2: Diagnosis 29g Cross-sectional Analysis. The most straightforward analysis of epidemiological data is simply counting without reference to cause or effect, a so-called "cross-sectional” analysis. This type of analysis provides information on the prevalence of certain data in the populations under study. In this study, the sample consisted of all 7th grade students in a junior high school who had received a grade equivalent score in reading of h.O or lower on the Stanford Achievement Test, a group of very poor readers. The most important prevalence rates are those of diagnostic categories a: mi $338616 _ H U less it h 3‘ 3“,! 99 and diagnostic profiles. Three different sources of diagnostic. categorization were available: the clinician, the computer decision rules and the computerized clinician. The clinician and computerized clinician both made initial diagnoses for 20 of the #2 children. The clinician's diagnosis included categorizations on all six of the MORAL Critical Performances as well as 30 other causal factors. The computerized clinician's diagnosis also included decisions on the six MORAL Critical Performances and on 58 other causal factors. Because the diagnostic classifications by clinician and computer were not available for all cases, and because a classification using less inference was desired, a third diagnosis was prepared using a set of decision rules based directly on raw performance data. Thresholds were established for the Critical Performances. If a student scored above the threshold, the diagnostic categorization would be "not problematic;" below the threshold, "problematic." Such thresholds could be established only for the Critical Performances and not for any causal factors. This rule-bassd diagnosis and the raw performance data will be the major dependent variables throughout the rest of the analysis. A comparison of the prevalence of Critical Performance problems across diagnostic sources is given in Table I3. The percentage of diagnosed cases with a problem in a particular Critical Performance is given along with the exact ratio of problematic cases to total cases. Instant ‘uc iesagn. i z I: 'we Jecocec dc letognizl. F'eb'em ‘63“lng locaeul a. 3'3) Rea: Instant Word Recognition Problem Decoded Word Recognition Problem Meaning Vocabulary Oral Reading Silent Reading Comprehension Listening Comprehension Without appears that students at this level of SAT performance trouble with diagnostic sources generally agree on this. even lOO Table Prevalence of Critical Clinician Diagnosis (N820) 952 (IQ/20) 1002 (20/20) 752 (15/20) 95% (19/20) 1002 (20/20) '752 (IS/20) attempting to cor all Critical Performances 13 Performance Problems Rule Diagnosis (N-h3) Computerized Clinician Diagnosis (N-ZO) 81% (3h/h2) 60% (12/20) (No decision possible on 6 cases) 90% (38/h2) 852 (17/20) (No decision possible on 3 cases) 672 (28/h2) 70* (lh/ZO) (No decision possible on 6 cases) 76% (32/52) 953 (l9/20) lOOZ (h2/h2) 502 (IO/20) (Differing definition of diagnosis) 92% 253 (05/20) (inappropriate weighting of diagnostic information) (39/52) relate the different diagnoses, it generally have in reading. The different The major differences in actual percentages seem to be due merely to the computerized clinician's milky letr, x 1h! QZ‘II listening :muteri :Finicia,» Illa 3‘90 :3"- be at Vent r more“, SWEET I [wtaqlng 323mg : ’eaCqu ”mute. .1! | ' slimc‘ lOl inability to categorize some cases on some Critical Performances.* However, there are a few significantly different prevalence rates for the different diagnostic sources. Note especially silent reading and listening comprehension. The prevalence rates for the diagnosis by the computerized clinician is one-half and one-fourth of that for the clinician or rule diagnoses, respectively. After careful examination of the algorithm used by the computerized clinician, this large difference can be attributed to a disagreement on the definition of problematic silent reading and inappropriate weighting of data for listening comprehension. The clinician and the decision rule evidently used a similar criterion - something like ”reading more than I year below 952g; glacement." The computerized clinician algorithm used the definition "reading more than one year below 325g recognition lgygl.“ From this second perspective, there were a number of students (l0) whose silent reading comprehension was limited by their ability to read words. The computerized clinician algorithm did not call this problematic: the clinician and rule diagnoses did. With these minor differences in high prevalence rates, one would expect the measures of association among the three diagnostic sources to be high. The measure used to compare the relative diagnoses is a simple proportion - the number of actual agreements divided by the number of possible agreements. It is based on the simple two by two table described in Table lh. * The computerized clinician had the option to decide that the evidence was too mixed to make a decision. The rule based decision did not allow this. The human clinician never took this option. SCL'CI .ateg: . scan: 3 e ‘a sec: 0 ”so: I: l02 Table lh Agreement Among Diagnostic Sources: The Statistic Definition Source I: Source I: Category M Problematic Category M Not Problematic Source J: Category M Problematic a b Source J: Category M Not c d Problematic The statistic is computed as (a+d)/(a+b+c+d). Note that this statistic computes agreement on one particular diagnostic classification across C3888 . The proportion data for the Critical Performance diagnoses is reported in Table l5. Table l5 Agreement Among Diagnostic Sources: Critical Performances Clinician and Clinician and Computerized Computerized Decision Rule Clinician and Clinician agreement Decision Rule agreement agreement Instant Word 65% (13/20) 95% (19/20) 70% (1u/20) Recognition Decoded Word 852 (17/20) 952 (19/20) 902 (18/20) Recognition Meaning 7:: (15/20) 80% (16/20) 75% (15/20) Vocabulary Oral Reading 902 (18/20) 85% (17/20) 852 (17/20) Silent Reading 502 (IO/20) lOOZ (20/20) 50% (IO/20) Comprehension Listening 50% (IO/20) 802 (l6/20) 30% ( 6/20) Comprehension DEII‘EI o'agh ins:a late the r Cfi‘g 703 A given proportion, like 652 for instant word recognition agreement between clinician and computerized clinician, means that of all students diagnosed by both clinician and computer, agreement on the status of instant word recognition performance occurred for 658 of the students. Note the very close correspondence between the clinician's diagnosis and the rule diagnosis. Clearly, because of the computerized clinician's inability to categorize certain students, its diagnoses are the most different of the three. Of course, these diagnostic categories do not occur in isolation for a student. Instead the clinical decision maker gives simultaneous judgements on many classifications for one student. The profiles of Critical Performances are defined in the following table. The first column gives the profile - the six symbols indicate the status of l)instant word recognition, 2)decoded word recognition, 3)meaning vocabulary, h)oral reading, 5)silent reading comprehension, and 6)listening comprehension in order. A "P" indicates that the corresponding Critical Performance is problematic; an "N" indicates not problematic. th Table I6 Agreement Among Diagnostic Sources: Profiles Diagnostic Profile Number of Number of Number of Clinician Rule Cases Computerized Cases with with Clinician Profile Profile Cases with (N-20) (N-h3) Profile (u-zo) l. P P P P P P 602 55% 02 2. P P P P P N lot 0% 252 3. P P N P P P IOZ IA: 52 h. P P N P P N IOX 02 I02 5. N P N P P P 5% 0% 02 6. P P P N P N 52 02 03 7. P P P P N P o: 02 lO% 8. N P P P N N 02 02 IOZ 9. N P P P N P 02 02 52 lO.N P P P P P 02 Oz 52 ll.N P N P N N 08 02 53 l2.N N P P N N 02 0% 52 l3.N N N P P N 02 02 52 lh.N N P N N N 02 0% ‘ 52 l5.P P P P N N 0% 02 52 16.P P N P N N 02 02 52 l7.N P N N P P 02 7* 02 l8.P N N P P P 02 52 02 l9.N P P N P P 0% 52 02 20.P N P N P P 02 22 03 2l.N P N N P N 02 22 02 22.P P N N P P 02 2% 0% 23.N P P P P N 02 22 02 2h.N N N N P N 02 22 02 25.P P P N P P 02 2% 02 The diagnostic profiles used by the clinician are relatively simple to understand. Eighteen of 20 students have problems with all reading skills and vary only on their language experience potential for reading (profiles I - A). Profile 5 is a student whose instant word recognition skills and meaning vocabulary are intact but who has a problem with mediating strategies for word recognition that might well be responsible for the problems with connected text. Profile 6 seems less likely. It describes a student who has problems with all forms of word recognition lill‘. CM? ’52 inc“ :ate Mi: '_ cg 'ecaiiir D'Ctlm cec7s'g () —-v~ ND C) c UEZECIe ICE and comprehension yet manages to read aloud adequately. The profiles generated by the computerized clinician seem to ilmdicate a greater diversity of student ability. Only eight of its pr1>files overlap at all with the clinician's diagnoses. However, recalling the different definitions for a silent reading comprehension problem. and the cases for which the computer was unable to make a dec:ision, the overlap could probably have reached as high as l2-lh out of 20. Of particular note, however, is that the computer diagnosis detected (supposedly) many students with no instant word recognition problems (profiles 8 - IA) and one student with really minimal problems (prkafile lh). Knowing that the computer algorithm and the decision rules are totally reliable (i.e., given the same data twice. the same diagnosis will result), these data are again demonstrating apparently different definitions of when a problem exists. The profiles of the diagnoses generated directly from the raw data bY decision rules agreed most closely with the clinician. Almost 703 of the‘se diagnostic profiles were accounted for by those first four clinician profiles. The rule-based diagnoses overlapped very little with the computer but did generate nine profiles not seen elsewhere. The data just described give the prevalence rates of various profiles. However, it did not directly compare total diagnostic categorization on the same case across diagnostic sources. When c""Ipar'ing the diagnosis of a specific student generated by two diagnostic sources, a single measure similar to the proportion described earlier can be computed. This statistic is also a proportion. But F? 3,1 0 36' OF; 33': 106 instead of being a proportion of agreement on one diagnostic categorization across cases, this new statistic is the proportion of agreement on one case across diagnostic categories. This proportion can then be averaged across all students diagnosed by the two sources. This average proportion describes how well the two diagnostic sources agree on total diagnoses across students. This measure of total agreement across cases is contained in Table I7. Standard deviations are included in parentheses. Table I7 Average Agreement on Total Diagnosis Across Cases Clinician and Clinician and Computerized Computerized Rule Diagnosis Clinician and Clinician Rule Diagnosis Diagnosis Critical 1 0.69 (0.22) 0.89 (O.ll) 0.67 (O.l7) Performances Diagnosis TOtal Diagnoses 0.311 (0.06) not available not available 0“ Critical Performances and Causal Factors The diagnoses of Critical Performances by clinician and decision "Ule are very close, agreeing 892 of the time. It is not surprising, then. that the agreements of the computerized clinician's diagnosis with the clinician and rule diagnoses were essentially the same, 69% and 672 respectively. The second row of the table deals with the more fine-grained diagnosis by computerized clinician and clinician only. The computer Used a total of 65 diagnostic categories; the clinician used 36. These two 0' 'eaai-ig any I (”i 1111': came diagno! elimiw. fI’IE'e PQC‘OrI 107 two differing vocabularies for describing diagnostic categories in reading were combined into #3 categories. Of these A3, II were mentioned only by the computerized clinician, l0 were mentioned only by the clinician. Therefore, there was automatic disagreement on 2l of the #3 categories. This is the reason for the low agreement on the total diagnosis. If the 2l categories on which agreement was not possible were eliminated, then. the agreement between clinician and computer reached 672 (s.d.-ll2), approximately equal to that on the Critical Performance diagnosis alone. During this first analysis, one fact became exceedingly clear. There is no well defined, operational construct for any of the Critical Performances of the model or any of their‘ effectors. The major differences among diagnostic categorization from the three sources seemed to come from differences in definition of the construct and differences in levels of inference. Apparently, the clinician used criteria very similar to the decision rule, namely, comparing performance on one subtest to some threshold. The computerized clinician was programmed to attempt more subtle levels of diagnosis by combining interview and selected performance data across reading tasks when making a decision. The two different diagnostic approaches give different results. It can be argued that the most reasonable interpretation is that the computerized clinician system has never been ”fine-tuned" on data from real children. The weighting system it uses should be revised to match more closely the clinician's diagnosis or the rule diagnosis. Or should it? Indeed, looking at the raw data, the clinician and rule diagnoses appear more reasonable. But this discrepancy has made it EDUDSE um vesea' fesear :e‘in? a C31 3130:: iOr 5 OS l08 abundantly clear that reading has no “pathologist” or "lab test" to tell which is right! It seems apparent that for the clinical and epidemiological research paradigms to succeed for any individual clinician or large research organization, to as great an extent as possible, the definitions of all constructs must be operationalized and maintained as a constant over time. For each construct, one must define the hypothesized function, specific procedures for measurement, and methods for combining measures into a final judgement. Retrospective Analysi . The next epidemiological analysis examines cause. and effect by looking backwards - a retrospective analysis. This means that groupings must be formed on the basis of diagnostic categorization and followed to differences in personal and environmental factors (D -> P,E). This type of analysis can assist in the search for causal hypotheses. The diagnostic categories examined were the Critical Performances of the MORAL model of reading. Although more categories were available in the clinician and computerized clinician diagnoses of some students, these more fine-grained. causal classifications are more inferential in nature. Since these classifications were not the direct target for measurment, many of the decisions were made on what few data there were (by clinician and computer) or by prior likelihoods (the clinician only). Therefore, these secondary classifications were ignored for this first analysis. tons ‘a clinic can:er DCVCE: I I e :r l09 Some of the personal and environmental characteristics available on the students included interview data, sex, and SAT scores. First consider the students' self perception of their reading ability based on clinicial interview. During interview, the students were queried concerning their self perception of their ability to l)recognize the words in their school books instantly, 2)sound out words, 3)read fluently, and h)understand what they read. The association of these self perceptions with the student's diagnostic categorization is presented in Table I8. The diagnostic source used was the decision rule diagnosis (which very closely paralleled the clinician's diagnoses). Table l8 Differences in Self Perception Based on Diagnosis Diagnosis Based on Self Perception Good Self Perception Bad Decision Rules ' Instant Word Recognition Good 502 ( l of 2) 508 ( l of 2) Bad 56% (ID of l8) an: ( 8 of l8) Decoded Word Recognition Good 0% ( O of l) l002 ( l of l) Bad le ( h of 19) 792 (IS of l9) Oral Reading Good 0% ( O of h) lOOX ( h of A) Bad I93 ( 3 of l6) 8l2 (l3 of l6) Silent Reading Comprehension Good -- -- Bad 35% ( 7 of 20) 65% (I3 of 20) The most obvious characteristics of these data is that the number of students without a problem for any given diagnostic category is 'ea' 23¢ nae: Jh'e Tnte (I “‘ Stat 253‘ sezq 330: A.“ 1 .-.- 3 0‘ “A Smile llO r'eally too small in this study to give a meaningful comparison. None of the differences among self perception for students with and without Faroblems were at all statistically significant, but with the small N tJ1at is not surprising. The general trend was that the majority of students had a poor self perception across the Critical Performances. TWais was true both for students who in fact had problems as well as for tJaose who performed adequately. What is likely is that these students, Mano are each having .problems in some reading area, generalize their particular weakness to the entire reading act. This is not entirely (Jrsreasonable given the way in which the various Critical Performances irateract with each other. The only exception is the high confidence poor students have in in their ability to recognize words instantly. In addition, for every case there was a greater percentage of poor Students who thought they were good than there were good students who thought they were poor. In fact, the data indicate that the student's self perception (measured as it was through interview) is not at all a 900d indicator of correct diagnostic categorization. One other check was run to see if another demographic variable, Sex, had confounded the results. Clinical and epidemiological research lee a simple technique called "standardization" to compensate for groupings that may differ on some significant personal or environmental Characteristics other than those under study (Roberts, I977, p.lh8). Basically, standardization is performed by stratifying the analysis on various levels of the confounding characteristic. Diagn: IIaIeE insta' 3ral R P hi lll Reexamining the association of self perception and diagnostic <:ategorization in light of sex differences gives the data in Table I9. Table I9 Differences in Self Perception Based on Diagnosis by Sex [)iagnosis Based on Decision Rules for Self ‘ Self Itale Students Perception Perception Good Bad liastant Word Recognition Good , l l Bad 9 3 Decoded Word Recognition Good 0 l Bad h 9 Oral Reading Good 0 A Bad 3 7 Si lent Reading Comprehension Good 0 0 Bad h l0 Diagnosis based on Decision Rules for Self Self Feumale Students Perception Perception Good Bad |hstant Word Recognition Good 0 0 Bad I 5 Decoded Word Recognition Good 0 0 Bad 0 6 Oral Reading Good 0 0 Bad 0 6 Silent Reading Comprehension Good 0 0 Bad 3 3 28' at: (u C) ll2 'This stratification indicates that in this sample of students the female~ students had a consistent (and correct) poor self assessment. It appears ‘to be the male students that are incorrectly assessing their ability. Prospective Analysis. The final epidemiological analysis examines <:ause and effect by looking forward, a prospective analysis. Groupings nqut be formed on the basis of personal and environmental cflwaracteristics and followed, then, to differences in diagnostic categorization (P,E -> D) . This type of analysis can be used to directly estimate the risk of contracting a certain problem given the presence or absence of a particular personal or environmental factor. The diagnostic categories examined were the MORAL Critical Performances again. The major personal factor used to predict the cliagnostic categorization were two' estimates~of language experience. TWwese two estimates were based on the judgement of test administrators iImmediately after test completion. They were estimates of a student's c>‘~Ierall language ability and of the student's mean length of utterance. TWwe overall language ability was judged to be high, average, or low: the Thean length of utterance was judged to be a word, a phrase, a sentence 'Or multiple sentences. The MORAL views language experience as a major determiner of reading ability and this analysis tested the association for these students. The relative diagnostic categorization of the groups defined by the test administrator's judgement of high, average, and low language ability is contained in Table 20. ll3 Table 20 Differences in Rate of Problem Diagnosis Based on Ratings of Language Abi l ity High Average Low Language Language Language Ability Ability Ability (N-lo) (Nel 7) (11-2 1) Instant Word Recognition 50.0% 70.62 95.23 Decoded Word Recognition 50.0% 911.2% 95.2% Meaning Vocabulary 0.02 58.52 85.73 Oral Reading 50.0% 611.7% 90.52 Si lent Reading l00.02 l00.0% l00.02 L i stening Comprehension 75.0% 88.23 l00.02 The most obvious aspect of these data is the increasing rate of problems with all Critical Performances as the test administrator judges language to be increasingly deficient. Before discussing these data fUrther the data on the second judgement of language ability, that of Mean length of utterance, is presented in Table 2l. In this table "Multiple" refers to the rating of "Mean Length of Utterance is Multiple Sentences." Similarly, "Single" refers to "Mean Length of Utterance is Single Sentence," "Phrase" refers to ”Mean Length of Utterance is Phrase," and "Word" refers to "Mean Length of Utterance is Word." 11h Table 21 Differences in Rate of Problem Diagnosis Based on Ratings of Mean Length of Utterance Multiple Single Phrase Word (N'9) (N'B) (N'13) (N'l 2) Instant Word 55.62 75.02 8h.62 lO0.02 Recognition Decoded Word 88.92 87.52 8h.62 100.02 Recognition Meaning Vocabulary 55.62 62.52 61.52 83.32 Oral Reading 66.72 752 61.52 100102 Silent Reading 100.02 100.02 100.02 100.02 Listening Comprehension 88.92 87.52 92.32 g l00.02 This second table generally demonstrates the same trend as the first. Apparently, language ability is a large determiner of diagnostic categorization. The differences between consecutive groups is not statistically significant but differences between the highest and lowest judgements are statistically significant at the 0.05 level for instant word recognition and meaning vocabulary. Summa y. Throughout this epidemiological analysis, several issues are raised. The first and simplest is a matter of sample size. For any of these analyses to provide guidance in the accurate diagnosis of reading problems or the eventual prevention of reading problems, quantities of data on non-problem readers must be collected. The relative specificity and sensitivity of test for diagnoses cannot be determined without such data. brll Hit! assi ’70: 9... 5‘1 em! $23: 21:; '53: L) "f. or:- 5:41 W 115 There are two more serious problems that the data reported here bring out. The first issue is causality. The data may, indeed, assist with the accurate diagnosis of reading problems by providing empirical associations among the various data. However, the findings actually did not examine causality. To study causality, one must in some way examine students before they develop the reading problems. Prospective studies might group children without reading problems on some personal and environmental characteristics and follow them forward until some students developed problems. A retrospective study would look back from existing problems to some point before the problem arose. This requires the examination of a store of pre-existing records. Based on the experiences connected with conducting this research, ansswering questions of causality may be more difficult for reading than for medicine. Just one obvious complication when looking for causes of reading problems is that there is a large cadre of reading teachers with C“ ffering abilities, philosophies, instructional styles, and so on. Not °nly do different students encounter very different teachers but one Student can, through the course of a school career, be remediated by a 3%? ies of teachers holding widely divergent views of what constitutes a Problem and its proper treatment. In some sense, there is no natural s1late. Conscious attempts to alter reading ability begin and continue throughout school. How, then, can all of these effects be disentangled? The last question raised by these data is the separability of reading problems. First, consider separation as to severity of problem. 1f one seventh grade student has an instant word recognition level one year below placement and another seventh grade student has an instant 116 word recognition level two years below placement, do they have the same problem? How about levels of one year and four years? Levels of one year and one-half year? If a second grade student is one-half year behind and a third grade student is one and one-half years behind, do they have the same problem? Next, consider separation based on Critical Performances. The MORAL says that a prerequisite for silent reading comprehension is word recognition. ls it appropriate, then, to diagnose as the computer system did and say that if silent reading performance matches word recognition performance then no comprehension problem exists? Or is the clinician's approach more valid? . Hopefully, this discussion has raised some of the more pressing problems introduced by inadequate definition of ’ constructs and measurements . Clinical Data Analysis: Q :3 I :3 Q The epidemiological analysis examined the first part of the clinical encounter dealing with diagnostic categorization based on personal and environmental characteristics. The remaining part to be examined is the clinical analysis of outcomes of treatments for particular diagnosis. Characteristics 9: Treatments and Outcomes. Although a cross-sectional analysis of clinical data is not possible (because treatment and outcome of treatment cannot exist at the same time), 325: 3.5: 9" LU- ‘! \l 117 descriptions of the ranges of treatment and outcomes must surely precede discussion of the retrospective or prospective data analysis. There were nominally three treatment groups in this study. The first group included twelve students who for various reasons were to receive no treatment with the reading clinician at all during the year. This group will be called "Group None.” The second and third group of students, 30 in number, started out together. However, by mid-year four of these students had progressed well enough that they were no longer deemed in need of assistance. Therefore, the second treatment group included four students who were instructed by the reading clinician for the first half of the year but received no such training during the second half. This second group will be called ”Group Half." The third group included 26 students who were instructed throughout the entire school year. This last group will be called ”Group Full.” The easiest indicator of the amount of instruction that each student in each group received is the number of days in attendance. Group None did not attend at all. Attendance patterns for Group Half and Group Full are in Table 22. Table 22 Mean Days of Instruction by Treatment Group First Half of Year Second Half of Year Group Half h2.5 (s.d. 7.0) (range 0 33-h9) Group Full h0.3 (s.d. 7.8) (range 2h.6 (s.d. l2.7) (range 18—u9) 0-39) Group None - no instruction at all SOC" ll8 During the first half of the year, detailed records were kept each day on the precise instruction received by the children. Such information as the total amount of time in class, the amount of time spent on word lists orally and silently, reading connected text :silently, etc., was recorded. Although not as reliable, estimates of ruanbers of words practiced in isolation and pages read in context were also recorded. Again, Group None received no instruction in the reading prwogram. The characteristics of instruction for Group Half and Group Ftill are contained in Table 23. The range of actual treatments within each group of instruction. is large. ._ :13 EM Time 5:7e 119 Table 23 Characteristics of Instructional Time by Treatment Group Group Half Group Full Total Time 801 (19h) 757 (177) in minutes (range 5&2-1000) (range 230-1053) Time on Isolated Words 161 (31) 220 (78) Orally (range 127-201) (range h9-h05) Time on Isolated Words 10.3 (12.7) 2h.h (27.8) Silently (range 0-26) (range 0-117) Time on Contextually 125 (77) 205 (81) Reading Orally (range 39-221) (range 25-380) Time on Contextual I69 (72) 87 (25) Reading Silently (range 65-223) (range 27-136) Time on Comprehension 77 (90) 2k (3k) Summaries (range 20-211) (range 0-125) Lost Time 13.5 (13.9) 9.h (8.5) (range 1-30) (range 0-28) Number of Words in 2609 (h29) 1511 (53k) Isolation (range 1088-213h) (range 553-2699) Number of Pages Orally lhl (71) 172 (86) (range 69-238) (range 0-358) Number of Pages Silent 115 (39) t 62.5 (16.3) (range 65-156) (range 0-180) Group None: no instruction in clinician's class at all During the first half of the year when observations were collected, thi‘s table indicates that the average student received around 775 Miifitates (12.9 hours) of total instruction across the 52 class sessions. Rewalling that the average number of classes attended was about #0. thef'e were 19 minutes per class. Of this 19 minutes, 5.3 minutes were Speryt on isolated word recognition orally, 0.6 minutes on isolated word recognition silently, h.8 minutes on oral reading of contextual ma: 1'13- 120 material, 2.5 minutes on silent reading of contextual material, 0.8 minutes on written comprehension summaries, and 0.3 minutes of explicitly lost time. This leaves h.7 minutes allocated to various other types of activities like class management, group instruction, and so on. Ths basic structure of the class was oriented toward word recognition drill and contextual practice. However, within this nominal treatment was a wide variety of actual treatments. The number of minutes in class ranged from 230 to 1053 (with attendance from the previous table ranging from 18 days to #9 days in the first half year). Similar to the problems with diagnostic categorization, the problem this variation raises is the definition of equality in treatment. Turning from treatment, the second important construct is outcome. In this analysis, two definitions of outcome will be examined: l)difference between diagnoses at the beginning and end of the year, and 2)differences in actual performance at the beginning and end of the jyear. Only the computer and rule diagnoses were generated at the end of tflwe year so the analysis could only be based on them. However, given the Prwablems with the computerized clinician's diagnoses described earlier, the diagnoses obtained from the decision rules will be the only ones Used in the analysis here. First consider the data across treatment groups. The prevalence rates of problems with the MORAL Critical Performances are contained in Table 2h. 11 if: its 121 Table 2A Prevalence Rates Before and After Treatment Across Treatment Groups Rule diagnosis Before Rule diagnosis After Treatment Treatment Instant Word 812 (Bk/#2) 822 (28/3h) Recognition Decoded Word 902 (38/h2) 532 (18/3h) Recognition Meaning Vocabulary 672 (28/h2) 852 (29/3h) OraI Reading 762 (32/h2) 852 (29/35) Silent Reading 1002 (hz/hz) 972 (33/3h) Listening Comprehension 932 (39/h2) 912 (31/3h) These results seem to indicate the students still have almost as many problems as when they began. The only large difference seems to be in the prevalence of decoded word recognition problems from pre-test to post-test 0 Because the students began the school year with extreme deficits, it is entirely possible that improvement did occur that was not mirrored 311 diagnostic classification changes. Therefore, the actual performances 0T1 the MORAL critical factors is given below: 122 Table 25 Distributions of Gains Across Treatment Groups Pre Post Gain Instant Word 75.2 (2h.9) 90.2 (28.h) 1h.9 (11.2) Recognition Decoded Word 8.h (6.7) 17.3 (11.0) 8.9 (12.0) Recognition Meaning 8.8 (h.9) 5.5 (h.8) -3.5 (3.6) Vocabulary Oral Reading 3.h (1.6) h.O (1.9) 0.53 (1.26) Silent Reading h.5 (1.5) 5.9 (l.h) l.h (1.5) Listening 3.2 (1.5) 3.9 (1.3) 0.7h (1.2h) Comprehension The first three rows give the numer of words recognized instantly, through mediation, and defined correctly respectively on the measurement task. The last three rows give direct grade level performances. These raw performance data indicates that although the diagnostic classifications did not change drastically, in fact, gains did occur. Instant word recognition gained 15 words which according to the SORT description is 3/h of a year grade equivalence. Decoding ability showed a good gain, doubling and adding almost an extra one-half year onto the students' word recognition ability. Silent reading comprehension gained a year. Realizing that these data are aggregated across treatment groups, let us now turn to the more interesting analysis of differences in treatment and gain. Mi DI 123 Retrospective Analysi . The next clinical analysis examines cause and effect by looking backward - a retrospective analysis. Groupings must be formed on the basis of outcomes and followed backward to differences in treatment, diagnosis, or other characteristics. For this analysis, one of the major questions of interest is the relationship between various aspects of the treatment and outcome. The MORAL model of reading emphasizes the importance of relevant engaged academic time for the improvement of the reading Critical Performances. The simplest measure of relevant engaged academic time is attendance. The 3h students remaining in the study at the end of the year were divided into various groupings depending upon their normalized gains on the MORAL Critical Performances: a score above zero was labeled high gain, a score below zero was labeled low gain. Similarly. students were separated based on high attendance and low attendance. The results are contained in Table 26. 12h Table 26 - Differences in Rate of Poor Attenders Based on Relative Outcomes Relative High Gain Relative Low Gain Total Gain 38:: (6 of 16) 592 (10 of 17) Instant Word 272 (h of 15) 632 (12 of 19) Recognition Decoded Word 302 (6 of 20) 712 (10 of 1h) Recognition Meaning Vocabulary 332 (5 of 15) 612 (ll of 18) Oral Reading . 352 (6 of 17) 592 (10 of 17) Silent Reading hh2 (8 of 18) 502 ( 8 of 16) Listening Comprehension 302 (7 of 23)‘ 752 ( 9 of 12) This table indicates unambiguously that gain for virtually all of the Critical Performances is dependent upon attendance. Only silent reading shows no tendency to relate attendance and gain when displayed in this way. This is most likely because very little instruction was aimed ’ specifically at comprehension of silent contextual reading. Given the numeric nature of both the performance gains (as opposed to diagnostic categorization gain) and the attendance, the obvious next step is to correlate attendance with performance gain. Such correlation are reported in Table 27. U llflF,c O). This is by far the most straightforward analysis relating outcome and treatment. As in the previous analysis, two presentations will be given, first the differences in diagnostic categorizations across treatment groups and, then, the differences in actual raw performances. 129 Table 29 Prevalence Rates Across Treatment Groups Pretest Rule Diagnosis Posttest Rule Diagnosis Instant Word Recognition Group None 6/6 5/6 Group Half O/h O/h Group Full Zh/Zh 23/2h Decoded Word Recognition Group None 5/6 3/6 Group Half 3/h 2/h Group Full 22/2h l3/2h Meaning Vocabulary Group None 2/6 5/6 Group Half 3/h k/h Group Full 1h/2h 20/2h Oral Reading . Group None 5/6 6/6 Group Half O/h O/h Group Full 23/2h 23/2h Silent Reading Group None 6/6 5/6 Group Half h/h h/k Group Full Zh/Zh 2h/2h Listening Comprehension Group None 5/6 h/6 Group Half Z/h 3/b Group Full 2h/2h Zh/Zh Looking over these data, several things are apparent. First, no student, no matter what the treatment, had much of a change in diagnostic categorization. The only significant drop in problems seems to be in decoded word recognition. In the group instructed for the full year, almost half of the deficits in decoding were eliminated. But notice that that same statement can be said about the Group None students who received no instruction! ' ”VF-'7 is 130 The second point is that the clinician was clearly correct in terminating instruction for the four students in the second group. This group was different from the other students treated for the full year, even in the initial prevalence of problems. In fact, it seems questionable that these students should have been retained for instruction at all, given the data on the non treated students. Although the instruction seemed to do little to change diagnostic categorizations, there may have been substantial change in raw performances. The differences in raw performances across treatment groups is described in Table 30. Instant Word Recognition Group None Group Half Group Full Decoded Word Recognition Group None Group Half Group Full Meaning Vocabulary Group None Group Half Group Full Oral Reading Group None Group Half Group Full Silent Reading Group None Group Half Group Full Listening Comprehension Group None Group Half Group Full 131 Table 30 Performance Gains by Treatment Group Pretest Performance 70.3 (26.1) 122.8 (3.5) 68.5 (17.2) 11.0 (6.0) 13 (10.0) 7.0 (6.0) 9.1 (5.5) 11.3 (6.0) 6.3 (11.0) 3.3 (1.8) 6.3 (0.5) 3.0 (1.2) 3.3 (1.0 3.3 (1.3) 2.8 (1.3) h.2 (1.2) 16.5 (1.7) 2.7 (1.3) Posttest Performance 77.2 (29.7) 1112.8 (13.0) 811.7 (19.6) 12.7 (11.5) 18.0 (9.0) 18.3 (11.3) 9.0 (7.8) 15.0 (11.5) 8.3 (11.5) (1.3) (1.3) (1.5) WNN Nmoo (1.1.) (1.0) (1.1.) MW: nootn (1.11) (1.3) (1.2) Dir-l" U‘CDU‘I Again, the raw performance data on instant word recognition, decoded word recognition, and meaning vocabulary are numbers of words recognized or defined on the performance test. The oral reading, silent reading, and listening data are grade equivalents. The raw performance data definitely present a different picture. First, the group of four students who received instruction for only half a year apparently learned quite a bit from the work. Their gain in 132 instant word recognition was a full year (using the SORT conversion factor of 20 words - 1 year). Their ability to decode started out higher than any other group and still increased by l/h years. The oral reading ability demonstrated by this group started out three grade levels higher than the other groups - yet they showed still the largest gain - over one year. Note also that their language ability started and ended higher than any other group - almost 2 year above the initial language ability of the other instructed students. It is mainly their silent reading comprehension that is problematic. The group of students who were instructed for the entire year also showed strong gains in word recognition. Although starting out significantly lower than the second treatment group, their instant word recognition ability was increased by 3/h years. Also their decoding ability was raised to the same high level as the second treatment group. They~ were able to increase their word recognition by almost 1 year through decoding. Finally, the no treatment group showed the lowest gains in all areas. Note that in only one Critical Performance did their abilities maintain pace with their grade placement. If the students here that received no instruction are, in fact, promoted to the next grade then they will begin that grade relatively further behind than they were the year before. The mean performance actually decreased for oral reading, probably because this performance continues to depend upon the ever more problematic word recognition ability. 133 Because of the small sample size, the differences in gain scores are not statistically significant at the 0.05 level with the single exception of the difference between the no instruction group and the full year of instruction group on gain in oral reading. However, the differences in initial performances of the groups are significant in a number of places. Summa y. The clinical analysis just completed raised the issue of definitions for treatment and outcome. The students in the study were divided into three treatment groups. Supposedly, the groups were to differ primarily on the amount of instruction received. However, the confounding variable of student attendance has in many cases confused this original categorization. There are students in Group Full (who supposedly received instruction all year) who in fact received less instruction than students in Group Half (who supposedly received instruction for only half a year). Clearly, actual treatment may differ substantially from intended treatment. But even ignoring the differences across treatments, the question remains as to whether or not the wide varieties of actual instruction should be treated as one treatment or as a set of closely related ones. Similarly, the definition of outcome is in question. The traditional educational approach would be to talk of gain scores on raw performance data. Note, however, that such an analysis masks the fact that diagnostic categorization changed little. 131. Other Analyses Grouping Decisions. The data from this study have so far been analyzed in the traditional methods of clinical and epidemiological research. Let us turn now to a content-area specific analysis, that of assigning students to reading groups. The first decision made by the clinician for which the collected data can provide guidance was the selection of the 30 students to receive instruction from the tested A2. Recall that of these AZ students, four students were eliminated from consideration for non-academic reasons. This means that eight students remained to be chosen for Group None, the no instruction group. The clinician's stated policy was to choose the eight students least in need of instruction for this group. If this were the actual decision criterion used, then the normalized scores for the eight students placed into Group None should have ranked 1-8, i.e., the best eight students would not receive instruction. Table 31 indicates the degree to which this was not true. Table 31 Decision: Choosing Students to Teach Ideal ranking 1 2 3 h 5 6 7 8 Ranking based on 2 A 5 7 10 11 1k 2k Total Performance Ranking based on 2 3 5 8 9 17 18 20 Word Recognition and Silent Reading 135 The first row shows ideal ranking given the stated policy; the second row shows actual policy based on total diagnosis: the third row shows actual policy based on word recognition and silent reading performance. Evidently, the first half of the children actually placed into Group None were chosen properly but the second half were not. The next decision was the splitting of the 30 students into 5 groups. The clinician, during interviews, stated that her grouping was intended to place people with the same reading level together. Rankings of total diagnoses were created for these 30 instructed students. Because of the managerial aspects of grouping decisions (e.g., conflicting class schedules), it is difficult to know how much the clinician's ideal groupings were bent by non-reading constraints. Regardless, there were six of the 30 students who were in groups substantially different from themselves (students who were the lowest or highest in their group and more than one-half standard deviations away ‘from the next closest group member). Both of these glimpses at the clinician's ability to informally rank student performance seem to indicate that immediate feedback of l“Elative class standings could change some very important instructional decisions. The only remaining concerns are those of the equivalence of a1ternate forms on the test battery and the reliability of various c1 1r1ical decisions and actions. First, consider the equivalence of a1 ternate forms . 136 Eguivalence 21 Alternate Forms. When the study was designed, it was decided that alternate forms of reading subtests were necessary.* Three forms of word lists and nonsense words were created by randomly reordering the words. Three alternate forms of the oral reading test (the Gray) were provided by the publisher. The design was set up so each student would use each form once. Each test session (pre-, mid-, or post-) had approximately 1/3 of the students on. each form. The students were randomly assigned to the form for pretest. Midtest and posttest forms for a student were determined by progression depending upon what test a student received at pretest (1-2-3 or 2-3-1 or 3-1-2). The allocation of forms for these measures is described in Table 32. Table 32 Distributions of Alternate Forms Test Pre Mid Post Total SORTl H: II: _ ll 39 SORTZ 13 13 12 33 SORT3 15 12 11 38 GMK 1 1‘1 111 11 39 GMK 2 13 13 12 38 GMK 3 15 12 11 38 GrayA 11: 1h 11 39 GrayB l3 13 12 38 GrayD 15 12 11 38 Statistical t-tests were run comparing the equivalence of these fur’ms to one another. For the various measures pulled from the SORT * Per the word lists, nonsense words, and oral reading paragraphs, this "<>" seems unnecessary given the long time span between the testing :ess ions and the type of performances measured. The test examiners all sgreed that student recall of the test materials between testing 933 ions for these problem readers seemed very unlikely. 137 (instant Iword recognition, mediated recognition, total recognition, and vocabulary correctness) there were no statistically significant differences at the 0.10 level for comparisons of form 1 to 2, 2 to 3, or I to 3. Similarly for the Gates-McKillop there were no statistically significant differences at the 0.10 level for comparing any pair of test forms on the measures of consonant cluster recognition, phonogram recognition, nonsense word recognition, or total recognition. The Gray versions were generally similar for comparisons between any pair of test forms on the last paragraph read, average reading time, average number of miscalls, pauses, punctuation errors, repetitions, or corrections. However, two seemingly significant differences were found. When comparing form 1 to form 3 for the Gray, the average number of miscalls per selection was statistically different at the 0.10 level (mean of 8.9 compared to 7.6). When comparing form I to form 2, average number of punctuation errors was statistically different at the 0.01 level (mean of 0.36 compared to 0.68). Given the small actual differences and the large number of t tests run, these two differences were ignored. It was assumed, therefore, that alternate forms of these three subtests (SORT, Gates-McKiIIOp, and Gray) were in fact equivalent. The design also required alternate forms for the comprehension measures. Here the assumed equivalence was more problematic. The MORAL defined silent reading comprehension and listening comprehension as two different modes of input for tapping the same semantic nets in memory. Therefore, not only would three alternate forms be needed for pre-, mid-, and post-test, but another three were desired to measure silent and listening comprehension with the same measures. Remember that no 138 existing silent reading test with graded paragraphs and enough alternate forms could be located. Therefore, the Spache with its two forms and the Durrell with its two forms were chosen and treated as equivalent forms for measuring silent reading comprehension and listening comprehension. As it turned out, the various forms of the silent and listening comprehension measures were substantially non-equivalent. This was primarily because the two forms of the Spache were substantially different from the two forms of the Ourrell. First, the major performances of interest on the silent reading measures were the highest selection read, the average reading time, the average percentage of total possible memories during sequential recall, the average percentage correCt on main idea questions, and the average percentage correct on inference questions.* These performances were substantially different when comparing the Ourrell forms to the Spache. They were not all that similar when comparing the two forms of the Spache to each other or the two forms of the Durrell to each other. Consider the measure "highest selection read." The means for these grade equivalent scores were A.59 (s.d. 1.22) for Spache A, h.21 (s.d. 1.52) for Spache form B, 6.13 (s.d. 1.01) for Durrell form A, and 6.17 (s.d. 1.39) for Durrell form B. The Durrell paragraphs have a mean that is statistically and meaningfully very different from that of the Spache, a difference of almost 2 years. To indicate the severity of this Problem, consider the fact that all four students who were ”graduated" * Recall that each comprehensions test consisted of a series of selections. For each selection a student had a reading time, a oercentage of possible memories, and a percentage of main idea and Inference question correctness. Clearly, these values can be averaged over selections. 139 from the reading program at midtest time were administered a form of the Spache as a pretest comprehension measure. They scored more than two years below grade placement and were accepted into the program. Other differences among forms are also apparent in Table 33. (Standard deviations are given in parentheses.) Table 33 Differences in Equivalent Forms: Silent Reading Spache Spache Durrell Durrell form A form B form A form B highest selection read b.59 h.21 6.13 6.17 (1.22) (1.52) (1.01) (1.39) average reading time 31.96 3h.57 30.58 31.85 per selection (12.89) (25.15) (8.26) (12.75) average percentage of 35.62 32.31 53.22 £5.17 sequential memories ( 15.81) (12.85) (11.92) (11.59) average percentage of h9.27 39.86 h8.25 hh.81 main idea (lh.63) (18.h7) (12.29) (17.72) average percentage of 51.79 59.23 68.52 56.33 inference (29.77) (39-33) (19.93) (25-73) Because of the extreme differences, the raw scores cannot meaningfully compared. In fact it is not known at this time, what form of what test (if any) is the most accurate indication of true silent reading performance. Therefore, following transformation will be applied to all raw data on silent reading performance. 1. Within each test session (pre-, mid-, post-) within each test (Spache A, B, Ourrell A, B), the raw scores will be standardized. 2. The standardized scores will be multiplied by the average standard form 1&0 deviation of the scores across the four test forms within a test session, and added to the mean of the scores across all test forms within a test session. This transformation assumes that the best estimate of actual silent reading performance for a particular testing session (pre-, mid-, or post-test), is the mean across all forms of a test during that test session. It further assumes that the best estimate for the standard deviation of actual silent reading performance for a particular test session is the mean of the standard deviations for the individual forms (NOT the standard deviation across all scores). All reports of silent comprehension measures were based on the data transformed in this way. Listening comprehension, then, had a similar problem. The performances of interest were the same as those for silent reading except that reading time was not a major concern since selections were read by the examiner not the student. Again, the performances were substantially different across the various forms of the listening comprehension data. 1h1 Table 3b - Differences in Equivalent Forms: Listening Spache Spache Ourrell Durrell form A form B form A form B average percentage of 32.78 23.81 h8.28 hh.83 sequential memories (11.h8) (08.52) (11.37) (10.50) average percentage of h5.33 3h.68 A3.65 £3.18 main ideas (11.53) (16.96) (12.17) (11.29) average percentage of h8.25 5h.60 6h.h8 56.2h inference (20.96) (17.95) (15.88) (15.36) Again, the raw scores are difficult to compare meaningfully. The analogous transformation that 2 was described for silent reading performances was applied to all raw: data on listening comprehension performance. All reports of these data were based on the transformed data. After these adjustments to the data were performed, the analysis of data based on the clinical and epidemiological research paradigm could proceed. Reliability 9: Data. The reliability of personal and environmental measurements, diagnostic categorizations, treatment selection and administration, and outcome measures is critical to the clinical and epidemiological research paradigms. Therefore, to as great an extent as possible, these concerns were examined here. .4' 152 The first reliability analysis consisted of reliability statements on the case and environmental measures. These measures were all part of the large multi-part test battery described elsewhere. Many of the subtests had multiple items and were, therefore, amenable to reliability estimates based on homogeneity. Aside from these measures, the only other data that commented directly on the question of reliability was a test-retest reliability using pre- and post-test data. Clearly this is a less than ideal estimate because specific instruction on the tested performance occurred between the two testing sessions, but this measure of reliability might serve as a lower bound on actual reliability. The data for these two measures of reliability on the MORAL Critical Performances raw data are described in Table 35. Table 35 Reliability of Critical Performance Measures Homogeneity Estimates: Lower Bound on Coefficient Alpha Reliability: Test-Retest Estimates: Pre-test to Post-test Instant Word 0.95 0.92 Recognition Decoded Word 0.90 0.16 Recognition Meaning Vocabulary - 0-73 Oral Reading - 0.76 Silent Reading 0.69 0.62 Comprehension Listening Comprehension 0.77 0.61 These data indicate that the reliability of the word recognition 11.3 measures is very high (0.9) while the reliability of connected text measures are only slightly lower (0.7). The only apparent discrepancy is the test-retest measure of decoding. In actuality, most of the students examined were brought up to some level of mastery on the decoding skills. Therefore, the alpha coefficient is a better estimate. These data point out something else. Because the test-retest reliability is high for all other performances, the instruction during the year did little to change the students' relative position. The second reliability concern is for reliability of diagnostic categorization. The approaches used in this study guarantee high reliability here. The decision rule diagnostic reliability and the computerized clinician's diagnostic reliability were essentially perfect: they were computerized procedures applied to raw data. The only source of error would be undetected typographical errors during data entry. The reliability of the written form diagnoses by the human clinician cannot be estimated here, but has shown to be as high as 0.7 in previous studies (Vinsonhaler, et al, in progress). For diagnostic categorization, then, the major limit on reliability seems to be the reliability of the test battery itself. The reliability of treatment measurement was the most problematic to determine. Measures of days attended are probably quite reliable. However, the reliability of measures of time on task are questionable. In the cramped quarters of the clinician's classroom, an attempt was made to_ estimate inter-observer reliability of the class activity measures. Subjective impressions are that the reliability was fair, but lhh no numbers are available. The very presence of the second observer in the room interfered with the perception of the first observer. As one observer shifted viewing from one part of the room to another, the other observer could not help but notice this and respond, thus, confusing the natural flow of the class. The only data reported here is for total time across 52 class sessions. Hopefully the variations among observers washed out with the large number of observation sessions. The final question of reliability deals with reliability of outcome. No direct measure of this reliability was possible. However, outcome was defined as the difference between pre- and post-test diagnoses or pre- and post-test raw performances. Both of these outcome definitions rely upon other measures with good reliability. Therefore, outcome measures themselves should be reliable. As a results of this analysis, it would seem that the major parts of the diagnostic and remedial activities were reliable and the clinical and epidemiological research paradigms could be applied. CHAPTER 5 Conclusions and Implications The basic problem that this thesis addressed was the methodology by which clinical problem solvers try to learn from experience. The typical method for such learning is the informal aggregation of experience in which everyone naturally engages. However, such informal learning can be problematic because of biases inherent in human thinking and the structure of typical clinical experience. As a result, the quality of clinical decisions can often be substantially less than that possible through systematic examination of actual experience. In some instances, informal aggregation of experience can lead to beliefs that are in direct opposition to those actually taught by the experience. ' The problems with informal learning from experience could exist in any clinical problem solving field. The field of medicine is one field in which such problems have existed in the past and still exist today. Medicine has, however, developed research paradigms that guide experience aggregation away from informal methods and more toward objective, scientific means. These paradigms are those of clinical and epidemiological research. The epidemiological paradigm is concerned with relating case (P), and environmental (E) characteristics to a diagnosis (0) while the clinical paradigm focuses on the relationships between diagnosis (D), treatment (T), and outcome(0). Both paradigms direct the careful systematic collection of data across different cases and clinicians and the analysis of the data in very specific ways. The key 1A5 1A6 to these paradigms is the existence of reliable diagnostic <:ategorizations and treatments that are differentially effective across these categorizations. Because these paradigms are based on the clinical garoblem solving model, it seemed likely that they could easily be generalized from medicine to other clinical fields. As practitioners and researchers in the field of medicine continue t1: improve the quality of medical care, another clinical problem solving f ield, the diagnosis and treatment of reading problems, is just beginning to grapple with conerns of how to learn most effectvely from experience. The obvious question that was raised was whether or not the czlinical and epidemiological research paradigm could be generalized to tflwe field of reading 'diagnosis and, if so, what would such a generalization reveal. The purpose of this thesis, then, was first to determine how to generalize the two medical research paradigms to the ‘field of reading and second to perform a research study guided by these paradigms searching for further understanding of reading diagnosis and treatment and for further understanding of what is required to make these formal methods of learning from experience work. Of course, it was not expected that a single such effort could rule on the adequacy of the research paradigms use in reading. However, it was expected to provide lessons in how to further examine their use in reading. The specific clinical and epidemiological study was carried out in the classroom of one seventh grade remedial reading teacher as she examined A2 children and taught 30 of these children throughout one school year. This setting did place many limitations on the generalizability of the results. The single greatest limitation was lack 1117 of variability. All of the children observed had reading problems of some form or another; most had severe problems. The clinician assumed that a single diagnosis was adequate to describe all students, that diagnosis being that the cause of most of the reading problems was a previous instructional deficit. This led to a single treatment plan for all students, specifically, carefully sequenced practice on isolated award recognition and on contextual reading using materials with increasingly more phonetic regularities. Treatment only differed with respect to entry level and pace. These various limitations on, all aspects of the clinical setting should be kept in mind as the results are reviewed. The major clinical and epidemiological data analyses described in the previous chapter are summarized in Table 36. Descriptive and Cross Sectional Retrospective Prospective ratings of mean length of utterance First consider the epidemiological began with the determination of prevalence 1A8 Table 36 Summary of Data Analysis Epidemiological Analysis P,E -> D *Prevalence rates of problems with Critical Performances *Agreement of different diagnostic sources across Critical Performances *Prevalence rates of diagnostic profiles of Critical Performances *Agreement of different diagnostic sources across diagnostic profiles *Differences in student self perception based on diagnostic categorization *Differences in student self perception based on diagnostic categorization stratified by sex *Differences in prevalence rates of diagnostic categorizations based on ratings of language ability *Differences in prevalence rates of diagnostic categorizations based on Clinical Analysis D -> T -> 0 *Days of instruction by treatment group *Relevant engaged academic time by treatment group *Changes in prevalence rates of Critical Performance problems after treatment across treatments *Distribution of gains in Critical Performances after treatment across treatment *Differences in attendance rates based on relative outcomes *Association attendance of gain and of outcome engaged *Association and relevant academic time *Differences in prevalence rate changes of Critical Performance problems after treatment by treatment *Differences in performance gains of Critical Performances after treatment by treatment data analysis. This analysis rates and profiles of 1A9 problems with the Critical Performances of the MORAL. The most obvious characteristic of the data initially was that virtually all of the students examined had severe problems with all six Critical Performances, i.e., problems with instant word recognition, decoded word recognition, meaning vocabulary, oral reading, silent reading, and listening comprehension. Three different diagnostic sources were used and all three were virtually unanimous in this judgement.* Although there was substantial agreement on this diagnostic assessment of the students, closer examination of the three types of diagnoses brought several facts to light. The three diagnostic sources generally agreed: the clinician and decision rule diagnoses were especially close. Across these three sources, however, there was an occasional sharp disagreement on the prevalence of a particular diagnostic problem. Closer inspection indicated that substantial differences in the definition of diagnostic problems existed. An example of such differences was the judgement on silent reading comprehension. The clinician and the decision rule considered a student's silent reading comprehension to be problematic if it was more than one year below grade placement: the computer simulated clinician considered it problematic if it was more than one year below word recognition level. In a very real sense, neither of these is more correct than the other. Either one of the definitions could be agreed upon as the "correct" one. What these distinctions highlight is that the precise definition of a behavior as well as a criteria for determining its adequacy are a * The three types of diagnoses were (l)those generated by the reading clinician guided by a written decision aid, (2)those resulting from the‘ computer application of simple decision rules to raw performance data, and (3)those given by a complex computer simulated clinician. 150 necessary prerequisite for clinical and epidemiological research. Maybe both of these diagnostic perceptions are valuable and should exist as separate diagnostic categories, but such decisions must be made consciously. Another point of interest confirmed by the epidemiological data is the human tendency (or requirement) to simplify a problem. There were 25 different diagnostic profiles generated by the three diagnostic sources. Of these 25, the clinician used six, the decision rule used 11, and the ' computer simulated clinician used 13. Apparently the clinician viewed the placement of the students into diagnostic profile equivalence classes from a simpler perspective than the other two diagnostic sources. Given the 11 categories used by the rule diagnoses and the elementary and non-inferential nature of the rules, these 11 categories are probably the best estimate of the number of actual distinctions. The human clinician used half of that. Of course, one possibility is that somehow the human clinician had already performed another round of grouping and had collapsed different diagnostic profiles into treatment equivalent profiles. However, with the separate applications of a written form to each case, and with the form requiring independent judgements on each Critical Performance, this seems unlikely. Given the reduction of the problem space for the clinician, one last implication is that actually different diagnostic classes could be incorrectly grouped together. Recalling the description in Chapter Two detailing the results of unreliable diagnoses, it is clear that such a reduction of the problem space when it is not warranted can result in errors in estimating the effect of treatment on these diagnostic 151 categories that are nominally the same. As the clinician makes this simplification, it is possible that valuable distinctions become lost. Only more empirical study could determine if this were true. As the study continued with the retrospective analysis of differences in personal or environmental characteristics given differences in diagnoses, several things were evident. First, for any epidemiological analysis, much more information than was collected here is required on non-problematic cases. Although in this study there were, in fact, students without a problem for any particular Critical Performance, no student examined was without problem. In this study, those that were healthy in some Critical Performance not only had problems with other ones but were also few in number. No one examined had good silent reading comprehension. The data indicated that there was a relationship between self perception and performance. There were more students who had a correct self perception of a given Critical Performance than there were students who had an incorrect self perception. When the technique suggested from epidemiology called standardization was applied to these data using the student's sex, the resulting stratified data indicated that virtually all of the incorrect self perceptions were voiced by the male students. Whether their actual perceptions were so wrong or they felt it more necessary to put up a front is unknown. In either case, standardization proved to be a straightforward technique and apparently worked well. The last epidemiological analysis was a prospective analysis of language ability and diagnostic categorization. Language ability was 152 measured in two ways: after detailed observation of reading performances the examiner was asked to rate the student's language ability as high, average, or low, and to estimate the student's mean length of utterance as multiple sentence, single sentence, phrase, or word. The association between language ability and problems with Critical Performances was pronounced. In fact, 1002 of all students rated with low language ability were problematic in almost every way and further diagnoses were actually redundant. Just as important as the strong relationship between language experience and diagnosis is the comparative lack of relationship between mean length of utterance and diagnosis. The expectation had been that these two measures would give identical results. That they did not indicates that the first relationship was more than just some artifact of the rating procedure. The clinical analysis of data began with descriptive measures of treatment and outcome across groups. The first problem raised by this analysis is similar to the problem raised by descriptive analysis of epidemiological data: problems of definition. The students in the study were nominally receiving the same treatment within each group. In fact, however, the treatments were substantially different because of student attendance. A factor of five separated the amount of instructional time received by the lowest class attender and the highest (203 minutes to 1053 minutes during the first half term for the group instructed for the full year). Treatments with differences of this magnitude cannot be called identical treatments. Much like medical treatments which require adherance to a schedule of 153 activities, instruction in this study required attendance. Obviously, many students could not meet this requirement. The question remains, then, when are treatments equivalent. The second question of definition that the descriptive analysis raised concerns the definition of outcome or gain. For this sample of students across the treatment groups, very little change in diagnostic categorization occurred between pre-test and post-test. However, noticable gain did exist when the raw performance data were examined. Although not significant enough to change drastically diagnostic categorization, the passing of a year did alter actual performance. Retrospective analysis of the clinical data indicated a strong relationship between relative class attendance and relative outcome (relative meaning compared to the rest of the A2 students). Without exception, the percentage of low attenders was smaller for high gain groupings and larger for low gain groupings. When actual correlations between raw performance gain scores and class attendance were computed, all correlations were in the right direction (more attendance give more gain): correlations for total gain and decoding skill gain with attendance were statistically different from zero. (This raises the question of whether or not patterns of data can be found to predict class attendance and used to choose which students receive some of the limited resources of remedial reading assistance.) Similar patterns were discovered when attendance was broken apart into the more fine-grained measures of relevant engaged academic time. This analysis also indicated that total time was a major determiner of total gain. Although none of the results were statistically significant, all relationships between 151. time and gain were in the proper direction. One note: there was little relationship between silent reading gain and time on tasks, but few tasks in class were relevant to silent reading comprehension. The final clinical analysis compared outcomes across treatment groups. This analysis indicated that treatment versus no treatment was a significant difference. The students who received no instruction averaged less than one year growth in all Critical Performances except silent reading comprehension. This means that at the beginning of next year, these students will start the year relatively farther behind than they were this year. The differences between the groups instructed for half year and full year were also large, but not so much in terms of gain as in terms of initial diagnosis. Remember that these two groups started out together and the students instructed for the half year were "cured" in the clinician's judgement at that time. It is not surprising, then, to discover that the initial diagnoses for these students were substantially higher than for the other treated students. Perhaps the largest difference is in entering listening comprehension ability: an average of A.5 grade equivalence for the group cured by half year and an average of 2.7 for the others. Continuing this comparison between the two instructed groups brings out another significant fact: the final performance of the two treated groups in terms of the amount of word recognition that decoding added to their instant recognition was essentially identical. After instruction both groups were able to raise their word recognition level by almost 155 one year with decoding. Both groups started out significantly worse than this. Such consistent gains were not seen elsewhere, so apparently, the treatment provided by the clinician is differentially effective for decoded word recognition. Across all of the treatment groups, one last point was re-emphasized by the prospective analysis, namely, differences in gain on raw performance measures are not necessarily reflected in differences in diagnostic categorizations. Specifically, although most of the treated students did show gain, even substantial gain, ALL STUDENTS STILL HAD READING PROBLEMS. At the end of the year, every one of these students still had problems with at least one Critical Performance in reading. The last specific analysis of data concerned the equivalence of alternate forms and the reliability of clinical measurements, decisions, and actions. Suffice it to say that problems with the actual non-equivalence of some alternate forms were accounted for and that the reliability of the personal, diagnostic, treatment, and outcome characteristics was adequate. The final question that remains is the implications of this research for reading and its clinicians and policy makers. First of all, it appears that the question of whether or not the clinical and epidemiological research paradigms work in reading has been replaced by a number of other questions that must be answered first dealing with triings like the ability to have reliable diagnostic categorizations and (Sifferential treatments. If these can be solved, it appears that the 156 single most difficult problem in generalizing these approaches to reading may deal with the practical matters of performing such studies. It requires work on the part of clinicians and reliability at all levels. Existing microcomputer technology could assist with the careful collection of data, the statement of diagnoses, the suggestion of treatments, and the measurement of outcomes. But what benefit is provided to the clinician? Indications from the clinician studied here are that the concerned clinician can act to make major potential improvements in treatment of ‘problem readers given a minimal amount of formally aggregated empirical information. Long before this report was completed, our clinician altered her teaching methods given the small piece of information that oral reading gains were very slight among her students. This empirical information was in opposition to what her informal perceptions of the situation dictated. With this information, she changed instruction. Aside from this type of information, the amount of data available to clinicians early in the development of a clinical and epidemiological data bank is small. At this time, no information is available on gains from other treatments on similar or different types of children. At this point we have data on only a very limited set of readers, with far too ‘few good readers in the set. The question remains how the results of the (data reported here would change given a larger sample of readers. Although the data available to individual clinicians may be limited iriitially, as the formal data base of clinical experience builds up, Imany other types of assistance would be possible. The varieties and 157 likelihoods of diagnostic profiles would be available. The empirical association of these profiles with measurement devices could lead to more reliable and valid diagnoses. Given particular diagnostic categorizations, prognoses could be determined and appropriate treatment plans suggested. As the differential effectiveness of various treatments becomes known, sequences of treatments could be prescribed, each working on a separate reading ability. Then the outcomes of each new case would serve to confirm or deny the existing practice and objective learning can OCCUI’ a This same type of information could also prove valuable to policy makers. For example, none of the children examined here had adequate silent reading comprehension at the end of the year. They will receive no special instruction next year so how will their reading ability survive? If their problem with silent reading starts to bring the other Critical Performances down with it, was it reasonable to treat them at all? Reading is very different from medicine. For almost any medical problem, the patient has armies of microscopic allies built into the body that can battle many problems and bring the patient to full health without assistance. There is a natural state called "healthy" in medicine. In reading, it is just the opposite. If anything, the psychological aspects of a poor reader are fighting on the other side. Even the data here indicated the relationship between poor self concept (and problems. Is it reasonable, then, to start curing a problem reader 1~hen one cannot complete the task? 158 Given empirical data on students, then, one can answer such questions. A policy maker might well decide that, with limited resources, it makes more sense to treat the readers with only moderate problems first and "save” their reading ability because it is saveable than to expend resources on children whose reading will only deteriorate again when instructional support is removed. The central concept here is to look for a change in diagnosis rather than a change in performance. Perhaps the last statement here should be a request that researchers and clinicians in reading (and possibly any other clinical problem solving field) at least consider the possibility of developing a clinical and epidemiological research program for the formal aggregation of clinical experience. The results of such work in medicine has been impressive: the potential for similar results in reading seem promising. Although the outcomes in terms of improved clinical problem solving cannot be eStimated without actually performing clinical and epidemiological research, the techniques of learning from experience by careful systematic recording and aggregation of the characteristics of that experience have an alluring common sense about them. To quote Roberts (1977, p.A) once again, clinical and epidemiological research techniques provide "possibly, the only methodological framework within vflwich science and rationalism can be integrated into the clinical inanagement of individual patients.” LIST OF REFERENCES List pi References Alderson, M. 5p introduction pp epidemiology. London: MacMillan Press, LTD, 1976. Anderson, L.M., Evertson, C.M., 8 Brophy, J.E. An experimental study of effective teaching in first-grade reading groups. The Elementary School Journal, 1979, 19, 193-223. Barnoon, 5., 8 Wolfe, H. Measuring ppp effectiveness pf medical decisions. Springfield, Illinois: Charles C Thomas, 1972. Benard, J. The medical decision. In DeDombal, F.T., 8 Gremy, F. help?. New York: North-Holland Publishing Company, 1976. Bender, G.A. Great moments 1p medicine. Detroit: Parke, Davis, 8 Company, 1961. Bloom, B.S. Human characteristics and school learning. New York: McGraw-Hill Book Company, 1976. Borg, W.R., 8 Gall, M.D. Educational research. New York: Longman, Inc., 1979. Brophy, J.E., 8 Evertson, C.M. Learning from teaching: A developmental perspective. Boston: Allyn and Bacon, Inc., 1976. Burdette, W.J., 8 Gehan, E.A. Planning and analysis 91 clinical studies. Springfield, Illinois: Charles C Thomas, 1970. Calfee, R. Cognitive psychology and educational practice. In D.C. Berliner (Ed.),Review pi research ip educationI 9. American Educational Research Association, 1981. Calfee, R., 8 Drum, P.A. How the researcher can help the reading teacher with classroom assessment. In L.B. Resnick 8 P.A. Weaver (Eds.),Theory ppg practice 9: early readingI volume ;. Hillsdale, New Jersey: Lawrence Earlbaum Associates, Publisher, 1979. Carroll, J.B. . Teacher's College Record, 1963, 55, 723-733. (Shall. J.S. Learning pp read: The great debate. New York: McGraw-Hill Book Company, 1967. 159 160 Ciba Foundation Symposium pp research and medical practice: Their interaction. New York: Elsevier/Excerpta Medica/North Holland, 1976. Coleman, V. Paper doctors: A critical assessment pi medical research. London: Temple Smith, 1977. Collen, M.F. (Ed.). Multiphasic health testing services. New York: John Wiley 8 Sons, 1978. Collen, M.F., Ruben, L., Neyman, J., Dantzig, G.B., Baer, R.M., 8 Siegelaub, A.B. Automated multiphasic screening and diagnosis. 53, 781-750. — — — Collen, M.F., Dales, L.G., Friedman, G.D., Flagle, C.D., Feldman, R., 8 Siegelaub, A.B. Multiphasic checkup evaluation study: A.Preliminary cost benefit analysis for middle-aged men. Preventative Medicine, 1973, 2, 236-2A6. Cutler, P. Problem solving ip clinical medicine: From data pp diagnosi . Baltimore: The Williams 8 Wilkins Company, 1979. Davis, F.B. The literature pi research 1p reading with emphasis pp models: Final repor . East Brunswick, New Jersey: IRIS Corporation, 1970. DeDombal, F.T., 8 Gremy, F. (Eds.). Decision making app medical care: Can information science help?. New York: North-Holland Publishing Company, 1976. DeDombal, F.T., 8 Horrocks, J.C. Computer aided diagnosis: Conclusions from an overall experience involving AA69 patients. In J. Anderson 8 J.M. Forsythe (Eds.),MEDINFO 192A. New York: American Elsevier Publishing Company, Inc., 197A. DeDombal, F.T., Leaper, D.J., Horrocks, J.C., Staniland, J.R., 8 McCann, A.P. Human and computer-aided diagnosis of abdominal pain: Further report with emphasis on performance of clinicians. British Medical Journal, l97A, 1, 376-380. DeGowin, E.L., 8 DeGowin, R.L. Bedside diagnostic examination, third edition. New York: MacMillan Publishing Co., Inc., 1976. Duffy, G.G. ”Fighting off the alligators: What research in real classrooms has to say about reading instruction”, State of the Art Conference, November 23, 1981, State University of New York at Albany. Duffy, G.G., 8 Roehler, L.A. ”An analysis of the instruction in reading instructional research”, National Reading Conference, December 2, 1981, Dallas, Texas. Durmin, M.T., 8 Biddle, B.J. The study pi teaching. New York: Holt, 161 Rinehart, and Winston, Inc., 197A. Einhorn, H.J. Learning from experience and suboptimal rules in decision making. In T.S. Wallsten (Ed.),Cognitive processes 1p choice and decision behavior. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Publisher, 1980. Einhorn, H.J., 8 Hogarth, R.M. Confidence in judgement: Persistence of the illusion of validity. Psychological review, 1978, 8 , 395-Al6. Ekwall, E. Diagnosis and remediation p: the disabled reader. Boston, Massachusetts: Allyn and Bacon, 1976. Elstein, A.S., Shulman, L.S., 8 Sprafka, S.A. Medical problem solving: Ap analysis pi clinical.reasoning. Cambridge, Massachusetts: Harvard University Press, 1978. approach. New York: Churchill Livingstone, 1975. Feinstein, A.R. Clinical biostatistics. St. Louis: The C.V. Mosby Company, 1977. Fisher, C.W., Filby, N.N., Marliave, R. Cahen, L.S., Dishaw, M.M., Moore, J.E., 8 Berliner, D.C. Teacher behaviors, academic learning time, ppp student pchievement: Final report pi phase iii-b, beginning teacher evaluation study. San Francisco: Far West Laboratory for Educational Research and Development, 1970. Fox, J.P., Hall, C.E., 8 Elveback, L.R. Epidemiology: Man and disease. New York: The Macmillan Company, 1970. Friedman, G. 0. Primer p: epidemiology, second edition. New York: McGraw-Hill Book Company, 1980. FFY. E.B. Reading instruction for classroom and clinic. New York: McGraw-Hill Book Company, 1972. Gagne, R.M. The conditions p: learning. New York: Holt, Rinehart, and Winston, 1977. Garrison, F.H. 5p introduction pp the history p: medicine, fourth edition. Philadelphia: W.B. Saunders Company. 1929. 511. D. Ip_ clinical problem solving behavior of classroom teachers _p they diagnose a child's reading performance in laboratory and classroom. Unpublished doctoral dissertation, Michigan State University, 1979. 900d. T.L., Biddle, B.J., 8 Brophy, J.E. Teachers make p difference. New York: Holt, Rinehart, 8 Winston, 1975. 500d. T.L., 8 Brophy, J.E. Looking 1p classrooms. New York: Harper, 162 Row, Publishers, 1973. Gram, R.R. Problem solving, systems analysis, and medicine. Springfield, Illinois: Charles C Thomas, Publisher, 1972. Hare, V.C., Jr. Systems analysis: A diagnostic approach. New York: Harcourt, Brace, 8 World, Inc., 1967. Hoffmeyer, L. The process and outcome pi diagnostic problem solving among reading clinicans. Unpublished doctoral dissertation, .Michigan State University, 1980. Holland, W.W. Data handling ip epidemiology. London: Oxford University Press, 1970. Holland, W.W., 8 Karhausen, L. (Eds) Health care and epidemiology. London: Henry Krimpton Publisher, 1978. Keen; P.G.W., 8 Morton, M.S.S. Decision support systems: A_ organizational perspective. Reading, Massachusetts: Addison-Wesley Publishing Company, 1978. Kurland, L.T., 8 Molgaard, C.A. The patient record in epidemiology. Scientific American, 1981, 2A5(A), 5A-63. Ipsen, J. Use of vital statistics. In W.W. Holland 8 L. Karhausen (Eds.),Health care and epidemiology. London: Henry Krimpton Publishers, 1978. Jackson, P.L. Life in classrooms. New York: Holt, Rinehart, and Winston, Inc., 1 68. Lange, H.J. Decision-making by doctors aided by computers. In F.T. DeDombal 8 F. Gremy (Eds.),Decision making ppp medical care: p52 information science help?. New York: North-Holland Publishing Company, 1976. Leaper, D.J., Horrocks, J.C., Staniland, J.R., 8 DeDombal, F.T. Computer-assisted diagnosis of abdominal pain using "estimates” provided by clinicians. British Medical Journal, 1972, 3, 350-35“- Leibert, R.E. (Ed.). Diagnostic viewpoints ip reading. Newark, Delaware: International Reading Association, 1971. Lilienfeld, A.M., 8 Lilienfeld, D.E. Foundations pi epidemiology, second edition. New York: Oxford University Press, 1980. Lilienfeld, D.E., 8 Lilienfeld, A.M. Epidemiology: A retrospective study. American Journal pi Epidemiology, 1977, 19p, AA5-A59. Lusted. L.B. Introduction ip medical decision making. Charles C Thomas, Springfield, Illinois, 1968. 163 Mausner, J.S., 8 Bahn, A.K. Epidemiology: pp introductory text. Philadelphia: W.B. Saunders Company, 197A. Mead Johnson Strategies ip clinical research. Evansville, Indiana: Mead Johnson, 1976. making pp p slippery disease. U.S. Department of Health, Education and Welfare, 1978. Nossal, G.J.V. Medical science and human goals. London: Edward Arnold, 1975. Peterson, P.L., 8 Walberg, H.J. Research pp teaching: Concepts, findings, ppp implications. Berkley, California: McCutchan Publishing Corporation, 1979. Roberts, C.J. Epidemiology for clinicians. Tunbridge Wells: Pitman Medical Publishing Company, 1977. Simon, A., 8 Boyer, E. Mirrors for behavior: pp anthology pi observation instruments. Philadelphia: Research for Better Schools, Inc., 1967. Simon, A., 8 Boyer, E. Mirrors for behavior: pp anthology pi observation instruments continued, 1920 suppliment, volumes 3 and p. Philadelphia: Research for Better Schools, Inc., 1970. Sherman, G.B. Untitled manuscript. In preparation, 1982-1983. Siegelaub, A.B., Friedman, G.D., Collen, M.F., 8 Kodlin, 0. Research applications of MHTS. In M.F. Collen (Ed.),Multiphasic health testing services. New York: John Wiley 8 Sons, 1978. Singer, H., 8 Ruddell, R.B. Theoretical models and processes pi reading, second edition. Newark, Delaware: International Reading Association, 1976. Small, W.P., 8 Krause, U. pp introduction pp clinical research. Baltimore: The Williams 8 Wilkins Company, 1972. Smith, F. ppperstanding reading; A psycholinguistic analysis pi reading and learning pp read. New York: Holt, Rinehart, 8 Winston, Inc., 1971. sDache. G.D. Diagnosing and correcting reading disabilities. New York: Allyn 8 Bacon, Inc., 1976. Spache, G.D. Investigating the issues pi reading disabilities. New York: Allyn 8 Bacon, Inc., 1976. Spache, G.D., 8 Spache, E.B. Reading ip the elementary schooL, third edition. New York: Allyn 8 Bacon, Inc., 1973. 16A observation and teaching models. Belmont, California: Wadsworth Publishing Company, Inc., 1977. Stratoudakis, J. The construction and ppp pi paper problems pp observe the diagnostic problem solving behavior pi reading clinicians. Unpublished doctoral dissertation, Michigan State University, 1980. Sullivan comprehension readers. Palo Alto, California: The Learning Line, 1980. Tversky, A., 8 Kahneman, D. Judgement under uncertainty: Heuristics and biases. Science, l97A, 185, 112A-ll3l. VanRoekel, J. The problem solving behavior pi reading specialists and learning disabilities resource teachers. Unpublished doctoral dissertation in progress, Michigan State University, . Vinsonhaler, J.F., Weinshank, A.B., Wagner, C.C., 8 Polin, R. Diagnosing children with educational problems: Characteristics of reading and learning disability specialists and classroom teachers. Reading Research Quarterly, In preparation, ppp approximately fall, 1982. Vinsonhaler, J.F., Weinshank, A.B., Polin, R., 8 Wagner, C.C. Model based training to improve diagnostic reliability in reading. Planned for Reading Research Quarterly, In preparation, ppp ppproximateiy spring, 1983. Vinsonhaler, J.F., Weinshank, A.B., Cureton, D., 8 Blatt, G. A model pi reading and learning pp read. Unpublished manuscript, I980. Weinshank, A.B. pp observational study pi the relationship between diagnosis and remediation ip_reading. Unpublished doctoral dissertation, Michigan State University, 1980. Wilson, P.D., Horrocks, J.C., Lyndon, P.J., Yeung, C.K., Page, R.E., 8 DeDombal, F.T. Simplified computer-aided diagnosis of acute abdominal pain. British Medical Journal, 1975, p, 73-75. Wulff. H.R. Rational diagnosis and treatment. Oxford: Blackwell Scientific Publications, 1976. Yerushalmy, J. The statistical assessment of the variability in observer perception and description of roentgenographic pulmonary shadows. In W. Winkelstein Jr. 8 F.E. French (Eds.),Basic readingp ip epidemiology. New York: MSS Information Corporation, 1972. References: Reading Tests Ourrell, D.B. Durrell analysis pi readipg difficulty: new edition. Harcourt, Brace, 8 World, Inc., 1955. Gates, A.J. 8 McKilIop, A.S. Gates-Mckillgp reading diagnostic tests. Teachers College Press, 1962. Gray, W.S. Gray oral reading tests. Bobbs-Merrill Company, Inc., 1967. Slosson, R.L. Slosson oral reading test (SORT). Slosson Educational Publications, Inc., 1963. 165 APPENDICES Appendix pi Test Battery Stimulus Materials The following pages include the stimulus materials for the measurement device. The students in the junior high school class used in the study had these materials in front of them, covered in plastic and in a three ring notebook. The notebook contained ' materials for all alternate forms of each subtest. The test examiner would direct the student to appropriate sections of the notebook. The notebook contains A3 pages. The contents presented in this appendix are identical to that used in the study except that the format has been altered slightly. 166 moowa‘ma-uN-a IO. 11. 12. 13. 1A. 15. 16. 17. 18. 19. 20. 61. 62. 63. 6A. 65. 66. 67. 68. 69. 70. 71. 72. 73- 7A. 75. 76. 78. 79- 80. . jump want look play come is can one see ball little three baby run down mother here uP help make hunger evening bench excuse grove desire perform destroy delicious understood safe against timid ocean damp stream . empty smash stone reward 21. 22. 23. 81. 82. 83. 88. 85. 86. 87. 88. 89. 9o. 91. 92. 93. 9A. 95- 96. 97. 98. 99- 100. 167 very under hill . road . dark . horse came PUPPY was bump with . ride along . friends first . food wish . what . basket live claimed forehead serious speechless distant dainty anger courage slumber common harness flakes develop vacant price future appearance promptly region silence A1. A2. A3. AA. A5. A6. A7. A8. A9. 50. 51. 52. 53- 5A. 55- 56. 57. 58. 59- 60. 101. 102. 103. 10A. 105. 106. 107. 108. 109. 110. 111. 112. 113. 11A. 115. 116. 117. 118. 119. 120. better hide large field around suddenly forest river sheep station breakfast across heavy lunch happen grass game farmer stars hope dignity generally gracious applause define cushion jungle haze profitable tailor fragrant obedient marriage presence terrace interfere ambition merchant custom extended 121. 122. 123. 12A. 125. 126. 127. 128. 129. 130. 131. 132. 133. 13A. 135. -l36. 137. 138. 139. 1A0. 181. 182. 183. 18A. 185. 186. 187. 188. I89. 190. 191. 192. 193. 1911. 195. 196. 197. 198. 199. 200. tremendous dungeon customary abundant malicious imaginary excellence consequently installed responsible yearning liquid medicine spectacular rebellion compliments infected inventory importance detained sojourn affable auspicious enamoured futility primordial inadequacy chastisement envisage pandemonium panorama compressible gustatory traverse simultaneous excruciating contraband decipher scrupulous facsimile 1A1. 1A2. 1A3. 1AA. 1A5. 1A6. 1A7. 1A8. 1A9. 150. 151. 152. 153. 15A. 155. 156. 157. 158. 159. 160. I68 reminiscence architecture environment counterfeit pensive tremor intricate administer continuously attentively industrious contemporary approximate standardize complexion exhausted malignant compassionate crisis society I61. 162. 163. 16A. 165. 166. 167. 168. 169. 170. 171. 172. 173. 17A. 175. 176. 177. 178. 179. 180. nucleus proportional articulate remarkably nonchalant memorandum deprecate contrasting whimsical evident prairies twilight intangible supplement formulated exuberant antique inducement irrelevance grotesque mN0\U1-FWN—fi 9. 10. 11. 12. 13. 1A. 15. 16. 17. 18. 19. 20. 61. 62. 63. 6A. 65. 66. 67. 68. 70. 71. 72. 73- 7A. 75- 76. 77- 78. 79. 80. . one little ball . play down mother see run baby Jump look make three is here help come want can up stone against ocean grove smash stream damp safe . excuse understood perform empty hunger reward destroy desire delicious evening bench timid 21. 22. 23. 2A. 25. 26. 27. 23. 29. 30. 31. 32. 33.. 3A. 36. 37. 38. 39- A0. 99- 100. 169 along with bump hill wish under horse came very food PUPPY was basket ride . road live first what dark friends . forehead . serious claimed . flakes . slumber promptly . courage harness . distant . vacant . anger . dainty . develop . appearance . silence . future . speechless . region price common 101. 102. 103. 10A. 105. 106. 107. 108. 109. 110. 111. 112. 113. 11A. 115. 116. 117. 118. 119. 120. lunch . hide field . heavy suddenly across river . hope game sheep breakfast happen . grass . around . forest station . stars . better large farmer fragrant tailor interfere terrace generally define custom ambition dignity applause extended merchant haze marriage cushion jungle gracious profitable presence obedient 121. 122. 123. 12A. 125. 126. 127. 128. 129. 130. 131. 132. 133. 13A. 135. 136. 137. 138. 139. 1A0. 181. 182. 183. 18A. 185. 186. 187. 188. 189. 190. 191. 192. 193. 19A. 195. 196. 197. 198. 199. 200. malicious yearning inventory installed customary consequently detained medicine dungeon compliments rebellion imaginary excellence abundant spectacular infected liquid importance responsible tremendous primordial futility enamoured decipher chastisement traverse contraband compressible scrupulous auspicious simultaneous inadequacy panorama excruciating sojourn envisage affable facsimile pandemonium gustatory 1A1. 1A2. 1A3. 1AA. 1A5. 1A6. 1A7. 1A8. 1A9. 150. 151. 152. 153. 15A. 155. 156. 157. 158. 159. 160. 170 standardize counterfeit tremor complexion environment approximate exhausted attentively pensive contemporary industrious administer compassionate architecture reminiscence crisis continuously malignant intricate society 161. 162. 163. 16A. 165. 166. 167. 168. 169. 170. 171. 172. 173. 17A. 175. 176. 177. 178. 179. 180. evident proportional contrasting memorandum grotesque prairies antique formulated articulate supplement twilight nonchalant whimsical intangible inducement exuberant deprecate remarkably nucleus irrelevance moowaxma-wN—u o 10. 11. 12. 13. 1A. 15. 16. 17. 18. 19. 20. 61. .62. 63. 6A. 65. 66. 67. 68. 69. 70. 71. 72. 73- 7A. 76. 77. 78. 79. 80. play make three help baby down is can UP one run want here ball Jump little come mother look stream desire evening understood destroy delicious ocean grove excuse damp hunger bench empty timid . smash perform stone safe against reward 99- 100. 171 . road . basket live . very . bump . ride . first . wish . what . horse . along . dark . came . with . under . food . hill - PUPPY . friends . was . dainty . price courage . develop .‘forehead . serious . flakes . distant . appearance . anger . promptly . speechless . common . slumber . region . future . harness . claimed vacant silence A1. A2. A3. AA. A5. A6. A7. A8. A9. 50. 51. 52. 53. 5A. 55. 56. 57- 58. 59- 60. 101. 102. 103. 10A. 105. 106. 107. 108. 109. 110. 111. 112. 113. 11A. 115. 116. 117. 118. 119. 120. station breakfast sheep field better heavy around river forest suddenly lunch happen large hope grass farmer hide stars game across extended define profitable obedient tailor applause gracious marriage interfere generally presence jungle terrace ambition custom cushion fragrant haze merchant dignity 121. 122. 123. 12A. 125. 126. 127. 128. 129. 130. 131. 132. 133. 13A. 135. 136. 137. 138. 139. 1A0. 181. 182. 183. 18A. 185. 186. 187. 188. 189. 190. 191. 192. 193. 19A. 195. 196. 197. 198. 199. 200. responsible malicious detained installed inventory imaginary excellence spectacular abundant tremendous medicine consequently yearning importance compliments dungeon liquid customary infected rebellion envisage pandemonium futility affable contraband panorama traverse enamoured compressible excruciating inadequacy ‘ chastisement auspicious decipher primordial simultaneous sojourn facsimile gustatory scrupulous 1A1. 1A2. 1A3. 1AA. 1A5. 1A6. 1A7. 1A8. 1A9. 150. 151. 152. 153. 15A. 155. 156. 157. 158. 159. 160. 172 complexion crisis approximate tremor intricate compassionate contemporary society standardize exhausted malignant reminiscence industrious pensive architecture environment continuously attentively administer counterfeit 161. 162. 163. 16A. 165. 166. 167. 168. 169. 170. 171. 172. 173. 17A. 175. I76. 177. 178. 179. 180. prairies nucleus antique contrasting twilight articulate memorandum supplement deprecate exuberant grotesque irrelevance nonchalant proportional formulated whimsical remarkably inducement evident intangible 3.st A.sl 5.ch 6.cr 7.gl 10.dw ll.pr 12.cl l3.wr 1A.th 15.tw 16.5w l7.wh 18.61 19.sh 20.dr 21.fl 22.tr 23.p1 2A. 25. 26. 27. 28. 29. 30. 31 32. 33- 3A. 35- 36. 37 38.‘ 39- A0. A1 A3. AA. A5. A6. idge er ate ite ind een ible .ock ell emp ickle ed .ew 1ck ark asp Coy A2. ight able ing ack 173 A7 A8. A9. 50 51 52 53. 511. 55. 56. 57 58. 59- 60. 61 62 63. 6A 65 66. 67 68. 69. .stind SNOW chible .whickle .proy .dwer twasp shable flark thate trock swite gright .plew .glemp clidge .bleen .drack sked .wrick fring crell 3.ch A.tw 7.dw 8.cl 9.bl 10.th ll.wh 12.sm l3.sw lA.sl 15.sh 16.fr 17.cr 18.tr l9.pl 20.fl 21.pr 22.sk 23.91 2A. 25. 26 27 28 29.’ 30 31 32. 33 3A 35- 36.’ 37 38. 39 A0. A1 A2 A3. AA A5 A6 ight idge .emp .er .ack .een .ark ickle .oy .ew asp 1ck .ed ind .ome ing .ock .ate ible .OW .able .ell 17A “7 A8 119. 50 51 52 53- 5A 55 56 57- 58 59 60 61 62 63. 6A. 65. 66 67 68. 69. .flark .chible SNOW .clidge .dwer .sked thate .gright .bleen .glemp stind .wrick .whickle .crell .swite .twasp plew prov trock .drack .shable fring slome 3.91 3' Oth U1 .tw .wh ~J' 0‘ .SW lO.pl ll.sm 12.sh l3.wr 1A.sk 15.bl 16.fl 17.cl l8.st 20.gr 21.pr 22.dw 23.fr 2A 25. 26. 27 28 29. 30. 31 32 33- 3A 35- 36. 37. 38 39 AO.’ Al A2 1.3. AA ‘15 A6 .oy idge ick .ate .able ing .OW .er asp .ack ind ight ible .een .ock .ell .ark ickle .ew .ed .emp 175 A7 A8. A9. 50. 51 52 53. 5A 55 56. 57 58. 59- 60 61 52 63 6A. 65. 66 67 68. 69. .shable twasp plew glemp .drack .slome SNOW .whickle .bleen crell .gright thate swite .clidge .fring .flark .wrick stind prov .dwer .chible trock sked 176 A3 One morning a boy made a boat. "Where can I play with it?“ he asked. Father said, "Come with me in the car. We will take your boat with us." Soon the boy called, "Please stop. I see water. May I play here?" "Yes," said Father. ”Have a good time." AA One day five children went out to play in the beautiful white snow. They played for a long time and then began to make snow animals. One of the animals was a dog. Soon the dog next door came out of the house. When he saw the snow dog he said, "Bow-wow.” The children laughed. “Now we have a dog that can bark.” A5 It was pet day at the fair. The children were waiting for the parade of animals to begin. They had trained their pets to do many different tricks. Among them was a tall boy whose goat made trouble for him. It kicked and tried hard to break away. When it heard the band, it became quiet. During the parade it danced so well that it won a prize. 16 Airpi freight. a dangerous or herds a our 200:. ‘ hiihways. 17 Hundre China, a la ”Ch. civil distant reg Years. 18 177 A6 Airplane pilots have many important jobs. They fly passengers, freight, and mail from one city to another. Sometimes they make dangerous rescues in land and sea accidents, and drop food where people or herds are starving. They bring strange animals from dense jungles to our zoos. They also serve as traffic police and spot speeding cars on highways. A7 Hundreds of years ago, most of Europe was a very poor region. But China, a large country in eastern Asia, had many of the comforts of a rich, civilized nation. Only a few people from Europe had visited this distant region. One was the famous Marco Polo. He learned some of the languages that were spoken in China and served its great ruler for many years. A8 The eager spectators who had cheered the plucky Warriors through eight hard-fought innings were silent. Only a run was required to defeat the much feared champions, who had previously defeated all opponents. The spectators had earlier criticized the umpire severely. Now their faces were tense with excitement as the players took their positions. 178 A9 The oil industry has been greatly increased by recent advances in science. Geologists have discovered new ways of locating veins of oil-producing rock. Problems of gusher control have been solved. Very ,effective also are newer methods of refining crude oil which have resulted in a higher ratio of quality fuel oil from a given volume of crude oil. A10 In response to the impulse of habit Joseph rose and spoke as in former_days. He spoke vigorously, continuously, and persuasively while the others listened attentively but in grim and contemptuous silence. Finally exhausted, Joseph hesitated for a moment: as often happens in such circumstances he became confused and was unable to resume speaking. A11 Many of the hypotheses about physical phenomena formulated by early philosophers were inconsistent and in most cases could not be universally applied. In order to develop accurate principles very capable physicists, mathematicians, and statisticians had to cooperate wholeheartedly over long periods of time to verify numerous basic facts and assumptions. In 4 contrasts the eartr‘ distance students 9111:5510 Ourii Ciambered ‘ “My beauiY nc Wm 51 Ca1m and 179 A12 In a concluding lecture on sidereal spaces, the astronomer contrasted the infinitesimal difference in the distance of the moon from the earth at apogee and at perigee with the great difference in the distance of the earth from the sun at aphelion and at perihelion. The students interrogated him, evidencing precociousness and lucidity in expression. A13 During a hiatus in the desultory firing, the apt lieutenant clambered wearily over the detritus piled against the redoubts. Beneath a canopy of empyrean blue lay the quiet, bucolic landscape, its pristine beauty now defiled by myriad diminutive promontories thrown up by the mortar shells, but radiating momentarily an inexplicable if spurious calm and peace. 33 . A b0) with a be: home." In Three kit: BA Alit 511! said, Hothe had a big ‘1twa 35 Twelv picture of ”Mr bags ”1Ed to IQgs‘ head Bribe. 180 B3 A boy had a wagon. He ran with it to a store. Soon he came back with a box. He called his dog and said, “Jump in, Happy. This is your home." In the morning the boy had a surprise. Happy was not in the box. Three kittens were there. BA A little girl ran out of a white house into a big yard. “Mother," she said, “my pet bird is gone. It went out of the open window." Mother laughed and said, "Look on my hat.“ When the girl looked she had a big surprise. A yellow bird with blue wings was on Mother's pretty hat. It was the bird that flew away. 35 Twelve boys were waiting in line at a party to play a game. A picture of a lion hung on the wall before them. They first put large paper bags over their heads so they couldn't see. Each of the boys then tried to pin a ribbon on the lion's tail. They put ribbons on the lion's legs, head, and body. All missed its tall. 50 none of them won the prize. 181 B6 One morning a big poster outside of Oak School told people about a basement bargain sale. Inside were long counters on which things collected by the children were displayed. Price tags were fastened to all articles. Most of the customers bought old but useful furniture. The sale was a huge success, and the money was used to purchase library books. 37 All of us admire the great skill of a good truck driver. He hauls many tons of things almost daily, including dangerous explosives. On mountain roads and in other isolated places he faces real dangers alone. He is his own mechanic. Sturdy and dependable, he will interrupt his schedule to help anyone who encounters real difficulty on a highway. B8 Rocky portions of the earth's surface are always changing. Many huge glaciers in the mountains carry along immense boulders which crush the rocks beneath. Chemicals in many streams penetrate rocks and dissolve them. Rocky surfaces are also broken up by processes of freezing and thawing which occur in most regions of alternate hot and cold weather. 182 39 After the American Revolution the colonies became states, each one having a governor. What was urgently needed was a federal government to insure domestic peace and to protect citizens from enemy attack. A constitutional convention was convened. After heated controversy, a constitution was prepared and submitted to the states for approval. 810 Beside the fireplace with its polished fixtures was a mohair chair which was in sharp contrast with a brilliant cover on a near-by footstool. Against the opposite wall stood a desk with stationery protruding from all its pigeonholes. But the object to which Alice's eyes returned repeatedly was a large flagon of incomparable value and startling beauty. 811 The visage of the pontiff was a familiar sight amidst the ornate decorations of the court. Famous for his politeness, he was as familiar with worldly affairs as with theology--a master strategist who could mold saints and sinners into a unified group or, if the situation justified such steps, discountenance unregenerates with a single quiet reproof. 183 812 An immediate rejection of customary rituals was unlikely as cultists steeped in traditional tribal lore advocated propitiation of imaginary deities. Their stubborn opposition to the abandonment of paganism subsequently brought opprobrium upon them, and historians record a cessation of eleemosynary enterprises until such customs were discontinued. 813 The ophthalmologist sent cultures to the microscopist requesting his opinion as to the causative organism of a painful conjunctivitis. A delay resulted when the expert on microscopy consulted the histomorphologist before giving a report. Meanwhile an anodyne was prescribed to forestall a recurrence of the patient's unfortunate sharp painful attacks. 18A 03 A cat wanted to find her kittens. She looked in the house and all over the farm. But she could not find them. Soon the mother cat saw a girl. "Mew," she said. ”Help me find my kittens." "Look,” laughed the girl. "Your kittens are coming to find you." OA One Saturday a black dog was walking down a pretty city street. He had no home and no one to feed or to pet him. At last he was so tired he lay down to sleep for a short time. Soon a small boy came by. "Are you lost?" he said. "Do you have a good home? I am glad I found you because I want a dog for a pet." 05 One bright summer day twin brothers walked to a lake with their uncle to fish. They sat still for a long time waiting for the fish to bite. Finally one boy got a bite. He became so excited that he dropped his pole into the water. The fish quickly swam away with it. Soon the pole disappeared. The surprised boy looked at his uncle and then laughed. 185 06 All over the world farmers face many difficult problems. Insects, birds, and other animals attack most of the farm crops and harm growing vegetables and fruit. At any season of the year the weather may also damage crops. Extreme heat, cold, or rain may ruin fruit, vegetables, or grain. However, most farmers usually succeed in having excellent crops. 07 Both city and country children love to visit-the zoo. A child who likes animals often finds all his familiar outdoor friends there. The fat woodchuck waddles as he always does in his native home. The playful seals swim in pools of cool water, flapping their flippers rapidly. Polar bears walk slowly back and forth paying little attention to people. 08 An airplane pilot eagerly hoped for a career in the Planet Air Force. His own occupation was very crowded and he was ambitious. He did not anticipate with pleasure the longer flight schedules, but the new position provided opportunitv for advancement. He might sometime even ciirect maneuvers of the Flying Demons, the ”mounted police" of outer space. 186 09 At last the captain had selected a sturdy vessel for his dangerous voyage and prepared to sail. His first mate was a reformed buccaneer who was a proud but clever knave, and a deadly opponent in hand-to-hand encounters. The crew too was a surly lot, but the best he could get on short notice. Even before they embarked he saw mutiny lurking in their eyes. 010 At first the energetic monk was heedless to the ferocity of the storm. He even accepted cheerfully nature's challenge to his fortitude. But soon the wind's incessant rasping in the bushes lessened his enthusiasm, and by the time he was within a furlong of the secluded monastery, he was anticipating eagerly temporary respite from the fury of the gale. 011 When we returned from the scenes of Asia Minor and the Grecian Archipelago, our ecstatic praise of the architecture of ancient Greece was equaled only by our laudations of the many archeological wonders of Egypt. We confess preference for the classical antiquities which evoke a mystic calm and imbue our inept minds with great reverence for the Acropolis. 187 012 Disconcerted by the spread of Origenistic opinion in Syria, Justinian condemned fourteen propositions espoused by The Alexandrian and compelled the Church to continue condemnation of his theories and to excommunicate Origen. Later, through the machinations of certain prelates a lengthy and mischievous controversy over the Monophysitic heresy was provoked. 013 The areas of regional metamorphism on the Meseta are formed of plutonic rocks passing upwards into a strata crystalline series in the Central Cordillera range. This series shows three levels: the lowest, augen-gneiss: an intermediate one, micaceous gneiss with schists and crystalline limestones introduced: and the upper one, lustrous schists and phyllites. 10. 11. 12. 188 Sam is wet. A bat can fly. Ned sat up in bed. A fox bit Tim on the leg. Bob and Pam sat on a raft. Bob had a fish in his hand. Pam had a ship. Frank sent Nan a gift. She gives him a kiss to thank him. A shark is a big fish that can harm a man. The man swims as fast as he can. Bill has come in first. He has won. Jack is last. He has lost. This bride has a long gown on. She stands in front of a church. We can play lots of games with cards. These three cards are the ace of clubs, the king of clubs, and the queen of spades. I eat three meals a day. For lunch I had a pound of ground meat, a slice of toast, and a cup of tea. This nice young man has lost his sight. He is now blind, but he has a smart dog for a guide. The player who throws the ball is the pitcher. The one who swings at it is the hitter. The man who crouches in back of the plate is the catcher. Mister Smith is a father. His wife is a mother. They are both taller than their son or their daughter. Jan White is washing her clothes in the washer and drying them in 1A. 15. 16. 17. 18. 19. 20. 189 the drier. When she is through, she will hang them up on the hangers. The boy and the girl are seated at their desks. The boy is reading a book. The girl is writing a letter. Winter is the coldest time of the year. The skies are gray. The days are cloudy, windy, and snowy. In the brightly lit big tent, a monkey is riding on the back of a donkey. Each child has money in his hand. Billy has a quarter and Jimmy has a dollar bill. The boy wearing the mask is trying to frighten the men and children. The puzzled poodle is barking at what he thinks is a witch. This woman has a bonnet on her head and a bracelet on her left wrist. She is sitting at the window and feeding pretzels to a sparrow, a pigeon, and a squirrel. In backing her car out of the garage, a woman driver hit a hydrant. Her luggage spilled out of the trunk in front of a cyclist who stopped all the traffic on the street. The youthful rider is mounted on a prancing stallion. In the flat prairie below, a train is just pulling out of the station. To the far right, storm clouds are forming over the tall mountain peaks. In the foreground, a barefoot woman in a bathrobe is using clothespins to pin up a bedspread. To her left, her younger brother is beating the dust out of a mattress. In the background, her uncle 190 is painting a bookcase. 191 2a Bob has a little red wagon. He likes to ride in it. He pulls it slowly up the hill. Then he rides it quickly down again. One day he took his dog with him. He pulled the dog up the hill. Then they rode down the hill. But the dog did not like to ride down. He jumped out of the red wagon. Bob went down by himself. Now he does-not try to take his dog in the wagon. 3a Bob has a brown and white dog named Spotty. He is called Spotty because he has brown spots on his nose. Bob always takes his dog on his trips to the woods. The dog helps scare the rabbits. Bob walks slowly, but his dog scampers through the leaves. One day Spotty left Bob and went off by himself. Bob called and whistled, but the dog did not come back to him. After a while Bob heard the dog barking a long way off. Bob walked toward the sound of the barking until he found the dog. Spotty thought he had caught a black and white kitten. But it wasn't a kitten, it was a skunk. That night the dog had to sleep outside. 192 Aa Yesterday Bob took a trip to a city market that was somewhat like a store but a great deal bigger. It didn't have any bread or canned goods like the grocery stores. But there were a great many big boxes of vegetables and fruits. Bob was hungry and wanted just one plum or cherry to taste. He wondered if one of the men would sell him just one plum. Everyone was buying the fruit and vegetables by the whole crate. When Bob asked the man to sell him one plum, he laughed and gave Bob an extra large plum wrapped in paper but wouldn't take any money. As he walked along eating the plum, Bob watched the men unloading the trucks and big trailers. They would chap open the top of the crate so that anyone could see the fruit. If a buyer liked the fruit, and was willing to pay the price, he might buy the entire truckload. 193 53 As a ship's boy, John Paul had all sorts of odd jobs on board. Sometimes he scrubbed decks or helped the cook. He cleaned the captain's cabin and ran errands, but he had other duties that pleased him more. He helped to clean the guns, which the merchant ship carried for A protection. And several times he stood behind the big wheel to steer the ship. Captain Benson wrote in the ship's log, or daily record, that the trip was calm and smooth-sailing. Nothing unusual happened, but every day was a real adventure for the new ship's boy. At the end of the voyage it was a thrill to sight land. When the ship docked near Fredericksburg, Virginia, John Paul was waiting to go on shore. John Paul's brother had a tailor shop in Fredericksburg and was very happy in his new home. He was eager to talk about the wonderful country, but John Paul already loved America. During the next few years John Paul visited America often. He became used to the free and democratic ways of the new country. Meanwhile he had learned to be an expert sailor. Although he was not tall, he was strong and quick. With his long arms he could haul or trim a sail with the best of men. 1911 6a Elephants are found wild today only in warm regions--in tropical Africa and in India. The story was very different 50 thousand years ago. Then, two species of the elephant family roamed North America and Europe in vast numbers. One of them was the mastodon. The mastodon lived in the eastern part of our country during the period of the Great Ice Age. In the swamps that were formed when the ice disappeared, many of the huge creatures were trapped and killed. We have found some of their skeletons. At a glance, the mastodon must have looked much like the elephants of today, except that it was covered with coarse, woolly hair and its tusks were much larger. It was probably heavier than the elephants we know, but not taller. Its head was flatter and its lower jaw longer. Its teeth were not like the teeth of the elephants of today. More than 200 years ago, the people of New England found bones of the mastodon when they dug ditches to drain swamps. At first they thought that the bones they found were bones of giant people. When they found teeth that weighed more than four pounds apiece, they decided that the giants were giants indeed. 195 7a Just as in driving a car, we use at least three speeds in reading. High gear in reading is called skimming, while studying is reading in low gear. Between these two, at a second gear, is what might be called a moderate speed of reading. As you may have heard, the good reader adapts his rate to the purpose of his reading. The rate he uses is determined by how much he wants to get out of the material he is reading. His rate is also influenced by the difficulty of the reading material. Thus, he shifts from gear to gear according to the amount he wants to retain or how difficult he finds the going. Skimming is useful for a number of situations in reading. We can use it when looking for a particular fact on a page or in a table. It is also appropriate when we have to cover a large amount of material that is not too interesting or too important. Skimming may also be used to determine the general trends or ideas of a selection when we do not have to know the fine details. It is also helpful when we are making a quick brush-up before recitation. Finally, it is very useful as the speed at which we would do pre-reading before studying intensively. 196 8a President Thomas Jefferson, in 180A, commissioned an expediton to go into the Northwest Territory to explore the land that was bought in the now-famous Louisiana Purchase. Meriwether Lewis, private secretary to the President, and Captain William Clark of the United States Army headed the 26 men who started up the Missouri River from St. Louis on May 21, 180A. On July 18 of of that year the group reached the southwest corner of the present state of Iowa, proceeded northward along the Missouri, and traversed parts of Iowa many times. The Lewis and Clark State Park west of Onawa was named in honor of these explorers. Sergeant Charles Floyd, a Kentucky backwoodsman and one of the most competent men of the party, became ill on August 19, 180A, and died the following day. His body was laid to rest upon a high bluff near the present site of Sioux City, where it is marked with a tall monument. The Floyd River and Sergeant Bluff were named in his honor. The expedition proceeded westward to the mouth of the Columbia River, and returned to Washington during the early months of 1807. Lewis was appointed governor of the Louisiana Territory and Clark was named governor of the Northwest Territory. 197 2b Bob was eating his dinner. ”Ding-dong,“ went the fire bell. The firemen were going to a fire. The engine went quickly down the street. Bob jumped up and left his dinner. He ran into the street. The fire was on the next corner. Bob watched the men put the fire out. "Ding-dong,” went the bell again. The engine started back down the street 0 Bob watched the engine go. Then he went back home and ate his dinner. 198 3b Mary was going downtown to watch the parade. She skipped and ran along the street because she could hardly wait to get there. She was early and found a good place to stand. Pretty soon she could hear the music of the bands coming down the main street. The men of the first band were dressed in scarlet, with white feathers in their hats. The men of the second band were clad-in dark blue, with red feathers in their caps. After them came the trucks loaded with flowers or fruit. Then there were cars filled with officers and their friends. Next came a company of soldiers in dark green uniforms. Last of all was another band dressed in white suits and yellow feathers. 199 Ab Mary's teacher took her class for a nature walk one sunshiny day last week. Every time the group came to a new plant, they would stop and examine it while the teacher explained its parts. She showed them how a bee gets its honey from flowers and how a bug had eaten part of the leaves from some plants. On a few plants, the flowers had fallen off, and seeds had begun to form. Later, while they were looking at some blossoms, one boy spied a nest hidden in a tree. They were very quiet, hoping that the mother bird would return to feed her young ones. Sure enough, she quickly came back with a fat, juicy worm in her bill. She fed the young ones, chirped a little, and then flew away after more worms. Mary's teacher said that birds eat a great deal every day. They help us by eating insects that would destroy our plants and by eating weed seeds. 200 5b Suppose that you have some beautiful poppies growing in your garden. Suppose, too, that you want to get some seed from them so that you can have more poppies like them next year. You must be sure, then, not to pick all the poppy flowers. If you do not leave some of the flowers on the plants, you will not have any seeds, for the flowers are the part of the plant that produces the seeds. There will not be any seeds if all the flowers are picked. Most seeds come from flowers. The seeds of pine trees and of the trees and bushes of the pine family are formed in cones. But most other seeds come from flowers. More than 190,000 kinds of plants produce seeds, and all but about 700 produce their seeds in flowers. Not all flowers are large and bright-colored like poppies. Probably you have seen many flowers that you did not know were flowers. Did you ever see any cottonwood flowers, or willow flowers, or grass flowers? Cottonwood trees and willow trees and grass have flowers, but their flowers are small and are not bright-colored. Many other plants have small flowers much like these. 201 6b When the early settlers came to America, trade was carried on by barter or by using such things as tobacco, sugar, and furs as money. Sometimes the settlers used Indian wampum. Wampum was shells that were made into beads and was used by the Indians as decoration and as money. Of course, when more people came from Europe to settle in America, they found they would need money to pay workmen. A mason did not always want to take furs for his pay. A furrier did not always want his wages in grain or tools. People had to have coins, so they used whatever was available--English shillings, Swedish and Dutch money, and Spanish dollars, or "pieces of eight." The colonists soon found there were not enough of these to go around. England would not let her Colonists make any money of their own. But in 1652, Massachusetts set up a mint and made her own coins anyway. Among these were the famous "pine-tree shillings." They were called this because the picture of a pine tree was stamped on them. These pine-tree shillings were made for thirty-four years, but they all had the same date on them. In this way the Colonists pretended that they were obeying England. 202 7b One of the most beautiful and lasting kinds of building stone is marble. Marble may be pure white or colored, or it may have streaks in it. It can be polished so that it has a very smooth surface. All marble was once limestone. Limestone, deep under the ground, may be changed to marble by heat and pressure. Granite is another very good building stone. It is formed from rock so hot that it is liquid. You may have seen pictures of liquid rock, or lava pouring out of volcanoes. Lava cools and becomes rock rather quickly. But granite is made from the hot liquid held underground. This rock cools very slowly. The liquid rock from which granite comes cools so slowly that the different materials in it separate from one another and form crystals. Granite is always a speckled rock because the different crystals in it are of different colors. Two minerals are always found in granite. They are quartz and feldspar. The dark speckles in granite are usually some other mineral. Granite makes very good building material because it is so hard. It can be beautifully polished, and the weather does not harm it. 203 8b One very important reason for slow reading is lack of pre-planning. Many slow readers have not learned to adapt their rate to the difficulty of the material they are reading. They can see objects quickly with their eyes, as in looking through the window of a moving car, but when they read, the same quick movements are not present. They stop to look at each individual word as though that were necessary for gaining ideas. They tend to read their textbooks, magazines, and even the newspaper at about the same rate of speed. It has been shown by many studies of good readers that the rate in fiction material should be two to three times as fast as that in non-fiction. Slow readers tend to "study" everything they read, while good readers vary their rate consciously according to the kinds of facts they want to get. For example, if a student is trying to find only one fact on a page, he certainly does not need to read the entire page. He can skim quickly over the page until he finds the fact he is seeking. If, on the other hand, he is expected to report critically on a piece of prose, he will need to read much more slowly and analytically. 20A 2a A hen had six little yellow chickens. One morning she took them for a walk. They looked for something to eat. They found some seeds and sand. A dog came to play with them. The hen did not like the dog. She flew at the dog and made him run away. 3a Three boys built a house in the woods. They put a table and two old chairs in it. There was a basket full of apples under the table. One afternoon they went away and left the door open. When they came back, they found two little pigs eating the apples. Aa A little girl got off the train all alone. There was nobody at the station to meet her. She asked the man inside the station where her mother was. He said that her mother could not get the car started. A man was trying to fix it. The little girl sat down to wait. A few minutes later a big car came around the corner with her mother in it. The little girl got in and they drove home. 205 5a About one hundred and fifty years ago, in France, the first man went up in a balloon. His balloon was made of paper covered with strips of cloth to make it strong. A long rope kept it from going too high. Later this man took a friend up in the balloon with him. On this trip they rose over five hundred feet. The trip lasted thirty minutes. They came down several miles from where they started. 6a Early settlers in America found that Indians would sell skins and land for glass beads. Many men earned their living by making glass beads and bottles. In 1827 a man invented a way to press molten glass into iron molds. The most famous glass works was in the town of Sandwich in Massachusetts. The Sandwich glass had a bright silvery appearance and it could be molded into very elaborate and attractive patterns. Beautiful lamps and candlesticks as well as all sorts of dishes were made from this glass. In many New England homes pieces of Sandwich glass are still found on display. 206 7a Basketball is one of the more recent games. It was devised by a college instructor who desired a game to interpose between the football and baseball seasons. The game demands precision of movement, concentration, and great endurance. It is more p0pular in those localities where it does not compete with hockey. Opinion differs as to whether it is a satisfactory game for girls. It has been modified to make it less strenuous for them by restricting the playing area of each player. Some of the large Western universities have audiences of over twenty thousand at their conference games. 8a Railroad communication developed rapidly just after the Civil War. Between 1865 and 1873, thirty-five thousand miles of track were laid. This doubled the distance people could travel by railroad. Some of the new roads connected important cities, and some extended westward beyond populated regions. Congress favored this sudden development by granting land to companies interested in furthering the expansion. Grants included territory lying within twenty miles of the proposed roadbed. 1Alternate sections were allotted to the railroad: those in between were reserved for homesteaders. The sale of sections of land owned by the rai lroad was made easier through this checkerboard arrangement. 207 2b The policeman has a big brown horse named Pep. The children like to pat him. They give him apples and sugar to eat. Pep can do two tricks. He can shake hands. When the policeman says, "Count three,” Pep paws the ground three times with his foot. 3b One morning Jack woke up late. He had to dress in a hurry. He put on one of his black shoes, but he could not find the other. He could find only an old brown shoe. He put it on and ran to school. He got there just as the bell rang. Ab Late in the summer a man started to build a house. He wanted to finish it before winter came. He had some men dig the cellar for him. Then he built the floor and the cellar steps himself. Before he could do anything more, he had to move away to another town. He left the house ,just as it was. He told the boys next door that they might use the cellar as a clubhouse. 208 5b The first trains were pulled by horses. Later, engines were used to pull trains. The first engines could not go very fast and often broke down. Once there was a race between a train pulled by a horse and a train pulled by an engine. At first the horse was ahead because the engine had to start slowly. Then the engine passed the horse, but something in the engine broke and the horse won the race. 6b Rubber came into general use about two hundred years ago. Columbus brought it back from his second voyage to America. It was named "rubber” because it was used to rub out pencil marks. Rubber comes mainly from Africa and Brazil. It is manufactured from the milky juice of the rubber tree. Rapid handling of the juice is necessary. It is collected early in the morning when the flow is greatest. At the factory, acid is added at once to make the rubber rise to the top. It is then treated in many different ways. Attempts to produce rubber in the United States have not been successful. 209 7b Every four years, athletes from all over the world compete in a festival of sports. These great contests are called the Olympic Games, since they were first conducted at Mount Olympus by the ancient Greeks. They were part of a religious celebration in honor of Zeus, father of the Greek gods. They were discontinued long ago and were revived only in 1896 by men interested in continuing the custom. The first of the modern games were staged appropriately in Athens. Since that time they have been held in various countries. They stimulate world friendship and help to maintain interest in physical perfection. 8b One of the most difficult political and economic problems of our Federal government is to prevent the development, throughout the country, of certéin commercial interests at the expense of others. One very persistent attack has been against the development of large corporations. DeWitt in his study of the history of this struggle states that it has three distinct objectives. First, to find some satisfactory Ineans to control and regulate the activities of large business corporations; second, to resist the tendency of corporations to exploit natural resources for their own benefit; third, to control tariffs which 'favor trusts and monopolies. Appendix pi Student Protocol Notebook This is an example of one student protocol notebook. It was used for directing the testing session and recording student responses. Each protocol notebook was generated separately for each student and each test session by a text processing system. Therefore, each student's protocol notebook contains the precise versions or alternate forms for him/her. The protocol notebook was essentially used as a script which was read and followed by the test examiner. The sample given here is for student id 117 who received test battery 601. Note that the notebook was changed slightly from pre- to post-test. Specifically, the nature of instructions and reminders to the test examiner were improved to guarantee a precise, standard use of the materials. The contents presented in this appendix are identical to that used in the study except that the format has been altered slightly. 210 1. 211 Please record today's date and the time: Date: Time: Please record your (the tester's) name: Name: The following materials should be present for the testing session: The materials for the student to use: 1) The 'Test Materials Notebook' The materials for the test administrator to use: 1) The 'Test Protocol Notebook' for the particular student that is to be tested. 2) An audio cassette taperecorder for recording the entire testing session. 3) A 90 minute cassette tape. A) A directional microphone. 5) A stop watch for timing various parts of the testing session. The order of presentation of materials to the student will be determined by the order of materials in the 'Test Protocol Notebook'. This order should be: h. Standard Interview form Instant Word Recognition Measure: Using the Slosson Oral Reading Test and non standard procedures Decoded Word Recognition Measure: Using the Gates-McKillop Test Oral Reading Measure: Using the Gray Oral Reading Test Sullivan Oral Reading Placement Task Reading Comprehension Measure: Using Spache and Ourrell Graded Selections and non standard procedures Listening Comprehension Measure: Using the Spache and Ourrell Graded Selections and non standard procedures Word Comprehension Measure: Vocabulary from the Spache and Durrell Comprehension Selections Test Administrator Debriefing Form 212 Check to make sure that the student's name and code number are on the front cover of the 'Test Protocol Notebook' and that his/her code number is on all successive pages of the same document. Avoid the use of the student's name during standard interview and oral reading tasks. Do not write the student's name on any of the materials. 00 NOT GIVE THE STUDENT HIS/HER TEST MATERIALS AT THIS TIME. Check out the audio tape recorder before the student arrives. Actually record something and play it back to check volume levels, proper functioning of tape and recorder and microphone, etc. 00 THIS EVERY TIMEIIllIl 213 Introduction pp Student After greeting the student and introducing yourself, orient the student to the session. If this is the first encounter with the student this year, use the first set of statements as your guide. If the student has already been diagnosed once this year, use the second set. Informally, and in your own words, orient the student using the appropriate guidelines. 1. STATEMENT SET 1: FOR THE FIRST DIAGNOSIS OF A STUDENT b. that you know the student has had problems with reading that the purpose of the session is to gather information so that you can figure out how to make the student a better reader that the session will last at least an hour. During most of that time the student will be reading either out loud or to him/herself. that during the session you will be jotting things down, using the stopwatch from time to time and taperecording the whole session. let the student ask you any questions s/he might have. 2. STATEMENT SET 2: FOR SUCCESSIVE DIAGNOSES OF A STUDENT that because of the way the reading program is setup here, since the student was tested at the beginning of the school year, we have to test him/her again. that, again, the session will last at least an hour. During most of that time the student will be reading either out loud or to him/herself. that, again, during the session you will be jotting things down, using the stopwatch from time to time and taperecording the whole session. let the student ask you any questions s/he might have. 21A Standard Interview Form 1. 2. Please record the time. Time: TURN ON THE TAPE RECORDERIII to record the entire testing session. Read the following to the student: "I'M GOING TO ASK YOU SOME QUESTIONS. YOUR ANSWERS WILL HELP US FIGURE OUT THE BEST WAY TO HELP YOU BECOME A BETTER READER." If this is the first encounter with the student this year, ask the 1 first set of questions. If the student has,already been diagnosed once this year, ask the second set of questions. DO NOT write down the student responses. The answers will be transcribed from the audio tape at a later date. QUESTION SET 1: FOR THE FIRST DIAGNOSIS OF A STUDENT be Why do you think you're having trouble reading? What kinds of books are easy to read for you? Do you remember when you started having trouble with reading? 1) If "Yes", then, ask "When?" 2) If "No", then, ask the student to think back to the fourth grade. Was everything alright then? Repeat until approximate grade established. When reading your school books, do you know most of the words right away? What do you do when you come to a word you don't know? When you listen to people read out loud, how can you tell if they're doing a good job? Are you having any trouble finishing your reading assignments in your classes? What do you read outside of school? When summarizing a fiction story, what kind of things should you include in the summary? What kind of reading is hard for you to summarize? Do your eyes bother you after you've been reading for a while? I I.“ a.-.t v“ 215 Do you have any trouble seeing things clearly? Do you have glasses? Should you wear them when reading? Do you? Do you have any trouble hearing people when they talk to you? What is punctuation for? Have you been in any special reading classes before? What kinds of things have you done in your reading classes? Is there anything particular that you want to be able to do in reading by the end of this year? QUESTION SET 2: FOR SUCCESSIVE DIAGNOSES OF A STUDENT When reading your school books, do you know most of the words right away? What do you do when you come to a word you don't know? When you listen to peOple read out loud, how can you tell if they're doing a good job? Are you having any trouble finishing your reading assignments in your classes? When summarizing a fiction story, what kind of things should you include in the summary? What kind of reading is hard for you to summarize? Do your eyes bother you after you've been reading for a while? Do you have any trouble seeing things clearly? Do you have glasses? Should you wear them when reading? Do you? Do you have any trouble hearing people when they talk to you? What is punctuation for? 216 ******************** Affect rating during this task: l.Did the student attend closely to the task? YES NO 2.0ther comments on affect during this task? 217 Instant Word Recognition Measure Please record the time. Time: Give the student his/her "Test Materials Booklet." Determine the appropriate form of the SORT for this student by looking at the next page of this protocol notebook. Then, turn to the proper page of the "Test Materials Notebook" for the student as indicated in its table of contents. ****THIS STUDENT WILL USE FORM SORT RA**** Read the following directions to the child: ”HERE ARE SOME LISTS OF WORDS. I'LL ASK YOU TO READ SOME OF THESE LISTS. IF YOU DON'T KNOW A WORD RIGHT AWAY, TRY AND SOUND IT OUT. WE'RE GOING TO START WITH THE VERY FIRST LIST." For each word, record student performance as follows: a. If the student reads the word with no hesitation, check the 'Instant' column. b. If the student correctly called the word within a slow count of five, check the 'Mediated' column. c. If the student does not correctly call the word within a slow count of five, write down verbal responses in the 'Miscall' column. An incorrectly accented word is considered a miscall. Write N.R. (No Response) if no response was made. d. After the five count, ask the student to continue with the next word. Let the student proceed through the lists, in order, until s/he can only read 5 or fewer words correctly. A word is considered correct 'whether called instantly or mediated. Close the student's ”Test Materials Notebook.” For each of the 20 (words in the last list the student attempted, ask the following question: 'What does mean?‘ Check the appropriate column: 'Correct Meaning' 'Incorrect Meaning' or 'No Response'. IREMEMBER: THE CRITERION IS 5 OR FEWER WORDS CORRECT - INSTANT OR MEDIATED. 218- Word Ins. Med. Miscall Cor. Inc. 1. jump 2. want 3. look A. play 5. come 6. is 7. can 8. one 9. see 10. ball 11. little 12. three 13. baby lA. run 15. down l6. mother 17. here u..- .4... I fi,—— -.' 23!. 18. up 19. help 20. make TOTALS Word 21. 22. 23. 211. 25. 26. 27. 28. 29. 3o. 31. 32. 33. 3A. 35. 36. 37. 38. 39. A0. very under hill road dark horse came PUPPY was bump with ride along friends first food wish what basket live TOTALS Ins. Med. Miscall 219 Cor. Inc. NR Word 111. A2. A3. AA. 115. A6. A7. 118. A9. 50. 51. 52. 53. 511. 55- 56. S7- 58. 59. 60. better hide large field around suddenly forest river sheep station breakfast across heavy lunch happen grass game farmer stars hope TOTALS Ins. Med. Miscall 220 Cor. Inc. NR Word Ins. 61. hunger Med. 221 Miscall Cor. Inc. NR 62. evening 63. bench 6A. excuse 65. grove 66. desire 67. perform 68. destroy 69. delicious 70. understood 71. safe 72. against 73. timid 7A. ocean 75. damp ___ 76. stream 77. empty 78. smash 79. stone 80. reward TOTALS Word 81. claimed 82. forehead 83. serious 8A. speechless 85. distant 86. dainty 87. anger 88. courage 89. slumber 90. common 91. harness 92. flakes 93. develop 9A. vacant 95. price 96. future 97- appearance 98. promptly 99. region 100. silence TOTALS Ins. Med. Miscall 222 Cor. Inc. NR u '7 winter—71‘2“ 223 Word Ins. Med. Miscall 101. dignity 102. generally Cor. Inc. 103. gracious 10A. applause 105. define 106. cushion 107. jungle 108. haze 109. profitable llO. tailor lll. fragrant 112. obedient 113. marriage 11A. presence 115. terrace 116. interfere 117. ambition 118. merchant 119. custom 120. extended NR TOTALS 22A Word Ins. Med. Miscall 121. tremendous Cor. 122. dungeon Inc. NR 123. customary 12A. abundant 125. malicious 126. imaginary 127. excellence 128. consequently 129. installed 130. responsible 131. yearning I32. liquid 133. medicine 13A. spectacular I35. rebellion 136. compliments 137. infected 138. inventory 139. importance 1A0. detained TOTALS flsmfllzui 2....- I Word 1A1. 1A2. 1A3. 1AA. 1A5. 1A6. 1A7. 1A8. 1A9. 150. 151. 152. 153. 15A. 155. 156. 157. 158. 159. 160. TOTALS 225 Ins. Med. Miscall reminiscence Cor. Inc. architecture NR environment counterfeit pensive tremor intricate administer continuously attentively industrious contemporary approximate standardize complexion exhausted malignant compassionate crisis society "3 H.,...— n W 2"01‘1- Word 161. 162. I63. 16A. 165. 166. 167. 168. 169. 170. 171. 172. 173. 17A. 175. 176. 177. 178. 179. 180. nucleus proportional articulate remarkably nonchalant memorandum deprecate contrasting whimsical evident prairies twilight intangible supplement formulated exuberant antique inducement irrelevance grotesque TOTALS Ins. Med. Miscall Cor. Inc. 7‘7.— -nn m6 i Word 181. 182. 183. 18A. 185. 186. 187. 188. 189. 190. 191. 192. 193. 1911.. 195. 196. 197. 198. 199. 200. 227 Ins. Med. Miscall sojourn Cor. affable auspicious enamoured futility primordial inadequacy chastisement Inc. NR envisage pandemonium panorama compressible gustatory traverse simultaneous excruciating contraband decipher scrupulous facsimile TOTALS wi- 228 REMEMBER: 1. Close the student's test booklet. 2. Ask the Student for the meaning of all words in the last list attempted. 3. Don't let the student spend more than a slow count of five on any one definition. A. Do not indicate to the student that these words are from the lists just read. If the student responds with a definition of a homonym, present the word again, spell it, and ask for the definition. *sssaesssssaessasste Affect rating during this task: l.Did the student attend closely to the task? YES NO 2.0ther comments on affect during this task? 229 Decoded Word Recognition Measure 1. Did you give the vocabulary test on the SORT just completed? If not, 00 IT NOW! Please record the time. Time: Reopen the student test booklet to the appropriate form of the Gates-McKillop for this student. USE THE TABLE OF CONTENTS. ****THIS STUDENT WILL USE FORM GMK RA**** Read the following instructions to the student: "HERE ARE SOME LISTS OF WORD PARTS AND NONSENSE WORDS. PLEASE READ DOWN EACH LIST FOR ME.” As the student reads, record all miscalled word segments and nonsense words by writing down the inappropriate response in the space provided. w u "II-'8 i. :2 j '1.sk 2.fr 3.st A.Sl 5.ch 6.cr 7.91 8.gr 9.sm 10.dw ll.pr 12.cl l3.wr 1A.th 15.tw l6.sw 17.wh______ l8.bl______ 19.sn_____ 20.dn_____ 21.fL_____ ZZJI 234H TOTALS 2A. 25. 26. 27.’ 28.‘ 29. 30. 31 32 33 311 35- 36 37 38. 39- A0 A1 A2. “3 AA. A5. A6 .emp .asp .oy 230 idge een ible .ock . ome .ell ickle .ed .ew ick ark ight .able ing OW .ack TOTALS A7. A8 A9. 50 51 52 53- 51.. 55- 56. 57 58. 59- 60. 61 62 63. 6A 65 66. 67 68. 69. .proy stind .smow chible .whickle .dwer twasp shable flark thate .slome trock swite gright______ .plew .glemp______ clidge______ .bleen .drack sked .wrick fring crell TOTALS w «___» j's a 231 ******************** Affect rating during this task: l.Did the Student attend closely to the task? YES NO 2.0ther comments on affect during this task? 232 Gray Oral Reading Test '0 2. Please record the time. Time: Now mum to the appropriate form of the Gray Oral Reading for this student.L5E THE TABLE OF CONTENTS. ***WHHS STUDENT WILL USE FORM GRAY RA**** Read the following instructions to the student: 'YOU ARE GOING TO BE READING SOME PARAGRAPHS ALOUD. WE'LL READ THEM ONE AT A TIME. THEY GET HARDER AS YOU GO ALONG AND YOU WILL BE MAKING MISTAKES. DON'T WORRY ABOUT THAT. JUST DO THE BEST YOU CAN.‘ Begin the oral reading with paragraph one. For each paragraph do the following: a. RESET THE STOPWATCH TO ZERO. Tell the student to begin. Then, you START THE STOPWATCH. b. c. Record the student's performance using the standard oral reading markings. d. When the paragraph has been completed stop the clock and record the time. e. Let the student continue through consecutive paragraphs until his rate exceeds two minutes for a given paragraph. f. Provide correct pronunciation (l)after 5 seconds or (2) on student request. Offer no other prompting. REMEMBER: THE CRITERION IS TWO MINUTES. PROVIDE PROMPTING AS DESCRIBED ABOVE. 233 A3 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: One morning a boy made a boat. "Where can I play with it?" he asked. Father said, "Come with me in the car. We will take your boat with us." Soon the boy called, "Please stop. I see water. May I play here?" "Yes," said Father. "Have a good time." AA START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: One day five children went out to play in the beautiful white snow. W a. ..un; Lin-"J I‘; i They played for a long time and then began to make snow animals. One of the animals was a dog. Soon the dog next door came out of the house. When he saw the snow dog he Said, “Bow-wow.” The children laughed. ”Now we have a dog that can bark.“ A5 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: It was pet day at the fair. The children were waiting for the parade of animals to begin. They had trained their pets to do many different tricks. Among them was a tall boy whose goat made trouble for him. It kicked and tried hard to break away. When it heard the band, it became quiet. During the parade it danced so well that it won a prize. 23A A6 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: Airplane pilots have many important jobs. They fly passengers, freight, and mail from one city to another. Sometimes they make dangerous rescues in land and sea accidents, and drop food where peOple or herds are starving. They bring strange animals from dense jungles to our zoos. They also serve as traffic police and spot speeding cars on highways. A7 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: Hundreds of years ago, most of Europe was a very poor region. But) China, a large country in eastern Asia, had many of the comforts of a rich, civilized nation. Only a few people from Europe had visited this distant region. One was the famous Marco Polo. He learned some of the languages that were spoken in China and served its great ruler for many years. A8 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: The eager spectators who had cheered the plucky Warriors through eight hard-fought innings were silent. Only a run was required to defeat the HRHHT feared champions, who had previously defeated all opponents. The spectators had earlier criticized the umpire severely. Now their faces were tense with excitement as the players took their positions. VII—VF*F”F‘ 235 A9 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: The oil industry has been greatly increased by recent advances in science. Geologists have discovered new ways of locating veins of oil-producing rock. Problems of gusher control have been solved. Very effective also are newer methods of refining crude oil which have resulted in a higher ratio of quality fuel oil from a given volume of crude oil. A10 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: In response to the impulse of habit Joseph rose and spoké as in former days. He spoke vigorously, continuously, and persuasively while the others listened attentively but in grim and contemptuous silence. Finally exhausted, Joseph hesitated for a moment: as often happens in such circumstances he became confused and was unable to resume speaking. All START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: Many of the hypotheses about physical phenomena formulated by early philosophers were inconsistent and in most caSes could not be universally applied. In order to develop accurate principles very capable physicists, mathematicians, and statisticians had to cooperate wholeheartedly over long periods of time to verify numerous basic facts and assumptions. 236 A12 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: In a concluding lecture on sidereal spaces, the astronomer contrasted the infinitesimal difference in the distance of the moon from the earth at apogee and at perigee with the great difference in the distance of the earth from the sun at aphelion and at perihelion. The students interrogated him, evidencing precociousness and lucidity in , expression. A13 START WATCH. HAVE STUDENT READ. RECORD ORAL READING. STOP WATCH. RECORD TIME. Time: During a hiatus in the desultory firing, the apt lieutenant clambered wearily over the detritus piled against the redoubts. Beneath a canopy of empyrean blue lay the quiet, bucolic landscape, its pristine beauty now defiled by myriad diminutive promontories thrown up by the mortar shells, but radiating momentarily an inexplicable if spurious calm and peace. *******************a Affect rating during this task: l.Did the student attend closely to the task? YES NO 2.0ther comments on affect during this task? 1. Did the student generally use proper inflection while reading the passages? (circle one) YES NO 2. Was the student reasonably fluent? (circle one) YES NO 1‘ i- 237 Sullivan Oral Readipg Placement Task 1. 2. Please record the time. Time: Turn to the Sullivan Reading Paragraphs. USE THE TABLE OF CONTENTS. Read the following instructions to the student: "YOU ARE GOING TO BE READING SOME SHORT PARAGRAPHS ALOUD. PLEASE START WITH PARAGRAPH NUMBER ONE AND CONTINUE UNTIL I TELL YOU TO STOP." Record the student's performance using the standard oral reading markings. Let the student continue through consecutive paragraphs until s/he makes three or more uncorrected miscalls in a paragraph. Offer no prompts or other assistance as the student reads. REMEMBER: THE CRITERION IS 3 OR MORE NONTRIVIAL UNCORRECTED MISCALLS. OFFER NO PROMPTING. 10. 11. 12. 238 Sam is wet. A bat can fly. Ned sat up in bed. A fox bit Tim on the leg. Bob and Pam sat on a raft. Bob had a fish in his hand. Pam had a ship. Frank sent Nan a gift. She gives him a kiss to thank him. A shark is a big fish that can harm a man. The man swims as fast as _ i 1 he can. t Bill has come in first. He has won. Jack is last. He has lost. This bride has a long gown on. She stands in front of a church. We can play lots of games with cards. These three cards.are the ace of clubs, the king of clubs, and the queen of spades. I eat three meals a day. For lunch I had a pound of ground meat, a slice of toast, and a cup of tea. This nice young man has lost his sight. He is now blind, but he has a smart dog for a guide. The player who throws the ball is the pitcher. The one who swings at it is the hitter. The man who crouches in back of the plate is the catcher. Mister Smith is a father. His wife is a mother. They are both taller than their son or their daughter. Jan White is washing her clothes in the washer and drying them in 13. 11“ 15. 165. 17. 18. 19. 12(3- 239 the drier. When she is through, she will hang them up on the hangers. The boy and the girl are seated at their desks. The boy is reading a book. The girl is writing a letter. Winter is the coldest time of the year. The skies are gray. The days are cloudy, windy, and snowy. In the brightly lit big tent, a monkey is riding on the back of a donkey. Each child has money in his hand. Billy has a quarter and’ Jimmy has a dollar bill. The boy wearing the mask is trying to frighten the men and children. The puzzled poodle is barking at what he thinks is a witch. This woman has a bonnet on her head and a bracelet on her left wrist. She is sitting at the window and feeding pretzels to a .sparrow, a pigeon, and a squirrel. In backing her car out of the garage, a woman driver hit a hydrant. Her luggage spilled out of the trunk in front of a cyclist who stopped all the traffic on the street. The youthful rider is mounted on a prancing stallion. In the flat prairie below, a train is just pulling out of the station. To the far right, storm clouds are forming over the tall mountain peaks. In the foreground, a barefoot woman in a bathrobe is using clothespins to pin up a bedspread. To her left, her younger brother is beating the dust out of a mattress. In the background, her uncle 2A0 is painting a bookcase. 5R******************* llffect rating during this task: 1.Did the student attend closely to the task? YES NO 2.0ther comments on affect during this task? 1 . Did the student generally use proper inflection while reading the passages? (circle one) YES NO 2.. Was the student reasonably fluent? (circle one) YES NO W 'WI .I’jA _ni 2A1 PleaSe record the time. Time: Take a five minute break if the student wishes. Turn the tape recorder off. After the break, TURN THE TAPE OVER, turn the tape and continue. recorder back on ‘L‘ h.‘ e s l 2A2 Silent Reading Measure 1. IS THE TAPE TURNED OVER? IS IT TURNED ON? IF NOT, DO SO. Please record the time. Time: Turn to the appropriate form of the Silent Reading Measure for this student. USE THE TABLE OF CONTENTS. ****THIS STUDENT WILL USE FORM SPACHE RA**** Read the following instructions to the student: "YOU ARE GOING TO BE READING SOME SELECTIONS SILENTLY ONE AT A TIME. I'LL BE ASKING YOU ABOUT EACH SELECTION AFTER YOU READ IT. I'LL TELL YOU WHEN TO START. PLEASE TELL ME WHEN YOU HAVE FINISHED READING.” For each selection do the following: a. RESET THE STOPWATCH TO ZERO. b. Tell the student to begin. Then, you START THE STOPWATCH. c. When the student has finished the selection, stop the clock and record the time and cover the selection with a card. For each selection conduct the comprehension procedure as follows: a. Ask the student. "TELL ME IN ONE SENTENCE WHAT THE MAIN IDEA IN THIS SELECTION IS." Check all responses made. If 'Other' write down response. Note: If student attempts to recite the whole selection, reprompt: "Please tell me just the MAIN idea.” b. Ask any questions which follow and check all responses made. If 'Other' write down response. c. Say to the student. ”TELL ME WHAT WAS IN THE SELECTION, STARTING AT THE BEGINNING." In the first column, number all memories in order of recall. When the student has finished, say, “CAN YOU REMEMBER ANYTHING ELSE?" Put a check in the second column next to the appropriate memory. Write any memories mentioned by the child which are not listed. d. Continue through all selections until the student takes more than 70 seconds or has fewer than 3 sequential memories. If, on the last selection, the student had fewer than 3 memories, then YOU reread the paragraph to him and reask the questions. REMEMBER: THE CRITERION RECALLS. 21.3 IS MORE THAN 70 SECONDS OR FEWER THAN 3 2AA Spache: 2a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. riding b. a wagon c. up and down hills d. (a boy: Bob) e. (a dog) f. other Do you think the dog enjoyed riding downhill in the wagon? Why? a. b. c. d. e. f. g. h. no he jumped out he was frightened he went too fast he thought he might get hurt other 'yes other 2A5 Tell me what the story was about, starting at the beginning. Bob has a little red wagon. He likes to ride in it. He pulls it __slowly __up the hill. __Then he rides it __quickly __down again. One day he took his dog with him. H“ He pulled . L I I L ‘__the dog __up the hill. A '__Then they rode ' down the hill but the dog did not like to ride down. He jumped out of the red wagon. Bob went down __by himself. __Now he does not try __to take his dog __in the wagon. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 2A6 Spache: 3a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. Bob; a boy . b. and Spotty: a dog c. take trips d. in the woods e. other Why did the dog have to sleep outside? a. he smelled bad b. he was being punished c. the skunk sprayed him d. he caught a skunk (half credit) e. other «steam—1 T .r'.'t" 21.7 Tell me what the story was about, starting at the beginning. Bob has a brown and white dog named Spotty. He is called Spotty because he has brown _spots on his nose. Bob always takes his dog on his trips to the woods. The dog helps scare the rabbits. Bob walks slowly, but his dog scampers through the leaves. __One day __Spotty _Ieft Bob __and went off ,__by himself. __Bob called and whistled, but the dog did not come back to him. After a while Bob heard the dog barking a long way off. Bob walked toward the sound of the barking until he found the dog. __Spotty thought he had caught a black and white kiten But it wasn't a kitten. it was a skunk. That night the dog had to sleep outside. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 2A8 Spache: Aa 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. - Time: Tell me in one sentence what the main idea in this selection is. a. Bob's: a boy's b. trip c. to the city market: to the fruit market d. (fruit) e. other Why did Bob have trouble buying fruit in a place that was filled with fruits and vegetables? a. they sell only in large quantities (by the crate) b. Bob didn't want to buy enough c. other I141. 21.9 Tell me what the story was about, starting at the beginning. Yesterday Bob took a trip to a city market that was somewhat like a store but a great deal bigger. It didn't have any bread or canned goods like the grocery stores. But there were a great many big boxes of vegetables and fruits. Bob was hungry and wanted just one plum or cherry to taste. He wondered if one of the men would sell him __just one plum. _Everyone __was buying the fruit __and vegetables ,__by the whole crate. _When Bob asked the man to sell him one plum he laughed and gave Bob an extra large plum wrapped in paper but wouldn't take any money. As he walked along eating the plum Bob watched the men unloading the trucks and big trailers. They would chop open the top of the crate so that anyone the price __could see ,__the fruit. ___If a buyer __liked the fruit __and was willing to pay 250 _ _he might buy _ _the entire truckload. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? . “-1-“ x .. - 251 Spache: 5a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. ship's boy: a boy: John Paul D. learns to be a sailor c. loves to sail d. (to America) e. other What leads you to believe that John Paul sailed on many voyages? a. he visited America often b. he learned to be an expert sailor c. he could sail with the best of men d. other Tell me what the story was about, starting at the beginning. 252 As a ship's boy, John Paul had all sorts of odd jobs on board. Sometimes he scrubbed decks or helped the cook. He cleaned the captain's cabin and ran errands but he had other duties that pleased him more. He helped to clean the guns which the merchant ship carried for protection. Several times he stood behind the big wheel to steer the ship. Captain Benson wrote in the ship's log or daily record that the trip was calm and smooth-sailing. Nothing unusual happened but every day was a real adventure for the new ship's boy. At the end of the voyage it was a thrill to sight land. When the ship docked near Fredericksburg, Virginia, John Paul was waiting to go on shore. John Paul's brother had a tailor shop in Fredericksburg and was very happy in his new home. He was eager to talk about the wonderful country but John Paul already loved America. During the next few years John Paul visited America often. He became used to the free and democratic ways of the new country. Meanwhile he had learned to be an expert sailor. Although he was not tall he was strong and quick. With his long arms he could haul or trim a Sail with the best of men. 253 NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 25A Spache: 6a 10 START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. description b. of mastodons: of ancient elephants c. (elephants) d. other Why do you suppose that the elephants of today do not have woolly hair? a. they live in warm regions b. they don't need it c. they are not the same kind of elephant d. they are a different kind e. other 255 Tell me what the story was about, starting at the beginning. Elephants are found wild today only in warm regions in tropical Africa and in India. The story was very different 50 thousand years ago. Then, two species of the elephant family roamed North America and Europe in vast numbers. One of them was the mastodon. The mastodon lived in the eastern part of our country during the peiod of the Great Ice Age. In the swamps that were formed when the ice disappeared, many of the huge creatures were trapped and killed. We have found some of their skeletons. At a glance, the mastodon must have looked much like the ants of today except that it was covered with coarse, woolly hair and its tusks were much larger. It was probably heavier than the elephants we know but not taller. Its head was flatter and its lower jaw longer. Its teeth were not like the teeth of the elephants of today. More than 200 years ago, the people of New England found bones of the mastodon when they dug ditches to drain swamps. __At first they thought that the bones they found were bones of ant people. __When they found teeth __that weighed more than four pounds apiece __they decided that the giants were giants indeed. IISIII III‘°IIII||IIIIIII|I“IIIIIIIIIIIIIIIIIIII NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 256 Spache: 7a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. using different reading speeds b. for different reading materials c. for different purposes d. (reading) e. other Why do you think slow readers tend to study everything they read. a. they're afraid they will miss something important b. they don't realize that reading speed can be adjusted c. other =1 war—m“ It; ' ' . I 257 Tell me what the story was about, starting at the beginning. Just as in driving a car, we use at least three speeds in reading. High gear in reading is called skimming, while studying is reading in low gear. Between these two, at a second gear is what might be called a moderate speed of reading. As you may have heard, the good reader adapts his rate to the purpose of his reading. The rate he uses is determined by how much he wants to get out of the material he is reading. __His rate is also influenced __by the difficulty __of the reading material. Thus, he shifts from gear to gear according to the amount he wants to retain or how difficult he finds the going. Skimming is useful for a number of situations in reading. We can use it when looking for a particular fact on a page or in a table. It is also appropriate when we have to cover a large amount of rial that is not too interesting or too important. Skimming may also be used to determine the general trends or t (1 0 of a selection when we do not have to know the fine details. It is also helpful when we are making a quick brush-up before recitation. Finally, it is very useful as the speed at which we would do pre-reading before studying intensively. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 258 Spache: 8a 10 START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what is the main idea in this selection. a. Exploration b. by Lewis c. and Clark d. of the Northwest Territories e. other Why did the expedition follow major rivers? provided water . provided a pathway . could use boats . other CLOUD It- 4 wan-"AQTT ’ i l I 259 Tell me what the story was about, starting at the beginning. President Thomas Jefferson, in 180A commissioned an expedition to go into the Northwest Territory to explore the land that was bought in the now-famous Louisiana Purchase. __Meriwether Lewis, __private secretary to the President, __and Captain William Clark of the United States Army headed the 26 men 1 who started up the Missouri River from St. Louis on May 21, 180A. On July 18 of that year the group reached the southwest corner __of the present state of Iowa, __proceeded northward along the Missouri, __and traversed parts of Iowa many times. The Lewis and Clark State Park west of Onawa was named in honor of these explorers. Sergeant Charles Floyd, a Kentucky backwoodsman and one of the most competent men of the party, became ill on August 19, 180A and died the following day. __His body was laid to rest __upon a high bluff __near the present site of Sioux City where it is marked with a tall monument. The Floyd River and Sergeant Bluff were named in his honor. The expedition proceeded westward to the mouth of the Columbia River, and returned to Washington during the early months of 1807. Lewis was appointed governor of the Louisiana Territory and Clark was named governor of the Northwest Territory. when“: 2 _ NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 260 ******************** Affect rating during this task: l.Did the student attend closely to the task? YES NO 2.0ther comments on affect during this task? 261 Listening Comprehension Measure 1. 2. Please record the time. Time: Take the test booklet away from the student. Turn to the appropriate form of the Listening Comprehension Measure for this student. USE THE TABLE OF CONTENTS. You will be reading the selections to the student FROM the STUDENT'S test booklet. ****THIS STUDENT WILL USE FORM DURRELL RA**** Read the following instructions to the student: "YOU ARE GOING TO BE LISTENING TO SOME SELECTIONS. I'LL BE ASKING YOU ABOUT EACH SELECTION AFTER YOU'VE LISTENED TO IT. Read at a moderate pace using normal inflection. Start the stopwatch when you begin reading and stop it when done. Record the time in the appropriate place for the selection. For each selection conduct the comprehension procedure as follows: Ask the student, "TELL ME IN ONE SENTENCE WHAT THE MAIN IDEA IN THIS SELECTION IS." Check all responses made. If 'Other' write down response. Note: If student attempts to recite the whole selection, reprompt: "Please tell me just the MAIN idea.” Ask any questions which follow and check all responses made. If 'Other' write down response. Say to the student, ”TELL ME WHAT WAS IN THE SELECTION, STARTING AT THE BEGINNING." In the first column, number all memories in order of recall. When the student has finished say, "CAN YOU REMEMBER ANYTHING ELSE?” Put a check in the second column next to the appropriate memory. Write any memories mentioned by the child which are not listed. Continue through all selections until the student has fewer than 3 memories in sequential recall. 262 Ourrell: 23 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. chickens: hen: chicks b. looking c. for food d. (dog) e. (bird) f. other Why did the hen fly at the dog? to frighten the dog away to protect the chicks because the dog bothered her the hen didn't like the dog to protect the food the dog was playing with her chickens other ill-9100.000 .Ch fit EST." ' Li ‘I—w—m ill 263 —.‘ ell me what the story was about, starting at the beginning. A hen had six little yellow chickens. One morning she took them for a walk. They looked for something to eat. They found some seeds and sand. A dog came to play with them. The hen did not like the dog. She flew at the dog and made him run away. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 26A Ourrell: 3a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. boys b. built house c. in woods d. (pigs) e. (apples) f. other Why were the pigs able to get to the apples? a. door was open b. apples were on the floor c. the boys were not there d. other ..l n. : L L 265 Tell me what the story was about, starting at the beginning. Three boys built a house in the woods. They put a table and two old chairs in it. There was a basket full of apples under the table. One afternoon they went away and left the door open. When they came back they found two little pigs eating the apples. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 266 Durrell: Aa I. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. little girl b. waiting: nobody there to pick her up c. at train station: after train ride: getting off the train d. for mother e. other Why did the girl have to wait at the station? 3. her mother wasn't there b. her mother's car broke down c. no one was there d. other How did the man inside the station know why the girl's mother was late? a. the mother called the station b. the mother sent a message c. other 267 .4 __A little girl __got off the train __all alone. There was nobody at the station to meet her. She asked the man inside the station where her mother was. He said that her mother could not get the car started. A man was trying to fix it. The little girl sat down to wait. A few minutes later a big car came around the corner with her mother in it. The little girl got in and they drove home. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? ell me what the story was about, starting at the beginning. 268 Durrell: 5a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. first man(note: the word FIRST must be used) b. went up c. in a balloon d. other On the second trip, why did they probably not use a long rope? a. balloon travelled too far b. wanted the balloon to go higher c. didn't have a long enough rope d. he knew how to handle the balloon e. he already knew it was safe f. so he could see how far and high they could go 9. they didn't need it (half credit) h. other 269 Tell me what the story was about, starting at the beginning. __About one hundred and fifty _years ago __in France, __the first man __went up in a balloon. __His balloon was made of __paper __covered with strips of cloth __to make it strong. __A long rope kept it __from going too high. __tater this man took a friend __up in the balloon with him. __On this trip they rose over five hundred feet. The trip lasted thirty minutes. They came down several miles from where they started. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? .'I "-1 ii 11 270 Durrell: 6a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: 2. Tell me in one sentence what the main idea in this selection is. a. glass b. early American c. uses of d. Sandwich glass e. (glass beads) f. other 3. How did the Sandwich glass works make its glass? a. the way the man did b. by using iron molds c. they melted it down d. other 271 Tell me what the story was about, starting at the beginning. Early settlers in America found that Indians would sell skins and land for glass beads. Many men earned their living __by making glass beads and bottles. In 1827 a man invented a way to press molten glass into iron molds. The most famous glass works was in the town of Sandwich in Massachusetts. The Sandwich glass had a bright silvery appearance and it could be melded into very elaborate and attractive patterns. Beautiful lamps and candlesticks as well as all sorts of dishes were made from this glass. __In many New England homes __pieces of Sandwich glass __are still found on display. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 272 Durrell: 7a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. basketball b. in college c. (invented) d. other Why was the playing area for each player restricted? a. to make it easier for girls b. they thought girls weren't as strong as boys c. other 273 Tell me what the story was about, starting at the beginning. __Basketball __is one of the more recent games. __It was devised __by a college instructor __who desired a game to interpose __between the football __and baseball seasons. __The game demands __precison of movement, __concentration, and great endurance. It is more popular in those localities where it does not compete with hockey. __Opinion differs as to whether it is a satisfactory game for girls. It has been modified to make it less strenuous for them __by restricting the playing area of each player. Some of the large Western universities have audiences of over twenty thousand at their conference games. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? 271. Durrell: 8a 1. START WATCH. YOU READ (if listening comprehension) or HAVE STUDENT READ (if silent comprehension). STOP WATCH. COVER SELECTION (if silent comprehension). RECORD TIME. Time: Tell me in one sentence what the main idea in this selection is. a. growth of b. railroads: trains: train companies c. after Civil War d. other Why do you think it would be important to extend the railroads beyond populated regions? a. people could settle new land b. resources could be transported back into the populated regions c. other How did Congress encourage the development of railroads? a. gave land b. other 275 5. Tell me what the story was about, starting at the beginning. __Railroad communication __developed rapidly __just after the Civil War. __Between 1865 and 1873, __thirty-five thousand __miles of track were laid. __This doubled the distance ‘__people could travel __by railroad. __Some of the new roads __connected important cities __and some extended westward beyond populated regions. _Congress ‘__favored this sudden __development __by granting land to companies ‘__interested in furthering ,__the expansion. ___Grants included territory __Iying within twenty miles __of the proposed roadbed. Alternate sections were allotted to the railroad. Those in between were reserved for homesteaders. The sale of sections of land owned by the railroad was made easier through this checkerboard arrangement. NOW ASK: CAN YOU REMEMBER ANYTHING ELSE? *aasesnsssaesasnaann Affect rating during this task: l.Did the student attend closely to the task? YES NO 2.0ther comments on affect during this task? 276 Word Comprehension Measure 1. 2. Please record the time. Time: For each of the words in the following two lists, ask the student the following question: "WHAT DOES MEAN?" For each word, check the appropriate column: 'Correct Meaning', 'Incorrect Meaning', or 'No Response'. 277 Vocabulary for SPACHE RA Word Cor. Inc. NR l.scamper 2.crate 3.merchant ship A.Ship's log 5.species 6.coarse (spell it) 7.tusks 8.skimming 9.moderate 10.retain ll.recitation 12.intensive1y l3.expedition lA.competent 4L. 278 Vocabulary for DURRELL RA Word Cor. Inc. NR l.elaborate 2.attractive 3.molten A.devised 5.recent 6.precision 7.endurance 8.1ocalities 9.modified lO.strenuous ll.restructuring 12.extended l3.allotted 1A.expansion ******************** Affect rating during this task: 'l.Did the student attend closely to the task? YES NO 2.0ther comments on affect during this task? '- not. allw 279 Test Administrator Debriefing Form ‘0 *kiulnt*icinh':********A*ka'cic*idh'c*********************** DO NOT DISMISS THE STUDENT YET. ************************************************** Examine the entire test booklet to make sure that all tests are completed. Check them off. Interview SORT SORT Vocabulary Gates-Mckillop Gray Sullivan Silent Reading Listening Comprehension Vocabulary If all are done, DISMISS the student. Please record the time. Time: Please answer the following questions concerning the just completed testing session. What was the student's overall affect? Animated Flat Sullen Hostile Passive Attentive Fidgety Nervous What was the student's mean length of utterance? One word Phrase Sentence Multiple Sentences What was the student's general sophistication of language? High Average Low Did the student's speech patterns demonstrate any major differences from standard English? No Yes. Specify: While reading silently and independently, what were the student's characteristics? Raced through material 100 Squirmed Was attentive Gave up Asked for assistance Other. Specify: 280 Did the student appear to have No Yes. Specify: any difficulty hearing? Did the student appear to have aspects of reading? Yes Rubbing eyes Watery eyes Keeping text close to eyes No Enter any other comments about recorded elsewhere. any difficulty with the visual the session that have not been Appendix Ei Sample Simulated Clinician Diagnostic Workup Below is a sample run of the computer diagnostic aid. Note that the system assumes that the standard test battery has been given and asks for data based on that assumption. Clearly, in a production system, it would be more effective to use a micro computer or terminal connected to a mainframe computer to assist with test administration and automatically collect, integrate, and feedback the data. 281 Wing-inane - 282 USER:XSZ8 TASK: 20A3 JUL 8,1981 15:55:00 The Multiple Object Simulated Encounter System (MOSES) Version. Experimental Number: 1.02 Date: August 29, 1980 Developed by the Institute for Research on Teaching Michigan State University No system messages. The Simulated Clinician based on the Model of Reading and Learning is now in control. In which of the following contexts do you wish the Simulated Clinician to operate? l.Computer Aid in Diagnosis and Remediation 2.Computer Aid in Followup 3.Computer Directed Diagnosis and Remediation A.Computer Directed Followup 5.Preceptor Instruction Enter a number from 1 to 5 corresponding to your choice: 1 Please identify the clinician. Enter the clinician's last name: WAGNER Enter the clinician's first name: CHRIS Enter the clinician's middle initial: C Enter the clinician's identification number (CIN): 1 Is the following data correct? CLINICIAN LAST NAME : WAGNER CLINICIAN FIRST NAME : CHRIS CLINICIAN MIDDLE INITIAL : C CLINICIAN IDENTIFICATION NUMBER : 1 Y Please identify the client. Enter the client's last name: SUBJECT ll7#l Enter the client's first name: Enter the client's middle initial: Enter the client's identification number (CIN): 283 0 Enter the client's grade placement as a number between 1 and 20. 7 Is the following data correct? CLIENT LAST NAME : SUBJECT 117#1 CLIENT FIRST NAME : CLIENT MIDDLE INITIAL . CLIENT IDENTIFICATION NUMBER : O O Y Certain routine information must be collected from the client. The information collection procedures will require approximately one hour of clinician time. The information collection procedures are described both here in the computer files and the external document entitled "Otto Junior High Initial Data Collection Procedures.“ Please follow the procedures and collect the required information from the client. Do you need a description of any of the information collection procedures? N Concerning the CLIPIR Standard Interview Procedure: Did the reader's response to the question: "When reading your school books, do you know most of the words right away?" indicate a potential problem with Instant Word Recognition ability? N Did the reader's response to the question: "What do you do when you come to a word you don't know?" indicate a potential problem with Decoded Word Recognition ability? Y Did the reader's response to the question: "When you listen to people read out loud, how can you tell if they're doing a good job?” indiate a potential problem with oral reading? Y Did the reader's response to the questions: ”What kind of reading is hard for you to remember?" or ”Do you have trouble remembering what you read?” or "What kind of books are hard to read?“ indiate a potential problem with silent reading? N Did the reader's response to the questions: ”Are you having any trouble finishing your reading assignments in your classes?" or Wait “Kari-.33 I i . 28A "Is there anything particular that you want to be able to do in reading by the end of this year?“ indiate a potential problem with motivation to read? Did the reader's response to the questions: ”What is punctuation for?” indicate a potential problem with punctuation usage? Y Did the reader's response to the questions: "Do your eyes bother you after you've been reading for a while?" and "Do you have trouble seeing things clearly? indiate a potential problem with visual acuity? Y Did the reader's response to the question: ''Do you have any trouble hearing people when they talk to you? indicate Y a potential problem with auditory acuity? Did the reader's response to the queStion: ”What do you read outside class?" or any other impressions from the session indicate N 'Standard a potential lack of adequate reading practice? Interview Data Collection Complete. OConcerning the CLIPIR Sort Procedure: ODid the Y How many 19 How many 0 How many 1 ODid the Y How many 15 How many 1 How many 8 ODid the Y How many 11 How many 3 How many 6 OOid the Y How many 6 reader attempt list 1? words did the reader recognize INSTANTLY on list words did the reader recognize through MEDIATION words did the reader MISCALL on list 1? reader attempt list 2? words did the reader recognize INSTANTLY on list words did the reader recognize through MEDIATION words did the reader MISCALL on list 2? reader attempt list 3? words did the reader recognize INSTANTLY on list words did the reader recognize through MEDIATION words did the reader MISCALL on list 3? reader attempt list A? words did the reader recogniZe INSTANTLY on list I? on list 1? 2? on list 2? 3? on list 3? A? m} J How many 2 How many 12 ODid the Y How many 8 How many 2 How many IA ODid the N 285 words did the reader recognize through MEDIATION on list A? words did the reader MISCALL on list A? reader attempt list 5? words did the reader recognize INSTANTLY on list 5? words did the reader recognize through MEDIATION on list 5? words did the reader MISCALL on list 5? reader attempt list 6? ONow, concerning the vocabulary measure on the last list. For list 1 For list 0 For list 6 5, how many words were correctly defined? 5, how many words were incorrectly defined? 5, how many definitions were not attempted? Review the pattern of miscalls on the lists. On how many of the miscalls were only the initial 9 On how many of the miscalls were only the final 0 For how many of shaped like the 21 For how many of O OConcerning the How many of the 13 How many of the 13 How many of the 3 letters correct? letters correct? the miscalls was the miscall a different word that was stimulus word? the miscalls was there confusion of similar letters? CLIPIR Gates-Mckillop Procedure consonant clusters were called correctly? phonograms were called correctly? nonsense words were called correctly? Review the miscues. Did the miscues indicate any confusion of similar letters? N OConcerning the CLIPIR Gray Oral Reading Procedure: Did Y How 35 How 0 Did Y .How 35 How 1 the reader attempt selection 3 many seconds did it take the reader to read selection 1 many errors did the reader make when reading selection 1 the reader attempt selection A many seconds did it take the reader to read selection 2 many errors did the reader make when reading selection 2 286 Did the reader attempt selection 5 Y How many seconds did it take the reader to read selection 3 A2 How many errors did the reader make when reading selection 3 3 Did the reader attempt selection 6 Y How many seconds did it take the reader to read selection A 52 How many errors did the reader make when reading selection A I. Did the reader attempt selection 7 Y How many seconds did it take the reader to read selection 5 98 How many errors did the reader make when reading selection 5 15 Did the reader attempt selection 8 Y . How many seconds did it take the reader to read selection 6 123 How many errors did the reader make when reading selection 6 17 Did the reader attempt selection 9 N Did the student recognize most words instantly in his grade level paragraph? (answer NO if not attempted) N Did the student decode most words not known immediately when reading at or above his grade placement? (answer NO if not attempted) N Did the pattern of student miscues indicate that visual discrimination was 0.K.? Y Did the student appear to use context to adjust miscues? N Did the student use adequate phrasing during oral reading? Y Did the student read at an adequate rate of speed? N Did the student observe punctuation while reading? Y Did the student attempt to correct any miscalls made? N OConcerning the CLIPIR Silent Reading Measure: Was the reading rate adequate in the grade level paragraph? (answer N0 if not attempted) N What was the number of the most difficult passage attempted by the reader? 6 287 Concerning selection 2 What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 3A What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 1A What percentage of the inference question was answered correctly (again, only the number)? 0 What was the reading time in seconds? 12 Concerning selection 3 What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 100 What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 1A What percentage of the inference question was answered correctly (again, only the number)? 100 What was the reading time in seconds? 66 Concerning selection A What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 3A What 2~memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 9 What percentage of the inference question was answered correctly (again, only the number)? 100 What was the reading time in seconds? 100 Concerning selection 5 What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 3A What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 2 What percentage of the inference question was answered correctly (again, only the number)? 0 What was the reading time in seconds? -99 Concerning selection 6 What percentage of the main idea concepts did the reader give (enter the 288 number only, e.g., 502 enter as 50)? 25 What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 0 What percentage of the inference question was answered correctly (again, only the number)? 0 What was the reading time in seconds? 71 OConcerning the CLIPIR Listening Comprehension Measure: What was the number of the most difficult passage attempted by the reader? 7 Concerning selection 2 What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 3A What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 21 What percentage of the inference quesion was answered correctly (again, only the number)? 100 Concerning selection 3 What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 100 What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 29 What percentage of the inference quesion was answered correctly (again, only the number)? 100 Concerning selection A What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 25 What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 50 What percentage of the inference quesion was answered correctly (again, only the number)? 100 Concerning selection 5 What.percentage of the main idea concepts did the reader give (enter the runnber only, e.g., 502 enter as 50)? 3A What.2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? 289 (again, enter the number only, e.g., 502 enter as 50) 30 What percentage of the inference quesion was answered correctly (again, only the number)? 0 Concerning selection 6 What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 0 What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 13 What percentage of the inference quesion was answered correctly (again, only the number)? 0 Concerning selection 7 What percentage of the main idea concepts did the reader give (enter the number only, e.g., 502 enter as 50)? 50 What 2 memories did the reader have in the sequential recall task (either in sequence or on further prompting)? (again, enter the number only, e.g., 502 enter as 50) 22 What percentage of the inference quesion was answered correctly (again, only the number)? 0 OConcerning the CLIPIR Vocabulary Measure: how many words were correctly defined? 2 How many words were incorrectly defined? 5 How many word definitions were not attempted? 21 Concerning the CLIPIR Tester Debriefing Procedure: Did the tester rate the reader's general sophistication of language as "low”? Y Did the tester rate the student's overall affect as hostile, sullen or flat? N Tester Debriefing Data Collection Complete. DATE IS:JUL 8, 1981 TIME 15:15:57:A3 INSTANT WORD RECOGNITION VITAL SIGN -1.000 DECODED WORD RECOGNITION VITAL SIGN -1.000 MEANING VOCABULARY VITAL SIGN -1.000 ORAL READING VITAL SIGN -1.000 '290 SILENT READING COMPREHENSION VITAL SIGN 1.000 LANGUAGE EXPERIENCE VITAL SIGN 0.667 MOTIVATION TO READ AND LEARN TO READ VITAL SIGN 1.000 VISUAL DISCRIMINATION OF WORDS AND WORD SEGMENTS AND LETTERS 0.667 SOUND SYMBOL ASSOCATION FOR CONSONANT BLENDS -1.000 SOUND SYMBOL ASSOCIATION FOR PHONOGRAMS -1.000 ABILITY TO BLEND RECOGNIZED WORD PARTS -1.000 ABILITY TO PARAPHRASE -1.000 ABILITY TO DISTINGUISH MAIN IDEAS AND DETAILS -1.000 ABILITY TO PREDICT MEANING FROM CONTEXT -1.000 ABILITY TO DETERMINE CAUSALITY 0.0 PHRASING OF SPEECH DURING ORAL READING 1.000 READING RATE DURING ORAL READING -1.000 READING RATE DURING SILENT READING -1.000 VISUAL ACUITY -l.OOO RELEVANT ACADEMIC TIME ENGAGED ON READING TASKS 1.000 ADEQUACY OF SELF FEEDBACK DURING READING PRACTICE -1.000 ATTITUDE TOWARD READING 1.000 INSIGHT PUNCTUATION GIVES SIGNALS FOR PROPER PHRASING 1.000 DATE IS:JUL 8, 1981 TIME 15:15:57:A3 CLIPIR STANDARD INTERVIEW PROCEDURE +1.000 IWR GRADE EQUIV CLIPIR INTERVIEW +1.000 DWR GRADE EQUIV CLIPIR INTERVIEW -l.000 ORAL READING CLIPIR INTERVIEW -1.000 SILENT READING CLIPIR INTERVIEW +1.000 ATTENTION MOTIVATION CLIPIR INTERVIEW +1.000 USE OF PUNCTUATION CLIPIR INTERVIEW -l.000 ‘VISUAL ACUITY CLIPIR INTERVIEW “i. 291 -1.000 AUDITORY ACUITY CLIPIR INTERVIEW -1.000 TIME ENGAGED CLIPIR INTERVIEW +1.000 CLIPIR SORT PROCEDURE +1.000 IWR GRADE EQUIV CLIPIR SORT -1.000 DWR GRADE EQUIV CLIPIR SORT -l.000 WC GRADE EQUIV CLIPIR SORT -l.000 USE OF INITIAL LETTERS FOR WORD RECOGNITION +1.000 USE OF FINAL LETTERS FOR WORD RECOGNITION +1.000 USE OF CONFIGURATION FOR WORD RECOGNITION -l.000 VIS DISC CLIPIR SORT +1.000 CLIPIR GATES MCKILLOP PROCEDURE +1.000 SSA CONS BLEND CLIPIR GATES MCKILLOP -1.000 SSA PHONOGRAMS CLIPIR GATES MCKILLOP -l.000 BLENDING CLIPIR GATES MCKILLOP -l.000 VIS DISC CLIPIR GATES MCKILLOP +1.000 CLIPIR GRAY PROCEDURE +1.000 OR GRADE EQUIV CLIPIR GRAY -1.000 IWR GRADE EQUIV CLIPIR GRAY -1.000 OWR GRADE EQUIV CLIPIR GRAY -l.000 VIS DISC CLIPIR GRAY -I.000 USE CONTEXT CLIPIR GRAY -l.000 PHRASING CLIPIR GRAY +1.000 ORAL READING RATE CLIPIR GRAY -1.000 INSIGHT PUNCTUATION CLIPIR GRAY +1.000 ORAL READING CORRECTION CLIPIR GRAY -l.000 CLIPIR SILENT PROCEDURE +1.000 SILENT READING RATE CLIPIR SILENT Au ‘ L 292 -1.000 RC GRADE EQUIV CLIPIR SILENT +1.000 MAIN IDEA CLIPIR SILENT -l.000 CAUSALITY CLIPIR SILENT -l.000 PARAPHRASE CLIPIR SILENT -l.000 CLIPIR LISTENING PROCEDURE +1.000 LC GRADE EQUIV CLIPIR LISTENING +1.000 MAIN IDEA CLIPIR LISTENING -1.000 CAUSALITY CLIPIR LISTENING +1.000 PARAPHRASE CLIPIR LISTENING -l.000 CLIPIR VOCABULARY PROCEDURE +1.000 WC GRADE EQUIV CLIPIR VOCABULARY -l.000 CLIPIR TESTER DEBRIEFING PROCEDURE +1.000 LANGUAGE EXPERIENCE CLIPIR TESTER DEBRIEFING -1.000 ATTITUDE CLIPIR TESTER DEBRIEFING +1.000 INSTANT WORD RECOGNITION VITAL SIGN -l.000 DECODED WORD RECOGNITION VITAL SIGN -1.000 MEANING VOCABULARY VITAL SIGN -l.000 ORAL READING VITAL SIGN -l.000 SILENT READING COMPREHENSION VITAL SIGN 1.000 LANGUAGE EXPERIENCE VITAL SIGN 0.667 MOTIVATION TO READ AND LEARN TO READ VITAL SIGN 1.000 VISUAL DISCRIMINATION OF WORDS AND WORD SEGMENTS AND LETTERS 0.667 SOUND SYMBOL ASSOCATION FOR CONSONANT BLENDS -1.000 SOUND SYMBOL ASSOCIATION FOR PHONOGRAMS -1.000 ABILITY TO BLEND RECOGNIZED WORD PARTS -1.000 ABILITY TO PARAPHRASE -1.000 ABILITY TO DISTINGUISH MAIN IDEAS AND DETAILS 293 -1.000 ABILITY TO PREDICT MEANING FROM CONTEXT -1.000 ABILITY TO DETERMINE CAUSALITY 0.0 PHRASING OF SPEECH DURING ORAL READING 1.000 READING RATE DURING ORAL READING -1.000 READING RATE DURING SILENT READING -1.000 VISUAL ACUITY -1.000 RELEVANT ACADEMIC TIME ENGAGED 0N READING TASKS 1.000 ~ ADEQUACY DE SELF FEEDBACK DURING READING PRACTICE ‘1.000 ATTITUDE TOWARD READING 1.000 INSIGHT PUNCTUATION GIVES SIGNALS FDR PROPER PHRASING 1.000 *Sstop Appendix 9; Clinician Diagnostic Workup A sample clinician diagnosis is included here. Such diagnoses were completed for a random sample of about one half of the pretested students (20). The diagnoses were based on the data from the student's ,m protocol notebook without any identification of the student. 29A 295 Does the student have a problem with INSTANT WORD RECOGNITION?(IWR) (Circle One) Yes No On what basis was this decision made? 1. *Describe one factor contributing to the problem with Instant Word Recognition. Deciding about the identity of a word based on one or two beginning letters. *Suggest remedial procedures for alleviating this factor. Approach all unknown or miscalled words from a DWR perspective (with some exceptions like "once”). See DWR l. 2. *Describe another factor contributing to the problem with Instant Word Recognition. Inattention to all elements of the visual display, particularly medial phonograms. *Suggest remedial procedures for alleviating this factor. Highlight, by size, the salient portions of the word, e.g., blOUse. The ou digraph is a common one and can be transferred to other words subsequently. 3. *Describe another factor contributing to the problem with Instant Word Recognition. Overreliance on a store of relatively few highly redundant words common to reading materials in grades one through three. *Suggest remedial procedures for alleviating this factor. See 1 and 2 above. A. *Describe another factor contributing to the problem with Instant Word Recognition. Insufficient extended reading practice which could act to expand store of instantly recognized words. *Suggest remedial procedures for alleviating this factor. See DWR #A. 5. *Describe another factor contributing to the problem with Instant Word Recognition. Auditory acuity problem. *Suggest remedial procedures for alleviating this factor. Screening with audiometer. Possible use of hearing aid or positioning student close to teacher so face to face contact can be maintained. 6. *Describe another factor contributing to the problem with Instant Word Recognition. Auditory discrimination problem. *Suggest remedial procedures for alleviating this factor. Wepman auditory discrimination test. If results positive, go to 5 above. 7. *Describe another factor contributing to the problem with Instant Word Recognition. 296 Visual acuity, fusion, tracking (etc.) problems. *Suggest remedial procedures for alleviating this factor. Vision screening with Keystone visual screening. If positive, to optometrist or opthalmologist for corrective treatment. refer n ..W'm 297 Does the student have a problem with DECODED WORD RECOGNITION?(DWR) (Circle One) Yes No On what basis was this decision made? *Describe one factor contributing to the problem with Decoded Word Recognition. Inadequate mastery of basic redundant sound-symbol associations. *Suggest remedial procedures for alleviating this factor. ' Use materials which begin with one-syllable words which incorporate the major redundant phonograms in isolation and in an extended contextual setting. The progression is from regular single vowel words (ship, black) to more irregular multiple voweled words (guided). Practice words in isolation and in extended contextual setting. *Describe another factor contributing to the problem with Decoded Word Recognition. ’ Inability to reliably differentiate root words from bound morphemic affixes. , *Suggest remedial procedures for alleviating this factor. Use materials which move from single syllable words to the affixed form such that practice is given in isolation and in an extended contextual setting. (star/starring: stare/staring: cut/cutter: cute/cuter: etc.) *Describe another factor contributing to the problem with Decoded Word Recognition. Absence of iterative strategies to (a)segment multisyllabic words, (b)transfer learned phonograms to each segment. (c)pronounce each segment, (d)recombine the segments into a spoken word, and (e)adjust the pronunciation if necessary. *Suggest remedial procedures for alleviating this factor. Use of materials which provide increaslingly lengthy multisyllabic words in isolation and in context, progressing from more to less regularity within segments. *Describe another factor contributing to the problem with Decoded Word Recognition. Insufficient extended reading practice with materials in a graduated sequence of difficulty (with respect to skills listed in factors 1, 2, and 3). *Suggest remedial procedures for alleviating this factor. Sufficient quantity of contextual material at every level in the hierarchy to consolidate recognition of major repetative phongrams. *Describe another factor contributing to the problem with Decoded Word Recognition. See IWR #5 (Auditory Acuity) *Suggest remedial procedures for alleviating this factor. See IWR #5 (Quditory Acuity) *Describe another factor contributing to the problem with Decoded 298 Word Recognition. See IWR #6 (Auditory Discrimination) *Suggest remedial procedures for alleviating this factor. See IWR #6 (Auditory Discrimination) *Describe another factor contributing to the problem with Decoded Word Recognition. See IWR #7 (Vision) *Suggest remedial procedures for alleviating this factor. See IWR #7 (Vision) *Describe another factor contributing to the problem with Decoded Word Recognition. Inadequate store of word meanings resulting in inability to adjust decoded word to correct spoken form. *Suggest remedial procedures for alleviating this factor. (I)Encourage use of context to support hypotheses about specific word meanings, (2)provide synonyms, actions, role playing, etc. to encourage educated guesses about word meaning. ' 299 Does the student have a problem with MEANING VOCABULARY?(MV) (Circle One) Yes No On what basis was this decision made? *Oescribe one factor contributing to the problem with Meaning Vocabulary. General sophistication of spoken language low, possibly due to inadequate or inappropriate language modelling. *Suggest remedial procedures for alleviating this factor. See DWR #8. *Describe another factor contributing to the problem with Meaning Vocabulary. Materials of sufficient complexity to enlarge semantic network are not accessible due to deficiencies in instant and decoded word recognition skills. *Suggest remedial procedures for alleviating this factor. See IWR/DWR. *Describe another factor contributing to the problem with Meaning Vocabulary. Ignorance of the meanings of common affixes, e.g., pre, sub, anti, inter, intra, un, de, dis, er, s, es, ed, etc. *Suggest remedial procedures for alleviating this factor. Always include meanings of affixed when doing decoding skills, and their effect on the root word. *Describe another factor contributing to the problem with Meaning Vocabulary. Ignorance of the etymology of common word parts, e.g., phono, bio, ology, graphy, micro, tele, etc. *Suggest remedial procedures for alleviating this factor. (None given by clinician) *Describe another factor contributing to the problem with Meaning Vocabulary. Insufficient practice in deducing word meaning from context. *Suggest remedial procedures for alleviating this factor. Provide a context for unknown words that is rich enough to support educated guesses about the meaning of the word in question. "1 “I. 300 Does the student have a problem with ORAL READING?(OR) (Circle One) Yes No On what basis was this decision made? 1. *Describe one factor contributing to the problem with Oral Reading. Instant word recognition inadequate. *Suggest remedial procedures for alleviating this factor. See IWR. 2. *Describe another factor contributing to the problem with Oral Reading. Decoded word recognition inadequate. *Suggest remedial procedures for alleviating this factor. See DWR. 3. *Describe another factor contributing to the problem with Oral Reading. Cognitive demands placed on reader to compensate for inadequate IWR/DWR skills results in insufficient attentional capacity to produce fluent, accurate translation from print to speech. *Suggest remedial procedures for alleviating this factor. Provide reading selections which parallel the sequence of skills built up from single to multisyllable and from more to less regular phonograms. Stress repeated reading for fluency and expressiveness. Use over-voicing with student. Model dramatic reading and have student repeat after you. Encourage self-evaluation: Does your reading sould less choppy? DOes it sound interesting to you? etc. Stress role of punctuation in regulating and enhancing fluent eXpression. ~ A. *Describe another factor contributing to the problem with Oral Reading. Inadequate performance in IWR/DWR results in slow, tedious and inaccurate reading which in turn discourages further practice. *Suggest remedial procedures for alleviating this factor. See #3 above. 5. *Describe another factor contributing to the problem with Oral Reading. Inability to use the sound system of the language to move beyond the single word stage. *Suggest remedial procedures for alleviating this factor. Repeated readings of short sections of text to a speed and accuracy criterion. This is a clinical procedure requiring one-on-one monitoring. Parental cooperation is usually necessary so student can practice at home. 6. *Describe another factor contributing to the problem with Oral Reading. Inappropriately rapid rate in the face of multiple miscues. *Suggest remedial procedures for alleviating this factor. See RX for DWR #1 and DWR #2. Stress accuracy over speed. Reward 301 stopping at unknown word for purposes of decoding. 302 Does the student have a problem with READING COMPREHENSION?(RC) (Circle One) Yes No On what basis was this decision made? *Describe one factor contributing to the problem with Reading Comprehension. IWR inadequate. *Suggest remedial procedures for alleviating this factor. See IWR. *Describe another factor contributing to the problem with Reading Comprehension. DWR inadequate. *Suggest remedial procedures for alleviating this factor. See DWR. *Describe another factor contributing to the problem with Reading Comprehension. Limited store of word meanings available to student. *Suggest remedial procedures for alleviating this factor. See MV #3, MV #A, MV #5. *Describe another factor contributing to the problem with Reading Comprehension. Dealing with difficulties in IWR/DWR/OR results in insufficient attentional capacity for systematic processing of incoming information. *Suggest remedial procedures for alleviating this factor. See IWR/DWR/OR remedial procedures. *Describe another factor contributing to the problem with Reading Comprehension. Inadequate or inappropriate cognitive strategies for anticipating. chunking, and finally retrieving content read silently. *Suggest remedial procedures for alleviating this factor. Provide comprehension decision aids to be used for organizing incoming material and for retrieving it. *Describe another factor contributing to the problem with Reading Comprehension. Inadequate volue of reading being done. *Suggest remedial procedures for alleviating this factor. Increase the amount of reading the student is required to read and organize/ both fiction and content area material. *Oescribe another factor contributing to the problem with Reading Comprehension. Inability to reliably deal with inferential questions. *Suggest remedial procedures for alleviating this factor. Help student articulate what prior knowledge s.he already has about the material read and how to trust that knowledge to fill in the gaps made by inferential questions. 303 Does the student have a problem with LISTENING COMPREHENSION?(LC) (Circle One) Yes No On what basis was this decision made? *Describe one factor contributing to the problem with Listening Comprehension. Limited store of word meanings. *Suggest remedial procedures for alleviating this factor. Increase the volume of reading the student is required to do. *Describe another factor contributing to the problem with Listening Comprehension. Inadequate strategies for organizing and then retrieving the gist of what was heard. *Suggest remedial procedures for alleviating this factor. See RC #5. These strategies should be transferrable to the listening setting. *Describe another factor contributing to the problem with Listening Comprehension. A gradual decrease in attention as the passage continues. *Suggest remedial procedures for alleviating this factor. This is more often a problem with the quality of the material being read than with the listener. Do not read material loaded with essentially unconnected detail, or materials that is poorly constructed in terms of logical structure and adequate redundancy. *Describe another factor contributing to the problem with Listening Comprehension. A gradual decrease in attention as the session continues. *Suggest remedial procedures for alleviating this factor. Do not read all sections consecutively. When student begins to fidget, proceed to next task and pick up listening again later. Does the student have a problem with ATTENTION / MOTIVATION?(A/M) (Circle One) Yes No On what basis was this decision made? *Describe one factor contributing to the problem with Attention / Motivation. Repeated failure at performing reading tasks results in avoidance behaviors. *Suggest remedial procedures for alleviating this factor. See RX for DWR/OR/RC/WM. Provide continuing stream of informational feedback: why a performance was inadequate: why a performance was better: why it was good. *Describe another factor contributing to the problem with Attention Wan-um: ‘17! .. a '. . ~ 301. / Motivation. Tediosness of performing reading tasks results in little or no practice of independent reading, contributing to further deterioration in performance and motivation. *Suggest remedial procedures for alleviating this factor. Provide time for sustained silent reading followed by oral reading and comprehension checks. Appendix pi Eguation pi Clinician ppp Computer Diagnoses The following pages contain lists of the following data in order: I. A list of all diagnostic terms used by the clinician. The numbers preceeding the list correspond to the combined vocabulary statement under which this factor is included. A list of the diagnostic terms used by the computer simulated clinician. The numbers preceeding the list correspond to the combined vocabulary statement under which this factor is included. The combined vocabulary. 305 l’ai’ ‘2'fl-L 306 (1) Instant word recognition vital sign (2) Decoded word recognition vital sign (3) Meaning vocabulary vital sign (A) Oral reading vital sign (5) Silent reading vital sign (6) Listening comprehension vital sign (7) Attention/motivation vital sign (8) Deciding about the identity of a word based on one or two beginning letters. (10) Inattention to all elements of the visual display, particularly medial phonograms. (A3) Overreliance on a store of relatively few highly redundant words common to reading materials in grades one through three. (1,38) Insufficient extended reading practice which could act to expand store of instantly recognized words. (11) Auditory acuity problem. (12) Auditory discrimination problem. (13) Visual acuity, fusion, tracking (etc.) problems. (17) Inadequate mastery of basic redundant sound-symbol associations. (18) Inability to reliably differentiate root words from bound morphemic affixes. (18,19,20) Absence of iterative strategies to (a)segment multisyllabic words, (b)transfer learned phonograms to each segment, (c)pronounce each segment, (d)recombine the segments into a spoken word, and (e)adjust the pronunciation if necessary. (38) Insufficient extended reading practice with materials in a graduated sequence of difficulty (with respect to skills listed in factors 1,2, and 3). (21) General sophistication of spoken language low, possibly due to inadequate or inappropriate language modelling. (1,2,3) Materials of sufficient complexity to enlarge semantic network' are not accessible due to deficiencies in instant and decoded word recognition skills. (22) Ignorance of the meanings of common affixes, e.g., pre, sub, anti, inter, intra, un, de, dis, er, s, es, ed, etc. (23) Ignorance of the etymology of common word parts, e.g., phono, bio, ology, graphy, micro, tele, etc. (37,38) Insufficient practice in deducing word meaning from context. (1,2,25) Cognitive demands placed on reader to compensate for inadequate IWR/DWR skills results in insufficient attentional capacity to produce fluent, accurate translation from print to speech. (l,2,2A,38) Inadequate performance in IWR/DWR results in slow, tedious and inaccurate reading which in turn discourages further practice. (17,18) Inability to use the sound system of the language to move beyond the single word stage. (2A) Inappropriately rapid rate in the face of multiple miscues. (l,2,A,5) Dealing with difficulties in IWR/DWR/OR results in insufficient attentional capacity for systematic processing of incoming information. (28,29,30) Inadequate or inappropriate cognitive strategies for anticipating, chunking, and finally retrieving content read silently. (38) Inadequate volume of reading being done. (30) Inability to reliably deal with inferential questions. (31) Inadequate strategies for organizing and then retrieving the gist mpg?!- Ffimfiet T" ‘5! 307 of what was heard. (A0) A gradual decrease in attention as the passage continues. (A0) A gradual decrease in attention as the session continues. (39,38) Repeated failure at performing reading tasks results in avoidance behaviors. (A2,38) Tediosness of performing reading tasks results in little or no practice of independent reading, contributing to further deterioration in performance and motivation. 308 (1) INSTANT WORD RECOGNITION VITAL SIGN (2) DECODED WORD RECOGNITION VITAL SIGN (3) MEANING VOCABULARY VITAL SIGN (A) ORAL READING VITAL SIGN (5) SILENT READING COMPREHENSION VITAL SIGN (6) LANGUAGE EXPERIENCE VITAL SIGN (7) MOTIVATION TO READ AND LEARN TO READ VITAL SIGN (1A) VISUAL DISCRIMINATION OF WORDS AND WORD SEGMENTS AND LETTERS (15) SOUND SYMBOL ASSOCATION FOR CONSONANT BLENDS (16) SOUND SYMBOL ASSOCIATION FOR PHONOGRAMS (19) ABILITY TO BLEND RECOGNIZED WORD PARTS (3A) ABILITY TO PARAPHRASE (35) ABILITY TO DISTINGUISH MAIN IDEAS AND DETAILS (37) ABILITY TO PREDICT MEANING FROM CONTEXT (36) ABILITY To DETERMINE CAUSALITY (25) PHRASING 0F SPEECH DURING ORAL READING (2A) READING RATE DURING ORAL READING (27) READING RATE DURING SILENT READING (13) VISUAL ACUITY (38) RELEVANT ACADEMIC TIME ENGAGED ON READING TASKS (Al) ADEQUACY OF SELF FEEDBACK DURING READING PRACTICE (39) ATTITUDE TOWARD READING (26) INSIGHT PUNCTUATION GIVES SIGNALS FOR PROPER PHRASING (11) AUDITORY ACUITY CLIPIR INTERVIEW (8) USE OF INITIAL LETTERS FOR WORD RECOGNITION (9) USE OF FINAL LETTERS FOR WORD RECOGNITION (10) USE OF CONFIGURATION FOR WORD RECOGNITION (29) MAIN IDEA CLIPIR SILENT (30) CAUSALITY CLIPIR SILENT (28) PARAPHRASE CLIPIR SILENT (32) MAIN IDEA CLIPIR LISTENING (33) CAUSALITY CLIPIR LISTENING (21) LANGUAGE EXPERIENCE CLIPIR TESTER DEBRIEFING (39) ATTITUDE CLIPIR TESTER DEBRIEFING 309 l)Instant word recognition 2)Decoded word recognition 3)Meaning Vocabulary A)0ral reading 5)Silent reading 6)Listening comprehension 7)Attention/motivation 8)Use of initial letters for word recognition ( 9)Use of final letters for word reocgnition (10)Use of configuration for word recognition (11)Auditory acuity (12)Auditory discrimination (13)Visua1 acuity (1A)Visual discrimination (15)Sound symbol association for consonant blends (l6)Sound symbol association for phonograms (l7)Sound symbol association general (l8)Segmentation (19)Blending (20)Adjustment (21)Language experience (22)Affix meanings (23)Word part meanings (2A)Oral reading rate (25)Oral reading fluency (26)Use of punctuation in oral reading (27)Silent reading rate (28)Silent reading paraphrasing (29)Silent reading main idea (30)Silent reading inference (31)Listening paraphrasing (32)Listening main idea (33)Listening inference (3A)General ability to paraphrase (35)General ability to get main idea (36)General ability to infer (37)Ability to use context (38)Relevant academic time engaged (39)Attitude toward reading (A0)Attention span (A1)Self feedback (A2)Interest (A3)0ther AAAAAAA" Appendix ii Instruction Observation Form Many iterations of the instruction observation form were used. Included here is the final version. The form consisted of two 8.5xlA inch pages taped together (reduced here, not proportionally). One form was used for each class. The most reliable information collected included type of task, time on task, materials, and instructor. Less reliable was the estimate of number of words or pages read in a task. Least reliable were measures of feedback on the second page, consisting of type and frequency of feedback from the instructor to the student. As can be seen, each entry in the form is divided into four parts one for each of four possible tasks in one class session. Rarely did more than four setting occur in one class session for one student. 310 i viiiIm. A A... s... inst... leek has... Lm\12 .I T++++++++ V + u + + + + + +......,,.,..,.,.. _mt. “ + + + + + +HIII m2. 2 + + + + __ +..111 rt. + + + + + “ +.....11 _.+ + + ++ + 3_ +111 _a++++++ “ +1.1 312 “when we»? 2.2 .. £2.22... N a. WNW 2 +++++++++++++++++++++++++ w +++++++++++++++++++++++++ T +++++++++++++++++++++++++ r +++++++++++++++++++++++++ w e++++++++++++++++++++++++ r +++++++++++++++++++++++++ We+++++++++++++++++++++++++ 313 Index abdominal pain, Al acute abdominal pain, Al agreement among diagnostic sources, 102, 106 agreement among diagnostic sources: critical performances, 102 agreement among diagnostic sources: profiles, lOA agreement among diagnostic sources: the statistic definition, 102 agreement statistic, 102 alderson, 26 allocation of maneuvers, 80, 85 alternate forms, 136 analytic procedures, 80 appendicitis, Al applying formal methods for aggregating experience to reading, 9 association of drug treatment and mortality for hypertension, 36 association of ebv virus and incidence of mononucleosis, 3A association of gain and class attendance, 125 association of mortality and surgical procedure, 35 association of mortality and surgical procedure stratified by liver function, 35 association of myocardial infarction and use of oral contraceptives, association of smoking and chronic bronchitis, 32 associations of outcome and time on class activities, 127 attendance, 117, 12A attention/motivation, 68 attributable risk,definition, 2A available data by student, 8A average agreement on total diagnosis across cases, 106 barnoon 8 wolfe, 1A benger, A .bernoulli, 39 biases in using experience, 19 blood-letting, 3, 39 bloom, A8 bronchitis, 31 brophy 8 evertson, 5O budd, 39 burdette 8 gehan, 8, 3A calfee, 53 carroll, A8, A9 case,definition, l3 center for disease control, A0 chall, 52 characteristics of diagnosis and cross-sectional analysis, 98 characteristics of instructional time by treatment group, 119 characteristics of treatments and outcomes, 116 cholera, 7 classic epidemiological study, 7 clinical data analysis, 116 clinical data analysis: d -> t -> o, 116 clinical problem solving model, 5, 12 clinical research, 6, 22 clinical research examples, 3A 33 311. clinical research,basic tenet, 23 clinical research,definition, 6, 23 clinical research,example, 3A, 36 clinical research,examples, 8 clinical research,improved care, 39 clinician diagnosis, 99 clinician,definition, 16 clinician,tasks of, 13 coleman, A2 collen, 29, A0 computerized clinician diagnosis, 99 conclusions and implications, 1A5 criteria for decision rule diagnosis, 7A critical perfomances,prevalence rates, 99 critical performances,decoded word recognition, 5A critical performances,instant word recognition,. 5A critical performances,listening comprehension, 55 critical performances,oral reading, 55 critical performances,silent reading comprehension, 55 critical performances,vocabulary, 55 cross-sectional research, 31 d - diagnostic characteristics, 5 data analysis,clinical, 116 data analysis,clinical,prospective, 128 data analysis,clinical,retrospective, 123 data analysis,epidemiological, 97 data analysis,epidemiological,diagnosis, 98 data analysis,epidemiological,prospective, 112 data analysis,epidemiological,retrospective, 108 data analysis,other,alternate forms, 136 data analysis,other,grouping decisions, 13A data analysis,other,reliability, 1A1 davis, 52 decision-rule diagnosis, 99 decision: choosing students to teach, 13A decoded word recognition, 5A, 6A decoded word recognition measure, 229 dedombal, Al dedombal, leaper, horrocks, staniland, 8 mccann, A1 degowin 8 degowin, 13 designs of research studies, 25 diagnosis, treatment, and outcomes in reading, A2 diagnosis,clinician, 99 diagnosis,computerized clinician, 99 diagnosis,decision-rule, 99 diagnosis,definition, l3 diagnosis,differences in,self perception, 109 diagnosis,differing definitions of, 101 diagnosis,in reading, A2 diagnosis,level one, 1A diagnosis,level two, 1A diagnosis,relationship to treatment, A5 diagnosis,reliability,procedures, 71 diagnosis,reliability,reading clinicians, AA 315 diagnostic sources, 99 diagnostic sources,agreement among, 102, 106 differences in equivalent forms: listening, lAl differences in equivalent forms: silent reading, 139 differences in rate of poor attenders based on relative outcomes, 12A differences in rate of problem diagnosis based on rating of language ability, 113 differences in rate of problem diagnosis based on ratings of mean length of utterance, 11A differences in self perception based on diagnosis, 109 differences in self perception based on diagnosis by sex, 111 distribution of alternate forms, 136 distributions of gains across treatment groups, 122 duffy. A6. A7 dunkin 8 biddle, A9 e - environmental characteristics, 5 ebv virus antibodies, 33 einhorn, 2, l8, l9 einhorn 8 hogarth, 2, l9 ekwall, A3 elstein, Shulman, 8 sprafka, 17 end-to-side portacaval shunt, 3A epidemiological data analysis, 97 epidemiological data analysis: p,e -> d, 97 epidemiological research examples, 31 epidemiology, 6, 22 epidemiologyibasic tenet, 23 epidemiologY.definition, 6, 22 epidemiology.example, 31, 32, 33 epidemiology.examples, 7 epidemiology,improved care. 39 equivalence of alternate forms, 136 execution of the study, 79 experience,biases in interpretation, l9 experience,breadth, l8 experience,learning from, 1 experience,reliability of interpretation, 20 experience,representativeness, l8 extrapolation, 80 factors affecting reading instruction, A8 false negatives, 19 false positives, l8 feinstein, 23, 25, 27 fisher, filby, marliave, cahen, dishaw, moore, 8 berliner, A8 followup,definition, 13 formal aggregation of experience, A formal aggregation of experience: the clinical and epidemiological research paradigms, 22 fox, hall, 8 elveback, 26 friedman, 1A, 23, 2A, 26, 31 gagne, A8 garrison, 3 general procedure, 60, 62, 71, 76, 79 generation,reading test battery, 85 316 911, AA good 8 brophy, A8, 50 gray oral reading test, 232 group instruction, 91 grouping decisions,. 13A guillain-barre syndrome, A0 gunshot wounds, A handling of instrusions, 8O hare, lA hoffmeyer, AA holland, 26 holland 8 kahausen, 26 homework, 91 hypertension, 36 i. procedures for defining a model of process. 59 ii. measurement device creation procedures, 61 iii. procedures for reliable diagnosis, 71 illusory correlation, 19 improved care. 39 incidence rates,definition, 2A informal aggregation of experience, I, 17 initial identification, 79, 85 instant word recognition, 63 instant word recognition measure, 217 instruction, 86 intake, 79, 82 introduction, 96 introduction to student, 213 ipsen, 2A iv. procedures for treatment observation, 75 jackson, A6, A8 kaiser - permanente medical care program, A0 kurland 8 molgaard, A0 leaper, horrocks, staniland, 8 dedombal, Al learning from experience. 1 learning from experience,formally, 2, 22 learning from experience,informally, 2, 17, 38 level one,diagnosis, 1A level one,treatment, 15 level two,diagnosis, 1A level two,treatment, 15 lilienfeld 8 lilienfeld, 3, 7,.26, 29, 32, 33. 39 listening comprehension, 55, 68 listening comprehension measure, 261 101.115. 3. 39 lusted, l2 maintenance plan, 79, 82 mausner 8 bahn, 26 mayo clinic, A0 mearIdays of instruction by treatment group. 117 measles, A0 measurement device creation procedures, 61 measurement device creation procedures,general, 62 measurement device creation procedures,reading, 62 317 measurement of transition, 80 measuring reading instruction, A9 methods and procedures, 57 midtesting of students, 9A model definition procedures, 59 model definition procedures,general, 60 model definition procedures,reading, 61 model,and transmission of knowledge, 37 model,case process, 37 model,case process,informal learning, 38 model,clinical problem solving, 12 model,medica1 diagnosis, 5 model,reading process, 52 models of the reading process, 51 mononucleosis, 33 moral, 52 moral,critical performances,decoded word recognition, 5A moral,critical performances,instant word recognition, 5A moral,critical performances,listening comprehension, 55 moral,critical performances,oral reading, 55 moral,critical performances,silen reading comprehension, 55 moral,critical performances,vocabulary, 55 multiphasic health testing service, A0 myocardial infarction, 32 neustadt 8 fineberg, A o - outcome characteristics. 5 objective, 79, 81 objectives, 57 objectives of study, 57 observation, 92 oral contraceptive use, 32 oral reading, 55, 66, 89, 90 other analyses, 13A outcome characteristics, 116 outcome measurement procedures. 78 outcome,change in prevalence of problems, 120 outcome,characteristics, 116 outcome,gain scores, 121 outcome,gain,relation to attendance, 12A outcome,gain,relation to treatment group, 130 outcome,relation to time on task, 126 outcome,relation to treatment group, 128 outcomes,in reading, A2 9 - personal characteristics, 5 pare, A performance gains by treatment group, 131 peterson 8 walberg, 50 portacaval shunt, 3A Portal hypertension, 3A posttesting of students, 9A Pretesting of students, 86 Prevalence of critical performance problems, 100 Prevalence rates, 99 Prevalence rates across treatment groups, 129 318 prevalence rates before and after treatment across treatment groups, 121 prevalence rates,critical performances, 99 prevalence rates,definition, 2A problem statement, 1 ' procedures in reading, 61, 62, 72, 77, 81 procedures,executing the study, 79 procedures,executing the study,general, 79 procedures,executing the study,reading, 81 procedures,measurement device creation, 61 procedures,measurement device creation,general, 62 procedures,measurement device creation,reading, 62 procedures,model definition, 59 procedures,model definition,general, 60 procedures,model definition,reading, 61 procedures,outcome measurement, 78 procedures,reliable diagnosis, 71 procedures,reliable diagnosis,general, 71 procedures,reliable diagnosis,reading, 72 procedures,treatment observation, 75 procedures,treatment observation,general, 76 procedures,treatment observation,reading, 77 prospective analysis, 112, 128 prospective research, 33, 36 purpose of study, 11 reading instruction in the classroom, A6 reading test battery generation, 85 reading,diagnosis, A2 reading,diagnosis,reliability, AA reading,outcomes, A2 reading,treatment, A2 related research, 12 relative risk,definition, 25 reliability in using experience. 20 reliability of critical performance measures, 1A2 reliability of data, 1A1 reliability,diagnosis,reading clinicians, AA reliable diagnosis procedures, 71 reliable diagnosis procedures,general, 71 reliable diagnosis procedures,reading, 72 research designs, 25, 26 results, 96 results of unreliable diagnosis, 20 retrospective analysis, 108, 123 retrospective research, 32, 3A roberts, 5, 1A, 18, 21, 22, 23, 36, 38 sample, 82 sequence of events, 85 sherman, 53 side-to-side portacaval shunt, 3A Siegelaub, friedman, collen, 8 kodlin, A0 silent reading, 90, 91 silent reading comprehension, 55. 67 silent reading measure, 2A2 S 1 m1 Si n s I 0 51112 $1112 Simon 8 boyer, 5O singer 8 ruddell, 52 slovic, 19 small 8 krause, 1A smallpox, 39 319 smoking characteristics, 31 snow, 7 sources of diagnostic categorization, 99 Spache. IA. A3. A5 stallings, 51 standard interview form, 21A statistic for agreement, 102 statistics,definition, 102 stratoudakis, AA student population, student,attendance, students,instruction, students,midtesting, students,posttesting, students,pretesting, subject,attendance, subjects, 82 subjects,instruction, subjects,midtesting, subjects,posttesting, subjects,pretesting, 82 117. 12A 86 9A 9A 86 117. 12A 86 9A 9A 86 subsequent identification, 80, 85 sullivan oral reading placement task, 237 summary, 95, 11A, 133 summary of data analysis, 1A8 summary of related research, 56 swine flu, A swine-flu, A0 t - treatment characteristics, 5 table,agreement among diagnostic sources: critical performances, 102 table,agreement among diagnostic sources: profiles, lOA table,agreement among diagnostic sources: the statistic definition, 102 table,association of table,association of table,association of table,association of table,association of liver function, table,association of contraceptives, table,association of drug treatment and mortality for hypertension, 36 ebv virus and incidence of mononucleosis, 3A gain and class attendance, 125 mortality and surgical procedure, 35 mortality and surgical procedure stratified by 35 myocardial infarction and use of oral 33 smoking and chronic bronchitis, 32 table,associations of outcome and time on class activities, 127 table,available data by student, 8A table,average agreement on total diagnosis across cases, 106 table,characteristics of instructional time by treatment group, 119 table,classic epidemiological study, 7 table,criteria for decision rule diagnosis, 7A table,decision: choosing students to teach, 13A table,differences in equivalent forms: listening, 1A1 ttt11i4 Iltttt. 320 table,differences in equivalent forms: silent reading, 139 table,differences in rate of poor attenders based on relative outcomes, 12A table,differences in rate of problem diagnosis based on rating of language ability, 113 . table,differences in rate of problem diagnosis based on ratings of mean length of utterance, llA table,differences in self perception based on diagnosis, 109 table,differences in self perception based on diagnosis by sex, 111 table,distribution of alternate forms, 136 table,distributions of gains across treatment groups, 122 table,mean days of instruction by treatment group, 117 table,performance gains by treatment group, 131 table,prevalence of critical performance problems, 100 table,prevalence rates across treatment groups, 129 table,prevalence rates before and after treatment across treatment groups, 121 table,reliability of critical performance measures, 1A2 table,results of unreliable diagnosis, 20 table,summary of data analysis, 1A8 table,test booklet characteristics, 70 table,typical clinical research study, 8 test administrator debriefing form, 279 test battery, 85 test battery generation, 85 test booklet characteristics, 70 the clinical and epidemiological paradigms and improved clinical care, 39 the clinical problem solving model, 12 the current state of diagnosis and treatment in reading, A3 the model of reading and learning to read (moral), 52 the nature of diagnosis and treatment in reading, A2 thio-tepa therapy. 8 time on tasks, 118 treatment characteristics, 116 treatment observation procedures, 75 treatment observation procedures,general, 76 treatment observation procedures,reading, 77 treatment of subjects, 85, 86 treatment,characteristics, 116 treatment,definition, l3 treatment.group instruction, 91 treatment,homework, 91 treatment,in reading, A2 treatment,level one, 15 treatment,level two, 15 treatment,cbservation, 92 treatment,oral reading, 89, 9O treatment,relationship to diagnosis, A5 treatment,silent reading, 90, 91 treatment,symptomatic, 15 treatment,time on tasks, 118 tversky 8 kahneman, 19 typhoid fever, 39 321 typical clinical research study, 8 v. procedures for outcome measurement, 78 vanroekel, AA vi. procedures for executing the study, 79 vinsonhaler, weinshank, cureton, 8 blatt, 53 vinsonhaler, weinshank, wagner, 8 polin, 1A, 20, AA, A5 vocabulary, 55, 65 weinshank, AA wilson, horrocks, lyndon, yeung, page, 8 dedombal, 30 word comprehension measure, 276 wulff, 1A yerushalmy, 20