)V153‘_) RETURNING MATERIALS: P1ace in book drop to LJBRARJES remove this checkout from .—J—_ your record. wig wiH be charged if book is returned after the date stamped below. TESTING THE RELIABILITY AND VALIDITY OF OCCUPATIONAL ASSESSMENT INSTRUMENTS DEVELOPED THROUGH RANDOM SELECTION OF PREVIOUSLY VALIDATED TEST ITEMS By Steven Craig Clark A DISSERTATION Submitted to Michigan State University in partia] fulfiilment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Teacher Education 1987 Copyright by STEVEN CRAIG CLARK 1987 ABSTRACT TESTING THE RELIABILITY AND VALIDITY OF OCCUPATIONAL ASSESSMENT INSTRUMENTS DEVELOPED THROUGH RANDOM SELECTION OF PREVIOUSLY VALIDATED TEST ITEMS BY Steven Craig Clark The purpose of this research was to calculate estimates of relia- bility and to gain measures of validity on selected examinations developed using the vocational/occupational test-item bank at Michigan State University. The writer investigated the possibility of develop- ing reliable and valid assessment instruments through random selection of test items from occupationally specific item subbanks. Using the Michigan State University vocational/occupational item bank. certified vocational/occupational education instructors from the fields of machine trades and mechanical drafting developed the tests using a test blueprint. Two hundred eighty-eight secondary vocational students were administered the test using test/retest methods. A randomly selected group of students was given the same form of the test on both occasions: the remaining students were administered a parallel form as a retest. The first sets of data were used for esti- mating test stability: the second sets were used for estimating test stability and equivalence. Steven Craig Clark Instructors were asked to rank (pass/fail) their students' knowl- edge at both the occupational level (across all subtest domains) and the domain level (individual subtest). These data were used as the criterion measure and were correlated with the actual test results to establish the concurrent validity of the test. The Pearson product-moment coefficients of stability were statis- tically significant at the .OT level for both occupational exams. The Pearson product-moment coefficients of stability and equivalence were also statistically significant for both instruments. Tetrachoric cor- relations of stability for the combined domain (duty) subscore ranking were statistically significant. as were the tetrachoric correlations for subscore stability and equivalence. Correlations between students' actual percentile rankings and instructor-provided rankings were statistically significant for machine trades. For mechanical drafting. the correlations were not significant. At the subtest (domain/duty) level. the results for both samples were statistically significant. However. based on the small sample size. the results lacked practical significance. The overall results indicated weak to moderate agreement between the two variables tested. The evidence led the researcher to conclude that the tests had little concurrent validity with regard to the instructors' rankings. ACKNOWLEDGMENTS For guidance. encouragement. and assistance. a special expression of appreciation is extended to Dr. George N. Ferns. As major professor. his advice and knowledge proved invaluable in the comple- tion of this degree. In addition. profound appreciation is extended to Dr. Irvin J. Lehmann. dissertation chairman. whose knowledge Of educational and psychological research methodologies provided the expertise necessary to complete this work. I also wish to acknowledge the contribution of my committee members: Dr. Rex E. Ray. whose insightful comments helped me maintain perspective on the practical importance of the study: Dr. Charles A. Blackman. whose availability and academic objec- tivity I will always admire: and Dr. Samuel A. Moore. whose wisdom and professionalism served as constant reminders of my goal. Sincere appreciation is eXpressed to Susan Cooley. who aided me as editor. typist. and adviser in preparing the dissertation manuscript. Dr. Chris M. Olson. whose reputation in the field of vocational/occupational education land commitment to instructors in Michigan. opened the doors for me to conduct this research. I am deeply indebted to my wife. Susan. for her patience and understanding. without which this research could not have been undertaken. I also acknowledge the support of my children. Lauren and Phillip. who have grown up in academe without complaining about the many sacrifices due to their father‘s time on task. Special thanks to my parents. here and on the Nest Coast. for their support and encour- agement throughout my educational career. vi TABLE OF CONTENTS Page LIST OF TMLES O O I O O O 0 O O O O O O O O O O O O O O O O 0 x LIST OF FIGURES ........... . ........... xii Chapter I. INTRODUCTION TO THE STUDY ........... . . . l Introduction . . . . . . . ...... . ..... . l , Background of the Problem . . . . . . . . . . . . . 2 Statement of the Problem . . ............ 4 Purpose of the Study ................ 5 Importance of the Study . . . . . . ...... . . 6 ‘ ReSearch Questions . . . .............. 8 Research Methodology . ............... 9 Assumptions . . . . . . . . . . . . . . . . . . . . lO Limitations of the Study ..... . . . . . . . . ll Definition of Terms . . . ....... . . . . . l2 Organization of the Remainder of the Dissertation. l8 II. REVIEH OF RELATED LITERATURE AND RESEARCH ...... 20 Introduction . . . ..... . . . ....... 20 Historical Overview of General and VOcational/ Occupational Education Evaluation . . . . . . . . 21 Effects of Evaluation ..... . . . . ...... 26 Tested Vocational/Occupational Education Evaluation Instruments . . . . . . . . . . . . . . 29 Estimating Reliability and Validity of Criterion-Referenced Tests . . . . . . ...... 34 Re1iab111ty O O O 0 O O O O O O O O O O O O O O O 34 Validity of Criterion-Referenced Tests . . . . . . 37 Test-Item Banking . . . . . . . . . . . . . . . . 4O smary O O O O O O O 0 O O I O O O O O O O O O O O 4] III. RESEARCH METHODOLOGY . . ........ . . . . . . . 43 Introduction . . . . . . . . . . . . . . . . . . . . 43 Design of the Study ................ 43 vii IV. v. Research Questions . . . . . . . . . ........ Population and Sample Inservice Sessions for Pilot-Test Instructors/ Administrators . . . . . . . . . . . . . . . . . Development of the Instruments . . ..... . . . The Tests . . . . . . . . . . . . . . . . . . . The Instructor Ranking Sheets ......... Data Collection . . . . . . . . . . . . . . . . . Data Processing and Analysis . ......... . RESULTS OF THE DATA ANALYSIS . . .......... Introduction . . . . . ReSUIts O O O O O O O O O O O O O O O O O O O O 0 Research Question Research Question Research Question Research Question Research Question Research Question Summary . . . . . . . $01-$de SUMMARY. CONCLUSIONS. AND RECOMMENDATIONS FOR FURTHER RESEARCH . . . . summary 0 O O O O O O O O O O O O O O O O O O O 0 Procedures . . . . . . ........ . . . . . Findings and Conclusions . . . . . . . . . . . . . . Machine Trades . . . Mechanical Drafting Synthesis ..... . Recommendations for Further Research ....... Epilogue . . . . . ................. APPENDICES A. B. C. VOCATIONAL PROGRAM EVALUATION MODEL . . . . . . . . MECHANICAL DRAFTING AND MACHINE TRADES TASK LISTINGS FROM THE v-TECS O O O O O O O O I O O C O O O O O O SAMPLE MECHANICAL DRAFTING AND MACHINE TRADES TEST BLUEPRINTS O O O O O O O I C O O ..... O O CORRESPONDENCE . . . . . . viii 95 lOZ lO7 E. SAMPLE INSTRUCTOR RANKING SHEETS FOR MECHANICAL DRAFTING AND MACHINE TRADES . . . . . . . . ..... F. DATA SUMMARY TABLES REFERENCES 0 O O O O O O O O O ..... O O O O O O O O I O 0 ix Table 3.1 3.2 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 LIST OF TABLES Duty Areas Selected for Testing and Number of Items REQUQStEdp by P110t Site 0 o o o o c o o o o o o 0 Number of Items in Each Duty Area Available for Inclusion in Tests . . . . . . . . . . . . . . . . Correlational Data for Machine Trades Test Form A2/ Retest Form A2 . . . . . . . . . . . . . . . . . . Correlational Data for Mechanical Drafting Test/ Retest. Form A2 . . . . . . . . . . . . . . . . . Correlational Data. Machine Trades Test Form A2/ Retest Form 82 . . . . . . . . . . . . . . . . . . Correlational Data for Mechanical Drafting Test Form A2/Retest Form 82 . . . . . . . . . . . . . . Subscore Ranking Agreement and Tetrachoric Correla- tions: Machine Trades. Test/Retest Form A2 . . . Subscore Ranking Agreement and Tetrachoric Correla- tions: Mechanical Drafting. Test/Retest Form A2 . . Tetrachoric Correlations for Combined Subscore Rankings on the Vocational/Occupational Assessment TeSt C O C O O O O I O O O O O O O O O O O O O O O Subscore Ranking Agreement and Tetrachoric Correla- tions: Machine Trades. Test Form A2/Retest Form 82 C O O O O O O O O O O O O O O O I O O O O Subscore Ranking Agreement and Tetrachoric Correla- tions: Mechanical Drafting. Test Form AZ/ Retest Form 82 O O O O I O O O O O I O O I O O O O Tetrachoric Correlations for Combined Subscore Rankings on the Vocational/Occupational Assessment Test 0 C C O O O O O O O O O O O O O O C I I O O O Page 50 53 64 64 65 66 68 68 69 7O 71 7l 4.ll 4.12 4.l3 4.14 4.l8 Students' Percentile Rank Achievement in Relation to Instructors' Rankings of Students: Machine Trades. All Sites Combined. in Frequency and Percentage of Total . . . . . . . . . . . . . . . Percentages and Chi-Square Results for Students' Rank Achievement in Relation to Instructors' Rankings of Students' Achievement: Machine Trades Students' Percentile Rank Achievement in Relation to Instructors' Rankings of Students: Mechanical Drafting. All Sites Combined. in Frequency and Percentage of Total . . . . . . . . . . . . . . . Percentages and Chi-Square Results for Students' Rank Achievement in Relation to Instructors' Rankings of Students' Achievement: Mechanical Drafting . . . . . . . . . . . . . . . . . . . . . Vocational/Occupational Assessment Test Subscore Averages in Relation to Instructors' Rankings: MaChine Trades. A1] Sites 0 o o o o o o o o o o o Instructors' Rankings in Relation to Students' Subscore Rankings: Percentage Agreement and Tetrachoric Correlation Data. Machine Trades . . . Vocational/Occupational Assessment Test Subscore Averages in Relation to Instructors' Rankings: Mechanical Drafting. All Sites . . . . . . . . . . Instructors' Rankings in Relation to Students' Subscore Rankings: Percentage Agreement and Tetrachoric Correlation Data. Mechanical Drafting xi Page 72 73 74 75 76 77 78 79 Figure 3.1 3.2 Research Design Sample Data Table LIST OF FIGURES xii CHAPTER I INTRODUCTION TO THE STUDY Introduction Cast in the foundation of vocational /occupational education is the conviction that evaluation is linked to program improvement. For more than 69 years. the national vocational education acts and subse- quent amendments have addressed with varying intensities the belief that some type of evaluation should be incorporated into the voca- tional education setting. The latest vocational education act. the Carl D. Perkins Vocational Education Act of 1984. P.L. 98-523. again emphasizes the need for evaluation. The thrust of the Perkins legis- lation concerns program improvement through a sound assessment pro- gram. The present legislation. if enforced. will mandate some local involvement in program evaluation. Providing reliable and valid evaluation procedures for voca- tional/occupational education programs is a formidable task. one with which the state of Michigan has started to grapple. Michigan has included two types of evaluation procedures in its attempt to evaluate the vocational experience. Process and product evaluation measures are addressed in Michigan's yearly planning process for vocational/ occupational education. Process evaluation is used to detect or predict defects in the procedural design of a program or course during the implementation or operational stage (Hentling. T980). Michigan evaluates process by using the Program Review and Planning (PRP) model (Michigan Department of Education. Vocational-Technical Education Service. 1982). Product evaluation is superficially addressed through the survey responses of students who were enrolled in a vocational/occupational program the previous year. Michigan's new Vocational Program Evalua- tion Model (MDE. V-TES. l986) (see Appendix A) has developed into an evaluation system that places new emphasis on student achievement as a part of vocational/occupational education product evaluation. This prompted the development of criterion-referenced tests for selected vocational/occupational programs. Yet data concerning the reliability and validity of these tests are nonexistent. Background of the Problem The Perkins Act has made program evaluation a requirement. To remain within the guidelines for reimbursement. over a five-year period 20% of the state fiscal agents must employ some measure of program evaluation each year. This scheme was developed to try to obtain lOOZ participation in some type of program evaluation by T990. Given these conditions. the Michigan Department of Education has adopted the following philosophy: I'Vocational education leads to competence and competence is linked to performance. Vocational educa- tion must. at some point. be concerned with the assessment of perform- ance' (Runkel. l986). Consequently. Michigan. along with many other states. needs to develop a vocational/occupational student competency assessment process and appropriate instrumentation. The evaluation requirements of the Perkins Act have caused con- siderable confusion and some controversy within vocational/occupa- tional education. The requirement of student assessment in programs funded under this act is troublesome for many districts. both adminis- tratively and programmatically. The question of how to assess can be approached from different positions. making the “letter of the law' seem negotiable. Assessment in vocational loccupational education may cause some discomfort to those versed in measurement of traditional general education subjects. The differences between vocational/occupational and general education are as notable as their similarities. The environment in which vocational/occupational students learn is unique to their specific program. That. in itself. tends to rule out a standardized examination. The notion of an occupationally specific standardized assessment tool falls short when one considers the vast array of regional and local employment needs within the same occupa- tional area. The differences in equipment and personnel within the same occupational specialty must also be considered. The need to assess vocational/occupational students' competency with valid and reliable instruments that reflect local programming provided the impe- tus for this research. Statement of the Problem The availability of occupational assessment tools is limited. Some vocational/occupational education programs have In: commercially produced evaluation instruments from which to select. whereas the focus of some nationally normed. commercialLy produced standardized tests is unsatisfactory (Seney. 1986). The dilemma of providing assessment instruments that meet the needs of local programs nationwide is substantiaL. Even more important is the question of local autonomy. Occupational program evaluation can be approached in many ways: no single evaluation procedure can be followed in all cases (Hentling. l980). As with any content area. one instrument cannot provide all programs with a satisfactory measure of student achievement because validity is specific rather than universal. In an attempt to alleviate the aforementioned problems. the Michigan Department of Education contracted with Michigan State Uni- versity to develop an item bank of vocational/occupational test ques- tions that address a variety of duty areas within selected vocational/ occupational specialties. Having such a test-item bank would enable local instructors of vocational education programs to cmstom design tests that address specific local needs. thus preserving Michigan's commitment to local control of educational programming. However. the reliability and validity of vocational/occupational tests constructed using the test-item-banking method developed by Michigan State Univer- sity have not yet been estimated. £_rpose of the Stqu Because. no current data. are available regarding estimates of reliability and validity of examinations developed using Michigan's vocational/occupational test-item bank. the researcher's purpose in this study was to calculate such estimates on selected tests constructed and administered during the pilot testing of PHchigan's occupational student assessment program. The vocational/occupational areas selected for this study were machine trades and mechanical drafting. Using data collected during the pilot study conducted in spring 1986. the researcher investigated the assumption that random selection of test items by duty and task areas would produce valid and reliable tests for the product evaluation of vocational/occupational students. By determining estimates of reliability and validity on selected Michigan pilot tests. the researcher obtained a preliminary measure of two important characteristics (reliability and validity) of an assessment instrument. The writer selected commonly accepted measures of test reliabil- ity and validity for use in this study. The two reliability estimates used were (a) stability and (b) stability and equivalence. In addi- tion. one type of criterion-related validity was studied: concurrent validity. The procedures employed in the research are discussed in detail in Chapter III. Importance of the Study The need to evaluate vocational education students is no dif- ferent from the need to evaluate students in other curriculum areas. The results of evaluation can be beneficial for the instructor of the vocational program. as well as for the student who participated in the evaluation process. Measurement and evaluation are essential to sound educational decision making (Mehrens & Lehmann. T978). The Carl Perkins Vocational Education Act of l984. P.L. 98-524. emphasizes the need for vocational program improvement and requires evaluation of both the program's process and product. The vocational education philosophy adopted in Michigan is that competence in vocational/occu- pational education must be linked to performance and. at some point. include an assessment of performance. For this reason. the Michigan Student Occupational Assessment model was developed for the product evaluation of Michigan's vocational students (MDE. V-TES. l986). The need for an assessment test became evident when the Michigan Department of Education. realizing that major concerns existed among Michigan vocational administrators concerning the assessment issue. reevaluated the requirements of the Perkins Act (five-year evaluation plan: 20% participation the first year. 40% the second. and so on. to be started in spring 1986). Personnel in the Department of Educa- tion's Vocational-Technical Education Service realized that a vast amount of research and development was needed for assessing the state's vocational/occupational students to comply with the Perkins Act. Therefore. the Act's requirements for participation in student assessment were relaxed to allow for an extra year of developmental work. This provision. however. forces 25% accumulative participation each year over a four-year period. The vocational program evaluation model was designed so that local vocational program instructors can identify specific vocational education program duties and tasks. using a master list developed by the Vocational-Technical Education Consortium of States (V-TECS: see Appendix B) for their vocational/occupational specialty. The selected duties and tasks are then entered into a computer program containing an unlimited number of test items. which are classified by duty and task. The MSU vocational/occupational test-item bank was developed using previously validated test items from occupational assessment tests used in V-TECS member states. The local vocational instructor requests a particular number of multiple-choice test items within the selected duties and tasks. and the computer randomly selects the items. This research has possible national significance because of V- TECS involvement. The 22 member statesin V-TECS are interested in the results of the Michigan pilot study. Research and development activities in Michigan intensified when the Michigan Department of Education V-TES awarded Michigan State University additional funds to further develop the assessment model. based on the pilot-testing effort. The format used in I'cross-walking" or coding duties and tasks for this research can be used in a state's occupational education program because the coding employs V-TECS lists. Thus. the item bank developed at Michigan State University theoretically could be used by any V-TECS member state. In addition. the findings of this study should help local educa- tion agencies and intermediate school districts understand the role this method of evaluation has in predicting the level of vocational/ occupational knowledge of students receiving training. Such knowledge should help with curriculum decisions. The study could be of value to the Michigan V-TES program. This research is the first investigation of Michigan's test-item-banking system for vocational/occupational studentassessment. Results of the study may help in determining whether future use of objective exams in vocational/occupational edu- cation programs is justifiable. in view of their reliability and validity. Limited information is available regarding test-item banks. How- ever. it seems appropriate to mention that Orenstein (l98l) found three major weaknesses in using test-item banks: (a) validity infor- mation was not available. (b) reliability estimates were unavailable. and (c) instructional materials that complement tests were not avail- able. Research uestions Each of the following research questions was posed to investigate the relative strength of the relationship that existed between sets of variables. Researchyggestion l: How stable are test scores obtained on the vocational/occupational assessment test? This question concerns the stability of test scores. Research_ggestion g: How stable and equivalent are scores obtained on parallel forms of the vocational/occupational assess- ment test? This question concerns the stability and equivalence of test scores. Research Question 3: How stable are subscore rankings obtained on the vocational/occupational assessment test? This question concerns the stability of subscores within the test. Such consistency is of paramount importance because the assessment instruments were constructed according to duties (domains). Research_guestion 4: How stable and equivalent are subscore rankings obtained on parallel forms of the vocational/occupa- tional assessment test? This question concerns the stability and equivalence of subscores within the test. Research uestion 5: How valid are the percentile rankings obtained on the vocational/occupational assessment test? This question concerns the concurrent validity of the test. Research_gyestion 6: How valid are the subscore rankings obtained on the vocational/occupational assessment test? This question concerns the concurrent validity of the subtest scores. Research Methodology To fulfill the purposes of the study. the first step was to conduct a thorough review of the literature concerning evaluation and testing of vocational/occupational education students. The review included not only the theories and models of evaluation for voca- tional/occupational education students. but a historical overview of IO educators' Views of testing student achievement for program improve- ment (process evaluation) and evaluation of program outcomes (product evaluation). Also. commercial tests developed for the evaluation of vocational/occupational education programs were examined. Finally. criterion-referenced reliability-estimating procedures were reviewed and analyzed. along with methods of measuring content and concurrent validity as related to the research questions under investigation. Based on the literature regarding the various ways to gain reliability estimates. the researcher chose to use test-retest methods. Measures of concurrent validity were obtained using the vocational/occupational instructors' ratings as criterion measures. Content validity was used to measure the validity of the tests. Assumptions The researcher assumed that the vocational instructor content specialists who participated in this research by validating and coding V-TECS test items to V-TECS duty and task lists were representative of vocational instructors in the field. He also assumed that the vocational instructors involved in the study would use local curricu- lum guides as well as outside resources (advisory committee members. local industry/business standards. and minimum performance objectives) in determining duty and task selections for construction of the test instrument. The investigator assumed that the vocational/occupational instructors involved in the study. having worked closely with the 11 examinees for not less than one school year. would possess the approp- riate knowledge and insight to permit them to give valid ratings. The investigator assumed that the students. classrooms. and instructors involved in this study were typical of their counterparts in the general population of vocational programs. In addition. he assumed that the students who participated in the study would answer the test items in both administrations to the best of their ability. The investigator assumed the tests were content valid. based on the manner in which they were constructed. Limitations of the Study The study was limited in the following ways. l. The writer did not attempt to validate the use of a test-item bank for any area outside of the vocational/occupational education curriculum at the secondary level. 2. The study was limited to selected areas of vocational/occupa- tional education that received state (M‘ federal government funding during the 1985-86 school year. 3. The study was limited to the specific vocational/occupational items entered into Michigan State University's mainframe computer by April 15. 1986. 4. The researcher did not attempt to evaluate programs (local. state. or national). program instructors. or students. 5. The investigation was limited to vocational education pro- grams of machine trades and mechanical drafting. whose instructors 12 volunteered to become involved in the study. Hence. instructors were self-selecting. Definition of Tepp; Inasmuch as this study relates to a specific area of education-- vocational/occupational education-~it is appropriate that definitions be provided in case certain terms are unfamiliar to the reader. Defi- nitions provided by the Michigan Department of Education and the Vocational-Technical Education Consortium of States are paraphrased to provide continuity of meaning. All terms are defined with regard to this research. Affective test item. An item designed to measure attitudes or values. Cluster program. A group of closely related occupational courses with a common core of skills. BEES- The process of assigning alphanumeric descriptors to items so that they can be identified. ngpitive test item. An item pertaining to knowledge (as defined in Bloom's taxonomy). which is usually evaluated by means of a written test. I Competency-based education (CBE). A systematic approach to instruction. with precisely stated student outcomes. carefully designed student-centered learning activities. established guidelines fOr time-on-task for mastery. and set criterion measures for moving from one task to another. 13 Concurrent validity. The relationship between the test scores or other* measures (predictors) and some independent external measure (criterion) obtained at the same time. Content validity. The degree to which the sample of test items represents the domain about which inferences are to be made. Cross-ydlking. The process of coding a test item to a specific task. Criterion-referenced_tg§§§. Tests whose results are interpreted in terms of whether students reach or do not reach an established criterion. Distpgctors. The incorrect responses to a multiple-choice test item. 'Qppdjn-referencedytests. Tests that draw a random or stratified ' sample of items from a precisely defined content area or domain. Qyty. One of the distinct. major activities involved in per- forming a job: each duty comprises several related tasks. Employability skills. The generic skills related to seeking. obtaining. keeping. and advancing in an occupation. Enabling objectiv . An objective related to background knowledge or skills that are prerequisite to mastering a certain task. Error variance. The total effect of the chance differences between persons that arise from factors associated with a particular measurement (e.g.. the participant's mood the day of the test). Task analysis. The process of identifying (a) the performance elements of a task. (b) the knowledge and skills required to perform 14 the task. (c) the tools and equipment necessary for job performance. and (d) the safety factors associated with efficient task performance. ltgdyreligpility. The correlation between the participants' responses to a particular item and their total test scores. Item validity. The correlation between the participants' responses to a particular item and their total scores on the criterion measure. Job analysis. The systematic collection and examination of job- relevant information. such as duties. tasks. tools. equipment. references. and demographic information. Job incumbent. An individual who is currently employed and per- forming competently in a work role. Local validation of duties. The process whereby local vocational instructors select duty areas from the V-TECS list that are included within their specific curriculum. Local validation of tasks. The process whereby local vocational instructors modify the larger. first form of their test by eliminating test items that they believe do not relate directly to their program's content/curriculum. Normjreferenced tests. Tests that relate an individual's scores to those of a well-defined group (the norm). The goal is to differen- tiate clearly among students at different levels of achievement. Such tests do not pinpoint participants' individual strengths and weak- nesses. just their total score relative to that of the norm group. 15 Objective referenced tests. Tests that measure learner perform- ance in a way described in a performance objective and evaluate the learner against the criteria stated in the objective. Occppdtional competency. The degree to which a student displays skills/knowledge in an occupational specialty. based on established criteria. Performgpce objectiv . A written statement that describes a task the student must perform. Pilot testing. Trying out an instrument or process on a group that is smaller than but similar to the actual test sample to deter- mine the usefulness of that instrument or process. Momotor test item. An item that measures specific perform- ance requiring a muscular or motor skill. Standardized test. A test that is administered using the same conditions (directions. materials. scoring. timing. identical ques- tions) for every person who will take the test. Such a test typically has norms. .S_t§_lfl. The statement or question portion of a multiple-choice test item. _S_t_udent achievement recording proces . The process by which a student's achievement is documented so that tasks in which the student demonstrated competence can be recorded. went certifigtion. The awarding of diplomas or documents certifying competency: or the advanced placement in a course sequence on the basis of test performance. 16 Subject-matter expgrt. A highly experienced job incumbent who assists in identifying task-performance steps and the skills and knowledge necessary to perform the job competently. Summative program evaluation. A method used to determine a pro- gram's overall merit relative to its competition. Iaak. A discrete unit of work that is performed by an indi- vidual. which has a definite and observable beginning and end within a limited time. Test administrator. A person formally charged with giving a test to a group of participants. Test ForJLnaAl. The preliminary form of the vocational student assessment test. It usually contains 40% to 50% more items than requested. to ensure sufficient items for the local validation pro- cess. Test Form A2. The modified version of Test Form Al. After local validation of test items. this is the final form of the test. Test FordyBl. The second form of the vocational student assess- ment test. hypothesized to be an equivalent form of Test Form Al. It normally contains 40% to 50% more items that requested. to ensure sufficient items for the local validation process. Test Form 82. The modified version of Test Form 81. After local validation of test items. this is the final form of the test. Test-item bank. A collection of criterion- or domain-referenced test items coded for input and retrieval by computer or manually. 17 Test reliability. The level of internal consistency or stability of the measuring device over time. Test stability. The ability of a test to yield consistent results over a period of time. given the same test and the same participants. Test validity. The degree to which a test measures what it purports to measure. TetraLchoric correlm. A correlational statistic used when both variables being correlated are in the form of artificial dichotomies. Vocagongaj education. Programs designed to prepare individuals for employment. or additional preparation for a related career requir- ing other than a baccalaureate or an advanced degree. Vocational process evaluation. A type of evaluation whose pri- mary goal is to detect or predict defects in the procedural design of a program or course during the implementation or operational stage (Hentling. 1980). Vocationalapppdpct evaluation. A type cfl’ evaluation that attempts to examine the outcomes of the vocational education effort. Vocational program. A program of study specifically designed to prepare individuals for employment in a specific occupation or cluster of closely related occupations or to prepare them for a career requiring further education. !;I§§§. An acronym for the Vocational-Technical Education Con- sortium of States. 18 V-TECS catalog. Contains a description of the duties. tasks. performance objectives. performance guides. and related data for a specified occupation. Organization of the Remainder of the Dissertatipp Chapter 11 contains a review of the literature related to this research. A wealth of material exists on the effects of evaluation on both general and vocational/occupational education programs. Philo- sophical concerns are reviewed because the results of this study should have major significance for curriculum. guidance. and educa- tional decision making. The second section of Chapter II includes a review of research and literature related to determining the reliabil- ity and validity of criterion-ldomain-referenced tests. In Chapter III. the writer chronicles the procedures followed in the study. from the development of the test-item bank through the inservice activities conducted with pilot-site instructors and admin- istrators. The population is described. and the rationale for its selection is explained. The data-collection procedures are reported. including testing procedures and random selection of the retest groups. Data processing is then described in terms of item analysis. the subdivision of tests into domain-specific units for analysis. and comparative statistics that were used to estimate reliability and validity. Chapter IV contains the estimates of reliability and validity. The findings of the pilot test are displayed in tabular format. 19 Chapter V contains a review of findings. conclusions drawn from the findings. recommendations. and suggestions for additional research. CHAPTER II REVIEW OF RELATED LITERATURE AND RESEARCH Introdgction Numerous reports have been published on field-tested evaluation measures for both criterion-reference and norm-referenced vocational/ occupational education instruments. Most of these reports concerned standardized measures. unlike the Michigan model. which uses a process of test construction based on local programming. The researcher learned that no research had been conducted on the estimations of reliability or measurements of validity on any assessment instruments developed and administered through the Michigan vocational program evaluation model. To provide a background. perspective. and rationale for this study. the review of related literature and research is focused on four topics: 1. A historical overview of both general and vocational/occupa- tional education evaluation. with emphasis on the latter. addressing program evaluation as well as student achievement. 2. A discussion of the effects Of evaluation. 3. An examination of the available vocational/occupational evalu- ation instruments that have undergone some degree of controlled field testing. 20 21 4. A review of pertinent literature related to securing relia- bility and validity estimates for criterion-referenced tests. Also discussed is the limited research that has been conducted in the area of test-item banking and its relationship to the present investigation. Historical Overview of General and Vogationalfi Qppdpational Edppatjon Evaluation Competitive examinations date back to 220 B.C. The Chinese used elaborate methods of evaluation in their civil service testing system (Hentling. 1980). Evaluation in occupational education may well have started in the early days of Greek sword and shield production. around 447 B.C. From Hill Durant's (1939) comment that 'the Spartan mother's farewell to her soldier son [was] 'Return with your shield or on it"l (p. 81). one gets the feeling that the first form of product evaluation was survival. In the early 19005. Thorndike established measurement techniques to assess changes in human behavior. This was the first major report of content goals and objectives being used in evaluating human subjects (Thorndike & Hagen. 1969). Later. in the 19305. Tyler directed the Eight-Year Study. which emphasized the evaluation of student outcomes. specifically using col- lege entrance exams for one group of students and matching them with students having similar backgrounds who were exempted from taking the exams to enter college (Chamberlin. Chamberlin. Drought. & Scott. 1942). In the Coleman Study of the late 19505. the vocational opportu- nities for minorities in the United States were evaluated (Hentling. 22 1980). Such studies examined students' behavior. as well as outcomes of the educational experience. In the 19605. a study paralleling Coleman's work was completed by John Flanagan of the American Institute for Research. He and his associates evaluated 440.000 subjects in school settings and then attempted to associate this evaluation of abilities with job success or failure. This work was titled Project TALENT (Flanagan et al.. 1964). Later. in 1969. Taylor initiated the National Assessment Project to assess school-to-school and state-to-state data. using student perform- ance data as the vehicle for assessment. The aforementioned projects were federally funded or supported by grants from private foundations. The evaluation of the effect of general education on American youths made good press and gained national attention. However. evaluation of vocational education pro- grams in the United States was largely ignored. partially because of the separation of these programs from general education. The evalua- tion effort in vocational education came about primarily through stimu- lation provided by federal legislation. With the passage of the Vocational Education Act of 1963 (P.L. 88-210). each state became responsible for evaluating its programs through the use of state advisory committees (Hentling. 1980). In theory. this was to have been the first real attempt to evaluate the vocational effort. In practice. however. this maiden attempt fell short of its goal. Two years after the Vocational Education Act of 1963 was passed. another piece of federal legislation was adopted. which forced major 23 strides in educational evaluation. Passage of the Elementary and Sec- ondary Education Act of 1965 (P.L. 89-10) I'may represent the most important change in national domestic policy since the New Deal period“ (Mernato. 1965). 'The ESEA legislation required that each project con- ducted under Titles I and III possess a specific evaluation component to include a plan for the evaluation of process and product' (Hentling. 1980). The Vocational Education Amendments of 1968 (P.L. 95-76) to the 1963 Vocational Education Act of 1963 re-emphasized the great need for program evaluation by the state advisory councils. The act referred to evaluation as follows: 1. The National Council shall review the administration and opera- tion of vocational education programs. including the effective- ness of such programs in meeting the purposes for which they were established. 2. The National Council shall conduct independent evaluations of programs carried out under the 1968 amendments. 3. The National Council shall review the possible duplication of programs at the post-secondary and adult levels within geo- graphic areas. 4. The State Advisory Council shall evaluate vocational education programs. services. and activities assisted under this act. In addition. the Vocational Education Amendments set standards for third-party evaluation teams. These teams reported to the project staff and the funding agency. part of the checks-and-balances system established by the new amendments based on program evaluation being directly tied to the funding of programs (Evans & Herr. 1978). The 1976 Education Amendments to the 1963 Vocational Education Act again emphasized and expanded the mandates for vocational education to 24 involve evaluation as part of its ongoing plan (P.L. 94-482. 1976). The Amendments specifically referred to 28 different forms of state and local evaluation under Title II of the Amendments. with dates for the completion of specific evaluation activities (Hentling. 1980). Specifically stated. the rules and regulations read: The state board shall. during the five-year period of the state plan. evaluate in quantitative terms the effectiveness of each formally organized program or project supported by federal. state. and local funds. These evaluations shall be in terms of: Results of student achievement as measured. for example. by: . Standard occupational proficiency measures . Criterion-referenced tests . Other examination of students' skills. knowledge. attitudes. and readiness for entering employment successfully. (P.L. 94- 482. sec. 104.402) In the late 19705 and early 19805. evaluation took on new meaning. The late 19705 marked the beginning of the 'effective-schools' movement (Brookover. 1979) and the 'effective-schools' research (Edmonds 8 Lezotte. 1979. 1980). both of which identified l'correlates'l (Edmonds. 1979) that were felt to have a profound influence on the effectiveness of schools. One of these correlates was 'careful monitoring of student achievement as a basis for program evaluation“ (Edmonds. 1979). Again. there is strong evidence for the need to assess students. regardless of the curriculum in which they are involved. I'for sound educational decision making' (Mehrens & Lehmann. 1978). In April 1983. the National Commission on Excellence in Education published a report entitled A Nation at Risk: The Imperative for Educa- tional Reform (National Commission of the States. 1983). In this 25 report. the term ''test" is used 11 times in reference to factors that have contributed to the 'risk' (Independent School. 1983). 'The excellence movement has given scant attention to the purpose of secondary and postsecondary vocational education" (Bottoms. 1984). This statement by the director of the American Vocational Association (AVA) expresses the feelings of most vocational educators after the reports on excellence in education and their subsequent recommendations were released. The Michigan Council of Vocational Administrators (MCVA) concurred with the AVA director's report and. in addition. stated that 'a concentration on the need to analyze. evaluate. and improve our current performance in delivering the basics in education to our youth is long overdue' (Pratt. 1984). Evaluation remains one of the most important factors in the educational process. With the volumes of educational reports and recommendations that emerged in the early 19805 reporting the benefits of student program evaluation. the Carl Perkins Vocational Education Act of 1984 (P.L. 98- 524. 1984) paralleled the national attitude--that student/program evaluation leads to improved programs. The Perkins Act requires both process and product evaluation of programs at the local level: 20% percent of the fiscal agents must evaluate students by the end of the 1985-86 academic year. The vocational education evaluation policy adopted in Michigan states that 'vocational education leads to compe- tence and competence is linked to performance. Vocational Education must. at some point. be concerned with the assessment of performance” (Runkel. 1986). These words from Michigan's top public school 26 administrator parallel those of the Perkins Act and respond to calls for accountability at all levels of education (Smith. 1985). Effects of Evaluation The National Institute of Education (1985) released survey results on the educational attitudes of the American people. The data showed that the American public strongly favored testing for promotion from junior high school. for graduation from high school. and for determin- ing teacher competency. However. teachers are not to neglect their students' social and personal development while they emphasize academ- ics (National Institute of Education. 1985). The effects of evaluation on both the school as an institution and the students within schools encompass a wide range of philosophical and sociological ideologies. Both evaluation for planning and evaluation for learning are addressed in the present study. with the analysis of programs by instructors. and development of student assessment tools based on the instructors' local curriculum content. As Dressel (1954) stated. Evaluation incorrectly done is at odds with the promotion of learning. Evaluation correctly done should enhance learning because it aids both the teacher in teaching and the student in learning. The student and the institution can be directly influenced by the evaluation process. The results of school achievement tests can alter people's attitudes toward education. Lezotte (1986) pointed out that an evaluation model that uses achievement testing to monitor student progress and gains in student achievement as measured on standardized tests has merit and is "getting a good reputation.“ 27_ The subject of testing often conjures up notions of exclusion. However. 'there is no good evidence that accurate feedback damages students' self-concepts. but there is much evidence that such feedback improves subsequent performance' (Mehrens & Lehmann. 1978). Students can even learn during the testing process. 'It is probably not extrav- agant to say that the contribution made to a student's store of knowl- edge by the taking of an examination is as great. minute for minute. as any other enterprise he engages in' (Stroud. 1946). The institution can benefit substantially from the evaluation process if the data are used as they were intended to be used--for sound educational decision making. Speaking to the General Subcommit- tee on Education of the House of Representatives' Committee on Educa- tion and Labor. Parnell (1973) stated: Measurement is the hand-maiden Of instruction. Without measure- ment. there cannot be evaluation. Without evaluation. there cannot be feedback. .Nithout feedback. there cannot be good knowledge of results. Without knowledge of results. there cannot be systematic improvement in learning. The improvement of learning and instruction through evaluation of students and analysis of test results leads one to the assumption that assessment testing can solve all of education's woes. That assumption is largely unproven: however. many writers have supported the use of assessment testing in United States schools (Olson. 1986). Gordon Cawelti (1986). executive director of the Association for Supervision and Curriculum Development. spoke of the effective-schools movement and evaluation: The effective-schools movement has provided a very useful focus for large numbers of schools to concentrate energies in the direction 28 of improving student achievement. It certainly has weaknesses. but it also has compelled school districts to clarify their academic goals and work on raising expectations for kids and monitoring progress. General education curriculum in the schools should focus as much on society as vocational education curriculun does on the world of work. I'Only rarely has curricUlum content not reflected what is hap- pening outside the school' (Apple. 1979). With “achievement“ and 'accountability' as the buzz-words in society today. the curriculum changes being implemented reflect those 'feelings (Naisbitt. 1984). Many educators believe that assessment for educational decision making at the building level is less than perfect. 'Most curriculum decisions are made by the building principals with quite a bit of input from teachers: parents and students. however. are generally left out' (Kimpston & Anderson. 1982). This leads one to believe that test scores are not considered in the change process and that little paren- tal input is taken into account. In ‘the ‘1ate '19305. many educational leaders adopted Frederick Taylor's 'scientific nmnagement' theories. Taylor (1938) asserted. 'The curriculum must be efficiently managed—and outcomes precisely predicted. Ten years later. in B_asic Principles of Cprriculum and Instrgction. Tyler (1949) asked four questions about the curriculum- development process: 1. Hhat educational purpose should schools seek to achieve? 2. How can learning experiences be selected that are useful in attaining objectives? 29 3. How will learning experiences be organized for effective instruction? 4. How can the effectiveness be evaluated? Tyler's model of curriculum development had evaluation as a major component. From the late 19405 until Sputnik. educational curriculum reflected the economic stability of the times. Stufflebeam (1971) developed one of the more generalizable curriculum and evaluation models. which has become popular today. His model includes four types of evaluation: context. input. process. and product. Ten years later. Stufflebeam (1981) chaired the Joint Committee on Standards for Educa- tional Evaluation. whose model for educational-program evaluation com- prised 32 standards categorized under four major headings: LNfility. fea5ibility. propriety. and accuracy. The last category was research based and dealt primarily with evaluation. At present. evaluation and assessment are keys to federal- and state-funded programs. Funding is tied directly to student and program (curriculum) evaluation (P.L. 98-524). Tested Vocational/Occupational Education Evaluation Instruments This section of the literature review includes an overview of field-tested measures of skills and knowledge that are occupationally specific (e.g.. machine trades). The tests included in this section encompass the field-tested measures compiled through computer searches of data bases such as the Educational Resources Information Center (ERIC). the Vocational and Occupational Information Center for 30 Educators (VOICE). Vocational Education CUrriculun Materials (VECM). and Dissertation Apatracts. The writer searched these sources to obtain information on instruments that had been developed or research that had been conducted within the past ten years. Other sources of information on occupational/vocational education testing were acquired through the following test-development institutions: the National Occupational Competency Testing Institute (NOCTI). the American Insti- tute for Research (AIR). and the Instructional Materials Laboratory of The Ohio State University. The vast majority of the verified. field- tested instruments were developed at these three institutions (Lewis & Martin. 1986). Through the National Center for Research in Vocational Education. vocational/occupational researchers Lewis and Martin (1986) under con- tract grant from the United States Department of Education. compiled a compendium to help vocational educators select appropriate test instru- ments to measure the learning outcomes of their programs. The compen- dium comprised 1.200 entries found in the Vocational Education Curriculum Materials (VECM) data base. Only 150 validated instruments were included in the compendium. This section of the review is focused on those instruments. Half of the 150 verified. field-tested evaluation instruments were developed at the National Occupational Competency Testing Institute (NOCTI). Of that number. 50 are teacher occupational competency tests. designed for experienced workers who wish to apply for teaching posi- tions in vocational education programs. The remaining tests developed 31 through the NOCTI--the Student Occupational Competency Achievement Tests--are specifically designed to measure students' occupational knowledge and skills. The NOCTI exams are standardized. norm-referenced instruments developed by the Institute. Users of these tests must ensure the security of the instruments and return all testing materials. Users purchase a testing service. not actual test materials (Lewis & Martin. 1986). The AIR tests include both written and performance sections. as do the NOCTI exams. The AIR tests also include a Hork Habits Inventory. a three-questionnaire packet containing the same 49 rating scales: the student completes two of the questionnaires. and the instructor com- pletes the remaining one. The students' first questionnaire concerns 'how you think the average employer wants employees to behave.“ and the second refers to “the extent to which each of the descriptions applies to you.‘ The teachers' questionnaire is designed to ascertain the degree to which the 45 descriptions apply to the student. General employers' ratings are included. to compare ratings across all ques- tionnaires. as recommended by the test developers. This questionnaire approach is relevant to the present research in regard to the criterion-related validity estimates the researcher made for the Michigan tests developed and used in this study. Subjective judgments. gathered through questionnaires completed by both students and instructors. are valuable tools in assessing specific measures of validity (Bloom. Hastings. & Madaus. 1971). Bloom et a1. commented on 32 these subjective judgments: 'It is true that careful subjective judge- ments by those [instructors] who have experience with performers may substitute as estimates until empirical evidence is collected' (p. 69). The Ohio tests are written tests designed to measure students' occupational knowledge and skills. Administered as part of the Ohio tests is the California Short Form Test of Academic Aptitude. which takes about one hour to complete. The developer recommends that the latter test be administered so that the students' achievement percen- tiles can be compared to their aptitude percentiles. This comparison is possible because both the Ohio tests and the California Short Form Test of Academic Aptitude were normed on the same population. Like the tests developed by the NOCTI. the Ohio tests are stand- ardized and norm referenced. Users of the Ohio tests must ensure the security of the tests and return all test materials. A testing service is purchased. rather than the actual test materials. The Ohio tests are meaningful to this research because they involve comparison of student percentile rank. as determined by the instruments. with a criterion measure of percentile rank. assessed by another instrument. both normed on the same population. The measured percentile rank and predicted percentile rank of the test population used in the present research were compared using much the same method- ology. The aforementioned occupational/vocational education test- developing institutions gain estimates of test reliability and validity using traditional methodologies. However. the test developers 33 considered in this section did not create their tests in the same manner in which the Michigan instruments used for this study were developed. Although many of the national tests (NOCTI. AIR. Ohio tests. and so on) can be considered criterion referenced in specific settings. they are by design occupationally specific. standardized. norm-referenced exams. The researcher would be remiss in eliminating them from this review: however. the direct relationship to the current research was tangential in nature. One occupationally specific test developer uses a data bank of test items. The Interstate Distributive Education Curriculun Consor- tium (IDECC) develops tests by individual competency or by occupation and level. The IDECC item bank has a 'descriptive test key.‘ which provides a rationale and reference for each multiple-choice test item. The existence of the IDECC item bank is the major reason the Michigan item bank does not include any nmmketing-education test items. How- ever. as testing of reliability and validity were of interest. this researcher contacted the director of IDECC testing and became keenly aware that test reliability and validity are not key concerns in the IDECC testing process (Gleason. 1986). Pollock and McDone (1974) developed a national item bank for tests of driving knowledge. Materials intended for operational licensing and educational agencies to use in developing driving-knowledge tests were prepared. Multiple-choice-item pools were generated for four classes of certification: approximately 2.200 items were developed. Field tests were conducted to collect psychometric. normative. and validation data 34 for one of the certification classes (Class C. Passenger car and light truck): however. no reliability estimates were given. O'Reilly (1976) summarized the development and validation of two different types of nufltiple-choice instruments. Thirty-six forms of the tests. which covered literal comprehension. were given to 5.722 students in grades 1 through 9. Upon examining the data. O'Reilly discovered that both forms of the test were 'generally' consistent. He also estimated test reliability using the Kuder-Richardson 20 statisti- cal formula. The reliability estimate for these exams was considered I'very good.“ Estimating Reliability and Validity of Criterion-Referenced Tests Reliability 1 Of primary interest in this study were measures of relationships. If research produces two sets of scores from the same group of sub- jects. it is often desirable to know the degree to which those scores are related (Mehrens & Lehmann. 1978). Fox (1969) defined test relia- bility as follows: Reliability . . . is the basic attribute which every procedure must possess. . . . By reliability we mean the accuracy of the data in the sense of their stability. repeatability. or precision. A perfectly reliable data-collection instrument is one which. if administered twice under the same circumstances. would provide identical data. (pp. 352-53) The researcher was also interested in criterion-referenced inter- pretation. Determining reliability is as important in studies of criterion-referenced tests as it is in research dealing with norm- 35 referenced tests (Martuza. 1977). What is needed. however. because of the domains within tests. are techniques that yield meaningful. interpretable results. regardless of the amount of variability present in the test-score distribution (Martuza. 1977). Several authors (Faggon. 1977: Mehrens & Lehmann. 1978: Popham 8 Husek. 1969: Sax. 1974) have agreed that determining the reliability of criterion-referenced measures can be problematic. Ebel (1979). how- ever. did not concur. He stated. Reliability is as important for criterion-referenced tests as it is for conventional norm-referenced tests. and it can be determined in much the same way. The notion that "classical'l test theory and "traditional“ methods of test analysis are inappropriate for criterion-referenced tests is based on a misconception. namely. that scores on a criterion-referenced test show no variability because all who take the test make perfect scores. answering all the questions correctly. While this is a theoretical possibility. it almost never happens. One could 'rig' a test to make it happen. but such a test would serve no useful educational purpose. The only reason for giving a criterion-referenced test is to identify students who have not achieved a certain educational objective. . . . The same procedures that are used to estimate the reliabili- ties of norm-referenced tests--that is test-retest. equivalent forms. split-halves. or Kuder-Richardson--can also be used to esti- mate the reliabilities of criterion-referenced tests. (pp. 281-82) Although it is not the researcher's intention to solve the controversy surrounding this issue. it is important that there be as much confi- dence as possible in the data and findings of any research. Fox (1969) explained that: To estimate reliability through the alternate-form procedure. the researcher must develop two parallel or equivalent forms of his instrument. administer both to the same people. and correlate the two sets of data obtained. The administration of the two forms can be with an intervening time interval. (p. 355) Ebel (1979) commented in this regard: 36 If the same test is given twice to the same group. if two different but equivalent forms are given to the same group. or if a single test is split into two equivalent halves that are separately scored. we use an index of the degree of agreement between the pairs of scores for each person as the basis for estimating the test reliability. (pp. 218-19) Popham and Husek (1969) stated that although the issue of varia- bility is at the core of the difference between norm-referenced and criterion-referenced tests. it is true that one almost always gets variant scores on any psychological test: but variability is not a necessary condition for a good criterion-referenced test. If the objectives within the test are substantially diverse. the items meas- uring them should be considered as different tests. not as a single. all-encompassing measure. Humbleton and Novick (1973) further suggested: 'An alternative measure of reliability might simply be the proportion of times that the same decision (pass/fail. master/nonmaster) would be made with the two parallel instruments" (p. 168). Carver (1970) agreed. but from a different perspective: The reliability of any test depends upon replicability. but repli- cability is not dependent upon test score variance. If examinees obtain similar (but not identical) scores on parallel forms of the same criterion-referenced test . . . near perfect replicability exists even though test reliability estimates using classical cor- relational methods (test-retest. parallel-forms. etc.) might be close to zero (no correlation). (p. 71) Carver went on to explain how he believed criterion-referenced test reliability can be assessed: 1. The reliability of a single form of a criterion-referenced device could be estimated by administering it to two comparable groups. The percentage that met the criterion in one group 37 could be compared to the percentage that met the criterion in the other group. The more comparable the statistics. the more reliable the test. 2. The reliability of a criterion-referenced test could be assessed by comparing the percentage of examinees achieving the criterion on parallel tests. (p. 56) Based on the preceding discussion. this researcher obtained esti- mates of reliability by employing three strategies suggested in the literature: 1. Test-retest reliability methodologies using the same sample. 2. Test-retest reliability methodologies using parallel forms of a test with the same sample. 3. Produced reliability estimates based on each subsection of the test. for both identical and parallel forms of the same test with the same sample. Validity of Criterion-Referenced Tests Unlike reliability. there is less controversy within the community of test theorists concerning acceptable methods for seeking estimates of test validity. Humbleton (1984) stated: “Criterion-related valid- ity studies of criterion-referenced test scores are not different in procedure from studies conducted with norm-referenced tests' (p. 21). Ebel (1979) described test validity as follows: 1. Validity is the most important quality a test can have. 2. Validity refers to the effectiveness of a test in measuring what it is intended to measure. 3. Validity has many different facets. takes many different forms. and can be defined in many different ways. 38 4. Validity can be demonstrated best by tryout after the test has been constructed. 5. To demonstrate validity. one must have criterion measures. real. synthetic. or hypothetical. (pp. 289-99) Sax (1974) indicated that tests have content validity if “the behavior and subject matter called for in the items corresponds to the behavior and subject matter identified in the specific objective.“ Scriven (1977) described how content validity of an instrument might be obtained: [Get] some external judgments as to the cohesiveness of the alleged goals. the actual content. and the test question pool. Hithout this. the validity of the tests and/or the utility of the curricu- lum will suffer. possibly fatally. There is no need at all for the individual judge at this task to be a professional evaluator: indeed. professional evaluators are frequently extremely bad at this. A good logician. a historian of science. a professional in the subject matter field. an educational psychologist. or a curric- ulum expert are possible resource categories. (p. 349) Predictive validity is one of two types of criterion-related validity. Sax (1974) wrote that: The term criterion-related validity . . . means the correlation of measurements within an external criterion. If the measurements are used to predict future behavior. these correlations are called predictive validity coefficients: if validity is estimated by cor- relations of measurements with currently obtainable criteria. these correlations are called concurrent validity coefficients. In determining the predictive validity of an instrument. the researcher must have an external criterion or other indicator for the variable under study. Fox (1969) wrote: Concurrent. congruent. and predictive validity express validity in a correlation and so provide an estimate of extent. . . . All of these procedures to produce the statistical estimate of validity require the researcher to have available some other measure for the variable under study. (p. 371) 39 Bloom et a1. (1971) described other measures that are considered viable in estimating the validity of a test as subjective judgments. Addressing knowledge acquisition. the authors noted that subjective judgments of those who have experience with the performers may substi- tute as estimates (validity) until empirical evidence is collected. Popham and Husek (1969) discussed the validity of a criterion- referenced instrument as follows: Criterion-referenced measures are validated primarily in terms of the adequacy with which they represent the criterion. Therefore. content validity approaches are more suited for such tests. A carefully made judgement. based on the tests' apparent relevance to the behaviors legitimately inferable from those delimited by the criterion. is the general procedure for validating criterion- referenced measures. (p. 164) Estimating reliability and validity of any test. either criterion referenced or norm referenced. depends on a number of factors. the most important being the student's score. Mehrens and Lehmann (1978) pointed out many of the problems inherent in testing students: A student's test score may vary for many reasons. The amount of the characteristic we are measuring may change across time (trait instability): the particular questions we ask in order to infer a person's knowledge could affect the score (sampling error): any change in directions. timing. or rapport with the test administra- tor could cause some variability (administrator error): inaccura- cies in scoring a test paper will affect the scores (scoring error): and finally such things as health. motivation. degree of fatigue of the person. and good or bad luck in guessing could cause score variability. (p. 7) Validity measures of the content- and criterion-related types. along with other validity measures. were discussed in the preceding section. Hith no empirical data existing on the Michigan vocational/ occupational student assessment model. this researcher obtained meas- ures of criterion-related validity using the subjective judgments 40 provided by the pilot-site instructors as his external criterion. Content validity was measured during the local validation process (described in Chapter III) performed by the pilot-site instructors and through the V-TECS test-item writing process. done before items were inserted into the item bank. Test-Item Banking Test-item banking systems are having an important influence on educational test development. A few item banks exist today. and the benefits that can be achieved through item-bank test development have been extolled (Orenstein. 1981). Numerous articles have been written about particular item-bank operations (Eaddy. 1985: Faggen. 1978: McCage. 1986: Orenstein. 1981). However. despite the widespread pro- liferation of item banking in a number of different subject areas. definitive and comparative data regarding the use of item banks are lacking (Orenstein. 1981). Orenstein discussed the strengths and limitations associated with the use of item banks for test construction. He described the strengths as follows: Time savings. cost savings. good item quality. test security. the construction of minimum competency tests. complete descriptions of content covered by items. schemes for classifying and coding items which provides for each item's retrieval. appropriate statistical information on item quality. and procedural operations for using the item bank. Orenstein cited the absence of “validity information. reliability esti- mates. and instructional materials that complement tests“ as limita- tions repeatedly voiced by test-item-bank users. 41 Test-item quality and validity have been a priority when devel- oping items for use in item pools (Orenstein. 1981). The Vocational- Technical Education Consortium of States (V-TECS. 1985) developed The Handbook for Developing Criterion-Referenced Test Items to establish a standard by which member states could construct and validate test items specifically identified for use in item banks as well as in traditional vocational/occupational exams. The Handbook contains five sections that explain the process of writing and validating each item. The process includes such steps as assisting users in locating and veri- fying tasks. through task analysis. and field testing strategies for validation.» Summar The present study is the first report on the student occupational assessment pilot study conducted at various sites in Michigan during spring 1986. Related concerns surrounding the present research-- history of general education and vocational/occupational education evaluation and the effects of evaluation--were discussed in the first sections of the review. Over the years. educators have seen that measurement of student achievement is of major concern both within education (for program improvement/evaluation of student achievement) and outside the educational arena (for justification and advancement). The requirement of evaluation in vocational/occupational education by the Carl Perkins Vocational Education Act of 1984 (P.L. 98-524) has 42 brought new attention to the process by which classroom evaluation is conducted. Of the 1.200 occupational/vocational evaluation instruments men- tioned in the literature. only about 150 have been field tested (Lewis. 1986). In addition to the Michigan State University occupational/voca- tional test-item bank. only two centers (IDECC and the driving- knowledge item bank) were discovered that develop occupational Ivoca- tional exams by accessing a data bank of multiple-choice test items. IDECC's method of developing exams parallels the procedure used to produce Michigan's exams. However. reliability and validity estimates for the IDECC exam were unavailable. The driving-knowledge item bank has been field tested. and both reliability and validity have been reported. However. the reliability estimates were gained using norm- referenced methodologies (Kuder-Richardson 20) as compared to test- retest methods. which were used in the present study. The final section of the review dealt with the reliability and validity of criterion-referenced tests. as well as of some norm- referenced measures. Types of reliability and validity were discussed. along with arguments for and against specific methods of estimating reliability and validity. The value of this study was underscored. based on the importance testing “experts“ have placed on estimating reliability and validity for sound assessment instruments. CHAPTER III RESEARCH METHODOLOGY Introduction Chapter III contains a discussion of the research methodology employed in the study. including the basic design of the study. the research questions. the population. pilot-test instructor inservice sessions. pilot-test instrument-development procedures. development and pilot testing of the instructor ranking sheets. data-collection techniques. and statistical-analysis procedures. Desigp_of the Stpdy The research design shown in Figure 3.1 is helpful in conceptual- izing the steps involved in conducting this study. The various proce- dures followed in carrying out the investigation are explained in the following pages. Research Questions The study methodology was designed to test the research questions posed in this study. Those questions are as follows: Question 1: How stable are test scores obtained on the voca- tional/occupational assessment test? Question 2: How stable and equivalent are scores obtained on parallel forms of the vocational/occupational assessment test? 43 44 .cmpmmv 388$; ”Tn 953“. cobc$oz 355m .8 ice—«30:00 $35 0:58... “.02.... we: no menswear 3:: m> a mwugzmm 3— _ 3 mm 9.985 «Em Mung 58 up cowummuummo 5539.2 58 IllLlll ”22:5 acmuusucou uo monument Jae—mum: m a 323.3 in >3:an Go 3652.26 .8 3: 35—3. umfiozuumob “co—unucowwun Ema ~¢ a ~< mace... .mwuoum uwwuozuumn’r c03m£~m> 30$ “8 3303sz S E ace 2 mine... no 22:: .5 33:8 _ E a 2 950m .555an amp... 9: no ”39503?! o5 L8 330959.: .5»... mEZmS xmfir men 3.5 82.; to 558:8 h _ 35.32: _ Li: “(gun—Eco mac—you: quiwml: _ can mtozum: nut—Lao... 3:62—1:08 \Ecomuooni go 33?: 28 d :< 45:8 L _ mucwguw c5322: uo macaw 9.5m — 9: co 3 can 2 DEB 58 $2.12. 6>u0up= man—uh. 8 S>ww Ou 0>Ah — _ mucwusum 08\Uo> .5953: co ~< .Eo... mugims £55933 “conga amen. _ No a ~< 2.50m “v.8. :5: out: :— a ~¢ munch mama? _ m> > mos 0 £058 u mxmo... in 833 .5595 .5303 no :ozooumm :éofiugumc— no.“ .3535 xmme new. .38 82.-., 9: Lo . dam. HO nw>m~8 Em muomnm 955(83285 9: 815,. _ 85:8". 2 HA Nogguumu: H fez—«TEDU . 2 Sub 95:55 :96; "035w TE conSLoQ 93 up Cass—«Econ: 4 mung rites—nu uo 3:35—02 can >a€3m> Ou gumbo“. cum: coffin—gm no floor—uggoooz 603.32 3.8550 no .615: 5:323: Ea 36:2, mkucwauwuicocmwumuu no 30min _ 2on6: Em £950sz motooE. Enough coins—gm 0:... no 233qu on» up 335: Ecoflaaooo>mcgumuo> COuuflgE «Eng 86 Saumgs ca — no 335m 45 Question 3: How stable are subscore rankings obtained on the vocational/occupational assessment test? Question 4: How stable and equivalent are subscore rankings obtained on parallel forms of the vocational/occupational assess- ment test? Question 5: How valid are the percentile rankings obtained on the vocational/occupational assessment test? Question 6: How valid are the subscore rankings obtained on the vocational/occupational assessment test? Population In March 1986. Michigan State University personnel asked voca- tional/occupational education directors at both the secondary and post-secondary levels to nominate programs and instructors to pilot test a new assessment procedure (see Appendix D). The study comprised 11 occupational specialties (see Appendix D). Included in these specialties were 46 vocational/occupational education programs. (For simplicity. the term “program“ as used in this dissertation refers to the specific site. instructor. and students enrolled in spring 1986.) Program inclusion was based on the following criteria: (a) the voca- tional/occupational education program received federal and/or state funding for the 1985-86 school year: (b) a certified vocational instructor was employed: (c) the program was an occupational specialty for which a test-item bank had been developed (a poOl Of items was available for 11 occupational specialties): (d) the program had admin- istrator and instructor support for the pilot-testing effort. and these individuals volunteered to become involved in the study: and (e) 46 the instructor had the flexibility in programming to allow for sched- uling of both test and retest dates. Each administrator and instruc- tor had an equal opportunity to participate in the vocational/occupa- tional pilot testing conducted by the Michigan Department of Education V-TES. From the vocational/occupational specialties meeting the afore- mentioned criteria. two occupational specialties--mechanica1 drafting and machine trades--were selected. All programs within these two spe- cialties were included in the study. Four mechanical drafting programs and seven machines trades programs were identified. a total of 11 pro- grams. The student enrollment in each program and the response rate (in parentheses) were as follows: Mechanic_1 Drafting_ Machine Trades Site H n = 42 (39) Site A n = 11 ( 10) Site I n = 25 (21) Site 8 n = 52 ( 37) Site J n = 23 (18) Site C n = 31 ( 25) Site K n = 27 (19) Site 0 n = 23 ( 21) 117 (97) Site E n = 20 ( 15) Site F n = 28 ( 22) SiteG n=___6_ ( 3) 171 (133) The total number of vocational/occupational education students involved in the test-retest procedure. and for whom participating instructors returned an instructor ranking sheet. was 230 or 80% par- ticipation. 47 Inservice Sessions for Pilot-Test Instrgetors/Admmpjstmators During March and April 1986. the researcher met with each selected pilot-test-site instructor to demonstrate the process of creating assessment instruments using V-TECS duty and task listings. These meetings had two main objectives: First. the test-item bank was coded (cross-walked) to the V-TECS duty and task listings. not to any state curriculum guide. Therefore. exposure to and understanding the listings was important to the success of the testing effort. Second. the instructor ranking sheets (see Appendix E) used as criterion meas- ures for validity were developed directly from the V-TECS duty and task listings. The ranking sheets were pilot tested at these meetings and at subsequent individual meetings between the researcher and pro- gram instructors. The procedure for constructing the student assessment instruments was explained through the use of examples and scenarios. The researcher informed the pilot-test instructors about the process to be used to develop their student assessment tests. Those procedures were as follows: 1. Instructors chose from V-TECS duty and task listings for their occupational specialty those that constituted the content of their programs. 2. Participants indicated the number of items within the duty and/or specific task they wanted to appear on the test. The number of items per duty and/or task was based on the instructors' evaluation of the curriculum's emphasis on a particular duty or task area. The 48 maximum number of items generated on any assessment test was not to exceed 100. 3. The researcher developed test Forms Al/Bl and returned them to the instructors for the last steps of test development. Instruc- tors validated all test items on the preliminary forms (Al/Bl) and checked stems. distractors. and answers (from the answer key) for accuracy. Participants also identified items to be eliminated from the final version of the vocational/occupational assessment test (Forms A2/82) for that instructor. 4. Instructors returned both forms of the assessment test to the researcher for modification (elimination of undesirable test items and/or addition of items) into the final version of the vocationall occupational assessment test (Forms A2/82). The instructors were also informed of the procedures they were to perform to ensure the quality and integrity of this research. At the conclusion of the inservice meetings. test and retest dates were confirmed for each of the 11 programs involved in the study. Instructors were asked to inform their students of the pilot testing dates to ensure attendance. Development of the Instrpmeppa The Tests Within a week of the inservice meetings. the pilot-site instruc- tors returned their selections from the V-TECS duty and task listings to the researcher. Table 3.1 shows the duty areas selected for test- ing. along with the number of items requested from each duty area. for 49 each pilot site. These data constituted the test blueprint upon which Forms A1 and 81 of the test were developed. The tests for the two occupational specialties selected for this research (mechanical drafting and machines trades) were constructed at Michigan State University. using the occupational test-item bank that Michigan State University personnel had developed for the Department of Education. At the time of pilot testing. the number of items in each duty area available for inclusion in the tests was as shown in Table 3.2. There were at least three items for each task. It is evident that not all duty areas were well covered by test items at the time of pilot testing. The researcher assumed. however. based on the duty and task listings returned by pilot-site instructors. that items existed in the bank to address the research questions. Both forms of the tests were developed using items available in the item bank. The reader is reminded that the instructors controlled all items included in the final form of the test. The Instrgctor Ranking Sheets To collect data for studying concurrent validity. the researcher used instructor ranking sheets as a criterion measure (see Appendix E). Concurrent validity pertains to the empirical technique of study- ing the relationship between the test scores and some independent external measures (criteria) (Mehrens & Lehmann. 1978). The instruc- tor ranking sheets were administered to the instructors at the same time as or within 10 days of the student retest. As stated in 50 Table 3.l.--Duty areas selected for testing and number of items requested. by pilot site. Pilot # of Items Site Duty Code 8 Description Requested Mechanical Drafting H F--Detailing Single Parts 25 G--Drawing Layouts 20 J--Drawing Layouts of Single Parts 20 L--Drawing Pictorials 5 M--Maintaining and Caring for Tools 20 N--Reproducing Drawings 10 I F--Detailing Single Parts 5 G--Drawing Layouts 10 J--Drawing Layouts of Single Parts 5 L--Drawing Pictorials 5 M--Maintaining and Caring for Tools 15 N--Reproducing Drawings 10 J F--Detailing Single Parts 25 G--Drawing Layouts 20 J--Drawing Layouts of Single Parts 25 L--Drawing Pictorials 10 M--Maintaining and Caring for Tools 15 N--Reproducing Drawings 5 K F--Detailing Single Parts 25 G--Drawing Layouts 25 H--Drawing Charts 5 I--Drawing Diagrams 5 J--Drawing Layouts of Single Parts 15 L--Drawing Pictorials 5 M--Maintaining and Caring for Tools 25 N--Reproducing Drawings 15 51 Table 3.1.--Continued. Pilot # of Items Site Duty Code & Description Requested Machine Trades A D--Performing Metalwork Operations 10 E--Performing Bench Work 10 F--Operating Drill Presses 20 G--Operating Grinding Machines 15 H--Lathe Operations 25 I--Operating Milling Machines 10 J--Operating Power Saws 5 0--Maintaining Shop Facilities and 5 Work Areas 8 A--Performing Supervisory Functions 5 B--Performing Mathematical Calculations 5 C--Designing and Planning Machine Hork l5 D--Performing Metalwork Operations 5 E--Performing Bench Work 15 F--Operating Drill Presses 10 H--Lathe Operations 25 I--Operating Milling Machines 15 0--Maintaining Shop Facilities and 5 Work Areas C C--Designing and Planning Machine Work 5 D--Performing Metalwork Operations 5 E--Performing Bench Work 5 F--0perating Drill Presses 15 G--Operating Grinding Machines 15 H--Lathe Operations 25 I--Operating Milling Machines 25 J--Operating Power Saws 0 C--Designing and Planning Machine Work 5 D--Performing Metalwork Operations 5 E--Performing Bench Hork lO F--Operating Drill Presses 15 G--Operating Grinding Machines 15 H--Lathe Operations 20 I--Operating Milling Machines 25 J--0perating Power Saws 5 52 Table 3.l.--Continued. Pilot # of Items Site Duty Code 8 Description Requested E A--Performing Supervisory Functions 5 B--Performing Mathematical Calcu- lations 5 C--Designing and Planning Machine Work 5 D--Performing Metalwork Operations 5 E--Performing Bench Hork 5 F--Operating Drill Presses lO G--Operating Grinding Machines 10 H--Lathe Operations 20 I--Operating Milling Machines 20 J--0perating Power Saws 5 L--Operating Shapers 5 N--Maintaining Machines and Tools 5 O--Maintaining Shop Facilities and 5 Work Areas F B--Performing Mathematical Calcu- lations 5 C--Designing and Planning Machine Work 10 D--Performing Metalwork Operations 10 E--Performing Bench Work 10 F--Operating Drill Presses 15 G--Operating Grinding Machines 15 H--Lathe Operations 30 I--Operating Milling Machines 30 J--0perating Power Saws 5 G B--Performing Mathematical Calcu- lations 10 C--Designing and Planning Machine Work 5 D--Performing Metalwork Operations 10 E--Performing Bench Work 5 F--0perating Drill Presses 10 G--Operating Grinding Machines 5 H--Lathe Operations 20 I--Operating Milling Machines 20 J--Operating Power Saws O--Maintaining Shop Facilities and 5 Work Areas 53 Table 3.2.--Number of items in each duty area available for inclusion in tests. Duty # of Items Code Description of Duty Available Mechanical Drafting A Designing Assembly Tooling 35 8 Designing Cutting Dies 3 C Designing Gages 3 0 .Designing Molds 11 E Designing Specialty Fixtures 82 F Detailing Single Parts 195 G Drawing Layouts 184 H Drawing Charts 49 I Drawing Diagrams 71 J Drawing Layouts of Single Parts 113 K Drawing Machine Diagrams 27 L Drawing Pictorials 59 M Maintaining and Caring for Tools 255 N Reproducing Drawings 48 O Trouble-Shooting and Correcting 6 Design Errors Total Hritten Items = 1.141 Machine Trades A Performing Supervisory Functions 12 8 Performing Mathematical Calculations 19 C Designing and Planning Machine Work 28 0 Performing Metalwork Operations 59 E Performing Bench Work 21 F Operating Drill Presses 79 G Operating Grinding Machines 80 H Lathe Operations 235 I Operating Milling Machines 122 J Operating Power Saws 30 K Operating Presses 3 L Operating Shapers 28 M Performing Production Machinist Line 3 N Maintaining Machines and Tools 65 O Maintaining Shop Facilities/Work Areas 9 Total Hritten Items = 7—— 54 Chapter I. the rankings were artificial dichotomies (pass/fail). col- lected at the duty level. These estimates were correlated with the artificially dichotomized student subscores (pass/fail) obtained on the specific subtests addressed in the vocational/occupational assess- ment test. Student subscores were placed into artificial dichotomies because of the relatively few participants at each site. The instructor ranking sheets included the maximum number of duty areas requested by the instructors. Therefore. not all V-TECS duties were used in constructing the ranking sheets. (For consensus agree- ments. see Table 3.1.) On the sheets. each duty area was preceded by a simple pass/fail box for the instructors to check. indicating their ranking of a particular student for that duty area. This system allowed instructors to indicate quickly whether cn' not the student possessed the skill/ knowledge in each of the duty areas tested. The pass-fail criterion was as follows. For a “pass“ the student was judged to possess enough skill/knowledge to obtain a ranking of 50% or higher on questions pertaining to a specific skill (duty) area. Stu- dents receiving a ''fail“ were judged not to possess enough skill/ knowledge in the stated skill (duty) area to obtain a ranking of 50%. The instructor ranking sheets required one additional item of information. the students' percentile ranking. To elicit meaningful rankings quickly. the researcher divided the 100 points of the percen- tile scale into five equal sectors (1-5) of 20 percentage points each (sector one = 0-20 points. sector two = 20-40 points. and so on). The instructors could easily rate students within one of the five sectors. 55 This criterion measure was then correlated with the percentage ranking obtained by students on the vocational/occupational assessment tests. The instructor ranking sheets were pilot tested in two stages. First. each instructor reviewed the format of these sheets during inservice meetings conducted at various sites throughout Michigan dur- ing March and April 1986. Second. vocational curriculum specialists. as well as research and testing professionals. critiqued the ranking sheets' construction and format. The researcher considered all opin- ions before constructing the final forms for the study. (See Appendix E for the final version of the instructor ranking sheets.) Data Collection The instructor ranking sheets were mailed or delivered to the two groups of pilot-site instructors--mechanical drafting and machine trades. Included was a cover letter with directions for completing and returning the instruments. as well as a stamped. addressed enve- lope in which participants were to return the completed sheets. (See Appendix D.) Each pilot-site instructor received a ranking sheet for each student in the class. The following information was already on the top of the sheet to insure proper coding: instructor code. stu- dent number/name. program code/title. and school/center code. The researcher conducted a telephone follow-up shortly after the testing and retesting had taken place. All the instructor ranking sheets were returned. During May. June. and July 1986. the researcher conducted a series of test-retests with the students from the 11 vocationall 56 occupational programs described earlier in this chapter. Each site included in the study was given a test and a retest date: thus 22 site visitations were required for data collection. The researcher con- ducted 20 of the 22 test/retest administrations of the assessment instrument. Twice he could not be present for either the test or the retest because of student motivation problems and/or time conflicts with other pilot-test locations. In both cases. the pilot-site instructors were well informed about the testing/retesting process and the proper administration of the instruments. The researcher used the following procedures with the students in both the test and retest phases: 1. Introduced the test-administration process to the students. .2. Discussed with students the reasons for the pilot testing/ retesting effort. 3. Gave students instructions for filling in the computer-scan answer sheet. 4. Encouraged the students to do their best and wished them well on the examination. On the test day. Form A2 of the vocational/occupational student assessment test was administered to all students in attendance. Stu- dents were given enough time to complete all items on the test. They were encouraged to take their time and were told that the researcher would stay after the scheduled class session if they needed extra time to complete the test. However. this was not necessary because all students finished their tests within the scheduled time. 57 The retest of the vocational/occupational student assessment test was conducted at all 11 test sites within five to seven days of the first administration. Some students who had taken part on test day did not show up for retest day: conversely. some students not involved on test day were present on retest day. Students were allowed to take the test the one day they attended class: however. only data from students in attendance on both days (test and retest) were used to answer the research questions. On the retest day. both ‘forms of the vocational/occupational education assessment test were administered. As students entered the room or after they were seated. the test administrator handed out the test booklets and answer sheets. The tests were handed out from the top of a preordered pile of exams. which alternated between Form A2 and Form 82. The students who took Form A2 of the assessment instrument on both occasions provided (in total or in part) the data used to answer Research Questions 1. 3. 5. and 6. The students who took Form A2 on test day and Form 82 on retest day provided (in total or in part) the data required to answer Research Questions 2. 4. 5. and 6. Data Processing and Analysis Once the testing period was over and both test and retest student answer sheets had been collected. the scoring process began. Header sheets and answer keys for the 22 forms (A and B) of the assessment instrument were prepared and taken to the Pfichigan State University 58 computer scoring laboratory for processing. FUll-item analysis sta- tistics were requested for each student. by school and by program. Data gathered from the instructor ranking sheets were entered by hand into a computer base and cross-checked by personnel in the Michi- gan State University Department of Applications Programming. The pass/fail rankings and percentile rankings were catalogued as the test-retest student data (by test site. program. and student number). Tables were constructed using the aforementioned data and delivered to the Department of Applications Programming of the Michigan State Uni- versity computer laboratory for analysis. (See Figure 3.2.) To answer the six research questions. the following methods were used. The students who took Form A2 of the assessment test on both occasions provided the data used to answer Research Question 1. Since the test data were in the form of continuous scores. the Pearson product-moment correlation (r) was computed to obtain a reliability estimate of test stability. The students who took Form A2 of the assessment test on test day and Form 82 of the instrument on retest day provided the data used to answer Research Question 2. Again. since the test data were in the form of continuous scores. the Pearson product-moment correlation (r) was used to gain the reliability estimates of test stability and equivalence. Test A2 and retest A2 scores were subdivided by duty area. which yielded a subscore for each duty tested on both occasions. These subtest scores were converted into artificial dichotomies (pass/fail) 59 Data table key TIIIIIfiIIIIIII m. awn m u.:n EC th ”nun. own 2. m.u:---- .t s e s a.K 1.9.? no» my. a» e T. m Wm m riotournou I I InstructorTPFEdIét ed 4T I l anyflmmhu Pass/Fail ecen 3e. erence . ercen 80 . Between A/BlRank Actualiflank PredictediGrade i Pass/Fail ore . I 1 e. 0". :Test Alfieteat AlRete:t B I I r I I I I I Site A J S I G Duty Areas Tested E F H D T‘P P l P i P TIP TIP 1 FIPIPIPIPIPIPIPIPIP: I? 5 1 7 u 5. 5 ID IPIPIPIPIFIFI P iOOlIli I F n F . up. lfill up. If. I1? up. I_w. up. LP + nu .__-“ F i F i F i F uPlFlFlFiFlPiFiF f T”? i 3u:-3 : 2: u :c-: {P 10031371 .-1 Wu... “H me nu... NH NH... NH... MM... r: 111 .r Yer L. .H FIF P I I I I F l FAT F FlFlFiFlFiFlFiF I- : A :c. P:P1P}F.F1F:F1P:FiFiFiFlPiPlFlF: u a 7 i .00 .37: 10081301 E71 1 7 1 HI E 16+ Sample data table. Figure 3.2: 60 and correlated with each other using the tetrachoric correlation (rt) to give a reliability estimate of stability within duty areas. When empty cells occurred in the 2 x 2 correlational matrix. the formula could not be used. Therefore. percentage agreements were computed between test and retest subscore rankings to help clarify the find- ings. These data provided the answer to Research Question 3. Test A2 and retest 82 scores were also subdivided by duty and converted to artificial dichotomies. which yielded another set of dichotomous data to be correlated using the tetrachoric correlation (rt) for reliability estimates of stability and equivalence. Again. when empty cells occurred in the 2 x 2 correlational matrix. the for- mula could not be used. Therefore. percentage agreements were com- puted between test and retest subscore rankings to help clarify the findings. These data were used to answer Research Question 4. Student test scores were converted to percentile ranks. lhese rankings enabled the researcher to identify each subject's performance relative to the group of participants from his/her test site. The test site instructors also provided individual student rankings. on the instructor ranking sheets. The relationship that existed between these two sets of data was measured using the chi-square test. The results were computed for each site. for each occupational division. and across all sites. These results provided the answer to Research Question 5. To answer Research Question 6. the researcher averaged each stu- dent's individual duty subscores from both the test and retest. This 6] resulted in one subscore for each duty area addressed on the tests. The relatively small number of participants necessitated converting these subscores into artificial dichotomies (pass/fail). The test site instructors provided individual duty rankings for each partici- pant. The relationship that existed between these two sets of data was computed using tetrachoric correlations for each duty area. The results were computed for each site. for each occupational division. and across all sites. The data correlated here were used to answer Research Question 6. For all analyses. an alpha level of .05 or less was established to indicate statistical significance. Results of the data analyses are presented in Chapter IV. CHAPTER IV RESULTS OF THE DATA ANALYSIS Intrpdaction Results of the analysis of data collected to answer the six research questions are presented in this chapter. Restated. the questions are as follows: Research Qaestion 1: How stable are test scores obtained on the vocational/occupational assessment test? Research Qgestion 2: How stable and equivalent are scores obtained on parallel forms of the vocational/occupational assess- ment test? Research Qgestion 3: How stable are subscore rankings obtained on the vocational/occupational assessment test? Resegeh Question 4: How stable and equivalent are subscore rankings obtained on parallel forms of the vocational/occupa- tional assessment test? Research Qpestion 5: How valid are the percentile rankings obtained on the vocational/occupational assessment test? Research Question 6: How valid are the subscore rankings obtained on the vocational/occupational assessment test? The data sources for this study were (a) the test/retest scores of selected Michigan vocational/occupational students involved in the occupational assessment pilot-testing program conducted during spring 1986 and (b) instructors' rankings of students' knowledge. also col- lected in spring 1986. The occupational specialties included in this research were mechanical drafting and machine trades. Seven test 62 63 sites were used in machine trades and four in mechanical drafting (machine trades test site G was eliminated from the analysis due to law student participation). In the following pages. the findings for both occupational spe- cialties are presented according to the research questions posed. Findings are presented by occupational specialty relative to the research questions. Results Research Question 1 How stable are test scores obtained on the vocational /occupa- tional assessment test? Machine trades. The Pearson product-moment coefficient of corre- lation (r) for the test/retest scores on Form A2 of the machine trades assessment test was computed for each site (see Table 4.1). The r's for three of the six sites (Sites C. E. and F) were significant at the .05 level. According to Rowntree (1981). the relationships found for Sits A. C. E. and F were either moderate or strong. Even though Rowntree's definition denotes moderate to strong relationships. the reader is advised to interpret the findings cautiously because of the small population size. Mechanical drafting. Table 4.2 contains the results for the analysis of the mechanical drafting test/retest scores. The Pearson r's for the test/retest scores were significant at the .05 level for three of the four test sites (Sites I. J. and K). The relationships were moderate to very strong (Rowntree. 1981). Again. given the small 64 population size. the reader is cautioned about the practical signifi- cance of the results. Table 4.l.--Correlational data for machine trades test Form A2/retest Form AZ. Site Pearson r Value A .590 (n = 5) B .116 (n = 16) C .820* (n = 13) D .371 (n = 10) E .687* (n = 7) F .723* (n = 9) *Significant at the .05 level. Table 4.2.--Correlational data for mechanical drafting test/retest. Form A2. Site Pearson r Value H .034 (n = 18) I .548* (n = 11) J .917* (n = 7) K .763* (n = 7) *Significant at the .05 level. Overall. the data pertaining to the measures of stability were moderate. Significant relationships were found at six of the ten sites (Sites C. E. F. I. J. and K). and at one other site (Site A) a moderate relationship was found. A Pearson r of .917 for mechanical 65 drafting Site J was the highest. whereas the r of .034 computed for Site H was the lowest. The results of the analysis seem to indicate stability in the machine trades and mechanical drafting test scores. Research_Qpestion 2 How stable and equivalent are scores obtained on parallel forms of the vocational/occupational assessment test? Machine trades. Table 4.3 contains the Pearson r's for the test/ retest scores on parallel forms of the machine trades assessment test. Strong relationships were found between sets of student scores on test Forms A2 and 82 at five of the six sites (Sites A. C. D. E. and F). The results at Sites C. D. E. and F were significant at the .01 level. whereas those at Site A were significant at the .05 level. These results suggest that Forms A2 and 82 of the machine trades assessment test are equivalent and that the scores appear to be stable. Table 4.3.--Correlationa1 data. machine trades test Form A2/retest form 82. Site Pearson r Value A .817* (n = 5) B .280 (n = 21) C .710** (n = 12) D .852** (n = 11) E .812** (n = 8) F .724** (n = 13) *Significant at the .05 level. **Significant at the .01 level. 66 Mechanical drafting. As indicated in Table 4.4. positive Pearson r's were found between test/retest scores on Forms A2 and 82 of the mechanical drafting assessment test. Three of the four test sites (Sites I. J. and K) had results that were significant at the .05 level. Even though the relationships can only be described as moder- ate (Rowntree. 1981). the mechanical drafting assessment tests seem to be equivalent. and scores on those tests appear to be stable. Table 4.4.--Correlational data for mechanical drafting test Form A2/ retest Form 82. ‘ Site Pearson r Value H .071 (n = 21) I .592* (n = 10) J .566* (n = 11) K .661* (n = 12) *Significant at the .05 level. The relationship between the scores on equivalent forms of the vocational/occupational assessment tests was significant at eight of the ten sites. Even though the relationships were moderate. the results were significant and appear to indicate the equivalence and stability of the tests and scores. respectively. That is. the tests can be used for path long-range prediction and inferences to a domain of knowledge. 67 Research Question 3 How stable are subscore rankings obtained on the vocational/occu- pational assessment test? Machine trades. Results of the test/retest of machine trades assessment test Forms A2. across all sites. are shown in Table 4.5. The tetrachoric correlations computed for each site indicated moderate to very strong relationships between students' subscore rankings on the test and their subscore rankings on the retest. The highest tetrachoric correlation (.737) was found at Site C: the lowest tetrachoric correlation was found at Site D (.124). At four of the machine trades sites (Sites A. C. E. and F) where students were tested and retested with the same form (A2) of the assessment instrument. moderate relationships (Rowntree. 1981) were noted between subscore rankings. Even though the relationships were moderate. at all four sites they were significant at the .05 level or lower. The results imply there was stability between test/retest student subscore rank- ings. Mechanical drafting. Table 4.6 contains the results of the analysis for mechanical drafting. The combined mean percentage agree- ments indicated relationships ranging from moderate to strong. Site H had the largest percentage agreement (94%). although a tetrachoric correlation statistic could not be computed. At three of the four sites (Sites I. J. and K) where FOrm A2 of the mechanical drafting test was administered twice. the relationships were significant at the .01 level. These results seem to indicate stability within students' subscore rankings on the mechanical drafting assessment tests. 68 Table 4.5.--Subscore ranking agreement and tetrachoric correlations: machine trades. test/retest Form A2. Mean Percentage Mean Percentage Tetrachoric Site Agreement Disagreement Correlation (rt) A 73 27 .583* (n = 40) B 70 30 .134 (n = 96) C 77 23 .737** (n = 104) D 54 46 .124 (n = 70) E 67 33 .47/* (n = 56) F 78 22 .733** (n = 54) Note: n = students x subtests (duty areas). *Significant at the .05 level. **Significant at the .01 level. Table 4.6.--Subscore ranking agreement and tetrachoric correlations: mechanical drafting. test/retest Form A2. Mean Percentage Mean Percentage Tetrachoric Site Agreement Disagreement Correlation (rt) H 94 6 --- I 71 29 .548* (n = 60) J 74 26 .640* (n = 66) K 88 12 .850* (n = 96) Note: n = Students x subtests (duty areas). *Significant at the .01 level. The tetrachoric correlations for subscore stability are shown in Table 4.7 for both occupational specialties. 'The tetrachoric 69 correlations were both significant at the .001 level. Mechanical drafting had the largest reported statistic (.871). whereas machine trades had the smallest (.561). The results indicated that students' subscores on the machine trades and mechanical drafting assessment tests were stable over time. Table 4.7.--Tetrachoric correlations for combined subscore rankings on the vocational/occupational assessment test. All Subscore Rankings Test A2/Retest A2 Machine trades .561* (n = 330) Mechanical drafting .871* (n = 222) Note: n = students x subtests (duty areas). *Significant at the .001 level. Research Question 4 How stable and equivalent are subscore rankings obtained on parallel forms of the vocational/occupational assessment test? Machine trades. The degree of relationship found between duty subscore rankings on parallel forms of the machine trades assessment test is shown in Table 4.8. All subscore rankings were combined by site and averaged to compute a tetrachoric correlation for each site. At two of the sites (Sites C and E). the relationships were signifi- cant. At the other sites. nonsignificant relationships were found. Site E had the highest percentage agreement (71%). whereas Site 0 had the lowest agreement between test/retest subscore rankings (54%). 70 Table 4.8.--Subscore ranking agreement and tetrachoric correlations: machine trades. test Form A2/retest Form 82. Mean Percentage Mean Percentage Tetrachoric Site Agreement Disagreement Correlation (rt) A 63 37 .387 (n = 40) B 64 36 .027 (n = 96) C 60 40 .280* (n = 104) D 54 46 .121 (n = 70) E 71 29 .596** (n = 56) F 61 39 .196 (n = 54) Note: n = students x subtests (duty areas). *Significant at the .05 level. **Significant at the .01 level. Mechanical drafting. The percentage of subscore ranking agree- ment on equivalent forms of the mechanical drafting test is shown in Table 4.9. The relationships recorded at Sites I and J were weak: however. the tetrachoric correlation at Site J was significant at the .05 level. Site H had the highest percentage agreement for the mechanical drafting sites (88%). Site K had the lowest percentage agreement (58%). followed by Site I (55%). The results of the tetrachoric correlations for both occupational specialties with regard to subscore stability and equivalence are shown in Table 4.10. The tetrachoric correlations were both signifi- cant at the .001 level. Mechanical drafting had the strongest reported relationship (.532). whereas machine trades had the weakest (.333). The results of the analysis suggest that students' subscores 71 on the machine trades and mechanical drafting assessment tests were stable and that the subtests (duty areas) were equivalent. Table 4.9.--Subscore ranking agreement and tetrachoric correlations: mechanical drafting. test Form A2/retest Form 82. Mean Percentage Mean Percentage Tetrachoric Site Agreement Disagreement Correlation (rt) H 88 12 --_ I 55 45 .351 (n = 60) J 73 27 .336* (n = 66) K 58 42 .136 (n = 96) Note: n = students x subtests (duty areas). *Significant at the .05 level. Table 4.lO.--Tetrachoric correlations for combined subscore rankings on the vocational/occupational assessment test. All Subscore Rankings Test A2/Retest 82 330) 222) Machine trades .333* (n Mechanical drafting .532* (n Note: n = students x subtests (duty areas). *Significant at the .001 level. 72 Research uestion 5 How valid are the percentile rankings obtained on the vocationall occupational assessment test? Machine trades. The percentage of agreement between students' achievement on the tests and instructors' rankings of the students across all sites is displayed in Table 4.11. In 30% of the cases. student performance matched instructors' rankings. More than half (52.5%) of the machine trades students did not achieve as well as their instructors expected they would. Seventeen percent of the stu- dents performed better than expected. Table 4.11.--Students' percentile rank achievement in relation to instructors' rankings of students: machine trades. all sites combined. in frequency and percentage of total (N = 130). Instructor Ranking Actual l 2 3 4 5 % of % of % of % of % of No. Total No. Total No. Total No. Total No. Total l l *.5 7 -5.0 7 -5.0 2 -l.0 2 3 +2.0 6 *4.5 11 -8.0 9 -7.0 5 -4.0 3 2 +1.0 8 *6.0 13 -l0.0 5 -4.0 4 1 + .5 7 +5.0 12 *9.0 9 -7.0 5 4 +3.0 8 +6.0 13 *I0.0 Key: *Percentage agreement = 30%. +Student achievement above ranking = 17.5%. -Student achievement below ranking = 52.5%. As shown in Table 4.12. the percentage agreement between machine trades students' percentile rankings based ("1 test achievement and 73 their instructors' rankings was weak. However. the chi-square value was significant at four sites (Sites A. C. D. and F). The highest agreement between rankings was recorded at Site E (53.4%): the lowest was recorded at Site 8 (18.9% match between students' achievement and instructors' rankings). On the average. 48% of the machine trades students performed at a rate below what their instructors expected. Conversely. 19.5% of the machine trades students achieved better than their instructors expected. These results seem to indicate little relationship between students' percentile rank achievement and their instructors' rankings of them. Table 4.12.--Percentages and chi-square results for students' rank achievement in relation to instructors' rankings of students' achievement: machine trades. Instructor Ranking/Student Rank Site Chi-Square % Agreement % Above % Below Results A 30.0 0 70.0 20.0** (n = 10) 8 18.9 29.7 51.4 18.9 (n = 37) C 44.0 24.0 32.0 26.1* (n = 25) D 19.0 0 81.0 19.4* (n = 21) E 53.4 20.0 26.6 21.3 (n = 15) F 27.2 9.1 63.7 37.5** (n = 22) *Significant at the .05 level. **Significant at the .01 level. Mechanical drafting. The overall percentage of agreement between students' achievement on the tests and instructors' rankings of their 74 students is illustrated in Table 4.13. In 30.9% of the cases. the mechanical drafting students' actual quintile rank matched the ranking provided by the site instructors. Forty-four percent of the students fell short of their instructors' expectations. and 25% achieved better than expected. Table 4.13.--Students' percentile rank achievement in relation to instructors' rankings of students: mechanical drafting. all sites combined. in frequency and percentage of total (N = 97). Instructor Ranking Actual 1 2 3 4 5 Rank % of % of % of % of % of No. Total NO. Total No. Total No. Total No. Total l 4 *4 2 -2 2 -2 3 -3 1 -l 2 3 +3 3 *3 5 -5 10 -10 3 -3 3 2 +2 7 +7 6 *6 7 -7 5 -5 4 2 +2 1 +1 5 +5 10 *10 4 -4 5 2 +2 3 +3 7 *7 Key: *Percentage agreement = 30.9%. +Student achievement above ranking = 25%. -Student achievement below ranking = 44.1%. Illustrated in Table 4.14 is the percentage agreement between students' rankings on the mechanical drafting assessment test and instructors' rankings of their students. Results of the chi-square analysis are also presented. Overall. the results appear to indicate a poor relationship between the two variables tested. 75 Table 4.14.--Percentages and chi-square results for students' rank achievement in relation to instructors' rankings of students' achievement: mechanical drafting. Instructor Ranking/Student Rank Site Chi-Square % Agreement % Above % Below Results H 33.3 53.8 12.9 19.0 (n = 39) I 14.3 9.6 76.1 5.9 (n = 21) J 50.1 0 49.9 27.2* (n = 18) K 26.3 10.6 63.1 4.3 (n = 19) *Significant at the .01 level. Research Question 6 How valid are the subscore rankings obtained on the vocationall occupational assessment test? flaghine trades. Shown in Table 4.14 are the frequencies and per- centages of the instructors' perceived rankings and students' actual subscore rankings in four cells: pass/pass. pass/fail. fail/pass. and fail/fail. Fifty-eight percent of the time (33% + 25%). instructors predicted accurately (by ranking each student on each duty area cov- ered in the test) students' achievement (pass/pass or fail/fail). Forty-two percent of the time. the students' performance was above or below their instructors' rankings. Ten percent of the students fared better than expected. whereas 32% achieved at a rate lower than their instructors' rankings. 76 Table 4.15.--Vocationa1loccupational assessment test subscore averages in relation to instructors' rankings: machine trades. all sites. Instructors' Rankings Students' Subscore Rankings Pass Fail Pass No. 308 84 % +33 -10 Fail No. 302 231 % -32 +25 Note: +58% agreement. -42% disagreement. The relationship between percentage agreements was significant at the .01 level. N = 925 (students x subscore rankings). Table 4.16 contains the results of the data analysis for machine trades. by test site and for all sites combined. The tetrachoric correlation between instructors' rankings and students' subscore rank- ings was statistically significant at all sites except Site 8. The tetrachoric correlations for Sites 0. E. and F were significant at the .001 level. whereas the correlations for Sites A and C were significant at the .05 level. The tetrachoric correlation for all machine trades sites combined was significant at the .001 level. The analysis results appeared to indicate that relationships between students' achieved percentile rankings and instructors' rankings of their students' achievement were valid. However. because of the small population size. the reader is cautioned about the practical signifi- cance of the results. 77 Table 4.16.--Instructors' rankings in relation to students' subscore rankings: percentage agreement and tetrachoric correla- tion data. machine trades. Instructors' Rankings/Subscore Rankings Mean Percentage Mean Percentage Tetrachoric Site Agreement Disagreement Correlation (rt) A 59 41 .332* (n = 10) B 45 65 .005 (n = 37) C 55 45 .227* (n = 25) D 64 36 .532** (n = 21) E 79 21 .490** (n = 15) F 77 23 .679** (n = 22) A-F 58 42 .371** (n = 130) *Significant at the .05 level. **Significant at the .01 level. Mechanical drafting. Illustrated in Table 4.17 are the frequen- cies and percentages of instructors' subscore rankings of students and students' actual subscore rankings Iil‘fOUP cells: pass/pass. pass/fail. fail/pass. and fail/fail. Sixty-nine percent of the time. students who achieved “pass“ rankings on duty areas were ranked by their instructors as possessing the knowledge to perform at a passing level. Twenty percent of the time. students who achieved “pass“ rankings on duty areas were rated by their instructors as possessing the knowledge to perform at a passing level. Thirty-one percent of the time. instructors' rankings and students' subscore rankings did not match: in 19% of these instances. instructors ranked students as passing but the students actually failed the duty area. 78 Table 4.17.--Vocationalloccupational assessment test subscore averages in relation to instructors' rankings: mechanical draft- ing. all sites. Instructors' Rankings Students' Subscore Rankings Pass Fail Pass No. 121 75 % +33 -10 Fail No. 120 304 % -19 +49 Note: +69% agreement. -31% disagreement. The relationship of percentage agreements was significant at the .001 level. N = 620 (students x subscore rankings). Table 4.18 contains the results of correlational analyses performed on the mechanical drafting data. by site and for all sites combined. The tetrachoric correlations revealed a weak relationship: only one test site (Site I) had a correlation that was significant at the .05 level. The correlational result for all mechanical drafting sites combined was significant at the .001 level. The individual-site data illustrated a poor relationship between instructors' rankings of student achievement (per subscore rank) and actual student subscore achievement. However. combining all mechanical drafting site data suggested a statistically significant relationship between the test and the criterion measure. Again. the practical significance of this finding must be viewed with caution. 79 Table 4.18.--Instructors' rankings in relation to students' subscore rankings: percentage agreement and tetrachoric correla- tion data. mechanical drafting. Instructors' Rankings/Subscore Rankings Mean Percentage Mean Percentage Tetrachoric Site Agreement Disagreement Correlation (rt) H 99 l --- I 53 47 .259* (n = 21) J 54 46 .185 (n = 17) K 86 14 .072 (n = 19) H-K 69 31 .494** (n = 57) *Significant at the .05 level. **Significant at the .001 level. Summar The results of the data analysis provided answers to the research questions regarding the reliability and validity of the vocational/occupational assessment tests. Strong in) moderate estimates of stability were reported for the vocational/occupational assessment tests. Stability and equivalence estimates for the tests were also moderate to strong: statistically significant results were found at 80% of the test sites. Subscore rankings yielded statisti- cally significant results 78% of the time. suggesting stability of subscores. Stability and equivalence estimates were significant 30% of the time within individual sites: these estimates were significant at the .001 level for all sites combined. Tetrachoric correlations 80 and chi-square values computed between the tests and the criterion measures were statistically significant at 50% of the test sites. The mechanical drafting group had the weakest relationships between stu- dents' achievement (percentage ranking on the test) and their instruc- tors' rankings (criterion measure). The test's concurrent validity was established by correlating students' subscore rankings with instructors' rankings of their students using chi-square and tetra- choric correlation. The result was statistically significant at the .001 leVel. A summary of the findings is presented in Chapter V. along with conclusions drawn from those findings and recommendations for further research. CHAPTER V SUMMARY. CONCLUSIONS. AND RECOMMENDATIONS FOR FURTHER RESEARCH Summary The researcher's purpose in this study was to obtain estimates of reliability and measures of validity on assessment tests developed using the occupational test-item bank at Michigan State University. The occupational specialties selected for this study were machine trades and mechanical drafting. The following research questions were investigated: Research Question 1: How stable are test scores obtained on the vocational/occupational assessment test? Research Question 2: How stable and equivalent are scores obtained on parallel forms of the vocational/occupational assess- ment test? Research Question 3: How stable are subscore rankings obtained on the vocational/occupational assessment test? Research Question 4: How stable and equivalent are subscore rankings obtained on parallel forms of the vocational/occupa- tional assessment test? Research Question 5: How valid are the percentile rankings obtained on the vocational/occupational assessment test? Research_Question 6: How valid are the subscore rankings obtained on the vocational/occupational assessment test? 81 82 Procedures The population was composed of 171 machine trades students and 117 mechanical drafting students (N = 288) from 11 vocational/occupa- tional education programs representing the machine trades and mechani- cal drafting occupational specialties. and the program instructors who volunteered to participate in the study. The data analyzed for the reliability portion of the study were the usable test/retest scores of 230 vocational/occupational students. For each occupational specialty. subjects were initially tested with Form A2 of the vocational/occupational assessment test (developed by their instructors in conjunction with Michigan State University test-item-bank personnel). From five to seven days after taking the test. the students were retested with either Form A2 or a parallel form (82) of the vocational/occupational assessment test. Stability and stability and equivalence estimates were computed. The test site instructors were asked to rank their students' knowledge at both the occupational and duty levels. using the Vocational-Technical Education Consortium of States (V-TECS) duty and task listings modified for this study. Data provided by the instructors constituted the criterion measure used in studying the concurrent validity of the assessment tests. Although content validity. per se. was not a part of this study. the manner in which the tests were constructed was taken as evidence that they were content valid. 83 Findings and Conclusions The findings and conclusions are presented separately for the two occupational specialties. Rowntree's (1981) labels (strong. weak. and so on) were used to interpret the strengths of the relationships. machine Trades 1. Pearson product-moment coefficients of stability averaged 0.55 for the machine trades assessment tests. which was interpreted as moderate. As sample size decreases. reliability estimates typically decrease as well (Ebel. 1979: Mehrens & Lehmann. 1984). making it increasingly difficult to obtain statistical significance. It is conceivable. because of the small number of participants. that the average stability coefficients for the machine trades tests were spuriously low. A larger sample might have resulted in a more hetero- geneous group. thereby yielding a higher reliability coefficient. Although the stability coefficient obtained in this study was inter- preted as moderate. the researcher is confident that one can general- ize from the score a student receives at one time to what he/she would Obtain had the test been administered at another time. 2. Pearson product-moment coefficients of stability and equivalence obtained on parallel forms of the machine trades assessment test averaged 0.70. which was interpreted as strong. Typically. vocational/occupational teachers do not prepare alternate forms of achievement tests. However. Michigan State University's vocational/occupational test-item bank was used to pro- duce parallel forms of assessment tests from occupationally specific 84 subbanks. Seemingly. tests developed with items selected at random from the same subbank (of similar items) should be equivalent and shOuld yield stable scores. This was the case with the machine trades assessment tests. Estimates of stability and equivalence are usually lower than stability estimates alone (Mehrens & Lehmann. 1984). This was not the case with parallel forms of the machine trades test. Based on this finding. the researcher concluded that examinees were as challenged in responding to Form 82 of the machine trades assessment test as they were in responding to similar items on Form A2. There- fore. the estimates of stability and equivalence are evidence that. given a different but similar set of items asked at a different time. students would obtain similar test results. 3. The tetrachoric correlation coefficient of stability obtained on students' subscore rankings (pass/fail) on the machine trades assessment test was 0.56. which was interpreted as mod- erate. Again. as occurred when stability coefficients on the machine trades assessment test were computed. similar results were obtained when the students' subscore (duty area) rankings were used. The small population. as well as the relatively small number of items represent- ing each subtest (duty area). may have caused the stability coeffi- cients to be spuriously low. With a larger group. the tetrachoric correlation might have been larger. resulting in a “stronger“ inter- pretation. Seventy percent of the machine trades students' subscore rankings remained consistent from one occasion to the next (test/retest). The tetrachoric correlation was significant at the .001 level. Based on this finding. the researcher is confident that 85 one can generalize from the ranking (pass/fail) a student obtained on one occasion to what ranking he/she would obtain had the test been administered at another time. 4. The tetrachoric correlation coefficient of stability and equivalence obtained on students' subscore rankings (pass/fail) on parallel forms of the machine trades assessment test was 0.33. which was interpreted as weak. Two factors cited in the literature as influencing test/retest reliability coefficients were (a) student answers on the retest are not totally independent of their answers on the initial test. and (b) general lack of interest on the students' part may make the retest a poor measure of his/her knowledge (Ebel. 1979). The tetrachoric correlation statistic used to obtain estimates of reliability (stability and equivalence) on subscore rankings obtained using parallel forms of the machine trades assessment test may well have been affected by both factors. Therefore. the reader is advised to interpret this finding regarding stability and equivalence of the machine trades subscore rankings with caution. Overall. the results were statistically significant at the .001 level. and it appears that. given a different but similar set of items asked at a different time. student subscore rankings would remain stable. 5. The chi-square value of 23.8 between students' percentile rankings obtained on the machine trades assessment test and the instructors' percentile rankings of students was signifi- cant at the .001 level and interpreted as strong. This result. however encouraging. must be interpreted with caution. More than one-half of the students (51%) obtained a rank below what their instructors had given them: however. only 21% of 86 those disagreed by more than 20 percentage points (see Appendix F. Table F.43). Based on this finding. the researcher concluded that there was little relationship between the test results (student percentile rank) and the criterion measure (instructors' rankings). In addition. this relationship between test and criterion measure was taken as evidence of the test's lack of concurrent validity. 6. The tetrachoric correlation between students' average subscore rankings (pass/fail) on the machine trades assessment test (test/retest) and the instructors' subscore rankings was 0.37. which was interpreted as weak. One of the most common misinterpretations made in educational research is to confuse statistical significance with practical significance (Borg & Gall. 1983). Although statistically significant results were obtained at five of the six machine trades test sites (A. C. D. E. and F). there seemed to be little relationship. at the duty level. between instructors' rankings of students' knowledge (criterion measure) and students' actual rankings on the machine trades assessment test. The rankings achieved on the test matched the rankings provided by the instructors only 30% of the time. It is conceivable that the machine trades instructors selected students' rankings based on their observation/knowledge of the students' ability to perform machining tasks (psychomotor skill development). rather than on the students' knowledge of machine trades practices and proce- dures (cognitive development). The evidence indicated that the tests did not have concurrent validity. 87 Mechanical Drafting 1. Pearson product-moment coefficients of stability averaged 0.57 for the mechanical drafting assessment test. which was inter- preted as moderate. Three of the four mechanical drafting test sites (I. J. and K) had results that were statistically significant at the .05 level. It is conceivable that the low reliability estimate at Site H could be explained. in part at least. by the lack of variance in the students' test scores (70% of the students' scores ranged between 20 and 29 points). Eliminating Site H from the mechanical drafting group resulted in a Pearson product-moment stability coefficient of 0.74. which was interpreted as strong. Based on this finding. the researcher concluded that one can generalize from the score a student receives at one time to what he/she would obtain had the test been given at another time. 2. Pearson product-moment coefficients of stability and equivalence obtained on parallel forms of the mechanical drafting assessment test averaged 0.47. which was interpreted as moderate. Although statistically significant results were obtained at three of the four mechanical drafting test sites (I. J. and K). the stability and equivalence estimates were considered moderate because of a very poor correlation at Site H (r = 0.07). If Site H had been eliminated. the correlations would have been “stronger.“ Also. estimates of stability and equivalence are usually lower than either stability or equivalence estimates alone (Mehrens & Lehmann. 1984). This was the case with parallel forms of the mechanical drafting assessment test. Therefore. the estimates of stability and 88 equivalence indicated that. given a different but similar group of items administered at a different time. students would obtain similar results. 3. The tetrachoric correlation coefficient of stability obtained on students' subscore rankings (pass/fail) on the mechanical drafting assessment test was 0.87. which was interpreted as strong. As occurred when stability coefficients on the mechanical drafting assessment tests were computed. similar but “stronger“ results were obtained when the students' subscore (duty area) rankings were used. Eighty-one percent of the mechanical drafting students' subscore rankings remained consistent from one occasion to the next (test/retest). The tetrachoric correlation was significant at the .001 level. The researcher concluded that one can generalize from the ranking (pass/fail) a student received on one occasion to what ranking he/she would receive had the test been administered on another occasion. 4. The tetrachoric correlation coefficient of stability and equivalence obtained on students' subscore rankings (pass/ fail) on parallel forms of the mechanical drafting assessment test was 0.53. which was interpreted as moderate. The students' subscore rankings on parallel forms of the mechanical drafting assessment test (test Form A2/retest Form 82) were consistent 69% of the time. Overall. the tetrachoric correlation was statistically significant at the .001 level. Thus. the researcher is confident that. had a similar yet different set of items been asked at a different time. students' subscore rankings would have remained stable. 89 5. The chi-square value of 14.1 between students' percentile rankings obtained on the mechanical drafting assessment test and the instructors' percentile rankings of students was not significant and was interpreted as weak. Overall. 69.1% of the rankings obtained by students on the mechanical drafting test differed from the rankings provided by their instructors. These results seem to indicate that instructors who teach and assess students primarily in the psychomotor domain may be ill-equipped (poor criterion measure) to rank or assess students in the cognitive and affective domains. Based on this finding. the researcher concluded that there was little relationship between the test results (ranking of students by percentage rank) and the criterion measure (instructors' rankings of students). The small relationship (between test and criterion measure) was taken as evidence that the test lacked concurrent validity. 6. The tetrachoric correlation between students' average subscore rankings (pass/fail) on the» mechanical drafting assessment test and the instructors' subscore rankings was 0.49. which was interpreted as moderate. Again. a common misinterpretation is to confuse statistical significance with practical significance (Borg & Gall. 1983). Although statistically significant tetrachoric correlation coefficients were computed using the combined mechanical drafting data. only one mechanical drafting test site (Site I) had a significant tetrachoric correlation between students' actual subscore rankings and their instructors' subscore rankings of them. It is conceivable. as with the machine trades group. that instructors selected test items based on their observation or knowledge of 90 students' ability to perform drafting skills. rather than their students' knowledge of drafting practices and procedures. Including more test items in each subtest (duty area) might have increased the test's content validity. However. the results indicated moderate agreement between the students' subscore rankings (the test) and the instructors' subscore rankings of students (the criterion measure). Based on the evidence. the researcher concluded that the test had little concurrent validity. Synthesis Although findings and conclusions were presented separately for each occupational specialty. the reader should note the similarities between the reliability and validity estimates of the assessment instruments developed for machine trades students and those developed for mechanical drafting students. The coefficients of stability and of stability and equivalence obtained on the occupational assessment instruments were statistically significant for both groups. Additionally. coefficients of stability and of stability and equivalence using the subscore rankings were also statistically significant for each group. The chi-square test indicated a statistically significant relationship between the machine trades students' percentile ranking obtained on the tests and their instructors' rankings of them. However. the chi-square test using data collected from the mechanical drafting students and their instructors failed to produce a 91 statistically significant result. It is possible that the mechanical drafting instructors ranked students according to their mechanical drafting skills rather than their cognitive understanding of drafting practices and procedures. The tetrachoric correlations between the students' subscore rankings and their instructors' rankings indicated that the tests had concurrent validity. Although the results of the tetrachoric correlations were statistically significant for both occupational specialties. the researcher believes the results lacked practical significance. It is conceivable that the significance resulted from the relatively large N (where N = students multiplied by the number of duty areas tested) rather than from any significant relationships between the two variables under investigation. Recnmmendations for Further Research Based on the findings of this investigation. tests developed using the Michigan State University vocational/occupational test-item bank seemed to possess reliability and validity. The item-bank method of developing vocational/occupational assessment tests should be investigated further with regard to reliability and validity. Tangential research topics include the following: 1. A similar investigation could be conducted for the remaining occupational specialties identified by V-TECS. 2. The present study could be replicated with a larger sample of vocational/occupational students and instructors. 92 3. The possibility of cross-walking all state vocational/occupa- tional duty and task listings to develop national listings incorporat- ing all common and geographically specific duties and tasks for each V-TECS occupational specialty could be studied. Epilogue For the past 18 months. the researcher has been involved in an investigation of vocational/occupational testing. specifically. tests developed at Michigan State University to assess vocationall occupational students in the areas of mechanical drafting and machine trades. Although the present study was focused on the reliability and validity of these tests. as an educator the researcher cannot overlook the people who are directly and indirectly affected by this assessment process. With regard to assessment. two major observations stood out as paramount after the study was completed. First. a large percentage of instructors overrated and/or underrated their students' knowledge of the subject matter. In some cases. more than 50% of the students were ranked higher by their instructors than they actually achieved on the tests designed for them. The researcher has observed labs and classrooms designed primarily for the “performance“ of job-related duties and tasks. not necessarily the cognitive development of the concepts related to those practices and procedures. As educators call for the rebirth of “problem-solving“ skill development rather than learning by traditional methods. perhaps vocational instructors are placing too much emphasis on “projects“ and psychomotor skill 93 development and paying too little attention to the broad scope of knowledge (math/science) encompassed in each occupational specialty. The second observations concerns the notion that a “core“ curriculum currently exists within each occupational specialty. This belief has not been supported by research. Actually. a vast diversity exists in the course content and equipment used to deliver what is theoretically the same vocational program (program title) within the same state. and in some cases the same city. Even programs that appear to be similar (if not identical) because of equipment and local labor market needs are assessed in different areas with varying numbers of items. The content (curriculum) is determined by each individual instructor. not by a statewide committee or by common state curriculum guides. Finally. testing is an interaction between the instructor and the students he/she instructs. The result should determine. in prescriptive terms. the area(s) in which improvement is required so decisions can be made regarding instruction. curriculum. equipment. and so on. A test alone should nut be used for decision making without supporting data. especially if the “cost“ of making an incorrect decision is high. APPENDICES APPENDIX A VOCATIONAL PROGRAM EVALUATION MODEL 94 Vocational Prolran Evaluation Model Secondary VocatxonaI Proarana Proceaa tvaluatton L Product Evaluation Long-lance PIannIna Y Proarae Inoroveeent Increased vocational enrollment. student occupattonal coeoetence. and related pIacenent. APPENDIX B MECHANICAL DRAFTING AND MACHINE TRADES TASK LISTINGS FROM THE V-TECS 955 480100Rv MECHANICAL DRAFTING IV) 02/01/85 A06 SEE CIPS:150202. 10802 48010; DOT 007. 181-010 - 0 AFT R. MECH NICAL THIS LISTING HAS DEVELOPED FOR THE VOCATIONAL-TECHNICAL EDUCATI N C NS RTIUM F STATES BY PERSONS IN THE STATE OF INDIANA. O O O O DUTY AND TASK LISTING DESIGNING ASSEMBLY TODLING ASSEMBLY MACHIN AUTOMATIC PARTSE HANDLING MACHINE CRIMPI NG DIE O C C C C O O O 2222!: P O n > 4— z; 6') —1|_ X— .a C :1 III Fl IXTURE STUD DRIVING FIXTURE NING CUTTING DIES N CUTOFF DIE N TRIMMING DIE NING GAGES N FLUSH PIN GAGE N PLUG GAGE N PROFILE GAGE NING MOLDS N DIE CAST MOLD N INJECTION MOLD NING SPECIALTY FIXTURES GRINDING FIXTURE INSPECTION FIXTURES ABRASION TEST FIXTURE CKING (GAGE) FIXTURE 0X FiXT URES ONIC CHECK FIXTURE c FIXTURE E R a "B n e O I" a ....-.-.-.—.-.-. M MMMMMMMU‘IMM III UNII tn IIIU'IUI III UIU'I In IIIU'III'IU'IUIUI 61096190630619 9 0‘) CI 9610 O on O 096396 rrrrrrrrrrrrrrrrrrrr r zzzzzzzzzz C C E L 0 AT FIXTURES B R IN H I FIX XTURES E LO ECT PPI R EPA NIC HELD FIXTURE INGLE PARTS H O L A V R O G S AST MACHINING PART E PED PART X D O O O A d "| ommouu-uuu— M ”-0 O u”... I O O O O O O O I I O O l O O O S DETAI I D m .4 S DEDP ART NG MACHINED PART PART AX CASTING SgR§T(VACUUM) CAST PART DUI MOLDED PART METALTPART p T S )DOU‘IOHMM M 0 SINGLE PART P I IP NG CONTOURING TEMPLATE RUBBER MOLDED PART DRAHING LAYOUTS ORAN DETAIL ASSEMBLY DRAW EXPERIMENTAL LAYOUT ASSEMBLY N—a—e—e—e—e—d—e—e—e OODOOOOOODODOOODO MMMMMMMMMMMMMMMMM 44444444444444444 >>>§>>>D>D>Db§>>>b N—c n ”MNWN-PQDMWINN— 96 A80100Mv MECHANICAL DRAFTING (V) 02/01/85 407 a. DRAW EXPERIMENTAL LAYOUT SUB-ASSEMBLY . DRAW INSTALLATIONA 2. DRAW LAYOUT ASSEMaL . DRAW LAYOUT SUB-ASSEMBLY &. DRAW PART IDENTIFICATION SSEMOLY . ORAw SPECIFICATION (OUTLINE) ASSE SEMOLY 3. DRAW SUB-ASSEMBLY 1 . DRAW WORKING ASSEMBLY 11. DRAW COMPRESSOR ROOM LAYOUT H. DRAWING CHARTS 1. DRAW BAR CHARTSRT 2. DRAW SECTORC 3. DRAW SEQUENCE HOF TOPERATION CHART I. DRAWING DIAGRAMS l. DRAW ELECTRICAL SCHEMATIC DIAGRAM 2. DRAW FLOW DI GRAM 2. DRAW HYDRAULIC SCHEMATIC DIAGRAM . DRAW PNEUMATIC SCHEMATIC DIAGRAM 5. DRAW WIRING DIAGRAM J. DRAWING LAYOUTS OF SINGLE PARTS DRAW CAST MACHINING PART DRAW DEVELOPED PART DRAW DIE CAST PART 0 O N ) I 1) r > XWN-NMNNN-fi e eeeeeee eeeee O z , I 1 C DRAW LAYOUT OF PRINTED CIRCUIT BOARD DRAW RUBBER MOLDED PART DRAWING MACHINE DIAGRAMS DRAW ELECTRICAL SCHEMATIC DIAGRAM OF MACHINE DRAWING PICTORIALS DRAW ASSEMBLY PERSPECTIVE DRAW SINGLE PART ISOMETRIC DRAW SINGLE PART PERSPECTIVE MAINTAINING AND CARING FOR TOOLS IT HEIGHT OF DRAFTING ARM OF ELBOW TYPE DRAFTING MACH NE GHT OF DRAFTING ARM OF TRACK TYPE DRAFTING MACH NE f‘ .n e e M tn NE TaACK TYPE MACHINE 1) m>->>)onm S N R TN NG MACH ON L A P PEN RINT FILE MAINTAIN COMPUTER REPRODUCING DRAWINGS FILE PRINTS MAKE MICROFILM APERTURE CARD OF DRAWING MAKE MYLAR PRINT OF DRAW NG MAKE PRINT FROM APERTURE ICARD ddddd‘ O O O O I O O O O I O O O O I z mmw—ommcmwn— 3 DEN- WM— 0 O O O 97 LBOIOOMV MECHANICAL DRAFTING (V) 02/01/85 L08 2. MAKE sEPIA PRINT OF DRAWINGS . RUN OLUEPRINT OF DRAWING 7. PRODUCE DRWING 0N PLOTTER O. TROUBLE-SHOOTING AND CORRECTING DESIGN ERRORS l. TROUBLH-SHOOT CAUS- OF DIE COMPONENT BREAKAGE 2. TROUBL.'SHOOT CAUS. OF FAILURE OF PARTS TO BE REMOVED FROM THE DIE z. TROUBL.-SHOOT CAUS. OF FAILURE OF PARTS TO BE REMOVED FROM THE FIXTURE . TROUBL.-SHOOT CAUS. OF FAILURE OF SCRAP TO BE REMOVED FROM THE DIE g. TROUBL.'SHOOT CAUS. OF FIXTURE COMPONENT BREAKAGE . TROUBL.'SHOOT CAUSr OF IMPROPER FIXTURE CLEARANCES a. TROUBLI-SHOOT CAUS. OF IMPROPER PART LOCATION IN FIXTURE . TROUBLE-SHOOT CAUS; OF INCORRECT PUNCH AND DIE CLEARANCE 3. TROUBL.'SHOOT CAUSr OF MISFITTING DIE COMPONENTS 1 . TROUBL.-SHOOT CAUS. OF MISFITTING FIXTURE COMPONENTS ll. TROUBL.'SHOOT COMPUTER PROGRAM 12. TROUBL.-SHOOT MOLDING PROBLEMS 9E) L80500Mv nACHlNE SHOP (v) 02/01/85 L51 SEE IPS: B80 OOT 280-0 23— MACHINIST O.E. CODE - 17.2302 THIS LISTING WAS DEVELOPED FOR THE VOCATIONAL-TECHNICAL EDUCATI N C N RTI M F STATES BY PERSONS IN THE STATE OF GEORGIA O O SO U 0 DUTY AND TASK LISTING PERFORMING SUPERVISORY FUNCTIONS COORDINATE ‘WORKERS WITH WORK TO BE DONE DE ERMIN NEA VAILABILITY OF SUPPLIES AND MATERIALS T MAINTAIN FILES AND ORDER AND RECEIVE STOCK SSLOH-UP ON END PRODUCT QUALITY CONTROL STANDARDS UP P ERVISE MACHINE USE SFVMSCH MAERTENANCE OF SHOP SAFETY EQUIPMENT. SAFE OPERATIONS AND SET NVENTORY SUPPLIES AND MATERIALS PERFORMING MATHEMATICAL CALCULATIONS APPLY DIMENSIONS OF PARTS FROM DLUEPRINTS 0R SPECIFICATIONS To JOB CALCULATE AMOUNT OF MATERIAL TO BE RE Mov E0 TO OBTAIN CORRECT LIMITS FOR RE ”OR CALCULATE CHANGE GEARING FDR THREADI NC CALCULATE CONVERSION OF REVOLUTIONS PER MINUTE (R PM) TO SURFACE FEET PER MINUTE SFPM CALCULATE DIMENSIONS OF SLOTS AND GROOVES ON SPECIAL SETUPS CALCULATE GEAR BLANK SPECIFICATIONS FOR IND CALCULATE MACHINE SPEEDS AND FEEDS BY FORMULASN CALCULATE STOCK UTILIZATION IN MACHINE WORK CALCULATE TOLERANCES 0R ALLOWANCES FOR PROPER FITS CONVERT To METRIC MEASUREMENT DETERMINE CLEARANCE. RELIEF AND RAKE 0F CUTTING TOOL S DETERMINE MATERIAL STRENCTH' ACCORDING TO STANDARD RAW STOCK SIZES TAKE MICRDMETER READIN USE MACHINING HANOBOOKS. CHARTS AND TADLES T0 AIO IN MATHEMATICAL CALCULATIONS DESIGNING AND PLANNING MACHINE WORK MAKE SKETCHES OF PARTS TO BE MACHINED F S S U I UNNNNd, on e eeeaoe NN-‘mWNmN-fi eeeeeoeeee ea dd—d—O n e PERFORM LAYOUT FOR PRECISION MACHINE WORK USING LAYOUT INSTRUMENTS PERFORM LAYOUT FOR PRECISION MACHINE WORK USING MILLING MACHINE . INSPECT. REMOVE. AND REPLACE PART(S) FOR REPAIR OR MACHINEW mo . REFER TO TECHNICAL ORDERS. STANDARDS. AND SPECIFICATIONS FOR MACHINE WORK . TEST FOR HARDNESS PERFORMING METALWORK OPERATIONS CLAMP WORK IN HOLDING DEVICE CUT META STOC aegRICAEET sngKAL METALWORKING TOOLS OPERATE CYLINDRICAL GRINDER OPERA TE HON PERFORM BENCH FILING POLISH ME AL USING DEPTH GA AGES MEASURE WITH DIAL TEST INDICA CHECK LAYOUT WITH GA GE BLOCKS DETERMINE ACCURACY OF PRECISI MEASURE WITH HEIGHT GAGES MEASURE WITH SINE BAR PERFORMING BENCH WORK CUT MATERIALS WITH HAND HACKSAWS CUT THREADS WITH HAND TAPS CUT THREAD DIES DISASSEMBLE AND ASSEMBLE PARTS HOME AND LAP SURFACES TOR ON MEASUREMENT INSTRUMENTS WITH GAGE BLOCKS O O C O O I I O O I O O O O —ddd— \flWN—P M wu—mmmwua O ”NH-- S99 NBOSOOMV MACHINE SHOP (V) OZ/OI/BS L52 HAND SHARPEN CUTTING TOOLS WITH ABRASIVE STONES REAM HOLES WITH HAND REAMERS REMOVE AND REPLACE HELICAL COIL WIRE SCREW INSERT (STI) REMOVE DAMAGED SCREWS AND OTHER NON-HARDENED THREADED HARDWARE WORK AND SHAPE METAL OPERATING DRILL PRESSES CENTER PUNCH HOLE LOCATI CONSIDER MATERIAL OF WOR COUNTERBORE TO DEPTH SPE COUNTERSINK H DETERMINE H DRILL HOLE d ”WNNN-P ‘ about» N PI IF WNG CE TO ED IN REQUIR —11XC3 E I GRINDING WHEEL EE HAHA=D gGAINST GRINDING HFOR DRILL PRESS OPERATION $§|EEILL PRESS SPINDLE ROPER FEEDS AND SPEEDS FOR OPERATION PERFORMED HMENT TO TAPH HOL E BLUEPRINT SPECIFICATIONS USE AUTOMATIC FEED ON DRILL PRESS OPERATING GRINDING MACHINES d dddd—‘dd—J .41 O)— D WNWN—o O O O O I C M m _g , 'U '0 I. ATTACH AND ALIGN MATERIALS FOR GRINDING OPERATION 2. BALANCE GRINDINGW HEEL z. CUT OFF OR PART MATERIALS WITH GRINDING MACHINES . DRESS AND TRUE GRINDING WHEEL S 2. INSPECT GRINDING WHEELS . MEASURE. INSPECT. AND REWORK WORKPIECE ON GRINDING MACHINES g. PERFORM GRINDING MACHINE HOPERATIONS AS PER SETUP . POLISH WITH GRINDING 8. SELECT AND SET SPEEDS nANDl FEEDS OF POWERA FEED GRINDERS I . SET UP AND PERFORM SURFACE GRINDING OPERA ONS II. SET UP GRINDER AND SHARPEN PLAIN MILLING CUTTERS 12. SET UP. GRIND. AND SHARPEN PRESHAPED LATHE TOOLS I2. SET UP GRINDER AND SHAPE CHISELS I . SET UP GRINDER TO RUN WORKPIECE BETWEEN CENTERS I2. SET UP GRINDER TO RUN ON MAGNETIC CHUCK I . USE UTILITY GRINDER H. SERVICING LATHES I. ALIGN LATHE CENTERS USING APPROXIMATE METHOD 2. ALIGN LATHE CENTERS USING ACCURATE MEASUREMENT z. DORE HOLES WITH THE . COUNTERBORE HOLES WITH LATHE g. COUNTERSINK HOLES USING LATHE . USING TAPER ATTACHMENT. CUT LONG EXTERNAL TAPEREO SURFACE g. USING COMPOUND REST CUT SHORT EXTERNAL TAPEREO SURF ACES . CUT INTERNAL THREADS WITH LAT HE 3. CUT EXTERNAL THREADS WITH LATHE I . CUT INTERNAL TAPEREO 5 MR ACES II. DIE CUT THREA ADS WITH LATHE. HAND THREADING I2. OIE CUT THREADS WITH LATHE USING DIE HEADS I2. DRILL HOLES WITH LA ATH HE I . SET UP LATHE AND FA ACE WORKPIECE HELD IN A CHUCK I2. MEASURE STOCK I . PERFORM CONTOUR. AN GULAR. OR RADII CUTS WITH LATHE Ia. PERFORM LAT FILI G I . PERFORM LATHE FILING TO DEDURRP RT I3. PERFORM SPINNING OPERATION USING FORMING TOOL 2 . REAM HOLES WITH LATH ET 2I. RECHASE THREADS ON LATHE 22. ROUGH CUT AND FINISH CUT HITH LATHE 22. KNURL PARTS WITH LATH HE 2 . SECURE TOOL HOLDER FIXTURES OR ATTACHMENTS 25. SELECT AND SET SPEEDS AND ME Ds 1(30 beasOOMv MACHINE SHOP (V) oz/OI/85 A53 26. 333 L. \DWN-‘WMNNN-PWMNNN-fi XNW‘WN-v o 0.0.... ZUN-‘IUN-‘I—NWNH 0.. 0 on. o 000 o .00... NNWN-n SET UP ENGINE LATHE SET UP TURRET LATHE FOR OPERATIONS TAP THREADSW HL ATHE SET UP TOOL POST GRINDER PERFORM GRINDING OPERATION OPERATING MILLING MACHINES ALIGN MIL ALIGN MI E L 3 xxx—nnbhnn X-E—IIU L WORK ' G MACHINES M L l I S L N G G 000’ 0000'! warm 55 m—rn 40m L CHINE OIL LLI ML UH UH Y I A R I M K A 332m14000r PERATIONS ATIONS NS ON THE HORIZONTAL MILL OS MILLING WORK mmvvvvvvv33333-00n mmmmmmmm—-—-zcfl rarrnxrzm-r-I—r-I—m-o— 22c"- ”moo Dawn-0):!) — —I 2 D O S UARE UP METAL USI OPERATING POWER SAWS CUT AND WELD BANDSAW BLA MEASURE MATERIAL AND CUT DETERMINE TYPE OF MATERI REMOVE AND REPLACE SAW B SAWING WITH A METAL BAND D EEDS ELECT APPROPRIATE BLAD OPERATING PRESSES SET UP AND PUNCH MATERIALS WITH PRESS SELECT ACCESSORIES AND ATTACHMENTS FOR PR SET UP PRESS AND ASSEMBLE PARTS WITH PRES SET UP PRESS AND DISASSEMBLE PARTS WITH P STRAIGHTEN MISCELLANEOUS PARTS WITH PRESS SET UP. REMOVE. AND REPLACE PARTS WITH PR OPERATING SHAPERS . ALIGN SHAPER ATTACHMENT. WORKPIECE. AND CUTTING TOOL SELECT AND SET SPEEDS AND FEEDS OF SHAPE S SELECT. SHAPE. AND SHARPEN CUTTING TOOLS FOR SHAPER OPERATION PERFORMING PRODUCTION MACHINIST LINE REMOVE AND INSTALL PINS REMOVE DEFECTIVE SPLINES AND REPLACE WITH NEW SPLINES REMOVE FROZEN OR SEIZED PARTS MAINTAINING MACHINES AND TOOLS INSPECT AND CHANGE DRIVE PULLEYS OR BELTS LEAN AND STORE HAND TOOLS. CUTTERS. FIXTURES. JIGS. OR ATTACHMENTS STALL. LEVEL. AND FASTEN DOWN MACHINES MOVE MACHINE PARTS PLACE AND ADJUST MACHINE PAR TS SPECT AND REMOVE. REPLACE. OR ADJUST MACHINE GUARDS RAPE AND PAINT MACHINES FOR CONTOUR SAWING WITH POWER HACK SAW ES OPERATIONS mmxlnm mmnmm N E E N C 'IOI AaosoOMv MACHINE SHOP (V) 02/0I/85 ASA NNNN—n—I—a—o—Od—o‘d- \hNN—o O UHU—PONDWW‘NN-PONDO PAIR HgND TOOLS FORM MAINTENANCE ON BANDSAW ECT COOLANTS CUTTING OILS. OR COMPOUNDS FOR MACHINING OPERATION AN ARBOR PRESgE “VIM S SAFE OPERATIONAL CONDITION R D TI S O SAFE OPERATIONAL CON I ON TAINING SHOP FACILITIES AND WORK AREAS OSE OF SCRAP METAL CHIPS OR SHAVINGS. AND TRASH OR WASTE MATERIALS ORM CUSTODIAL TASK N AND MAINTAIN WORK AND ADJACENT WORK AREAS T FACILITIES P AND CLEAN SHOP FACILITY m2"‘ 2 V? APPENDIX C SAMPLE MECHANICAL DRAFTING AND MACHINE TRADES TEST BLUEPRINTS 'IOZ INSTRUCTOR'S NAME SCHOOL/CENTER A ADDRESS CITY/ZIP TEST DATE: ESTIMATED DATE TEST WILL BE RECEIVED IN RESOURCE OFFICE AT MSU: INSTRUCTIONS: REQUEST TEST ITEMS AT THE FULL TASK LEVEL, DUTY LEVEL, OR INDIVIDUAL TASK LEVEL BY INSERTING THE REQUESTED NUMBER OF ITEMS IN THE CORRESPONDING BOX. FOR HELP, SEE ATTACHED EXAMPLE. ETASKLIST ID: “OOIOOMV FCCHANICAL DRAFTIFB (Copyright - 3-16-8‘0) A. DESIGNING ASSEMBLY TODLING 1. Design Asseebly eachine 2. Design automatic parts handling euchine 3. Design Crieping die a. Design locating fixture 5. Design Welding fixture 6. Design stud driving fixture TOTAL FOR DUTY: A B. DESIGNING CUTTING DIES I. Design cutoff die 2. Design trio-ing die TOTAL FOR DUTY: B C. DESIGNING BABES DU DUDE DDUDDDUU I. Design flush pin gage R. IO. II. IE. 13. I“. IS. 16. 17. IO. 19. 20. :° pa 9 UUDUUDDUUDDD DUDUDUUDUUDDU an ya a 103 Detail Machined part Detail plastic Molded part Detail powered eetal part Detail purchased part Detail revised part Detail sand casting part Detail sheet eetal part Detail tabulated part Detail single part Detail piping Detail contouring template Detail rubber eolded part TOTAL FOR DUTY: F DRAWING LAYOUTS Drew Draw Draw Draw Drew I] ii p E H. 1. DUI] 2. detail asseebly experieental layout asseebly experimental layout subasseebly installation assembly layout asseebly layout subasseebly part identification asseebly specification (outline) asseebly subasseebly working asseebly coepressor rooe layout DUTY: G DRAWING CHARTS Drau Draw bar charts sector chart INSTRUCTOR'S NAME SCHOOL/CENTER ADDRESS CITY/ZIP TEST DATE: 104 ESTIMATED DATE TEST WILL BE RECEIVED IN RESOURCE OFFICE AT MSU: INSTRUCTIONS: REDUEST TEST ITEMS AT THE FULL TASK LEVEL, DUTY LEVEL, OR INDIVIDUAL TASK LEVEL BY INGERTING THE REOUESTED NUMBER OF ITEMS IN THE CORRESPONDING BOX. FOR HELP. SEE ATTACHED EXAMPLE. ETA-(LIST ID: «Doom mount on (Copyright - 1240-74.) A. I. E. 3. 6. 5. 6. '7; B. J. E. [II] [Inn [II] [1000080 6. PERFORMING SUPERVISORY FUNCTIONS Coordinate Mortars eith Work to be done Detereine availability of supplies and eaterials Maintain files and order and rOCeive stock Follou-up on end product quality control standards Supervise sechine use Suoervise eaintenance of shop safety eouioaent. safe operations and set up of sachinery - Inventory supplies and eaterials TOTAL FOR DUTY: A PERFORMING MATHEMATICAL CALCULATIONS Aoply dieensions of parts fros blueprints or specifications to Job Calculate asount of aaterial to be rseoved to obtain correct lieits for resort Calculate change gearing for threading Calculate conversion of revolutions per sinute (RPM) to surface feet per einute (SFPHI UDUDDUDUU UUUDDDUEI [I DUUUUDDUUD 6. 7. B. R. 10. 11. IE. 13. I“. 105 Calculate dieensions of slots and grooves on special setups Calculate gear blank specifications for indexing Calculate eachine speeds and feeds by foreulas Calculate stock utilization in eachine uork Calculate tolerances or alloeances for proper fits Convert to eetric eeasureeent Detereine clearance. relief, and rake of cutting tools Detereine aaterial strength according to standard ran-stock sizes Take eicroeeter readings Use eachining handbooks, charts, and tables to aid in eatheeatical calculations TOTAL FOR DUTY: B DESIGNING AND PLANNING MACHINE WORK Make sketches of parts to be eachined Perform layout for precision eachine work using layout instrueents Perfore layout for precision eachine eork using eilling eachine Inspect, reeove. and replace partIs) for repair or eechine sork Refer to technical orders. standards. and specifications for eachine nor Test for hardness TOTAL FOR DUTY: C D. I. E. 3. 6. '6e 7. PERFORMING METALWORK OPERATIONS Clasp eork in holding device Cut eetal stock Fabricate special eetaleorking tools Neat treat eetal Operate cylindrical grindeR Operate hone Perfore bench filing Polish eetal [IIJDUUDDDUI] DUDDDDDDUDDD [IUDIJDDD 0 e 10. II. IE. 13. IA. ‘106 Using depth gages Measure with dial test indicator Check layout with gage blocks Detereine accuracy of precision eeasureeent instrueents with gage blocks Measure with height gages Measure with sine bar TOTAL FOR DUTY: D o‘ O a PERFORMING BENCH WORK Cut eaterials with hand hacksaws Cut threads with hand taps Cut thread dies Disasseeble and asseeble parts Hone and lap surfaces Hand sharpen cutting tools with abrasive stones Raae holes with hand reaeers Reeove and replace helical coil wire screw insert (STl) Reeove daeaged screws and other nonhardened threaded hardware Work and shape eetal TOTAL FOR DUTY: E F. I. E. 3. A. OPERATING DRILL PRESSES Center punch hole location 'Consider eaterial of workpiece to be drilled Counterbore to depth specified in blueprint Countersink hole to drawing requireeents Detereine holding technique Drill hole to size Mount and secure work in holding device Reae hole to size Sharpen drill using grinding wheel APPENDIX D CORRESPONDENCE 'IO7 MEMORANDUM TO: Student Assessment Advisory Committee/Pilot Site Contacts FROM: Chrisbua Olson and Steagn C. Clark RE: Core Curriculum DATE: February 18, 1986 Some of you have expressed an interest in the capability of comparing your student assessment test scores with those of similar students throughout the state. The primary problem is that there is no guarantee that all students have studied the same material and developed competencies in the same tasks. As you know, Michigan does not have a 'core curriculum' but rather curriculum guides that allow teachers to Choose the tasks to be included in the program with input from local resources. In an attempt to assist those that would like to compare their students with others in like program areas, we would like to survey participating teachers to identify the tasks taught in their program. The Curriculum Resource Team will compare responses to identify those tasks taught in every program. To assist us, would you please identify a teacher (or teachers) in each of the following program areas who will be nominated to participate in the 'student assessment pilot test' in April, May and June of this year. He will then send them a task list and request that they 'highlight‘ the tasks taught in their programs. He will then be able to compare the results and report back to you at a later date. He would like to have the responses back from teachers by the end of February. You do not need a person in each category, only those who you wish to participate in the pilot testing. Thank you for your assistance. 108 Nominations for Participation in Student Assessment Pilot Test 2:99:99 Cashier/Checker Cosmetology Data Processing Diesel Mechanics Electronics Food Preparation Horticulture Machine Trades Mech. Draiting Nurse Assisting Helding Return to: Chris M. Olson, Ph.D. 102 Hills House Michigan State University -DR- East Lansing, MI 48824 Phone (in Michigan): 1-800-831-7523 Iggghg[-Nggg (Address if diiferent than yours) Steven Clark, Research Assistant 102 Hills House Michigan State University East Lansing, MI 48824 109 MICHIGAN STATE UNIVERSITY CURRICULUM RESOURCE TEAM DEPARTMENT OF AGRICULTURAL AND EXTENSION EDUCATION COLLEGE OF AGRICULTURE AND NATURAL RESOURCES TELEPHONEISVI 353““ March 4, 1986 Dear You have been identified as a potential participant in a student assessment pilot test. by . Your participation will depend on the avail- ability Of test items. and you will be contacted concerning this later. A first step in the process. however, requires us to identify any connonali- ties among programs in different areas Of the state. Would you please highlight on the attached task list(s). those tasks taught in your program. YOur response by March 17 will be appreciated. Sincerely, fl. M54... Chris M. Olson. Ph.D. Project Director and Associate Professor CC: Mack Seney Steve Clark MSU Curriculum Resource Team 101 Will: House Michigan State Univemw East Lansing. Muchugan 0624-1050 I'IO MICHIGAN STATE UNIVERSITY CURRICULUM RESOURCE TEAM DEPARTMENT OF AGRICULTURAL AND EXTENSION EDUCATION COLLEGE OF AGRICULTURE AND NATURAL RESOURCES TELEMONE(517)353-M61 April 22. 1986 T0: Pilot Site Instructors FROM: StevefiyEAark. Research Assistant RE: Development of Test Fonn Al/Bl (Directions) Enclosed please find a copy of the V~TECS duty and task list for your program. As discussed during the meeting last month, you are to choose from the task listing. the duties and tasks that most correctly parallel your curriculum content. A test will be developed based on this infor- mation (Form A1 and parallel Form 81) and sent to yOu. Please include the number of items you wish for each duty area. I will include an add- itional 25% for each area. than you will be able to select only those items which you want to see on your final test (Form A2 and 82). Form A1 and 81 come in two parts; (1) the test questions and, (2) the answer key. Fonn Al/Bl will contain at least 25% more items than you wish on you final test (Form AZ/BZ). 1 Read each of the test questions carefully. Determine whether on not the content of the question reflects content taught in your curriculum. 2. Check the answer key carefully. Make sure the answer is the correct one. 3. 0n the answer key. circle those questions YOU WISH TO DELETE. This is local selection (validation) of test content. You will have a test that reflects y0u local curriculum and is content valid. 4. If you should delete more items than the 251. additional items may be added. Contact me as soon as possible. 5. Return the answer key (with the questions circled that you wish deleted)to me for modification into Fonns A2 (test) and 82 (retest). 6. Indicate the number of students that will be tested so I will have enough answer sheets with me on test day. Thank-you. Call if you have any questions. I will see you on your scheduled test day. MSU Cumculum Resource Team 101 Wills House MICHI'IR Slate Unnvemlv [an LIIIIIII', Michigan “.2610” III MICHIGAN STATE UNIVERSITY unnmmnvcoulnmmiowlmmusanlnmunuo aun-enemas-Immsouv-aumsua. uumnwsumuasnmmmmi auiuwmummuutnlmnumwo mn ”was April 23. 1986 Mr. Steven C. Clark Counseling. Educational Psychology and Special Education Erickson Hall Dear Mr. Clark: Subject: Proposal Entitled. ”Testing the Reliability and Validity of Occupational Competency Assessment Instruments Constructed Via Random Selection of Previoggly Validated Test Items" I am pleased to advise that I concur with your evaluation that this project is exempt from full UCRIHS review. and approval is herewith granted for conduct of the project. You are reminded that UCRIHS approval is valid for one calendar year. If you plan to continue this project beyond one year. please make provisions for obtaining appropriate UCRIHS approval prior to April 23. 1987. Any changes in procedures involving human subjects must be reviewed by the UCRIHS prior to initiation of the change. UCRIHS must also be notified promptly of any problems (unexpected side effects. complaints. etc.) involving human subjects during the course of the work. Thank you for bringing this project to my attention. If I can be of any future help. please do not hesitate to let me know. Sincerely. / C.— Henry E. Bredeck Chairman. ”CRIBS HEB/jms cc: Dr. Chris M. Olson mb-WWWW' APPENDIX E SAMPLE INSTRUCTOR RANKING SHEETS FOR MECHANICAL DRAFTING AND MACHINE TRADES NZ 2!. 829.. .82.... .88 . ounce 039:0 255.5 he use»: 395 . {on 3550 395 . 9.8 ~38. a8...» :95 . use... 9.32.0 ocma 395 . {on 83.6.. :95 . ton 0320.59 395 . ts. 839.. 0:8... .68 . 9.8 3:28! :95 . 9.8 v2.2.8... 9.3.8.. :95 . anon ammo 3o tek— . PNM e-c-s- fiPNMIU‘OFQO‘e 5.33.. 9.2:. 395 .m 9.8 08325.. :95 . 5320 «.3333 03.2558 395 .3 fun 9320mm game 395 . 5.3:. 3.29.8 232.5. .38 .m £5... mo...» .6 use»: 9.5.3 . mill—U 5.32.. so: :95 .N 3....— mo...— ..ucwmao Sassoon 33.583 395 .p «56.... 2.3.8 .. VIII—“HH— ta 838 .88. :38 .8 :a... a»... 828.8 2.380 :88 .9 ER... :38 .2 9.2.0 c0398.? ..o 353...» 395 .m 9.8 035.. :38 4.. face .308 .68 .~ “Me. 78233 :38 .o. 3.5.... can :95 .p 9.. ~30... coon» :38 .9 E55 23...... ... Hill—[IL .28 9.328 Ba... :38 .3 do.“ one; 9.8 33.6.. :38 .9 ts. v8.2.9.2. :38 .~. .50?— loo. 3339.8 sub .2 to; ~38. veteran :38 .2 24:32. 2.3.5: :95 .2 9.3 830... 033:. :38 .9 .3532...» .58 .o 9.8 8:28.. :38 .a >353: 2.53:3 5.3023an :95 .m 9.3 ammo Ages: 0.5395 so“ :38 .o £288.. 838:382 ta .58 g ts. 9.326 :5 .8. :38 ... 39.6333 .503. :95 .0 fan 9.3.8 338 .0 39.603 030...: :95 .m acme 005:8... 9.3.3.. 338 .m 32.93 832.32.. :95 ... ts. 83.»... :38 .. 3965326 .5022 incestuaxu no.5 .m foo ammo out :38 .m 39.53.. 0:32 Emcee—Loo; :95 .N foo “xx—cumin. =38 .N .358. :33 .58 .. .2... 8228.. .80 :38 .. $8.: 2.2.... .o _ _.. _ mt... No...» 23:8 .. EU So... n2... 3...... one; wcwzc FEE 3.3 E .8.» 3.6 8.3» s... 55... .83.. to Sm ..o 9.39. a 33:8 33.382... 3 cmoo—xocxanEn 3.8.... 3030.. go: meow eucugon 0:...an €2.03»qu an SHE ¢ .8 gain 60.... 3.6 033» 05 55:. .23: .5 mom .0 9.3.... a 26an 3:...nnoocsn 3 omoo.so§\n:§n .339... 30838 coco—San 39.3.. 62.03950 .3 «mm: ¢ .8 mega ll :8 a .8. :28 B... 3.. 2a .38 a .8 28. .x... 8m 8.83.8 so 9:3 .sfifixiq 8582:... .52. 8 82.. :8 .o 8a.. 35.... s88...” 2.» B 8838.. 38.88.}23a s... x8e... 88:. .gngfimfi >592... "coco 5.30.... .008 aoocom .88 .8852. .683... €83» guzw zomhusmzm ggfigu Egg .5ng 05.53. .54: 113 0.0.2.0 0:0 050 50» .0.— 33 3:05 copuom «Rio... omno: omsom 9.0 a _ a _ _ b u .026. 5.32.8.3... :0 50.3 .002 :0. 2:9. .8030 35 .00.. 8.. 5...: S .920 00380.8. .8 05 .85 0005300.. 03000200 .50» :0 0000: .030330300165. .0 0000—03000 :0 33000 0000: 00.. .00000 .80! 5 .0... 00003 .00000 000008 0 00500 50> 5 000.3 .0000: 35.. 00:00:00 5:0 00305 0000: 0 5030.00 5005...: . 0:. 05.003: 50050: . :00 5.030030. c008 . .3 .0223 00:03 . 05:00.. 0&0 0.00.0 :0 03: 000: 00m . 05:000 0.50.0.0 0005.03 .0. .0000... =0 0530.0 02.00... .0 .358 08. 3...... .a 05.5.0 .o 0580:... .5. .0 00305.0 :005 .m 3530.0 .0 05.... 0300 0x3. .m 000000 0008 4. 0.00 0.50.000 05.... 05.0 0.3.. .0 00:05.00... 35000.0 .508 .0 0530.0 .00 05.0 .03- 93. .m 00.00: 0006 .m 0530.0 .0 0.00 0.50.000 50.0.00. 00...: .~ 0.8.. 3... 08.. 05.80... 5:. .. 3...... 0:. .. 8.8... 03:... .m 853...... 02.0805... ... H 05:00.. 30... 000.. 050...... 0...... 38.0 .o I... 9.3.0... .o 05.0.. 003.3 .~ 03.0828 0...... 305.. .68 .0 05:00.0 00.00305 0.0.. 0350 30.5 .~ 35000.0 0050 30:00 .0 I5 05000.0 .00 0:50: 000.0: .— 030080.00 3030000 30.: .— 58.. S... 05...... e... 02.2.4055. .z DU 538...... 0.0.5... J H :0... 000.. :0... 000.. 3.: E5 3:: Ex— 114 .0: 050 :0.) 0.5000: .3 00?» 0:50: :0 0: 0.5000: .9 5000... 000:0 000 3.0: .9 8.8... 0000 0...? 3005.00... 0.0.6.0.. 0008...... _ 0005.00.00... 05300.0 00 000.5000 05.0.0000 .N. 00:00.0:000 .0:00 000 030.00 0000.50 000.00: .0 00.003 000w :00: 00000: x005 .2 0.005 30.00 05: :00 500:0: 000000. 000 000.000 .o .000505 0000 50.0 :0... 0.0000: .3 0.0.50. 000: :05... 0000: 50¢ .0. 0030 :0000 00: .0 00030 0:00.00 :00... 3000 950050 000.0:0 00a. .0 .30... 00:9. .0 8000.30 0: 08 08.. .m 00:: 0000: 0.00.00 .0. 00.00 03.5000 000 03.500005 .0 000: 000.000 .0 .0000 000: :05... 0000.00 08 .m .08.... Eurasia 0.0.08 .m 000... 000.. 5... 008.... 06 .0 0000... 000.0 000: .0 0300300: 000: :05... 0000.000... 05 .— 38.. 003.330... .2800 38..."... .m 00.0.. :90... 0205.00. .0 H 0.0000 5000... 05 .m :0... 0000 00.8.. 00.39. 0. 0:9. 0.5.0 .— mzogzro 0.55:0: 02.50.00. .0 H 08.0.5. .8 .00. .0 :0... 0000 3.03 05:00... .00 005000250000 0050030500 500000550... 5 000 50.000000 6.00.0 000.5000 00 .0000. .m 0.0 00 00300 000 .00.0:0 60.000000: 055002. 00... .3 0.0: 05:00... .0 005000.. 00005.50. 0x0... .9 5000.. .00 3.0.00 00030.. 000 .0500. .000005 .0 0050 0.0000 :0. 05:00... @0530... 0.000000 00 050.0000 59.0.00 000.000... 05.0.0000 .m. 050: 0.9. 05:00... 00.05000 .00 00000” 0.00.00 .m 05000 000005.005 0:000— w500=0 00 0x0. 000 .0020. .000000050 050.0000 .2 050: 3.9. 05:00... 05500.0 .00 000.000 0.00.00 .N 000005000... 05.00... 00 0.0.08 .0— 005.000... 0: 00 00.00 .00 00:000.... 003.. .0 B: .0080 .8. 0800.620 .0 82.0.0.3 0.0.8.3 .0 0.0... 02.00.: 0.0225. 02.. 02.28000 .0 H 0.00... 05:00... 5 00.000300... 30000 0005:0000 .o :0... 0000 00:250.. .3 00000000 000000 05:00... 000500500 .0. 050005 .00 005000200000 x003 .00» 0003055 6 0000.000... 000 00:00.0 >..00:0>0H .0. 000000 50.0000 ~0:05:00... .0 0: 000 0:0 000000.000 0.000 00 0.0.60.0 000 00050 .0 000000050 0000:0000 .m 5:005:00 000000 00:0 .00 000000050... 003.008 .0 3.00%. 0005... .00 000.. 0000.00 00 $.05 00: 05:03. 000.0000.“ .m 00:5... .00 0050530. .0 050.0250 000—0055 ... 00.000000 30030.. .0.— 0005— 000..00 500:0 00.0000 0030.6 00000.0 000 :0 03:30:00 .0 00 00.6.5. 00 00 05.000... .00 000000 0003000“. .m «.0000 03000. 000 000.0 000 00:.— 5005£ .m w5000..:0 .00 w5.00u 0w:0:0 0001.55 .N 20:00! 0000:0 00“. 00 0000000200000 000 00:00.00 00 32305000 005300000 .N .0 005.005: 00.. 00.00 .0 000500050 .303 .F 0:00 0: 00 3.03 :0? 0.00.9. 00050.08 .— 02805085 05:50.5: 02.0.5.0... .0 fllL I _ 0820...... .80....an 00.08.00. .0 H :0... 0000 :0... 0000 05.5 >50 mfiz: Eb: LII]! .111! .00.0 00:0 000000 0:0 5:00: .050: .0 new 00 0500. 0 0.0200 03000000000 00 0000030050330 000000 000030.000 0000 00001.00 03000.. 60050002... .5 3:..— < .6 man—Em In 60.0 >000 000000 5:00: 00:3: .0 Sm .0 0500. 0 03300 3 0000000000 00 0000;050:003 :9600 000000000 00:00:00 008.50.. .00050003 an in: < .8 gm ill ill 4.0M. 0 .00 3000: 000 was 000 .0000 0 .00 0.600 0:0 wow 05.00000 0:0 050: 30000000.. 03000750 .000 00 9000: :0... .0 0000 .0050. 0000300 0:0 .3 000000000 000016050330 0:0 0.00:0 000000 .guhgga >508: .008 00.0.0... .0000 “00:00 .0000 3000.505 ".00: 00.0000” Emzw zo—hounufi cohgfimzm ham—Emma: Hagan “2.5m... 81:0 115 00... 0.000 2.... 00.00 300.. 00 0.0000 . 0000 »0.0.>.0 00.00 300-. 00 000000 . 0...! 0.0:... .o. 88. 0.... 0.00..» .3 0.... .800 . 00.000... 9.2:... 3.000.000 000 00 000 00 000200000 »02:0 0.000000 0.000000 . 000200000 0 09.00.0000 0000000 . 000.000000 »0...000 0.000000 . 000.000000 »0...000. 0.000000 . »0......_ .0000 0.000000 . 000.00.000 00.000020 0.000000 . 00.2.0 000 0000.00 . 00000000000 000 00000.0 9:00 000.0 30.000. 2.: . 0.00» 2.: . 0.03 30.000200 2.: . 00.000 30.0000 00 2.: . 0.»:0 00 2.: . x003 2.... 0000.950 000000. . 00.0000 »02:... 020000 000 00 0000. .03 . 8.....8... 05...... 0.... 8.9. 2.... . .20.. 30.0000 08 . 30.0000 0.6.00. 00 0000 . 80.2... 5...... o. 0.8 . 000.000... »0.2.... 00.... 00.00 0.00 . x00: 2.0 0000.050 0.0.0003 . 00000000000 00.000... »02:... 0»..0 . 00.00.: 00.000... »0.2.0 0»..¢ . 002.000: 02...... 00.0.5.0... . 0000..» >0...00 000 . 00000 0.000»0e 00 00. 00 0000.0» 00 000 . 0000000 0003000 000.0000: 000 00 0000..» 00 000 . 0.00.00 00000 000 0000..» 00 000 . 0.000 0000. 000000000 000.000 000 .00..» . 00 000 . 000.00.000 »0.00..» 0000.00 0000.00 000 00 000 . 0000000 »0....E 0.0.0 000.000 000 0000..» 00 000 . 00000..» 0000 .0300 00 00000 000 000000 000 000 000.00 . 2.0.02 050...... .0... 5...... . 00000 .00 00 000.00.000 00.000... »0.00..» 5000.00 . 000 .000... 00.00..» 00 000.0000: 0.0030. 000 .000000. .000000: . 0.00.0. »0.00.0» 000000. . 0.0003 »0.00.0» 0000 000 00000 . 000.000... 00.00..» 00... 0.0.0000... 0.00 .0 000 08 . .000) 00.00..» 0000.3 . 000200000 9300..» .00 03.0000... 5.20 000 000000 . 002.000: 02.02.00 02.0.0095 . math NNN PNMRU‘ONQO‘RPN 0-0-0-0-0-0-0-0-0- NN H—NMBID‘DNDO‘E UFNMRU‘O FDO‘ .HWHU 0000 003000000 2.00.5 0.000000 . 0000.0» 0000 .000 .0. 000 . 0...... .0... 008.... 0.... . 000200000 000 0000. 000.50 E. 000 . 0...... 0......» .5 .00 . 800.... 0.... 008. .8 0.... 08.00 . 38.08.... .3 .80.... .83.... .8. 0.080 . 0...... .0... 00...... .05. . 0...... .0... .8 .00... 0.... .8 00...... . 00.... ..o 2.00.... 89.8.. . 0...... o... .0... 8.9. 8.... . .000 00.300 »...00 00.000000 »0.00.00 3.00.00 . 0...... 0...... 008...... . .0... 0.5.00 0. 0...... 0...... E80... . 0000. 00... 0000 2000 00 .00.:000 3500000 0000000 . 0.0000 0.0.008. . 08.... a c. 0.0.. 08.9.09. 8... 0.... 6...... .5 .00 . 0...... o... 0.... 8.0.. 3.... . 0.00.. o... 0...... 0...... 0.... 0.09.... .8 o... . 0.080.... 0...... 6...... .0... 080.... .8 u... . 008...... .0080. 02.0.... .8 . 0...... 0.... 089.... .2000... .8 . 0...... 0.... 0.00.... 0.0.0.... .8 . 00000.50 0000000 3000000 00000 000 .0000 0.50960 »0.0.. . 00000000 0000000 3000000 »..0. 000 3.0010300 00000 00.0: . 0...... 0...... 9...... 4:20.58 . 0000. 00... 00.00 0000.00.58 . 0000. 000 00... 00.00 0.8 . 0000000000... 00000000 9.30 0000000 0000. 0020 . 000000 000580000 00.00 0.00000 0000. 0»..< . 95.0.5003 UPS . H 0000 000.0 2.00 00 0000 0.008000 00... . 0.2.8.030... 00.00:... c. 0.0.. 00. 0...... . 0.9. a... o. .858... 0.0.0... 8.. . 000.0009. 0000008 . 00800.00 00.000000 .00 000000 000 00000 000000 000 00000 2.00 000 . 2.00 0.00 00 00003 000.00 . 0.00.00 000.0 2.00 0. 000000 00 .000 0.5000 . 00.000000 00000 2.00 .00 000000 00 .000 000.00 . 000000000 000 30000 000 .00.! 0.00:... .20.... 0...... 8... .3 2.... 00......0 . .00.! 9:00.00 »0.00 2.00 00000.0 00.0 00 0.00 .80.. . 00200 00.0.00 0. .000: 0.5000 000 0.0.9. . 00.0 00 0.00 2.00 . 83.08. 8.0.9. 8.30.8 . 000050.000. »0.3000 00 0.00 0.0.0.0058 . 00.0000... 0. 00.0.0000 00000 00 00000000000 . 02...... 0.. o. 08.9....) .0 3.00.0... 000.08 . 00.0000. 0.00 00.50 000000 0000000 2.00 2.0.5.000 In. P’NOVI'IDONO H 20... 000... H 20... 000.. H6 .foto v5 83 .50» new :8 3:05. oopuom 2.:8 omnc: onus Sac ~_____ a 42:: 233033305. co 333 Ring :2 3:9. ecu—25o 25 H08 :0» so?! 5 3:95 vuuufioiv .89 05 x820 gag—Dug“. $300215 .59» :0 323 .denxmmvwgofi ho 5.23:qu :o .233» v32 ac: nvnao seal 5 o..- ouuagu .833 09608 a 3:33 393 :83 van 302m . 2.3.203 05mm . 33a {oz anyone? tea 3.8: 5358 !B :38 . 9.93 33326 58.5.. . 33.35.“... 32! Lo snub new 69:55 go 320 :32. amgon no macaw—a . m5: 55: 92 3:383... 82m Pin 055523: . .0683 33.50.59: 0038 . 3039.80 magma L0,. 308 new 3.8% 98 new 307% . avg: 89:3 3 :8 23 ~38. a 5:. 9:38 . 383 33 00:30.. was .265: . 333 on 3 “3:39: .3 0&3 veg-:38 . 53025038 Nmsm 9:33 .5350 m.- ~uam unma— 5? 3238. to “.8 E. 3233 :58: . .8. £85 3 883 2882 3o: 8.. a6 . ”MD ml» 5.5: 235.95 . II «PS8 .59» 5 ova...» .533 3c: 3:093: 3...» 0335 one»: o N MEMO h 3.."— no!— APPENDIX F DATA SUMMARY TABLES 117 Table F.1.--Vocationalloccupational assessment test results, by site: machine trades, test Form AB. Cumulative Cumulative Scores Frequency Frequency Percent Site 50 - 60 I 1 I 10 I 100 A 40 - 49 I 4 I 9 I 90 30 - 39 I 3 I 5 I 70 20 - 29 I 2 I 2 I 80 Site 80 - 90 I 4 I 37 I 100 B 70 - 79 I 0 I 33 I 89 6O - 69 I 3 I 33 I 89 5O - 59 I 0 I 31 I 84 40 - 49 I 4 I 31 I 34 30 - 39 | 13 I 37 I 73 30 - 29 I 15 I 15 I 41 Site 60 - 70 I 3 I 35 I 100 C 50 - 59 I 4 I 23 I 88 4O — 49 I 8 I 18 I 73 3O - 39 I 3 I 3 I 12 Site 60 - 70 I 8 I 21 I 100 D 50 - 59 I B I 19 I 90 40 - 49 I 10 I 11 I 58 30 - 39 I 1 I 1 I 5 Site 70 - 80 I 1 I 15 I 100 E 60 - 69 I 3 I 14 I 93 50 - 59 I 0 I 11 I 73 40 - 49 I 7 I 11 I 73 30 - 39 I 3 I 4 I 27 30 - 39 I 1 I 1 l 6 Site 60 - 70 I 8 I 32 I 100 F 50 - 59 I 13 I 14 I 64 4O - 49 I 3 I 8 I 9 Site 50 - 60 I 1 I 3 I 100 B 40 - 49 I 3 I 2 I 67 Table 118 F.2.-Vocational/occupational assessment test results, by site: machine trades, retest From 82. Cumulative Cumulative Scores Frequency Frequency Percent Site 50 60 I 1 I 5 I 100 A 40 49 I 2 I 4 I 80 30 39 I 2 I 2 I 40 Site 50 60 I 1 I 21 I 100 B 40 49 I 1 I 20 I 95 30 39 I 5 I 19 I 90 20 29 I 3 I 14 I 67 10 19 I 1 I 1 I 5 Site 50 60 I 4 I 12 I 100 C 40 49 I 2 I 4 I 33 2O 29 l 2 I 2 I 17 Site 60 70 I 2 I 11 I 100 D 50 59 I 4 l 9 I 82 40 49 I 4 I 5 I 45 3O 39 I 1 I 1 I 9 Site 60 70 I 2 I B I 100 E 50 59 I 1 I 6 I 75 40 49 I 1 I 5 I 63 30 39 I 1 I 4 I 50 20 29 I 2 I 3 I 38 10 19 I 1 I 1 I 13 Site 70 80 I 1 I 13 I 100 F 60 69 l 4 I 12 I 92 50 59 I 6 I 8 I 62 40 49 I 1 I 2 I 15 30 39 I 1 I 1 I 8 Site 60 70 I 1 I 2 I 100 B 50 59 I 0 I 1 I 50 40 49 I 0 I 1 I 50 30 39 I 1 I 1 I 50 119 Table F.3.-Vocational/occupational assessment test results, by site: machine trades, retest Form A2. Cumulative Cumulative Scores Frequency Frequency Percent Site 40 - 50 I 4 I 5 I 100 A 30 - 39 I 1 I I 1 I 20 Site 80 - 90 I 2 I 16 I 100 B 70 - 79 I 0 I 14 I 88 60 - 69 I O I 14 I 88 50 - 59 I 0 I 14 I 88 40 — 49 I 2 I 14 I 88 30 - 39 I 1 I 12 I 75 20 - 29 I 11 I 11 I 69 Site 50 - 60 I 5 I 13 I 100 C 40 - 49 I 4 I B I 62 30 - 39 I 4 I 4 I 31 Site 60 - 70 I 1 I 10 I 100 D 50 - 59 I 1 I 9 I 90 40 - 49 l 7 I 8 I 80 30 - 39 I 1 I 1 I 10 Site 50 - 60 I 3 I 7 I 100 E 40 - 49 I 1 I 4 I 57 30 - 39 I 3 I 3 I 43 Site 60 - 70 I 6 I 9 I 100 F 50 - 59 I 2 I 3 I 33 4O - 49 I 1 I 1 I 11 Site 40 - 50 I 1 I 1 I 100 120 Table F.4.-Machine trades summary, all sites: test Form A2. Cumulative Cumulative Scores Frequency Frequency Percent Sites 80 - 90 I 4 I 133 I 100 A - B 70 - 79 I 1 I 129 I 97 60 - 69 I 18 I 128 I 96 50 - 59 I 26 I 110 I 83 40 - 49 I 37 I 84 I 63 30 - 39 I 26 I 47 I 35 20 - 29 I 21 I 21 I 16 Table F.5.-Hachine trades summary, all sites: retest For m A2. Cumulative Cumulative Scores Frequency Frequency Percent Sites 80 - 90 I 2 I 61 I 100 A - 8 70 - 79 I 0 I 59 I 97 60 — 69 I 7 I 59 I 97 50 - 59 I 11 I 52 I 85 40 - 49 I 20 I 41 I 67 30 - 39 I 10 I 21 I 34 20 - 29 I 11 I 11 I 18 Table F.6.-Hachine trades summary, all sites: retest For m B2. Cumulative Cumulative Scores Frequency Frequency Percent Sites 70 - 80 I 1 I 72 I 100 A - G 60 - 69 I 9 I 71 I 99 50 - 59 I 17 I 62 I 86 4O - 49 I 13 I 45 I 63 30 - 39 I 13 I 32 I 44 20 - 29 I 17 I 19 I 26 10 - 19 I 2 I 2 I 3 121 Table F.7.-Vocationalloccupational assessment test results by site: mechanical drafting, test Form A2. Cumulative Cumulative Scores Frequency Frequency Percent Site 30 - 40 I 9 I 39 I 100 H 20 - 29 I 27 I 30 I 77 10 - 19 I 3 I 3 I 8 Site 50 - 60 I 10 I 21 I 100 1+ 40 - 49 I 8 I 11 I 52 30 - 39 I 3 I 3 I 14 Site 50 - 60 I 2 I 18 I 100 J 40 - 49 I 3 I 16 I 89 30 - 39 I 11 I 13 I 72 20 - 29 I 2 I 2 I 11 Site 70 - 80 I 3 I 19 I 100 K 60 - 69 I 6 I 16 I 84 50 - 59 I 9 I 10 I 53 40 - 49 I 0 I 1 I 5 30 - 39 I 1 I 1 I 5 + Adjusted scores Table F.8.-Mechanical drafting summary: all sites, test Form A2. Cumulative Cumulative Scores Frequency Frequency Percent Sites 70 - 80 I 3 I 97 I 100 H - K 60 - 69 I 6 I 94 I 97 50 - 59 I 21 l 88 I 91 4O - 49 I 11 I 67 I 69 30 - 39 I 24 I 56 I 58 20 - 29 I 29 I 32 I 33 10 - 19 I 3 I 3 I 3 Table 122 F.9.-Vocational/occupational assessment test results by site: mechanical drafting, retest Form 82. Cumulative Cumulative Scores Frequency Frequency Percent Site 30 - 40 I 5 I 21 I 100 H 20 - 29 I 14 I 16 I 76 10 - 19 I 2 I 2 I 10 Site 40 - 50 I 5 I 10 I 100 1+ 30 - 39 I 4 I 5 I 50 20 - 29 I 0 I 1 I 10 10 - 19 I 1 I 1 I 10 Site 50 - 60 I 1 I 11 I 100 J 40 - 49 I 5 I 10 I 91 30 - 39 I 4 I 5 I 45 20 - 29 I 1 I 1 I 9 Site 60 - 70 I 3 I 12 I 100 K 50 - 59 I 4 I 9 I 75 4O - 49 I 3 I 5 I 42 30 - 39 I 1 I 2 I 16 20 - 29 I 1 I 1 I 8 + Adjusted scores Table F.10.--Mechanical drafting summary: all sites, test Form A2. Cumulative Cumulative Scores Frequency Frequency Percent Sites 60 - 70 I 3 I 54 I 100 H-K 50 - 59 I 8 I 51 I 94 4O - 49 I 10 I 43 I 80 30 - 39 I 14 I 33 I 61 20 - 29 I 16 I 19 I 35 10 - 19 I 3 I 3 I 6 123 Table F.11-Vocational/occupational assessment test results by site: mechanical drafting, retest Form A2. Cumulative Cumulative Scores Frequency Frequency Percent Site 30 - 40 I 3 I 18 I 100 H 20 - 29 I 13 I 15 I 83 10 - 19 I 2 | 2 I 11 Site 50 - 60 I 7 I 11 I 100 1+ 40 - 49 I 3 I 4 I 36 30 - 39 I 1 I 1 I 9 Site 50 - 60 I 2 I 7 I 100 J 40 - 49 I 3 I 5 I 71 30 - 39 I 1 I 2 I 28 20 - 29 I 1 I 1 I 14 Site 60 - 70 I 6 I 7 I 100 K 50 - 59 I 0 I 1 I 14 40 - 49 I 1 I 1 I 14 +Adjusted scores Table F.12.-Mechanical drafting summary, all sites: retest From A2. I Cumulative Cumulative Scores Frequency Frequency Percent Sites 60 - 70 I 7 I 43 I 100 H - K 50 - 59 I 8 I 36 l 84 40 - 49 I 7 I 28 I 65 30 - 39 I 5 I 21 I 49 20 - 29 I 14 I 16 I 37 10 - 19 I 2 I 2 I 5 124 Table F.13.-Hachine trades mean score summary, all sites: test Form A2, retest Form A2. Avg. Test Avg. Retest Avg. Test-Retest Site Score Score Difference A I 42.2 I 43.6 I 1.4 B I 33.0 I 35.8 I 2.8 C I 43.0 I 43.8 I 0.8 D I 50.4 I 47.3 I 3.1 E I 47.1 I 50.1 I 3.0 F I 58.7 I 60.1 I 1.4 G I 48.0 I 42.0 I 6.0 Table F.14-Machine trades mean score summary, all sites: test Form A2, retest Form 82. Avg. Test Avg. Retest Avg. Test-Retest Site Score Score Difference tn'nmcanwb 36.0 45.6 43.3 49.1 46.7 57.2 50.0 J‘UlgUl-L‘I'UJ‘ 0% chm UIanJE>u3Lnru 125 Table F.15.-Mechanical drafting mean score summary, all sites: test Form A2, retest Form A2. Avg. Test Avg. Retest Avg. Test-Retest Site Score Score Difference H I 23.0 I 25.5 I 2.5 I I 50.0 I 49.4 I 0.6 J I 44.1 I 42.0 I 2.1 K I 64.0 I 63.7 I 0.3 Table F.16.--Mechanical drafting mean score summary, all sites: test Form A2, retest Form 82. Avg. Test Avg. Retest Avg. Test-Retest Site Score Score Difference H I 27.3 I 25.9 I 1.4 l I 45.8 I 38.6 I 7.2 J I 33.9 I 40.2 I 6.3 K I 57.5 I 50.5 I 7.0 126 subscore achievement rankings, by site and duty Table F.17.-Vocational[occupational assessment test/retest area: machine trades, test Form A2/retest Form A2. T/Rt Fail/Fail Frequency T/Rt Fail/Pass Frequency T/Rt Pass/Fail Frequency T/Rt Pass/Pass Frequency Duty Area 02405532 00020022 13010000 40120001 DEFBHIJS A Site 11 8 10 13 12 11 242324 344100 000110 CDEFHI Site 35605141 1 1 11202112 1.s.1.1.nv1.qup. 32426053 CDEFBHIJ C Site 0405331 1241413 2132053 7332313 DEFGHIJ D Site 32251422 22003011 21201031 02322313 CDEFSHIJ E Site 127 continued. Table F.17. T/Rt Fail/Fail Frequency T/Rt Fail/Pass Frequency T/Rt Pass/Fail Frequency TlRt Pass/Pass Frequency Duty Area 100633 108112 101021 696233 EFBHIJ Site 00110010 10000001 00000000 01001100 128 Table F.18.--Vocational/occupational assessment test/retest subscore achievement rankings summary, by site: nachine trades, test For: A2/retest Form 02. I I Test/Retest I Test/Retest I Test/Retest I Test/Retest IDutiesI Pass - Pass I Pass - Fail I Fail - Pass I Fail - Fail SiteITestedIFrequrncy (%)lFrequency (%)IFrequency (%)IFrequency (X) A ID-E-F-I I I I n=5 lG-H-I-I 8 (20%) I 5 (12%) I 6 (15%) I 21 (53%) IJ-S I I I I B IC-D-E-I I I I n=16IF-H-I I 2 (2%) I 12 (12%) I 17 (18%) I 65 (68%) C IC-D-E-I I I I n=13IF-G-H-l 30 (29) I 14 (13%) I 10 (10%) I 50 (48%) II-J I l I I D ID-E-F-I I I I n=10IB-H-I-I 22 (31%) I 16 (23%) I 16 (23%) I 16 (23%) I3 I I I I n=7 IF-G-H—I 16 (28%) I 10 (18%) I 9 (16%) I 21 (38%) n=9 IH-I-J I 29 (54%) I 5 (9%) I 7 (13%) I 13 (24%) G lC-D-E-I I I I n=1 IF-G-H-I 3 (38%) I 0 (0%) I 2 (24%) I 3 (38%) II‘J I I I I 129 Table F.l9-Vocational/occupational assessment test/retest subscore achievement rankings, by site and duty area: nachine trades, test Form A2/retest Form 82. T/Rt Fail/Fail Frequency T/Rt Fail/Pass Frequency T/Rt Pass/Fail Frequency T/Rt Pass/Pass Frequency Duty Area 11512420 21032002 12010010 11001123 DEFBHIJS A Site 11 10 12 15 13 14 9.9.6.9.1.1. 695365 200111 CDEFHI Site 15564643 71332411 nva611.1.1.1.6.4 6.Q.fiuphfi.1.1.a. CDEFBHIJ C Site 0111571 6.7.0f6AUn:.1 1.1.q.1.q.nvv. 7273322 DEFBHIJ D Site u. 02444243 20121210 23000203: 43323232 CDEFBHIJ E Site 130 Table F.19.-continued. T/Rt Fail/Fail T/Rt Fail/Pass T/Rt Pass/Fail T/Rt Pass/Pass Duty Areas 01111112 10011110 01100000 10000000 CDEFGHIJ B Site I3I Table F.20.-Vocational/occupational assessment test/retest subscore achievement rankings summary, by site: machine trades, test Form A2/retest Form BE. I Test/Retest I Test/Retest I Test/Retest I Test/Retest Dutiesl Pass - Pass I Pass - Fail I Fail - Pass I Fail - Fail TestedIFrequency (%)IFrequency (%)IFrequency (%)IFrequency (%) Site D—E-FI I I I A B-H-II 9 (23%) I 5 (12%) I 10 (25%) I 16 (40%) J-S I I I I Site C-D-EI I I I 8 F-H-II 5 (4%) I 34 (27%) I 12 (10%) I 75 (59%) Site C-D-El I I I C G-H-II 23 (24%) I 17 (18%) I 22 (23%) I 34 (35%) I-J I I I I Site D-E-FI I l I D G-H-II 26 (33%) I 16 (21%) I 19 (25%) I 16 (21%) J I I I I Site C-D-EI I I l E F-G-HI 22 (24%) I 10 (16%) I 9 (14%) I 23 (36%) I-J I I I I Site E-F-GI l I I F H-I-JI 36 (46%) I 13 (17%) I 18 (23%) I 11 (14%) Site C-D-EI I I I G F-B-HI 1 (6%) I 2 (13%) I 5 (31%) I 8 (50%) 132 Table F.21.--Vocational/occupational assessment test/retest subscore achievement rankings, by site and duty area: mechanical drafting, test Form A2/retest Form A2. Duty I T/Rt I T/Rt I T/Rt I T/Rt Area I Pass/Pass I Pass/Fail I Fail/Pass I Fail/Fail I Frequency I Frequency I Frequency I Frequency Site F I 0 I 0 I 0 I 18 H 8 I O I 0 I 0 I 18 J I 0 I O I 0 I 18 L I 0 I 0 I 2 I 16 H I 0 I 1 I 0 I 17 N I O I 2 I 2 I 14 F I 10 I 0 I 0 I 1 G I 9 I 2 I 0 I 0 Site J I 4 I 1 I 4 I 2 I L I 2 I 4 I 3 I 2 M I 6 I 2 I 0 I 3 N I 6 I 0 I 2 I 3 F I 0 I 0 I 0 I 7 8 I 2 I 3 I 0 I 2 Site J I 1 I 1 I 1 I 4 J L I 2 I 0 I 1 I 4 H I 3 I 0 I 2 I 2 N I 1 I 3 I 0 I 3 F I 7 I O I 0 I 0 G I 6 I O I 1 I 0 Site H I 4 I 0 I 2 I 1 K I I 6 I 0 I O I 1 J I 7 I 0 I 0 I O L I 7 I 0 I 0 I 0 M I 5 I 1 I 0 I 1 N I 0 I 1 I 2 I 4 133 Table F.22.--Vocationalloccupational assessment test/retest subscore achievement rankings, summary by site: mechanical drafting, test Form A2/retest Form A2. I Test/Retest I Test/Retest I Test/Retest I Test/Retest I Fail - Pass I Fail - Fail Testelerequency (%)IFrequency (%)IFrequency (%)IFrequency (%) Dutiesl Pass - Pass I Pass - Fail Site F-G-JI I I I H L-H-NI 0 (0%) I 3 (2%) I 4 (4%) I 101 (94%) Site F-G-JI I I I I L-M-NI 37 (56%) I 9 (14%) I 9 (14%) | 11 (16%) Site F-G-JI I I I J L-M-NI 9 (21%) I 7 (17%) I 4 (10%) I 22 (52%) Site F-G-HI I I I K I-J-LI 42 (75%) I 2 (4%) I 5 (9%) I 7 (12%) H-N I 134 Table F.23.-—Vocationalloccupational assessment test/retest subscore achievement rankings, by site and duty area: mechanical drafting, test Form A2/retest Form 82. Duty I T/Rt I T/Rt I T/Rt I T/Rt Area I Pass/Pass I Pass/Fail I Fail/PAss I Fail/Fail I Frequency I Frequency I Frequency I Frequency F I 0 I 0 I 1 I 20 8 I 0 I 0 I 0 I 21 Site J I 0 I O I 0 I 21 H L I 0 I O I 1 I 20 M I 0 I 5 I 0 I 16 N I 0 I 8 I O I 13 F I 2 I 0 I 6 I 2 8 I 3 I 5 I 0 I 2 Site J I 1 I 2 I 0 I 7 I L I 2 I 5 I 0 I 3 H I 2 I 3 I 1 I 4 N I 2 I 2 I 3 I 3 F I 0 I 1 I 3 I 7 G I 0 I 0 I 4 I 7 Site J I O I 0 I 3 I 8 J L I 2 I 0 I 3 I 6 M I 0 I 1 I 1 I 9 N I 1 I 1 I 1 I 8 F I 9 I 0 I 3 I 0 B I 5 I 1 I 5 I 1 Site H I 2 I 2 I 7 I 1 K I I 8 I 2 I 2 I 0 J I 7 I 3 I 2 I O L I 9 I 2 I O I 1 H I 3 I 4 I 3 I 2 N I 3 I 0 I 3 I 6 135 Table F.24.-—Vocational/occupational assessment test/retest subscore achievement rankings, summary by site: mechanical drafting, test Form A2/retest Form A2. All I Test/Retest I Test/Retest I Test/Retest I Test/Retest DutiesI Pass - Pass I Pass - Fail I Fail — Pass I Fail - Fail TestedIFrequency (%)IFrequency (%)IFrequency (%)IFrequency (%) Site F-G-JI I I I H L-M-NI 0 (0%) I 3 (2%) I 4 (4%) I 101 (94%) Site F-B-JI I I I I L-M-NI 37 (56%) I 9 (14%) I 9 (14%) I 11 (16%) Site F-G-JI I I I J L-M*NI 9 (21%) I 7 (17%) I 4 (10%) I 22 (52%) Site F-B-HI I I I K I-J-LI 42 (75%) I 2 (4%) I 5 (9%) I 7 (12%) H-N I I I I 136 Table F.25.-Vocational/occupational assessment test-retest subscore agreement and correlational data: machine trades, test Form A2/retest Form A2. DutyIPercent Agreement IPercent DisagreementlTetrachoric AreasIPass-Pass/Fail-FailIPass-Fail/Fail-Pass ICorrelation D I 80 I 20 I -- F4 E I 40 I 60 I --- Site F I 100 I 0 I 1.0 I A G I 40 I 60 I -- H I 100 I O l 1.0 I I 100 I 0 I 1.0 J I 60 I 40 I --- _ ____ §_-_1 _________ éQ___-___-i __________ flQ-_-__--_L__-_::::__ Avg.D-S I 73 I 27 I ---- C I 69 I 31 I --- Site D I 50 I 50 I --- B E I 63 I 37 I ---- F I 75 I 25 I .410 H I 88 I 12 I --- -1 I ____ 75 -1 __________ 35 _ I ’2::.. Avg.C-I I 70 I 30 I --- C I 85 I 15 I --- D I 54 I 46 I --- Site E I 77 I 23 I --- C F I 92 I 8 I --- G I 85 I 15 I ---- H I 85 I 15 I ---- I I 69 I 31 I ---- J I 69 I a; I -——;__ Avg.C-J I 77 I 23 I -- D I 70 I 30 I -- E I 70 I 30 I .607 Site F I 30 I 70 I -- D G I 70 I 30 I .549 H I 60 I 40 I --- I I 40 I 60 I .181 J I 40 I so I .39z__ Avg.D-J I 54 I 46 I --- 137 Table F.25.--continued. DutyI Percent Agreement IPercent DisagreementITetrachoric AreaIPass-Pass/Fail-FailIPass-Fail/Fail-Pass ICorrelation C I 43 I 57 I --- D I 57 I 43 I -- Site E I 72 I 28 I -- E F I 100 I 0 I 1.0 G I 43 I 57 I .151 H I 100 I 0 I 1.0 I I 43 I 57 I .151 J I 73 I 23 I _-é11__ Avg.C-J I 66 I 33 I --—- E I 78 I 22 I .574 Site F I 100 I 0 I 1.0 F 8 I 67 I 33 I -- H I 89 I 11 I -- I I 67 I 33 I .529 J I L 92 L ___ 33 I .529_ Avg. E-J I 78 I 22 I --- 138 Table F.26.-Vocationalloccupational assessment test-retest subscore agreement and correlational data: mechanical drafting, test Form A2/retest Form A2. DutyI Percent Agreement IPercent DisagreementITetrachoric AreasIPass-Pass/Fail-FailIPass-Fail/Fail-Pass ICorrelation F I 100 I 0 I 1.0 Site 6 I 100 I 0 I 1.0 H J I 100 I 0 I 1.0 L I 89 I 11 I -- H I 94 I 6 I --- ______ B-_-L- 78 I -----§§--------L----:::: Avg. F-N I 94 I 6 I --- F I 90 I 10 I --- B I 82 I 18 I --- Site J I 55 I 45 I --- I L I 36 I 64 I --- H I 82 I 18 I -- ______ fl---1---------§§--------L----_-----1§--------L---_::::_ Avg. F-N I 71 I 29 I -- F I 100 I 0 I 1.0 G I 57 I 43 I -- Site J I 71 I 29 I --- J L I 86 I 14 I -- H I 71 I 29 I -- N- I ----§Z_-------L __________ &§--_- I-- "'2:- Avg. F-N I 71 I 29 I -- F I 100 I 0 I 1.0 G I 86 I 14 I .607 Site H I 71 l 29 I -- K I I 100 I 0 I 1.0 J I 100 I 0 I 1.0 L I 100 I 0 I 1.0 M I 86 I 14 I --- N I 57 I 43 I ---- Avg. F-N I 88 I 12 I -- 139 Table F.27.-—Vocational/occupational assessment test-retest subscore agreement and correlational data: machine trades, test Form A2/retest Form 82. Duty I Percent Agreement IPercent DisagreementITetrachoric Areas IPass-Pass/Fail-FailIPass-Fail/Fail-Pass ICorrelation D I 40 I 60 I .262 Site E I 40 I 60 I .262 A F I 100 I 0 I 1.0 B I 20 I 80 I --- H I 60 I 40 I —-- I I 100 I 0 I 1.0 J I 80 I 20 I -- ------ §---L_----_---éQ- I - 40 _---L----::::- Avg. D-S I 63 I 37 I --- C I 62 I 38 I .137 Site D I 48 I 52 I --- B E I 57 I 43 I --- F I 76 I 24 I .291 H I 67 I 33 I --~ __----I---l --------- Zl------_-L ---------- €2--- I “" Avg. C-I I 64 I 36 I -—- C I 42 I 58 I --- Site D I 67 I 33 I .549 C E I 67 I 33 I .549 F I 67 I 33 I .475 G I 75 I 25 I .723 H I 58 I 42 I .140 I I 42 I 58 I .140 3 I 58 I -fl§-_------1--_--§21--- Avg. C-J I 60 I 40 I —-- D I 64 I 36 I --- Site E I 27 I 73 I .417 D F I 73 I 27 I -- G I 36 I 64 I .240 H I 72 I 28 I --- I I 82 I 18 I -- J I 27 I 73 I ---~ Avg. D—J I 54 I 46 I --- Table F.27-continued. Duty 140 I Percent Agreement IPercent DisagreementITetrachoric Areas IPass-Pass/Fail-Faill Pass-Fail/Fail-PassICorrelation C I 50 I 50 I -- Site D I 63 I 37 I ---r E E I 88 I 12 I -- F I 75 I 25 I -- G I 88 I 12 I -- H I 50 I 50 I --- I I 88 I 12 I --- J | 63 _I 37 I ---- Avg. C‘J I 71 I 29 I --- E I 85 I 15 I --- Site F I 62 I 38 I --- F B I 62 I 38 I .054 H I 39 I 62 I .342 I I 46 I 54 I .102 ____ J I 69 I _____ 31 I __::::_fl_ Avg. E-J I 61 I 39 I -- 141 Table F.28.-Vocational/occupational assessment test/retest subscore agreement and correlational data: mechanical drafting, test Form A2/retest Form 82. Duty I Percent Agreement IPercent Disagreement ITetrachoric Areas IPass-Pass/Fail-Faill Pass-Fail/Fail-Pass ICorrelations F I 95 I 5 I --— Site 8 I 100 I 0 I 1.0 H J I 100 I O I 1.0 L I 95 I 5 I -- M I 76 I 24 I --- N I 62 l 38 I ---- Avg. F-N I 88 I 12 I -- F l 40 I 60 I —-—- Site 6 I 50 I 50 l --- l J I 80 I 20 I --- L I 50 I 50 I -—- H I 60 I 40 I --- N I 50 l 50 I ---- Avg. F-N I 55 I 45 I —-- F I 64 I 36 I --- Site 8 I 64 I 36 I --- J J I 73 I 27 I --- L I 73 I 27 I --- H I 82 I 18 I --- N I 82 I 18 I ---- _ Avg. F-N I 73 I 27 I -—-- F I 75 I 25 I --- Site 8 I 50 I 50 I -- K H I 25 I 75 I .627 I I 67 I 33 I -- J I 58 I 42 I —--- L I 83 I 17 I -—- H I 42 I 58 I -- N I 75 I 25 I ---- Avg. F-N I 58 I 42 I --- 142 Table F.29.-Vocationalloccupational assessment test subscore aveage in relation to instructors' subscore rankings, by site and duty Area: machine trades. INSTRUCTOR RANKING------DUTY SUBSCDRE RANKING Fail/Pass I Fail/Fail Duty I Pass/Pass I Pass/Fail Area Frequency I Frequency I Frequency I Frequency 11945503 01122109.. 36033450 750008 IIIIIIII will!!! 620100.55 DEFGHIJS Site 214.6782 CDEFHI Site CDEFBH‘IJ Site 2006460 1.004000 N324657a “1105320 04462315 17852458 CDEFBHIJ Site 143 Table F.29-continued. Duty _-__1a§I&u§198_&eNELagzzzzzzgulx_§u§§QQB§_EeuuLa§ Area I Pass/Pass I Pass/Fail I Fail/Pass I Fail/Fail I Frequency I Frequency I Frequency I Frequency E I 19 I 3 I 0 I 0 Site F I 20 I 2 I 0 I 0 F 6 I 17 I 3 I 0 I 2 H I 3 I 4 I 5 I 10 I I 12 I 8 I 0 I 2 J I 16 I 5 I 0 I 1 C I 1 I 2 I O I 0 Site D I 1 I 2 I 0 I O B E I 0 I 3 I O I 0 F I 1 I 2 I O I 0 G I 1 I 2 I 0 I 0 H I 1 I 2 I 0 I 0 I I 0 I 3 I 0 I 0 J I 1 I 2 I O I 0 144 Table F.30.-Vocationa1loccupational assessment test subscore aveage in relation to instructors' subscore rankings, by site and duty area: mechanical drafting. Duty -___lN§IBQQIQE_EENELN§ ------ QUTY SQEEQQEE_BQNELE§_ Area I Pass/Pass I Pass/Fail I Fail/Pass I Fail/Fail I Frequency I Frequency I Frequency I Frequency m - ----- F I 0 I O I 0 I 39 Site 8 I 0 I 0 I 0 I 39 H J I 0 I O I 0 I 39 L I 0 I 0 I 2 I 37 M I O I 0 I 0 I 39 N I 0 I 0 I 0 I 39 F I 0 I 0 I 14 I 7 Site 6 I 0 I 0 I 15 I 6 I J I 5 I 9 I 2 I 5 L I 11 I 9 I 0 I 1 M I 9 I 11 I 0 I 1 N I 8 I 13 I 0 I 0 F I 0 I 0 I 1 I 17 Site 8 I 0 I O I 5 I 13 J J I 1 I 17 I O I O L I 5 I 5 I 0 I 8 M I 5 I 13 I 0 I 0 N I 3 I 12 I 1 I 2 F I 17 I 1 | 0 I 1 Site 8 I 4 I 1 I 12 I 2 K H I 6 I 8 I 2 I 3 I I 7 I 0 I 9 I 3 J I 14 I 1 I 3 I 1 L I 8 I 0 I 9 I 2 M I 15 I 4 I 0 I 0 N I 3 I 16 I 0 I 0 145 Table F.31.--Vocational/occupational assessment test subscore average in relation to instructors' rankings, summary by site: machine trades. _ lNSTRUQIQB_BQNElflE:::::::QQIX-§QE§QQEE-EENELNE- Dutiesl Pass/Pass I Pass/Fail I Fail/Pass I Fail/Fail TestedIFrequency(%)IFrequency(%)IFrequency(%)lFrequency(%) Site D-E-FI I I I A H-I-JI 19 (13%) I 24 (30%) I 9 (11%) I 28 (46%) J-S I I I I Site C-D-EI I I I B F-H-II E9 (13%) I 103 (46%) I 20 (9%) I 70 (32%) Site C-D-EI I I I C F-G-HI 51 (25%) I 59 (30%) I 29 (15%) I 61 (30%) I-J I I I I Site D-E-FI I I I D G-H-II 76 (52%) I 48 (33%) I 5 (3%) I 18 (12%) J I I I I Site C-D-EI I I I E F-G-HI 40 (33%) I 25 (21%) I 16 (13%) l 39 (33%) I-J I I I I Site E-F-BI I I I F H-I-JI 87 (66%) I 25 (19%) I 5 (4%) I 15 (11%) Site C-D-EI I I I B F-BrHI 6 (25%) I 18 (75%) I 0 (0%) I 0 (0%) I46 Table F.32.--Vocational/occupational assessment test subscore averge in relation to instructors‘ rankings, summary by site: mechanical drafting. ——-—-—_—~————_--—-—-—_‘--—__-—~—_~—n~——nn—-~——‘__-_—--——n— DutiesI Pass-Pass I Pass-Fail I Fail-Pass I Fail-Fail TestedIFrequency(%)IFrequency(%)IFrequencyIX)IFrequency(%) Site F-G-JI I I I H L-H-NI O (0%) I 0 (0%) I 2 (1%) I 232 (99%) Site F-B-JI I I I I L-H-NI 33 (26%) I 42 (33%) I 31 (25%) I 20 (16%) Site F-B-JI I I I J L-H-NI 14 (13%) I 47 (44%) I 7 (6%) I 40 (37%) Site F-G-HI I I I K I-J-LI 74 (49%) I 31 (20%) I 35 (23%) I 12 (8%) M-N I I I I I47 Table F.33.-Vocationalloccupational assessment test/retest, pass/fail achievement, for machine trades, all sites: test Form A2/retest Form A2. TEST A2 Pass Fail I I I R I 110 I 67 I E Pass I I I T I (26%) I (16%) I E I I I S I _ _ I________________I T I I I 62 I 189 A I I 2 Fail I (14%) I I I I I I I Table F.34.--Vocationalloccupational assessment test/retest pass/fail achievement for machine trades, all sites: test Form A2/retest Form 82. TEST A2 Pass Fail I I I R I 122 I 95 I E Pass I I I T I (25%) I (19%) I E I I I S I _ I I T I I I I 97 I 183 I B I I I 2 Fail I (20%) I (36%) l I I I I I I I I I I48 Table F.35.-Vocational/occupational assessment test/retest pass/fail achievement for mechanical drafting, all sites: test Form A2/retest Form A2. TEST A2 Pass Fail I I I R I 88 I 22 I E Pass I I I T I (32%) I (8%) I E I I I S l_______-______--l l T I 21 141 I A I 2 I I I I I I I I I I Fail I (8%) | (52%) I I I I I I Table F.36.--Vocationa1/occupational assessment test/retest pass/fail achievement for mechanical drafting, all sites: test Form A2/retest Form 82. TEST A2 Pass Fail I I I R I 61 I 46 I E Pass I I I T I (18%) I (13%) I E I I I S I I I T I I I I 53 I 188 I B I I I 2 Fail I (15%) I (54%) I I I I I I I I I I I49 Tables F.37-F.42.-Hachine trades student percentile rank (achievement) in relation to instructor's ranking of students: machine trades, all sites, in frequency and (percent). Table F.37-Site A Instructor Ranking 1 a 3 4 s _ I I I 1 I I I 1 I I I (10%)I I I I I I I I I I I I I 3 I I a I I I I (30%)I I Actual I I I I I _I Ranking I I I I a I I a I I I I (30%)I I I I I I I _ __I I I I I a I I 4 I I I I (20%)I I I_ I _--__-_l _____ I _ I I I I I I I 1 I 5 I I I I I (10%)I I I I I I Note: Percentage agreements = 30 Student achievement above rating = 0 percent Student achievement below rating = 70 percent Table F.38--Site B Instructor Rating 1 2 3 4 5 _ I I 2 I 2 I I 1 I 1 I I (5.4%)I (5.4%)I I (2.7%)I I I I I I I I 2 I 2 I4 I 1 I 1 I 2 I (5.4%)I (5.4%)I (10.8%)I (2.7%)I (2.7%)I Actual I I I I I I Rating I I 2 I I 4 I I 3 I I (5.4%)I I (10.8%)I I I I I I I I I I 1 I 3 I 3 I 1 I 4 I I (2.7%)I (8.1%)I (8.1%)I (2.7%)I I I I I I I I l I 3 I 3 I 2 I 5 I I I (8.1%)I (8.1%)I (5.4%)I I I I Note: Percentage agreements = 18.9 Student achievement above rating = 29.7 percent Student achievement below rating = 51.4 percent ISO Table F.39.--Site C Instructor Rating _____1 _______ 3 3 _-_‘1 ________ ii _____ I I 2 I 3 I I I 1 I I (8%) I (12%)I I I I I I I _ l_ __I I 1 I 1 I 1 I I I 2 I (4%) I (4%) I (4%) I I I Actual I ________ I I _I _ I ______L Rating I I I 5 I 1 I 1 I 3 I I I (20%)I (4%) I (4%) I I____ I _ I I I _ _____L I I I 3 I 3 I I 4 I I I (12%)I (12%)I I I_.________l____-___|_______-_I_-_____.._|__-_-____J_ I I I 1 I 1 I 2 I 5 I I I (4%) I (4%) I (8%) I l--------l___-_-__L_-___--_L____-___L____-___l Note: Percentage agreements = 44 Student achievement above rating = 24 percent Student achievement below rating = 32 percent Table F.40.--Site D Instructor Rating ____-.1_ ________ 3. ________ 3 _______ ‘1 ________ § _____ I I I 1 I I 1 I 1 I I I (4.8%)I I (4.8%)I l________l_-______L____-___l__-__-__1________L I I I I 5 I 2 I 2 I I I I (23.8%)I (9.5%)I Actual I___ I I I I I Rating I I I I 2 I 2 I 3 I I I I (9.5%)I (9.5%)I I I _L_ I I I I I I I I 4 I 4 I I I I I (19%)I I I I __ I I I I I I I I 4 I 5 I I I I I (19%)I I I I I I I Note: Percentage agreements = 19 Student achievement above rating = 0 Student achievement below rating = 81 ISI Table F.41.--Site E Instructor Rating 1 _ 3 _-__§ _______ fl ________ § _____ I I 3 I I I I 1 I I (20%)I I I I l- | I ___-l____-___l-_______L I I 3 I 1 I I I 2 I I (20%)I (6.7%)I I I Actual I I I _ I _I________I Rating I I I 1 I I I 3 I I I (6.7%)I I I l---__-_-1-_______l-_______L----_-__l_--_____I I I I 1 I 3 I I 4 I I I (6.7%)I (20%)I I l__-___-_l________l ________ I -_ I __I I I I I 2 I 1 l 5 I I I I (13.3%)I (6.7%)I I I I I I Note: Percentage agreements = 53.4 Student achievement above rating = 20 percent Student achievement below rating = 2 6.6 percent Table F.42.--Site F Instructor Rating ____-i ________ a ________ a _______ e ........ a _____ I 1 I I I I I 1 I (4.5%)I I I I I l____-__-l________l________I____-___l_______-l I I I 5 I I 1 I 2 I I I (22.7%)I I (4.5%)I Actual I_ I I I I _I Rating I I I 2 I 3 I 1 I 3 I I I (9.1%)I (13.6%)I (4.5%)I I I I I I I I I I I 1 I 4 I 4 I I I I (4.5%)I (18.2%)I I I I ___I I I I I I I 2 I 2 I 5 I I I I (9.1%)I (9.1%)I | I I Note: Percentage agreements = 27.2 Student achievement above rating = 9.1 percent Student achievement below rating = 6 3.7 percent I52 Table F.43.-Student percentile rank achievement in relation to instructors' rankings of students: machine trades, Sites A-F combined, in frequency and (percent). Sites A - F Instructor Rating 1 2 3 4 if: _ I l I 7 I 7 I I 2 I 1 I (.5%)I (5%) I (5%) I I (1%) I I I I I I I I 3 I 6 I 11 I 9 I 5 I 2 I (2%) I (4.5%)I (8%)I (7%) I (4%) I Actual I__ I I I I I Rating I I 2 I 8 I 13 I 5 I 3 I I (1%) I (6%) I (10%)I (4%) I I I I I I I I I 1 I 7 I 12 I 9 I 4 I I (.5%)I (5%) I (9%)I (7%) I I _ I I I I I I I I 4 I 8 I 13 I 5 I I I (3%) I (6%)I (10%)I I I I I I Note: Percentage agreements = 30 Student achievement above rating = Student achievement below rating = 19 percent 51 percent Tables F.44-F.47.-Student percentile rank in relation to instructors' rankings of students: mechanical drafting, all sites, in frequency and (percent). Table F.44--Site H Instructor Rating I I I 2 I 1 I 3 I 2 (5.1%)I (2.6%)I (7.7%)I (5.1%)I 1 2 3 4 5 4 I 1 I I I 1 (10.3%)I (2.6%)I I I I I I I 3 I 2 I 2 I 2 I 2 (7.7%)I (5.1%)I (5.1%)I (5.1%)I Actual I I I I Rating 2 I 7 I 5 I I I I I I I I I I I 2 I 1 I I I (5.1%)I (2.6%)I I I I I I I I I I 3 I (5.1%)I (17.9%)I (12.8%)I I I I I I I I I I Note: Percentage agreements = 33.3 Student achievement above rating = 53.8 percent Student achievement below rating 8 12.9 percent 153 Table F.45--Site I Instructor Rating 1 3 -__-§ ..... ‘I Q _____ I I I 1 I 2 I I 1 I I I (4.8%)I (9.5%)I I I_ I I___ I I I I I I I 4 I 1 I 2 I I I I (19%)I (4.8%)I Actual I__ I_ _I I__ I I Rating I I | I 4 I 2 I 3 I I I I (19%)I (9.5%)I I________I___ I __ I I I I I I 1 I 2 I 2 I 4 I I I (4.8%)I (9.5%)I (9.5%)! I I _I____ I _ I I I I I I 1 I 1 I 5 I I I I (4.8%)I (4.8%)I I I I I Note: Percentage agreements = 14.3 Student achievement above rating = 9.6 percent Student achievement below rating = 76.1 percent Table F.46.--Site J Instructor Rating 1 a __g 4 5 _ I I 1 I I I I 1 I I (5.6%)I I I I I I I I I I I I 1 I 3 I 3 I I a I I (5.6%)I (16.7%)I (16.7%)I I Actual I I I I I I Rating I I I 1 I 1 I 1 I 3 I I I (5.6%)I (5.6%)I (5.6%)I I I I I I I I I I I 4 I I a I I I I (22.2%)I I I I I I I I I I I I I 3 I 5 I I I I I (16.7%)I I I I I I I Note: Percentage agreements = 50.1 Student achievement above ratin g = 0 percent Student achievement below rating = 4 9.9 percent 154 Table F.47.-—Site K Instructor Rating l 2 3 4 5 _ I I I 1 I 1 I 1 I 1 I I I (5.3%)I (5.3%)I (5.3%)I I I I I I _I I I I I 1 I 2 I 2 I I I I (5.3%)I (10.5%)I Actual I I | I I I Rating I I I I 2 I 2 I 3 I I I I (10.5%)I (10.5%)I I I I I I I I I I 1 I 2 I 2 I 4 I I I (5.3%)I (10.5%)I (10.5%)I I I I I I I I I I I 1 I 3 I 5 I I I I (5.3%)I (15.8%)I I I I I Note: Percentage agreements = 26.3 Student achievement above rating Student achievement below rating 10.6 percent 63.1 percent Table F.48.-Student percentile rank in relation to instructors' rankings of student rank: mechanical drafting, sites H-K combined, in frequency and (percent). Sites H - K Instructor Rating 1 2 3 4 5 I 4 I 2 I 2 I 3 I l I 1 I (4%) I (2%) I (2%) I (3%) I (1%) I I I I I I I I 3 I 3 I 5 I 10 I 3 I 2 I (3%) I (3%) I (5%) I (10%)I (3%) I Actual I I I I I I Rating I 2 I 7 I 6 I 7 I 5 I 3 I (2%) I (7%) I (6%) I (7%) I (5%) I I I I I I I I 2 I 1 I 5 I 10 I 4 I 4 I (2%) I (1%) I (5%) I (10%)I (4%) I I I I I I I I I I 2 I 3 I 7 I 5 I I I (2%) I (3%)I (7%) I I I I I I I Note: Percentage agreements = 30.9 Student achievement above rating Student achievement below rating 25 percent 44.1 percent REFERENCES REFERENCES American Vocational Association. Fact Sheet. Arlington. Va.: American Vocational Association. March l984. Apple. Michael. “Curriculum in the Year 2000: Tensions and Possibili- ties.“ Educational Leadership (January l983). r Barr. Arvil 5. “Research Methods.“ In Encyclopedia_9f Educational Research. pp. ll60-66. Edited by Chester N. Harris. New York: Macmillan. I960. Bloom. Benjamin 5.: Hastings. Thomas J.: and Madaus. George F. Hand- book on Formative and Summative Evaluation of Student Learning. New York: McGraw-Hill Book Co.. l97l. BOrg. N. R.. and Gall. M. 0. Educational Research: An Introduction. New York: Longman. I983. Bottoms. Gene. “Moving Toward Excellence.“ American Vocational Education Journal (November/December 1984). Brookover. Hilbur: Beamer. Laurence: Efthim. Helen: Hathaway. Douglas: Lezotte. Lawrence: Miller. Stephen: Passalacque. Joseph: and Tornatzky. Louis. CreatingIEffective Schools: An Inservice Pro- gram for Enhancigg School Learning Climate and Achievemen . Holmes Beach. Fla.: Learning Publications. I982. Carver. R. P. “Special Problems in Measuring Change Nith Psychometric Devices.“ In Evaluation Research: Strategies and Method . Washington. D.C.: American Institute for Research. 1970. ' Cawelti. Gordon L. Effectivea§chools Movament and Evaluation. Asso- ciation for Supervision and Curriculum Development. Chamberlin. Dean: Chamberlin. Enid: Drought. N. 0.. and Scott. H. E. Did They Succeed in College? New York: Harper and Brothers. l942. Cronbach. Lee J. “Course Improvement Through Evaluation.“ In Cur- riculum Evaluation. Edited by David A. Payne. Lexington. Mass.: 0. C. Heath. 1974. 155 156 Dressel. Paul L. Evaluation as Instrpction. Proceedings of the I953 Invitational Conference on Testing Programs. Princeton. N.J.: Educational Testing Service. 1954. Durant. William. TheaIife of Greece. New York: Simon & Schuster. I939. Eaddy. Kenneth. Director of V-TECS. Telephone interview. February 25. I986. Ebel. Robert L. Essentials of Educational Measurements. 3d ed. Englewood Cliffs. N.J.: Prentice-Hall. l979. Edmonds. Ronald R. “Effective-Schools Correlates.“ Education Week. January l5. I986. Evans. Rupert N.. and Herr. Edwin L. Foundations of Vocational Educa- tion. Columbus. Ohio: Charles E. Merrill. I978. Faggen. Jane. Decision Reliabilit1,and CIassification Validity for Decision Oriented Criterion-Referenced Tests. Bethesda. Md.: ERIC Document Reproduction Service. ED l59 22l. l979. Flanagan. J. C.: Davis. F. B.: Dailey. J. T.: Shaycroft. M. F.: Orr. D. B.: Goldberg. 1.: and Neyman. C. A.. Jr. Project TALENT. The Igentification. Development.aapg Utilization of Hagan TaJents: The American High School Student. Pittsburgh: University of Pittsburgh Press. I964. Finch. C. R.. and Crunkilton. J. R. Curriculum Development in Voca; tional and Technica] Education. Boston: Afllyn and Bacon. l979. Fox. Robert L. The Research Process in Education. New York: Holt. Rinehart. and Winston. I969. Glaser. Robert. and Nitko. Anthony J. “Measurement in Learning and Instruction.“ In Educational Measurement. 2d ed. Edited by Robert L. Thorndike. Washington. D.C.: American Council on Edu- cation. l97l. Gleason. James. Director. IDECC. Telephone interview. March 1986. Harnisch. Delwyn L. Computer AppIication Issues in Certification apg Licensure Testing. Bethesda. Md.: ERIC Document Reproduction Service. ED 26l 079. I986. Harris. Chester W. Techniques for Analyzing Test Response Dat . Bethesda. Md.: ERIC Document Reproduction Service. ED ll0 496. I975. 157 Hillway. Tyrus. Introdgction to Research. Boston: Houghton-Mifflin. I956. Humbleton. Ronald K.. and Novick. Melvin R. “Toward an Integration of Theory and Methods for Criterion-Referenced Tests.“ Journal of Educational Measurement lO (Fall I973). Independent School. A Nation at Risk: The Imperative for Educational Reform. An Open_Letter to theigmerican PeopI_. Independent School. October l983. Kimpston. Richard D.. and Anderson. Douglas H. “A Study to Analyze a; Curriculum Decision Making in School Districts.“ Educational ‘ Leadership 40 (November I982): 63-66. Klein. Stephen P.. and Kosecoff. Jacqueline. “Issues and Procedures in the Development of Criterion-Referenced Tests.“ In Readings in Measurement and Evaluation in Education and Psychology. Edited by William A. Mehrens. New York: Holt. Rinehart. and -' Winston. l976. v Koch. William R.. and Reckase. Mark 0. Problems in Application of Latent Tgait Models to Iajloreg Testing. Bethesda. Md.: ERIC Document Reproduction Service. ED l77 l96. I980. Kromhout. Ora. Center for Instructional Development and Services. Tallahassee. Florida. Telephone interview. February l5. I986. Lathrop. Robert. Center for Instructional Development and Services. Tallahassee. Florida. Telephone interview. Lee. Connie W. Third Progress and Information Report of the VocationaI-Technical Education Consortigmaof States. Atlanta: Southern Association of Colleges and Schools. July 1976. Lewis. Morgan V.. and Martin. Scott. Measures of Occgpationally Specific and NonoccgpationaIIy Specific Knowledge and Skills: A Compendium. Columbus: National Center for Research in Voca- tional Education. The Ohio State University. l986. Lezotte. Lawrence. “On the Success of the Effective Schools Movement by Lynn Olson.“ Education Week. January l5. I986. Martuza. V. R. Applying Norm-Referenced and Criterion-Referenced Measures in Education. Boston: Allyn and Bacon. I977. McCage. Ronald. [Vocational-Technical Education Consortium of States] Newsletter to Member States. July I986. 158 Mehrens. William A.. and Lehmann. Irvin J. MeasurementIapd Evaluation in Education and Psychology. New York: Holt. Rinehart and Winston. 1984. Meranto. Phillip. “The Politics of Federal Aid to Education in l965.“ In A History_of Congressiona] Acts amd Court Decisions. Ann Arbor: University of Michigan Press. l974. Michigan Department of Education. Competency-Based Education In- Service Monges. Lansing: Department of Education. l974. . Vocational-Technical Education Service. Memo from the Superintendent of Schools. Phillip Runkel. February ll. I986. Miller. P. L. “Change Agent Strategies: A Study of the Michigan-Ohio Regional Educational Laboratory.“ Dissertation_Apstracts Immer- national 29 (I968): l684A. (University Microfilms No. 68-17. ll2 Millman. J. “Criterion-Referenced Measurement.“ In Evaluation in Education.4§prrent Applications. Edited by W. J. Popham. Berkeley. Calif.: McCutchan. l974. Morsh. J. E.. and Archer. W. B. Procedural Guide for Conducting_0ccp; pational Surveys in the United States Air Force. Lackland Air Force Base. Texas: Personnel Research Laboratory. September I967. Naisbitt. John. Megatrends. New York: Warner Books. l984. National Center for Educational Statistics. VocationaI Edggation DQLQ Survey. Preliminary statistics for l980-l98l. National Institute of Education. §prvengesults on the Educational Attindes of the American Peopl . Washington. D.C.: National Institute of Education. I985. Oberstein. Diane R. “Item Bank Utilization.“ Ph.D. dissertation. University of California at Los Angeles. l98l. Oliver. Dale. Professor of Education. Division of Vocational- Technical Education. Virginia Polytechnic Institute and State University. Telephone interview. February 16. I986. Olson. Chris. Projects Coordinator. Vocational Education Curriculum Project. Michigan State University. Personal interview. November I985. Olson. Lynn. “Effective Schools.“ Education Week. January IS. I986. 159 O'Reilly. Robert P.. and others. Validation of a Multiple-Choiga Clpae Test of_theral Comprehension: Summary Repor . Bethesda. Md.: ERIC Document Reproduction Service. ED l20 265. I976. Parnell. Dale. Hearings before the General Subcommittee on Education of the Committee on Education and Labor. House of Representa- tives. 93rd Cong.. lst sess. In Elementary and Secondary Edppa; tion Amendments of 1973. Pollock. William T.. and McDole. Thomas L. Devel0pment of a National Item Bank for Tasts of Driving_Knowledg . Bethesda. Md.: ERIC Document Reproduction Service. ED 128 403. I977. Popham. James. and Husek. T. R. “Implications of Criterion-Referenced Measurement.“ Journal of Educational Measurement 6 (Spring I969): l-9. Pratt. William. “Michigan Council of Vocational Administrators' Response to A Nationiat Risk.“ November I983. Runkel. Phillip. “Approval of Secondary Vocational Program Evaluation Model.“ Lansing: State Board of Education. February 25. l986. Sax. Gilbert. Principles of Educational Measurement. Belmont. Calif.: Wadsworth Publishing Co.. 1974. Scriven. Michael. “The Methodology of Evaluation.“ In Curriculum and Evaluation. Edited by Arno A. Bellack and Herbert M. Kliebard. Berkeley. Calif.: McCutchan Publishing Corp.. I977. Seney. Mack. Michigan Department of Education. Personal interview. March 7. l986. \ Smith. Nick. U. S. Senator. “Accountability in Education.“ Senate Bill I95. Lansing. Mich.: Senate Analysis Section. 1986. Stroud. J. Psychology in Education. New York: McKay. 1946. Stufflebeam. D. L.: Foley. W. J.: Gephart. W. T.: Guba. E. 6.: Hammond. R. L.: Herriman. H. 0.: and Provus. M. M. Educational Evaluation and Decision-Making in Education. Itasca. Ill.: Peacock. l97l. Taylor. Fredrick. Scientific Managemen . New York: Harper and Row. I947. Thorndike. R. L.. and Hagen. E. Measurement and Evaluation in Psychology and Education. New York: Wiley. I969. 160 Tyler. Ralph. Bas c Principles of Cgrriculum and Instruction. UOS. 1 Chicago: University of Chicago Press. l949. Congress. The Vocational Education Act of l963. Public Law 88-2lO. Washington. D.C.: Government Printing Office. l963. . Elementary apg Secondary Education Act of 1965. Public Law 89-l0. Washington. D.C.: Government Printing Office. 1965. . The Vocational Edagation Amendments of l968. Public Law 95-76. Washington. D.C.: Government Printing Office. l968. . The 1976 Vocational Education Amendments. Public Law 94-482. Washington. D.C.: Government Printing Office. l976. . The Carl Perkins Vocational Eduaation Act. Public Law 98- 524. Washington. D.C.: Government Printing Office. 1984. U.S. Department of Education. National Commission on Excellence in Education. A Nation at Risk. Washington. D.C.: Government Printing Office. l983. Vocational-Technical Education Consortium of States. Handbook for Developing Criterion-Referenced Test Itemg. Atlanta: V-TECS. l985. Wentling. Tim L. Evaluating Occupational Education and TrainingaPro- grams. Urbana. Ill.: Griffon Press. l980.