w..— .1.- w‘ ‘5'“,— Miclflgan state "marshy THEE” This is to certify that the dissertation entitled THE DESIGN, DEVELOPMENT, AND FIELD TEST OF A PROTOTYPE PROGRAM EVALUATION APPROACH FOR MEDICAL RESIDENCY ROTATIONS presentedby ERIC GAUGER has been accepted towards fulfillment of the requirements for The Ph.D;_deg1-eein EDUCATIONflL SYSTEMS DEVELOPMENT VIIIJ“ ‘ Major professor Date June 6, 1985 MS U i: an Affirmative Action/Equal Opportunity Institution 0-12771 MSU LIBRARIES m RETURNING MATERIALS: Place in book drop to remove this checkout from your record. FINES will be charged if book is returned after the date stamped below. THE PRO”: heparin THE DESIGN, DEVELOPMENT, AND FIELD TEST OF A PROTOTYPE PROGRAM EVALUATION APPROACH FOR MEDICAL RESIDENCY ROTATIONS By Eric Gauger A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Counseling, Educational Psychology, and Special Education 1985 3(08—5354 @1986 ERIC GAUGER All Rights Reserved (I THE l PROTC The study desc systematic approaCh graduate medical edu the literature revealr and field tested for e suitable, in Its entire evaluations of their judging program mer attempts to evaluate and requirements st: luund in the litera structure of rotati conducting program de518W and condu imllemented that Iliations. The compom borrowed from th. Illicit was iielr ABSTRACT THE DESIGN, DEVELOPMENT AND FIELD TEST OF A PROTOTYPE PROGRAM EVALUATION APPROACH FOR MEDICAL RESIDENCY ROTATIONS BY Eric Gauger The study described in this dissertation addressed the need for finding a systematic approach to program evaluation that fits the unique structure of graduate medical education (GME) programs—-i.e., residency rotations. A review of the literature revealed that no models or approaches had been designed, developed, and field tested for evaluating rotations, and no model could be identified that was suitable, in its entirety, for this purpose. Residency and rotation directors conduct evaluations of their programs for purposes of accreditation, decision-making, and IUdging program merit. Problems arise for the residency or rotation director who attempts to evaluate GME programs. Guidelines for accreditation are often vague, and requirements specified in only general terms. And program evaluation models found in the literature are general, and do not take into account the unique structure of rotations. Residency and rotation directors are rarely trained in conducting program evaluations. The study reported in this dissertation, then, was designed and conducted to determine whether an approach could be developed and implemented that addressed the specific requirements of evaluating residency rotations. The components and concepts of the approach developed were derived or borrowed from the program evaluation models reviewed in the literature. The aPProach was field tested by evaluating a multidisciplinary residency rotation. A number 0i me implementatior Amaw‘ merit of the components. director oi the all the mater? the consultar approach. It the metaevalr research que: lollowing cor drawn: l. Th dir 2. ll 3. II I. l ir Finall IESle'Ch ar number of methods were used to collect data about instructional intents and implementation and curriculum relevance. A metaevaluation was designed and conducted to summatively determine the merit of the prototype evaluation approach. The metaevaluation had several components. First, a structured interview was conducted with the rotation director of the program evaluated. Second, a metaevaluation consultant reviewed all the materials used in the primary evaluation. Next, the researcher met with the consultant and answered the consultant's questions about the prototype approach. Then, a structured interview with the consultant was conducted. Next, the metaevaluation data were synthesized and answers were generated to the four research questions. From the results of the field test and the metaevaluation, the following conclusions about the prototype program evaluation approach have been drawn: 1. The approach was useful in providing information to the rotation directors. 2. The approach was practical. 3. The approach was ethical. 4. The approach collected and conveyed to the rotation directors accurate information about the rotation. Finally, recommendations for future program evaluators and for further research are presented. To It and wortl To Jackie. Her approval, encouragement, understanding, and love made this study not only possible, but worthwhile. hhau ellorts of all goal. My motl and always e completion of my wife, who herself. They Many pr ways. First, dissertation, Iorm. llarler component5r worked throc deadlines. 1 major role ir Placed on he 0i co doctoral to greatly app] uttered mur he directed medical ed tarly Impet hellllll rer encouyggyn Illengtheh ACKNOWLEDGEMENTS It is a most difficult task, in the few pages allotted, to acknowledge the efforts of all those whose contributions have helped me to achieve this present goal. My mother, Frieda, certainly, because she recognized the value of education and always encouraged me to pursue mine more seriously. I trust that the completion of my program indicates that her concern was rewarded. And Jackie, my wife, who promoted my development and my ambitions at great sacrifice to herself. They are most responsible. Many people contributed to this specific project in different and important ways. First, Marlene Dodge word—processed not only the final copy of this dissertation, but the many iterations it went through before reaching its present form. Marlene also typed and word—processed countless documents used in various components of the study which do not appear in this volume. She came in early, worked through lunch, and stayed late on many occasions as I attempted to meet deadlines. Her work was impeccable, kept the study manageable, and played a major role in seeing that it was completed. And despite the constant time demands placed on her, Marlene always worked with good humor. Of course, I want to acknowledge those who served as members of my doctoral committee--their professional guidance and time was invaluable and greatly appreciated. Bruce Miles was my committee chairman, a role in which he Offered much advice. Bill Anderson gave freely of his knowledge and expertise as he directed the disseration. Bill Abbett provided perspective from the graduate medical education administrators‘ viewpoint. Cass Gentry provided much of the early impetus under which the study was launched. Rebecca Henry furnished many helpful recommendations throughout the study and was always ready with an encouraging word. Finally, Steve Yelon offered succinct advice which significantly strengthened the study. I thank each of them for their contributions and I am iii udhuutm I want turnout mwmdl rmNnfimt umh"whl hh his uhahg law whmtd hehme NNNam Cmn mayacknc and hope Immdh henhe hamm Ndubj hyyw vitally in handy hwem “huh hmpg A "ludicur confident they will see the effects of their labor in my professional growth. I want to thank Doug McKeag and Dave Hough for inviting me to examine their rotation. They both gave willingly Of their valuable time and allowed me access to all components of the rotation. Sue Curtis and Vicki Curley provided the cooperation which allowed me to obtain information "on the other side of the tracks" without which the study could never have been completed. I thank them both. It is also from Sue, Vicki, and Mike Curley that I learned the fine art of tailgating. I also want to acknowledge my metaevaluation consultant. Kent Sheets performed admirably and professionally in this role, and did so on very short notice. The time and expertise Kent brought to the metaevaluation effort was clearly evident and sincerely appreciated. Completing this current project provides an appropriate vehicle by which I may acknowledge my finest teachers, those who have shaped the person I am today and hope to become tomorrow. I want to thank Red for the many lessons I have learned from him, and for the style he uses in teaching them. Ted taught me that the price of independence is frequently high. But never too high. Many years ago, on a mountain in Vermont, Roy Sheldon prepared me for my eventual graduate studies by stressing that work needs constantly to be gone over before it is right. Roy—-I've gone over and over my work. And he's ready. Steve Appell has been vitally instrumental in my development. Perhaps the most important lesson I have learned from Steve is perspective. But there are others, too. Red, Ted, Roy and Steve share many personal attributes. Humor is the one I prize most and attempt to emulate. I admire and respect these four master teachers. Their influence has been profound. A number of people contributed to this project by helping me maintain a modicum of sanity throughout my stay in Michigan. Howard Teitelbaum provided iv advice and Pmi andeitorts. Fr that made thi! Celtics." Ken exploring a net Valerie and K endeavors, an negatively on I by making bo‘ their visits. Two cou Bill and Valet performed ori the high seas Finally, Space, mater program. Hi respected org advice and professional knowledge on many occasions. I am grateful for his time and efforts. Fred Benjamin, a native son of New England, provided many moments that made this marathon tolerable despite his inability to remember "the old Celtics." Ken Sheets shared with me the many adventures of discovering and exploring a new noosphere. Bill Yust and Steve came out to run the Old Kent, and Valerie and Kathy Zaffino visited, too. We were courageous in our athletic endeavors, and the fact that Steve could never beat me should not reflect negatively on him. Gary and Mary Reeves visted twice, and Fitz showed his roots by making both trips over Big IO football Saturdays. I appreciated and enjoyed their visits. Two couples also helped out on the eastern front. Burt and Betty Conley and Bill and Valerie are valued friends. Burt, another native son of New England, performed original nautical research while I was away, and perfected and patented the high seas donut maneuver (HSDM). Both couples deserve commendations. Finally, I want to thank Clack Maatsch and OMERAD for providing me office space, materials, and particularly secretarial assistance while completing my program. His concern for the troops is just another reason that OMERAD is the respected organization that it is. Chapter Or Intros Back- Stat Pury Res Lirr Del Ovr (Sham lnt Ev TABLE OF CONTENTS Chapter One Statement of the Problem Introduction Background Graduate Medical Education Evaluation of Residency Programs The Rotation in Sports Medicine Statement of the Problem Purpose of the Study Research (Metaevaluation) Questions Limitations Definition of Terms Overview of the Dissertation Chapter Two Review of the Related Literature Introduction Evaluation of Graduate Medical Education Programs Program Evaluation Models The Countenance of Educational Evaluation Model CIPP Evaluation Model CSE Evaluation Model Discrepancy Evaluation Model Summary of Program Evaluation Models Metaevaluation Summary vi lntroduct Backgrou Design 0 Spt Prr Field Te Th Pa lmplemr Id I I Instrur Chapter Three Methodology and Procedures Introduction —“ \ Background Design of the Approach Specific Evaluation Questions Process Used to Implement the Approach Field Test of the Approach The Rotation in Sports Medicine Participants Implementation of the Approach Identification of Rotation Intents Examination of Documents Data Collection Former Residents Questionnaire Current Residents Questionnaire External Review Panel Questionnaire Observation and Interview Data Documentation of Discrepancies Preparation and Delivery of the Evaluation Report Discrepancy Resolution Improvements to the Rotation Instruments Former and Current Residents Questionnaires Pilot Testing (Former and Current Residents) External Review Panel Questionnaire Pilot Testing (External Review Panel) Validity of the Questionnaires Validity of Former and Current Residents Questionnaires Validity of External Review Panel Questionnaire Reliability of the Questionnaires vii so so 69 76 76 77 80 so so so 81 82 83 85 85 85 85 86 86 89 89 9O 9O 90 92 93 Meta Met Sur mu lnt Checklists Data Analysis Procedures Metaevaluation of the Approach Participants Data Collection Rotation Directors Consultant Metaevaluation Instruments Structured Interview Guide (Rotation Directors) Structured Interview Guide (Consultant) Validity and Reliability of the Metaevaluation Structured Interview Guides Data Analysis Procedures Summary Chapter Four Results Introduction Results of the .Primary Evaluation Rotation Context Variables Evaluation Question 1 Evaluation Question 2 Evaluation Question 3 Availability of Faculty and Staff Injured or III Athletes Facilities Budget Educational Materials Instructional Variables Evaluation Question 4 Evaluation Question 5 Evaluation Question 6 Evaluation Question 7 viii 95 95 95 96 97 97 98 98 98 100 100 100 100 100 101 104 104 105 105 105 106 107 107 107 107 107 Evaluat lmplen Discus Resul' Summ SUmr Sumr hm hm Sum Evaluation Variables Evaluation Question 8 Evaluation Question 9 Implementation of Curriculum and Objectives Competencies Learned Through Discussion Competencies Learned Through Observation Competencies Learned Through Performance Relevance of Curriculum and Objectives External Review of Objectives Implementation of Additional Evaluation Variables Evaluation Question 10 Evaluation Question 11 Evaluation Question 12 Discussion of Findings (Primary Evaluation) Summary Of the Evaluation Meeting Results of the Metaevaluation Summary of Metaevaluation Results Research (Metaevaluation) Questions Research Question I Research Question 2 Research Question 3 Research Question 4 Design Criteria General Metaevaluation Data Summary of Metaevaluation Results Summary of the Chapter Chapter Five Summary and Conclusions Introduction Summary of the Research Problems Encountered Data Collection Strategies 108 108 109 110 III 113 114 116 118 121 121 123 128 130 131 132 134 134 134 135 136 138 139 141 142 143 145 145 145 146 1R7 Conc. Reco Recc Sumr Relerence hppendicr Apr her her he do Evaluation Question Clusters Strengths and Weaknesses of the Approach Conclusions Recommendations for Program Evaluators Recommendations for Further Research Summary References Appendices Appendix A: Appendix B: Appendix C: Appendix D: Appendix E: Appendix F: Appendix G: Appendix H: Appendix I: Appendix J: ' Appendix K: Appendix L: Appendix M: Appendix N: Participants Questionnaire Survey Of Important Components Of A Family Practice Rotation In Sports Medicine Checklist for Observing and Interviewing Faculty Checklist for Observing and Interviewing Residents Metaevaluation Transcription (Rotation Director) Metaevaluation Transcription (Consultant) Structured Interview Guide (Rotation Director) Structured Interview Guide (Consultant) Summary Data Of Competencies Learned Through Discussion Summary Data of Competencies Learned Through Observation Summary Data Of Competencies Learned Through Performance External Review Panel Ratings of Individual Competencies Summary Data of Implementation of Instructional Strategies Summary Data of Residents Achievement Of Prerequisites 150 158 158 161 162 163 171 189 201 202 204 215 225 227 230 231 236 240 241 I The 6 Dat LIST OF FIGURES Figure 1 2 The Countenance of Educational Evaluation Model The CIPP Model The CSE Model The Discrepancy Model The Prototype Program Evaluation Approach for Evaluating Residency Rotations Data Collection Techniques Matrix xi #1 1+5 70 81+ my; Evaluati Questior Curricul Discrep: Discrep Discrep Curricu Compe' Discrep Conten Knowlr Reside LIST OF TABLES Table Evaluation Questions Questionnaire Specifications Curriculum Descriptions Discrepancies in Implementation of Discussion Competencies Discrepancies in Implementation of Observation Competencies Discrepancies in Implementation of Performance Competencies Curriculum Mean Ratings Competencies Rated at or Below Desirable (2.00) Discrepant Competencies Content of Pre- Posttest Knowledge, Skill and Attitudes of Former Residents Residents‘ Assessments of Rotation Faculty 91 110 112 114 115 117 119 120 126 127 129 Instr on office deportmer lnstroctior designing, education One medical evoloutio evoluulio medicol systems onolyze r Ins Chapter One Statement of the Problem Introduction Instructional developers are often employed in medicine. They may work for an office of medical education in a medical school, for an individual clinical department, or for a graduate medical education (residency) program. Instructional developers generally help physicians and other health professionals in designing, developing, and evaluating individual courses and clerkships, or entire education programs. One of the more frequent responsibilities for Instructional developers in medical education is evaluation. They are often requested to assist in the evaluation of the performance of medical students or residents, and in the evaluation of the effectiveness of educational programs. Whether working in medical schools or residency programs, instructional developers often help design systems for evaluating learners, develop evaluation instruments, and collect and analyze data on the performance of medical students and residents. Instructional developers are often asked to help document the effectiveness of the education programs of medical schools and residencies. Such data are used for an internal evaluation of a medical school, clinical department, residency program, or residency rotation. Program evaluation data are periodically required, too, by external accrediting agencies. From numerous sources the medical school or residency program must assemble data that document the organization's ability to meet accreditation criteria. When attempting to conduct an evaluation of a residency rotation program, Instructional developers face several problems. First, there are no validated approaches for evaluating a rotation. Second, the existing models are unsuited to sources reQI rotations or on broad, 9! There and progra education, that they I - tar resider I document The prototype education: rotation, Practice conducte: lmetaeva accurate practical Context Continui pIQCIICi EValuol F—_———’ 2 unsuited to evaluating a residency, both in their assumptions and in the data sources required. Finally, there are no definitive criteria for evaluating residency rotations or programs. Administrators of residency education programs must rely on broad, general criteria provided by external accrediting agencies. There is a need for a validated approach for evaluating residency rotations and programs. With a decrease in the funds available for graduate medical education, residency program directors are under increasing pressure to document that they provide high-quality education, research, and patient care experiences for residents. Currently, there is no systematic approach for residency directors to document the quality of their graduate medical education programs. The purpose of this study has been to design, implement, and evaluate a prototype program evaluation approach for one component of graduate medical education: the residency rotation. The prototype approach was validated with one rotation, the rotation in sports medicine, offered by the Department of Family Practice at Michigan State University. After the program evaluation was conducted with the prototype approach, an evaluation of the results (metaevaluation) was conducted to determine whether the approach yielded accurate and useful information, and to determine whether the approach was practical and ethical. W The background section is divided into three parts. The first part provides a context for the study by describing relevant aspects of the graduate phase in the continuum of medical education. In the second part, the current evaluation practices and problems in residency programs are discussed. Finally, the program evaluated in the study is briefly described. Gradu education. specialty ii education I tor medicr medicine a them for it Phys graduate ‘. school by environme teaching I specialty Altl lacilities hospital altiliatic Gr. training resident or inter calendo 0 resid pragm- lour m Graduate Medical Education Graduate medical education is an integral part of the continuum of medical education. Physicians enroll in residency programs to develop expertise in a specialty field in medicine. Residency education is essential since undergraduate education leading to the MD. degree is no longer sufficient to prepare a student for medical practice. Graduate training thus prepares physicians to practice medicine at the level of current knowledge and technology as well as preparing them for future developments. Physicians in graduate medical education are called residents. During their graduate training, residents expand the knowledge and skills learned in medical school by assuming progressive responsibility for patient care in supervised clinical environments and through other activities such as seminars, conferences, and teaching rounds. In addition, residents seek to complete training requirements of a specialty board (e.g., family practice). Although residents may be assigned to clinics or educational or research facilities for a portion of their programs, most graduate medical education is hospital based. The hospitals are frequently owned by, operated by, or have affiliations with a medical school and are called teaching hospitals. Graduate medical education is organized by specialty. The duration of training is from three to seven years. In addition to training in their specialty, residents engage in time-based curriculum components in other medical specialties or interest areas. These are called rotations. Rotations frequently operate on a calendar-month basis. Residency programs typically specify the number of months a resident must rotate through other specialties (e.g., a particular residency program for family practice may require a second—year resident to rotate through four months of internal medicine, and two months each of pediatrics, psychiatry, general sur various age W Resir director a the reside programs Residency protessior assures n programs Residenc Americar protessia Re: Graduate Commit Council. strouser Sliecialt Medical (Anne Sometir Policies inadeqt IOI IQ” general surgery, and electives). Residency programs are periodically reviewed by various agencies to assure their quality. Evaluation of Residency Programs Residency training programs are organized and directed by a residency director whose staff is responsible for the education, training, and supervision of the residents. Specialty boards in the various areas generally require that these programs be reviewed and approved for accreditation by organizations called Residency Review Committees. The purpose of this accreditation is to provide a professional judgment of the quality of the program. The process of accreditation assures medical students, residents, specialty boards, and the public that training programs are in essential compliance with published standards (AMA, I983, p. 8). Residency Review Committees are composed of members appointed by the American Medical Association (AMA), by the specialty boards, and sometimes by professional associations of specialty fields. Residency training programs are accredited by the Accreditation Council for Graduate Medical Education (ACGME) or directly by the Residency Review Committee if the Committee has been given authority by the Accreditation Council. The Council is made up of members from five parent organizations that sponsor the Residency Review Committees: the American Board of Medical Specialties (ABMS), the American Hospital Association (AHA), the American Medical Association (AMA), the Association of American Medical Colleges (AAMC), and the Council of Medical Specialty Societies (CMSS). Problems sometimes arise because these agencies frequently disagree on standards, review policies, and procedures. Even when they do agree, standards may prove inadequate or untimely to effect program change. Among the special requirements for family practice program evaluation, for example, are the following: One measurement of the quality of a program is the performance of its graduates in the certification It is change in Hon Committe the progrr To require: specialty addition Portions responsi lequirer examinations of the American Board of Family Practice. The degree of resident attrition and the presence of a critical mass of residents are also factors to be considered i2n6)the overall evaluation of the program. (AMA, I983, p. It is uncertain whether a change in standards, for example, would effect a change in performance by residents taking a certification examination. However, the accreditation process begins when a Residency Review Committee sends a surveyor to the residency program site. The surveyor reviews the program and submits a report to the Committee. At this point the Residency Review Committee reviews the program information in detail, evaluates the program and determines the degree to which it meets the published education standards. It decides upon an accreditation status for the program and identifies areas of non-compliance with standards, if any. (AMA, I983, p. I) To be accredited, residency programs must meet both the general requirements for graduate medical education and the special requirements for a specialty. These are the standards against which programs are judged. (In addition, the program must be sponsored by an institution, e.g., a university.) Portions of the general and special requirements deal with evaluation responsibilities of the residency director and his staff. Part I of the general requirements for all residency training programs, for example, states: Institutions, organizations and agencies offering programs in graduate medical education must assume responsibility for the educational validity of all such programs. This responsibility includes . . . providing for . . . program planning, program review and evaluation of participation . . . While educational programs in several fields properly differ from one another, as they do from one institution to another, institutions and their teaching staffs must ensure that all programs offered are consistent with their goals and meet the standards set forth by them and by voluntary accrediting agencies . . . The teaching staff and administration . . . must (a) establish the general objectives of graduate medical education; . . . (c) review instructional plans for each specific program; . . . (e) develop methods for evaluating, on a regular basis, the effectiveness of the programs and the competency of persons who are in the programs. (AMA, I983, p. 9) Section Ltd < Part I special requ‘ oi the spec follows: A pro Special rec that the 9 resources the Progrc Siiii of re Section LIA of the general requirements further states: A periodic analysis of each program by representatives of the concerned departments, the residents, and the administration should be developed: These analyses should include the appraisal of: a) The goals and objectives of each program, b) The instructional plans formulated to achieve these goals, c) The effectiveness of each program in meeting its goals, and, d) The effectiveness of utilization of the resources provided. There should be documentation of these analyses and of the mech)anisms to correct identified deficiences. (AMA, I983, p. to Part II of the requirements for accreditation of programs consists of the special requirements for training in the various branches of medicine. Section VI D of the special requirements for residency training in family practice runs as follows: Evaluation of the program: The Family Practice residency must incorporate all elements of the 'Special Requirements.‘ The program should be evaluated within the context of the educational needs of the residents, teaching responsibilities of the faculty and the availability of health care resources within the community. This evaluation should include an appropriate balance between education, research and service. Continuing medical education should occupy a prominent place in the program and residents should be encouraged to develop learning patterns to continue their education. (p. 26) A problem begins to emerge for the residency director from the general and special requirements above. The problem is not that there are no guidelines, but that the guidelines are so vague. Requirements for curriculum content, program resources and personnel, duration of training, evaluation, and other dimensions of the program are specified in only general terms. Yet the director and teaching staff of residency training are responsible for defining and implementing the goals and objectives of the program. . .. An additional responsibility of the program director is to select the various evaluation techniques employed. . . . (AMA, p. ll) The required i programs, No "An addit eualuatio: important are left The prog someone best etlc Re is not a Such evr and stru worth. ideally 5 be exan TI evaluat lessons trailer kinds 0 The problem is compounded by the fact that residency directors are not required to have a background in evaluation. In family practice residency programs, for example, the program director must have had a minimum of two years full-time professional activity in the field of family practice. The director should be a diplomate of the American Board of Family Practice. It is highly desirable that the Program Director have had teaching experience in a Family Practice residency program. (AMA, I983, p. 24) No evaluation methodology is offered to program directors. To reiterate: "An additional responsibility of the program director is to select the various evaluation techniques employed." (AMA, p. ll). Thus, methods for determining the importance of various data and strategies for collecting and analyzing these data are left to local—level program directors who often lack expertise in these areas. The program directors are then forced to rely on the chance availability of someone who knows something of evaluation methodology or to resort to their own best efforts. Residency programs and rotations may be evaluated even when accreditation is not an issue. Periodic internal review in any organization can be beneficial. Such evaluation can yield useful information for decision-making about planning and structuring change, for control over ongoing activities, and for judging program worth. Unfortunately, program evaluation models found in the literature are not ideally suited to the unique structure of residency education. Reasons for this will be examined in Chapter Two of this dissertation. The graduate medical education program director is mandated to do evaluations of various programs for the purpose of accreditation. For a variety of reasons, the program director may also choose to do internal evaluations. Yet the problem of a lack of methodology to guide such endeavors constrains both of these kinds of evaluation attempts. The unit rotation in sp Department 0 Professors of MD, also set and David 0 designed in a and the servir Represe dermatology, consultation, obstetrics/9y podiatry, ps) multidisciplir athletes att area of ion Primary cor Educatianor Thraug Various acti referral clir Pristine“1 c(insults wt- ”Searches. The r limes each The Rotation in Sports Medicine The unit of graduate medical education under study in this dissertation is the rotation in sports medicine at Michigan State University, which is offered by the Department of Family Practice in the College of Human Medicine. Two Associate Professors of Family Practice serve as rotation directors. Douglas B. McKeag, M.D., also serves as Coordinator of Sports Medicine at Michigan State University, and David O. Hough, M.D., is Director of Sports Medicine. The rotation was designed in a multidisciplinary fashion and has a significant amount of structure and the services of a wide range of specialists and consultants. Represented are such specialties as athletic training, cardiology, dermatology, exercise physiology, gastroenterology, internal medicine, legal consultation, manipulative therapy, nephrology, neurology, nursing, nutrition, obstetrics/gynecology, opthalmology, optometry, orthopedics, otorhinolaryngology, podiatry, psychiatry, pulmonary diseases, radiology, surgery, and urology. This multidisciplinary approach is designed to both meet the needs of the varsity athletes at the university as well as to improve graduate training in the specialty area of family practice. The curriculum is designed to provide residents with primary care experiences in the areas listed above, while stressing patient education and injury prevention techniques. Throughout the rotation, the resident observes, discusses, and participates in various activities such as an informal clinic in a training-room setting, a formal referral clinic in an ambulatory clinical setting, on—site medical coverage of team practices, and on-site coverage of varsity athletic events. In addition, the resident consults with various coaches, trainers, equipment managers, and consultants, and researches and writes a paper on some aspect of sports medicine. The rotation in sports medicine operates on a calendar-month basis, eight times each year. The recommended length of the rotation is one month. The rotation is 0‘ residency prC fourth-year n The ro reasons. Fir various comp rotation was participants conducted. ' Becaus more complv more tradit' rotation din the rotatior rotation in : Resid “Milli? ar mechanism ihis task. ngram 9, “It left to fact ppm 1 Maroon. Standards, EVal rotation is available to residents in family practice at MSU, residents from other residency programs who are in the second or third year of their residencies, and fourth-year medical students at MSU. The rotation in sports medicine was selected for study for a number of reasons. First, it had been in existence for a number of years and had adopted various components that were used repeatedly for instructional purposes. The rotation was offered eight times or more each year, and a significant number of participants were involved. No formal evaluation of the rotation had ever been conducted. Thus, the effectiveness of the rotation was not known. Because of the multidisciplinary nature of the rotation, the curriculum was more complex and the experiences of the participants were more diverse than in more traditional rotations. The evaluation also served as an opportunity for the rotation directors to document the effectiveness of the rotation externally. And the rotation directors were interested in and receptive to an evaluation of the rotation in sports medicine. Statement of the Problem Residency directors are required by accrediting agencies periodically to analyze and to appraise their programs and to document their analyses and mechanisms to correct identified deficiencies. A number of problems complicate this task. Among them are that only Vague evaluation guidelines are provided, program goals and objectives may vary (within limits), and evaluation techniques are left to the residency director. These problems are often compounded by the fact that the residency directors are not required to have and often do not have a background in evaluation. Further, sponsoring agencies often disagree on standards, review policies, and procedures. Evaluation of various components of graduate medical education are sometimes too. Progrv structure 0‘ The p graduate rr evaluating typically it The 1 field test education, were delir were colle The accurate W08 prac metaevalv metaevalv Rec and into e\ialuatio iUestions The sec: questing tiresome question "its ol l0 sometimes undertaken for reasons other than accreditation. Problems arise here, too. Program evaluation models found in the literature are not well suited to the structure of graduate medical education. The problems described here make systematically evaluating a program in graduate medical education difficult. There is a need for a viable approach to evaluating programs with the special environmental and contextual constraints typically found in a rotation. Purpose of the Study The purpose of the study was twofold. The first was to design, develop, and field test a prototype approach for evaluating one component of graduate medical education, the residency rotation. Thus, information requirements of a rotation were delineated, strategies for accumulating the data were developed, and the data were collected and analyzed and the results reported to the evaluation audiences. The second purpose was to determine whether the approach used yielded accurate and useful formative evaluation information and whether the approach was practical and ethical. Metaevaluation criteria were thus established, metaevaluation data were collected and analyzed and reported to the metaevaluation audience. Because of the twofold purpose of the study, two different sets of questions and two different data sources are presented. One set of questions was the l2 evaluation questions that guided the primary evaluation. The I2 evaluation questions and sources of data for answering them are presented in Chapter Three. The second set of questions presented is the four research (metaevaluation) questions that guided the study. The research (metaevaluation) questions are presented in the next section of this chapter. The research (metaevaluation) questions and sources of data for answering them are also presented in Chapter Three of this dissertation. There was the de evaluation 1 second com The questic approach w l. Wt di' 2. W 3. W i. D The to validatv study was ellect cor results fr interence. The just the iinnily p, not on m' A t the selev islectiar I I Research (Metaevaluation) Questions There have been two major components of this study. The first component was the design, development, and implementation of an approach to program evaluation tailored to the unique needs of a particular residency rotation. The second component was an evaluation of the first component, or a metaevaluation. The questions the study attempted to answer about the program evaluation approach were: Was the approach useful in providing information to the rotation directors? 2. Was the approach practical? 3. Was the approach ethical? it. Did the approach collect accurate information about the rotation? Limitations The limitations of the study are several. First, the purpose of the study was to validate an approach for evaluating a particular residency rotation. As such, the study was not designed as an experiment that could yield inferential, cause—and- effect conclusions, but as a contextual evaluation of actual goals, treatments, and results from the specific program processes undertaken in that rotation. Thus, inferences to different rotations should be made with caution. The data for the approach were collected from faculty and participants of just the rotation under study. A random sample of residents across different family practice rotations was not taken. And data were collected only once and not on multiple occasions. A third limitation is that the faculty of the rotation under study assisted in the selection of a panel of external evaluators, and hence may have biased that selection. A final I study. The designing and raise the ques fled] Council for < specialized p determined 1 Review Cami a protessionc P- 7). M will or shoulv such audienv w recognition qualificatior M medical stu liltiil. Cine being essen obIeCjIVes ( deieimine I2 A final limitation concerns the multiple roles played by this researcher in the study. The researcher helped design and perform the evaluation as well as designing and performing the summative metaevaluation. These circumstances raise the question of conflict of interest and cooptation. Definition of Terms Accreditation (of a program)—-The process by which the Accreditation Council for Graduate Medical Education (ACGME) grants public recognition to a specialized program which meets certain established educational standards as determined through initial and subsequent periodic evaluations by the Residency Review Committee. The primary purpose of the accreditation process is to provide a professional judgment as to the quality of the educational program (AMA, I983, P. 7). Audience (evaluation)—-A group, whether or not they are the client(s), who will or should see and may use or react to an evaluation. Typically there are many such audiences to an evaluation (Scriven, I980, pp. ll-l2). Certification (of a physician)--The process by which a specialty board grants recognition to an individual physician who has met certain predetermined qualifications, as specified by the board (AMA, I983, p. 7). ClerkshipuSupervised undergraduate medical experiences designed to give medical students their initial exposure to the clinical setting within a specialty area. Competencies-«Stated learner outcomes that have been recently verified as being essential for success in the field. They are a level above general behavioral objectives and are the skills and attitudes used to produce a product or service. Evaluation Variables--A cluster of five evaluation questions used to determine whether rotation context and instructional variables were evaluated in any systematic fashion. f_o_r_r purpose 0' _F_<_>_r_v primary a 9g part at it competer In; documen Lig governm within tl M_e and judo adequoc slrengtl I3 Formative Evaluation-~lnformation provided to program developers for the purpose of modifying and improving the program. Formative Metaevaluation-—A metaevaluation that is intended to guide a primary evaluation (Stufflebeam, l98l, p. l6l). Graduate Medical Education-—Comprised of internships and residencies, that part of the formal training of a physician which concentrates on developing clinical competence in a specialty area. Instructional Variables-~A cluster of four evaluation questions used to document important instructional aspects of the rotation in sports medicine. Licensure (of a physician)—-The process by which an agency of state government grants permission to an individual physician to practice medicine within the jurisdiction of that state (AMA, I983, p. 7). Metaevaluation—-The process of delineating, obtaining, and using descriptive and judgmental information about the utility, practicality, ethics, and technical adequacy of an evaluation in order to guide the evaluation and publicly to report its strengths and weaknesses (Stufflebeam, l98l, p. l5l). Primary Evaluation—-An evaluation that is the subject of metaevaluation (Stufflebeam, l98l, p. l6l). Residency--The period of graduate medical education that begins at graduation from medical school and ends after the educational requirements for one of the medical specialty certifying boards have been completed. The length of graduate medical education varies from three to seven years (Association of American Medical Colleges, l98l, p. 7). Rotation Context Variables-—A cluster of three evaluation questions used to document important aspects of the environment in which the rotation in sports medicine operates. Bgtgtj specialty 0| physicians \ mMMN S_tg_kg immediate interested 3R) M purpose at M completed Chap described limitation The SUlitiorts " 0i the p: madels c reviewed Ch. sUbleets Proceduy l4 Rotations-—Short-term clinical and independent study blocks for learning in specialty areas. Rotations are under the direct supervision of one or more physicians whose control over the resident's learning experiences varies depending on the organizational setting. Stakeholders--Organizations, groups, individuals, and other units in the immediate environment of a program who directly participate in or become interested in a program evaluation and its results (Rossi and Freeman, I982, p. 3l0). Summative Evaluation--Information provided to program consumers for the purpose of judging worth or merit. Summative Metaevaluation-—A study that judges the worth and merit of a completed evaluation (Stufflebeam, l98l, p. l6l). Overview of the Dissertation Chapter One provided the background of the problem, outlined the problem, described the purpose of the study, listed the research questions, discussed limitations of the study and defined key terms. The review of the literature in Chapter Two examines the research that supports the l2 program evaluation questions serving to make up the content areas of the primary evaluation. In addition, several well-known program evaluation models are reviewed and discussed. Finally, the metaevaluation literature is reviewed as it relates to the problem under study. Chapter Three presents the design of the study, including the sample of subjects and their selection, the instruments used in gathering data, the field test procedures, the metaevaluation procedures and statistics to be used. Chapter Four presents the results of the study in tabular and narrative form. Chapter Five summarizes the results of the study. Conclusions are drawn, implications stated, and recommendations for further study suggested. In t reviewed. used in r section cc the final metaeval1 Fir: Three da IERIC); l Systems addition, metaeva Piogrann evaluati. literatgy mugrqny 0v these c SChneeu Chapter Two Review of the Related Literature Introduction In this chapter, two major areas and one minor area of literature are reviewed. The first section reviews program evaluation approaches that have been used in residency programs. Because this literature was found deficient, the section continues with a review of more general models of program evaluation . In the final section, the literature on metaevaluation is reviewed, because a metaevaluation was conducted as part of this study. Evaluation of Graduate Medical Education Programs First reviewed were models for evaluating graduate medical education. Three data bases were reviewed: the Educational Resources Information Center (ERIC); Medline, a computerized division of the Medical Literature and Retrieval Systems (MEDLARS); and the Family Medicine Literature Index (FAMLI). In addition, related dissertations, program evaluation texts, evaluation journals, metaevaluation literature and materials related to the accreditation of residency programs were examined. The focus of this part of the literature review was evaluation of residency rotations. No research in this area was found. Some literature was found, however, on the broader subject of evaluation of residency programs. Only seven studies of evaluation of residency programs were found. Many of these attempted to infer the quality of the program by indirect measures. Schneeweiss (I98l), for example, wrote: Surely the best guide to the success of a given program has to be the performance of its graduates. This can be measured in several ways, including Board examinations and graduate surveys aimed at how well prepared residents felt for their future practice. (p. l086) I5 'The afinferer add: CR success < nototype hauingl the facu recorde To hcaHyc indruvn ihefdinv Tl five Orv ewenp, indyqp helav rehosp T eVOIUa' l6 The Special Requirements for family practice (AMA, I983) make identical use of inference with regard to resident performance on board examinations. They add: The degree of resident attrition and the presence of a critical mass of residents are also factors to be considered in the overall evaluation of a program. (p. 26) Of the several more systematic attempts to collect information about the success of programs, Corley (I976) provides the most extensive compilation of prototype instruments to be used for data collection. He proposes systematic, in- training formative evaluation as a means of providing feedback to the resident and the faculty, to identify problem areas and chronicle resident progress, and to record whether objectives are being met. Corley writes: Residents are receptive to reliable information referable to their accomplishments. Are they, or are they not, acquiring desired skills and competencies? Can they be confident that they are achieving their professional goals? Conscientious faculty share the same anxieties. Are they providing valid learning experiences? Can they be confident of the product of their labors? (p. 3) To help provide answers to these questions, Corley presents a number of locally developed checklists, oral and written exams, questionnaires and evaluation instruments, as well as recommending others that are commercially available (e.g., the Minnesota Outline of Core Knowledge for Family Practice). The instruments are grouped into categories designed to elicit information in five areas: I) implementation and administration of the residency program, 2) entering characteristics of the residents, 3) evaluation of the residents, 4) annual in-training examination, and 5) four evaluations by the residents: a) evaluations of the faculty, b) the hospital rotations, c) other departmental programs, and d) retrospective evaluations by alumni. Two strengths of Corley's book are the comprehensiveness of areas that his evaluations cover and that he supplies a large number of instruments that may be adopted or data are no of instrum even at th the Corie} provide pr program e Corl including procedure residents The recommr second instrume Th residenc develop. fails to Progran W Riograr lulalio c(inside I7 adopted or modified for local use. A weakness, however, is that implementation data are not provided. Thus, it is impossible to determine whether the collection of instruments has been successful in gathering useful formative evaluation data even at the residency program where they are used. A major limitation then, of the Corley book is that while it offers various evaluation resources, it fails to provide pragmatic strategies for design, implementation, and analysis of an overall program evaluation. Corley (I983) revises and adds to his original operational prototype by including even more checklists, questionnaires, rating scales and observation procedures. Corley concentrates primarily on evaluating the performance of residents and faculty during the residency training period. But, he adds, there remains a need for program evaluation: evaluating the residency's merit as an educational program. Each residency has a series of implicit and explicit educational goals. The function of program evaluation--the final task in evaluating residency training—-is to report the extent to which these goals have or have not been achieved. (p. 29f) There are two strengths of Corley's second edition. First, Corley recommends a formative evaluation approach for improving residents' skills. A second strength is the 46 appendices, which contain evaluation tactics, instruments, and principles field tested by Corley. The weakness of the book is that while acknowledging the need for evaluating residencies, Corley devotes less than two pages to the topic. He reviews a model developed for determining the merits of a residency program. But the description fails to provide strategies for the design, implementation, and analysis of the program. Whitman (I983) provides an approach to evaluating family practice residency programs, considering three objects of evaluation (residents, instructors, clinical rotations) and two kinds of evaluation (formative and summative). The evaluations consider five different areas: resident satisfaction with the program, how closely ‘ methods and (for program of the three using variou chart review multiple dot the evaluativ instructors, integrated t- Whitrr he recammv data for ce residents a review). H Ihalithas| I Jelly itSIdencyt collectedc be(tins whe iiressmeny iburces 0f iivnited to Offered Sp Suurceg (1 Inform“, l“IOIIIIGI‘R I8 how closely the teaching methods and content of the program match the intended methods and content, resident achievement, resident application of knowledge, and (for program evaluation only) impact on patients (was care improved?). For each of the three objects, both formative and summative evaluation data are gathered, using various combinations of observation, peer review, self-report, tests, and chart review. Whitman's approach has these strengths. First, he recommends using multiple data sources, which should raise the level of confidence with the results of the evaluation. Second, the same data sources can be used to evaluate residents, instructors, and clinical rotations. Thus, these data sources can be combined and integrated to achieve efficiency in an overall evaluation of the three. Whitman provides no examples of actual instrumentation, however. Rather, he recommends that a particular kind of evaluation technique be used to gather data for certain levels of information requirements (e.g., to determine whether residents are applying course content, he recommends observation and chart review). He offers no implementation data. A major limitation of his approach is that it has never been field tested. Jelly and Friedman (I980) also propose a resident evaluation system for residency training. Their approach, like Whitman's, provides for information to be collected about residents, faculty, and rotations. Implementation of the approach begins when faculty assess residents and residents assess themselves, using "self- assessment tests, clinical rating scales and psychological instruments." Other sources of information are also used. These other sources include (but are not limited to) rating forms from external rotations, in—training exams, information offered spontaneously from patients, preceptors, and others, and additional data sources developed at each site. Jelly and Friedman emphasize the use of information collected and not specific techniques or instruments for collecting the information. This approach reflects their assumption that data-gathering methods such as In of the sci education assessed, gathered. Imp various ir serve as provide c resident‘: going prc far the n A . practice specific of the s friedmc how the One str residen- meeting eulcom inform. iv iivgrar lie ski iitult} 19 such as in-training examinations, rating scales, and simulations have received most of the scholarly attention to date, leaving unexplored the ethical, political, and educational issues surrounding the use of information. Thus the content, or what is assessed, is less at issue than what to do with the information once it has been gathered. Implementation continues with a conference for sharing with the resident the various information collected. The results of the conference are written down to serve as a guide to the resident in making adjustments in learning activities and to provide a record of progress. The reports are then used to make changes in the resident's program and in elements of the program itself. The evaluation is an on- going process, with each outcome of a given cycle becoming an information source for the next cycle. A variation of the approach is reported to have been implemented at a family practice residency, but instrumentation and data are not reported because "the specific structural features and organizational climate that generated the details of the system at this residency program may not apply elsewhere“ (Jelly and Friedman, I980, p. 76). So, it is not known what specific data were collected, or how they were analyzed or used. The approach appears to have several strengths. One strong point is that multiple sources are used to gather information about the resident. Another strength is the periodically scheduled information-sharing meeting. A final strength is that the evaluation is an on-going process, with each outcome of a given cycle being an important data source for the next cycle of information collecting, information sharing, and decision making. Marshall and Davanzo (I983) address one component of an overall formative program evaluation. They developed a procedure and methodology for assessing the skill of family practice faculty teaching in a residency program. To evaluate faculty skill, a sixty-item questionnaire was developed and administered to each resident iI questionnc personal interperso items to r also offer Res member's member) Senior re Particula each lac written question in their specific match is 5: Collect at g QSSeSSI' “id at liflIIOn. 20 resident in the program. Faculty were also asked to rate themselves. The questionnaire was divided into five parts: an overall rating, teaching skills, personal characteristics, interpersonal relationships with patients, and interpersonal relationships with staff. A research assistant read the questionnaire items to residents and recorded their responses on a Likert Scale. Residents could also offer additional comments about the faculty. Residents' responses about each faculty member were averaged. A faculty member's team (who were more likely to have more contact with the faculty member) and those not on the team were differentiated, and responses averaged. Senior residents were also differentiated from first- and second—year residents. Particularly high and low scores and additional comments were then collected on each faculty member and compiled and reported in narrative form. Besides a written narrative summary, a summary of the residents' responses to each questionnaire item was made. The data were used to try to assign faculty to teach in their areas of strength. Thus, the possibility of a match of teaching skills to specific areas of instruction within a residency is optimized (or the lack of such a match is brought to light). Reliability and validity data are not provided in the Marshall and Davanzo study. Therefore, it is difficult to determine whether resident feedback about faculty teaching skills would serve as a valuable component in an overall program evaluation. Several authors reported individual components of program evaluation—~data collection strategies, for example. Spain and Woiwode (I979) reported the results at a survey designed to provide residency program administrators with an assessment of the residency training program and information about the experience and attitudes of the graduated residents, which was to be used for comparison with national trends. Spain and Woiwode provide no instrumentation or procedures, nor do thi however, add suppo Rey taken one and who concludev program (among satisfact rating sc specializ part cor program A residenc they ha over pi procedv T evaluat Iaund. resider who pp evalua reside IRIegy 2| nor do they provide implementation techniques or data analysis. They conclude, however, that follow—up surveys can "help to clarify problems of the residency and add support to those responsible for improving the program." (p. IllO) Reynolds and Chanel (I979) conducted a mail survey of residents who had taken one or more years of psychiatry training at a particular residency program and who had been out of the program one or more years. Reynolds and Chanel concluded that a follow—up study could be a useful means for evaluating a residency program and recommended doing a survey. They designed questionnaires to elicit (among other data) information about residents‘ perceptions of, use of, and satisfaction with their training. The first part of the questionnaire consisted of a rating scale for assessing the effectiveness of training in treatment modalities and specializations required for accreditation and of the total program. The second part consisted of open-ended questions to assess more general aspects of the program. A strength of the study was that respondents, who had participated in the residency program over a period of 20 years, were stratified according to when they had participated. Thus, it was possible to detect differences in the program over time. But, as with Spain and Woiwode, no specific instrumentation, procedures, implementation strategies or data analysis techniques are provided. The focus of the first part of the search of the literature was on the evaluation of residency rotations. Nothing on the evaluation of rotations was found. Some citations were found on the more general subject of the evaluation of residency programs, however. Perhaps the most helpful was Corley (I976, I983), who provided some assistance to program evaluators by offering a large number of evaluation instruments and tactics for evaluating residents and faculty during residency training. Whitman (I983) recommended using multiple data sources and integrating these data into a comprehensive evaluation. Jelly and Friedman they built purpose at routine to source for results of where the evaluatior The inpartan‘ evaluatin evaluatin sufficien‘ Several I evaluatic field tes epproacl the evalv Th make do Studies, Impiehr be impc venom, general fi 22 Friedman (I980) also recommended the use of multiple sources of data. In addition, they built regular meetings of faculty and residents into their approach for the purpose of discussing evaluation data. Jelly and Friedman also proposed a cyclic routine for evaluation, with outcomes from the previous cycle becoming a data source for the next cycle. Marshall and Davanzo (I983) recommended that the results of resident evaluations of faculty be used to assign faculty to teach in areas where they excelled. And Reynolds and Chanel (I979) recommended a two-part evaluation approach, using follow-up surveys. These approaches were found inadequate for several reasons. First, and most importantly, the focus of the approaches reviewed was on the broader subject of evaluating residency programs rather than on the more specific subject of evaluating of residency rotations. Several of these researchers fail to supply sufficient instrumentation, implementation data, or data analysis procedures. Several failed to provide an adequate description of the elements covered in the evaluation. Others provided this information in greater detail but had not done field testing. Finally, several recommended program evaluation strategies as approaches. These strategies were insufficient for use as an overall approach to the evaluation of residency rotations. The literature discussed in this section presented insufficient information to make determinations of the viability of the various approaches. Most were case studies. Some of the approaches were recommended even though they had not been implemented. Several of the articles presented so little information that it would be impossible for an instructional developer to assess the viability of the approach recommended. Because of this lack of detailed information, a review of more general program evaluation models was undertaken. This review is presented next. A revi search at It comprehensi program ew literature is Steele II973 education or Despi‘ just several seems to be If this attempt th deemed irng address thi in it “voiyzed. lite made" Process w imPlemen disadvany. imposed. 23 Program Evaluation Models A review of general program evaluation models became necessary when a search of the literature about graduate medical education failed to identify a comprehensive approach for evaluating residency rotations. There are numerous program evaluation models. Most of these models are found in the evaluation literature for either postsecondary educational programs or social action programs. Steele (I973), for example, reviewed 56 different models for use in evaluating education and social programs. Despite the large number of models available, evaluators appear to depend on just several. In a review of program evaluation models House (I978) writes, "There seems to be fairly good agreement on what are the major models" (p. it). If this is true, one major task for the instructional developer appears to be to attempt the selection of the model that most closely incorporates the criteria deemed important for the particular evaluation setting. Borich and Jemelka (l98l) address this point. The problem for the evaluator becomes one of choosing the conceptualization or model most appropriate to the evaluation problem. Because the evaluation models appearing in the literature are purposely general (so as to be applicable to a wide variety of educational problems), the task of choosing that conceptualization of evaluation most appropriate to a specific purpose becomes even more arduous. (p. I62) In this section, four of the frequently cited models will be reviewed and analyzed. For each, the description will be in four parts: the major concepts of the model will be identified and explained; the various steps in the evaluation process will be reviewed; a graduate medical education example illustrating the implementation of the model will be provided; and the advantages and disadvantages of the model for evaluating the rotation in sports medicine will be proposed. The n that each originally d an educatic the Itterat models wev in ‘Norther in suiticiev are those v Mt The State‘s C designed procedure checklists and stanc pracedun evaluatic observatt 0i Progrr Sta hoW to r that pro based n Occurrev based q 24 The models were selected on the basis of several criteria. One criterion was that each model be a program evaluation model. As such, each model was originally developed to provide information about or to judge the merit or worth of an educational program. A second criterion was that each model be prominent in the literature of educational program evaluation. To satisfy this criterion, the models were selected from a compilation of educational program evaluation models in Worthen and Sanders (l973). A final criterion was that each model be described in sufficient detail that a review and analysis could be done. The models selected are those of Stake (I967), Stufflebeam (l97l), Alkin (I969) and Provus (l969). The Countenance of Educational Evaluation Model (Stake, I967) The first program evaluation model to be reviewed and analyzed is Robert E. Stake‘s Countenance of Educational Evaluation Model (I967) (Figure l). Stake designed the model to attempt to persuade educators to rely on more formal procedures for evaluation. Stake acknowledged that such formal procedures as checklists, structured visits by peers, program comparisons with similar programs and standardized testing of students had been documented, but noted that these procedures were little used because the procedures were not tailored to individual evaluations. Stake also noted that informal evaluation procedures such as casual observation of programs, implicit goals, intuitive norms, and subjective judgments of programs frequently yielded superficial and distorted results. Stake's model does not dictate what should be measured in an evaluation nor how to measure it. The Countenance of Educational Evaluation is a general model that provides a framework for individual evaluation studies. It is an objectives- based model in that it compares that which was intended with what actually occurred. Objectives-based models focus on assessing outcomes. In an objectives- based approach, evaluation consists of comparing actual performance against the prespecified levels of performance. in the Stake model, the evaluator is a specialist the progrt The being evc continger in the r “transact The being 8V1 the goal: Th the prog prograzr methods motriee Progran and ins‘ A Dmgrar interes studen- melho. may b when SChool 25 specialist who collects, processes, and interprets data and makes judgments about the program. The major concepts in Stake's model include a rationale for the program being evaluated, description and judgment data matrices, and the dimensions of contingency and congruency. Several additional important concepts are imbedded in the data matrices. Stake refers to these concepts as "antecedents," "transactions," and "outcomes." The rationale describes the philosophy and basic purposes of the program being evaluated. The purpose of the rationale is to provide one basis for evaluating the goals of the program. There are two data matrices in Stake's model. One matrix is used to describe the program being evaluated; the second, to judge the program. To evaluate a program, various data are gathered from different sources and by different methods. When the evaluator seeks descriptive or judgment data to complete the matrices, three information sources are used: conditions for entry into the program (antecedents), instructional methods used in the program (transactions), and instructional outcomes of the program. Antecedents are conditions affecting students before their entry into the program being evaluated—for example, their previous experience, aptitudes, and interests. instructional methods or transactions are the variety of encounters of students with teachers, materials, or other students. A lecture is an instructional method or transaction. Outcomes are the consequences of the transactions. They may be immediate, as with a certain score on an examination, or long range, as when a student ultimately selects a college major on the basis of a certain high school course. RATION‘ One r transq that c ColUrr The Ir Ore er of ti 26 Figure l The Countenance of Educational Evaluation Model iNTENTS OBSERVATIONS STANDARDS JUDGMENTS l ANTECEDENTS RATlONALE TRANSACTIONS OUTCOMES l DESCRIPTION MATRlX JUDGMENT MATRlX There are two ways of processing evaluation data in the description matrix. One way is to determine what Stake calls "contingencies" among antecedents, transactions, and outcomes. Contingencies are a series of "if—then" statements that compare entry conditions, instructional methods, and outcomes. in the intents column of the description matrix, contingencies are called "logical contingencies." The logic is that if certain entry conditions exist and certain instructional methods are employed, then a certain outcome is to be expected. in the observation column of the description matrix, contingencies are called "empirical contingencies" because ' processinr the descr goals and happened matrix it listed in congruen is the co Th When co general lithe e Thus, tr instruct similar Specifie descrip- evaluat matrix Slandar lr deterT Often ieslder than might 27 because these contingencies are what actually occurred. The second way of processing evaluation data in the description matrix is congruency. The columns in the description matrix are labeled "intents" and "Observations." To Stake, program goals and intents are synonymous. Observations are descriptions of what actually happened in the program. The logic is that for each goal listed in the description matrix in the intents column, there is a corresponding observation that will be listed in the observation column. The observations determine the degree of congruency between the original goals and the observed results. Congruency, then, is the comparison of what was planned with what actually happened. The columns in the judgment matrix are labeled "Standards" and "Judgments." When completing the standards column, the evaluator has the option of specifying general standards of quality or standards specific to the program being evaluated. if the evaluator specifies general standards, a relative comparison is necessary. Thus, to arrive at judgments, the evaluator would compare entry conditions, instructional methods, and outcomes of the program being evaluated to other similar programs. These data come from the description matrix. if the evaluator specifies specific standards, on absolute comparison must be made between the descriptive data gathered and the standards of excellence set. Whether the evaluator uses relative or absolute criteria, the judgment column in the judgment matrix is completed by comparing data from the description matrix to the standards. implementation of the Stake model begins when the evaluator attempts to determine the rationale of the program to be evaluated. Program rationales are often implied. Thus, it could be necessary for the evaluator to elicit from a residency director, for example, a statement of the basic purposes of the residency program. in this case, part of the rationale for a residency program or rotation might be to satisfy the requirements of a specialty board while meeting the changing the speCi commUnl has been imp (intents) stateme methods Thus, if students have al The din examln these p betwee instruc halt ol should 28 changing health care needs of the community. With the specific requirements of the specialty board delineated, and the specific changing health care needs of the community determined, one basis for judging the individual goals of the program has been established. implementation continues with the development of three types of goals (intents). These goals are written out as planning statements. The three planning statements describe intended entry conditions of students (antecedents), intended methods of instruction to be used (transactions), and intended learning outcomes. Thus, the residency director might plan to accept as residents only those medical students who graduate in the upper half of their classes. The director may plan to have all new residents begin their service with four months of internal medicine. The director may plan for all the residents to achieve a certain passing grade on an examination at the end of the four-month period. When the evaluator examines these plans, he examines the contingencies. He would examine the contingencies between entry conditions and methods of instruction, and between methods of instruction and outcomes. He notes that if residents are selected from the upper half of their classes and given four months of internal medicine service, then they should pass the examination. Next, the evaluator determines congruence between the goals and observations for entry conditions, instructional methods, and outcomes. He may find that one resident was accepted into the residency program who had graduated in the lower half of his medical school class. A second resident fell ill and missed nearly a month on the service. But all residents passed the exam at or above the criterion level. The evaluator continues to check congruence between what was planned and what actually took place. This process aids in identifying program weaknesses. Here, the evaluator may discover that the exam given at the end of the service has less relevance to the service than was originally thought. The judgment Descripti their spe last five while SOt is soccer Th evaluate adaptab iramewi observa congrue Another summal improvi value a now us: require the re would r0tatir Within it "in Workir ctntir 29 The evaluator continues implementation of Stake's model when making judgments of the residency program based on data from the description matrix. Description of the program may indicate that all residents are expected to pass their specialty board examinations. And data may indicate that all residents in the last five years have passed their board exams. Thus, the evaluator may report that while some elements of the residency program require improvement, the program is succeeding in meeting its goals and is operating in congruence with its rationale. There would be both advantages and disadvantages to using Stake's model to evaluate the rotation in sports medicine. One advantage is that the model is adaptable to a rotation setting. The model is adaptable because it is a general framework that emphasizes the systematic collection of data (goals and observations), provides general methods for analyzing the data (contingencies and congruency) and provides two methods for judging a rotation (absolute or relative). Another advantage is that it makes it possible to make both formative and summative evaluations of the rotation. Thus, the model could be helpful in improving program components as well as making an ultimate judgment of the value of the rotation. The Stake model is more formal and objective than methods now used in evaluating the rotation in sports medicine. The Stake model would also require the setting of specific objectives. This would indicate areas of priority in the rotation as well as areas of success and lack of success. The Stake model would also provide immediate information about educational outcomes of the rotation. Finally, it is possible, with the Stake model, to employ an evaluator from within the College of Human Medicine or the Department of Family Practice. if on "inside" evaluator could be identified, less time would be required to build a working relationship. in addition, the inside evaluator would be available on a continuing basis. The medicine. models 5 behaviorc is measo designs. emphasis of the ra odminist Us: rotation lmpleme scope oi to provi construr not pror matrix. 81 PUrpose iniormr 0 desc C(tight: Cinditi tVOiUQ' it! use % 30 There are disadvantages in applying the Stake model to the rotation in sports medicine. Attempts to specify behavioral objectives when using objectives—based models such as Stake's can result in oversimplified objectives. Specifying behavioral objectives can also reduce the stated purposes of a rotation to only what is measurable. Prespecified objectives often require prespecified evaluation designs. Such evaluation designs may be unresponsive to shifts in a rotation’s emphasis. Stake's model would also be not well suited for evaluating components of the rotation not definable in performance terms (e.g., rotation organization and administration, scope and sequence of instruction). Use of the Stake model would require a good deal of the evaluator‘s and the rotation director's time—-more time than they are likely to be able to give. implementing the Stake model can also be expensive, depending on the depth and scope of the evaluation. But perhaps the greatest disadvantage is that Stake fails to provide adequate methodology for obtaining information on many of the key constructs of the model. Two critical areas for which adequate methodology are not provided--standard setting and judgment making--are located in the judgment matrix. Stake's Countenance of Educational Evaluation model was derived for the purpose of describing and judging educational programs by a formal rather than informal process. The essential components of the model are a program rationale, a description matrix, a judgment matrix, dimensions of contingencies and congruencies, and three different data sources. The data sources are entry conditions, instructional methods, and outcomes. Stake's model is applicable to the evaluation of rotations, but the disadvantages of the model render it less than ideal for use in evaluating the rotation in sports medicine. CIPP Evaluation Model (Stufflebeam, l97l) The next model to be reviewed and analyzed is that of Daniel L. Stufflebeam. Stuiilebe< which the model: at developm and Secor that time The ClPl does not will be u in provides model, ‘ Rather, decisior Er providir it is as bt the . T may be model define. lniorrr EVQILK come; We 3i Stufflebeam's model is often referred to as the "CIPP" model. The acronym by which the model is known stands for the four levels of evaluation contained in the model: of context, of input, of process and of product. The stimulus for the development of the model was the evaluation requirement in the 1965 Elementary and Secondary Education Act (ESEA). Program evaluation models that prevailed at that time were thought to be unresponsive to the special needs of ESEA programs. The CIPP model is a decision—management or decision—making model. As such, it does not assess merit directly. instead, its focus is on gathering information that will be used by decision makers. in the CIPP model, as in Stake's model, the evaluator is a specialist who provides evaluation information. But unlike the role of the evaluator in the Stake model, the evaluator in the CIPP model makes no judgments about the program. Rather, the evaluator in the CIPP model reports evaluation data to the program decision-makers. Evaluation using the CIPP model is a process of delineating, obtaining and providing useful information to be used by the decision maker to judge alternatives. it is assumed that the merit of the various alternatives is taken into consideration by the decision maker. The CIPP model is presented in Figure 2. The major concepts in the CIPP model are the four types of evaluation that may be done and the three steps that make up the process of evaluation. The model provides information for the purpose of making decisions. Stufflebeam defines four kinds of decisions and provides a type of evaluation that provides information for planning changes needed in the educational system. Context evaluation also provides a basis for judging eventual outcomes. The purpose of context evaluation is to diagnose problems underlying needs in the system and to judge whether currently proposed objectives are sufficiently responsive to the assessed such as sl lnp change a program strategie is to ide procedur projects also co strategi Pr implem process implem intonnr Proces: Proarar rototic PIOdur contin reloci than Unlhlr descr ildgn 32 assessed needs. Context evaluation is accomplished through a variety of means, such as surveys, interviews, and diagnostic tests. input evaluation is a type of evaluation used in the CIPP model to structure change activities identified in context evaluation and to provide a basis for judging program implementation. in input evaluation, possible sources of support, solution strategies, and procedural designs are identified. The purpose of input evaluation is to identify and assess capabilities within the system, alternative strategies, and procedural designs. input evaluation is conducted with such strategies as pilot projects, literature searches, and visits to exemplary programs. input evaluation is also conducted by inventorying and analyzing available resources, solution strategies, and procedural designs for relevance. Process evaluation is a type of evaluation in the CIPP model that is used to implement and refine the design and procedures of a program. The purpose of process evaluation is to identify or predict defects in the procedural design or implementation of a program. Other purposes of process evaluation are to provide information for the preprogrammed decisions and to record and judge procedures. Process evaluation is carried out by specifying actual processes to be used in the program, by interacting with and observing activities of the program staff (e.g., rotation director), and by observing barriers to prespecified procedures. The final type of evaluation in the CIPP model is product evaluation. Product evaluation is used to make decisions about a program regarding continuation, termination, and modification. Product evaluation is also used to refocus a change activity in a program and to provide a record of the effects a program has made. These effects may be both positive and negative, intended and unintended. The purpose of product evaluation in the CIPP model is to collect descriptions and judgments of outcomes and to relate the descriptions and judgments to context, input, and process information. A final purpose of product evaluatic evaluatic criteria, judgmen‘ Un individuc services organizc evaluati resident educatir 33 evaluation is to determine the worth of the program being evaluated. Product evaluation is accomplished by operationally defining and measuring outcome criteria, by performing both qualitative and quantitative analyses and by obtaining judgments of the program from stakeholders (e.g., students, administrators). Under the CIPP model, evaluation is not concentrated as much on guiding an individual program evaluation study as it is in providing ongoing cyclical evaluation services to decision makers in an organization. According to the ClPP model, an organization (e.g., a residency program) periodically needs to undergo a context evaluation. Such an evaluation would examine the needs of students (e.g., residents) and clients (e.g., patients), expose new opportunities (e.g., advanced educational technologies), collect data about problems in the organization, and Figure 2 The ClPP Model I- —————— -) ——————————————— t l DECISIONS i liliiliillllAcnvrrrEs 1 l -EVALUATIONS J Nromobliisticj Change cmzz>rv It Incremental Change l l l T | | | MOZ)IO Homeosenrlc i Chan-e u. nilghrene Pers stence Installation i ll Termination L_._. ___ .__ ._._ ___ ___ __..__..__ ______..__ __.___ __..__. ___ __ ___ ___ .__ .__ __ __. __ __ ._. assess orgar would, ideail lino soiutic the underta' solution str- sirategy bet be conducti product evc strategy re reached, thi evaluations level of pr longer wor‘ continue w At er are decisio system are change iss l5 Gpparen “0i Opparr SOlvtion is the Sister traduct er The Process r delermini 31+ assess organizational goals and priorities. The results of a context evaluation would, ideally, lead to decisions about whether to introduce changes in the system. if no solutions were apparent for the changes required, the CIPP model indicates the undertaking of an input evaluation. The input evaluation would result in a solution strategy's being selected by the decision maker. To test the solution strategy before the strategy is installed in the system, a process evaluation would be conducted, to detect any defects in the design of the strategy. Finally, a product evaluation would be performed on the strategy to determine whether the strategy resulted in the desired level of performance. if the desired level was reached, the strategy would be installed in the system. if not, process and product evaluations would be repeated until the modified strategy resulted in the desired level of performance or until the decision maker decided the strategy was no longer worthy of further effort. The evaluation cycle under the CIPP model would continue with the next periodic context evaluation. At each stage of evaluation (i.e., context, input, process, and product) there are decisions to be made. One such decision justifies whether or not changes in the system are warranted. This decision comes as a result of context evaluation. if a change is warranted, a decision must be made about whether a satisfactory solution is apparent. if a solution is apparent, it is installed in the system. if a solution is not apparent, an input evaluation should be done to identify a solution. When a solution is identified by input evaluation, a decision must be made to install it in the system immediately or to test the solution by submitting it to process and product evaluation. The remaining major concepts of the CIPP model are the three steps in the process of evaluating: delineating, obtaining, and providing. Delineating is determining what information will be collected, and is a collaborative effort of the evaluator and the decision maker. Obtaining information, the second step of the proc final step, to report i the results and providr impact. lmplr to be unde: particular regular ca relating or a product whether evaluatior the intent rotation d The Providing etaluotio residency Preceptol ”lot be informat rotation leaching ieSidenc coiieCiic 35 of the process of evaluation, is done by the evaluator. Providing information, the final step, is also done by the evaluator. This process essentially determines how to report the information gathered. The evaluator does three things in reporting the results of an evaluation: prepares the evaluation reports, disseminates them, and provides for follow-up activities to help to ensure that the evaluation has some impact. implementation of the ClPP model begins by selecting the type of evaluation to be undertaken. For example, a residency director and the rotation director of a particular rotation may agree that some residents require experiences beyond the regular curriculum. Product evaluation is intended to provide information for relating outcome information to prespecified objectives. Having thus chosen to do a product evaluation, the residency director and rotation director must decide whether the evaluation will serve a decision—making purpose (formative evaluation), on accountability purpose (summative evaluation), or both. Because the intent of the evaluation is to improve the rotation, the residency director and rotation director agree that the evaluation should serve a decision-making purpose. The next steps in implementation are defining the delineating, obtaining and providing questions to be addressed in a decision-making (formative) product evaluation. "Delineating" defines the questions to be asked. in the example, the residency director and rotation director may ask questions about additional preceptor and physical plant resources or alternative instructional strategies that may be employed in the rotation. in the "obtaining" step, how the needed information will be collected is decided upon. Here the residency director and rotation director would specify information sources such as hospital administrators, teaching faculty at the affiliated medical school, and current residents in the residency program. The residency director and rotation director also specify data— collection instruments and procedures for collecting and analyzing data. in would 5 the exa themse prograr is unlit compel special i sports system above, cancer result, medic Was ir evalur provic decisi kept Prose Conie °PPor State Pravi bite: impr 36 in the third step, "providing," the residency director and rotation director would specify how and to whom the specified information would be reported. in the example, the residency director and rotation director may decide that besides themselves, the information would be of benefit to all residents in the residency program and the rotation directors from all other services. in the example cited, it is unlikely that the residency director and rotation director would have all the competencies needed to do the evaluation: it would be necessary to consult a specialist. There are advantages in using the ClPP model for evaluating the rotation in sports medicine. One advantage is that the model encourages continuous and systematic evaluation in planning and implementing a program. As described above, the model is geared more to a "systems“ view of education. Thus, the model concentrates on providing ongoing evaluation services to decision makers. As a result, the model would probably not be applied just to the rotation in sports medicine, but to the entire family practice residency program. if the ClPP model was installed in the entire residency program, however, continuous and systematic evaluation of all rotations in the residency would be encouraged. The model also provides a rationale for helping the rotation directors to be accountable for decisions made in implementing the rotation. The rationale is the result of records kept during each evaluation. These records document objectives, strategies, processes, attainments, and other data upon which decisions were based. Because context evaluation is a continuing process in the ClPP model, needs and opportunities in the dynamic interest area of sports medicine could be systematically identified and incorporated into the rotation. The model also provides for a narrowly focused evaluation (as in the example cited) or for a broad— based evaluation of an entire rotation or residency program. Finally, in addition to improving the rotation, the ClPP model can be used to judge its overall worth. Bi rotatior implemu provide act rat Lincoln decisio the ba: Perhap rotatic admini time ( and 0: avatar evaior oi l9i ior m comp trace idelir the it of it How idea 37 But there are also disadvantages to using the ClPP model to evaluate the rotation in sports medicine. The first disadvantage is that specific steps for implementing the ClPP model (e.g., instructions, procedures, forms) are not provided. A second disadvantage is that the model assumes that decision makers act rationally and that the decision-making process is an Open process (Cuba and Lincoln, l983, p. l6). Decision makers do not always act rationally and the decision-making process is often political. When the evaluation decision is made on the basis of political rather than rational motives, the decision can be biased. Perhaps the greatest disadvantages to using the CIPP model to evaluate the rotation in sports medicine, however, are that the model is hard to implement and administer, is expensive to maintain, and takes a good deal of the decision makers' time (rotation directors and residency directors). Difficulties in implementation and administration, as well as the high expense, often ensue from the several evaluations taking place in the system at one time. The ClPP program evaluation model was designed to provide responsive evaluation for programs sponsored by the Elementary and Secondary Education Act of 1965. The basic purpose of the model is to provide knowledge and a value base for making and defending decisions about an educational program. The essential components of the CIPP model are the four types of evaluation (context, input, process, and product) and the three steps that make up the process of evaluation (delineating information needs, obtaining the required information, and providing the information to the necessary audienCes). The three-step process is used in each of the four types of evaluation. The ClPP model can be used in a rotation setting. However, the disadvantages associated with the ClPP model make it, too, less than ideally suited for evaluating the rotation in sports medicine. CSE Evaluation Model (Alkin, l969) The next model to be reviewed and analyzed is that of Marvin C. Alkin. Alkin's ! because Center Elemen' respons L' manage makers CIPP r Rather considr in the decisic the es COUiSG what , Progrr Which "PTOgI prog- “Dora and c Plese 0m Uppp 38 Alkin's model of program evaluation is sometimes referred to as the "CSE model" because it was developed at the Center for the Study of Evaluation at UCLA. The Center was sponsored by the U5. Office of Education under Title W of the Elementary and Secondary Education Act (ESEA) of l965, and was charged with the responsibility of developing a basis for measuring program effectiveness. Like the ClPP model, the CSE model of program evaluation is a decision— management model. Thus, the model is intended to supply information to decision makers for the purpose of improving the effectiveness of a program. Like the ClPP model, the CSE model does not assess educational outcomes directly. Rather, it is assumed that the value of alternative outcomes is taken into consideration by the decision maker in choosing among them. In the CSE model, as in the ClPP model, the evaluator is a specialist who provides information to decision makers. The CSE model differs from the CIPP model, however, in that the evaluator in the CSE model is expected to make judgments about various courses of action. The major concepts in the CSE model are the five areas of evaluation, or what Alkin calls "need areas,” and the four-step evaluation process that provides program decision makers with information for decision-making. The five areas in which evaluation may be done are what Alkin refers to as "systems assessment," "program planning," "program implementation," "program improvement," and "program certification." The steps in the evaluation process are selecting the appropriate area to evaluate, selecting appropriate information to seek, collecting and analyzing the information, and reporting the information. The CSE model is presented in Figure 3. The first of Alkin's need areas is systems assessment. Systems assessment is a means of determining the range and specificity of educational objectives appropriate for a particular situation. in systems assessment, the entire educatio (Systems model.) Th attempt planning process The rat etiectir oi inpu T implert to wh’ progra the Cl' Evalur descri the Sr Unanl evaiu and a Ptovi Wart (Prov 39 educational system is evaluated in terms of its broad goals and objectives. (Systems assessment in the CSE model is similar to context evaluation in the ClPP model.) The second need area is program planning. Evaluation for program planning attempts to provide information that will enable the decision maker to make planning decisions. Planning decisions are used to select among alternative processes to determine which alternative should be introduced into the system. The role of the evaluator in program planning is to assess the potential relative effectiveness of different courses of action. (Program planning is the equivalent of input evaluation in the ClPP model.) The third need area in the CSE model is the evaluation of program implementation. An evaluation of program implementation determines the extent to which the implemented program meets the description formulated in the program planning. (Program implementation corresponds to process evaluation in the CIPP model.) The next need area in the CSE model Alkin calls "program improvement." Evaluations of program improvement provide two kinds of data: The first kind describes the extent to which a program is meeting its prescribed objectives, and the second assesses the impact of a program on other processes or programs. Unanticipated program outcomes are often detected during program improvement evaluations. (Program improvement evaluations are used to modify the program, and are similar to process evaluations in the ClPP model.) The final need area is called "program certification." Program certification provides information for decision makers to use in making judgments about the worth or merit of a program and the potential generalizabiiity of the program. (Program certification is comparable to product evaluation in the ClPP model.) The planning, model's ' processr and prog areas, 5 educatio medical last thr prograr Br the mo determ? decisior selectir the ev develop data,c (0 the model, evaluc variou the in the C Vlewe Unite 40 The evaluation need areas Alkin calls "systems assessment," "program planning," and "program certification" are similar, respectively, to the ClPP model's context evaluation, input evaluation, and product evaluation. However, process evaluation in the ClPP model is conceptualized as program implementation and program improvement in the CSE model. in the CSE model, the first two need areas, systems assessment and program planning, are used to evaluate entire educational systems (e.g., a residency program affiliated with a university-based medical school). By contrast, the evaluation of instructional programs involves the last three needs areas: program implementation, program improvement, and program certification. Besides the areas of evaluation in the CSE model, the other major concepts in the model are the steps in the evaluation process. The first step is the determination of a need area for evaluation. This determination is made by the decision maker, not the evaluator. The second step in the evaluation process is the selection of information appropriate to the need area. This task is carried out by the evaluator and includes such components as designing the evaluation and developing appropriate instruments. The third step is the collection and analysis of data, also tasks of the evaluator. The final step is the reporting of summary data to the decision maker by the evaluator. There is a major difference in the CSE model, and the ClPP model at this step. in the CSE model, the role of the evaluator requires the evaluator to make judgments or recommendations about various decision alternatives to the decision maker. implementation of the CSE program evaluation model is nearly identical to the implementation of the ClPP model. There are subtle differences, however. in the CSE model, the needs areas of systems assessment and program planning are viewed as evaluations of entire systems (e.g., a residency program affiliated with a university—based medical school). The last three needs areas, program impemer embdk reddency inher emhmh had beer mdd. h idmma CErn emhm’ Isl implementation, program improvement and program certification are viewed as evaluations involving instructional programs (e.g., a particular rotation within a residency). The evaluation of an individual program like the evaluation suggested in the example in the ClPP model would ordinarily not be undertaken until evaluation in the first two need areas (systems assessment and program planning) had been accomplished (Figure 3). No such assumptions are made in the ClPP model. in the CIPP example, a product evaluation was undertaken to provide information on what additional experiences would benefit residents. Under the CSE model, this type of evaluation would be called a program improvement evaluation. in the CSE model, the evaluator would take an interventionist role in Figure 3 CSE Model Evaluating Educational Syatoma Systems . Program Assessment Decrsion Planning L i l l Evaluating Instructional Programs Program ' Implementation Decisions Decision Program Certification Decision Decision Program Improvement T attempll the rota present rotation residenc Ti the wt: evaluat' the rub cyclica would r would r system help to i disadv. A seri maker iniorn CSE n The p CSE Possit ditiic time. the r 42 attempting to provide information that would lead to immediate modification of the rotation. The evaluator would identify problems, collect and analyze data, and present the data immediately to the decision makers (residency director and rotation director). The evaluator would also make recommendations to the residency director and rotation director about potential decisions. There are advantages to using the CSE program evaluation model to evaluate the rotation in sports medicine. One advantage is that using the CSE program evaluation model would force the specification of clear instructional objectives for the rotation. Another advantage is that evaluation in the CSE model is part of a cyclical process of program managment. Thus, decisions made about the rotation would continually be evaluated to the betterment of the rotation. The CSE model would also help put the rotation in sports medicine in sharper focus in the greater system (the residency program). This could help to eliminate educational gaps and help to define sports medicine as a family practice interest area. The CSE program evaluation model presents some disadvantages. One disadvantage is that little methodology is provided for conducting the evaluation. A serious disadvantage is that in the CSE model, the evaluator, and the decision maker (e.g., rotation director) have a shared responsibility. The evaluator provides information. The decision maker uses the information to make decisions. in the CSE model, the evaluator is sometimes called on to recommend courses of action. The problem is that no one person is responsible for program improvement. The CSE model shares with the ClPP model several additional disadvantages for possible use in evaluating the rotation in sports medicine. The CSE model is difficult and complex to implement and administer, is expensive to maintain, and time—consuming for the rotation director and an evaluator. The CSE model implies the requirement of an evaluation specialist. Thr eitectivr decision evaluate The essr the tour assessm improve evaluati individc program The prr tive ev and am is possi oi the Sports Discre \ Pittsb comm other in edi halo city 143 The CSE program evaluation model was designed to assess and improve the effectiveness of Elementary and Secondary education Act (ESEA) programs. A decision-management model, the CSE program evaluation model permits the evaluator more latitude in influencing decision alternatives than similar models. The essential components of the CSE model are the five areas of evaluation and the four-step evaluation process. Alkin calls the five evaluation areas "systems assessment," "program planning," "program implementation," "program improvement," and "program certification." The first two evaluation areas involve evaluating the entire educational system. The last three evaluation areas address individual programs in the system. in the CSE model, utilizing one of the three program evaluations assumes that the first two have already been accomplished. The process of evaluation includes the following steps: determining which of the five evaluations to do, selecting appropriate information to be collected, collecting and analyzing the information, and reporting summary data to decision makers. it is possible to use the CSE model in a rotation setting. However, the disadvantages of the model render this model less than ideally suited to evaluate the rotation in sports medicine. Discrepancy Evaluation Model (Provus, l969) The final program evaluation model to be reviewed and analyzed is the Pittsburgh Evaluation Model developed by Malcolm Provus. The model is more commonly known as the Discrepancy Model for its reliance on identifying differences between previously set standards (objectives) and actual performance in educational programs. The Discrepancy Model was designed to attempt to apply evaluation theory and management theory to the evaluation of programs in large city school districts. The model attempts to determine whether to improve, maintain, or terminate an educational program. The impetus and support for the development of the model was the l965 Elementary and Secondary Education Act (ESEA). Ti in that ' them it educati purpose consist: the ob: decisio odditio organiz implen Discrej special design stages Provo: "COST- ln the and e. Provo Qipec orll r and r DTSCr ilacr illitgj 1+4 The Discrepancy Model has somewhat of a decision-management orientation in that the model requires the evaluator to aid program administrators by supplying them information. But the Discrepancy Model is objectives-based—-focusing on educational outcomes. Objectives-based models have as a premise that important purposes of education can be expressed by behavioral objectives. Evaluation then consists of comparing actual student performance against the prespecified levels in the objectives. On the basis of the degree to which objectives are achieved, decisions are made to continue, modify, or terminate educational programs. In addition, in the Discrepancy Model, Provus includes process objectives for program organization and procedure. Here, evaluation consists of comparing the actual implementation of the program against the prespecified objectives. ln the Discrepancy Model, what is referred to as the "evaluator" is actually a team of specialists who counsel program administrators in such technical areas as research design, data processing, and test construction. The major concepts of the Discrepancy Model include five kinds of evaluation stages and a three-step process for carrying out each stage of the evaluation. Provus calls the evaluation stages "design," "installation," "process," "product," and "cost-benefit." The cost~benefit stage is not applicable to all program evaluations. In the first of the three steps in the evaluation process the program administrators and evaluation staff come to an agreement about objectives (called "standards" by Provus). Next, the evaluator determines whether there are discrepancies between aspects of the program and the objectives governing those aspects. Finally, using any discrepancy information, the evaluator identifies weaknesses in the program and recommends solutions. This process is carried out in each evaluation stage. Discrepancy information at any evaluation stage leads to a decision whether to proceed to the next stage of evaluation or to change either program objectives or program operations. Advancement through the various stages of evaluation is coming at the l to terrr m L_ "desig evalo: direct comp Mode leocl objec also to r low in t 45 contingent upon attaining congruence between program operations and objectives at the previous stage. If congruence is not possible, the recommendation is made to terminate the program. The Discrepancy Model is presented in Figure 4. Figure 4 The Discrepancy Model terminate terminate terminate C/B Analysis based on new in the first of the five evaluation stages in the Provus Discrepancy Model, the "design" or "definition" stage, the program description is documented. The evaluator obtains from the program staff (e.g., residency director, rotation directors, other instructors) comprehensive descriptions of various program components. The program components are often referred to in the Discrepancy Model as inputs (e.g., facilities, students), processes (e.g., student interactions with teaching staff and instruction) and outputs (e.g., the achievement of prespecified objectives). The descriptions are compared to the program definition, which has also been generated by the program staff. Discrepancies are then noted and used to modify the definition of the program so that it is congruent with the actual input, process, and output components. Program definition (stage one evaluation) in the Discrepancy Model attempts to gather approximately the same information as does input evaluation in the ClPP model, systems assessment in the CSl matrix off In tt observatior being cont intended. the progra is the equi evaluation The componen in expectr stage is t process st the progn The cOmponer Objective lest behi Program. of sum Similar t. ntOdel, (1| The stage, i: compen 1+6 in the CSE model, and contingencies in the intents column of the description matrix of Stake's model. in the second stage of evaluation, "installation," the evaluator makes observations of the components of the program (inputs, processes, outputs) as it is being conducted to determine whether the program is being implemented as intended. Discrepancy information at this stage is used to modify the way in which the program is being implemented. The installation stage in the Discrepancy Model is the equivalent of process evaluation in the ClPP model, program implementation evaluation in the CSE model, and the concept of congruency in the Stake model. The third stage," process evaluation," determines whether program components (inputs, processes, and outputs) are resulting in intermediate changes in expected student behavior. Discrepancy information generated at the process stage is used to modify either the program components or the objectives. The process stage of evaluation is similar to the concept of formative evaluation and the program improvement stage of the CSE model. The fourth stage, "product evaluation," determines whether program components (inputs, processes and outputs) are leading students to the ultimate objectives specified for the program. During product evaluation pretest and post- test behavior change measures are often used to determine achievement of program objectives. This stage of the Discrepancy Model is similar to the concept of summative evaluation. Product evaluation in the Discrepancy Model is also similar to product evaluation in the ClPP Model, program certification in the CSE model, and the concept of outcomes in the Stake model. The fifth stage of evaluation in the Discrepancy Model, the "cost-benefit" stage, is not always applicable in a program evaluation. To be applicable, competing programs must be available. The cost-benefit analysis then determines which of cost. in Stuttlebc dhector addition some to a stage ' Fa first stc hutaHat agreemr evaluati A: rotatior achievir second identil) requirir would ' be mar make 1 Choice Selecti reside: dltecti 47 which of the alternative programs achieves the common objectives at the lowest cost. In the previously cited example used to illustrate implementation of Stufflebeam's CIPP and Alkin's CSE models, a residency director and rotation director agreed that some residents in a particular rotation required experiences in addition to the regular curriculum. in the Discrepancy Model, this need would come to light as a result of discrepancy information provided by the evaluator from a stage three (process) evaluation. For purposes of this example, it is assumed that evaluation has begun at the first stage, program definition, and proceeded through the second stage, program installation. it is further assumed that the first step in the process of evaluating, agreement on objectives between the residency director, rotation director, and evaluator has already been achieved. As a result of process (stage three) evaluation, the residency director and rotation director would learn from the evaluator that some residents were not achieving the objectives specified for them in the rotation. This constitutes the second step in the evaluation process. in the third step, the evaluator would identify the weaknesses in the rotation as it exists that result in some residents requiring additional experiences to attain prespecified objectives. The evaluator would then inform the residency director and rotation director of the decisions to be made to improve the program. The residency director and rotation director make the decisions on the basis of the information supplied by the evaluator. The choice between decision alternatives to improve the rotation as well as the selection of criteria for generating the alternatives is the responsibility of the residency director and rotation director. The residency director and rotation director could also decide to change the objectives to meet the observed behavior. Although ll decide to tr additional 6 objectivesl Once the evaluat objectives« would procr With reviewed a purposes 0' Discrepanc advantage program 5‘ model. Tr orprograrr lor the dir desirabilit} roles of b clearly de lormative lldgmentr Ther evdlquej eVGlUGte 48 Although it is unlikely, the residency director and rotation director could also decide to terminate the rotation entirely. The most likely course of action is that additional experiences could be identified to help all residents attain the rotation objectives (e.g., writing a short research paper). Once the change in the rotations' objectives or operations have been made, the evaluator would determine whether there still were discrepancies between objectives and observations. if no further discrepancies were noted, the evaluation would proceed to stage four, product evaluation. With the Discrepancy Model, as with the previous program evaluation models reviewed and analyzed, there would be both advantages and disadvantages for the purposes of evaluating the rotation in sports medicine. One advantage of the Discrepancy Model is that it forces the statement of explicit objectives. A second advantage is that a high degree of communication between the evaluator and program staff (e.g., residency director and rotation director) is built into the model. This model is flexible in its provision for adjustment of program processes or program objectives. Unlike other objectives-based models, this model provides for the direct evaluation of program processes rather than attempting to infer the desirability of processes from their effects on student behavior. The evaluation roles of both the evaluator and personnel connected with the program are also clearly delineated in the this model. Finally, this model provides for both formative evaluation for program improvement and summative evaluation for judgment of program worth. There would be many disadvantages to using the Discrepancy Model to evaluate the rotation in sports medicine, however. First, the model is designed to evaluate major systems such as undergraduate medical curricula, and not just individual programs such as single rotations. Another disadvantage is that in order to be measurable, objectives must be quite specific. Specific objectives can obscure point in objectivi content Th skills. 5 The ma director stondarc and pro medicin time co the ma several several is very T may be are ref an app betwee (llSCrep lnCOrpr Clearlp the m. l0lger 49 obscure more general but equally important objectives. This may be an important point in evaluating a program like the rotation in sports medicine, in which many objectives are not definable in performance terms (e.g., scope and sequence of content and the quality of the rotation organization and administration). The Discrepancy Model requires a large staff of evaluators with diverse skills. Such a staff is unlikely to be available to the rotation in sports medicine. The model provides little methodology for eliciting objectives fr0'n residency directors or rotation directors. It assumes a classroom setting in which standardized or criterion—referenced exams can be administered to produce process and product evaluation data. There is no such setting in the rotation in sports medicine. Further, such a setting is not even relevant to the rotation. The major time commitment required of sports medicine rotation personnel for this task and the many other tasks associated with the model is unlikely. Finally, because several program staff members (e.g., residency director and rotation director) and several outside evaluators are required to do a variety of ongoing tasks, the model is very complex and quite expensive to implement. The essential components of the model are the five types of evaluation which may be performed and the three—step evaluation process. The types of evaluation are referred to as program design, installation, process, and product, and there is an optional cost-benefit evaluation. The evaluation process includes agreement between the evaluator and program personnel on objectives, generation of discrepancy information, and identification of program weaknesses. The model incorporates aspects of decision-management program evaluation models but is clearly an objectives-based model. This model has some relevance to evaluating the rotation in sports medicine, but only if evaluation were implemented on a much larger scale (e.g., the entire family practice residency). analyzer criteria were pr models analysis (l96ll, Stake's Alkin's, evaluat approoi E educat and pr svsterr summr Progra when . in Spa was, t Sports eClvca Outsir l0 ir cOmn lllSlri 50 Summary of Program Evaluation Models In this section, four program evaluation models have been reviewed and analyzed. The models were selected on the basis of several criteria. Among these criteria were that the models evaluated programs and not students, that the models were prominent in the educational program evaluation literature, and that the models were described in the literature in sufficient detail that a review and analysis could be accomplished. Four models were identified: those of Stake (I967), Stufflebeam (l97l), Alkin (I969) and Provus 0969). Two of the models, Stake's and Provus's, were objectives-based models, and two, Stufflebeam's and Alkin‘s, were of a decision-management orientation. These four general program evaluation models were reviewed because no adequate program evaluation model or approach could be identified in the graduate medical education literature. Each of the general models reviewed made contributions to the evaluation of educational programs. Some models required the delineation of specific objectives and provided immediate answers for decision-making. Other models assumed a systems orientation and provided ongoing evaluation services. Both formative and summative evaluation were possible with various models. Some models considered program processes as well as program products. But problems became apparent when attempting to select any one model to implement in evaluating the rotation in sports medicine. The size and scope of evaluations designed using the models was, in each case, of larger scale than was needed to evaluate the rotation in sports medicine. Several of the models were designed to evaluate major educational systems and not individual programs. The models required the use of outside evalution specialists or large evaluation staffs. The models were difficult to implement and administer, were expensive, and implied a large time commitment by a rotation staff. Several of the models provided no specific instructions or procedures for implementation. The of of the rotati objectives-b which is me which stanr making mo evaluator ar aprogram. to shifts in that would! An at sports medi ln thi 0i the me Conducted Clthponent review, de metaevalui how metae Scriu 197%, new to refer 1 Subsequent SI The objectives-based models were not well suited to evaluate those aspects of the rotation not definable in performance terms (e.g., scope of content). Thus, objectives-based models may reduce the stated purposes of a rotation to only that which is measurable. Objectives-based models often assume classroom settings in which standardized or criterion—referenced tests are available. The decision— making models shared responsibility for program improvement between the evaluator and the administrator. Thus, no one person was responsible for improving a program. Finally, predetermined general evaluation designs can be unresponsive to shifts in program emphasis. As a result, no model in its entirety was identified that would be ideally suited to evaluate the rotation in sports medicine. An attempt was then made to design an approach to evaluate the rotation in sports medicine. The approach is described in Chapter Three. Metaevaluation In this section the literature on metaevaluation will be reviewed. A review of the metaevaluation literature is included because a metaevaluation was conducted as a part of this study. Thus, it is necessary to select concepts and components of metaevaluation models from the metaevaluation literature. In this review, definitions of metaevaluation will be provided and the purposes of metaevaluation will be described. The review will also provide a description of how metaevaluations are implemented. Scriven introduced the term "metaevaluation" in I969 and Stufflebeam (I974, I978, I98l) has been its leading proponent. Scriven used the term "metaevaluation" to refer to the process of evaluating evaluations. Other writers have offered subsequent definitions. A proposed operational definition is that metaevaluation is: the process of delineating, obtaining, and using descriptive and judgmental information about the utility, practicality, Stuf‘ the Cook summative and one 51 II-92l. To actively, metaevalr designed occountal lUdgment: Primary r 52 ethics, and technical adequacy of an evaluation in order to guide the evaluation and publicly to report its strengths and weaknesses. (Stufflebeam, l98l, p. I51) We use . . . the term "metaevaluation" to refer only to the evaluation of empirical summative evaluations-—studies where the data are collected directly from program participants within a systematic design framework. (Cook & Gruder, I978, p. 6). Stufflebeam's definition reflects a wider scope for metaevaluation than does the Cook and Gruder definition. Stufflebeam suggests that both formative and summative metaevaluations be conducted, and he provides four formative designs and one summative design for conducting metaevaluations (Stufflebeam, I974, pp. 77—92). To Stufflebeam, metaevaluation serves two purposes. Performed pro- actively, metaevaluation supports decision—making. Proactive or formative metaevaluations provide recommendations about how evaluations should be designed and conducted. Metaevaluations performed retroactively have an accountability function. These retroactive or summative metaevaluations produce judgments of the merits of a completed evaluation. Smith (I98l) cites as the primary reason for conducting metaevaluation: To understand and improve the practice of evaluation itself. More specifically, individual meta-evaluation studies may be undertaken . . . to accomplish one or more of the following purposes: -—to assess the quality, impact or utilization of evaluation work; -—to study the nature of the evaluation process; —-to redress a possible evaluation abuse; —-to certify evaluation work, providing for accountability in evaluation; --to illuminate and control for bias in evaluation work; or --to assess the utility of new approaches to evaluation. (p. 267) Cook and Gruder (I978) are more succinct: "The purpose of metaevaluation is simply to help evaluators meet their goals by providing diagnostic feedback and helpful advice about what to do" (p. 6)- the general it several evaluat metaev Sanders check“ 1. judging cnterh iypesr the ad PIOCES lntonn lwnhi the ct OnSWe dedgr the cr S3 Metaevaluation models are not found in the education literature as readily as general program evaluation models. Smith (I98I) addresses this lack. The theory and practice of meta—evaluation is in its infancy. This is to be expected since the widespread, systematic practice of evaluation is itself only in its second decade. Evaluators have consequently had little practice in conducting meta-evaluations and the literature on the subject is sparse. . . .The number of actual meta-evaluations is still very small, and I know of no comparative studies of meta—evaluation procedures. (p. 266) While few actual metaevaluation studies have been reported in the literature, several authors have deveIOped checklists, approaches, and models for guiding evaluation studies. Stufflebeam (I974) and Cook and Gruder (I978) provide metaevaluation models, and Roth (I982) developed a metaevaluation approach. Sanders and Nafziger (I976) and Millman (l98l) developed metaevaluation checklists. The Sanders and Nafziger checklist was designed to provide a basis for judging the adequacy of evaluation designs. This judgment is made by applying criteria in the form of questions to the evaluation design under consideration. Four types of criteria are included in the Sanders and Nafziger checklist: criteria for the adequacy of evaluation planning, for the adequacy of the collection of and processing of information, for the adequacy of the presentation and reporting of information, and for ethical considerations and the evaluation protocol. To use the Sanders and Nafziger checklist, the metaevaluator first becomes familiar with the major components of the evaluation design. Then each item on the checklist is considered with regard to the evaluation design. Reviewing the answers on the checklist provides a judgment of the adequacy of the evaluation design, and provides a basis for giving feedback to the evaluator. Millman‘s checklist (l98l) covers two areas of an evaluation. The first area is the components of the program evaluation: l) the conditions existing prior to the inception 0‘ evaluation, covered by evaluator o Unlik is not to metaevalur worth of ti p. 3“). Ho Meta Cook and formative metaevalu metaevalu An l3thlU<1tto| known se deVGlOpec ElGluotio America, ASSOCiqfi me“titers 54 Inception of the program and the evaluation, 2) the effects of the program and the evaluation, and 3) the utility of the program and the evaluation. The second area covered by the Millman checklist is the quality of the judgments made by the evaluator of the program being evaluated. Unlike the Sanders and Nafziger checklist, the goal of the Millman checklist is not to generate a series of yes or no answers or to derive a type of metaevaluation score. Rather, "it is to come away with a sense of the merit or worth of the evaluationuwhere it makes it, and where it falls down" (Smith, l98l, p. 3”). How the metaevaluator gains this sense Millman does not explain. Metaevaluation models and approaches were supplied by Stufflebeam (I974), Cook and Gruder (I978), and Roth, I982. Stufflebeam provides four proactive formative metaevaluation models and one retroactive model for summative metaevaluation. In distinguishing between formative and summative metaevaluations, Stufflebeam (l98l) writes: Summative metaevaluation is the fundamental metaevaluation role. It sums up the overall merit of an evaluation, and usually is done following the conclusion of a primary evaluation. It holds evaluators accountable by publicly reporting on the extent that their evaluation reports meet standards of good evaluation practice. Finally, summative metaevaluations determine how seriously they should take the primary evaluation's reported conclusions and recommendations. (p. l5l) An important assumption about metaevaluation is that the quality of an evaluation can be determined by comparing the evaluation performance against a known set of standards or concepts. Currently, such a set of standards has been developed and published by The Joint Committee on Standards for Educational Evaluation (I981). The Joint Committee was made up of representatives from the American Educational Research Association (AERA), the American Physchological Association (APA), the National Council on Measurement in Education (NCME), and members from several additional groups. The Committee agreed that a set of profess Because it such stant practice evaluatior utility, le maybe to Met Perlorme program: should be metaeval purpose. evaluatia evaluatic accounto lmj tour SIEj conducte leo.,wh evaluati. tools of eVuluatt they We metaew leg“ v encount 55 of professional standards could help improve the quality of educational evaluation. Because no adequate standards existed, the Committee deveIOped and published 30 such standards. As the Committee perceived it, the standards would upgrade the practice of evaluation and develop more efficient and effective educational evaluations. The standards contain advice about four areas of an evaluation: utility, feasibility, propriety, and accuracy. Using these standards, an evaluation may be formatively guided or summatively judged. Metaevaluation may be conducted to serve a decision-making purpose. Performed for this reason, metaevaluation is conducted proactively with the program evaluation to provide recommendations concerning how evaluation studies should be designed and conducted. This type of metaevaluation is formative metaevaluation. Metaevaluation may also be used to serve an accountability purpose. In this case, metaevaluation is conducted retroactively to the program evaluation, and produces public judgments of the merits of the completed program evaluation. Retroactive metaevaluation conducted for the purpose of accountability is summative metaevaluation. Implementing Stufflebeam's model of summative metaevaluation consists of four steps. In the first step, the metaevaluator determines why the evaluator conducted the evaluation study and what the goals of the evaluation study were (e.g., what audiences were to be served, what the intended audience want from the evaluation). Data are then gathered for judging the evaluator's intents and the goals of the study. Next, the metaevaluator compiles information about the evaluation design that was chosen (e.g., what other designs were considered, why they were rejected). In the third step of implementing metaevaluation, the metaevaluator determines how well the chosen evaluation design was carried out (6.9., whether the design fully carried out, what specific problems were encountered). Finally, the metaevaluator should consider the results produced (e.g., whe intormati both to tt Coo Three mc the metc evaluatio whetherr metaeval used. T conducte metaeva Two oft Specific Illodel It] lea, tt metaevc Progran las Oppc evaluati TI Review externo e\loluat Publish. Ullllty “leflsur I__————’ ' T“ 56 (e.g., whether objectives were achieved, what information was produced). The information is compiled by the metaevaluator and a written report is disseminated both to the program evaluator and to all the audiences of the program evaluation. Cook and Gruder (I978) proposed seven different metaevaluation models. Three major factors differentiated these models. The first factor concerned when the metaevaluation was to take place, either simultaneously with the program evaluation or subsequent to the program evaluation. The second factor concerned whether or not data from the program evaluation was to be manipulated during the metaevaluation. The third factor was the number of independent data sets to be used. Three of the models described by Cook and Gruder are designed to be conducted simultaneously with the program evaluation. They are formative metaevaluation models. The remaining four models were of a summative nature. Two of these summative metaevaluation models (Review of the Literature About a Specific Program——Model 2; and Empirical Re-evaluation of Multiple Data Sets-- Model 4) required the program under consideration to have been multiply evaluated (e.g., the many evaluation studies of "Sesame Street"). A third summative metaevaluation model described by Cook and Gruder (Empirical Re-evaluation of a Program Evaluation-—Model 3) focused an answering questions about the program (as opposed to the evaluation of the program) that were not asked in the program evaluation. The implementation of the final summative metaevaluation model (Essay Review of an Evaluation Report——Model l) is accomplished by either internal or external metaevaluators who publish reports of the evaluation after the program evaluation is completed, but before the final program evaluation report is published. Using this model, the metaevaluation task is to concentrate on the utility of individual research questions and on such technical issues as sampling, measurement, and data analysis. Ther model. Tl questionin determine with ther to validity data. Tc before fir metaevalu the met accountal Coc metaeval reviewed scale pro for the ct Re In Roth' undertak PTOgram elticienr determir Th Vurious comtilet dECislon Collecte 57 There are two main advantages in using the essay review metaevaluation model. The first advantage is that new interpretations may be revealed without questioning the data collected. A second advantage is that the model may determine specific weaknesses in the program evaluation that can be overcome with the data already collected. A major disadvantage of the model is that threats to validity cannot be examined because the metaevaluator does not re—analyze the data. To avoid this problem, Cook and Gruder suggest conducting the review before field research is completed. In so doing, however, the focus of the metaevaluation would be shifted from a summative to a formative nature. Hence, the metaevaluation would then serve a decision-making rather than an accountability purpose. Cook and Grader discuss the advantages and disadvantages of all seven of the metaevaluation models they propose. With the exception of the model just reviewed, however, the metaevaluation models are designed to be used for large- scale program evaluations (e.g., school integration). Thus, they are not well suited for the current study. Roth (I982) provided an approach for metaevaluating a program evaluation. In Roth's approach, the metaevaluator (who is also the program evaluator) undertakes formative metaevaluation procedures for the purpose of modifying the program evaluation for future use. Roth's approach attempts to determine the efficiency, completeness, and degree of focus of a program evaluation, and to determine the extent to which each evaluation objective was achieved. The Roth approach is implemented by the metaevaluator who c0nstructs various charts and tables and attempts to estimate percentages of efficiency, completeness, and focus. As an example, completeness refers to the percentage of decisions made about the program which are made at least partly with the data collected in the program evaluation. Program decisions made without the use of data colic (Roth, I9l that infor major dis maintain what pros metaeval Iormativr Scr and Grud "collecte Iramewo metaeva metaeva Th designs. °l lhe I ll°9tatt “5 Md and We. both th 58 data collected in the program evaluation represent the "error of completeness." (Roth, 1982, Book V1, p. 12) An advantage of Roth's metaevaluation approach is that information is provided for modifying subsequent program evaluations. A major disadvantage of the approach is that the program evaluator is required to maintain a detailed record or log of all decisions made regarding the program and what program evaluation data, if any, were used in reaching each decision. Roth's metaevaluation approach is further not suited for the current study since it is a formative metaevaluation approach. Scriven originally defined metaevaluation as evaluation of evaluation. Cook and Gruder limited their definition of metaevaluation to studies in which data were "collected directly from program participants within a systematic design framework" (p. 6). Stufflebeam's definition placed no such limitations on metaevaluation, and indeed proposed both formative and summative models of metaevaluation. The various metaevaluation models each contain concepts particular to their designs. Stufflebeam (1978), however, notes in general that a proper conceptualization of meta evaluation should meet several conditions. It should present and define all the specific concepts that are required to explain the general concept, and this presentation should be internally consistent. The conceptualization of meta evaluation should suggest hypotheses to be tested in a program of research on meta evaluation, and it should give direction for developing Clear guidelines for the proper conduct of meta evaluation. 21 (p. There are several virtues to attempting to perform metaevaluation regardless of the model used. The first virtue is that metaevaluation helps to ensure that program evaluations are conducted as adequately, efficiently, fairly, and usefully as possible. A second virtue is that metaevaluation publicly exposes the strengths and weaknesses of a completed program evaluation. Thus, metaevaluation protects both the evaluator and the consumer from the application of faulty conclusions and recomme evaluatic evaluatic Tht that like a metae even wa: lock of lack of need for Smith (p metaevc evaluati In emphas': Commit The sta tor set intervie GPPIOOC PUblish hundrer Partici lequire Posed | be use Slottdo —7—” EFF" " W 59 recommendations. A third virtue to performing metaevaluation is that meta- evaluations involve very little cost, especially when compared to the program evaluation. There are also limitations to performing metaevaluation. One limitation is that like the program evaluation itself, metaevaluations are subject to error. Thus, a metaevaluation may not solve the problems of the program evaluation and may even waste additional resources. But perhaps the more significant limitation is the lack of methodology to be found in the literature. Stufflebeam (1974) reported a lack of detail on the mechanics of performing metaevaluation and described the need for both conceptual and technical development in metaevaluation. In 1981, Smith (p. 263) observed that there had still been relatively little work done in the metaevaluation, and what little had been done focused on the development of evaluation standards and not methodology. In developing the metaevaluation component for the current study, an emphasis was placed on the standards for evaluation developed by the Joint Committee (The Joint Committee on Standards for Educational Evaluation, 1981). The standards were chosen as a basis for developing the metaevaluation component for several reasons. The major reason was the desire to develop structured interview guides that would be helpful in summatively evaluating the prototype approach used to evaluate the rotation in sports medicine. The standards, published in 1981, represent the current state-of-the-art of evaluation. Several hundred evaluators with backgrounds in education, social science, and other areas participated in the formulation of the standards. More importantly, the standards require the evaluator to gather metaevaluation information relevant to questions posed by program evaluation clients and audiences. The standards are designed to be used in either a formative or a summative metaevaluation. Finally, the standards may be used with a variety of evaluation methods such as surveys and observati Usi utility, is evaluatir program of the st approact describes will be r In reviewer medical rotation broader evaluati viability PIOgran Fl anaIyZe Criteric models the ma The 101 “0 ode 9101M] 60 observation—-methods used in the program evaluation. Using the standards, instruments were developed to assess the degree of utility, feasibility, propriety, and accuracy attained by the prototype approach for evaluating residency rotations. The metaevaluation component and the prototype program evaluation approach are separate entities. The metaevaluation component of the study is not intended to serve as a part of the prototype program evaluation approach. The individual components of the structured interview guides will be described in Chapter Three of the dissertation. The results of the metaevaluation will be reported in Chapter Four. Summary In this chapter, two major bodies and one minor body of literature were reviewed. The chapter began with a search for models for evaluating graduate medical education programs. The focus of the search was evaluation of residency rotations. No research was found. Some literature was found, however, on the broader subject of the evaluation of residency programs. The residency program evaluation literature proved insufficient for the purpose of determining the viability of the various evaluation approaches suggested. Thus, a review of general program evaluation models was undertaken. Four major program evaluation models were identified, reviewed, and analyzed. The models were selected on the basis of several criteria. Among these criteria were the following: The models must evaluate programs-—not students; the models must be prominent in the educational program evaluation literature; and the models must be described in sufficient detail to permit a review and analysis. The four general program evaluation models were reviewed and analyzed because no adequate program evaluation model or approach could be identified in the graduate medical education literature. Fin included approach was not metaeva were ult Standard several developi utility, ‘ approacl Th evaluati evaluate Propose the purj 61 Finally, the metaevaluation literature was reviewed. This review was included because a metaevaluation was conducted on the rotation evaluation approach proposed in this dissertation. Although the literature in metaevaluation was not as extensive as the literature in general program evaluation, several metaevaluation models and checklists were identified. Structured interview guides were ultimately developed using standards proposed by the Joint Committee on Standards for Educational Evaluation (1981). The standards were selected for several reasons, prominent among them that the standards proved useful in developing summative metaevaluation structured interview guides for assessing the utility, feasibility, propriety and accuracy of the prototype program evaluation approach proposed in the dissertation. The residency program evaluation literature and general educational program evaluation literature provided no approach or model that appeared ideally suited to evaluate the rotation in sports medicine. In Chapter Three, such an approach is proposed. In addition, metaevaluation structured interview guides are provided for the purpose of accountability. l approc used t rnahor deterr redde redde procet Inetae progn is lot devel the 5' Iron meta Procc Chapter Three Methodology and Procedures Introduction A purpose of this study has been to design, develop and implement an approach for evaluating residency rotations and to determine whether the approach used would work in gathering accurate and useful formative evaluation infor- mation. A review of the literature on program evaluation in Chapter Two determined that there was no single model that was ideally suited for evaluating residency rotations. Therefore, an approach was designed and implemented in one residency rotation. The approach was then assessed by means of a metaevaluation procedure. In this chapter, the methodology used to design, implement, and metaevaluate the evaluation approach is presented. The reasons that existing program evaluation models were deemed inadequate are discussed. This discussion is followed by a schematic diagram and description of the prototype approach developed to evaluate the residency rotation. The rotation and the participants in the study are described. In the final part of the chapter, the research questions from Chapter One are presented again, and the implementation of the metaevaluation, the metaevaluation instruments, and metaevaluation data analysis procedures are described. w Many models for the evaluation of educational programs were identified in the literature. All presented severe limitations for use in evaluating residency rotations. Many models, including the four reviewed and analyzed in Chapter Two, 62 were orlt Elementc be applic general 1 school 5) different were di' Decision substant' objectiw as gener providec Or for eva? secondj imwma describe medicin models. easilyi require that su must r hetWee methor Plogra 63 were originally developed as a result of the evaluation requirement in the 1965 Elementary and Secondary Education Act (ESEA). These models were designed to be applicable to a number of large-scale educational programs and are purposely general for this reason. Several models were, in fact, designed for large—scale school systems. Some models required large evaluation staffs with expertise in different areas. Many of the models were expensive to implement. Some models were difficult to implement and administer because of their complexities. Decision-making models assumed an ongoing evaluation effort. However, a substantial time commitment was also expected with the use of models with an objectives-based orientation. Finally, because many of the models were designed as general guides, specific instructions or procedures for implementation were not provided. One purpose of this study was to design, develop and implement an approach for evaluating specific programs of a smaller scale--residency rotations. (The second purpose was to determine whether the approach yielded accurate and useful information, and whether the approach was practical and ethical.) The approach described in this chapter was designed for a specific program, the rotation in sports medicine. Many of the components in the approach were borrowed from existing models. The approach was designed to consider a number of criteria: I) it must be easily implemented by an external evaluator, 2) it must reflect the informational requirements of the rotation to be evaluated, 3) it must be designed in such a way that subsequent evaluations can be implemented by an internal evaluator, 4) it must cost relatively little money and time, 5) it must identify discrepancies between instructional intents and actual implementation, 6) it must include the methodology necessary for implementation, and 7) it must place responsibility for program improvement on the program administrators (rotation directors). Ma The app specific evaluate tailored compone this stuc in this d H describe M Ts context Tl importc rational environ qUeStIOI I. 2. SlUllle 64 Design of the Approach Most program evaluation models focus primarily on the process of evaluation. The approach described in this chapter, however, has a second basic component: specific evaluation questions of interest to the audiences of the rotation being evaluated. It is the combination of process and specific evaluation questions tailored to the program under evaluation that makes the approach useful. The two components are equally important and highly integrated. A major contribution of this study to residency rotations is the specificity of detail in the design reported in this dissertation. First, the specific evaluation questions of interest to the rotation will be described, and then the process used to implement the approach. Specific Evaluation Questions Twelve evaluation questions were divided into three clusters: 1) rotation context variables, 2) instructional variables, and 3) evaluation variables. The questions about rotation context attempt to identify and describe important aspects of the environment in which the rotation operates. The rationale for rotation context questions is to provide documentation of the environment in which the rotation is operating. Three rotation context variables questions form this evaluation question cluster. They are: I. 2. Are there written policies and procedures to guide the rotation? Are there written job descriptions and qualifications for faculty and staff positions? Are there written descriptions of the following resources? —-availability of faculty and staff --injured or ill athletes --facilities (e.g., clinical spaces, rehabilitation equipment) ——budget «educational materials (e.g., library, slides) Several of the more general program evaluation models address similar areas. Stufflebeam (1971), for example, recommends the use of context evaluation to describe th purpose of c an evaluati faculty) an< program pru Provus (1% description. being evalt prerequisite clearly des rotation Ct Hammondl Thes It. 5. Cone qUestlon c “967): for anteceden. describe the desired and actual conditions of an educational program for the purpose of diagnosing particular problems and targeting improvement efforts. Such an evaluation could expose resources available to the program (e.g., adjunct faculty) and assess the efficacy of program goals and priorities (e.g., discovering a program procedure that hinders residents from making the best use of their time). Provus (1969) describes rotation context variables under the heading of "program description." being evaluated only after the program's antecedent conditions (e.g., resident prerequisites), transactions (e.g., policies and procedures) and purposes have been clearly described. Other models also include components that correspond to the rotation context variables of this study--among them those of Tyler (1942), Hammond (1969), and Alkin (I969). The second cluster of questions is about instructional variables. They are: 4. 5. Concepts similar to those underlying the instructional variables evaluation question cluster are also reported in the program evaluation literature. (1967), for example, with regard to his data matrices, discusses the concept of antecedents (e.g., resident prerequisites). Stufflebeam speaks of input evaluation. Are there written prerequisites for residents’? Are there written objectives for the rotation? Is there a written plan for implementing the curriculum (content areas addressed in the rotation)? Is there a written description of instructional strategies to be used to implement the curriculum? The purpose of input evaluation is to provide information for determining how to utilize resources to achieve project objectives. This is accomplished by identifying and assessing 1) relevant capabilities of the responsible agency, 2) strategies for achieving project objectives and 3) designs for implementing a selected strategy. (Worthen and Sanders, 1973, p. 137) In the Provus design, the evaluator can be confident about what is Alkin I implemt local le l|9671 a teachin Sr Two 015 among characl residen be imp trainini should objecti I to (tell (my sy: 66 Alkin (1969) describes the instructional variables component in the program implementation part of his model. Hammond's (1969) model for evaluation at the local level contains similar instructional content and methodology variables. Stake (1967) also notes that "to evaluate an educational program we must examine what teaching. . .is intended" (p. 531). Several of the graduate medical education approaches reviewed in Chapter Two also included components corresponding to instructional variables. Prominent among these was that of Corley (1983), who devoted 30 pages to entering characteristics of residents, because "without an adequate data base of each resident's entering characteristics, professional progress in residency training will be impossible to evaluate" (p. 33). In addition, the special requirements for training in family practice state that: "All major dimensions of the curriculum should be structured educational experiences for which written goals and objectives. . .exist" (Directory of Residency Training Programs, 1983, p. 22). The third cluster of questions is about evaluation variables, and it attempts to determine whether rotation context and instructional variables are evaluated in any systematic fashion. The questions are: 8. Is the curriculum systematically evaluated for «implementation? «relevance? 9. Are rotation objectives evaluated for «implementation? «relevance? 10. Are instructional strategies evaluated for «implementation? «effectiveness? 1 I. Are residents evaluated for «prerequisites? «knowledge? Mart questions. that the j only at tl term." componer Pro procedurr decisions process 6 the curre similar tr Model, p data. 1r result of different Instrucfi evaluatir Alkin's I SUCCess c0”lions dlSCTEDC «skills? «attitudes? 12. Are faculty and staff of the rotation evaluated for «qualifications? «availability? «teaching skills? Many existing program evaluation models help to justify the inclusion of such questions. Stufflebeam (in Worthen and Sanders, 1973, p. 138), for example, writes that the purpose of product evaluation is to "measure and interpret attainments not only at the end of a project cycle, but as often as necessary during the project term." Process evaluation in the Stufflebeam model also corresponds to the components of evaluation variables in the prototype approach. Process evaluation in Stufflebeam's CIPP model detects defects in the procedural design or its implementation, provides information for program decisions, and provides a record of procedures. The information provided in process evaluation of the CIPP model is similar to the type of information used in the current approach. The process stage of the Provus Discrepancy Model (1969) is similar to the evaluation variables cluster. In the process stage of the Discrepancy Model, program adjustments are made on the basis of interim formative evaluation data. In the prototype approach, improvements to the rotation are made as a result of discrepancy information. Discrepancy information is generated from the differences noted between intents documented in the rotation context and instructional variables clusters and what actually occurred, as documented in the evaluation variables cluster. In addition, the program improvement component of Alkin‘s CSE Model (1969) is designed to provide information about the relative success of various program components. Thus, the program improvement component of the CSE Model provides essentially the same information as the discrepancy information generated in the prototype approach. 39191129 I. Art 2. Art p05 3. Ar: I_n_str_uc_t_i l. Ar 5. Ar 6. Is 68 Table 1 Evaluation Questions Rotation Context Variables I. Are there written policies and procedures to guide the rotation? 2. Are there written job descriptions and qualifications for faculty and staff positions? 3. Are there written descriptions of the following resources? «availability of faculty and staff «injured or ill athletes «facilities (e.g., clinical spaces, rehabilitation equipment) «budget «educational materials (e.g., library, slides) Instructional Variables 4. Are there written prerequisities for residents? 5. Are there written objectives for the rotation? 6. Is there a written plan for the implementation of the curriculum (content areas addressed in the rotation)? 7. Is there a written description of instructional strategies to be used to implement the curriculum? Evaluation Variables 8. Is the curriculum systematically evaluated for «implementation? «relevance? 9. Are rotation objectives evaluated for «implementation? «relevance? 10. Are instructional strategies evaluated for «implementation? «effectiveness? ll. Are residents evaluated for «prerequisites? «knowledge? «skills? «attitudes? 12. Are faculty and staff of the rotation evaluated for «qualifications? «availability? «teaching skills? T h compone compon€ residenc llhitmar contains the Carl request it absence answere additior the slit measure any of provide rotatior method PM I the pn identit Occurn relevaj lmPtOI Rutati 69 The evaluation approach outlined by Marshall and Davanzo (1983) contains a component on faculty and staff teaching skills, as does the evaluation variables component of the prototype approach. The Jelly and Friedman (1980) approach for residency training contains components similar to the evaluation variables. Whitman's (I983) guide to systematic data collection for residency programs also contains resident, faculty and program components similar to this cluster. Finally, the Corley (1983) approach contains a number of checklists, questionnaires, surveys, request cards, and other instruments for evaluating similar components. The twelve evaluation questions attempt to determine the presence or absence of various evaluation data available in a rotation. If the questions are answered in the affirmative, and if the data are sufficient, accurate, and current, additional evaluation is unnecessary. If the evaluation questions are answered in the affirmative, but data are insufficient, inaccurate, or not current, appropriate measures should be undertaken to improve the existing system of evaluation. If any of the 12 questions are answered in the negative, however, there is a need to provide additional information. The information needs are agreed upon by the rotation director and the researcher. Then, through a variety of data collection methods (e.g., questionnaires) the data are obtained and analyzed. Process Used to Implement the Approach The second component of the approach was the process used to implement the program evaluation. The process component consisted of six elements: 1) identifying the intents of the rotation, 2) collecting data about what actually occurred during the rotation, 3) documenting discrepancies, 4) reporting relevant discrepancies, 5) resolving discrepancies, and 6) implementing improvements. The Prototype Program Evaluation Approach for Residency Rotations is presented in Figure 5. The first element of the process component of the current approach is ‘l Identi' lnstru: Intent collec‘ first 1 voriqb The f. Syllob focuH lolqti 70 Figure 5 The Prototype Program Evaluation Approach for Residency Rotations Conduct Rotation ll identify Col lec: Document instructional Implementation DiscrepanCies intents ’ And Relevance ’ Data Preoare Ana Resolve 3e IVE' Cisc‘eaanc es lm;rcvem8fit5 r Evaluation To Rotation Report collecting data about the intents of the rotation. Such data are generated from the first two clusters of evaluation questions, about rotation context and instructional variables. There are two primary sources of information about rotation intents: The faculty and staff of the rotation, and documents about the rotation (e.g., the s>’llabus). The evaluator conducts this part of the evaluation by interviewing the faculty and staff and by collecting and examining the documents. Data about rotation intents may be collected throughout the course of the evaluation. Howeve: the instr about tl identify 0967), t it occurs evoluot inlervie prepare and cor observe doto-cc residen evoluot oi the course C collect 08 o l instruc (I971) I nearly elemer “969), lnslrUr kinds . Gllill’eg However, since the evaluator designs subsequent questionnaire items depending on the instructional and contextual intents, it is necessary to collect most of the data about these intents first. Similar concepts to the first element in this approach, identifying rotation intents, are advocated by Tyler (I942), Bloom (I956), Stake (I967), Hammond (I969), and Provus 0969). The second element of the process component is collecting data about what occurs in the rotation, and about the relevance of the curriculum. The three evaluation question clusters serve as the basis for data collection. The evaluator interviews faculty, staff, and current residents in the rotation. The evaluator also prepares and disseminates data-collection instrunents (questionnaires) for former and current residents and an external review panel. Finally, the evaluator actively observes the rotation in progress and records pertinent observations. The different data—collection techniques are used at various times. Questionnaires to former residents and the external review panel may be distributed at any time during the evaluation. The evaluator collects data from current residents at or near the end of the rotation in which they participate. The evaluator observes throughout the course of the rotation. Collecting data is an element of all evaluation models. The kind of data collected vary with the model used. The three evaluation question clusters served as a basis for data collected in this approach: rotation context variables, instructional variables, and evaluation variables. Alkin (I969) and Stufflebeam (I97I) propose similar elements to rotation context variables using the same or nearly the same terminology. Stake (I967) and Provus (I969) describe a similar element in their models but use different terms. Stake (I967), Tyler (I942), Provus (I969), Stufflebeam (I97I), and Alkin (I969) all address various parts of the instructional variables evaluation question cluster in the current approach. The kinds of information produced by the evaluation variables question cluster are also addressed by these and other model authors. Th between evaluate various specific modifie< there is process it (1969) rr who he Ball (I9 T? evaluat directo Anders- I98I), l report discrep l solving l0 deI discrel Fina”; lhree Select Model lllput 72 The third element of the process component is documenting discrepancies between the intents of the rotation and what actually occurs in the rotation. The evaluator does this by comparing the intents with the data collected from the various data sources. Questionnaire items and evaluator observations for each specific question in the three evaluation question clusters are developed or modified by the evaluator. The data collected are then compared to the intents. If there is a discrepancy, it is recorded for reporting in the fourth element of the process component. Tyler (I942) originated the discrepancy concept, and Stake (I967) and Provus (I969) more fully developed the idea in their models. Authors and model builders who have contributed to this concept include Datta (in Berk, I98I), Anderson and Ball (I978), Patton (I978), Rachal (I982) and Arter (I982). The fourth element of the process component is the evaluation report: the evaluator prepares and delivers the results of the evaluation to the rotation director. The evaluation report element of the process is based on the writings of Anderson and Ball (I978), Stake (in Worthen and Sanders, I973), Datta (in Berk, l98l), Rachal et aI. (I982), Arter et aI. (I982) and Cuba and Lincoln (I983). The report documents discrepancies and offers recommendations for eliminating major discrepancies. The fifth element of the process component is resolving discrepancies or solving problems. The faculty and staff of the rotation and the evaluator attempt to determine the cause of the discrepancy. When the cause or causes of a discrepancy have been determined, a list of possible corrective actions is made. Finally, the rotation director selects the best course of corrective action. These three steps--determining the cause, generating potential corrective actions, and selecting the best alternative—~are based on a similar part of the Discrepancy Model (Provus, I969). Other models also contain such elements (e.g., Stufflebeam's input evaluation). The s correctiver but improv Corrective intent to at the other implement: improveme implement: lormative improve pr worth or it many of tI necessity producing evaluating every disc necessary Thes EXisIlng n Componen ChGpter. Prog Illusion-r "Nels e: Diagram melll dir GbOUf dEI The sixth and final element of the process component is implementing the corrective action in the rotation. This may be done with the aid of the evaluator, but improving the rotation is clearly the responsibility of the rotation director. Corrective action may be of two sorts: one is to modify or eliminate a rotation intent to achieve more congruence with the actual implementation of the rotation; the other is to modify or eliminate a portion of the rotation so that actual implementation of the rotation is more congruent with the intents. The improvements are the alternatives which are selected in the fifth element and implemented by the rotation director. This element is justified by the concept of formative evaluation: formative evaluation information is used to modify and improve programs, while summative evaluation information is used to judge the worth or merit of programs. The reliance on formative evaluation is apparent in many of the major models. Stufflebeam (I97I) and Alkin (I969) both emphasize the necessity of structuring evaluation so as to serve decision-making purposes by producing appropriate and timely information. Provus's (I969) approach to evaluating entire school districts was also formative. In the Discrepancy Model every discrepancy results in a problem-solving routine that ultimately defines the necessary corrective action. These six elements of the process component have been extrapolated from existing models and integrated with the specific evaluation questions (the first component of the approach) to form the prototype approach presented in this chapter. Program evaluation models are frequently classified as having either a decision-making orientation or an objectives-based orientation. Decision-making models essentially provide information to be used in making decisions about the Program being evaluated. Models using a decision-making orientation do not assess merit directly. The focus of decision—making models is on gathering information about decisions to be made. It is assumed that the merit of alternatives is consider evaluatr decisior a decisi evaluat leads tc E‘ objecti‘ The ev: betwee 0n the change Object Progra against an objr I design was a dissert focus *0 sh had St more and r odditi retail 74 considered when decisions are made. Using a decision—making approach, the evaluator supplies information about particular alternatives being considered to the decision maker. The decision maker then selects the alternative. Evaluation using a decision-making approach is a cyclical process of program management in which evaluation leads to decisions. The results of these decisions are evaluated, which leads to further decisions and so on. Evaluation using an objectives-based approach has a different focus. In an objectives-based approach, the focus is on assessing the outcomes of education. The evaluation consists of comparing what is with what should be, the differences between what was intended and what actually occurs—-often called discrepancies. On the basis of these discrepancies, the program administrator makes decisions to change the program or the objectives (or in rare cases to terminate the program). Objectives-based models need not focus exclusively on student performance. Program administration may also be evaluated--program implementation compared against intended plans. The prototype approach described in this chapter generates an objectives-based, formative evaluation approach. A major problem encountered in the study had to do with changes made in the design of the approach during the conduct of the study. The reason for the changes was a transition in directorship of the researcher's doctoral committee. A new dissertation director was appointed after data collection had been completed. To focus the evaluation more tightly, the new director suggested an important change -—a shift in the emphasis of the existing evaluation questions. The shift in emphasis had several results. One result was that the individual evaluation questions were more specifically focused on important issues, and as a consequence, more specific and relevant findings were made about the rotation. A second result was that additional evaluation questions were developed about the context in which the rotation operated, and about instructional aspects of the rotation. A third result was thr the ca compor rolatio several in dire added I obtain instruc in this more : used i questiu criteri evaluc equn. CODCG sever: QVOIUI SUIfic liquii evalu ellic onsw ddlo 75 was that all evaluation questions were grouped into three areas: questions about the context in which the rotation operated, questions about instructional components of the rotation and questions about evaluation components of the rotation. The questions that comprised the rotation context variables cluster and several questions in the instructional variables cluster were added after the change in directorship took place. The questions about rotation context variables were added because such data would be very useful to the rotation directors and could be obtained post hoc fairly easily and inexpensively. Additional questions about instructional intents were developed to ensure that sufficient data were collected in this important cluster. The three clusters of questions made data collection more systematic and provided an example of evaluation questions which could be used in replicating the study. Finally, the shift in emphasis of the evaluation questions addressed an important design criterion of the approach. The design criterion was that the approach identify discrepancies. Many of the original evaluation questions did not reflect a discrepancy approach to evaluation. The expanded and more focused evaluation questions addressed the discrepancy concept. The effect of the shift in emphasis of the original questions was that, in several cases, the researcher attempted to fit data already collected to new evaluation questions. For some of the new evaluation questions, the data were sufficient. For others, the data collected did not furnish all the information required to answer the evaluation questions. Despite the problems encountered, the researcher was able to complete the evaluation without collecting entirely new data. The problems resulted in a loss of efficiency and, in some cases, effectiveness. It was possible for the researcher to answer the l2 evaluation questions, however, because of the multiple sources of data designed into the study. (J prototy and the chapter coHect field tr were p liwgiig l Practh goal5r the co and tr howev rotati redde The r condo “PPIO Iepre eXper Preje Provi the r the r 76 Field Test of the Approach On the basis of the special evaluation requirements of a residency rotation, a prototype approach was designed for the rotation in sports medicine. The rotation and the participants in the field test are described in the following sections of the chapter, as is the implementation of the field test. The development of data- collection instruments is reported. Data gathered with the instruments during the field test, summarized in a written evaluation report prepared by the researcher, were presented to the rotation directors in January I985. The Rotation in Sports Medicine The rotation in sports medicine is offered by the Department of Family Practice in the College of Human Medicine at Michigan State University. The goals of the rotation are to acquaint the resident with and involve the resident in the comprehensive care and treatment of athletes. A major portion of the care and treatment is for illness or injury to athletes. An emphasis of the rotation, however, is the future prevention of such illnesses and injuries to athletes. The rotation is available to residents in family practice at MSU, second- or third-year residents from other residency programs, and fourth year medical students at MSU. The rotation is one month long. The field test of the evaluation approach was conducted during the course of one rotation. The rotation in sports medicine was chosen to field test the prototype approach for several reasons. First, the rotation in sports medicine was representative of rotations in general. The rotation consisted of clinical experiences with increasing resident responsibility for treating patients, a research project, and observation of athletes performing. The rotation in sports medicine provided a multidisciplinary approach to treatment. In addition, the duration of the rotation was a calendar month, like many other rotations. Like other rotations, the rotation in sports medicine was designed to provide residents with the basic skills c more 1 rotatia had ev Finally the ph in spa coIIec The s collec group entire rotati area i in set Sport; could sougl form third Prac- Ohio the Pedir 77 skills and knowledge required in one area. The rotation had been in existence for more than four years and approximately 50 residents had participated in it. The rotation was offered eight or more times each year, but no formal evaluation of it had ever been conducted. Thus, the effectiveness of the rotation was not known. Finally, the rotation directors were interested in and receptive to an evaluation. Participants Three groups provided data for the field test. The first group consisted of all the physicians and allied health professionals who had participated in the rotation in sports medicine since its inception in I977 until the beginning of the data- coIIection period in November I982. This group was designated Former Residents. The second group consisted of physicians who participated during the data collection periods of November I982, and January, February, and March I983. This group was designated Current Residents. The information was sought from the entire populations, not from a sample. The third group consisted of physicians who had not participated in the rotation but who were recognized for their various contributions to the interest area of sports medicine. This group differed from the former and current residents in several important ways: they had no previous connection to the rotation in sports medicine, and they were selected from among a large number of people who could have been selected (rather than the entire population). The information sought from the third group was of a nature different from that sought from the former and current residents. Finally, the practice backgrounds of many of the third group were different. An effort was made to identify physicians who, besides practicing sports medicine, are opinion leaders in this field. An effort was made to obtain a diversity of perspectives. Thus, an external review panel was drawn from the specialties of family practice, internal medicine, orthopedic surgery, pediatrics, general surgery, and cardiology. The task presented to panel members was to r< medicine medicine A . subjects 49 subjer Universi‘ acceptec were par during t9 March | family j Pflefl: recogniz Pc An atte cOmplel cOmplet rotatior of each thelwe Q kelChai theque T 0i eitt medlCir 78 was to rate the importance of the curriculum components of the rotation in sports medicine at Michigan State University and to identify areas important for sports medicine but not being addressed in the rotation. A total of 87 of the 90 possible subjects participated in the study. The subjects represented these three groups: Former Residents: N = 48 of the possible 49 subjects who had completed the rotation in Sports Medicine at Michigan State University at some point between October I978, when the first resident was accepted, and the October I983 rotation. Current Residents: N = 8 subjects who were participating in the rotation in sports medicine at Michigan State University during the data collection periods of November I982 and January, February, and March I983. (Of the eight, three were fourth-year medical students, four were family practice residents, and one was a podiatric resident.) External Review Big: N = 3I of the possible 33 subjects, all of whom are physicians, and are recognized for their various contributions to the interest area of sports medicine. Participants were contacted for participation in the study in different ways. An attempt was made to obtain responses from all 49 participants who had completed the rotation since its inception. For the eight current residents, completing the questionnaire at the end of the rotation was mandated by the rotation director. An hour was set aside for this task, a clay or two before the end of each rotation. Compensation was not offered the residents for their time, but they were given a small gift--an MSU keychain. Questionnaires were mailed to the 49 former residents. A token gift -—the keychain--was sent with the questionnaire as an inducement to complete and return the questionnaire. The external review panel was not intended to be necessarily representative of either orthopedic surgeons or primary care physicians who practice sports medicine. A four-step process was used in the selection. F Y§£§.! Ihg_£[ §ggrt§_ ponelv The b< hokfing contrk as tea among t l thtc names I leoder Amon Ortho sporh level, Sport Thk. 79 First, listings of sports medicine physicians were assembled. These sources were used: The American Orthopedic Society for Sports Medicine Roster (I982), The American College of Sports Medicine 1982 Membership Directory, and E Sports Medicine Book (I978). Next, bases for possible inclusion or exclusion on the panel were discussed and agreed upon by the rotation directors and the researcher. The bases for being considered an opinion leader in sports medicine included holding or having held office in a sports medicine society, editing or frequently contributing to sports medicine journals, directing sports medicine clinics, serving as team physician for either a college or professional team, having a reputation among peers, or practicing sports medicine to a large degree. Next, the researcher and the rotation directors discussed the names, in the light of the criteria generated in step two. This process resulted in a list of 30 names. Finally, the researcher asked the rotation directors to suggest further opinion leaders. Three additional names were thus added to the list of panel members. Among those identified as leaders were six past presidents of the American Orthopedic Society for Sports Medicine, the current president, six directors of sports medicine clinics, eleven team physicians on the professional or major college level, a past president of the American College of Sports Medicine, editors of two sports medicine journals, and frequent contributors to sports medicine journals. This group proved the most difficult to obtain data from, and ultimately provided the lowest response rate (3l/33). The external review panel members, like the other participants, were offered no compensation for their time, but were sent a token gift-—a lottery ticket——intended as an inducement to complete and return the questionnaire. lmp specific rotationc answered lobe pro W Ex; writtenc to the Cl was titli schedule of rotat rotation posttest as a has Compare A 62 item The leg several 1heles‘ MC D lhree ( evalaai lacuh) Implementation of the Approach Implementation of the approach began when the researcher presented the l2 specific evaluation questions to the rotation directors. In discussion with the rotation directors, the researcher determined that none of the l2 questions could be answered entirely in the affirmative. Thus, additional evaluation information had to be provided. Identification of Rotation Intents Examination of documents. The researcher collected a copy of all the written documents about the rotation in sports medicine which had been distributed to the current residents of the September I982 rotation. The package of documents was titled, "Sports Medicine Curriculum in Family Practice." It contained a schedule for the month, an overview of the curricular content of the rotation, a list of rotation objectives, the resident's daily schedule, examples of forms used by rotation personnel, several reprints of articles of interest, a copy of the pre— posttest and answers, and a sports medicine bibliography. These documents served as a basis for the intents of the rotation against which implementation data were compared. A final document reviewed in the evaluation was the existing pre- posttest, a 62 item objective test given to residents on the first and last days of the rotation. The test had been developed by the rotation directors and had been in use for several years. There was one form of the test. No use was made of the results of the test. The researcher examined the test for content validity. Data Collection Data were collected on the twelve evaluation questions, divided into the three clusters: l) rotation context variables, 2) instructional variables, and 3) evaluation variables. The evaluator put together the data by interviewing the faculty and staff, through observation, and by review of rotation documents. The questiOI deternn rotatior inaplern compor revievv by inte data at inatrix each 6 coHect f the m residei and tt by the anphc atfixe the q Vbhfli lwen the t “nah about aboU' rotat vanc 8l questions about rotation context and instructional components attempted to determine the presence or absence of formal policies and procedures to guide the rotation, and to serve as the goals or standards against which actual implementation of the rotation was compared. Data for the evaluation variables component were obtained from current and former residents and the external review panel by a written questionnaire, from the faculty and staff of the rotation by interviews, and by observation. The evaluation variables component collected data about implementation, to compare against the intended goals or standards. A matrix is provided for determining which data collection techniques were used for each evaluation question (Figure 6). ln field testing the approach, data were collected first on the evaluation variables component. Former residents cyestionnaire. Data collection began in February 1983, with the mailing of a letter to all former residents. The letter informed the former residents that a questionnaire was forthcoming, explained the purpose of the study, and the importance of participating in it. The letter was followed one week later by the mailing of the questionnaire (Appendix A). Also included were a letter emphasizing the importance of the study, a return envelope with first-class postage affixed, and a token gift-~a keychain——used as an incentive to complete and return the questionnaire. Former residents were promised confidentiality. A plainly visible code number appeared on each questionnaire for follow-up purposes. Twenty-one of the 49 potential respondents returned their questionnaires within the first two weeks. The data sought from former residents were data about implementation——about what actually occurred during the rotation. These data about implementation were collected by using the questions in the third cluster, about evaluation variables. These data were used to compare with the data on rotation intents from the questions in the first two clusters (about rotation context variables and about instructional variables). Two additional included . Fifteen at Twc letter onl to out of three ren returns, t he was n< (lg residents rotations Therese. rotation. gitt--a k participc W82, tw< cOfltpleti impleme These ir The Que determi Dr Observa- iMmmq 82 Two weeks after the questionnaire was mailed, a first follow—up letter and additional questionnaires were sent to the 28 nonrespondents. This mailing included another postage-paid return envelope. Another incentive was not sent. Fifteen additional responses resulted from the first follow-up letter. Two weeks after the first follow-up mailing, a second follow-up-—a reminder letter only--was sent. lt generated ten more responses, for a total to that point of 46 out of a possible 49. Finally, two weeks after the second follow-up mailing, the three remaining nonrespondents were telephoned. This resulted in two additional returns, for a final total of 48 out of 49 possible responses. (The 49th resident said he was not willing to participate in the study.) Current residents questionnaire. Data were collected from the current residents at the end of each rotation during the data collection period, the rotations conducted in November I982, and January, February, and March I983. The researcher scheduled an hour with the residents on or about the last day of the rotation. The current residents questionnaire was given to each resident. A token gift—-a keychain--was also given. During the data collection period, eight residents participated in the rotation in sports medicine: one in the rotation in November I982, two in January l983, three in February, and two in March. All eight residents completed questionnaires. The data sought from current residents were implementation data of the same kind as that collected from former residents. These implementation data were also used to compare with rotation intents data. The questionnaires for former residents contained additional items that sought to determine how much sports medicine they now practiced. Data were also collected by three other means during the study: interviews, observation, and examination of rotation documents. The researcher conducted informal interviews both with the instructional staff of the rotation and with current residents. The instructional staff included trainers, the clinic nurse, and faculty p regarding problems and other _E_xt review p explainer participa (Appendi with p05 completi contiden tor folla The datc assessm. lorotat the cont T\ qUestior follow-r follow-r naire. telepho resPans 9 WOW TOTQfiO : V 83 faculty physicians. An attempt was made to collect data on a number of topics regarding rotation implementation: administrative aspects of the rotation, problems arising during the rotation, instructional strategies used in the rotation, and others. External review panel questionnaire. Collecting data from the external review panel was begun in February [983, with the mailing of a letter that explained the purpose of the study and the importance of the physician's participation in it. The letter was followed one week later by the questionnaire (Appendix B), a letter emphasizing the importance of the study, a return envelope with postage affixed, and a token gift--a lottery ticket-—as an incentive for completing and returning the questionnaire. The panel was also promised confidentiality, but a plainly visible code number appeared on each questionnaire for follow-up purposes. The initial mailing generated 22 of a possible 33 responses. The data sought from the external review panel were primarily used as an external assessment of the validity of the content of the rotation. The data were compared ‘ to rotation intents data (question clusters one and two) to determine the validity of the content the rotation. Two weeks after the initial mailing, a first follow—up letter and additional questionnaires were sent to the II nonrespondents. (No gift was included in the follow—up.) This effort generated 3 additional returns. Two weeks after the first follow—up 0 second follow-up letter was sent, but not providing another question- naire. The second follow-up resulted in 2 additional responses. Finally, a follow—up telephone campaign was conducted, which accounted for 4 responses, and a total response from the external review panel of 3l out of a possible 33. Observation and interview data. Implementation data were also collected through observation of residents, faculty and activities during the February I983 rotation. The researcher joined the residents as they participated in the rotation Evaluatio Question 84 Figure 6 Data Collection Techniques Matrix Evaluation Question i * * -x- 2 -x- -x- 3 * -)(- 4(- 4 * * -)(- 5 * * 9(- 6 * * * 7 * -x- * 8 * * -x- * 9 * * * as to * * * l l * * * and ob: team e lintorrr clinica Februa M T by the the ar variab the ev r1993 recon direci disser revie Discr \ durir part solvi Poss' majr lg, el6r and observed them and the various faculty during formal clinics, informal clinics, team events, and in orthopedic surgery. It was during this time that the interviews (informal questionning sessions) were held. Interviews with the trainers and clinical personnel were also conducted as questions arose, primarily during the February [983 rotation. Interviews with the faculty physicians have been ongoing. Documentation of Discreflncies The researcher compiled a list of discrepancies between the goals identified by the questions about rotation context variables and instructional variables and the actual implementation of the rotation (data generated from the evaluation variables question cluster). The discrepancies were included as an integral part of the evaluation report. Prgparation and Delivery of the Evaluation Report The researcher prepared a comprehensive report of evaluation findings and recommendations for improving the rotation, formally delivering it to the rotation directors in January of I985. More narrowly focused reports were prepared and disseminated to other audiences of the evaluation (e.g., former residents, external review panel members). Discrepancy Resolution The researcher and the rotation directors discussed major discrepancies during the delivery of the evaluation report and during a follow—up meeting. As part of the discussions, the researcher asked three major questions as a problem— solving routine: I) Why is there a discrepancy? 2) What corrective actions are possible? and 3) Which corrective action is best? Corrective solutions for the major discrepancies were agreed upon by the researcher and the rotation directors. Improvements to the Rotation The researcher presented the list of corrective solutions generated during element five, discrepancy resolution, to the rotation director. The researcher also ofterec T questic for cu extern Rotati aid in intervi Forme sectio rotati: impro asking practi first 1 Medii Medir offered to help the rotation director implement the improvements. Instruments Three separate questionnaires were developed for this research: I) a questionnaire for former residents (Participants Questionnaire), 2) a questionnaire for current residents (Rotators Questionnaire), and 3) a questionnaire for the external review panel (Survey of Important Components of A Family Practice Rotation in Sports Medicine). The evaluator developed two checklists —-one as an aid in observing and interviewing the rotation faculty, the other in observing and interviewing the residents. Former and Current Residents Questionnaires The first two questionnaires were of similar design and content, including sections focusing on various experiences encountered during the course of the rotation, and the resident‘s perceptions of the rotation and how it could be improved. In addition, the questionnaire for former residents contained a section asking for information pertaining mostly to the degree that they currently practiced sports medicine. The items for the questionnaires were developed by the researcher. As a first step, an internal document called "Minimal Competencies Required for Sports Medicine Family Practice Physicians" was produced by the Coordinator of Sports Medicine, the Director of Sports Medicine, and the researcher. The document covered five broad areas of concern to the sports medicine faculty: the philosophy of sport, basic science of sport, preventive sports medicine, patient care, and research in sports medicine. Within each of these five areas were listed the minimal skills and abilities the faculty expected from the residents by the completion of the rotation. Items were then developed to determine how often during the rotation the residents practiced these skills and abilities, observed them, or discr A made c a reas prereqi dynam primar (catch value staten reside withov were was tl Thus, taken lllichi comp level Whetl deter studi quart QUQS' slrat list 87 or discussed them with the faculty. As a second step, the Coordinator of Sports Medicine and the researcher made a list of the prerequisite training necessary to ensure that the resident stood a reasonable chance of success in the rotation. The original list included prerequisites in areas such as anatomy, physiology, pharmacology, pharmaco- dynamics, kinesiology, nutrition, the physical exam, psychology, concepts of primary care, physical therapy, advanced cardiac life support and in a general (catch—all) category. The prerequisites were thought too detailed to be of practical value in designing questionnaire items. A second try produced a more general statement of prerequisities. The questionnaire item asked former and current residents about the number of terms they studied various disciplines and topics, without an attempt to specify the exact content of the courses. Two assumptions were made in developing the Minimal Competencies document: One assumption was that most medical schools courses offered essentially the same information. Thus, if a resident had studied at a different school, the resident's education was taken to be not substantively different from that of a resident who studied at Michigan State University. The second assumption was that successfully completing courses in the prerequisite areas on the medical school or graduate level sufficiently constituted meeting the prerequisite. Thus, to determine whether residents had attained the necessary prerequisites, it was necessary to determine which courses had been taken and which broad topical areas had been studied. To avoid confusion between the differing lengths of semesters and quarters at different schools, the word "term" was used in constructing questionnaire items. As a third step, the researcher prepared a list of all the instructional strategies thought to be used by the faculty during the rotation. A check of this list of strategies with the faculty physicians and the nurse manager led to the inclusi itennS' rate tt l copies forspt change doctor these instru 88 inclusion of some additional strategies and the exclusion of others. Questionnaire items were then developed regarding these strategies. The residents were asked to rate the relative effectiveness of the strategies for learning sports medicine. Next, items were developed for the remaining evaluation questions, and typed copies of the instruments were submitted to the faculty physicians of the rotation for specific recommendations about technical accuracy and wording. When these changes had been made, the questionnaires were submitted to the researcher's doctoral committee and a research consultant for further recommendations. With these additional recommendations incorporated into the questionnaires, the instruments were prepared for pilot testing. A data-collection strategy used early in the study was to acquire a broad range of information about both former and current residents. This information was intended to serve as a data base for possible future research. As a result, many questionnaire items were developed that did not relate to the evaluation questions. A number of additional questionnaire items were used to collect data but were not used in reporting the results. The questionnaire items and the evaluation questions for which the items provided data are listed in Table 2. Evaluation Questions I through 7 were answered by examination of rotation documents, observation of the participants and activities of the rotation, and discussion with faculty, staff, and residents. To answer Evaluation Question 2, the researcher also reviewed the resumes and curricula vitae of the faculty and staff of the rotation. The matrix shown in Figure 6 provides the data—collection techniques used to obtain data for each of the l2 evaluation questions. Table 2 lists the particular questionnaire items that provide data to answer Evaluation Questions 8 through l2. @1911; P' former and agr time o comple explain timed comple researc improv T incorpr C Becaus the ad former in layc EM during exterr compr Physic of y, 89 Pilot Testing (Farmer and Current Residents) Pilot testing consisted of administering the respective questionnaires to two former residents and one current resident. Two former residents were identified and agreed to respond. Only one resident was participating in the rotation at the time of pilot testing, and he agreed to respond. In each case, the resident completed the questionnaire while the researcher remained in a nearby room to explain any ambiguities or answer any problems that arose. The researcher also timed the pilot tests to determine whether the amount of time required to complete the questionnaire was excessive. At the end of each pilot test, the researcher talked with the resident to try to see whether the instruments could be improved. The former residents made suggestions about format and layout, which were incorporated. One major problem emerged in the pilot test with the current resident. Because of the format of one question, and the resident‘s diligence in answering it, the administration required 80 minutes (compared with 25 and 35 minutes for the former residents). The question was subsequently reformatted, and minor changes in layout made. External Review Panel Questionnaire The questionnaire for the external review panel of physicians was developed during the late fall of I982 and early winter of I983. lt attempted to provide an external assessment of the validity of the content of the rotation. As a first step, items were generated from the list of minimally required competencies. The questionnaire was typed and submitted for recommendations to 0 physician who treats athletes but was not connected with the rotation. This physician made several recommendations, which resulted in changes in the format of the instrument as well as the addition of several items. With these changes incorpr physici sugges and 9 ltenas nfleva questh EH91] l physic inrwor loHow Sports testc dead buhdi provh toHov vahdi qUest demc qUest dunr Vahd 9O incorporated, the instrument was submitted to a research consultant, the faculty physicians, and the researcher's doctoral committee. They made minor suggestions, which were incorporated in the questionnaire. Evaluation questions 8 and 9 had two parts: one part about implementation, the other about relevance. Items one through fifty-five of the questionnaire provided data used to answer the relevance parts of evaluation questions 8 and 9. Items 56 to 59 of the questionnaire seek information of a general nature. Pilot Testing (External Review Panel) Pilot testing of the questionnaire consisted of resubmitting the form to the physician who made recommendations to the first iteration. Several minor changes in wording were recommended and incorporated. The questionnaire was given the following title: Survey of Important Components of a Family Practice Rotation in Sports Medicine. Validity of the Questionnaires Ebel (I979, pp. 299—300) states that "validity is not so much a property of a test as it is of the use made of it." Valid use of a test is facilitated by stating clearly in the test specifications what the test is intended to measure and then building the test to meet those specifications, checking its reliability, and providing clear instructions on how to use it. These recommendations were followed during the development of the three questionnaires. To address content validity, physicians who practiced sports medicine were asked to review the questionnaires. Acting on the advice of Ebel (I979, p. 298) "Validity can be demonstrated best by tryout after the test has been constructed," each questionnaire was pilot tested before implementation. Validity of former and current residents questionnaires. A primary concern during the design of all three questionnaires was content validity. To be "content valid", the questionnaires had to "sample representatively and adequately the Evaluat' implemr Evalua relevar I I r i Evalu Evali skills EVal (ital , _,,_,C____,~_.________z_ _ 9I Table 2 Questionnaire Specifications Evaluation Question 8: Is the curriculum systematically evaluated for implementation and relevance? Implementation Data: Participants (former residents) Questionnaire (and corresponding Rotators (current residents) Questionnaire) items 30 (discussion competencies), 3t (performance competencies), 32 (observation competencies), 36-368 (implementation of the curriculum according to written materials), and 65 (curricular areas reported not addressed). Evaluation of implementation of the curriculum is made by the evaluator who analyzes from which curricular area individual items are reported not implemented. Additional data were generated from examination of rotation documents. Relevance Data: External Review Panel (Survey of Important Components of A Family Practice Rotation in Sports Medicine) Questionnaire items I-25 (Patient Care), 26-3I (Basic Science of Sport), 32-43 (Preventive Sports Medicine), 44-60 Research in Sports Medicine), and SI-55 (Philosophy of Sport). Evaluation Question 9: Are rotation objectives evaluated for implementation and relevance? Implementation Data: Participants Questionnaire (and corresponding Rotators Questionnaire) items 30 (discussion competencies), 3t (performance competencies), 32 (observation competencies), 35—358 (implementation of individual objectives according to written materials), and 65 (individual objectives reported not addressed). Additional data were generated from examination of rotation documents. Relevance Data: External Review Panel Questionnaire items l-SS, examined as individual objectives irrespective of the curricular area in which they were contained. Evaluation Question to: Are instructional strategies evaluated for implementation and effectiveness? Participants Questionnaire (and corresponding Rotators Questionnaire) item 33. Additional data were generated from examination of rotation documents. Evaluation Question ll: Are residents evaluated for prerequisites, knowledge, skills, and attitude? Participants Questionnaire (and corresponding Rotators Questionnaire) items 6, 7, I2, l3, l4 (for knowledge and skills), 9, I8, 67 (for attitude), and 29 (for prerequisites). Additional data were generated from examination of rotation documents. Evaluation Question [2: Are faculty and staff of the rotation evaluated for qualifications, availability, and teaching skills? Participants Questionnaire (and corresponding Rotators Questionnaire) items 58-59 (qualifications), 60—6l (teaching skills). conter questi wiht I were compr areas. ndnin Sport teach quest rotat incor then doct< verd were the ques cont rose the intc vah ens enc 92 content of the course of instruction" (Ebel, I979, p. 303). In designing the questionnaires for the current and former residents, the researcher worked closely with the Coordinator of Sports Medicine. Five areas of concern in the rotation were identified. The researcher and the Coordinator produced a list of minimal competencies to be attained by the family practice physician in each of the five areas. Questionnaire items were developed by the researcher from the list of minimal competencies. The researcher also worked closely with the Coordinator of Sports Medicine in developing questionnaire items about resident prerequisites and teaching strategies used in the rotation. The researcher developed questionnaire items for the remaining evaluation questions and submitted them for review to two of the faculty physicians (the rotation directors). The physicians made recommendations, which were incorporated in the next iteration of questionnaire items. The questionnaires were then submitted to a research consultant and several members of the researcher's doctoral committee. Recommendations were made and incorporated into the final version of the questionnaire. Validity of external review panel questionnaire. Content validity concerns were approached somewhat differently in developing the questionnaire directed at the external review panel. The original items for the external review panel questionnaire were generated, as before, from the list of minimally required competencies. The questionnaire was then submitted for review and recommendations to a physician who treats athletes but was not connected with the rotation. The physician made several recommendations that were incorporated into the questionnaire. The external review panel questionnaire served as a validity assessment of the content of the rotation in sports medicine. To help ensure that no content areas were omitted, the questionnaire was designed to encourage respondents to write in important competencies that were not addressed in the revievv commi' liejhtj C Questh was th l98h p. half c curren I Practi the tis :— (D 9... / tillers InIEr' 90thr APpe 93 in the rotation. As a final validity measure, the questionnaire was submitted for review to a research consultant and several members of the researcher's doctoral committee. Suggestions were made and incorporated into the final version. Reliability of the Questionnaires One questionnaire item in both the Participants and the Rotators Questionnaires was constructed as a scale item. The reliability of this scale item was therefore assessed. The assessment used was Cronbach's Alpha (Hull & Nie, I98I, p. 256), which produces a coefficient of internal consistency based on all split- half combinations of scale items. Internal consistency for the 56 former and current residents on the scale was: Questionnaire Item 33 = .68 The scale items used in the Survey of Important Components of A Family Practice Rotation in Sports Medicine were the individual item responses in each of the five broad curricular areas. Internal consistency for the curricular areas was: Philosophy of Sport : .44 Basic Science of Sport : .59 Preventive Sports Medicine = .86 Clinical Science : .90 Research in Sports Medicine : .77 Checklists The researcher developed two checklists -—one as an aid in observing and interviewing the faculty and staff of the rotation, the other in observing and interviewing the residents. The checklists were designed to assist the researcher in gathering data about implementation of the rotation. They are presented in Appendices C and D. questi the rc repor4 tables narra' Michi quest for st lortl rep the me' evc the DI( 94 Data—Analysis Procedures Data analyses were selected that were appropriate to the clusters of questions. Thus, data are reported primarily in a descriptive, narrative format for the rotation context variables and instructional variables. Quantitative data are reported for the evaluation variables. These evaluation variables are presented in tables using frequencies and percentages and accompanied by explanatory narrative. Responses to open-ended questions are listed in appendices. Data from the questionnaires were prepared for optical scanning at the Michigan State University Scoring Office. Data were transcribed from individual questionnaires to IO-column scoring sheets. The data were entered and transferred for storage to magnetic tape. The data were analyzed by SPSS (Statistical Package for the Social Sciences: Nie, Hull, Jenkins, Steinbrenner, and Bent, I975). Metaevaluation of the Approach Metaevaluation was defined by Stufflebeam (I98l) as: The process of delineating, obtaining and using descriptive and judgmental information about the utility, practicality, ethics and technical adequacy of an evaluation in order to guide the evaluation and publicly to report its strengths and weaknesses. (p. I5I) Stufflebeam's definition identifies two main roles of metaevaluation. The guiding role is formative metaevaluation. One role is to provide information to guide on evaluation during its progress. The guiding role is what Stufflebeam refers to as "formative metaevaluation.“ The second main role of metaevaluation is the public reporting of the strengths and weaknesses of an evaluation. The public report of these strengths and weaknesses is what Stufflebeam refers to as "summative metaevaluation.“ A summative metaevaluation was designed and conducted to evaluate the prototype approach, and focused on several areas: the usefulness of the information gathered using the approach; the feasibility of the approach; the propriety of the approach; and the accuracy of the information gathered by the appror rotafi and n dheC' the n 0m after conh dhec secor dhec direr QUdIr Appr cons rnec the 95 approach. The metaevaluation answered four questions presented in Chapter One: I. Was the approach useful in providing information to the rotation directors? 2. Was the approach practical? 3. Was the approach ethical? 4. Did the approach collect accurate information about the rotation? Participants Two sources provided data for the metaevaluation: the two directors of the rotation in sports medicine, and an external consultant experienced in evaluating and metaevaluating graduate medical education programs. One of the two rotation directors and the consultant responded to metaevaluation procedures designed by the researcher. Data Collection Rotation Directors. Metaevaluation data were collected in January I985, after discrepancies were resolved, the fifth step of the evaluation. The researcher contacted the rotation directors and arranged a meeting. One of the two rotation directors could not come to the meeting, and subsequent efforts to arrange a second meeting were unsuccessful. The researcher interviewed the one rotation director, using a structured interview guide. The researcher asked the rotation director a number of open-ended questions, and responses were recorded on audiotape and transcribed for analysis. The transcription is presented in Appendix E. Consultant. The researcher identified a qualified medical education metaevaluation consultant, experienced in both the evaluation and metaevaluation of graduate medical education programs. The consultant was contacted and agreed to evaluate the approach used in the primary evaluation. This was done in February I985. The research rotathsn on meth recommi instrunae evaluati- researct ofany q At anange consultc meeting dhterer consult cansulh quesfior torana New T One gu second 96 researcher sent the consultant 0 copy of the evaluation report delivered to the rotation directors. The report included an executive summary as well as sections on methodology, presentation of data, and discussion of results, conclusions, and recommendations. The report also included several appendices: sample instruments used to collect data during the primary evaluation, a diagram of the evaluation approach, and conclusions and recommendations reached by the researcher. The consultant was instructed to review the report and prepare a list of any questions he had about the design and conduct of the primary evaluation. After the consultant had reviewed the evaluation report, a meeting was arranged between the researcher and the consultant. The researcher answered the consultant's questions about the design and conduct of the evaluation. At a second meeting, the researcher interviewed the metaevaluation consultant, using a different structured interview guide. The researcher asked the metaevaluation consultant 0 number of open-ended questions. In addition, the researcher asked the consultant to supply responses to the researcher's four research (metaevaluation) questions. The consultant's responses were recorded on audiotape and transcribed for analysis. The transcription is presented in Appendix F. Metaevaluation Instruments Two separate metaevaluation structured interview guides were developed. One guide was developed for the directors of the rotation in sports medicine, the second for the metaevaluation consultant. Both guides were designed to supply data that were used to help summatively determine the effectiveness of the prototype approach used to evaluate the rotation in sports medicine. The first step in developing each of the instruments was to identify evaluation standards related to the research questions. The source of the standards selected was Standards for Evaluations of Educational Programs, Prg‘ects and Materials (The Joint Committee, I98I). One or more standards were selected from Hts pt questic dander researr approa rnetae' areas: practi inlorn effect guide propr typed com the g m prin the (tile 6V0 97 this publication for each of the research questions. Specific metaevaluation questions were then developed that addressed the concepts within each of the standards. Both guides contained items which were not specifically related to the research questions but which dealt with the effectiveness of the evaluation approach. The guides designed for both the rotation directors and the metaevaluation consultant are presented in Appendices G and H. Structured Interview Guide (Rotation Director) The guide for the interview of the rotation director focused largely on two areas: the utility of the information provided by the primary evaluation about the practical information needs of the rotation directors; and certain additional information not directly related to the four research questions, but about the effectiveness of the evaluation approach. Additional questions appeared on the guide prepared for the interviews of the rotation directors: about the feasibility, propriety, and accuracy of the primary evaluation. A first draft of the guide was typed and submitted for review to a member of the researcher's doctoral committee, who made several recommendations, which were incorporated before the guide was prepared for use with the rotation directors. Structured Interview Guide (Consultant) The guide for the interview of the metaevaluation consultant focused primarily on three areas: the accuracy of information obtained using the approach, the propriety of the approach, and additional information not specifically related to the research questions but about the effectiveness of the evaluation approach. Questions about accuracy of information attempted to determine whether the primary evaluation produced sound information and whether conclusions reached in the evaluation were logically linked to the data. Questions about propriety attempted to determine whether the rights of people affected by the primary evaluation approach were protected. the WGI' COT) Val' vali car The @ Re sin re: III! 98 A first draft of the guide was typed and submitted for review to a member of the researcher's doctoral committee, who made several recommendations, which were incorporated before the guide was prepared for use with the metaevaluation consultant. Validity and Reliability of the Metaevaluation Structured Interview Guides In developing the metaevaluation guides, the researcher focused on the validity of these instruments. Most of the questions were developed directly from concepts in evaluation standards that were related to the four research questions. The source of the evaluation standards was Standards for Evaluations of Educational Programs, Projects and Materials (The Joint Committee, I98I). Reliability of the guides was not addressed because one guide was designed for a single respondent and the other guide for just two respondents. Data Analysis Procedures Data from the summative metaevaluation were used to answer the four research (metaevaluation) questions. The responses of the rotation director and the metaevaluation consultant served as the basis for formulating responses to the research (metaevaluation) questions. Metaevaluation data were organized by listing the data in tables. Using data contained in the tabular format, the researcher then provided summary answers to the four research (metaevaluation) questions. The results are presented in Chapter Four. Mm An evaluation approach was designed that was more ideally suited to the special evaluation requirements of a residency rotation than existing program evaluation models. The approach was field tested on one rotation, the rotation in sports medicine. The evaluation was conducted using various methods and sources of data. Data were analyzed using descriptive statistics. An evaluation report based on metaeval and accu on the r presente 99 based on the data was prepared and presented to the rotation directors. Finally, a metaevaluation was performed to determine the usefulness, feasibility, prOpriety and accuracy of the evaluation approach. The results of the evaluation performed on the rotation in sports medicine and the results of the metaevaluation are presented in Chapter Four. evr for de Chapter Four Results Introduction This chapter presents the results of the primary evaluation, the delivery of the evaluation report, and the results of the metaevaluation of the approach used to evaluate the rotation in sports medicine. First, the results of the primary evaluation for each of the I2 evaluation questions, and interpretation of the results for each evaluation question. Next, a discussion of the evaluation report and the delivery of the report to the rotation directors. Third, the results of the metaevaluation of the evaluation approach, and interpretation of the results for each of the four research (metaevaluation) questions. Finally, a summary of the chapter. Results of the Primary Evaluation The [2 evaluation questions designed into the evaluation approach were clustered into three areas: I) rotation context variables, 2) instructional variables, and 3) evaluation variables. Each question will be listed and followed by the presentation and interpretation of data collected for the question. One primary procedure was used to answer all three rotation context questions: examination of rotation documents. In addition to the primary data- collection procedure, data from the questionnaires administered to former and current residents were used to support the researcher's observations and discussions. Such data were frequently supplied from response to Open-ended questions on the questionnaires. Bgtation Context Variables Evaluation Question I. Are there written policies and procedures to guide the rotation? I00 USI Po an dir IOI Information obtained in the evaluation revealed that policies and procedures used to guide and direct the rotation are currently not formalized in writing. Policies and procedures are not currently systematically disseminated to faculty and staff, but are often made in an ad hoc fashion based upon the rotation directors' knowledge of the rotation. Evaluation Question 2. Are there written job descriptions md qualifications for faculty and staff positions? Data gathered during the evaluation revealed that job descriptions and qualifications for faculty and staff positions in the rotation do not exist. Such descriptions are necessary for properly matching individuals to job tasks. One reason that there are no descriptions and qualifications is that sports medicine is not a specialty area of family practice or orthopedic surgery. It is an interest area of practicioners in those (and other) fields. Thus, there are simply no nationally established criteria for those in the field. In the absence of such criteria, the researcher attempted to determine the qualifications of the faculty and staff of the rotation in sports medicine, by a two-step process: discussing with the rotation director the minimum faculty and staff required to teach residents participating in the rotation, and reviewing the resumes and curricula vitae of the faculty and staff identified by the rotation director. From the review, the researcher attempted to determine whether the faculty and staff seemed qualified to teach sports medicine. The rotation director identified the following faculty and staff as the minimal personnel required to conduct the rotation: two faculty physicians, one head trainer, and one registered nurse. The researcher's assessments of these faculty members qualifications for teaching sports medicine follows. The first of the two rotation faculty physicians holds both B.S. and MD. degrees. During his family practice residency program he was associated with an l02 early leader in the area of sports medicine. Following his residency, this physician came to MSU as a Family Practice Fellow, and was subsequently appointed to the faculty of the Department of Family Practice. He was board certified in Family Practice in I976. He has been a member of the American College of Sports Medicine since I976, Director of Sports Medicine at MSU since I980, and a member of the Big l0 Collegiate Conference Team Physicians since I977. He has taught various courses in sports medicine and given numerous presentations in sports- related areas. He has authored or coauthored fifteen publications in sports medicine. He has coordinated eight local and regional sports medicine conferences and won several grants for research of sports medicine. On the basis of his long and continued interest and involvement in various aspects of sports medicine practice, teaching, service, and research, it seems that this physician must be qualified to teach sports medicine in the rotation. The second faculty physician holds B.S., M.S., and MD. degrees. He came to MSU as a Family Practice Fellow and was subsequently appointed to the Family Practice faculty. He was board certified in Family Practice in I976. He has been a member of the American College of Sports Medicine since I970, the Society of Teachers of Family Medicine since I978, and the United States Sports Academy since l98l. He has been Coordinator of Sports Medicine at MSU since I980, and a team physician at MSU since I977. He has taught a variety of sports medicine courses and lectured frequently on various aspects of sports medicine. He has numerous publications in the area of sports medicine, and is a frequent lecturer and presentor on a variety of sports medicine topics. Because of his long and continuing interest and involvement in the interest area of sports medicine, including practice, teaching, service, and research, it appears that this faculty physician must be qualified to teach residents in the rotation in Sports Medicine. In addition to the rotation directors, three other physicians serve as faculty I03 physicians. Two of these physicians hold D.O. degrees, the third holds an MD. degree. The faculty physicians are board certified in Pediatrics, Maxillofacial Surgery, and Family Practice. Each of the faculty physicians has published articles on various aspects of sports medicine. The faculty physicians have a combined total of l4 years teaching experience in the rotation in sports medicine and additional teaching experience in other medical areas. One of the rotation faculty physicians has served as team physician in a high school and at the United States Olympic Training Center. Individually and as a group, the faculty physicians serving the rotation in sports medicine appear to be well qualified to teach residents. The head athletic trainer hold a bachelor's degree in physical education and a master's degree in athletic training. He has been an athletic trainer for l9 years, at Colorado State University and Michigan State University. He has taught undergraduate and graduate courses in injury prevention, care and rehabilitation, and related areas to both students and coaches for I2 years. In addition, he has published seven articles on subjects related to sports medicine. He has given numerous lectures on a variety of sports medicine and athletic training topics. He is a member of both the National Athletic Training Association and the American College of Sports Medicine. Currently he serves the Department of Intercollegiate Athletics at Michigan State University as Coordinator of Athletic Training and Rehabilitation, and the Physical Education Department as an educational specialist. On these bases the head athletic trainer appears well qualified to teach residents in the rotation in sports medicine. Finally, the nurse/manager of the Sports Medicine Clinic (formal referral clinic) holds a bachelor's degree in nursing and is licensed by the State of Michigan. She has I5 years of nursing experience and for several years has been a clinical instructor in a local nursing program. She has published an article on Sports - ‘r ".1; .... . l04 medicine. She has assumed increasing administrative responsibilities throughout her career. On these bases, she appears well qualified to perform her current duties as nurse/manager of the formal clinic. Evaluation Question 3. Are there written descriptions of the following resources? -availability of faculty and staff -injured or ill athletes -—facilities (e.g., clinical spaces, rehabilitation equipment) —budget -—educational materials (e.g., library, slides) Information obtained in the evaluation indicated that such descriptions do not exist. Some information may be gleaned from rotation documents, but the documents are often outdated and unavailable to residents and faculty. Providing these descriptions permits faculty, staff and residents to obtain information so that they may more effectively conduct and participate in the rotation. . Availabili’gy of faculty and staff. Three documents indicated the availability of various faculty and staff of the rotation. The Resident's Daily Schedule on Rotation listed by day and time the activities in which the residents were to be engaged. Rehabilitation and journal-review activities were scheduled to be conducted with trainers. Coverage of team practices was to have included consultation with equipment managers, coaches, trainers and other consultants, and visits to local high school programs. The other two documents were essentially memoranda given to the residents at the orientation meeting. These memoranda were titled figrts Coveragefiebruary I983 and Sports Medicine Plysician Training Room Coverage. The first memo listed the dates, times, and locations of various varsity athletic events, and the faculty physician and residents assigned to cover the event. The second memo listed the days on which and locations at which each faculty physician attended the formal and informal clinics. Comments made by l05 residents (questionnaire data) indicated that residents sometimes had difficulty meeting with the rotation directors. "A schedule of Dr. McKeag's and Dr. Hough's free time would be valuable so I could talk to them, or schedule free time regularly." "I would have liked l—2 hours sometime in the rotation set aside to answer specific questions about sports medicine and my future plans." Finally, "A more formal schedule should be devised." The researcher concludes that the Resident's Daily Schedule be updated to reflect more accurately the actual activities engaged in by residents and the availability of faculty and staff of the rotation. Injured or ill athletes. An important "resource" of the rotation in sports medicine is the athletes who incur the various sports related injuries and contract the illnesses. Residents observe, discuss, diagnose, and treat these injured and ill athletes during the rotation. A review of rotation documents revealed no description of ill or injured athletes to be treated by residents during the rotation. Some information could be gleaned from the educational objectives given to residents, however. The first objective required residents to be able to manage twenty common sports medicine problems that were listed with the objectives. There was no mention of the percentages of varsity, intramural, recreational, elite, or high school athletes to be seen (or who had been seen in the past). One resident's comment points out the need for such description. "I thought I was going to get some exposure at the high school level." Facilities. One of the rotation documents, Sports Coverage, February l983, listed various facilities used in event coverage during the rotation. The document failed to list other facilities such as athletes' training rooms, clinical and rehabilitation spaces, and rehabilitation equipment. 3:519:24. There is no formal budget for the rotation in sports medicine. In discussions with the rotation director, the researcher found that the facilities and IO6 supplies used in the rotation were provided by the Department of Intercollegiate Athletics (DIA). Such supplies include all medications and first-aid materials. In addition, the DIA budget allocates monies for laboratory, X-rays, and other fees. The rotation directors‘ time is allocated to the rotation through agreement with the Chairman of the Department of Family Practice in the College of Human Medicine. Since the rotation in sports medicine has no budget, one other document, the Faculty Activity Form, is pertinent. The Faculty Activity Form documents the number of contact hours the rotation directors and faculty physicians spend with the residents and in other rotation—related activities. Regarding the rotation in sports medicine, the Faculty Activity Form is used for two main purposes. One purpose is to account for the amount of time the rotation directors and faculty physicians spend on the rotation. This document is helpful when planning with the chairman of the Department of Family Practice for future faculty support to the rotation. The second purpose of the form is to document involvement of the rotation faculty in the Sports Medicine Division. This documentation is valuable in annual contract negotiations with the DIA. While monies are not allocated to the rotation itself, they are allocated to the Sports Medicine Division. Because the rotation in sports medicine is an extension of the Sports Medicine Division, any increase in resources allocated to the division enhances the position of the rotation (e.g., if the DIA agrees to budget more faculty hours, residents will have increased access to physicians while they are practicing sports medicine). Educational materials. A review of rotation documents reveals no description of the educational materials available to residents. A sports medicine bibliography is passed out to residents. The bibliography lists books, journal articles, monographs, and the major sports medicine journals to be reviewed by residents. Instructional Variables Evaluation Question 4. Are there written prerequisites for residents? Evaluation data revealed that specific resident prerequisites required for admittance to the rotation do not exist. Some general educational criteria have been agreed upon by the rotation directors, but are sometimes ignored when admitting residents to the rotation. Evaluation Question 5. Are there written objectives for the rotation ? Information obtained in the evaluation indicated that objectives for the rotation do exist. The objectives were written in general rather than specific terms, however. The information obtained also indicated that the written objectives do not accurately reflect the emphasis of the curriculum of the rotation as it is implemented. Evaluation Question 6. Is there a written plan for the implementation of the curriculum (content areas addressed in the rotation)? Data obtained during the evaluation indicated that a format plan for implementing the curricular areas addressed in the rotation does not exist. A document was distributed to residents which listed the areas of curriculum to be studied. This document was sometimes misconstrued by residents as additional educational objectives. Evaluation Question 7. Is there a written description of instructional strategies to be used to implement the curriculum? Evaluation data revealed that a formal plan for implementing the curriculum does not exist. One document distributed to residents lists some, but not all, of the activities required of residents as they participate in the rotation. The document does not match specific objectives with activities for attaining the objectives, however . [08 Evaluation Variables Evaluation Question 8. Is the curriculum systematically evaluated for implementation and relevance? The information collected during the rotation indicates that no formal (written) evaluation is conducted for either implementation or relevance of the curriculum. In addition, evaluation data disclosed no informal feedback sessions between faculty members or between faculty and residents about implementation or relevance of the curriculum. Additional data were required to answer the evaluation questions. Because there was insufficient evaluation information about the rotation curriculum and objectives, and because of the importance of such evaluation information to the prototype evaluation approach, the researcher developed three questionnaires. The instruments supplemented existing data, and contributed to the assessment of the implementation and relevance of the rotation in sports medicine. Since there was no clear specification of the rotation curriculum and objectives, the researcher, in collaboration with the rotation director, combined the two existing curriculum and objectives documents. The new document was titled Minimal Competencies Recyired For Sports Medicine FamilLPractice Phflfians and provided a single source of curriculum and objectives for the rotation. The researcher used the minimal competencies to help design the questionnaires to provide data on the implementation and relevance of the rotation. The questionnaires reflected the competencies listed in each of the five curricular areas of the minimal competencies document. With it, the researcher attempted to make two determinations: whether or not the curriculum in general and the specific objectives within the curriculum were implemented as planned, and whether or not the curriculum and the objectives were relevant for beginning the practice of sports medicine. l09 Three questionnaires were designed. One questionnaire was to assess the implementation of the curriculum and objectives of the rotation from the perspective of former residents: Participants Questionnaire (Appendix A). A second questionnaire-«Rotators Questionnaire--was to assess the implementation of curriculum and objectives of the rotation from the perspective of current residents of the rotation. The final questionnaire, Survey of Important Components of a Family Practice Rotation in Sports Medicine (Appendix B), was designed to assess the relevance of the curriculum and objectives of the rotation from the perspective of a panel of sports medicine experts. This external review panel consisted of well-known sports medicine primary care physicians and orthopedic surgeons. The former and current residents thus provided an internal assessment of the implementation of the curriculum and objectives of the rotation while the external review panel validated the relevance of the curriculum and objectives. In developing the instruments, the researcher assessed the implementation and relevance of both the curriculum and objectives together. Rather than report identical data twice, the data are combined and presented once in Evaluation Question 9. Evaluation Question 9. Are rotation objectives evaluated for implementation and relevance? The data collected during the evaluation indicate that no formal (written) evaluation is conducted for either implementation or relevance of rotation objectives. Additional evaluation data disclosed no informal evaluation procedures designed to elicit such information. The data used to answer the implementation portion of Evaluation Question 9 came from two sources: rotation documents and written responses to questionnaires. The objectives were found not to be precisely defined or systematically listed, and rotation objectives not evaluated. To obtain information ?—-——T "— 110 about the implementation and relevance of both the objectives and curriculum, the researcher designed the Participants Questionnaire (for former residents), the Rotators Questionnaire (for current residents), and the Survey of Important Components of a Family Practice Rotation in Sports Medicine (for the external review panel). The findings from the three questionnaires are presented next. Implementation of Curriculum and Objectives The study was originally designed to separate the responses of former and current: residents, and to detect shifts in curricular emphases. When data were collected, however, no differences were detected in response patterns of former and current residents. The responses of former and current residents, therefore, have been collapsed into single totals. Absolute frequencies and adjusted percentages are used in the tables which follow. The percentages are adjusted to reflect the number of respondents on each particular questionnaire item. The data regarding curriculum descriptions are presented in Table 3. Table 3 Curriculum Descriflions No Questionnaire Item X93 & Response 35. Did you receive a description of the curriculum? 4O 4 12 35a. Was the description clear? 37 2 17 35b. Was the description consistent with the actual implementation of the rotation? 23 3 3O 36. Did you receive descriptions of the Specific behaviors expected of you during the rotation? 39 6 11 36a. Were the descriptions clear? 36 1 19 36b. Were the descriptions consistent with the 2 36 actual implementation of the rotation? 18 NOTE. N .-: 56 Most residents reported that they had received and understood descriptions of the curriculum. Fewer reported that the descriptions of the curriculum were consistent with the implementation of the rotation, however. The reports were similar about objectives of the rotation. The first area assessed in the Participants and Rotators Questionnaires was the curriculum of the rotation. A second area assessed by these questionnaires was the specific competencies in each curricular area. Questions about the competencies to be acquired were devel0ped for both questionnaires. Both former and current residents were asked how often competencies were implemented in the rotation. The purpose of these questions was to compare the intents of the rotation as stated in the minimal competencies document with the actual implementation of the rotation. The questions were designed to provide data about knowledge and skill components of the rotation curriculum. The researcher sought to determine how many times residents discussed, observed, or practiced competencies to be acquired. Competencies were divided into three kinds: competencies that could be acquired by residents primarily through discussion with the rotation faculty--e.g., understanding the contrasting sports medicine philosophies of orthopedics and family practice (Participant's Questionnaire item 30, Appendix A); competencies that could be acquired primarily through observation-~e.g., assessing injuries such as dislocations and fractures (Participant's Questionnaire item 32); and competencies that required some kind of performance by the residents--e.g., developing treatment plans, counseling athletes about dietary fads (Participant's Questionnaire item 3|). Absolute frequencies and adjusted percentages for the items relating to the implementation of the curriculum and objectives are presented in Tables 4, 5, and 6. The following tables report how often the residents said they were engaged in the experience during the rotation. Competencies learned through discussion. The rotation directors indicated a number of competencies would be acquired through discussion. Some of the 112 competencies to be discussed were equivalent to those listed in the minimal competencies document. Others were components of overall competencies to be attained by residents during the rotation. Data from both former and current residents were tabulated and summarized, and are presented in Appendix I. In analyzing the data, the researcher sought discrepancies between what the rotation directors intended the residents to learn and the implementation (actual resident experiences) of the rotation. Discrepancies greater than 50% between the intended discussions and residents reporting no such discussions were noted and described. Twenty-three competencies were intended to be attained through discussion. Nineteen competencies fell below the 50% cutoff score. Half or more than half of the residents reported four competencies were not discussed during the rotation. They are presented in Table 4. Table 4 Discrepancies in Implementation of Discussion Comgtencies mestionnaire Item Number of Discussions -_0_- _l_-_5 6 or more 30d. How many times during the rotation did you discuss the equipment and supplies that the school must furnish before an 31 25 -0- athletic event begins? (55%) (45%) (0.0%) 30n. How many times during the rotation did you discuss the difference in treatment plans (medication) for high school 28 25 3 and college athletes? (50%) (45%) (5%) 30u. How many times during the rotation did you discuss medical care considerations for handi- 42 ll 3 capped athletes? (75%) (20%) (5%) 30w. How many times during the rotation did you discuss re- search methods in sports 31 22 3 medicine? (55%) (39%) (5%) Note. Percentages may not sum to 1.0 because of rounding. NOTE. N = 56 An average of 59% of the residents responding reported having no discussions with rotation faculty on the topics listed in Table 4. For all other discussion—taught competencies the average for no discussions was 2l%. Competencies learned through observation. in addition to discussion, the rotation directors indicated that a number of competencies would be addressed by residents through observation. These observations were to be of 20 medical problems of interest. Residents were to observe the diagnosis and treatment of the medical problems as components of several competencies to be acquired through observation during the rotation. As with competencies to be learned primarily through discussion, the researcher sought discrepancies between the intended and implemented curriculum. Discrepancies between competencies that were intended to be learned through observation and which 50% or more of the residents reported never having observed were noted. Half or more than half of the residents reported three competencies were not observed during the rotation. They are presented in Table 5. Seventeen competencies were observed by residents during the rotation and are not reported in the table. For the competencies listed in Table 5, an average of 65% of the residents reported never observing the medical problem. For the 17 other medical problems, just [3% of the residents reported having no observations. Data from both former and current residents were tabulated and summarized, and are presented in Appendix J. An average of 65% of the residents responding reported having no observations of the medical problems listed in Table 5. For all other observation- taught competencies the average for no reported observations was just l3%. Three problems narrowly missed the 50% cutoff score. Twenty-seven residents (48%) reported observing no cases of myositis ossificans (questionnaire item 32c). Twenty-five residents (45%) reported observing no cases of disturbances of special senses (questionnaire item 32t). Finally, twenty-two residents (39%) reported 32g 32;. 32f obsl actl lear disc inte con Pra< disc (Orr SUm 114 Table 5 Discrepancies in Implementation of Observation Competencies uestionnaire Item Number of Observations ~Q- 1:2 6 or more 2g. How many times during the 35 2 rotation did you observe (63%) (314%) 04%) puncture wounds? Zj. How many times during the rotation did you observe 28 25 2 metabolic disturbances? (51%) (45%) (4%) 2t. How many times during the rotation did you observe disturbances of special 45 10 1 senses? (80%) (18%) (2%) NOTE. Percentages may not sum to 1.0 because of rounding. NOTE. N = 56 >serving no cases of disturbances of consciousness (questionnaire item 32r). Competencies learned throungerformance. The rotation directors also ‘tended that 51% activities be performed by residents during the rotation. Some of e activities to be performed were components of competencies to be acquired. her activities constituted complete competencies. Regardless of whether the tivities were competencies or components of competencies, they were to be irned through practice. As with the competencies to be acquired through cussion and observation, the researcher sought discrepancies between the ended and implemented curriculum. The cutoff score for performance 1petencies was 50%. If 50% or more of the residents reported that they never cticed the competency during the rotation, the researcher counts this a :repancy. Discrepancies were not reported about 37 competencies. Seventeen ipetencies, however, were reported discrepant and are presented in Table 6. mary data from both former and current residents were tabulated and 'narized, and are presented in Appendix K. guestit 31e. ing. 310. ilu 115 Table 6 Discrepancies in Mementation of Performance Competencies Questionnaire Item Number of Performances -Q— 1~_5 6 or more 31e. How many times during the rotation did you transport or supervise the transportation 38 16 2 of an injured athlete? (68%) (29%) (4%) 3lf.1. How many times during the rotation did you develop treatment plans for high 29 19 7 school athletes? (52%) (35%) (13%) 31g.3. How many times during the rotation did you prescribe 31 20 5 corticosteroids? (55%) (36%) (9%) 310. In caring for an athlete, how many times during the rotation did you contact 1. parents 40 14 2 (71%) (25%) (4%) 2. coaches 32 20 4 (57%) (36%) (7%) 3. officials 49 7 ~0- (89%) (13%) (0.0%) 31t. How many times during the rotation did you prescribe exercise for these athlete groups? 3. physically impaired athletes 46 8 -0— (85%) (15%) (0.0%) 4. athletes with chronic illnesses 30 23 3 (54%) (41%) (5%) 5. prepubescent athletes 40 15 1 (71%) (27%) (2%) 7. geriatric athletes 48 7 -O- (87%) (13%) (0.0%) 31u. How many times during the rotation did you prescribe the following treatment modalities? 1. cybex 27 23 4 (50%) (43%) (7%) 2. TNS units 27 23 3 (51%) (43%) (6%) 3. orthotron 45 8 l (83%) (15%) (2%) 4. cold pressure boot 37 14 1 (71%) (27%) (2%) 5. paraffin baths 35 17 2 (65%) (31%) (4%) 9. EMS units 39 12 l (75%) (23%) (2%) 13. orthoplast splints 28 24 2 NOTE. Percentages may not sum to (52%) (44%) (4%) 1.0 because of rounding. NOTE. N : 56 reshler compe lielev< of the port 0 object wheft suffic obtoi physi £3919 curri oddi poss ocqi desi COST CUi' que (ES 0: H6 For the l7 competencies listed in Table 6, an average of 67% of the residents reported never practicing the competency. The average for the 37 competencies not reported as discrepancies was l5%. Relevance of Curriculum and Objectives The first part of Evaluation Questions 8 and 9 dealt with the implementation of the curriculum and objectives of the rotation in sports medicine. The second part of Evaluation Questions 8 and 9 deals with the relevance of the curriculum and objectives. Relevance of the curriculum and objectives was assessed to determine whether the competencies being taught in the rotation were adequate and Sufficient to begin the practice of Sports medicine. The data used to answer the relevance portion of questions 8 and 9 were obtained solely from the external review panel: l7 surgeons and 14 primary care physicians. Using the SurveLof important Components of a Family Practice Rotation in Sports Medicine questionnaire, the panel rated the importance of curriculum and objectives (the minimal competencies) of the rotation and offered additional competencies they felt were necessary that a family practice physician possess in order to begin the practice of sports medicine. The panel rated the curriculum and objectives of the rotation as follows: I : acquiring the competency is unimportant, 2 : acquiring the competency is desirable, but not essential, and 3 = acquiring the competency is essential for beginning the practice of sports medicine. The mean ratings for all the competencies in a curricular area make up the rating for that area of the curriculum. Not all members of the external review panel responded to each questionnaire item. Rather than omit the cases in which all items were not responded to, only missing items within cases were dropped. Thus, not all cases in a given curricular area are based on the same number of items. One respondent, for e: curric of res the i medic curri< Curr Pail: Spc Basi of Pre1 Me 117 for example, inadvertently omitted about half the questionnaire items in the curricular area of patient care. Thus, these item means were based on the number of respondents who actually responded to the item. The data in Table 7 show that surgeons consistently rated somewhat lower the importance of the curricular areas that make up the rotation in sports medicine. Primary care physicians rated philosophy of sport as the most important curricular area. Surgeons rated preventive sports medicine highest. Table 7 Curriculum Mean Ratings Curricular Area P.C.-)'(. S.D. Surgeon Y S.D. Overall Y SD. n : 114 n = 17 n '-'- 31 Patient Care in Sports Medicine 2.63 .24 2.52 .20 2.57 .22 Basic Science of Sport 2.65 .22 2.57 .28 2.61 .26 Preventive Sports Medicine 2.61 .31 2.60 .29 2.61 .29 Research in Sports Medicine 2.30 .38 2.20 .34 2.25 .36 Philosophy of Sport 2.83 .31 2.52 .32 2.66 .35 Mean Y + Mean 5.1). 2.60 .30 2.48 .29 2.511 .29 w 1.0 The curricular area is unimportant = The curricular area is desirable, but not essential : The curricular area is essential P. . : Primary Care Physicians Surge inedk forSL mdyt rated remd GbOU' One pone spor dcqu conn Hsfi‘ secc in e phy: roh pon lde the db< 2i ll8 Surgeons rated philosophy of sport third (tie with patient care aspects of sports medicine). The mean for primary care physicians was nearly identical to the mean for surgeons on preventive sports medicine. Yet preventive sports medicine rated only fourth for primary care physicians. Primary care physicians and surgeons each rated research in sports medicine as the least important curricular area. The remaining four areas were rated quite closely in importance (mean range = .09). External review of objectives. Two sources were used to collect information about the objectives (competencies) to be attained by residents during the rotation. One source was the external review panel's ratings of individual competencies. The panel rated each of the minimal competencies (objectives) for the rotation in sports medicine as follows: 1 : acquiring the competency is unimportant, 2 : acquiring the competency is desirable, but not essential, and 3 = acquiring the competency is essential for beginning the practice of sports medicine. A complete listing of the ratings for individual competencies is presented in Appendix L. As a second source of data, the external review panel offered additional competencies in each of the curricular areas. The panel recommended that family practice physicians possess the competencies before beginning to practice sports medicine. in analyzing the data, the researcher sought discrepancies between what the rotation directors intended the residents to learn and what the external review panel considered unimportant. The researcher established a cutoff score of 2.00 (desirable, but not essential). Overall mean ratings of the importance of acquiring the competencies was calculated. Sixty-three of the competencies were rated above 2.00 and are not reported. Twelve competencies were rated at or below the 2.00 level by the panel, however, and are reported as discrepancies in Table 8. (Note. Objectives calculated at exactly the 2.00 level were included in the table since either primary care physicians or surgeons rated the objectives below the cutoff level.) l exterr compe 1185‘- 26. 38. 1ll. ‘17. l8. l9. 119 For the twelve competencies listed in Table 8, the mean rating by the external review panel was 1.92, or less than "desirable." The mean rating for competencies for which discrepancies were not found was 2.67. Table 8 Competencies Rated at or Below Desirable (2.00) Questionnaire Item P.C. X Surgeon? Overall 3(— n = 14 n = 17 n = 31 9. How important is identifying medical care considerations for special athlete groups (e.g., mentally retarded)? 1.92 2.06 2.00 17. How important is properly prescribing the following? b. TNS units 1.93 1.81 1.87 (1. cold pressure boot 2.07 1.73 1.90 e. paraffin baths 1.93 1.44 1.67 h. EMS units 2.14 1.87 2.00 k. medcolator with ultrasound 2.14 1.81 1.97 26. How important is identifying appropriate references for information about sports related problems? c. unpublished professional papers 1.83 1.94 1.89 38. How important is designing a community- based program of sports medicine care? 1.92 2.00 1.96 41. How important is properly applying the basket-weave ankle-strapping (taping technique)? 2.14 1.88 2.00 47. How important is designing a research study in sports medicine? 1.93 1.81 1.87 48. How important is conducting a research study in sports medicine? 1.93 1.93 1.93 49. How important is writing the results of a research study in sports medicine? 1.93 2.00 1.97 120 In addition to determining which competencies were held unimportant, the researcher attempted to determine which competencies external review panel members disagreed about. A cutoff point of .50 was established. Thus, if the ratings by primary care physicians and surgeons on a particular competency differed by .50 or more, a discrepancy about the competency was noted. Agreement E reached about seventy-one of the competencies (i.e., differences in the ratings of primary care physicians and surgeons was less than .50). Four competencies were rated above the cutoff level, however, and are reported in Table 9. Table 9 Discreflit Competencies Competency P.C. Surgeon Difference n : 14 n = 17 R x 21. How important is learning to counsel athletes regarding dietary fads? 2.86 2.19 .67 29. How important is learning to identify current dietary fads engaged in by athletes? 2.64 2.06 .58 46. How important is learning to determine strategies for reviewing the sports medicine literature? 2.64 2.13 .51 51. How important is learning to differentiate between the contrasting sports medicine philosophies of primary care and orthopedics? 2.83 2.17 .66 NOTE. P.C. : primary care physician In summary, both the external review panel and former residents rated the curricular areas highly. Of the five curricular areas, only research in Sports medicine was rated below the 2.50 level on the three-point scale used. The panel also rated individual competencies (objectives). Twelve of the competencies (16%) lZI taught in the rotation were rated at or below the "desirable, but not essential" level. Two areas of competencies were rated least important by the panel: learning to prescribe certain treatments properly, and designing, conducting, and writing up the results of a research study in sports medicine. Implementation of Additional Evaluation Variables Evaluation Question 10. Are instructional strategies evaluated for implementation and effectiveness? Evaluation data revealed that instructional strategies used in the rotation were evaluated for neither implementation nor effectiveness. The evaluation data revealed neither formal (written) evaluations designed for use by residents nor informal feedback sessions designed to elicit such information. Data used to answer Evaluation Question 10 came from two sources. The first source was rotation documents. Two documents provided useful information: the memo of October 6, 1978, from the Athletic Medicine Rotation and the daily schedule given to residents in the handout package. The second source of data was the questionnaire responses by former and current residents of the rotation. The researcher designed a questionnaire item to provide the required data. The questionnaire item was number 33 in the Participants Questionnaire. The item was designed to rate the implementation and the effectiveness of the fifteen strategies which had been identified. First, implementation of the instructional strategies will be presented. Then, the effectiveness of the strategies will be presented. Residents were asked to rate instructional strategies on a four—point scale. A cutoff point of 50% was established to determine implementation. Thus, if 50% or more of the residents indicated that they did not use the instructional strategy or that they could not recall using the strategy, the researcher judged that there was a discrepancy between the intended and implemented curriculum. Between former l22 and current residents no differences in response patterns were detected, and the responses of both groups are combined into single totals. Data from both former and current residents were tabulated and summarized, and are presented in Appendix M. Just two instructional strategies were judged discrepant. One strategy was sitevisits. Sitevisits are an instructional strategy by which residents acquire competencies at a location not directly associated with the rotation (e.g., the Youth Sports Institute, on area high school). The other strategy was active research or experimentation. This strategy consists of conducting an experimental research project during the course of the rotation. Forty-four of the 56 former and current residents (79%) reported not using orlnot recalling making site visits. Thirty-six of the 54 former and current residents who responded to the item (67%) reported not performing active research or experimentations. The discrepancy rate between intended and actual implementation as reported by residents for the thirteen instructional strategies that did not meet the 50% criterion was I8%. Two of these thirteen instructional strategies, however, came close to the 50% criterion: Twenty~six of the 56 former and current residents (46%) reported never having taken the pretest, and twenty-six of the 53 former and current residents (49%) reported never having taken the posttest. After assessing the instructional strategies used in the rotation for implementation, the researcher assessed the strategies for effectiveness. A cutoff score of 50% was established: if 50% or more of the residents reported an instructional strategy to be excellent, the researcher counted the strategy most effective. As before, when differences in response patterns between former and current residents could not be detected, the data were combined and are presented as single totals. fif- l23 Two strategies met the 50% criterion for excellence. The first strategy rated excellent was the informal clinic. Residents observe, discuss, and participate in the treatment of varsity athletes on a walk-in basis using the informal clinic strategy. Thirty—five residents (63%) rated the informal clinic as an excellent instructional strategy. The second strategy which met the criterion was interaction with trainers. Forty-two residents (75%) rated interacting with trainers as an excellent instructional strategy. The two strategies are interrelated to a degree, since the informal clinic provides the setting for much of the interaction between residents and trainers. An average of just 30% of the residents rated the remaining thirteen instructional strategies as excellent. Three of these thirteen instructional strategies that did not meet the 50% criterion were rated close to it. The pretest, for example, was rated by fourteen residents (47%) as an excellent strategy. Twenty-five residents (49%) rated the formal (referral) clinic as an excellent strategy. Finally, twenty-six residents (47%) rated event coverage as excellent. in Evaluation Question ID, two instructional strategies were identified as intended but not fully implemented. Those strategies were sitevisits and active research or experimentations. In addition, two instructional strategies were identified as especially effective. These strategies were the informal clinic and interaction with trainers. Several strategies came close to, but did not meet, the criterion levels for implementation and effectiveness. Evaluation Question ll. Are residents evaluated for prerequisites, knowledge, skills and attitude? The data collected during the evaluation indicate that residents are not evaluated for prerequisites, knowledge, skills or attitude during the rotation. One measure, a pre- posttest is administered to many residents. The pre- posttest does not accurately reflect the curriculum of the rotation, however. in addition, the results of the pre— posttest are not used in any way. No systematic, informal feedback sessions are scheduled to determine residents prerequisites, knowledge, skills and attitude. Some observation and supervision of residents is conducted by faculty and staff members during certain activities (e.g., formal clinic). Systematic evaluation is not designed into rotation activities, however. Residents are required to complete a short research paper by the conclusion of the rotation. The research papers are not systematically reviewed or evaluated. Evaluation Question II will be answered in two parts. ln the first part, resident prerequisites will be considered, and in the second part, resident knowledge, skills and attitude. Because there was insufficient data about residents prerequisites, and because of the importance of such evaluation information to the approach used to evaluate residency rotations, the researcher developed a questionnaire item about residents prerequisites. The questionnaire item is Participants Questionnaire item 29. The questionnaire item was designed to determine how much study of prerequisite skills and knowledge had been undertaken by residents before they participated in the rotation in sports medicine. The residents were asked how many terms in college or medical school they had studied each prerequisite area. in analyzing the data, the researcher sought discrepancies between prerequisites the rotation directors intended residents to possessland those prerequisites the residents actually did possess. A cutoff point of 50% was established. Thus, if 50% or more of the residents reported that they had never studied a prerequisite area, it was counted a discrepancy. (Data from both former and current residents were tabulated and summarized. As before, no differences in response patterns were detected between former and current residents, and the data are combined and ____ . _ 9 , ,u 125 presented as single totals in Appendix N.) Only one such discrepancy was found. Forty-one residents (75%) reported that they never studied how to interpret laboratory tests for an exercising person (Participants Questionnaire ltem 290). Two other discrepancies failed to meet the 50% criterion, but were reported frequently by residents. Twenty-four residents (44%) reported never having studied how to critique research literature (item 29N). Twenty-five residents (45%) reported never having studied physical/rehabilitation medicine (item 29R). Related research competencies had also been among the least important as rated by the external review panel (Table 8). In the second part of Evaluation Question ll, resident evaluation for knowledge, skills, and attitude will be addressed. The major rotation document examined was the Sports Medicine Pre— Posttest. The pre— and posttest consists of 62 objectively answered (multiple choice) questions. The identical form is used for the pretest and the posttest. The researcher conducted a content validity assessment of the pre- posttest. The results are presented in Table l0. The curricular areas of clinical science and preventive sports medicine account for 85% of the items included in the pre- and posttest. Two curricular areas, philosophy of sports medicine and research in sports medicine, are ignored. The pre- and posttest attempts to assess only residents' cognitive knowledge. Skills and attitudes to be acquired by residents are not assessed in the test. Finally, the information tested for comes solely from assigned readings. The researcher determined that the pre- and posttest was not content valid for evaluating the knowledge, skills, and attitudes of residents in the rotation. Additional data were required. Because there were insufficient data about residents' knowledge, skills and attitude, and because of the importance of such evaluation information to the approach used to evaluate residency rotations, the researcher developed questionnaire items to assess these areas. 126 Table 10 Content of the Pre- Posttest Number of items Philosophy of Sports Medicine —0— (0.0%) Basic Science of Sports 9 (15%) Preventive Sports Medicine 25 (40%) Clinical Science 28 (45%) Research in Sports Medicine -0- (0.0%) Because assessing specific competencies possessed by former residents was not possible, the researcher designed questionnaire items that would more generally indicate residents' knowledge, skills and attitudes about sports medicine. These more general indicators are presented in Table ll. Most former residents (92%) reported that they continue to study sports medicine (Participants Questionnaire item 7). Far fewer residents report learning by formal means such as classes. Most former residents continue to learn sports medicine through such informal means as reading sports medicine journals and books, from peers, and in their own practices (item 6). Thirty—seven former residents (77%) reported that they currently practice sports medicine (item l4). Yet only seven former residents (l9%) indicated that they served as team physician for any group of athletes (item 18). Only twelve residents (27%) indicated partici— pating on a community level in sports medicine (item 9). Community participation often includes such activities as performing preparticipation physical exams, teaching sports medicine techniques to allied health professionals, and participating in coaching seminars. A related questionnaire item (item 27), not reported in Table II, attempted to determine whether residents participated in local sports medicine networks. Fifteen residents (43%) indicated that there were 127 Table 11 KnowledgELSkill, and Attitudes of Former Residents . . Questionnaire Item I_e_s _N_o Meiji—13% 6. Has your study included formal training (e.g., classes, CME, internships)? 9 39 -0- (19%) (81%) 7. Have you continued to study sports medicine since your participation in 44 4 -0- the rotation? (92%) (8%) 9. In the last year, have you participated on a community level in sports medicine (e.g., 12 33 3 perform pre—participation physicals)? (27%) (73%) 12. Have you published an article that dealt 3 45 -0— with sports medicine? (6%) (94%) 13. Are you currently involved in any 1 47 -0- sports medicine research? (2%) (98%) 14. Do you currently include sports medicine 37 11 -0- in your practice? (77%) (23%) 18. Do you serve as team physician for any 7 30 11 group? (19%) (81%) NOTE. N = 48 such networks in their communities. Twenty residents reported that they did not know whether there were such networks or did not respond to the question. Just four residents (27%) indicated that they were an active part of the networks. Eleven residents (73%) responded that they did not participate in the networks. Thirty-three of forty-eight former residents did not respond to the question. Finally, former residents reported that they rarely engaged in sports medicine research (item 13) or published articles in the area of sports medicine (item 12). Former and current residents were also assessed for their attitudes specifically about the rotation, by their comments to an open—ended questionnaire item (item 67). Former and current residents were enthusiastic about and had positive attitudes about the rotation. 128 Evaluation Question ll attempted to identify missing resident prerequisites and discrepancies in the evaluation of residents' knowledge, skills, and attitudes. Just one resident prerequisite was found to be systematically lacking. Two other prerequisites failed to meet the 50% criterion for discrepancies but were frequently reported (44% and 45%). Evaluation Question 12. Are faculty and staff of the rotation evaluated for qualifications, availability, and teaching skills? The information obtained during the evaluation indicates that no evaluations of faculty or staff of the rotation are conducted to determine qualifications, availability or teaching skills. The data collected revealed neither formal instruments by which residents could provide feedback to the faculty and staff, nor informal means for obtaining such information. Data used to answer Evaluation Question 12 came from responses of former and current residents to written questionnaire items. A discussion of the qualifications of the various faculty is presented in Evaluation Question 2, and assessments of the faculty‘s qualifications for teaching in the rotation in sports medicine. In the present section, ratings of the qualifications, availability and teaching skills of the faculty will be presented. Because of the importance of faculty and staff evaluation data to the approach used to evaluate residency rotations, the researcher designed questionnaire items to provide information about the sufficiency of the backgrounds of the faculty and staff, whether faculty were up—to-date in their fields, and whether the faculty were prepared and interested in teaching. Additional information was supplied from an open—ended question. As before, when differences in response patterns were not detected, the results of former and current residents were combined and presented as single totals. The data are presented in Table 12. 129 Former and current residents strongly agreed in their assessments of the rotation faculty. The residents reported that the background of the faculty were sufficient for them to learn sports medicine, that the faculty were up-to—date, seem prepared, and were interested in teaching. Responses to open—ended question 67 indicated that some residents felt that faculty members should be more generally available (e.g., "A schedule of Dr. McKeag‘s and Dr. Hough's free time would be valuable so I could talk to them, or schedule free time regularly." "I would have liked 1-2 hours sometime in the rotation set aside to answer specific questions about sports medicine and my future plans." Finally, 25 residents (54%) reported that faculty members such as the rotation directors and trainers contributed most to their learning during the rotation (question 49, not reported in Table 12). Table 12 Residents' Assessments of Rotation Facultj Number Missing Questionnaire Item leg 1&3 Undecided Cases 58. Were the backgrounds of the faculty sufficient for you 48 1 5 2 to learn sports medicine? (89%) (2%) (9%) 59. Were the faculty up-to—date 50 —0- 4 2 in their fields? (93%) (0.0%) (7%) 60. On a daily basis, did the 50 -0— 4 2 faculty seem prepared? (93%) (0.0%) (7%) 61. Did the faculty seem 47 2 4 3 interested in teaching you? (89%) (4%) (8%) NOTE. Percentages may not sum to 1.0 because of rounding. NOTE. N = 56 l30 ln Evaluation Question 12, five questionnaire items provided the basis for information about the qualifications, availability, and teaching skills of faculty and staff of the rotation in sports medicine. The data from both former and current residents showed strong support for the qualifications and teaching skills of the faculty and staff. Several residents commented, however, that the rotation directors should have been more generally available. Discussion of Findings (Primary Evaluation) Several findings in the primary evaluation merit additional discussion. The discussion is limited to unexpected or unusual findings. Some objectives were not attained by residents during the rotation because of a lack of opportunity to observe, discuss, and/or treat athletes with certain medical problems (e.g., physically impaired athletes). Other objectives were not attained because certain types of athletes were rarely or never seen in the clinics (e.g., pre—pubescent, geriatric, and elite athletes). These objectives were important, since the specific skills are not taught elsewhere in residency training. An unexpected finding was the curricular ratings of the rotation by members of the external review panel. Both primary care physicians and surgeons indicated considerable consensus in their responses about the curriculum and most of the individual objectives. Surgeons rated prevention as the most important curricular area, perhaps because they see the often irreversible results of severe sports injuries. Primary care physicians rated philosophy of sport highest. A major portion of primary care philosophy is the prevention of sports related injuries. Both primary care physicians and surgeons agreed in their assessments of the importance to the beginning physician of designing, conducting and reporting the results of sports medicine research. The panel agreed these skills were l3l unimportant. Residents also lack the prerequisites required to meet these objectives. Residents rated differently two similar instructional strategies used during the rotation. The strategies were onsite coverage of team practices and of events. Covering events was rated as a much more effective strategy for learning sports medicine. Perhaps the excitement of attending actual events caused some residents to remember particular injuries. Finally, the data revealed that residents continue to add to their sports medicine knowledge and skills when they have completed the rotation. They do so, however, not to promote sports medicine on a community basis, as was the intent of the rotation directors, but to add sports medicine to their private practices. Summary of the Evaluation Meeting A summary report of the evaluation of the rotation in sports medicine was prepared and delivered to the rotation directors in January 1985. A meeting to discuss the researcher‘s findings was scheduled about a week afterward. The intent of the meeting was to present and discuss with the rotation directors the results of the primary evaluation. One of the two rotation directors could not come to the meeting. During the meeting, the researcher presented a brief background of the study, reviewed procedures and methods used, and presented the results. Finally, the researcher presented recommendations based on the results of the 12 evaluation questions. Because of the length of the evaluation report, the presentation took longer to complete than originally planned. The researcher was able to present the results of all twelve evaluation questions, however, and to present all of the major recommendations. ‘ . 132 Results of the Metaevaluation The results of the primary evaluation of the rotation in sports medicine were reported in January 1985. After the results of the evaluation were reported, a metaevaluation was conducted to summatively determine the merit of the prototype approach. The metaevaluation consisted of collecting data to answer the four research questions presented in Chapter 1. The four research (metaevaluation) questions are: I. Was the approach useful in providing information to the rotation directors’? 2. Was the approach practical? 3. Was the approach ethical? 4. Did the approach collect accurate information about the rotation? The metaevaluation consisted of four steps. The first step was identifying evaluation standards that related to the four research (metaevaluation) questions. One or more evaluation standards were identified from Standards for Evaluations of Educational Programs, Projects and Materials (The Joint Committee, l98l). In the second step, specific metaevaluation questions were developed, using the standards as guidelines. Two sets of specific metaevaluation questions were developed: one set for the rotation directors of the rotation in sports medicine, and the other set for a metaevaluation consultant. The questions for both the rotation directors and an external metaevaluation consultant were assembled in the form of structured interview guides. Many of the questions which made up the guides were borrowed from Standards for Evaluations of Educational Programs, Projects and Materials (The Joint Committee, l98l). Other questions were based on the design criteria presented in Chapter Three, and on additional specific information requirements identified by the researcher. The additional questions solicited information about the general effectiveness of the approach for use in evaluating l33 residency rotations. The rotation directors' and metaevaluation consultant's responses in the interviews provided the basis for the answers to the research questions. The questions for the rotation directors and the metaevaluation consultant are presented in Appendices G and H. In the third step, the researcher obtained the metaevaluation data from the rotation director and then from the metaevaluation consultant. First, the researcher sent the rotation directors copies of the evaluation report. A week later, the researcher met with one rotation director to discuss the findings of the evaluation and the researcher's recommendations for improving the rotation. A week after the evaluation meeting, the researcher met with the rotation director again. At this meeting, the researcher used the interview guide developed for the rotation directors to guide a question—and-answer session. The interview was recorded on audiotape and transcribed into hard copy. The researcher delivered a number of materials to the consultant, including a copy of the written evaluation report. These were intended to familiarize the consultant with the design of the prototype evaluation approach. The consultant reviewed the materials for about l0 days. The researcher and the consultant then met to discuss the design of the evaluation approach. The researcher explained ambiguities in the design and answered the consultant's questions. At a second meeting, the researcher used the interview guide developed for the consultant to guide a question-and-answer session. The interview was recorded on audiotape and transcribed into hard copy. In the fourth and final metaevaluation step, the researcher analyzed the metaevaluation data in several stages: first, clustering the responses to similar interview questions from the two interviews; next synthesizing the data of the clustered questions; finally, generating answers to the four research (metaevaluation) questions, the questions about the design criteria of the approach, I34 and the additional general questions. In generating the answers, the researcher evaluated the transcribed data in conjunction with his personal appraisals of the data. The personal appraisals of the data were based on his knowledge of the prototype evaluation approach and the problems and successes encountered in implementing the approach. A summary of the results of the metaevaluation is presented next. Summary of Metaevaluation Results Metaevaluation data are presented for each of three areas. The three areas in which metaevaluation data were collected were the areas of the four research (metaevaluation) questions, the design criteria, and a general area of metaevaluation data. Research (Metaevaluation) Questions Research Question 1: Was the approach useful in providing information to the rotation directors? An analysis of the metaevaluation data revealed that complete, comprehensive, and relevant information was collected about the rotation. in addition, the procedures used to interpret the information collected were sufficiently described so that the bases for value judgments made by the researcher could be understood by the rotation directors. A great deal of metaevaluation information was collected about the primary evaluation report. These data indicated that the primary evaluation report contained sufficient detail for the rotation directors to understand the information collected, the conclusions reached by the researcher, and the recommendations the researcher made. ln addition to being understandable to the rotation directors, both the oral presentation and written report were delivered soon enough for the information to be useful to the rotation directors. 135 The data clearly indicate that both the rotation director and the metaevaluation consultant felt that the researcher was both trustworthy and competent to perform the evaluation. As a result, the evaluation findings were credible to the rotation directors and were accepted by them. Finally, the rotation directors indicated that the primary evaluation results and recommendations would be of much help to them in planning future rotations and another closely related program (a sports medicine fellowship). indeed, the rotation directors immediately began to implement into the rotation some of the recommendations listed in the primary evaluation report. On the basis of the analysis of these metaevaluation data, the researcher concluded that the evaluation approach was useful in providing information to the rotation directors. Research Question 2: Was the approach practical? An analysis of the metaevaluation data collected indicates that both the rotation director and the consultant concurred on the practicality of the approach. Both felt that the presence of the researcher during the activities of the rotation occasionally disrupted the flow of activities or affected the interpersonal dynamics between various participants of the rotation (e.g., faculty and residents). The rotation director stated, however, that no change in service to patients resulted, and, at most, the presence of the researcher occasionally slowed the pace of the activities. Both the rotation director and the consultant similarly agreed that the funds invested in the evaluation were worth the information received. The rotation director specifically commented that the comprehensive and complete evaluation information received was worth the money invested. The consultant noted that, with one exception, practical procedures were employed in the evaluation. The exception noted was that the evaluator spent a full month observing the activities of the rotation and interviewing faculty, staff, and residents. The consultant I36 commented that most rotation directors could not afford to hire an evaluator for an entire month of observation and interviewing. The researcher concurred with the consultant on this point. The researcher felt that observation of the rotation during the field test of the approach was necessary to collect the required data. The researcher carried out this strategy while meeting the requirements of a doctoral dissertation. Subsequent evaluations conducted by internal evaluators, such as rotation directors, would obviate the need for employing such a labor intensive strategy to the same degree, because the evaluator would already be present during rotation activities. Thus, the field test as conducted was worth the investment. In addition, by already being familiar with many rotation activities and personnel, subsequent evaluations conducted by internal evaluators would also be cost-effective. The approach may not be worth the money invested or practical, however, if subsequent evaluations were conducted by external evaluators who deemed it necessary to observe on a full- time basis. The researcher feels, however, that full-time observation is probably not required. Instead, necessary data can be obtained through a combination of part-time observation and structured interviews with faculty, staff and residents. On the basis of the analysis of the metaevaluation data collected, the researcher concluded that the evaluation approach during the field test was practical. Disruption was kept to a minimum and information of sufficient value was produced to justify the resources expended. The practicality of subsequent evaluations depends on the relationship of the evaluator to the rotation. Research Question 3: Was the approach ethical? An analysis of the metaevaluation data collected to answer this research question indicates that the rotation director and the metaevaluation consultant agreed in their assessments of the ethicality of the evaluation approach. These assessments were based on procedures implemented by the researcher during the l37 conduct of the evaluation and by criteria regarding the oral and written evaluation reports. Both the rotation director and the metaevaluation consultant agreed that the researcher had instituted and upheld procedures for guaranteeing anonymity to the sources of data. The metaevaluation data also indicated that the design of the evaluation approach was adequate in providing for the rights and welfare of other participants in the rotation (e.g., the patients). Both the rotation director and the consultant indicated that the written and oral evaluation reports were well balanced in reporting strengths and weaknesses of the rotation. Both the rotation director and the consultant responded that the written evaluation report was open, direct, and honest in reporting the evaluation findings. The consultant did not comment about the oral presentation of the report because he did not attend the meeting. The rotation director's comments about the presentation of the findings which were delivered orally at the meeting were similar to his comments about the written report. Finally, both the rotation director and the metaevaluation consultant agreed that conclusions presented in the evaluation report were based on the data collected in the study. On the basis of the analysis of the metaevaluation information collected, the researcher concluded that the evaluation approach used in the primary evaluation was ethical. The evaluation results were not compromised by conflict of interest and the oral and written evaluation reports were open, direct, and honest. in addition, the design of the evaluation approach respected and protected the rights of human subjects. Finally, the written and oral evaluation reports were complete and fair in reporting strengths and weaknesses of the rotation. I38 Research Question 11: Did the approach collect accurate information about the rotation? As with the preceding research questions, the analysis of the metaevaluation data collected to answer Research Question 4 indicate that the rotation director and metaevaluation consultant agreed in their assessments regarding the accuracy of the information collected about the rotation. The preponderance of metaevaluation data gathered to answer this research question was supplied by the metaevaluation consultant, however. The analysis of the metaevaluation data indicates that, using the evaluation approach, the researcher described information sources in sufficient detail to determine the adequacy of the information collected and the validity of the information produced. Similarly, the metaevaluation consultant indicated that data collection instruments and procedures were described in sufficient detail to determine the validity and the reliability of the data. The metaevaluation consultant noted, however, that reliability of the instruments was a lesser issue since it is unlikely the instruments would be used again without modification. The metaevaluation consultant also reported that, with the exception of observation data, sufficient evidence was provided that both qualitative and quantitative data were collected, analyzed, and reported systematically. Observation data were not systematically recorded by the researcher. The analysis of the metaevaluation data indicate that the context in which the rotation functions was sufficiently examined to determine an appropriate evaluation approach. Similarly, the analysis of the data indicate that an accurate description of the rotation was produced. The analysis of the metaevaluation data indicate that the conclusions reached by the researcher were supported by the data, and that recommendations for improving the rotation were based on the results of the evaluation questions. I39 Finally, the analysis of the metaevaluation data indicate that the procedures used in the evaluation approach protected the results and the evaluation report from being purposely distorted. Thus, on the basis of the analysis of the metaevaluation data collected, the researcher concluded that the evaluation approach was successful in collecting accurate information about the rotation. Design Criteria Additional metaevaluation data was collected about the applicability of the evaluation approach for residency rotations. This additional metaevaluation data focused on the design of the approach. Both the rotation director and the metaevaluation consultant contributed to the design of the approach. The researcher contributed additional metaevaluation data in the area of the design of the evaluation approach. Metaevaluation data were previously generated which addressed the design criteria of the ease of implementing the approach, low cost and time, the discrepancy concept, the required methodology, and the responsibility for rotation improvement. These results will not be further discussed. One design consideration was that the evaluation approach could be subsequently implemented by the rotation directors. The metaevaluation consultant reported that sufficient information about methodology was provided for subsequent implementations, but that future implementation would depend greatly on the rotation directors' committing the necessary time to the task. The metaevaluation consultant gave as an example of time commitment, the time that would be required to modify the existing data—collection instruments so that the instruments would be useful in subsequent iterations. The rotation director was also optimistic about the possibility of his performing future evaluations. He commented that he understood the evaluation approach and many of the procedures used during the evaluation. l__—k—_‘ MO The evaluation approach reflected the specific information requirements of the rotation in sports medicine, another design criterion. First, the l2 evaluation questions are designed to address the general information needs of any rotation. Second, the questionnaire items for all three questionnaires were developed with the assistance of the rotation directors and from existing rotation documents. Utilizing the input of the rotation directors and the documents helped to ensure that the specific informational requirements were met. That the recommendations made by the researcher are being addressed by the rotation directors further attests to the degree to which useful information was acquired. Although the design criterion of meeting the informational requirements of the rotation was achieved, a major problem in collecting the proper data was revealed. Data collection was implemented without developing a table of test specifications. Test specifications were not developed because the focus of the evaluation was not clear in the early stages of the evaluation. A table of test specifications would have matched specific questionnaire items with the evaluation question for which the items were intended to supply data. Several problems resulted. One problem that resulted from not having developed a table of test specifications was that a large amount of extraneous data were collected. These data did not answer the l2 evaluation questions. Another problem was that the opportunity to obtain additional data was lost because guidelines for obtaining observational data were not developed. A final problem resulting from the absence of a table of test specifications was that additional questionnaire items were not developed. These additional questionnaire items could have been used to supply fuller data for answering the l2 evaluation questions. On the basis of the analysis of the metaevaluation data generated, the researcher concluded that the design criteria had been addressed in the approach. l4l _Gglgral Metaevaluation Data One final set of metaevaluation data were collected: information not directly related to the research (metaevaluation) questions or the design criteria, but indicating the general effectiveness of the approach for use in evaluating residency rotations. Both the rotation director and the metaevaluation consultant supplied these general metaevaluation data. The metaevaluation consultant reported that the design of the approach should have included cognitive and performance assessments of the residents. The researcher did not include such assessments since provision for these assessments had been made by the rotation directors (the pre- posttest) and because the object of the evaluation in the study was the program-mot students. The pre- posttest did not adequately assess the achievement of competencies by residents during the rotation, however. The researcher reported the inadequacy of the pre- posttest in the evaluation report and described the lack of appropriate systems of evaluation in the rotation. A recommendation by the researcher to the rotation directors was that appropriate systems of evaluation be developed. The metaevaluation consultant correctly pointed out the large amount of extraneous data collected by the researcher. The extraneous data reflected an early data-collection strategy. Before the study became more tightly focused, the data-collection strategy had been to obtain a wide range of data. Some data were to be used in the current study, and other data stored for possible use in future studies. This strategy resulted in the collection of extraneous data and in several other problems. These problems were presented in the design criteria section. Modifications of the data—collection instruments and tables of test specifications should permit the acquisition of pertinent data with reduced effort in subsequent evaluations. l42 The rotation director and the metaevaluation consultant agreed that a strength of the evaluation approach was the comprehensive and complete data collected. The metaevaluation consultant also commented that the information addressed specific characteristics of the rotation and that information was collected from a variety of sources. The rotation director noted that an additional strength of the evaluation approach was the unusually high response rate of former residents and external review panel members (96%). The metaevaluation consultant also revealed an important weakness in the evaluation approach. The evaluation approach included the [2 evaluation questions of importance to residency and rotation directors. The approach, however, failed directly to identify the procedures for developing the "second-level" questions (questionnaire items). These second-level items will vary from rotation to rotation. More explicit criteria for developing second-level questions should have been included in the design. A final general metaevaluation question dealt with the impact of the evaluation using the prototype approach. The rotation director said he had received valuable information about the rotation and intended to implement the recommendations made by the researcher. The weaknesses of the approach were such that they could be corrected or eliminated in subsequent evaluations using the approach. Thus, despite the problems and weaknesses discovered in the field test of the approach, the approach could be successfully implemented in the future. Summarj of Metaevaluation Results The results of the metaevaluation summatively demonstrated the merit of the prototype approach used in the primary evaluation. The rotation directors received useful, accurate and specific information about the rotation in sports medicine. The information was obtained at an acceptable cost to the rotation directors and without undue disruption to rotation participants or activities. Finally, the approach was ethical in its design. The results also disclosed several important problems and weaknesses. One problem was that the study did not become sharply focused until after many data had been collected. Another problem was that test specifications were not developed for matching questionnaire items with the evaluation questions they were intended to help answer. The results were that extraneous data were collected, some data—collection opportunities were missed, and potential questionnaire items were omitted. A weakness of the design is that unless subsequent evaluations are conducted by a rotation director, the time commitment required of the evaluator may render the approach impractical. Results of the metaevaluation which have implications beyond this study are discussed in Chapter Five. Summary of the Chapter Two sets of data were reported in this chapter. First, data from the primary evaluation were reported. These data were reported in clusters made up of questions about the context in which the rotation operates, questions pertaining to instructional components of the rotation, and questions about evaluation practices of the rotation. Data were presented in tables and by descriptive narrative. Results of the metaevaluation of the approach used to conduct the primary evaluation were also presented. Data for the metaevaluation were gathered primarily from two people--one of the two rotation directors, and a metaevaluation consultant. Both participated in metaevaluation structured interviews. Their responses provided information that answered the four research (metaevaluation) questions, the design criteria questions and additional general questions about the approach. The researcher provided additional metaevaluation data about the design of the approach. llili The last chapter of the dissertation contains a summary of the research. The results of the field test of the approach are discussed and conclusions are drawn. Recommendations for future medical evaluators are provided. Finally, recommendations for further research are presented. Chapter Five Summary and Conclusions Introduction In this chapter the research is summarized and the major issues related to the results of the study are discussed: problems encountered in the study, the effectiveness of the data collection strategies used, the utility of the data collected in each evaluation question cluster, and the relative difficulty of collecting the data, and the strengths and weaknesses of the prototype program evaluation approach. Conclusions are drawn, and finally, recommendations for future graduate medical education evaluators and for further research are made. Summary of the Research The study was undertaken to address the problems encountered by residency and rotation directors who wish to evaluate components of their graduate medical education programs. One major problem was revealed in the review of program evaluation models. No program evaluation model was identified that had the correct components or degree of specificity to make it relevant for use in evaluating rotations. The research reported in this dissertation is the result, then, of the attempt to design, develop, field test, and metaevaluate a prototype approach for evaluating residency rotation programs. Major components of the design of the prototype approach were synthesized from the literature on program evaluation or borrowed from existing models of program evaluation. The components were selected to address seven design criteria established by the researcher. These design criteria were adopted to attempt to develop an approach that would be useful to all residency rotation directors. M5 —_'—i7~ l46 The prototype program evaluation approach was field tested on one rotation, the rotation in sports medicine. Data were collected between November l982 and March l983, by a number of procedures. The data were analyzed, and written and oral evaluation reports were prepared and presented to the rotation directors. Finally, a metaevaluation was conducted to summatively judge the merit of the prototype program evaluation approach. Problems Encountered The first problem encountered had to do with the design of former and current residents questionnaires. No statement of specifications for matching questionnaire items to evaluation questions had been developed. Thus, the data collected were insufficient to answer some of the evaluation questions. The researcher also collected a large amount of unusable data. The data proved to be unusable as a result of two decisions: on early data—collection strategy to collect as many data as possible about the former and current residents, and a shift in emphasis of the evaluation questions. A great deal of time was wasted by respondents in completing the questionnaires, and by the researcher in collecting, coding, and analyzing data that were not used. The completion of the study was not threatened, since the researcher was able to answer all l2 evaluation questions. By devising a table of specifications in advance, however, the researcher would have collected more specific and targeted information, and would have saved a great deal of time for both himself and the respondents. A second problem was that systematically recorded and retrievable observation and interview data were not collected. The problem was the result of the lack of specificity of checklists. Two checklists were developed by the researcher to assist in observing the rotation and interviewing faculty, staff, and residents participating in the rotation. Because the data-collection strategy at the time of observation was to gather as much information as possible, the checklists I47 developed were of a general nature. As before, no table of specifications was developed. The result was that researcher observation was not effective. Specific data for answering the evaluation questions were not obtained. The researcher was able to answer evaluation questions on the basis of data supplied from other sources, however. Thus, while useful obserrvation and interview data were not collected, the evaluation was completed. Data Collection Strategies The researcher used seven strategies in collecting data for the primary evaluation: l) examination of rotation documents, 2) observation of the faculty, staff, residents, and activities of the rotation, 3) informal interviews and discussions with faculty, staff, and residents, 4) the Rotators (current residents) Questionnaire, 5) the Participants (former residents) Questionnaire, 6) the Survey of Important Components of a Family Practice Rotation in Sports Medicine (external review panel), and 7) review of faculty members' vitae or resumes. The seven data—collection strategies will be reviewed for the effectiveness with which each produced valuable data. The examination of rotation documents was especially effective in helping the researcher to establish the instructional intents of the rotation. Many of the rotation documents had been written in the early existence of the rotation. They showed the original instructional intents of the rotation directors. Some instructional intents had been modified or dropped from the rotation over time. Several new instructional intents had been added. The documents, however, had not been updated to reflect these changes. Thus, the documents served as a basic framework for exhibiting instructional intents. A second effective data-collection strategy was the questionnaires which were developed for former and current residents and which provided quantitative and qualitative data about the implementation of the rotation. The data obtained l48 from the two questionnaires also spanned the entire history of the rotation, from the first resident to complete the training until the most recent. Thus, data were obtained from nearly the entire population. Another data-collection strategy that proved to be especially effective was the questionnaire developed for the external review panel of opinion leaders in sports medicine, a national panel representing primary care physicians and orthopedic surgeons engaged in sports medicine on the collegiate and professional levels and in private practice. Data from this questionnaire clearly documented the validity of the curricular areas addressed in the rotation and of the individual objectives in each of the curricular areas. Because there are no national, agreed- upon standards, surveying opinion leaders to validate the curriculum and objectives was an especially useful strategy. One data-collection strategy provided information of less significant value than the other six, perhaps: the review of vitae and resumes. From them, the researcher attempted to determine the qualifications of the faculty and staff for teaching in the rotation. Making judgments here was highly subjective, and relied on criteria such as degrees earned and publications to infer qualifications. In the absence of more direct criteria, the strategy provided some data, but their validity is open to question. Components and tactics of two overall successful strategies ultimately proved to be superfluous or unproductive. An early strategy was to obtain a wide range of data for possible future use, and the questionnaires developed for former and current residents collected such data. These ultimately extraneous data slowed the progress of the evaluation and the completion of the dissertation. Finally, a tactic used by the researcher was to develop separate questionnaires for former and current residents--a tactic designed to detect shifts in the emphasis of the implementation of the rotation over time. When data were analyzed, however, no differences were detected. Evaluation Question Clusters Here the clusters will be examined for the utility of the data collected and the relative difficulty encountered in collecting them. Valuable information was obtained from the cluster of questions about rotation context (Evaluation Questions l through 3). The information generated from these questions is useful for purposes of internal decision—making. Decisions made with rotation context information would most often be in the areas of planning and structuring changes to the rotation, controlling ongoing activities of the rotation, establishing or changing administrative intents of the rotation, or in judging the merit of the rotation. Data collected to answer rotation context questions were qualitative and were obtained by the study of documents and review of vitae and resumes. The data were comparatively easy and inexpensive to collect and analyze. Valuable information was also generated in the cluster of questions about instructional variables (Evaluation Questions 4 through 7). The information obtained from this cluster is useful in addressing the accreditation issues of definition and implementation of rotation goals and objectives. The cluster provided data documenting the existence of formal rotation objectives, plans for implementing the curriculum, and intended instructional strategies. Information generated from this cluster of questions was also qualitative, and was obtained by examining rotation documents and discussions between the researcher and rotation faculty and staff. The data were comparatively easy and inexpensive to collect and analyze. Valuable data were also collected by the cluster of questions about evaluation variables (Evaluation Questions 8 through l2). The information generated by this cluster is useful in documenting implementation of the instructional intents, in externally validating the curriculum and objectives, in gaining accreditation for the I50 program, and in making improvements to the rotation. Data obtained from the evaluation variables cluster of questions were primarily quantitative, but were supplemented by qualitative data. The data were obtained by using all the strategies except review of vitae and resumes. These data were comparatively difficult and expensive to collect and analyze. The researcher spent a great deal of time in the development of the three questionnaires. The questionnaires were expensive to print and mail. Follow—up procedures cost additional time and money. Analysis of the data required coding and the use of a computer. The difficulties and expenses were justified, however, by the utility of the data collected. The utility of the data generated from the evaluation variables cluster of questions is threefold. First, the data may be used to make improvements to the rotation. Second, a portion of the data generated in the evaluation variables cluster externally validates the general curriculum and the specific objectives. Finally, data generated in this cluster help to satisfy accreditation standards regarding the review of instructional plans for graduate medical education programs and the development of evaluations of both the program and the faculty. In summary, valuable data were collected from each of the three clusters of questions. Because of the strategies used, data from two of the clusters-«rotation context variables and instructional variables——were collected and analyzed with comparatively little difficulty or expenditure of funds. By comparison with the other clusters, data collected from the cluster of questions about evaluation variables were difficult and expensive to collect and analyze. The rotation directors felt that the difficulty and expense were justified, however, by the utility of these data. §tr_equths and Weaknesses of the Approach Some of the strengths and weaknesses of the approach are inherent in its design. Others were manifested during the field test of the approach. Several of the weaknesses which were manifested in the field test of the approach, however, may be overcome during subsequent implementation. First, the strengths: l. The prototype program evaluation approach uses specific questions that address the informational requirements of rotations. The evaluation models and approaches reviewed by the researcher concentrated only on the process of evaluation. The prototype approach developed by the researcher, however, consists of two components. One component is the process by which the evaluation is conducted. The other component consists of l2 evaluation questions of specific interest to rotation directors. Thus, a strength of the prototype approach is that it provides both the mechanics (i.e., process) and the tangibles (i.e., specific questions) to which the mechanics are applied. 2. The prototype program evaluation approach requires that specific rotation goals and objectives be developed. Developing specific goals and objectives for the rotation is a strength of the approach because the focus of evaluation becomes the educational outcomes of the rotation. When specific goals and objectives have been developed, the focus of the evaluation approach becomes the discrepancies between goals and objectives and the implementation of the rotation. What modifications the rotation requires are thus determined. Requiring specific rotation goals and objectives permits the evaluation of instruction in the rotation and guides residents in their learning. There is one final strength from using the objectives—based approach. As stated earlier, on objectives-based approach focuses on discrepancies between instructional intents and rotation implementation. Discrepancies are revealed and modifications are made in the intents or to the implementation. Using the concepts of instructional intents, rotation implementation, and determining discrepancies between the two involves program evaluation as an integral component of rotation development. Thus, the ongoing development of the I52 rotation is based on systematic data collection and analyses. 3. The prototype program evaluation (pproach collects qualitative md quantitative data from internal and external sources to provide oomprehensnve Information. There are three reasons that collecting both qualitative and quantitative data from sources both internal (e.g., residents) and external (e.g., the external review panel members) is a strength of the proposed approach. One reason is that comprehensive evaluations with multiple purposes (e.g., determining intents and implementation) often require both qualitative and quantitative data. The data required to answer questions in the cluster about rotation context and instructional variables were most efficiently obtained by means of a qualitative approach. Implementation and relevance data in this research were most efficiently obtained by means of quantitative methods. A second reason that collecting qualitative and quantitative data is a strength is that the two types of data complement each other. When both qualitative and quantitative data are collected, the analyses can provide insights that neither source alone could provide. A third reason that collecting both qualitative and quantitative data is a strength is the correction of biases. Qualitative and quantitative methods often have different biases. Using both qualitative and quantitative data helps to correct for the biases present in each method. 4. Responsibility for rotation improvement is clearly defined in the prototype program evaluation approach. Some evaluation models require the expertise of large evaluation staffs that supply evaluation information to decision makers. In other models, the evaluator supplies information that is to be used in decision-making. A problem sometimes arises in that program improvement is not clearly the responsibility of either the program directors (decision makers) or the evaluators. There is no such ambiguity lS3 in the prototype approach. Responsibility for improving the rotation clearly rests with the rotation director. The continuing development of the rotation should thus be facilitated. 5. After the initial evaluation has been conducted by an external evaluator, subsequent evaluations using the prototype program evaluation approach may be conducted by internal evaluators. The evaluation approach has been designed in such a way that evaluations subsequent to the initial evaluation performed by the researcher may be performed by internal personnel of a rotation (e.g., the rotation director). The two-part approach (evaluation questions and process for implementation) was discussed in both the written and oral evaluation reports delivered to the rotation directors. Copies of the various evaluation instruments and a discussion of procedures were included in the written evaluation report. The ability of an internal evaluator to do subsequent evaluations permits rotation directors to implement evaluation as an ongoing and integral component of developing the rotation. In addition, the prototype program evaluation approach is designed in such a way that the majority of the development is by the external evaluator during the first implementation. Once the development (e.g., designing questionnaires) is completed by the external evaluator, subsequent evaluations require comparatively minor developmental modifications. 6. The prototype program evaluation approach emphasizes formative evaluation, but contains a summative component. Overall, the prototype evaluation approach emphasizes formatively evaluating the rotation in order to revise and improve it. One component of the approach, however, provides data to validate the curriculum and objectives of the rotation. That component consists of data collected from members of the external review panel. Program validation is of a summative nature. The combination of formative and summative evaluation is a strength of the approach. The summative component is used to validate the content of the rotation. The formative l54 component is used to revise and improve the activities of the rotation by which the content is delivered. The implication for the rotation being evaluated is that the summative component need not be implemented with each evaluation. Once validated, the curriculum and objectives do not require frequent revision or improvement. The validity of the rotation curriculum and objectives should remain stable for some time. Thus, subsequent evaluations of a rotation would focus on revising and improving the activities (e.g., sitevisits, clinical experience). 7. Evaluations conducted using the prototype program evaluation approach produce comprehensive information about the rotation. Using the prototype approach, information is produced about the context in which the rotation operates, about instructional intents, and about implementation of the rotation. Information is provided about administrative aspects of the rotation, the discrepancies between intents and implementation, and the validity of the curriculum and the objectives of the rotation. The information produced may be used to improve the rotation, to validate the content of the rotation, or to gain accreditation for the residency program. The prototype approach may thus be implemented for a number of reasons. Now, the weaknesses of the approach: I. The initial implementation of the prototype program evaluation approach requires the services of a trained evaluator. The evaluator conducts the major developmental work necessary to implement the evaluation--for example, the production of the observation and discussion checklists and the questionnaires for various rotation participants. As discussed in #5 of the strengths of the approach, once the primary development has been conducted by the external evaluator, subsequent evaluations may be implemented by internal evaluators. But the initial evaluation requires the services of a trained, external evaluator. l55 2. The initial implementation of the prototype program evaluation approach requires a substantial investment of both time and money. The design of the prototype approach requires the investment of a great deal of the external evaluator's time. The external evaluator must be or must become familiar with the various components of the rotation to be evaluated. The external evaluator must develop and pilot test various data-collection instruments. The external evaluator must code and analyze data. The external evaluator further must spend time observing the participants and activities of the rotation and discussing the rotation with the directors, faculty, and residents. Finally, the external evaluator must prepare and deliver both oral and written evaluation reports to the rotation directors and other audiences of the evaluation. A concomitant weakness is the dollar cost of implementing the initial evaluation. The field test of the prototype program evaluation approach was conducted as a part of the researcher's doctoral research. So, the rotation directors were not required to compensate the researcher (external evaluator) for his time. In other circumstances, however, the cost of compensating the external evaluator may be prohibitive. Other expenses are incurred in the conduct of the prototype approach—— printing and mailing questionnaires. Follow—up procedures are a further expense. Coding and analyzing data take money. Secretarial time required to type or word— process data—collection instruments, letters, and the final report must be considered. All this may be within the means of many rotations. But coupled with paying for the external evaluator's services, the expenditure may be prohibitive for most rotation directors. 3. The prototype program evaluation approach requires the cooperation of faculty, staff, residents, and patients being seen during the rotation. The prototype program evaluation approach requires the collection of data from a number of respondents by means of several techniques (e.g., researcher I56 observation, questionnaire completion). Obviously, the various respondents' cooperation is needed. In the field test of the approach, completion of the current residents questionnaire was mandated by the rotation director. No such mandate was possible regarding former residents or members of the external review panel. Individual rotation faculty (e.g., head trainers, nurse manager of the formal clinic) control various aspects of the rotation evaluated during the field test. The researcher had to have their cooperation in allowing him access to their physical spaces and their vitae. He had to have the cooperation of the residents when he observed and discussed the rotation. Patients were given the option of including or excluding him from examination rooms. Without the cooperation of each of these individuals or groups, the researcher would not have had access to important evaluation data. Thus, a weakness of the approach is that the cooperation of a number of people must be granted if the data are to be obtained. 4. The prototype program evaluation approach provides inadequate methodology for establishing judgment criteria. Criteria for making judgments in several areas of the evaluation were not provided in the design of the approach. One example of the lack of criteria related to the I2 evaluation questions. Each of the 12 questions may essentially be answered yes or no. If the answer to any of the 12 questions is no, additional data are sought to provide the required information. The first seven evaluation questions seek to determine whether there are formal (written) documents about a number of variables of importance to the rotation. Whether there are such documents is relatively easy to determine. The final five evaluation questions do not ask about the simple existence or nonexistence of some program elements, but seek information about such abstract concepts as the relevance of the curriculum and of its content, residents' knowledge and attitudes, and faculty qualifications. The prototype approach does not provide criteria for judging whether or not these concepts are being implemented. As a consequence, the evaluator must rely on other than well defined criteria for reaching a judgment in answering these five evaluation questions. Further, the design of the approach does not provide for procedures for defining explicit criteria such as cutoff points at which instructional strategies are said to be fully implemented. 5. The success of subsequent evaluations using the prototype program evaluation approach depends to a degree on the objectivity of the internal evaluator. The data obtained using the prototype approach may be used in several ways. One important use of the evaluation data is to externally validate the curriculum and objectives. Another important use of the data is to make improvements to the rotation. A third use is to supply evidence for program accreditation. The objectivity of the internal evaluator is essential in reporting the results of subsequent evaluations. Continuing development of the rotation and accreditation based on fact are at issue. The potential for bias by internal evaluators, thus, is a weakness of the prototype approach. 6. Attention to medical concerns by the internal evaluator may outweigh attention to evaluation concerns. A problem for the internal evaluator who is also a member of the rotation faculty (e.g., rotation director, head trainer) will always be one of priorities. The researcher suspects that medical concerns will routinely be given a higher priority during rotation activities than will matters of program evaluation. This may be a weakness of the prototype approach. (There may be an exception to this when the internal evaluator is not a faculty member of the rotation--is, for example, a fellow.) So, the strengths of the approach include the specific evaluation questions component, the requirement of specific rotation goals and objectives, the I58 collecting of both qualitative and quantitative data, the clear definition of responsibility for rotation improvement, the possibility of subsequent evaluations performed by internal evaluators, the inclusion of both formative and summative evaluation components, and the comprehensiveness of information obtained by means of the approach. And the weaknesses of the approach include the need for a trained external evaluator, the substantial initial investment in time and money, the need for the cooperation of a number of people, the inadequate or nonexistent methodology, the dependency on the objectivity of the internal evaluator, and the relative priorities of medical and evaluation concerns by internal evaluators. A field test was conducted with the prototype program evaluation approach. Data were collected and analyzed as a part of the field test. On the basis of the results from the field test, the following conclusions are drawn about the prototype program evaluation approach: I. The approach was useful in providing information to the rotation directors. 2. The approach was practical. 3. The approach was ethical. 4. The approach conveyed to the rotation directors accurate information about the rotation. Recommendations for Program Evaluators The ultimate value of the study reported on in this dissertation is the utility of the prototype approach for instructional developers and program evaluators charged with the task of evaluating graduate medical education programs. Program evaluation is an integral component in any instructional system in l59 which systematic development and improvement and planned change are valued. The prototype approach presented in this dissertation can be employed by instructional developers to generate the information required to develop and plan change in graduate medical education programs systematically. Internal evaluators may benefit from the prototype approach as they learn from external evaluators how to conduct their own program evaluations. There is a list of recommendations for both external and internal graduate medical education program evaluators who wish to implement evaluations using the prototype program evaluation approach: The cooperation of the various groups and individuals upon whom the I. evaluation depends should be secured in advance. Securing cooperation of the participants in advance ensures that the necessary data can be collected by means of the data-collection strategies built into the design. Criteria for making judgments should be agreed upon in advance by the 2. evaluator and the rotation director. Agreement in advance on criteria for making judgments would prevent potential misunderstanding between the rotation director and the evaluator from arising during data analysis and interpretation. An internal evaluator should be designated for each residency program. The internal evaluator should work closely with the external evaluator on the initial rotation evaluation. 3. The designated residency internal evaluator could perform evaluations on various rotations within a residency program. Thus, the services of an external evaluator would be required only once. This would save development time and money. The initial development costs of doing evaluations would also be spread over a number of rotations, and the unit cost of doing an evaluation on an individual rotation in a residency program would be lowered. Finally, the designated residency internal evaluator would be more likely to attend to I60 evaluation concerns than medical concerns during the evaluation. 4. Evaluation questions should be tightly focused an informational needs to facilitate data collection. The evaluation questions should be agreed upon in advance by the rotation directors and the evaluator. Agreement on evaluation questions helps to ensure that most of the information obtained will be of direct utility to the rotation under study. 5. A table of specifications for matching questionnaire items to evaluation questions the items are intended to answer should be developed early in the study. Similarly, a table of specifications should be developed for the checklists used to guide observation and interviews during the rotation. Such tables of specification would help to ensure that sufficient data are obtained to answer the evaluation questions and that extraneous data are not gathered. 6. Arrangements should be made for periodic systematic external reviews of the primary evaluations. The periodic, systematic external reviews (metaevaluations) would serve as an assessment of the merit of the primary evaluation. Metaevaluations also provide a hedge against internal evaluator bias. Before the ultimate value of the evaluation approach presented in this dissertation is assessed, further evaluations must be conducted: first, subsequent evaluations of the rotation in sports medicine to determine whether the design can indeed be implemented by internal evaluators. Primary evaluations of other rotations by external evaluators will be needed as well, followed by subsequent evaluations of these rotations by internal evaluators. Only after this additional evidence is collected can more definite conclusions be drawn about the merit of the prototype program evaluation approach presented here. l6l Recommendations for Further Research The study described in this dissertation was designed to be applicable to a wide range of rotations. A limit of the study was that it was conducted by an external evaluator. Further, only one rotation was evaluated by means of the prototype approach. Additional research is therefore necessary to determine the ultimate value of the approach to internal evaluators in different settings. Here are some recommendations for further research: I. The prototype program evaluation approach should be field tested on different rotations in different settings. The field test of the approach was conducted on a multidisciplinary rotation with multiple training sites. Future research should focus on rotations with single disciplines that are conducted at one location (e.g., ambulatory settings). A second purpose of applying the evaluation approach in different settings is to determine whether the l2 evaluation questions are adequate and sufficient, thereby satisfying the design criterion of reflecting the informational requirements of the rotation being evaluated. Another purpose is to determine possible modifications in the evaluation process. Yet another purpose is to determine whether the weaknesses detected in the field test conducted in this study manifest themselves in other settings. 2. Further research should be conducted to determine whether the prototype program evaluation approach can be implemented by internal evaluators. The approach was designed in such a way that the initial evaluation was done by an external evaluator. The external evaluator designed and developed the data- collection instruments, collected and analyzed the data, and prepared and delivered the evaluation reports. Further research should be conducted to determine whether an external evaluator is required to perform the initial evaluation in each rotatiOn setting, or if internal evaluators can determine appropriate subsequent evaluation information requirements, modify the data collection instruments I62 provided in this study, and conduct subsequent evaluations. 3. Additional research should also address the question of how often, comparatively, external and internal sources should supply information. Using the prototype program evaluation approach, the researcher collected information from sources considered internal to the rotation (e.g., former and current residents) about rotation intents and implementation. He also obtained information from external sources (i.e., the external review panel) to validate the curriculum and objectives of the rotation. Further research should be done on the optimal frequency with which rotation evaluations should be conducted by obtaining data from internal sources, and on the frequency with which external data are required to validate the curriculum and objectives of a rotation. 4. Further research should be conducted on the data-collection procedures employed in the prototype program evaluation approach. The researcher collected redundant information in the field test of the prototype approach. Further research is required to determine the minimum number of data-collection strategies actually needed to obtain valid and reliable data about a rotation during an initial evaluation. Summary The study described in this dissertation was conducted because of a void perceived in the area of program evaluation models to meet the unique requirements of residency rotations. A prototype program evaluation approach for residency rotations was designed, developed, and field tested on one rotation. The prototype approach was then metaevaluated. The results from the primary evaluation and from the metaevaluation of the primary evaluation suggest that the prototype approach may be useful to residency and rotation directors. Subsequent evaluations by internal evaluators of the rotation evaluated, and additional primary and subsequent evaluations in other rotation settings, will ultimately determine the value of this prototype program evaluation approach for residency rotations. REFERENCES REFERENCES Alkin, M.C. (1969). Evaluation theory development. Evaluation Comment. _2_(1), Alkin, M.C., 6c Fitz-Gibbon, C.T. (1975). Methods and theories of evaluating programs. Journal of Research and Development in Education, _8_ (3), 2~15. American College of Sports Medicine: 1982 Membership Directory. (1982). Indianapolis: Author American Medical Association. (1983). 1983/1984 Directog of residency training programs. Chicago: Author. American Orthopedic Society for Sports Medicine. (1982, October). Membership roster. Chicago: Author. Anderson, 5., dz Ball, 5. (1978). The profession and practice of program evaluation. San Francisco: Jossey-Bass. Arter, Judith, A., and others. (1982) State—of-the—art in evaluation reports and in selecting exemplary reports; AERA competition for best report of an evaluation. Northwest Regional Educational Laboratory. (Eric Document Reproduction Service No. ED 220 508) Baker, E.L., 6t Alkin, M.C. (I973). Formative evaluation of instructional development. Audio-Visual Communications Review, E (4), 389-420. Baron, II.B., 6c Baron, R.M. (1980). In search of standards. In R. Perlofi (it E. Perlott (Eds.), Values, ethics, and standards in evaluation (New directions for program evaluation, Number 7). San Francisco: Jossey—Bass. Berdie, D.R., 6c Anderson, 1F. (1974). Questionnaires: Design and use. Metuchen, N.J.: Scarecrow Press. Berk, R.A. (1981). Educational evaluation methodology: The state-of—the-art. Baltimore: Johns Hopkins University Press. Bland, C.J., Stoller, J.E., Ciriacy, E.W., 6c Houge, D.R. (1978). An internal review of a residency training program. Journal of Medical Education, 2, 426-429. Blank, W.E. (1982). Handbook for developing competengy—based training programs. Englewood Cliffs, NJ: Prentice-Hall. 163 164 Bloom, B.S. (Ed.). (1956). Taxonmy of educational objectives. New York: David McKay. Borich, G.D. (Ed.). (1974). Evaluating educational prggrams and products. Englewood Cliffs, NJ: Educational Technology. Borich, G.D., 6c Jemelka, R.P. (1981). Definitions of program evaluation and their relation to instructional design. Educational Technology, 31-38. Brethower, K.S., 6c Rummler, G.A. (1977). Evaluating training. Improving Human Performance Quarterly, _5_ (3—4), 103-120. Britan, G.M. (1978). Experimental and contextual models of program evaluation. Evaluation and Program Planning, 229-234. Burton, N.W. (1977). Performance standards. (Occasional Paper No. 11). Kalamazoo, MI: Western Michigan University, Evaluation Center. Canfield, P.R. (1976). Family medicine: An historical perspective. Journal of Medical Education, 2, 904-911. Chawhan, A.R., 6c Shahabi, A. (1979). Perception of teaching competencies by medical educators. Medical Education, Q, 298—302. Cook, T. D., 6: Gruder, C. L. (1978). Metaevaluation research. Evaluation Quarterly, 2(1), 5-51. Corbett, T.C. (1979). Evaluation of post-implementation strategies and systems: Introduction. In P.P. LeBreton, V.E. Bryant, D.L. Zweizig, D.G. Middaugh, A.G. Bryant, 6: T.C. Corbett (Eds.), The evaluation of continuing education for professionals: A systems view. Seattle: University of Washington. Corley, J.B. (1976). In-training residency evaluation. The Journal of Family Practice, 2 (5), 499—504. Corley, J.B. (1976). Evaluating residency training, an operational prototype. Charleston, SC: Medical University Press. Corley, J.B. (1983). Evaluating residency training (2nd ed.). Lexington, MA: Collamore Press. Cronbach, L.J. and Associates. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass. Cronbach, L.J. (1982). Designing evaluations of educational and social program . San Francisco: Jossey-Bass. Datta, LE. (1981). In R.A. Berk, Education evaluation methodology: The state of the art. Baltimore: Johns Hopkins University Press. Ebel, R. L. (1979). Essentials of educational measurement. Englewood Cliffs, NJ: Prentice- Hall 165 Fitz-Gibbon, C.T., 6: Morris, LL. (1978). How to design a program evaluation. Beverly Hills: Sage. Forman, D.C. (1980). Evaluation of training: Present and future. Educational Technology, 2000), 48—51. Geyman, J.P. (1975). Continuity of care in family practice, Part 4: Implementing continuity in a family practice residency program. The Journal of Family Practice, 2(6), 445-447. Geyman, J.P., at Brown, T. (1976). An in-training examination for residents in family practice. The Journal of Family Practice, 3(4), 409-413. Geyman, J.P., at Brown, T. (1976). A network model for decentralized family practice residency training. The Journal of Fa_r_ni1y Practice, 3 (6), 621-627. Giesen, L.A. (1979). An evaluation of Grotelueschen's model for evaluating an adult education program. Minneapolis: University of Minnesota. (Eric Document Reproduction Service No. ED 201 764) Glass, G.V. (1977). Standards and criteria. (Occasional Paper No. 10). Kalamazoo, MI: Western Michigan University, Evaluation Center. Gjerde, C.L., 6t Coble, R.J. (1982). Resident and faculty perceptions of effective clinical teaching in family practice. The Journal of Family Practice, l_4 (2), 323-327. Gough, H.G., 6: Hall, W.B. (1977). Physician's retrospective evaluations of their medical education. Research in Higher Education, Z, 29-42. Gowin, D.B., 6c Millman, J. (1978). Can meta-evaluation give a direction for research on evaluation? Ithaca, NY: Cornell University. (Eric Document Reproduction Service No. ED 161 938) Grenough, J.L., 6c Dixon, R.W. (1982). Using 'utilization' to measure training results. Training/HRD, 40—42. Grotelueschen, AD. (1980). Program evaluation. In A.B. Knox 6c Associates (Eds.) Developing, administering and evaluating adult education. San Francisco: Jossey—Bass. Guba, E.G. (1975). Problems in utilizing the results of evaluation. Journal of Research and Development in Education, _8 (3), 42—54. Guba, E.G., 6: Lincoln, Y.S. (1981). Effective evaluation. San Francisco: Jossey- Bass. Hambleton, R.K., 6r Powell, S. (1983). A framework for viewing the process of standard setting. Evaluation 6: The Health Professions, 6 (1), 3-24. Hammond, R.L. (1969). In B.R. Worthen 6c J.R. Sanders, Educational evaluation: Theory and practic . Belmont, CA: Wadsworth. 166 House, E.R. (1978). Assumptions underlying evaluation models. Educational Researcher, 7 (3), 4-12. House, E.R. (1980). Evaluating with validity. Beverly Hills: Sage. Howard, G.S., Maxwell, S.E., Wiener, R.L., Boynton, K.S., 6t Rooney, W.M. (1980). Is a behavioral measure the best estimate of behavioral parameters? perhaps not. Applied Psychological Measurement, _4 (3), 293-311. Howard, (3.5., Schmeck, R.R., & Bray, J.H. (1979). Internal invalidity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement, 1_6_ (2), 129-135. Hubbard, J.P., Furlow, L.T., 6c Matson, DD. (1967). An in-training examination for residents as a guide to learning. New England Journal of Medicine, 2_7_6_ (8), 448—451. Hubbard, J.P. (1978). Measuring medical education (2nd ed.). Philadelphia: Lea and Febiger. Hull, C.H., 6c Nie, N.H. (Eds.). (1981). SPSS Update 7-9: New procedures and facilities for releases 7-9. New York: McGraw-Hill. Hunt, D.D., Irby, D., Bauer, 5., & Loebel, J.P. (1981). An appraisal of methods of evaluating training in psychiatry. Medical Education, l_5, 332—334. Hunt, B. (1978). Who and what are to be evaluated? Educational Leadership, 32 (4), 260-263. Innes, A.H. (1982). Threats to validity in evaluation models. Charlottesville, VA: University of Virginia. (Eric Document Reproduction Service No. ED 218 352) Iwanicki, E.F. (1974). An approach to improving the evaluation design of ongoing educational programs. Chestnut Hill, MA: Boston College, Center for Field Research and School Services. (Eric Document Reproduction Service No. ED 097 342) Janella, R., and others. (1982). A guide to the development of evaluation reports. The Louisiana round table of proggm evaluators. Louisiana State Department of Education, Baton Rouge. (Eric Document Reproduction Service No. ED 230 621) Jelly, E.C., 6t Friedman, C. (1980). An evaluation system for residency training. The Journal of Family Practice, 1_0 (1), 73—80. Jemelka, R.P., 6c Borich, CD. (1979). Traditional and emerging definitions of educational evaluation. Evaluation Quarterly, 3 (2), 263—276. Joint Committee on Standards for Educational Evaluation. (1981). Standards for evaluations of educational programs, projects and materials. New York: McGraw-Hill. Klintberg, LG. (1976). A responsive evaluation of two programs in medical education. Studies in Educational Evaluation, _2_ (1), 23—30. 167 Levit, E.J. (1969). In—training evaluation of learning. Archives of Dermatology, 2, 342-349. Madaus, (E.F., Scriven, M., or Stufflebeam, D.L. (Eds.). (1983). Evaluation models. Boston: Kluwer-Nijhoff. Marshall, W.R., 6t Davanzo, C.C. (1983). Family practice faculty evaluation: Systematic resident feedback. Family Medicine Review, 2(1), 77-87. Matson, DD. (1966). An in-training evaluation of residency training programs and trainees. Journal of Medical Education, Q, 47-51. Markert, R.J. (1981). Medical education at Wright State University: An evaluation during the first year of residency by the class of 1980 and their residenc supervisors. Dayton, OH: Wright State University, School of Medicine. (Eric Document Reproduction Service No. ED 223 684) Matuszek, P.L., 6: Lee, A. (1977). Evaluating evaluation. Austin, TX: Austin Independent School District. (Eric Document Reproduction Service No. ED 137 417) Millis, J.S. (1966). Thggraduate education of physicians: The report of the citizens commission on graduate medical education. Chicago: American Medical Association. Millis, J.S. (1969). The objectives of residency training. Canadian Medical Association Journal, 1%, 599-602. Millman, J. (1981). A checklist procedure. In N.L. Smith (Ed.), New technigues for evaluation: New perspectives in evaluation (Vol. 2). Beverly Hills: Sage. Mirkin, (3., 6c Hoffman, M. (1978). The sports medicine book. Boston: Little, Brown. Moser, C.A., 6c Kalton, G. (1972). Survey methods in social investigation, New York: Basic Books. National Board of Medical Examiners. Evaluation in the continuum of medical education: Report of the committee on gokals and micrities of the national board of medical examiners. (1973). Philadelphia: Author. Nevo, D. (1983). The conceptualization of educational evaluation: An analytical review of the literature, Review of Educational Research, 5_3.(1), 117—128. Nie, N.H., Hull, C.H., Jenkins, J.G., Steinbrenner, K., or Bent, D.H. (1975). Statistical package for the social sciences (2nd ed.). New York: McGraw- Hill. Nilsson, N., (St Hogben, D. (1983). Metaevaluation. In E.R. House (Ed.). Philosophy of evaluation. New directions for_prpg_ram evaluation: No. 19. San Francisco: Jossey Bass. Patton, M.Q. (1978). Utilization-focused evaluation. Beverly Hills: Sage. 168 Patton, M.Q. (1980). Qualitative evaluation methods. Beverly Hills: Sage. Peterson, G.W. (1982). A meta—evaluation of a generic skillspmxoach to evaluating academic programs. Tallahassee, FL: Florida State University, Center for Educational Technology. (Eric Document Reproduction Service No. ED 219 398) Provus, M.M. (1969). Evaluation of ongoing programs in the public school system. In B.R. Worthen 6t J.R. Sanders, Educational evaluation: Theory and practice (pp. 170-209). Belmont, CA: Wadsworth. Rachal, Janella, and others. (1982). A guide to the development of evaluation reports. Baton Rouge: Louisianna State Department of Education. (Eric Document Reproduction Service No. ED 230 621) Reynolds, M.M., & Chanel, P.E. (1979). Evaluation of a residency training program. Journal of Medical Education, 23, 957-959. Richards, R.K. (1978). Continuing medical education: Pergpectives,problems, prognosis. New Haven: Yale University Press. Rossi, P.H., 6c Freeman, H.E. (1982). Evaluation: A systematic approach (2nd ed.). Beverly Hills: Sage. Rotem, A., or Bandaranayake, R. (1983). How to plan and conduct programme evaluation. Medical Teacher, 2 (4), 127-131. Schneeweiss, M.B. (1981). Requirements for residency training in family practice. Journal of Family Practice, l_2_ (6), 1085-1086. Scriven, M. (1969). An introduction to meta-evaluation. Educational Product Report, 2, 36-38. Shepard, L.A. (1977). A checklist for evaluating large-scale assessment programs. (Occasional Paper No. 9f Kalamazoo, MI: Western Michigan University, Evaluation Center. Smith, M.E., O'Callaghan, M., Corbett, A.J., Morley, B., 6r Kamradt, LL. (1977). An example of meta-evaluation from industry. Improvig Human Performance Quarterly, 5 (3—4), 168-182. Smith, N.L. (1981). Criticism and meta—evaluation. In N.L. Smith (Ed.), New techniques for evaluation: New perspectives in evaluation (Vol. 2). Beverly Hills: Sage. St. Pierre, R.G. (1980). Follow through: A case study in metaevaluation research. Cambridge, MA: ABT Associates. (Eric Document Reproduction Service No. ED 187 721) Spain, F., 6: Woiwode, D. (1979). Graduate follow—up survey in a university—based family practice residency. The Journal of Family Practice, 2 (6), 1108-1110. Stake, R.E. (1967). The countenance of educational evaluation. Teachers College Record, Q, 523-540. 169 Steele, S.M. (1973). Contemporary approaches toirogfam evaluation and their implications for evaluating programs for disadvantgged adults. Syracuse, NY: (Eric Document Reproduction Service No. ED 075 715) Stevenson, J.F., Longabaugh, R.H., 6c McNeil, D.N. (1979). Metaevaluation in the human services. The Evaluator and Mana ement, Sage, Beverly Hills, 37-54. Stufflebeam, D.L. (1968). Evaluation as enlightenment for decision-making. Columbus: Ohio State University Evaluation Center. Stufflebeam, D.L. (1971). Educational evaluation and decision—making. In B.R. Worthen 6r J.R. Sanders, Educational evaluation: Theory and practice (pp. 128-150). Belmont, CA: Wadsworth. Stufflebeam, D.L. (1974, April). Toward a technology for evaluatingevaluations. Paper presented at the annual meeting of the American Educational Research Association, Chicago. Stufflebeam, D.L. (1978). Metaevaluation: An overview. Evaluation and the Health Professions,l(l), 17—43. Stufflebeam, D.R., 6c Webster, W.J. (1980). An analysis of alternative approaches to evaluation. Educational Evaluation and Policy Analysis, 2(3), 520. Stufflebeam, D.L. (1981). Metaevaluation: Concepts, standards, and uses. In R. Berk (Ed.), Educational evaluation methodology: The state of the art. Baltimore: Johns Hopkins University Press. Taylor, D.B. (1976). Eeny, meeny, miney, meaux: Alternative evaluation models. The North Central Association Quarterly, _52 (4), 353-358. The Task Force on Graduate Medical Education of the Association of American Medical Colleges (1980). Graduate medical education: Proposals for the eighties, Washington: Association of American Medical Colleges. Tyler, R.W. (1942). General statement on evaluation. Journal of Educational Research, g, 492—501. Weiss, C.H. (1982). Measuring the use of evaluation. In E.R. House, S. Mathison, J. Pearsol, ér H. Preskill (Eds.), Evaluation Studies Review Annual (Vol. 7). Beverly Hills: Sage. Wentling, T.L., 6c Klit, J.A. (1973). Meta-evaluation applied: The evaluation of a large scale evaluation system. University of Illinois. (Eric Document Reproduction Service No. ED 075 509) Whitman, N. (1983). Evaluating family practice residency programs: A guide to systematic data collection. Family Medicine, 1_5 (1), 6-10. Willis, B. (1979). A design taxonomy utilizing ten major evaluation strategies. International Journal of Instructional Media, 6 (4), 369-388. Wilson, J.L., 6c Dutt, J.S. (1983). A family practice preceptorship: Design, implementation, and evaluation. Family Medicine, Q (4), 134-138. 170 Worthen, E.R., dc Sanders, J.R. (1973). Educational evaluation: Theory and practice. Belmont, CA: Wadsworth. Yelon, S.L. (1984). How to use and create criterion checklists. Performance dc Instruction Journal. April, 1—4. APPENDICES APPENDIX A PARTICIPANTS QUESTIONNAIRE MICHIGAN STATE UNIVERSITY ROTATION IN SPORTS MEDICINE PARTICIPANT'S QUESTIONNAIRE INSTRUCTIONS The questionnaire is divided into three sections. Each section requires only several minutes to complete. If you cannot complete the entire questionnaire at one time, you may find it convenient to complete one section at a time. There is a code number on your questionnaire. The code number is for follow-up purposes. It is used to prevent you from receiving reminder letters, not to identify you with your responses. 1. Please answer all questions in order. 2. Most questions can be completed by checking an answer space. If you do not find the exact answer that fits your case, please write a short answer in the margins. 3. Some participants in the rotation were not members of a residency program. Thus, some questions refer to "rotators" and not to residents. 4. The accuracy of your responses is vital as we seek to improve the rotation. Please be straightforward in your answers. You will not be identified with your answers. 171 172 Part I This part of the instrument dea is primarily with your practice since you participated in the rotation in Sports Medicine. The questions will not be used to identify you. They are included to determine the effect your experiences may have had on your answers to other items in the instrument. Please check the month and year in which you participated in the rotation. January April July October February May August November 1978 March June September December 1979 § _ fi _ ‘1980 ‘1981 1982 If you participated in the rotation more than once, please indicate the month(s) and year(s). Where are you currently practicing? City suburban rural other (please specify) Are you board eligible or board certified in any specialty area(s)? Yes (please specify) No Did you have formal training in sports medicine (eg. class, clerkship, . internship, residency) before you elected the rotation in Sports Med1cme? Yes (please specify) No Since you participated in the rotation, have you had any additional formal 7 training (eg., Classes, clerkships, internships, reSidenC1es) in sports medicme . Yes (please specify) o Since you participated in the rotation in Sports Medicine, from which sources have you continued to learn about sports medicine? a I have discontinued my study of sports medicine b journal articles c books d conferences e peers f office experience g sports teams for which I have served as team physician h other (please specify) If you Checked "a" above, please go to question 9. Rank order the three sources (from question 7 above) from which you continue to learn the most about sports medicine by placing the appropriate letters in the spaces provided. learn most about sports medicine learn next most learn 3rd most In the last year, what have you done on a communit level in sports medicine? (eg., participated in coaching seminars) Please list any sports in which you participate regularly for personal recreation. Please check any of these journals which you read on a regular basis. The Physician and Sports Medicine The American Journal of Sports Medicine Medicine and Science in Sports and Exercise Physicial Fitness Research Digest *_Other sports medicine related journal (please specify) Have you ever published an article that dealt with sports medicine? Yes (please indicate the journal(s) and date(s)) No Are you currently involved in any sports medicine research? Yes (please describe briefly) ——N° 173 Please estimate how much (percentage) of your current total patient care involves sports medicine (eg., managing sports medicine problems, preventing sports injuries, rehabilitation, performing sports physical exams, etc.). 96 If you practice n_o sports medicine (0%) please go to question 29. Please indicate the three most frequent reasons for sports medicine patient visits to your practice: 1._seen most a) acute injury f) rehabilitation 2.___seen 2nd most b) chronic injury g) sports physical exams 3._seen 3rd most c) overuse injury h) exercise prescription (1) acute illness 1) sports advice; guidance e) chronic illness j) other (please specify) Please indicate the three age groups you see most frequently when you practice sports medicine: I. seen most a) 0-4 years (pre—school) 2. seen 2nd most b) 5—10 years (grade school) 3 seen 3rd most c) ll—13 years (junior high) (1) 14—17 years (high school) e) 18—22 years (college) f) 23 and over (recreational) g) 23 and over (elite--please specify sport(s)) When practicing sports medicine, what percentage of each sex do you see? 96 female % male 100% Do you serve as team physician for any group? Yes (please check each age group for which you serve as team physician) No (please go to question 19) 0-4 years (pre~school) 5—10 years (grade school) 11-13 years (junior high) 14-17 years (high school) 18—22 years (college) 23 and over (recreational) 23 and over (elite--please specify sport(s)) llll If yes, check either or both of the following that you attend in person: practices games, events, etc. 174 20. 21. 22. 23. 24. 25. During a practice or event, have you ever: administered CPR? administered ACLS? removed an athlete by stretcher? Has an athlete ever died while competing in a sport for which you were the team physician? If yes, did the athlete die during practice game _pother (please specify) Were you present? Yes No When practicing sports medicine, do you work with a certified athletic trainer (A.T.C-)? Do you recommend or prescribe rehabilitative exercise programs for your athlete patients? Do you perform screening sports physical exams? Can you now diagnose sports medicine problems that you were unable to diagnose before participating in the rotation? Have you found it necessary to modify any sports medicine techniques you learned in the rotation to fit your practice? If yes, please describe or give an example. 175 Don't Know / Recall 26. 27. 28. Don't Know/ Yes No Recall Have you been able to apply any sports medicine skills and/or knowledge to your non—athlete patients? If yes, please give an example. Is there a SErtS medicine network in the community where you practice? If yes, are you an active part of the network? Yes No As a result of participating in the rotation, which of the following are you now, or have you engaged in? read sports medicine journals attend sports medicine conferences participate in a sports medicine fellowship treat sports medicine injuries become affiliated with a community sports medicine program (e.g., become the local high school sports physician) other (please specify) Part II This part of the instrument is directed primarily toward your experiences while you were participating in the rotation. 29. In medical or graduate school, or any other formal training, which of the following topics have you studied or been trained in? Indicate by checking the number of terms prior to the rotation in sports medicine your courses or clerkships have addressed these topics. More One Than One None Term Term Obtaining a medical history __ _ _ Performing a complete physical exam __ __ __ Performing Advanced Cardiac Life Support (ACLS) _¢_ __ __ Interpreting laboratory tests for an exercising person __ — _— Interpreting laboratory tests in health and illness __ _— — (continued) 176 7 29. (continued) None f. Interpreting laboratory tests for a non—exercising person g. Gross Anatomy (e.g., head, neck-spine, thorax) h. Microscopic Anatomy (e.g., histology of muscle, tendons and ligaments) Physiology (e.g., cardiovascular, respiratory, electrolytes) Patho-physiology (e.g., concepts of in- flammation, swelling and repair) 5... H- o k. Pharmacology (e.g., drug absorption, distribution, elimination and side effects) Kinesiology (e.g., muscle fibre arrange— ments, muscle attachments, types of muscle contractions) m. Nutrition (e.g., fundamentals, nutrient composition of foods, role of nutrients in physiological systems) Critiquing Research Literature (e.g., statistical terms and methods, and interpretation) 0. Emergency Management of the acutely injured (e.g., ACLS, suture technique, focusing the clinical appraisal, cast application) ’— D .3 Psychology/behavioral sciences Psychiatry Physical/rehabilitation medicine Use of community medical and health resources __ E”?.O’O 30. For the following items, please estimate the number of times during the rotation that you discussed each of the topics below with your superVisors. NOTE: To accurately complete this questionnaire, when an item refers to your supervisor it means any person or persons from whom you received instruction during the rotation (e.g., faculty phy51C1ans, trainers, nurses, etc.) -0— 1-5 6 or more a. The contrasting sports medicine philosophies of orthopedics and family practice. _ _ __ 13- Equipping a bag for on-site event coverage. __ _ _ c. What must be carried on the person during on-site event coverage. . __ __ __ d- The equipment and supplies that the school must furnish before an athletic event begins. _ __ ____ e. The rules and regulations of various sports. _ _ __ (continued) 177 — ‘ 30. tracer: PST—F 0 ~97? £5 31. (190.00” TT‘ 1'" 3' (continued) The multi-disciplinary approach to sports medicine. Inappropriate conditioning or coaching techniques. Injury—preventing equipment (e.g., shoulder pads, helmets). Appropriate playing surfaces or environments. Psychological motivation of different athletes. The role of diet in specific conditioning programs. Dietary fads practiced by athletes. Age—specific pre-participation physical exams. The difference in treatment plans (medication) for high school and college athletes. Exercise physiology. On—the—field exams (immediately after injury). Clinical judgments regarding whether a player is allowed to return to action or is removed from the event. Treatments for common sports injuries. Rehabilitation programs for sports injuries. The overuse syndrome. Medical care considerations for special athletes. (e.g., handicapped). The major sports medicine journals. Research methods in sports medicine. lllll lllll lllll l l lll For the following items, please estimate the number of times during the rotation that you: Attempted to determine an individual athlete's motivation for participating in a sport. Obtained medical histories of athletes. Counseled athletes regarding improper diet. Performed first aid on athletes. Transported or supervised the transportation of an injured athlete. Developed treatment plans for: High School athletes College athletes Prescribed the following drug groups: NSAID analgesics corticosteroids Diagnosed acute injuries at the informal clinic. Treated acute injuries at the informal clinic. _ . Diagnosed acute injuries at the formal (referral)'clin1c. Treated acute injuries at the formal (referral) clinic. (continued) 178 -o- 1—5 6 or more P (continued) Diagnosed chronic injuries at the formal (referral) clinic. Treated chronic injuries at the tomeferral) clinic. — _ — Treated athletes exhibiting symptoms of the overuse syndrome. Contacted the following in caring for an athlete: parents coaches officials Performed differential diagnoses on the following body parts: knee ankle shoulder neck head Referred to the following for information regarding sports medicine basic science. textbooks journals conference papers consultants Referred athletes to other health care professionals for treatment (e.g., trainers, consultants, nurses). __ Designed rehabilitation programs for ill or injured athletes. _ Prescribed exercise for the following athlete groups: athletes recovering from illness athletes recovering from injury physically impaired athletes athletes with chronic illnesses pre-pubescent athletes recreational athletes geriatric athletes _ _ __ Prescribed the following treatment modalities: cybex __ __ __ TNS units __ _ __ orthotron __ ~_ __ cold pressure boot __ __ _ l l l paraffin baths _ _ _ medcason/ultrasound __ __ __ ice _ __ __ heat __ _ __ EMS units _ _._ __ pressure wraps __ _ _ taping _ _ _ protective padding _ _ __ orthoplast splints _ _ _ stretching (for warming up) _ __ _ contrast bath _ __ __ casting applications __ __ __ 179 _ A. 32. Please estimate the number of times during the rotation that you observed each of the following problems: ' ' —0- 1- Vi 6 or more contusion hematoma myositis ossificans open wounds llll llll llll abrasions lacerations puncture wounds blisters llll llll llll skin infections metabolic disturbances strains and sprains bursitis dislocations/subluxations tenosynovitis stress fractures fractures (osseous, chondral, and osteochondral) muscle cramping nerve injuries disturbances of consciousness disturbances of special senses lll lll lllllll lllllll llll llll lllll 33. Please indicate how effective each of the teaching methods listed below was for learning about sports medicine. \r—l CUP-l a) £38 :4 u Ci (1) (U a) w .. s 3.. 2 g g Zu 8 3 m 'UE‘ :< re :: v-«o [:1 ¢ H QC} a. Orientation Meeting b. Pre-test __ __ — Formal Teaching Sessions 9 d. Informal Clinic (overall rating) _ _ _ _ observation _ _ _ __ discussion _ _ __ _ participation (treating athletes) _ _ _ _ (continued) 180 33. (continued) D—h o m. n. O. [3. Formal (referral) Clinic (overall rating) observation discussion participation (treating athletes) Interaction with Consultants (overall rating) observation discussion participation (treating athletes) Interaction with Trainers (overall rating) observation discussion participation (treating athletes) Site-Visits (overall rating) Youth Performance Institute Center For the Study of Human Performance Area High Schools Practice Coverage (overall rating) observation discussion participation (treating athletes) Event Coverage (overall rating) observation discussion participation (treating athletes) Journal and Literature Readings (overall rating) assigned self-study Active research/experimenrations Independent Study Research Paper Post-test Other (please specify) 181 Excellent Adequate Inadequate Did Not Use/ Don't Recall 34. 35. 36. 37. 38. Did you attend the orientation meeting? a. If yes, was the orientation meeting sufficient to explain what was expected of you during the rotation? If it was not sufficient, please comment. Did you receive a description of the curriculum? a. b. If yes, was the description clear? Was the description consistent with the actual implementation of the rotation? If no, please comment. Did you receive descriptions of the specific behaviors expected of you during the rotation? a. b. If yes, were the descriptions clear? Were the descriptions consistent with the actual implementation of the rotation? If no, please comment. Did you receive a description of the time commitments expected of you? If yes, was the schedule consistent with the actual implementation of the rotation? Did you receive a schedule of activities in which you were expected to participate? If yes, was the schedule consistent with the actual implementation of the rotation? 182 Don't Recall 39. #0. 41. 42. 12 Don't _ Yes No Recall Were there expectations of you that were implied, but not stated? a. If yes, what were they? b. How did you learn about them? Were you told who would suErvise you during the rotation? If yes, did these people, in fact, supervise you? Were you personally evaluated by your supervisors during the rotation? (Do not consider any final reports, sent to your residency director) If your answer was "no", or "Don't Recall", please go to question 43. a. If yes, were you told how you would be evaluated? — b. Were you told who would be evaluating you?— C. Do you feel you were, in fact, evaluated according to what you were told? d. If no, according to what criteria were you evaluated? e. How did you learn of these criteria? Please check each of the following items that you think were used to evaluate you during the rotation. Then, rank order the three you think were used most to evaluate you (not all items are necessarily relevant). a. How well I managed common sports medicine problems b. My ability to discuss preventive methods of sports injuries C- My competence in on-site coverage of sports events d. My rehabilitation program designs for various sports injuries e. My competence in working with the various individuals involved with the total treatment of the athlete f- My attitude 5;- My personality h . My attendance 1. My interpersonal interaction skills j. My physical examination skills k- My skill at taking histories 1- My overall sports medicine knowledge -——_ . Other (please specify) 3 n. Other (Please specify) used most to evaluate my performance (place letters in spaces provided) used 2nd most used 3rd most 183 43. till. 45. 46. 099?? trace HU’ T-P'PPPBT‘?" Don't Yes No Recall During the rotation, were you receptive to being evaluated? How often do you think evaluation should take place? Should this evaluation be formal (eg., exams, patient management problems) informal (eg., periodic questioning from your supervisors) both Please indicate the degree to which you feel the following resources were available to you during the rotation. More Than Sufficient Sufficient Insufficient Don't Know/ Can't Recall other rotators acutely injured athletes chronically injured athletes acutely ill athletes (common illnesses) chronically ill athletes (common illnesses) llll l recreational athletes __ _ physically impaired athletes clerical assistance consultants faculty physicians lllll trainers nurses coaches __ parents __ team practices __ team events __ variety of sports environments __ informal clinical space (Jenison, Olin and Football Building) __ __ __ __ formal clinical space (Clinical Center) __ __ __ __ rehabilitation facilities __ __ l lllllll strength testing equipment __ __ __ sports medicine library __ office space (faculty and staff) __ (continued) 1 l 184 U c: ‘16. (continued) 3 0 ~14 EH EH :1 1.1 1-4 V) I: \r—‘l u G) 3 m C C1 w-l O U m cu o I: w ,1: -.-1 -.-i :4 m H U 1H w-l :44 u u <11 u-i :1 — ~ 1.1 1+: m i: i: o :1 c: o to z m Pl Q o x. educational materials (videotapes, slides, books, journals, etc.) y. medical supplies 2 office supplies aa. Other (please specify) 47. For the items below, please indicate how much time was made available to you during the rotation to: More Than Sufficient Sufficient Insufficient Don't Know/ Can't Recall manage common sports medicine problems gain a working knowledge of how to prevent sports injuries become competent in on~site event coverage design appropriate rehabilitation programs for various injuries work with various individuals (athletes, parents, coaches, trainers, officials and other medical personnel in the community) who are involved in the total treatment of the athlete. __ __ 185 Part III The final part of the instrument deals primarily with how you feel about the rotation and how we can change to improve it. 48. 49. 50. 51. 52. As 'a result of the rotation, please list the three areas of sports medicine in which you think you have the most knowledge. In addition, please list the three areas of sports medicine which you think you must study more. Most Knowledge Need More Study What person or activity contributed most to your learning during the rotation? Why? What person or activity contributed least to your learning during the rotation? Why? What changes would you make to improve the person or activity which contributed least to your learning? What were your reasons for electing the rotation in sports medicine? (check all that apply). Then, rank order the three most important reasons that you elected the rotation. There was a void in my schedule. The reputation of the rotation. The reputation of the faculty. Advised by my residency director. Advised by another rotator. Interest in sports medicine. A desire to fill a professional need. I am a former athlete. I plan to practice sports medicine to some degree. 1 l ‘.-:-'.:rqo 1'"? an 0'1» llllllll Other (please specify)_________________________ Most important reasons Ielected the rotation. (Place letters in spaces provided) most important reason 2nd most important 3rd most important 186 l 16 53. What were your expectations of the rotation? (check all that apply) To learn about the philosophy of sports medicine. (e.g., ethics, medical—legal issues, primary care vs. orthopedic approach, etc.) To learn basic science of sport. (e.g., exercise physiology, biomechanics, kinesiology, pharmacology of sport) To learn preventive sports medicine. (e.g., conditioning and training techniques, injury prediction and prevention, epidemiology of exercise and injury, etc.) To learn patient care aspects of athletes. (e.g. management of common sports injuries, rehabilitation techniques for ill or injured athletes, medical care of special athlete groups, etc.) To study research in sports medicine I had no well defined expectations Other (please specify) 54. To what degree were your expectations met? The rotation: Exceeded my expectations. Met my expectations. Fell somewhat below my expectations. Met very few of my expectations. 55. How well did the rotation prepare you to include sports medicine in your practice? I feel well prepared to practice sports medicine. I feel comfortable in practicing sports medicine. I feel somewhat uncomfortable. I feel unprepared to practice sports medicine. 56. To what degree are the skills and knowledge you acquired during the rotation incorporated in your practice? always seldom often never sometimes 57. To what degree was the rotation consistent with your previous education and training? totally consistent some inconsistency mostly consistent not at all consistent For the following questions, please indicate whether the answer is "yes", "no“, or whether you are "undecided." 58- Were the backgrounds Yes No Undec1ded of the various faculty members sufficient for you to learn sports medicine? __ _ __ 187 59. 60. 61. 62. 63. 64. 65. 66. 67. 68. Yes No Undecided Were the faculty were up to date in their fields? On a daily basis, did the faculty seem prepared? Did the faculty seem interested in teaching you? ' What part of the rotation did you like the best? Why? What part of the rotation did you like the least? Why? What needs or problems did you have during the rotation that went unmet? What one or two sports medicine topics that were IE addressed in the rotation would have been valuable for you to study? Did this questionnaire address what you think were the important topics of the rotation? yes no (please identify the topics missed) Please list any other comments you have about the rotation: Check here if you wish to receive a synopsis of this study. . . velo - Please return your completed questionnaire in the enclosed, stamped en pe Thank you. 188 APPENDIX B SURVEY OF IMPORTANT COMPONENTS OF A FAMILY PRACTICE ROTATION IN SPORTS MEDICINE Survey of Important Components Of A Family Practice Rotation In Sports Medicine INSTRUCTIONS This survey is an attempt to determine the specific skills and knowledge which should be possessed by the Family Practice physician who is beginning to include sports medicine in his/her practice. The skills and knowledge are taught in a month-long rotation in sports medicine offered by the Department of Family Practice at Michigan State University. The skills and knowledge have been divided into the following areas: Patient Care Aspects of Sports Medicine Basic Science of Sport Preventive Sports Medicine Research in Sports Medicine Philosophy of Sport NOTE--Some skills and knowledge do not seem to fit perfectly under the areas in which they have been included. This is of less concern than the importance of the item, itself. I. Please indicate the importance of the skills and knowledge according to this scale, circling only the one (I) answer that fits best: I Attaining this skill or knowledge is essential in order to begin practicing sports medicine. 2 Attaining this skill or knowledge is desirable, but not essential in order to begin practicing sports medicine. 3 Attaining this skill or knowledge is not imErtant in order to begin practicing sports medicine. 4 I am uncertain of the im rtance of this skill or knowledge for beginning to practice sports medicine, or, I do not understand the meanipg of this skill or knowledge. 2. Most questions can be answered by circling a number in one of the columns. If you do not find the exact answer that fits yours pomt of view, please write a short aiswer in the margins. 3. Your advice and opinions are vital as we seek to improve our rotation in Sports Medicine. Please be straightforward in your answers. You Will at be identified with your responses. 189 There is a code number on your questionnaire. It is used for follow-up purposes, and will prevent you from receiving reminder letters. The code number is not used to identify you with your responses. — PATIENT CARE ASPECTS OF SPORTS MEDICINE This section deals with patient care aspects of sports medicine. For our purposes, patient care means the clinical skills and knowledge necessary to treat sports medicine injuries and illness. With regard to patient care aspects of beginning to practice sports medicine, please indicate the importance of attaining the following skills and knowledge: Uncertain of importance/ Don't understand meaning +.> 3 r— 44 .D (U C ._ ,5 .. +2 4—> l'" CU C 5- IC !— CU O -r- .0 U) D— +.> m m E C 5— CI.) -r— 3 '5 ,_, ,_, m (I) O O LLI D C 2 I. Developing a working relationship between the team physician and the coach. I 2 3 4 2. Identifying the egipment which should be carried in the team physician's bag. | 2 3 4 3. Identifying the equipment the team physician must carry on his/her person. I 2 3 4 4. Identifying the equipment and supplies which should be furnished by the . 4 school before an athletic event can begin. I 2 3 5. Performing emergency care on the injured 3 4 athlete. I 2 6. Diagnosing and treating the injured athlete 3 4 on the field. I 2 7. Transporting/supervising the trans- 3 4 portation of the injured athlete. l 2 190 Designing and prescribing appropriate rehabilitation programs for ill or injured athletes. Identifying medical care considerations for special athlete groups (e.g., mentally retarded). Determining when to use consultants in medical decision-making. Determining which consultants to use in medical decision-making. Determining when to talk with coaches, parents, and significant others in medical decision—making. Performing the following physical examin— ations and arriving at correct differential diagnoses: Knee Ankle Shoulder Neck Head Understanding and discussing the concept of the overuse syndrome. Diagnosing the overuse syndrome. Properly treating the overuse syndrome. 191 Essential Desirable, but not essential N Not important Uncertain of importance/ Don't understand meaning b-i-‘J—‘J—‘D b I7. Properly prescribing the following: 20. 2|. 22. 23. Cybex (iso—kinetic testing) Essential TNS units (transcutaneous nerve stimulators) l Orthotron cold pressure boot paraffin baths ice heat EMS units (electrical muscle stimulators) presswe wraps taping medcolator with ultrasound (e.g. Medcason) protective padding orthoplast splints stretching (for warming up) Taking accurate and complete histories of athletic injuries. Making distinctions between appropriate medication treatment plans for high school and college level athletes. Identifying the proper medications with, which to quip the bag for on-site event coverage. Counseling athletes regarding dietary fads. Formulating appropriate treatments for common sports injuries. Devising treatment plans for athletes with common illnesses. 192 Desirable, but not essential NN Not important “WU-’00 to www tance/Don't under— stand meaning Uncertain of impor— .l-‘J-‘J—‘D J—‘DJ—‘J—‘h J—‘J—‘J—‘J—‘b .l.‘ 24. 25. s'. . O S— D. OJ E '0 +3 'l— C 3 l— 4—’ 3 U7 .0 (U C ‘4— C -.— re 0 4.) -r— n 4..) +3 - C '— 0) C L C C N) (U I_- OJ C 'l" 0 OJ 'r— .0 U) D- “ Q E p to m E 4-’ \ E L G.) -r- L Cl.) "U G, .,_ w u C m m -I—’ +3 U C (U V) G) O O C (G 44 Lu D E Z 3 44 (It Counseling athletes with regard to their current psychological states. 2 3 4 What additional patient care aspects of sports medicine skills and knowledge would you recommend a Family Practice physician possess before beginning to practice sports medicine? (please rate your recommendations as either I (Essential), or 2 (Desirable, but not essential).) l 2 | 2 | 2 l 2 | 2 193 BASIC SCIENCE OF SPORT This section deals with the basic science of sport. For our purposes, basic science means the study of the normal systems by which the human body functions, and the effects of exercise on these systems (e.g., physiology, anatomy, kinesiology, nutrition etc.) With regard to the basic science of sport for beginning to practice sports medicine, please indicate the importance of attaining the following skills and knowledge: 73 26. Identifying appropriate references 2:: for information about sports 5; related problems. L” texts I journals I unpublished professional papers I 27. Recalling pertinent anatomical and kinesiological information (e.g., muscle groups, attachments, functions, etc.) when treating specific injuries. | 28. Assessing the value (which drug to use, when to use it, appropriate dosage, duration of prescription, side effects) of the following drug groups: NSAI D I analgesics I corticosteroids I 194 Desirable, but not essential NN Not important woo Uncertain of importance/ Don't understand meaning J.‘ 29. 30. 3|. Identifying current dietary fads used by athletes. Understanding and discussing exercise physiology concepts and effects on the human body. What additional basic science of sport skills and knowledge would you recommend a Family Practice physician possess before beginning to practice sports medicine? (please rate your recommendations as either I (Essential), or 2 (Desirable, but not essential).) 195 — Essential Desirable, but N not essential W Not important Uncertain of impor— 3‘ tance/Don't under— stand meaning PREVENTIVE SPORTS MEDICINE This section deals with preventive sports medicine. For our purposes, preventive sports medicine involves actively changing the medical care system to prevent various problems (e.g., epidemiology, screening, special population concerns, etc.) With regard to preventive sports medicine for beginning to practice sports medicine, please indicate the importance of attaining the following skills and knowledge: i . O S— D. Q) E n -I—) 'r- C 3 r-- «l—> 3 D (U c 9— ". f6 0 U C S 8 8 2 g .8 32. Conducting age—specific but non—sport- specific pre-participation physical exams. l 2 3 4 33. Discriminating between appropriate, and inappropriate conditionim techniques for various sports. I 2 3 4 34. Discriminating between appropriate and inappropriate training techniques for various sports. I 2 3 4 35. Discriminating between proper and improper coachng techniques. I 2 3 4 36. Diagnosing injuries resulting from in- appropriate and improper conditioning, training and coaching techniques. I 2 3 4 37. Prescribing appropriate exercise for athletes according to: age I 2 3 4 presence of chronic illnesses l 2 3 4 temporary physical impairments l 2 3 4 rehabilitation from illness I 2 3 rehabilitation from injury I 2 3 4 38. Designing a community-based program of 2 3 4 sports medicine care. I 196 stand meaning 39. 40. 4|. 42. 43. Identifying appropriate injury-preventing protective equipment (e.g., mouth guards, safety glasses, pads, etc.) Determining the proper use of injury- preventing protective quipment. Properly applying the basket-weave ankle strapping (taping technique). Evaluating the sports environment to determine potential hazards to the athlete (pre-competition injury pre— vention technique). What additional preventive sports medicine skills and knowledge would you recommend a Family Practice physician possess before beginning to practice sports medicine? (please rate your recommendations as either I (Essential), or 2 (Desirable, but not essential).) 197 Essential Desirable, but not essential N Not important (.0 Uncertain of importance/ Don't understand meaning 4.‘ IO RESEARCH IN SPORTS MEDICIbE This section deals with research in sports medicine. For our purposes, research means identifying and assessmg sources of sports medicine information, and designing and conducting a research study in sports medicine. With regard to research in sports medicine for beginning to practice sports medicine, please indicate the Importance of attaining the following skills and knowledge. I 1 L L 0 CU Cl'U E c 4..) -r— 3 3 r— 4) U3 _o n: c '4— 4-> : :1— (O O - :r— n 4..) +3 C C l" G) C L C O (U (U r— 0.) O 'l— D CU .,_ _Q m o. to E 44 to m E 4—> \ C L CI.) ‘r- 5— Q) 'U a) .,_ a) u c U) U') P -I-) U C (U U) G.) O O C (U -I—’ LLI D E Z I) +9 U) 44. Identifying the major sports medicine journals. I 2 3 4 45. Discriminating between fact and opinion in the sports medicine literature. I 2 3 4 46. Determining strategies for reviewing the sports medicine literature. I 2 3 4 47. Designing a research study in sports medicine. I 2 3 4 48. Conducting a research study in sports medicine. I 2 3 4 49. Writing the results of a research study in sports medicine. I 2 3 4 50. What additional research in sports medicine skills and knowledge would you recommend a Family Practice physician possess before beginning to practice sports medicine? (please rate your recommendations as either I, (Essential) or 2 (Desirable, but not essential).) l 2 l 2 l 2 l 2 l 2 198 PHILOSOPHY OF SPORT This section deals with the philosophy of sport. For our purposes, philosophy of sport refers to the Emedical ramifications of sports (e.g., economics, ethical considerations, philosophical approaches of different specialties, etc.) With regard to philosophy of sport for beginning to practice sports medicine, please indicate the importance of attaining the following skills and knowledge. L . O L D. Cl.) E -o +3 'r— C 3 r— -l-’ 3 U3 .0 (U C '4— C ’I— f6 0 4—> r- . -l-’ -i—> - i: r—- G) C L C C (5 f5 l— O) O -r- O G.) ..._ _C) in D. (U Q E +4 it; w E +—’ \ C L (1) 'r— L G) ‘O , 3 '5 p p 8 8 % 5|. Differentiating between the con- Lfl g g 2‘? :5, 3 1;; trasting sports medicine philosophies of primary care and orthopedics. l 2 3 4 52. Recognizing that the athlete's welfare has priority over the out- come of an event. I 2 3 4 53. Determining the individual athlete's psychological motivation for participating in his/her sport (e.g., weight loss, to turn professional, etc. . l 2 3 4 54. Asking qiestions, td