‘ ,2 . .4"; f} 9"] . e FEWUO’ E531». . .I .m A... ABSTRACT A CASE STUDY IN PROGRAM EVALUATION BY James L. Snook This investigation is concerned with the lack of impact of evaluation and institutional research on aca- demic programs in American higher education. From the perspective and literature of institutional research on the one hand, and sociology, on the other, the case study method is used in an effort to identify interpersonal, institutional, and research factors and dynamics which enhanced or inhibited impact of evaluation research efforts at Kalamazoo College during the three year period, 1966-69. The findings are largely supportive and illustrative of numerous conclusions in the literature which are noted. Four summary conclusions regarding strategy for achieving evaluation impact are offered, with special rele- vance for small, private colleges. In revealing tOp level resistance to evaluation research findings and evidence of it being overcome, this case study contributes more broad- ly to filling a gap in the literature noted by Reginald Carter: There are few reports on how to overcome resistance at the tOp management level to the findings of social science research evaluations. A collection of case histories where management resistance was successfully overcome would be of great benefit to future researchers, and a study of organizational and interpersonal condi- tions would be an important initial step toward changing James L. Snook the dominant latent function of evaluation research from more ritualism to a rationale for change.1 1Reginald E. Carter, "Clients' Resistance to Negative Findings and the Latent Conservative Function of Evaluation Studies," The American Sociologist, Vol. 6 (May, 1971), p,.123. ‘1 '3 Department A CASE STUDY IN PROGRAM EVALUATION BY James L . Shook A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Administration and Higher Education 197 2 I wish in recognition endeavor: To Paul To Kal! the opportunit: iigher educath To my 1 patience and e1 6\ ACKNOWLEDGMENTS I wish to express my appreciation to the following in recognition of their significant contribution to this endeavor: To Paul L. Dressel for his counsel and guidance. To Kalamazoo College, her faculty and staff, for the opportunity to share in the agony and ecstacy of higher education. To my wife, Martha, and children for their loving patience and endurance. ii Clapter I. THE F Rev Kal The Com ’1 we: * & O “883 4' Anal 80111 The Con: III. HISTOE Chapter I. II. III. IV. V. TABLE OF CONTENTS THE PROBLEM AND INTRODUCTION . . . . . . . Review of the Literature . Kalamazoo College . . . . The Case Study . . . . . . Comments . . . . . . . . . METHODO I’OGY O O O I O O O O O O O O O 0 Analyses . . . Sources of Supporting Evidence The Interviews . . . . . . . . Comments . . . . . . . . . . . HISTORY’ . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . GOVERNANCE . . . . . . . . . . . . . . . . EVALUATION RESEARCH AND ITS IMPACT . . . . History of the Office of Institutional Re- search and Program Evaluation . . . . Danforth.Study . . . . . . . . . . . . . Impacteee eeeeeeeeee Senior Independent Project Program Evaluation........... e InpaC‘b e e e e e e e Studies Requested by the President . Attrition Study . . . . Other Studies Having Identifiable Impact Summary . . . . . . . . . . . . . . . . iii 73 86 91 101 102 102 10 5 103 fn .14 Woe-D ' r: "" O oh".¢ a. Chapter Page VI. PROBLEMS IN PRODUCING AND USING EVALUATION RE- SEARCH: IMPLICATIONS AND.RECOMMENDATIONS . . 110 Summary of Major Weaknesses, Contributing Factors, and.Resistance, Existing and Overcome, Regarding the Two Major Evalua- tion Research Efforts: Danforth.Study and SIP Evaluation 0 O O O O O O O O O O O O O 110 Institutional Research: Organizational and Interpersonal Conditions . . . . . . . 119 Further Illustration of the Literature . . . 128 Summary and Conclusions . . . . . . . . . . . 13h BIBLIOGRAPHY O O O O O O O o O O O O O O O O 1- 0 O O O 0 114.0 iv Agra 1. How the LIST OF FIGURES Figure Page 1. How the Kalamazoo Plan Operates . . . . . . . . 53 Review of Th research 1' rather imp 50m. 1% offices of evaluation On higher 98 "In by dem the so by the admini their ‘ t0 eVa instit' future Th ”Eluetion £91 revolui \ - 1E1 Tesearch 0r. ' °° eance hive“ ity . . 2Fr' 3‘. institut .1 CHAPTER I THE PROBLEM AND INTRODUCTION Review of the Literature The surge of growth and popularity of institutional research in American higher education in recent years is rather impressive (Rourke and Brooks, 1966; Cross, 1967; Boyer, 1967; Stecklein, 1970), with college and university offices of institutional research becoming a locus of evaluation efforts in higher education. Boyer states: One of the most prominent features on the American higher education landscape is the looming mass known as "Institutional Research." Besieged on all sides by demands for "excellence," hard pressed to justify the soaring costs of their activities, and tempted by the flowing cornucopia of foundation money, college administrators from.Boston to Santa Barbara have given their blessing to a welter of research programs designed to evaluate past and present developments in their institutions and, hopefully, to cast some light on future trends.1 This surge in institutional research offices and evaluation research also is seen as emphasizing the manager- ial revolution of the last decade and a half,2 institutional lErnest L. Boyer, The Impact of Institutional Research on the Academic ProgrmeAlbany: OTTIce of the Vice Chancellor for University-Wide Activities, State University of New Ybrk, 1967), p. 1. 2Francis E. Rourke and Glenn E. Brooks, "The Growth of Institutional Research," The Managerial Revolution in Higher Education (Baltimore: Jehn Hopkins Press, 1966). l geeds for 53 35513 for de Whil ;-;:pose, OPS affices of 1 game and B. 1955), Cross Typic been cone problems Notiz research and mdinprove e significant c to better, :11: Parent instit E516 by comps There addresses extent be \___ _‘, 3Paul '3‘} h‘d' H fsfr Educat rstztutional 2 needs for self-knowledge,3 and a felt need for more rational basis for decision making and action.h While the response has produced great variety in purpose, organization, personnel, role, and activity of offices of institutional research (Birch, 1970; Lins, 1963; Rourke and Brooks, 1966; Stecklein, 1962, 1970; Stickler, 1968), Cross nevertheless observes: Typically, offices of institutional research have been concerned with the operational nuts-and-bogts problemm involving the allocation of resources. Noting that "a strong link between finstitutionag research and efforts of colleges to strengthen management and improve efficiency"6 has been followed by evidence of a significant contribution by institutional research offices to better, more efficient non-academic management of their parent institutions, Boyer finds the academic side to be pale by comparison. He states: There is, however, no body of literature that addresses itself to the central question: "To what extent has institutional research actually helped 3Pau1 L. Dressel and Sally B. Pratt, The World of Higher Education (San Francisco: Josey-Bass, I971), p. 9. Aw. Hugh Stickler, "The Managerial Revolution in Higher Education: An Overview," Introductory Papers on Institutional Research, ed. E. F. Sdhietinger (Atlanta: §3uthern Regional Education Board, 1968). #Reginald E. Carter, "Clients' Resistance to Negative Findings and the Latent Conservative Function of Evaluation Studies," The American Sociologist, Vol. 6 (May, 1971), pp. llB-lZH. SPatricia K. Cross, When Will Research Improve Education? (Berkeley: Center for Research andeevelopment in Higher Education, 1967), p. 2. 6Boyer, op. cit., p. h. improve C 0] program in safer further 4 While 13 and inf such as on developmen the heart This concl vent liter and the de neth ds re ago. It is ispect of eva] ncgren in hi; vestigation i: A num‘ tiel demand 1‘ evaluation of :iel crisis, criticismfi 1 societmlo ir (“8-, Jencks JOARSOD, and educational e \ 71m 83m figmm Reg: 1171), pp 1 3 improve collegiate life generally and the academic program in particular?"7 Boyer further concludes: While institutional research--organized both formal- ly and informally--also has studied educational matters such as curriculum, teaching methods, and student development, this effort has had only minor impact on the heart of collegiate life--the academic program. This conclusion is supported by the paucity of rele- vant literature, the comments of informed observers, and the degree to which many academic tenets and methgds remain essentially what they were generations ago. It is this problem of lack of, or at best limited, impact of evaluation research, especially on the academic program in higher education, to which this case study in- vestigation is directed. A number of factors seem.destined to create substan- tial demand for more, better, and effective (having impact) evaluation of the academic program: the continuing finan- cial crisis, student unrest and disenchantment, and public criticism;9 increasing awareness of our rapidly changing society;10 increasingly effective authoritative criticimm (e.g., Jencks and Riesman, 1968; Chickering, 1969; Dressel, Johnson, and Marcus, 1970); and increasing possibilities for educational alternatives to compete with conventional 7Ibid., Synopsis BIbid. 9Frederick w. Bolman, "University Reform and Insti- tutional Research," The Journal of Higher Education, Vol. Al (1971): PP0 86-970 10A1v1n Toffler, Future Shock (New York: Bantam Book by Random.House, Inc., . giger educa In 3 (.11 researc gut filth E. istrators "(3 time that c hat one of keep remindi conjunction :ion and dev easily becon Dre: Faii reachim means t] rationai In terms of 3:033 concl higher education.11 In addressing herself to the core question, "When will research improve education?" Cross concludes in agree- ment with H. s. Dyer-12 that most of what faculty and admin- istrators "do as educators is in fact based on blind assump- tions that could turn out to be wrong."13 Dyer contends that one of the functions of institutional research is to keep reminding them of this fact. Cross also cautions in conjunction with T. R. McConne1114 that educational innova- tion and development in the absence of evaluation "may easily become quackery." Dressel has long cautioned that: Failure to engage systematically in evaluation in reaching the many decisions necessary in education means that decision by prejudice, by tradition, or by rationalization is paramount. In terms of higher education, existing or contemplated, Cross concludes: The weight of the evidence to date is that a care- ful program of research and development and evaluation is our most promising approach to improvements in 11Ernest L. Boyer and.George 0. Keller, "The Big Move to Non-Campus Colleges," Saturday Review, (July 17, 1971). PP- hé-SB- 12H. S. Dyer, "Can Institutional Research Lead to A Science of Institutions?," The Educational Record, (Fall, 1966). PP- h52-u66- lBCross, op. cit., p. h. lhT. R. McConnell, "Research or Development: A Reconciliation," Phi Delta Kappa Monograph (Bloomington, Indiana: 1967). 15Paul L. Dressel and Associates (eds.), Evaluation in Higher Education (Boston: Houghton.Mifflin, 1961), p.6. lg views ninerily rich the of the ad: nae: “ice of F theoret higher education.16 Furthermore, according to Boyer: While most institutional research has focused on management matters . . . a few investigations have probed aspects of college life more directly related to the academic program. In some instances these inquiries have emerged from within the milieu of institutional research itself, but more often they have been carried on by behavioral scientists--psycholo- gists and sociologists--not directly identified with the fraternity.17 Dyer notes similarly the prevalence of two differ- ing views of institutional research: one is concerned primarily with the Operational problems of institutions in which the institutional researcher is "strictly a servant of the administrator"18 and is generally attached to the office of the president; the other view is characterized by theoretically oriented, long term.studies of students and the inner workings of institutions and is beholden to the social sciences. Dyer acknowledges that "research of an operational variety is unquestionably important," but counters that 90perationa1 research, almost by definition pays little or no attention to the fundamental purposes of an institution or to the value systems that control it."19 Asking further, "What kind of research will help college-sponsored offices of institutional research contri- bute to the maximum.development of their colleges?," Cross l6Cross, op. cit., p. u. 17Boyer, 0p. cit., p. 7. 18Dyer, cp. cit., p. ASh. 19M. 1:; oke S Dy‘ Cf: operat "fesea' of edu differ consti Operat where the so what 13 tion 8 3:38, Cros ;erspectiv research. D? 1:: and en is to ' someho tional tional Di EM 21290353, . @112 3P in be a 6 invokes Dyer's caution that: Offices of institutional research would neglect Operational problems at their peril, but he warns that "research that does not go beyond the instrumentalities of education is unlikely to make any fundamental difference in the impact of the institution on its constituents or on the community it hepes to serve. Operational research, uninformed by theory, goes no- where . . . it may be concerned with ways of ensuring the solvency of the institution, but it does not ask what purposes are being served by keeping the institu- tion solvent."20 Thus, Cross' question may beg a broader social science perspective than that typically encompassed by institutional research. Dyer surmizes: If institutional research is to have a positive and enduring impact on institutional quality, if it is to become a science in any acceptable sense, it must somehow integrate both of these points of view @pera- tional and theorgiIEEIJ in a common attack on institu- tional problems. Discussing "social research and educational policy," martin Trow has identified the following three broadly different kinds of research--each having somewhat different purposes, methods, strategies, and problems:22 predictive - aims to tell us what will result from some specific educational practice or pattern and often involves establishing a close statistical connection between one or more characteristics of students and a predicted educational outcome. directive - arms to assimilate policy making to research, intervening directly into educational practice with statements of what ought to be done on the basis of its findings. ZOCI‘OCI, OE. Cite, p. 20 21Dyer' OE. OitO’ p. 14-514. 22Martian. TrOw, fSocial Research and Educational Policy," Research.in Higher Education: Guide to Institutioge a1 Decisions (New York: #5311ege Entrance Examination Board, 9P0 ' --' : z. 0.: 1 vs problems 7 illuminative - aims to explore and illuminate the nature of educational institutions and processes-- to show the connections among student characteristics, organizational patterns and policies, and education- al consequences. It is concerned especially with the ways in which.these and other components of educational systems interact with one another. Sociologist Trow argues for illuminative research as the best kind of institutional research. He contends that this kind of research sidesteps the distinction between "pure" and "applied" research on the grounds that what may be most valuable to policyhmakers is a fuller under- standing of their own institutions and what goes on within them. . . . Thus, this mode of research assumes that nothing is necessarily outside its terms of refer- ence and aims for clarification and understanding on the basis of which wiser and.more effective policy decisions can be made.23 Dressel and Pratt note that institutional research with its focus generally being on studies specific to the problems of a particular institution or larger system of higher education has been typically predictive or directive in.nature. In contrast to institutional research, research.on higher education starts with the identification of issues and problems and undertakes to make a contribu- tion to the understanding or the resolution of themm2u Hence, research on higher education, a broader concept than institutional research, is typically more illuminative in nature. ‘Yet, the question of impact of research on.higher education is no less disquieting than the question of the impact of institutional research. Dressel and Pratt observe: Despite the volume of research, its impact on 23Ibid. 2LLDressel and Pratt, op. cit., p. l. institt rather and PW the re: bring ' sane 1" differl search! conclu. sions, nation on pub with r: many or betweel As safer lite: numerous a1 proving in: seen to cm 3153918, D: researcher, fiance, WaI'T 25 p30 . 27 caving Ed 8 institutions and on higher education generally may be rather limited. The problems of interpreting research and putting it into action are many. However, much of the research is illuminative and is not oriented to bring about action. The results of research in the same field are often contradictory because of the differences in assumptions or in the values of the re- searchers. A great deal of the research is simply in- conclusive, and, to a considerable extent, final deci- sions, whether on a single campus or at a state or national level, are based on political maneuvering and on publicly stated values or secretly held prejudices, with research findings having limited impact. And, in many cases, it is difficult to trace the co ection between the research.and the actions taken.2 As is evident in the recent annotated guide to the :major literature in higher education by Dressel and Pratt, numerous authors offer suggestions or principles for imp proving institutional and evaluation research which would seem to contribute to increased impact of evaluation. For example, Dressel and Pratt suggest that the institutional researchers' investigations take on new depth and signifi- cance, warrant greater confidence, and are more effective when their investigations evidence awareness of the large body of research.on higher education. McConnell?6 in noting what has paid off in other fields such.as the natural sciences, sounds a prominent theme in the literature: agreeing with Dyer, he suggests that significant improvement in impact on education by institutional and other evaluation research requires that they focus on questions which have reference to theoretical-explanatory principles (such as learning and motivation27) on the one hand and relevance to 251bid., p. 8. 26Mcconne11, op. cit. 27L. J. Cronbach, "The Role of the University in Im- proving Education," Phi Delta Kappan, (June, 1966), pp. 539- SAS . Dve :eseaI‘Ch tc First: it deal tions 1. problem Second, measure that a}: refined cation recogni bilitie which I sults.£ Pr a educat i hypothe who ther measure habitua The not in; imports kind of saying Surate Ag:- “8 Princip :55 more in i‘ylade‘ZL‘nac'qv lea; “know Pact, Asse latter how 9 practical problems in education on the other. Dyer suggests necessary conditions for institutional research to achieve positive and enduring impact: First, institutional research must be so organized that it deals with the real problems of particular institu- tions while at the same time endeavoring to fit these problems into some sort of evolving generalizations. Second, it must develop more dependable and believable measures to get at the many institutional variables that appear important as our observations become more refined. And third, it must acquire much.more saphisti- cation about how to conduct genuine experiments which recognize the restrictions while exploiting the possi- bilities in the data peculiar to its own domain and which lgad to a body of interpretable, cumulative re- sults.2 Practically any question one can raise about the educational enterprise can be formulated as a testable hypothesis so as to get a reasonable indication of whether the opportunities we provide students make a measurable difference in the kinds of thinking they habitually do and the kinds of people they become. 9 These three conditions are difficult but probably not impossible to fulfill. In view of the central importance of institutions of higher learning to our kind of society in our kind of world, it goes without saying that the effort to meet them should be commen- surate with.the magnitude of the problem..30 Agreeing only in part with Lewis Mayhew that one of the principal reasons "that institutional research has not had more impact on the conduct of institutions . . . is the inadequacy of available instrumentation,"3l Dyer neverthe- less acknowledges the centrality of measurement to such im- pact. Asserting that it is the measurable things, no :matter how subjectively or inaccurately measured, that tend 28Dyer, op. cit., p. h66. 29Ih1d., p. A65. 3OIbid., p. nee. 311bid., p. h58. :3 effect ed: thr°° ma seefiurem any genu ins’citut 80313 8r factoI‘S not: :as been ‘19" the idea or input is beg tat it is 11“- research the gave: ‘oei‘ol‘e t] otheI’WIS‘ 811(1 j_ff 13] interest‘ and purP‘ cros: In 0“ place Wh‘ % when W9 - practice It i: aspects to 131'. he evaluatic Bore usable a academic prog tpical insti noted above. P950 urces, pe: “--------- 32Tyel 34cm 10 to affect educational practice, Dyer notes three major classes of institutional problems in which measurement, if rightly conceived, is indispensable to any genuine solutions. They are the definition of institutional goals, the determination of how well the goals are being met, and the identification of the factors that facilitate or impede reaching the goals.32 Noting that most measurement in higher education has been devoted to measurement of inputs, Dyer believes the idea of judging quality education by relating output to input is beginning to take hold. Furthermore, he cautions that it is important also that the ultimate consumers of the research-~that is, the administration, the faculty, and the governing body--have a.hand in defining the variables before the computer starts grinding out the numbers. Otherwise, the numbers may be regarded as irrelevant, and if they are regarded as irrelevant by any of the interested parties, they are irrelevant for all intents and purposes. "“ Cross' conclusion seems apt: In other words, improvement in education will take place when we study the right kinds of questions and when we Iearn how to apply research knowledge to the practice of education. It is evident that there are at least two major aspects to the problem.of limited impact of institutional and evaluation research. The first, making such research more usable as a basis for improving education including the academic pregram, will apparently require more than the typical institutional or educational research approaches as noted above. Certainly, the organization, perceived role, resources, personnel, etc. of an institutional research ‘— 32Dyer, op. cit., p. #59. 33Ibid., pp. ABC-A59. 3uCross, op. cit., p. 1. ence Hill 9763 Possib: The of those uh! gay be no le would bring useful inst; my be desi: May} and researci did have 011 the c and an 1 results reasona‘e Exar 5398 Of Suce 511d educatie l Dumbo: tions w} selecti. Often a. ‘36 reveals, seriously 0. “Suwani 11 office will have much to do with whether this is likely or even possible. The second, getting such research into the thinking of those who make educational policy and program decisions, may be no less critical. Presumably, institutions which would bring such resources together so as to produce more useful institutional and evaluation research would in some- way be desirous of it. Mayhew offers examples of institutional research and research on higher education that did have an impact on the practice of higher education. On the other hand, there have been both notable failures and an enormous number of situations in which research results were not utilized even though findings were reasonably clear.35 Examing the results of efforts to evaluate the de- gree of success or failure of social action, industrial, and educational innovations, Carter notes a number of case studies that indicate specific condi- tions where evaluation research.findings have been selectively ignored. Thus, even if evaluation research is competently conducted, the fate of the regglts is often determined by another set of criteria. He reveals, for example, that clients:may desire an image of actively supporting scientific evaluations and/or may never seriously consider the possibility that the results might be negative-~i.e., not agreeing with the client's (and 35Lewis B..Mayhew, "Educational Research, Its Capabilities and Limitations," Research.in.Higher Education: Guide to Institutional Decisions (New York: College En- trance Examination Board, I965), p. 6. 36Carter, 02. cit., p. 118. memes th Latter point as the main - little impac Cart my clients : tion researc. incomtered . Positively I" system of th. 1) the p: reSeaJ 2) the 0.1 3) the f, ‘4) past 1 Searc; 5) map?) Seller; Hence 395:1me 111 re institut: potentia: I”$112137 ' ' ' Ami groups he call}, a dc t° make a effecmVG pWential ' a apprc Interest on the is 12 sometimes the researcher's) anticipated results. This latter point is suggested by sociologist Peter Rossi37 as the main reason why negative results generally have so little impact. Carter notes that there are a number of reasons why clients may choose not to accept the results of evalua- tion research in addition to the "resistance frequently encountered by social scientists when their findings do not positively reinforce the expectations and ideological value system of their client." These includez38 1) the psychological characteristics of clients and/or researchers ) the quality of the research 3) the feasibility of implementing the recommendations A) past negative experiences with social science re- search 5) inapproPriateness of data and/or researcher to generate policy recommendations Hence, in addition to producing higher quality, more meaningful research, Boyer points out that institutional research will never achieve its maximum potential impact until its results are transmdtted1more forcibly to the peOple who hold decisionémaking power. . . . Amid.the welter of interested individuals and groups he he researchexj must first and most emphati- cally address himself to the peeple who hold the power to make a firm.decision on a given matter. To be effective, the institutional researcher must define his potential audience carefully and sensitively and plan his approach in terms of capturing the attention and interest of the persons who are prepared to take action on the issue at hand. In short, he must have a 37Peter Rossi, Boobytraps and Pitfalls in the Evalu- ation of Social Action Programs,Froceedings offtheISocial StatisticsfSEcti on of:the Annual Meetings of the American Statistical Association (Washington, D. 0.: American Statistical Association, 1966), pp. 127-137. 380arter, Op. cit., p. 118. strategy car“ Ther‘ at the '5‘ science I historie: overcome and a at tions we the domi: from men In 3 that this at A survey tete of e corresj rephies 1 very lit“ complete! carefull' Kalamazoo Co” M In 1' the "Kalamaz. fihfitentiall- “mg 31118111 ] Duriz =4» .wempted a J self-evaluat; l3 strategy for getting results.39 Carter concludes that: There are few reports on how to overcome resistance at the top management level to the findings of social science research evaluations. A collection of case histories where management resistance was successfully overcome would be of great benefit to future researchers, and a study of organizational and interpersonal condi- tions would be an important initial step toward changing the dominant latent function of evaluation research from mere ritualism to a rationale for change. 0 In sum.then, it is to the situation noted by Boyer that this study is directed: A survey of the literature makes clear that the recent state of research activity has not been accompanied by a corresponding surge of influence. Research bibliog- raphies multiply like fruit flies, yet we still know very little about what happens when a project has been completed, conclusions drawn, and statistical tables carefully filed awayJLl Kalamazoo College In 1961, Kalamazoo College in Kalamazoo, Michigan adapted an academic programewhich has come to be known as the "Kalamazoo Plan." This innovative plan has contributed substantially to an increased prominence of that institution among small private liberal arts colleges in America. During the 1966-67 academic year, Kalamazoo College attempted a formal self-evaluation. An outgrowth of that self-evaluation effort was the establishment of a full-time Office of Institutional Research the following year. That office was terminated two years later. 39Boyer, op. cit., p. 28. MOCarter, op. cit., p.123 hlBoyer, OE. cit., p. 2. This £1891! or i :c-o (301.188e aeCffice c extent that :hsotudy 1 getting evaI faculty and Sollege ope: It c tmotioning Iejor claim: md its acac' mid (also a toutees or lane, a Btl 153101151 anc- Chterh? is within its E Mt °f Kala of the E°Ver evaluation 1' But respec In t; inattempt 1: \\ thar la The Case Stag This case study is an attempt to assess the impact or lack of impact of evaluation studies conducted at Kalama- zoo College under the auspices of the formal self-study and the Office of Institutional Research from 1966-69. To the extent that lack of impact is identified, a major focus of this study is consideration of difficulties, if any, in getting evaluation research findings into the thinking of faculty and administration so as to be evidenced in the College operation and progrmm. It does not require excessive visibility into the functioning of higher education in America to recognize nmjor claims of proprietorship and management of the college and its academic program by the administration on the one hand (also as representative of the governing board of trustees or regents) and by the faculty on the other. Hence, a study of management resistance and related organi- zational and interpersonal conditions as called for by Carterh2 is important to this case study and is included within its scope. Accordingly, an analysis of the develop- ment of Kalamazoo College and the Kalamazoo College Plan, of the governance of the College, and of the impact of the evaluation research noted above comprise Chapters III, IV, and V respectively. In the concluding Chapter (VI) of this case study, an attempt is made to identify both (a) major weaknesses in hgcarter, op. cit., p. 123. seleCted 6V8] nmibut ing 3353381133, search, and t :tercomeuif aria conclud tich enhanc 'epact at Ka sture and to planning and In a investigatic evaluation : this case st (1) author of ti College, fix mistant tc imitufiom the Office < emitence (; hand faflilig Q‘aent °bjec 1 (2) efferta uncle 15 selected evaluation research under investigation and factors contributing to such weaknesses, and (b) major sources of resistance, if any, to the findings of that evaluation re- search, and circumstances by which such resistance was overcome--if indeed it existed and was overcome. In sum, this concluding chapter is an attempt to identify factors which enhanced or inhibited evaluation research and its impact at Kalamazoo College in light of the selected liter- ature and to derive implications and recommendations for planning and carrying out such evaluation. Comments In addition to the salience of the problem.under investigation and efforts at Kalamazoo College to attempt evaluation research, additional factors are favorable to this case study on Kalamazoo College at this time: (1) A time lapse of two years has passed since the author of this case study was affiliated with Kalamazoo College, first as full-time consultant and technical assistant to the faculty director of the 1966-67 formal institutional self-study, and then as full-time Director of the Office of Institutional Research during its two year existence (1967-69), hOpefully yielding a blend of first hand familiarity with the subject at hand with the subse- quent objectivity of an "outside" point of view. (2) It is now two to five years after the evaluation efforts under investigation, allowing time for impact to occur 0 Jar-nu .‘lhifi <3) T‘r £1 personnel he Kalamazoo ed the evalu 15200 3011889 retiring soon {a} President Erofessor Ray figire in the by the facult l6 (3) The Kalamazoo Plan is ten years old, yet nearly all personnel who have been involved in the development of the Kalamazoo Plan, subsequent governance of the College, and the evaluation studies under investigation are at Kala- :mazoo College at this time. However, two key people are retiring soon after this investigation is conducted: (a) President Weimer K. Hicks, in December, 1971, and (b) Professor Raymond Hightower, in September, 1971; a key figure in the deve10pment and passage of the Kalamazoo Plan by the faculty, and.Dean under the new Kalamazoo Plan. After :Mcademic A1” early in March son which at t proposed study resources fron mnyreceived 1971 letter f: The ix basic to the ‘ them out. The (1) de mazoo (2) gc tional (3) in Sation (’4) pr into 1;, CHAPTER II METHODOLOGY After discussing this tentative study with the Dean of Academic Affairs at Kalamazoo College, Douglas Peterson, early in March, 1971 the author sent a letter to Dean Peter- son which at his request noted the purpose and tenor of the preposed study and what would be needed in terms of time and resources from.Kalamazoo College and its personnel. The study received approval as formally noted in an April 7, 1971 letter from.Dean Peterson. Analyses The intended analyses introduced in Chapter I are basic to the types and sources of evidence needed to carry them out. These analyses lie in five areas: (1) development of Kalamazoo College and the Kala- mazoo Plan: Chapter III (2) governance of the College (including organiza- tional and interpersonal conditions): Chapter IV (3) impact of the evaluation research under investi- gation: Chapter V (k) problems of getting evaluation research findings into the thinking of the faculty and administration, 17 and c impac Chapt aortas of SL‘ Analy tense from tr. and records, (1) 2 history of K5 pertaining tc 2h01min: of cation and 01 Study where 1 3. III, and \ (2) ; interviewees I Videli'd‘rom a1 Policy Con he Office of Number of r mm“ are m (3) _I_ noted as a De; 18 including resistance factors which inhibited or pre- cluded impact and were or were not overcome: Chap- ter VI (5) implications and recommendations for planning and carrying out evaluation research so as to have impact, particularly in a small private college: Chapter VI Sources of SupportingEvidence Analyses in these five areas rest primarily on evi- dence from three sources: (1) the literature, (2) documents and records, and (3) interviews conducted by the author. (1) The literature is that directly related to the history of Kalamazoo College and that noted in Chapter I pertaining to institutional and evaluation research from the point of view of institutional research in.higher edu- cation and of sociology. It is noted throughout this case study where relevant, appearing most frequently in Chapters I, III, and v1. (2) Documents and records noted or donated by the interviewees, or known to, or found by the author vary widely--from.news releases and position papers to Education- al Policy Committee minutes and evaluation research reports. The Office of Academic Affairs was very helpful in locating a number of relevant materials and records. Documents and records are most evident in Chapters III and V. (3) Interviews with Kalamazoo Collegg:personnel were noted as a part of this case study in the March 16 letter to lean Peterson: flaws and a t‘ are included at he was 8' Listed indica ccnld exp“t Inter f-llege facul (a) have play of the Kalama influential" the inception administrativ the Kalamazoo toltenent in The 0 EL‘COI‘ding t0 Primarily fI‘O researCh, and ducted with i lifit of int er :ritel‘ia "an schedule item Mom in th at “aired in ital list 01‘ View” being at 19 Dean Peterson. The prhmary topics of the preposed inter- views and a tentative list of 22 persons to be interviewed were included. Dean Peterson's response of approval noted that he was sending a supporting note to those persons listed indicating the nature of the project and that they could expect to hear from.the author in the near future. Interviews were sought with.members of Kalamazoo College faculty, administration, and board of trustees who (a) have played crucial roles in the conception and/or birth of the Kalamazoo Plan (see Chapter III), (b) have been "most influential" in the College during the ten years following the inception of the Kalamazoo Plan, (c) have held an administrative position with.major responsibility related to the Kalamazoo Plan or program, (d) or have had direct in- volvement in the evaluation research under investigation. The original list of 22 interviewees was drawn according to the above criteria as perceived by the author primarily from his prior Kalamazoo College experience, research, and knowledge. The first interviews were con- ducted with interviewees meeting criteria "c" and "d"; the list of interviewees was subsequently amended in terms of criteria "a" and "b" as trends in response to interview schedule item.C (asking "who have been the most influential persons in the College?") emerged. In sum, the amended list of desired interviewees was comprised of 18 from the orig- inal list of 22 plus six additions. Of these 2h, 22 were interviewed between June 3 and August 13, 1971, all inter- views being arranged and conducted by the author. The wining t'w'O :is hish rank ieParted iron 301139? ”0 i Of 131' three of the criteria; the faculty membé tiewees, 13 l or board of t birth of the teen on leave six more bece next three ye Clear the President R the life crrent high nized as havi there: of in me POSition cf t3.1116. Of . zoo College fa Present ill-flue latter are Hid Ellen“ 0128 and \ 53 {.0116 OJ 20 :remaining two were not interviewed: one having resigned Ihis high ranking, influential administrative position and ideparted from.Kalamazoo College in 1967; the other failing to keep two interview appointments without notice. Of the 22 persons interviewed, eight met at least three of the four criteria; nine satisfied two of the criteria; the remaining five (four administrators and one faculty member) met only one criterion. Of these 22 inter- ‘viewees, 13 have been members of the faculty and/or staff or board of trustees at Kalamazoo College since before the birth of the Kalamazoo Plan in 1960 (though several have been on leave for one to three years during this period) and six more became affiliated with Kalamazoo College within the next three years . Clearly, the Chairman of the Board of Trustees and the President have been among the "most influential" through- out the lifo of the Kalamazoo Plan. In addition, of four current high ranking awmtnistrators who are widely recog- nized as having substantial though somewhat different spheres of influence within the College, two* have held the same position and been influential for an extended period of time. Of these four, three have previously been Kalwma- zoo College faculty members, and three* owe meat of their present influence to the position they hold. Two* of the latter are widely thought to be influential primarily as extensions and representatives of the President's power and *One of these is one of the persons not interviewed as previously noted. gluenc' gong til me, on 0:13: in adviser :ehaz'io: Preside: have met Q payed : rently I 531133 1 (a) beh 3? memb. “Pk f0: of the 4 “ah th $101131 j P3988111- (more 1* 2l influence. Five of these six, and six other administrators are among those interviewed. 0f the remaining six administra- tors, only one is recognized as being very influential, and only in recent years, primarily in the informal role of adviser to the President in the area of student life and behavior which is increasingly beyond the realm of the President's traditional influence. Nevertheless, all six have met criterion "c" on a continuing basis, and.thus have played some part in the governance of the College. Of these ten administrators (excluding Board Chairman, Light) four were previously full-time faculty members at Kalamazoo College, and one of these four plus three others also cur- rently handle some part-time teaching responsibilities. Interviewee comments noting the various spheres of faculty member influence can be summarized as follows: (a) being an Opinion leader and listened to by other facul- ty'members, (b) being willing and having the dedication to work for the betterment of the College or for a constituency of the whole or part of the faculty, (c) being influential with the President by virtue of one's personal and func- tional relationship to the President or one's persuasion, pressure, or expertise which is recognized by the President (more fully elaborated in Chapter IV). Typically, in becoming influential with their colleagues, faculty members also hold important leadership positions in A.A.U.P., the Faculty Senate or formal commit- tee structure, or as dean of academic affairs. It is from sch Pontions muontial dw 3