.R 1,4 'n a Q "I U;‘L-.‘.-..B.A .‘ t ’ x .39’“— ...-,u.... ... - .11.: -: CS ... x . '~ "(‘—'.".f‘-- 1 .'_ ."h, i:h.\lt 1.“; ‘s «f: .; IllItlllzuzlylllullmlmulinwill" This is to certify that the thesis entitled THE IDENTIFICATION OF CPITEPIA FOP EVALUATING GRADUATE PPOGPAMS IN HIGHER EDUCATION AT DIS TEN INSTITUTIONS V presen e by Albert Lynd has been accepted towards fulfillment of the requirements for Doctor of Philosophy degree inAdministration and Higher Education ’ 7 Major professor BMW/fig MM QM ‘ . {1 mx muosnv ms. ', " vi LIBRARY muons ! WHILIICIIW H .. BA‘I‘ ABSTRACT THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS BY Albert Lynd The Purpose A desire for highest quality seemingly needs no defense, regardless of the product or craft involved. Yet the consideration of quality in any setting seems to provoke controversy. Perhaps a part of the controversy regarding quality traces to the concept itself. Today, assessment of quality is among the most complex issues facing institutions of higher education and Higher Edu— cation as an integral part of American higher education faces the same current critical questions put to all graduate study. The identification of criteria for evaluating graduate programs in Higher Education at Big Ten institutions was the centrality of this study. This was accomplished by determining the extent of agreement among department chairmen at those Big Ten institutions which offer the graduate degree in Higher Education and seven recognized experts in the field of Higher Education. Albert Lynd Methodology Utilizing a modified Delphi Method, the study was executed through a three-stage mail survey to the chairmen of the departments of Higher Education at the eight insti- tutions of the Big Ten which offer graduate proqrams in Higher Education, and to seven recognized authorities in the field of Higher Education. Program characteristics developed by the author through a review of the literature were listed under five major headings: Students, Personnel, Program, Finance, and Facilities. The first stage of the survey asked the participants to rank the characteristics in order of importance for judging quality in doctoral programs. The second opinionnaire, based on the first, contained all the characteristics in the order of their rated importance. In addition, the first responses were included for each panelist. Participants again ranked the criteria and indicated reasons why their second rank- ings differed from the first modal rankings. The third and final stage of the survey contained the top five cri- teria from the second stage and the minority opinions provided by the panelists. These Opinions were listed under two headings, one for those contending an item should be "more important" and the other for those con- tending an item should be "less important" than the modal ranking. Participants again ranked the criteria in order Albert Lynd of importance. Frequency tabulations were compiled on the importance of each characteristic. In addition, mean rankings were compiled and the criteria arranged in order of rated importance. Major Conclusions The following briefly relates some major conclu- sions of the study as derived from the literature and findings: 1. Higher Education as a field of study is feasible and defensible. 2. A clarity of program purpose and plan is the major concern in evaluating doctoral programs in Higher Education. 3. Student satisfaction with degree programs is continuing to be important as a measure of quality. 4. The success in placement of graduates and their rise to positions of leadership has become an important measure of quality. 5. Scholarly and research competency of faculty members should be of the highest quality. 6. All departments should insure that disser- tations are of the highest quality. 7. The adequacy of library holdings is a good indication of program quality. 8. Adequate financial support is essential to achieving and maintaining program quality. Albert Lynd Recommendations 1. An intensive study of the history of Higher Education is needed. 2. Specific objectives, related courses, and learning experiences should be developed. 3. Higher Education departments should undertake their own evaluation. Consensus of opinions concerning quality should be obtained from the faculty, students, and the department chairmen. 4. Other groups, such as faculty members, stu- dents, and professional organizations might have similar or different opinions concerning quality. Consensus across groups as well as among groups should be determined. 5. Discussion of the results of studies about quality should be initiated at professional or associ— ation meetings. 6. Summaries of criteria should be used as a framework for a model to assess quality which could be tested empirically. 7. Implementation of a system of evaluating quality should always consider the interdependency of the evaluative criteria. 8. A clearing house should be established for research being conducted into Higher Education by Big Ten institutions. 9. A study focusing on enrollment trends uSIng selected measures of student quality should be undertaken. THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS BY (‘9‘0 a<* Albert-Lynd A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Administration and Higher Education 1976 DEDICATED TO Illene ii ACKNOWLEDGMENTS The author is deeply grateful for the guidance, support, encouragement, and friendship of Dr. Richard L. Featherstone, major adviser and Committee Chairman. To committee members Dr. William Sweetland, Dr. George Myers, and Dr. Lawrence Sarbaugh, the author expresses his thanks for their time, support, and con- structive criticism. The participants in this study are applauded for their c00peration and continued interest in this venture. A special word of appreciation to Dr. Van C. Johnson, Chairman Department of Administration and Higher Education, Michigan State University for his official and unofficial assistance in carrying out this project. iii Chapter I. II. III. TABLE OF CONTENTS INTRODUCTION . . . . . Statement of the Problem Purpose of the Study. . Definition of Terms . . Limitations of the Study Design . . . Overview of the Study . Summary . . . . . . REVIEW OF THE LITERATURE . Introduction . . . . Criteria and Assessment Techniques for Graduate Education. . Reputational Studies . Some Objective Indicators of Excellence. Scholarly productivity. . . . . . Degrees, awards, and other faculty traits . . . . Student quality . . Physical facilities. Client Satisfaction Ratings. . . . . Higher Education as a Field of Study . . The Program of Study. . Assessment of Individual Programs . . . . . Summary . . . . . . METHODOLOGY . . . . . Introduction . . . . Procedure . . . . . Step One . Step Two . iv Institution l4 19 20 23 23 29 30 33 33 34 34 40 40 41 42 42 44 46 S3 58 61 63 63 64 64 64 Chapter Step Three . . . Step Four . . . Summary. . . . . IV. ANALYSIS OF INFORMATION Introduction . . . Analysis . . . . First Opinionnaire Introduction. Students . . Personnel. Program . . Finances . . Facilities . Summary . . . . Second Opinionnaire Introduction. Students . . Personnel. Program . . Finances . . Facilities . Summary . . . . Third Opinionnaire Introduction. . Convergence: The Oracle Summary List of Criteria Speaks. . . for Evaluating Graduate Programs in Higher Education Summary . . . . V. SUMMARY . . . . . Introduction . . . Overview of Study . Conclusions . . Methodology. . . Graduate Education 66 69 71 72 72 73 73 73 74 75 75 83 83 83 84 84 85 95 97 98 98 99 100 100 107 109 111 113 113 115 117 118 119 Chapter Higher Education . . . . . . . . 121 Summary . . . . . . . . . . . 124 Recommendations . . . . . . . . . 125 EPILOGUE . . . . . . . . . . . . . . 130 APPENDICES APPENDIX A. PARTICIPATING INSTITUTIONS . . . . . . 131 B. PARTICIPATING AUTHORITIES . . . . . . 132 C. COVER LETTERS AND FIRST OPINIONNAIRE. . . 133 D. SECOND OPINIONNAIRE . . . . . . . . 141 E. THIRD OPINIONNAIRE. . . . . . . . . 147 F. RESPONSE BY PARTICIPANTS. . . . . . . 152 G. COMMENTS . . . . . . . . . . . . 153 SELECTED BIBLIOGRAPHY . . . . . . . . . . 157 vi Table LIST OF TABLES CHARACTERISTICS OF STUDENTS FOR JUDGMENTS ABOUT THE QUALITY OF PH.D. PROGRAMS; OPINIONNAIRE, STAGE ONE . . . . . . . 76 CHARACTERISTICS OF STUDENTS FOR JUDGMENTS ABOUT THE QUALITY OF PH.D. PROGRAMS; OPINIONNAIRE, STAGE TWO . . . . . . . 86 CHARACTERISTICS OF STUDENTS FOR JUDGMENTS ABOUT THE QUALITY OF PH.D. PROGRAMS; OPINIONNAIRE, STAGE THREE . . . . . . 102 CONVERGENCE OF OPINION ON CHARACTERISTICS CHOSEN "NUMBER 1 .' C O O O I O C C O 10 8 SUMMARY LIST OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION . . 110 RESPONSE BY PARTICIPANTS . . . . . . . 152 vii CHAPTER I INTRODUCTION The University in America is in deep trouble. Ironically, the major source of that trouble lies in its past success: as it succeeded in advancing cultural values, contributing new knowledge, and providing ser- vices to society, the university became a major contrib- utor to the new and drastically different social dynamic which is currently evolving in this country. The char- acter of that dynamic threatens the nature of all insti- tutions, and especially those concerned with higher education. During the postwar years, opportunity for edu- cation beyond the high school was extended. During the decade of the 19608, education was racked by controversy, frustrations, and sometimes violence. And now, during the Seventies, education is experiencing changes. The changes grow out of a re-examination of purposes, out of a realization that segments of the population have been bypassed as we have worked for equality of oppor- tunity, out of a surge of experiments and innovations in programs, and a result of a trend toward systems of institutions. If we need a fresh sense of direction to guide colleges and universities in the midst of change, this direction, in part, will be found by observing some trends of the past. For two centuries after the founding of Harvard University in 1636, American education followed tradi- tional patterns. It then began to develop a character of its own. Church-founded colleges established by the several denominations to preserve and advance particular elements of culture, dotted the landscape. As the fron- tier of civilization moved westward, each of the states initiated a public university as the capstone of its public school system. The federally encouraged colleges of agriculture and mechanic arts, beginning a little over a century ago, gave impetus to research, to occupational training, and to types of extension education that had been frowned on by universities of Europe. The beginning of this profound change in character occurred as the wave of democracy, spirited by Jefferson and brought to a head during the Jacksonian period, swept over the new nation. Alarmed by the extension of the ballot to illiterate persons, the educators advocated universal literacy as the solution. And so, for the first time anywhere in the world, the compulsory study of reading, writing, and arithmetic by all children was advocated. This necessitated the establishment of public normal schools for the training of teachers. It also led to a recognition of the responsibility of the public and the use of tax revenues so that educational needs may be adequately met. During the nineteenth century, a basic pattern for higher education became clear. Private colleges multiplied in numbers and flourished in an atmosphere of relative freedom from state supervision or inter- ference. Parallel with them, however, each state founded a state university and state normal schools and colleges. Shortly thereafter, the Morrill Act became the first significant move by the federal government to stimulate higher education. Therefore, the structure of higher education has been changing. Universities as organized centers of learning date back a thousand years. Seen in this per- spective, these changes in education in the United States during the past century and a half are all the more remarkable. The innovative spirit has permeated American higher education; and the ability to change in the future is supported by these many and varied innovations of the past (Henderson, 1970). Tradition has been accepted when it has proved its merit; but changes have been made as the need for change has been evident. In addition to looking at the past for guidance, it is essential to discover new direction by examining contemporary needs. The ability and thrust for change 9 . . A raises questions about the nature and quality to be ”pkg; a 1.4-7” U ! . e" . . .zu‘fl‘, (L. inifavfiuqu c‘, if .‘.p ‘ offered. Weinberg (1968) suggestsmthat thengjggllg7 mot—WW problem in assessing universities quality has its origin in the essential differences between the university's view of excellence and society's view of it. He notes that universities are, by nature, discipline-oriented, and their idea of excellence is whatever deepens under- standing of or insight into the problems generated or resolved within the various disciplines. A desire for highest quality seemingly needs no defense, regardless of the product or craft involved. Yet, the consideration of quality in any setting seems to provoke controversy. Whenever evaluative judgments are made and a rank order is established, someone is offended. Perhaps a part of the controversy regarding quality traces to ambiguities within the concept itself. As Cartter (1966) notes: "In an operational sense, quality is someone's subjective assessment, for there is no way of objectively measuring what is in essence an attribute of value." Blackburn and Lingenfelter (1973) suggest that although quality may be more than the sum of its parts, communication is facilitated and controversy is mitigated when quality is defined in terms of its component parts. If the objectives of an insti- tution or program are defined, an assessment of its quality principally considers the degree to which those objectives are attained--"the degree to which excellence obtains along specific dimensions." Additionally there is agreement that while quality and excellence exist, some institutions and programs have more of it than do others. And quality and excel- lence do matter. Further, they matter in important ways. Research shows unequivocally that faculty behavior is associated with institutional quality (Wilson, 1942; Parsons & Platt, 1968), that institutions differ on scales of excellence (Berelson, 1960), and on other characteristics, as this study will attempt to show. Also, despite the controversy which sometimes accompanies assessment, systematic evaluation of all graduate programs is becoming increasingly important and useful. Widely accepted projections of Ph.D. supply and demand (Brode, 1971; Cartter, 1971; National Science Foundation, 1971; Wolfe & Kidd, 1971) predict surplus doctorates in almost every field. Although these pre— dictions have drawn criticism (Letters, Science, 1971; Moses, 1972), the magnitude of the predicted surplus is too great to be dismissed lightly. Financial and market pressures are likely to force cutbacks and reallocations of resources in graduate education. Careful regular assessment of excellence is necessary to insure that whatever actions are taken relative to graduate programs be guided by a clear perception of their strengths, weaknesses, and social contributions. Limited resources should be invested where they can produce the greatest return. Of course, assessment is necessary for positive reasons as well. The expansion of knowledge continues unabated, and the rates of change in society and in the modes of transmitting and utilizing knowledge are accelerating without pause. Sound research and teaching at the graduate level is critical both to develop needed technological and social innovations and to cultivate deeper understanding of the human situation. Given the rate of change in the environment, regular re-evaluation of the effectiveness of graduate education is necessary to forestall obsolescence and irrelevance. Finally, quality assessment possesses an internal virtue of no small consequence, one which by itself justifies the incorporation of program as an operational procedure as regular as the annual audit. The assessment process qua process has salutary consequences. New ideas emerge; better practices are introduced, concerned self- analysis questions long standing assumptions, protects against dysfunctionalism, and vitally important, generates a climate for healthy growth and development. In these and other concerns of government and industry and education, excellence in graduate education is a sine qua non. Regular assessment, no matter how agonizing at times, is fundamental, especially in graduate education. Now to assess quality, then, is no idle concern. Doctoral education in this decade has clearly entered a period of major reassessment and readjustment. One major reason for the review of priorities is the serious financial stress experienced by the higher edu- cation community generally and doctoral institutions particularly. This stress has been exacerbated by the curtailment of government support at all levels. In addition, the demand for doctorates characteristic through the 19608 has been largely satisfied. According to many prognosticators, the nation may be forcing a glut of doctorates in the next decade and beyond in many fields, unless the present rate of doctorate production is moderated. Finally, there has been growing a general concern and dissatisfaction over how the universities actually have been serving individuals and the nation. Without delving too deeply into its historical development, it at least should be recalled that the modern American graduate school is the result of the grafting of the German concept of postgraduate study upon collegiate institutions which evolved from the English model and whose course of study traditionally ended with the A.B. degree. To the extent that the A.M. degree was awarded prior to its rehabilitation during the period 1853-1881, it represented the recognition only of "bachelors of arts who are engaged in literary or pro- fessional pursuits and who pay to their college a fee prescribed by its regulations" (Eells, 1963). The Ph.D. was first granted as an earned degree on the completion of a formal thesis at Yale in 1861. It was, however, the opening of Johns Hopkins in 1876, of Clark University in 1888, and of the University of Chicago in 1890 that focused attention on the potential of post-baccalaureate study to meet the needs of an increasingly complex culture for specialized education. During the last part of the nineteenth century, the College of Arts and Sciences at the major American universities typically organized graduate councils to supervise post-baccalaureate degree programs within the college. As the Ph.D. degree began to attain prestige approaching that of the professional degrees in medicine, law, and theology, it became recognized as a desirable goal of post-baccalaureate studies in subjects other than those supervised by the faculties of Arts and Sciences. Gradually, over the years, the privilege of granting this degree has been extended to other faculties through the devise of splitting off the graduate council from the faculty of Arts and Sciences to form a university- wide graduate school representative of all faculties, but dominated by the traditional arbiters of the post— baccalaureate liberal arts degrees. In 1900, about 250 legitimately earned doctorates were conferred by American universities (Berelson, 1960). In contrast, some 18,000 doctorates were awarded in 1966, and 29,872 in 1970. Less than fifty institutions granted the philosophical doctorate in the nineteenth century. The comparable number was 213 for 1960-66, and had grown to 286 by 1969-70 (National Board on Graduate Education, 1972). In 1900, fourteen institutions organized the Association of American Universities with a principal objective of cooperating on the standardization of graduate programs. At that time, this group of insti- tutions was granting 88 percent of all Ph.D.‘s in the United States. For the period of 1960-66, the same fourteen institutional organizations granted only 31 percent of the doctorates, and the Association itself, now grown to forty-six U.S. institutions, awarded 67 per- cent of all philosophical doctorates given in the country. Their graduate deans are currently represented by the 10 Association of Graduate Schools of the Association of American Universities. The enormous broadening of the number of insti- tutions offering post-baccalaureate liberal arts programs has led to the formation of the Council of Graduate Schools, the basic requirement for membership in which is the granting of thirty A.M.'s and M.S.'s or Ph.D.‘s in at least three fields of study over a period of three years. Nearly three hundred institutions have qualified for membership in Council of Graduate Schools. Thus, there has been considerable concern nationally that this proliferation of new doctoral pro- grams and the greater growth of enrollment in those programs which are not of the highest quality may have created an overall dilution in quality of doctoral edu- cation. Yet continuing interest has been shown in ways to analyze graduate programs in American colleges and universities. This interest has taken several directions: First, the publication of a list of graduate schools which have been found competent to conduct prOgrams for the doctorate degree; and secondly, self-appraisal. Yet there seems to be little systematic and organized effort to develop criteria for evaluating quality in a variety of settings and providing reliable information about the diverse aspects of graduate education. As a 11 result, there has, in all probability, developed a varied surge of graduate programs and a more varied quality of work offered at institutions. It seems logical to expect to find that any specialized field of study must have developed over a period of time rather than to have emerged suddenly from a corn field as an active volcano. If there has been such development, one should expect increasing status and recognition. Although the study of Higher Education has not yet reached maturity, it possesses many of the qualities of a scholarly field. Programs designed to prepare persons to enter into professional work in Higher Edu- cation exist in great numbers. Departments of Higher Education and Centers and Institutes with hundreds of faculty members sponsor research into higher education as well as provide degrees and public service programs. An expanding literature encompasses all aspects of the higher education enterprise. Dressel and Mayhew (1974) cite the growing bureaucracy associated with Higher Education as a discipline and the problem of trying to view it without the multitude of individuals and organi- zations and reports and studies which relate to its understanding. Higher Education permits individuals to begin work or scholarship from a variety of formal academic disciplines and the contributing disciplines 12 and authors are so numerous and varied as to preclude any dominant theory. Thus, Higher Education as a disci- pline lacks those critical unifying attributes associated with other professional and scholarly fields. Professional training for college teaching has historically counted of little or nothing more than sub- ject matter mastery and training in research. Neither was much attention given to the problems of the higher education institutions themselves: their governance, organization, and financing, their responsibilities to the society and the culture, their role in national and international affairs. But gradually a new dimension in education emerged and has begun to show definite form. First by way of isolated courses, then in programs of courses, sometimes in separate departments, and in semi-independent insti- tutes and centers for the study of higher education, this new dimension has proceeded to widen its scope and strengthen its claim to attention and respect. This development has been relatively rapid and especially accelerated in the early 605 and 705 (Ewing, 1963). Here and there across the country in colleges and uni- versities sensitive and responsive to educational needs, the study of Higher Education has been instituted, either in modest, rudimentary form or in more sophisti- cated and elaborate structure. 13 The earliest of such efforts (to be treated in Chapter II) have been traced back to the 18905, when a few isolated individual teachers instituted courses dealing with the general subject of higher education. These, however, proved to be false starts, premature and identified in almost every instance with the interest and effort of a single individual. These pioneer courses vanished with their originators and appear to have borne little or no fruit. It was not until after the First World War that courses of permanence and continuity were established which are the true antecedents of the present- day courses in Higher Education (Young, 1952). Between 1920 and 1945, progress was steady and colleges and universities initiated instruction in Higher Education at the rate of about one additional institution each year. Following the Second World War, interest in higher education was unprecedented. And this pressure stimulated the efforts of colleges and universities to study and research into all facets of Higher Education that the institutions might be prepared for the future. These forces made possible the expansion of Higher Education as a field of study because of the underlying widespread belief in the value of higher education and of research on higher education. The period of the late 19503 and early 19605 appears in retrospect to have been almost euphoric in this 14 regard: people expected higher education to be able to solve the most vexing social problem and research on it to produce the answers needed to do so. If American higher education is in its decade of decision, prominent among the reasons is the growing problem of an overabundance of faculty and administrative personnel to meet the present and projected enrollments. Higher Education, as an integral part of American higher education cannot be exempt from this national concern. Clark (1974) suggests that identifiable charac- teristics and indicators be deve10ped which can be used for program self-study and improvement, for the benefit of prospective students and for program evaluation by appropriate outside groups. In the matrix of serious considerations and questions concerning the status of graduate education and Higher Education in particular, the basis for this study was found. Statement of the Problem Two problems confront the assessment of excellence in graduate education. First, ascertaining the appropriate criteria for excellence; and second, quantifying the cri- teria so as to permit comparisons among programs. The selection of criteria for excellence is by no means an easy task. It is plagued both by political and conceptual difficulties. For obvious reasons, indi- viduals and organizations favor criteria which focus on 15 their own strengths. Since institutions compete for students, faculty, and funds, a commonly accepted defi- nition of excellence has political complexities aside from conceptual ones. Not withstanding the political problems, even a dispassionate, disinterested observer has serious difficulty selecting a set of noncontradictory criteria. In order to establish criteria for excellence, program evaluators first must decide what they value. From their values they then fashion objectives, which in turn, establish the criteria for excellence. Criteria vary with the objectives sought. For example, evaluators whose primary objective is the production of new knowledge establish programs in terms of faculty and student scholarly output. They will rate a program with a distinguished faculty which produces much valuable research and a few outstanding, research-oriented scholars much higher than evaluators who hold the train- ing of college teachers as the primary objective of graduate programs. Appropriate criteria for the second evaluator might emphasize the numbers of qualified col- lege teachers produced by a program rather than the quality of faculty and student research. The objectives of a program, of course, are rarely unidimensional. Furthermore, different objectives usually are neither mutually exclusive. 16 For example, it may not be possible to maximize both the generation of outstanding scholarly work and the production of competent college teachers. However, it may be possible and desirable to maintain an optimal mix by balancing competing values in single institutions or by seeking different objectives in different programs (Warren, 1967). Ideally the criteria established for evaluation of graduate programs will be formed by a conscious review of all relevant objectives and a con- scious weighting of those objectives on the basis of an hierarchy of values. The established value hierarchy, and consequently the criteria for evaluating graduate programs, will vary with the perceived needs of the institution, state, or nation. Certainly it is possible for graduate programs emphasizing somewhat different objectives to attain excellence and to receive due rewards. After the basic task of selecting explicit cri— teria has been completed, the second critical problem in evaluation is encountered. What is a valid measure of relative degrees of excellence on a stated criteria? For example, what is a valid measure of a faculty's scholarly abilities? Is it peer evaluation, or research grants, or number of publications? 17 What is a valid measure of departmental effec- tiveness? Is it alumni salaries? Placement of graduates? Ph.D.‘s per faculty? While most of the techniques necessary for evalu— ating graduate programs have already been developed or can be developed relatively easily, problems of measure- ment cannot be dismissed lightly. However, where measure- ment difficulties exist, the weakness of one technique may be offset by the strengths of another. Hence, by utilizing several measures of effectiveness, a relatively comprehensive and valid evaluation may occur. Each of the three elements of the evaluative process--objectives, criteria, and assessment indices-- have unique problems. But the most critical prerequisites of successful evaluation are that each element of the process be clearly defined and the relationships between objectives, criteria, and indices be logical and explicit. If these conditions are not met, the evaluative process creates unnecessary confusion and controversy. While the study of Higher Education has not yet reached maturity, it possesses many of the characteristics of a scholarly field. Programs have been designed to prepare persons for entry into professional work in higher education; faculty are engaged in service, instruction, and research; and departments or centers have been formed to sponsor research and offer degrees. 18 There is an ever-growing quantity of literature about all parts of the Higher Education domain, and numerous organizations have arisen that produce publications and new knowledge about Higher Education. With Higher Education clearly a field of study in this sense, it is desirable to identify the charac- teristics of a graduate program in Higher Education with those disciplines which have evolved in reasonably dis- tinctive fields of study and scholarly activity. Yet there has been little systematic effort to develop pro- cedures to evaluate quality and to provide reliable information about various aspects of Higher Education as a graduate study. If the standards of the colleges and universities are maintained at present levels, it is apparent that graduate schools must plan programs which are reputable and at the same time designed to attract and train quality students. Although academic communities are quick to establish guidelines and offer suggestions for improving conditions outside their walls, there is great reluctance for self-evaluation. Yet such evaluation is necessary, perhaps not to eliminate weak programs, but to offer specific suggestions for improvement. Evaluative cri- teria can be viewed as a basis for a taxonomy of estab- lished and emerging Higher Education departments, but such a use would be peripheral to the major purpose in 19 developing them, for one could certainly never justify spending much time discussing them if their only function were to serve as categories. The primary purpose would be to serve as standards against which departments assess themselves. The mention of evaluation all too often engenders some negative reactions such as fear that guidelines will lead to unjust ranking of departments against some arbitrary hierarchy. Such reactions, while understandable, emphasize only the abuses of guidelines and overlooks an important positive use of them--that of aiding a department to grow, develop, and attain its objectives, through an on- going self-evaluation. The primary goal of criteria should be formative, by providing information with which departments themselves can select their objectives, analyze strengths and weaknesses, devise and develop strategies, and assess progress toward the objectives chosen. Purpose of the Study As part of this study, a review is made of the history of the development of Higher Education as an area of scholarly study and research in American colleges and universities. In addition, those institutions in the Big Ten which currently offer instructional courses and graduate degree programs in the area of Higher Edu— cation are identified. Since the identification of 20 characteristics related to quality in graduate programs in Higher Education is the centrality of this study, those Big Ten institutions offering such a graduate program are surveyed to develop criteria for evaluation of quality. Acknowledged experts in the field of Higher Education are also included in the survey to provide an extra-institutional viewpoint. The study is designed to provide information, based on the extent of agreement among department chairmen and recognized authorities, about doctoral programs characteristics most important .to judgments about quality. Definition of Terms Certain terms are defined in order to clarify the concepts represented by the terms as they will be used in this study. higher education.--This term refers to the broad area of education beyond high school; and any and all kinds of collegiate level post-secondary school training. As used here it is not specific, referring simply to level and not to kind. Higher Education.--This is a particular segment of higher education; that particular field of study and/or research where interests center on and are restricted to the affairs, activities, and problems associated exclusively with community colleges, senior colleges, 21 and universities. The term is used to designate the study of these higher education institutions as edu- cational enterprises, as teaching, learning and research organizations, and as social institutions, which leads to a master's degree, educational specialist, or other two-year certificate or degree, or doctorate whether oriented toward teaching, service, institutional research, or scholarship. Big Ten Institutions.--This term refers to the following institutions: University of Illinois (Urbana- Champaign); Indiana University (Bloomington); University of Iowa (Iowa City); Michigan State University (East Lansing); The University of Michigan (Ann Arbor); Uni- versity of Minnesota (Minneapolis); Northwestern Uni- versity (Evanston, Illinois); Purdue University (West Lafayette, Indiana); The Ohio State University (Columbus); and University of Wisconsin-Madison (Madison). Center or Institute for the study of Higher Education.--This term refers to a special organizational entity or agency somewhat separate from or semi-indepen- dent of the School or College of Education or the uni- versity to which it is attached. It is generally not as exclusively involved with the teaching aspect as is the department, nor is its organizational structure parallel to that of a department. Centers or Institutes 22 generally intend to serve both the parent university and educational agencies outside the university. Research is an integral goal; instruction may or may not be a prominent activity. Instructional course.--This refers to a formally organized and structured course, as listed in a college or university bulletin or time schedule, and for which hours or units of credit are recorded for those who register for and satisfactorily complete the required work. Generally, in an instructional course as used here, it is thought that students shall meet regularly at scheduled times in classes or groups for instruction or discussion, rather than for purely instructional direction. However, directed individual study courses, or independent study courses, are included. Theses and dissertations are excluded from the category of instruc- tional courses. General course.-—This term refers to groupings or sequences of courses where it is presumed that a single course does not offer adequate coverage of a total subject area for anyone presuming seriously to study the subject. A number of more or less related, but generally discrete, nonduplicating courses, each of which treats some facet of a total subject, and when all courses comprising the program are considered, a reasonably complete coverage of a subject is intended. 23 Graduate Program(s).--As defined for the pur- poses of this study, refers to programs leading to degrees beyond the master's degree level. Limitations of the Study The author realizes that there exists within the country many quality graduate programs in Higher Education with wide and diverse offerings. Many have achieved national and international recognition. Yet they cannot be considered since this particular study considers only those graduate programs in Higher Education offered at institutions who are members of the Big Ten. A further limitation is that the study will not be evaluative. No attempt is made to evaluate the pro- grams at the participating institutions, only the criteria for evaluation is determined. Design The study, employing a modified Delphi Method, was designed as a three-stage mail survey to the eight institutions of the Big Ten who offer graduate programs in Higher Education; and to seven recognized authorities in the field. The first Opinionnaire asked each par— ticipant to rank the various characteristics in order of importance. The second Opinionnaire contained all the characteristics of the first and a modal ranking for each. The participants again ranked the 24 characteristics and provided reasons why their rankings differed from the modal rankings. The third and final stage of the survey contained only the top five char— acteristics and the opinions given. The participants were again asked to rank the characteristics in order of importance. Frequency tabulations were compiled on the importance of characteristics. In addition, mean ratings were computed so that program characteristics could be arranged in order of their rated importance. As a background for the study, literature in the field of graduate study and Higher Education specifi— cally was reviewed to determine the history of graduate education and Higher Education, its scope, present status, and projected future. Program characteristics listed under each heading were identified through this review. Helpful materials include: Barak (1973); Berelson (1960); Blackburn and Lingenfelter (1972); Brown (1970); Cartter (1966); Clark (1974); Dressel and Mayhew (1974); Ewcing (1963); Har— cleroad (1972); Heiss (1970), and Higgins (1968). The characteristics were then grouped under five major head- ings: Students, Personnel, Program, Finance, and Facilities. Participants in this study are those institutions in the Big Ten which offer graduate programs in Higher Education. In addition to representatives from these 25 institutions, a selected cross-section of recognized authorities in the field of Higher Education agreed to participate. In order to gather data to be used to determine criteria for an effective evaluation of graduate Higher Education programs, an Opinionnaire was designed and sent to the chairmen of Higher Education departments at the following institutions: (1) University of Illinois (Urbana-Champaign); (2) Indiana University (Bloomington); (3) University of Iowa (Iowa City); (4) Michigan State University (East Lansing); (5) The University of Michigan (Ann Arbor); (6) University of Minnesota (Minneapolis); (7) The Ohio State University (Columbus); and (8) Uni- versity of Wisconsin-Madison (Madison). The same Opinionnaire was sent to recognized authorities in the field of Higher Education, soliciting their participation in this study. Those experts who participated were: Patricia Cross, Paul Dressel, Lyman Glenny, Fred Harcle- road, Algo Henderson, Lewis Mayhew, and Dyckman Vermilye. Essentially, the polling method consisted of repeated sampling of the opinions of the administrators and authorities regarding their perceptions of what cri- teria should be used in evaluating graduate Higher Edu- cation programs. The participants did not, in the con- text of this study, meet face-to-face. Instead they completed opinionnaires and submitted them to the author 26 by mail. The avoidance of face-to-face discussion was purposeful. In addition to the economic burdens and limitations, face—to-face communications create serious problems in attempting to achieve convergence of opinion, as previous research has revealed. Among the problems are the following: 1. Dominant individuals tend to control the dis- cussion and have greater influence on group opinion than their knowledge might necessarily warrant. 2. Too much discussion time often is devoted to irrelevant or biased views of individuals. 3. Individual judgment can be distorted by the presence of others, lessening the reliability and usefulness of the response. Still, it is considered necessary to provide some controlled interaction among the participants. Each of them needs to know the opinions of others and to consider them independently in order to derive an informed opinion. A modified form of the Delphi technique, developed by the RAND Corporation as a means of obtain- ing greater consensus among experts dealing with problems of national defense, was selected as the basic procedure for this study. 27 The Delphi technique consists of having each participant complete a series of questionnaires inter- spersed with controlled feedback on the responses of the other participants. In addition to the advantages already mentioned vis-a-vis face-to-face meetings, this method provides anonymity to the participants, thus minimizing the influence of personal and political interests on their decisions. As originally developed, the Delphi technique consists of four steps: 1. Each participant is asked to write his opinion on a specific topic. Each participant is asked to evaluate all the opinions in terms of a given criteria. Each participant receives the list and a summary of the responses, and if his views differ from the most frequent responses, he is asked to revise his opinion or to indicate his reason for not doing so. Each participant receives the list with an up- dated summary including minority opinions and is asked to repeat or revise his own opinion. A prime value of the Delphi technique is that it preserves the virtue of independent thought but simul— taneously permits the participant to draw on the knowledge 28 of others. In other words, the technique assures that opinions arrived at independently will nevertheless be informed opinions. Polling of the participants began when the instrument was mailed to each of them. The first opinion- naire contained instructions to complete and return it within fourteen days. The criteria indicated were separated, categorized, and analyzed to determine the modal ratings, or those most frequently stated. Thirteen forms were returned for a total response rate of 87 per- cent. After the initial analysis was completed, a second Opinionnaire was mailed. This contained all responses given on the first and the modal responses. Instructions to the participants indicated explicitly that they were not to be concerned with their own previous responses. They were asked to note the modal responses and, if their responses differed from the modal responses, they were to indicate, if possible, one or two reasons for their choices. After these opinionnaires were returned, the modal responses were calculated again for each of the criteria. Additionally, minority reasons for differing from the modal responses were summarized. These reasons were listed under two headings, one for those contending a criteria should be more important than the modal rating indicated; the other for those contending a goal should 29 be less important than the modal rating indicated. Four- teen completed forms were returned for a total response rate of 93 percent. A third and final Opinionnaire was sent to the participants. Modal responses were again indicated. Participants were asked to place the opinionnaires side by side so they could see both the modal responses and the reasons some participants did not agree. Then each participant was asked to indicate for the last time his criteria for evaluation and return the opinionnaire. A final summary was made. Fourteen were returned for a total response rate of 93 percent. Overview of the Study Chapter I contains the introduction to the study, the statement of the problem, a discussion of the objec- tives of the study, limitations of the study, definitions of terms, and organization of the study. Chapter II presents a review of the literature and history of graduate education in the United States and the emergence of Higher Education as a field of study. This chapter will discuss the early and later history of Higher Education, the problems and issues facing graduate education, and some prospects and needs for self—study and improvement. 30 Chapter III is concerned with the design of the study, the population of the survey, and the instrument used. Chapter IV contains an analysis of the information supplied by the participants surveyed. Chapter V contains a summary of the findings, some conclusions, recommendations, and suggestions for further research. Summary Universities are remarkably flexible and resilient organizations. But conflicting demands on their resources and financial stringency have produced serious new stresses within them. In the past, these institutions were capable of growing in many directions without having to assess mission or scope and without being specifically accountable to funding agencies, the public, faculty, or students. That period has ended, and universities increasingly are being asked to justify themselves. The problems of identifying and measuring the components of such complex organizations or of analyzing and evaluat- ing their performances are enormous. These problems are complicated by uncertainties about how to identify and demonstrate the quality and quantity of education, research, and public service. 31 The accomplishments of a university--in the large, and in the long run--are the distinctive con- tributions of its graduates and its faculty to the world of ideas, expression, and action. A university that is doing what it should in new scholarship is a stimulus, shelter, and testing ground for innumerable individual and collective creative efforts in science, intellectual invention, critical insights, and the arts. Such an institution will be well represented in the networks throughout the world for sharing new findings and new approaches in the scholarly disciplines and the pro- fessions. Although institutions need to keep these long— term issues very much in mind, they must concentrate attention on institutional and scholarly processes, resources, and short-term indicators of a university's position and prospects. Because ultimate results are so long-range and so difficult to assess, an operational approach can examine the view currently held about the university and its academic quality by those whose decisions affect it. From these indicators, a university can and must know where it stands with respect to academic quality as well as quantity. Finally, a qualitative study based on a single department or discipline rather than a university-wide reputation serves many practical purposes. It little 32 helps deans or department chairmen to be told that they rank third, eleventh, or sixty-seventh; it may help them a great deal to have a rough indication against which to view their relative strengths. The present study does not pretend to reflect all possible aspects of eminence or its absence, but it provides an approximate View by judges who are intimately involved with programs both as teachers and as administrators. The first step in the improvement of an institution or a department is self-knowledge; such knowledge can be extremely useful even though it may not be flattering. It is hoped that the information from this survey will be useful in strengthening graduate programs in Higher Education. CHAPTER II REVIEW OF THE LITERATURE Introduction The purpose of this study is to identify cri- teria for evaluating graduate programs in Higher Edu- cation at Big Ten institutions. This was accomplished by determining the extent of agreement among department chairmen at those Big Ten institutions which offer the graduate degree in Higher Education as well as the opinions of seven recognized experts in the field of Higher Education. In addition, a history of the growth and development of Higher Education as a field of study is included. Given these two purposes, the review of literature has been divided into two general sections: (1) the materials concerning evaluation of graduate education in general and Higher Education specifically; and (2) the growth and development of Higher Education as a disci- pline in colleges and universities in America. 33 34 Criteria and Assessment Techniques for Graduate Education The desire for high quality as the goal of every Ph.D. program needs no defense. Yet different tasks for different disciplines suggests different dimensions for consideration in developing procedures to assess quality. Reputational Studies The most widely known, heralded, and criticized evaluation of graduate programs has been the reputational studies. Hughes (1925) conducted the first of these in the 205. He replicated his study ten years later for the American Council on Education. In that study he clas- sified graduate departments on the basis of their adequacy and their distinctiveness. In a study in 1957, Keniston asked department chairmen to rate the relative positions of twenty-five major universities with respect to the quality of their graduate programs. More recently, Cartter (1966) repli- cated and expanded the Hughes and Keniston surveys in a study initiated by the American Council on Education and supported conjointly by the National Science Foun- dation, the National Institute of Health, and the U.S. Office of Education. Cartter's survey included the assessment of twenty-nine departments in 106 graduate institutions by nine hundred department chairmen and approximately three thousand faculty. 35 In a 1959 study, McGrath expressed much skepticism over the contributions of the graduate school. He felt that graduate education had become more important than undergraduate study with the eventual specialization creeping into the liberal learning normally associated with the undergraduate years. In addition, graduate education with its emphasis on research would weaken the significance of and respect for teaching. Berelson's study of graduate education published in 1960 covers the history of graduate study, offers an analysis of the present, and projects future trends. He recommends several major changes in the degree program. Also, he contends that quality in graduate education must be maintained and that it is in danger with the increasing number of institutions embarking on graduate study. He concludes that the major responsibility for graduate study should stay in the presently established institution of the top and middle prestige rankings. His examination covered much of the literature about graduate education; employed some independent studies by himself; and, on the whole, determined that graduate education was in good health. Carmichael (1961) offers many criticisms of graduate education and some recommendations for its improvement. The Carmichael study does need to be considered with one caution: it was made in 1961 when 36 there was a shortage of college teachers; therefore, many of the recommendations deal with ways to relieve the shortage. He feels that the goals and purposes of graduate education have not been clearly defined and the students suffer from this lack of definition. Respondents in the Cartter study were asked to indicate which among six given terms in their judgment best described the quality of the graduate faculty in their field in each of the institutions in the sample. They were also asked to rate the effectiveness of the doctoral program in their field in each of the insti- tutions by indicating which of the terms best described the competence and accessibility of the faculty, the curricula, the educational and research resources, and the quality of their graduate students. Using numerical ratings for each of the descriptive terms, Cartter drew up tables of the leading departments by rated quality of the graduate faculty, and of leading departments by rated effectiveness of their graduate programs. Higgins (1968) effectively summarizes the history of graduate school assessment in his unpublished rating of doctoral programs in education. Heiss (1970) in her book, Challenges to Graduate Schools, sought to study the components of excellence in graduate education. Information was obtained from graduate deans, academic deans, department chairmen, 37 faculty and graduate students. They were asked to com- ment on issues in the areas which most affected them. Roose and Andersen, sponsored by the American Council on Education in 1971, developed one of the most methodologically sophisticated and widely published studies of graduate education. This study is a repli- cation of the Cartter survey in methodology, but includes more disciplines, institutions, and rates the changes taking place during a five-year span. Included were thirty-six fields of study in 150 institutions granting the doctor's degree. The ratings compared faculties of the top 50 graduate institutions with the 80 institutions surveyed. It also presented the distribution of ratings by disciplines as well. The survey also categorized 50 institutions along geographic distributions to determine if any section of the country was deficient in quality graduate programs. It recommended that greater care be taken in accepting students into doctoral programs and a watchful eye be kept on the ever-increasing number of doctoral programs established and degrees offered. Millman and Toombs (1972) collected and organized the basic data of their study so as to give comparisons that had special significance for the state of Pennsyl- vania. The same data from Pennsylvania are compared with five other states and also with the top-ranked schools in the study. It is concerned with the 38 relationship between enrollment and ranking, the number of fields and ranking, and resident student enrollment and ranking. In 1972 the Board of Regents of New York State appointed a Commission on Doctoral Education which was charged to make recommendations for developing policy to meet present needs and to guide the future development of doctoral education in the state of New York. The Commission recommended the following objectives guide graduate education: (a) maximum quality on an insti— tutional and statewide basis; (b) maximum economy, efficiency, and effectiveness in the use of the resources for doctoral education on an institutional and statewide basis; and (c) equity in access to doctoral education for all doctoral students. Blackburn and Lingenfelter (1972) in a report prepared for the Regents of the State University of New York addressed what they felt were the critical problems in the assessment of excellence in doctoral programs: the determination of what criteria to measure excellence; and the difficulty of dealing with the information in order to make comparisons among the various programs. In a 1974 report, funded jointly by the Graduate Record Examinations Board and Educational Testing Ser- vice, Clark asked sixty-three graduate deans to rate the importance of many program characteristics to 39 judgments about quality, and then to rate the adequacy of several possible measures of the most important char- acteristics. The survey gave special attention to several different program purposes, such as the training of scholarly researchers, teachers, or practitioners of other kinds, and the differences between academic disci— plines. Focus of the survey was on identifying the kinds of information expert graduate educators thought would be most helpful in assessing program quality. It seems that peer evaluation characterizes all the reputational studies. A panel of scholars rates the quality of an institution's faculty and/or graduate pro- gram in a given discipline. By combining individual panelists' responses, an aggregated rating of the graduate program is calculated. Although the criterion of excellence which domi- nates reputational studies is the scholarly ability of a faculty, the technique of peer rating can be used to obtain measures of other criteria. For example, in addition to scholarly ability, studies use "program effectiveness" as a criterion. Other criteria, such as "supportiveness of graduate students," "effectiveness of teaching," or "student quality" could be used. However, since widely scattered outside evaluators are less likely to be aware of conditions pertinent to such criteria than they are of a faculty's scholarly output, 40 reputational assessments or other criteria cannot be defined as easily as can assessments of scholarly ability. Some Objective Indicators of Excellence Several observers of graduate education have selected one or more objective characteristics as bench- marks of excellence. Some objective characteristics, labeled "correlates of quality," have been established by examining various programs rated in the American Council on Education studies and distilling a cluster of objective traits associated with high quality ratings. Other evaluators have selected indicators on an a priori basis, defining quality in terms of certain characteristics and proceeding to develop an index of those characteristics. Calvert, et a1., (1971) combines several objec- tive indicators to construct a general index of quality. Scholarly productivity. Stallings and Singhall (1971) use scholarly output as an important criterion of excellence. Their index assigns weights to books, articles, reports, patents, etc.; and a productivity score is calculated by aggregating the quantity of credits in each category. Several researchers have found a relationship between the most prolific departments and those receiving 41 top rankings in reputational studies (Berelson, 1960; Cartter, 1966; Crane, 1965). Blackburn (1972), however, contends that productivity changes with time. He con- tends that since a number of variables, such as age, rank and tenure, affect performance, the development of a productive faculty mix must take into account the relevant factors. Clark (1957) used scholarly productivity as a measure of program quality, by requiring that other scholars utilize a piece of work, and calculates the number of citations in subsequent scholarly publications. Citations of work that is more than ten or fifteen years old are given extra weight on the assumption that dura— bility is an indicator of unusual quality. Margolis (1967) discusses the strengths and weaknesses of the citation index in great detail. Degrees, awards, and other faculty traits. Some observers have used other faculty characteristics as indicators of excellence. Bowker, in 1965, examined the percentage of faculty with the Ph.D., years of experience, the percentage of instructors teaching graduate students, the percentage of faculty publishing in the past five years, the percentage publishing on subjects other than their Ph.D. dissertation topic, and the percentage of foreign specialists who have traveled abroad. 42 Student quality. Another technique of evaluating doctoral programs has been to measure the quality of students enrolled in a program. In one sense this is a reputational measure, because faculties tend to select the best possible students and good students are attracted to programs with a reputation for quality. In another sense, however, well-qualified students are an essential element of an excellent program. Thus student quality can stand in its own right as a criterion of excellence. Perkins and Snell (1962) compared history graduate students to students in other disciplines on the basis of undergraduate grade point averages, I.Q. scores, and Graduate Record Examination scores. These indices can easily be used to compare students in different doctoral programs in the same discipline. The distribution of Woodrow Wilson fellows among graduate programs was utilized by Bowker (1965) as an index of quality. Physical facilities. An obvious facility required for doctoral programs is the library. Several researchers-—Perkins and Snell (1962), Jordan (1963), and Cartter (1965)--found that highly rated institutions have larger libraries and spend more per student on librarian salaries. No systematic studies of other physical facilities for doctoral education (laboratories, office space, 43 computer capabilities, seminar rooms, etc.) exist, possibly because these are rarely designed solely for graduate instruction. However, evaluation of such facilities is appropriate in the assessment of new or existing programs. Since the facilities required differ among disciplines, specialists must be utilized for the evaluation of physical facilities. Attention should be given to four evaluations that were conducted by outstanding institutions. Uni- versity of California-Berkeley by Heiss in 1964; Harvard University by Elder in 1958; Columbia University by Barzum in 1968; and University of Minnesota by Alciatore- Eckert in 1968. In these studies the institutions con- front themselves publicly on the effectiveness of their graduate programs. In an effort to discover the standards upon which institutions of higher education were judged, Hatch (1964) examined the literature on file in the Clearing House of Studies in Higher Education and found the following factors listed as indices of outstanding institutional quality: the institution is disposed to make a dis- tinction between the acquisition and the examination of knowledge; it provides adequate learning resources, jealously guards academic freedom, rewards good teaching, administers its counseling program for institution—wide impact, and performs its institutional research on 44 important matters. The program is characterized by its flexibility, permissiveness, openness to experimentation, uniqueness, provision for independent study, and high but attainable goals. The course work challenges the students to develop their own initiative, develops their critical faculties, recommends extensive reading, requires a large block of out-of—class study time and offers little instruction labeled as remedial. Bissel (1968) cites four characteristics which he considers to be earmarks of quality in a university. The institution is a stronghold of scholarship in the pure theoretical subjects that lie at the basis of any expansion of knowledge. The great university has graduate and undergraduate divisions that are both strong. The great university maintains a balance between its long- range goals and its short-range obligations, or between its obligation to pure scholarship and its obligation to the society of which it is a part. Client Satisfaction Ratings An obvious means of evaluating an enterprise of any kind is to ask those it serves for their opinions. The consumers of doctoral education are students and the employers of graduates. Both of these groups have been utilized in evaluative studies. Alciatore and Eckert asked a group of Ph.D. recipients to evaluate their training at the University 45 of Minnesota. Their opinion of their preparation for tasks such as teaching, research, etc. was solicited as well as their general satisfaction with their career and the doctoral program that launched it. A similar study was conducted at Florida State University in 1957. Such surveys of student opinion are most valuable when they include both current and past, successful and unsuccessful students so that a wide range of opinions is included in the study. Student opinion surveys have been a part of several major studies of graduate education that have been previously cited. Berelson (1960) surveyed 3,843 recent doctoral recipients in order to gain information about their experiences and opinions about possible reforms. Heiss conducted her extensive surveys of student opinion at ten major graduate schools and devoted much of her study to the analysis of their suggestions and complaints. Tucker's 1964 study of attrition and Gotlieb's 1961 study of graduate student socialization have used graduate student or alumni opinions as explicit evalua- tive instruments. Graduate and professional education reform is discussed by Mayhew and Ford (1974). The authors, con— cerned with changes and innovations in both professional 46 and graduate education, examine attempts to change or reform both schools. Problems that have plagued graduate study for years are mentioned to serve as springboards for consideration of issues never before faced until now. Yet, in providing guidelines for change, the authors suggest that systematic evaluation and self-appraisal are necessary for continued growth. The preceding review indicates that assessment of quality in graduate education has been the source of much concern. There has been, however, and continues to be, great ambiguities and subtleties of concept surrounding quality, and the proper procedure for its assessment. Yet quality does exist, thus its assessment, however difficult, is of central importance to graduate education. Higher Education as a Field of Study Higher Education as field of study has had a short history but a long past. A number of studies have been done in the area of the development of the study of Higher Education. None furnishes the historical perspective that is accomplished by Burns Byron Young in his dissertation at Stanford. Young (1952) traced the historical development of the first courses in Higher Education up to and including the development of departments of Higher Education. Young had no 47 previous studies on which to build and found most of his data in original documents, letters, reports, minutes, and college and university bulletins. The Young study clearly illustrates the difficulty of piecing a puzzle without all the pieces. The develop- ment of the study of Higher Education is not well docu- mented. The pioneers in the field were concerned about training for college and university teachers and not training to prepare academic administrators. A second concern, and a significant factor in the development of the study of Higher Education, was to study and research institutions of higher education as social, political, and economic organizations. The Young study is related to the present study in that it outlines the historical origin of instructional courses in Higher Education and the development of Higher Education as a field of study. Young pointed out that the first course in Higher Education was offered by G. Stanley Hall at Clark University in 1893. Later, another course entitled "Organization of Higher Education" was offered by Dean James, College of Education, University of Minnesota, 1908-1909. Following these sporadic offerings, regular course work to provide professional preparation for careers in college administration was started in 1920 48 at the University of Chicago, Ohio State University, and Teachers College, Columbia University. No doubt in these early beginnings, daring to offer courses in something called "Higher Education" must have taken some courage and a pioneer spirit. Ewing (1963) investigated the proliferation of graduate courses in Higher Education at an increasingly large number of institutions. Included in Ewing's study were the institutes and centers for the study of Higher Education. His research was the first historical review of the development of Higher Education as an area of scholarly study and research since Young. When Ewing conducted his study in the academic year 1962-63, eighty-seven institutions were offering 560 courses in the field of Higher Education as compared to thirty- one institutions offering 110 courses at the time of Young's study. Ewing classified the content of the courses that were being offered for graduate study as well as those courses in interdisciplinary programs. Ewing in his study stressed the weakness of using catalogs as a source of obtaining data. Some differences do exist between Young's and Ewing's dating on the establishment of Higher Education programs at specific institutions. Among Ewing's conclusions were that Higher Education was reaching an equal status with elementary and secondary education; and that organization 49 and administration was one of the important subject matter areas within the field. McGrath (1969) commented, "Until very recently the guild of practitioners in higher education could hardly satisfy the criteria used to determine whether a particular vocation could properly be classified as a profession" (p. 2). He suggested that formerly those who wanted to develop expertise in teaching, administration, or research had to acquire these skills through an appren- tice system in much the same way that one became a phy— sician before medical schools were established. Sagan (1969) in reviewing the literature concerning the history and development of Higher Education observed: It is surprising that higher education has waited so long to talk about itself. Perhaps it wished to wait until its impact was truly profound. Perhaps it wished to wait until its influence could be traced to almost every major advancement and to every portion of the national structure. (pp- 1-2) Ewing and Stickler (1964) stated: Interest and activity in the discipline has increased at an accelerated pace as the need for better trained college teachers and administrators intensified in response to political, economic and social forces. From about 1956 on, the appearance of institutes and centers for the study of higher education has added what may--at least in some instances--prove to be an important new agency for study and research in the field. (p. 401) Higher education has gained stature and is now taking its place alongside other recoqnized fields of edu- cation. National attention to higher education as a field of study is constantly increasing. (p. 402) 50 Currie (1969) reviewed graduate school catalogs for 1966-67 and found that at least 106 universities were offering courses related to one or more of the sub— areas of Higher Education. By 1968, there were approximately 335 faculty members teaching one or more courses in Higher Education, according to a list provided by the American Association for Higher Education (1968). The "List of Faculty Members Teaching Courses in Higher Education," which was distributed by the American Association for Higher Education, included a total of 698 names, representing 166 colleges and universities. An analysis of this listing showed that each of six institutions-—Columbia University, Pennsylvania State University, Southern Illinois University, University of California at Berkeley, University of Minnesota, and University of Wisconsin listed sixteen or more faculty teaching in Higher Edu- cation. The range was from one to twenty—four. Dibden (1965) established a rationale for formal programs in Higher Education: intellectual curiosity and educational maturity, the inherent and practical importance of higher learning, and the commitment of academic man. He also identified five problems which are characteristic of departments in this area. He indicated further some possible ways in which such a department could render important service to the university. 51 Overholt (1967) surveyed 151 institutions to determine how many had a department, center, or insti- tute studying Higher Education. Of the 121 replies, 31 had such a department, 3 were establishing one, 81 taught courses in this area, and 64 indicated a research center for the study of their own institution or Higher Education in general. In an attempt to document the incidence and scope of offerings in Higher Education, James F. Rogers (1969) sent a questionnaire to 180 institutions; 137 responses were included in his study. Of the 86 programs reported, including 53 that offered areas of major concentration at the doctoral level, 84 of them employed 468 faculty and offered 889 courses; 49 major programs had an enrollment of 2,174 graduate students; 44 minor programs enrolled 842 students at the doctoral level; 37 major programs awarded 316 doctorates; and 27 minor programs had 354 doctoral recipients in 1967-68. Higgins (1971) provided some unique data to support the growing status and recognition of Higher Education as a field of study. He compared two studies undertaken by himself in 1968 and 1970. With the data received, he was able to compare for the first time ratings of five doctoral areas in education, including Higher Education, at the fifteen top ranked graduate schools in the United States. Higher Education had 114 52 faculty names submitted with a total of fifty-seven doctoral programs, which was the smallest number in both categories for the five areas. On a geographical basis, Higgins found that Higher Education programs were evenly distributed among the East (14), South (15), Midwest (15), and the West (13). In comparing the results of the 1968 study in Higher Education with that of 1970, Higgins found that the highest ratings in the former were the University of California at Berkeley, The University of Michigan, Columbia University, and Stanford University; this order for 1970 was The University of Michigan, University of California at Berkeley, Stanford University, University of California at Los Angeles, and Columbia University. One of the values of this study is that it isolated and identified administration in Higher Education as a spe- cific field in professional education. Orr (1971) commented that recently at the Uni- versity of Alabama the decision was reached that one option for meeting the foreign language requirement for the doctoral program was that of scheduling fifteen semester hours in Higher Education. Fletcher (1972) added that in terms of a recent decision at the University of Nevada at Reno seventeen semester hours in Higher Edu- cation may be used as one option for meeting the foreign language requirement. 53 The Program of Study If Higher Education is a specialized field of study with a documented history, there should be specific objectives and goals, related courses, and the graduates of these programs should occupy college and university positions doing what they were trained to do. There is general confusion or lack of agreement about the goals and objectives for Higher Education pro- grams just as there is confusion about admissions. How— ever, this is not surprising. The Chronicle of Higher Education (1972) pointed out that a meeting of the Modern Language Association gave strong indication that English as a discipline is in a process of marked change and that departments vary in objectives, goals, and programs. Williams and Richman (1971) reported on a survey of graduate preparation in psychology. The department chairmen indicated that teaching ability and research ability were important competencies, yet only approxi- mately one-fifth to one—sixth of the graduate students were provided training in these competencies. Burnett (1972) observed from reading graduate bulletins and in conversation with colleagues, that three general career objectives emerge in graduate training programs: administration, teaching, and research. Some programs in Higher Education indicate a principal function, while mentioning related ones. 54 Institutions have developed specialized programs probably in response to needs within their respective states; such as the preparation of Junior or Community College adminis- trators or faculty in states where these institutions occupy an important place in the statewide system of higher education. Overholt (1967), from a total of 121 institutions, found the following seven areas of course offerings: (1) Junior or community college (2) College and university administration (3) History, philoSOphy, and issues in higher education (4) Preparation of teachers and curricula (5) Preparation of student personnel workers in higher education (6) Research in higher education (7) Miscellaneous Rogers (1969) established somewhat different program areas on the basis of sixty-nine institutions reporting major and minor programs and eighty—four institutions that reported courses in subspecialties. He reported the following: (1) Student personnel work (including administration) (2) Academic administration 55 (3) Administration of business affairs (4) General administration (5) Higher Education (general) (6) College teaching (7) Junior college (including administration) (8) Teacher education (9) Other (primarily adult education) Waldron studied doctoral programs in Higher Edu- cation at forty different institutions. His inquiry centered around programs, admission requirements, courses offered, and financial assistance for graduate students. He found that these programs ranged in size from two to twenty-four courses with 12.9 courses as the average. The most common areas were General Higher Education and Student Personnel; Teacher Education and Business Affairs in Higher Education were least common. Unique in this study was Waldron's attempt to obtain follow-up infor- mation on graduates and the qualifications of faculty members offering courses in Higher Education. Ten of the institutions studied had comprehensive information about their graduates; all, however, indicated that graduates were being placed in the area of specialization for which they were prepared. 56 One of the most comprehensive studies of graduate programs in Higher Education was completed in 1970 by Palichak, et a1. One hundred and forty-one responses to 275 questionnaires provided the following: 1. Approximately 86 percent of the respondents indicated that they included administration, student personnel work, research, and college teaching in the field of Higher Education. Even a higher percentage (97) included the study of both two-year and four-year insti- tutions. Seventy-nine percent agreed there was a spe- cific body of knowledge in the field of Higher Education. Most of the respondents preferred courses be taught at the doctoral level; fewer than one percent preferred the master's level. The Ph.D. degree was preferred over the Ed.D. degree. The following careers for graduates were rated from high to low: administrators, student personnel workers, researchers, faculty for Higher Education programs, and college teachers. The greatest preference of program elements was shown for research problems, curriculum planning, internships, and student personnel laboratory/internships. In an attempt to gain information about members of the Association of Professors of Higher Education, Ross (1974) polled 200 members, of which 164 replied. While not specifically addressing the question of content for Higher Education programs, she did receive infor- mation about the specialties of each respondent. They are, in order of frequency: (1) Governance, administration (2) Community college (3) Curriculum 57 (4) History, sociology, philosophy (5) Student personnel (6) Research and evaluation (7) College teaching While these categories only reflect individual specialties, it is possible to conclude that these specialties have great impact on the overall content of programs of Higher Education. A similar conclusion can be drawn from research done by Carr (1974). This study was conducted to identify and analyze the educational and employment characteristics of doctoral graduates in Higher Education as a specialized area of study. The population included the 1963, 1966, 1969, and 1972 doctoral graduates in Higher Education from nine universities. Of specific interest are the subfields of study identified. They are, in order of frequency: (1) Academic administration (2) Student personnel (3) Community college (4) Curriculum and Instruction (5) Business (6) Institutional Research 58 Although reflecting only subfields of specific graduates at specific institutions, an inference to program content can be drawn. Travelstead in his 1975 survey of programs in Higher Education concluded that a diversity of program purposes exists, with research into higher education, and the training of administrators ranking first and second respectively. Other program purposes included: faculty preparation for departments of Higher Education; consulting; staffing for state, regional, national, and private agencies; and teacher education at the college level. In addition, he concluded that, although pro- lific, the study of Higher Education is theoretically not well defined nor are its programs. Assessment of Individual Institution Programs An investigation of the pertinent related litera- ture and research uncovered numerous follow-up studies; however, only a few studies have been conducted which seek to evaluate graduate programs in Higher Education at various American institutions. Lokers (1958) evaluated the doctoral program in Education at the University of Michigan, from the institution's viewpoint as well as that of the graduate. His conclusions supported evaluation as a continual process, especially for the institution if it is to 59 meet the various needs of its students as well as render a positive service to them. The doctoral program in Higher Education at Indiana University was reviewed by Broertjes in 1968. This study, based on a follow-up survey of graduates attempted to evaluate the quality and effectiveness of the graduate program and make recommendations for improvement. Brice (1970) surveyed the doctoral program in Higher Education at North Texas State University, Denton. The emphasis of the program of study is on college teach- ing and administration. He determined that the compe- tencies gained in educational research and statistics and the residency requirements are valuable. In 1972, Carl Blackwell conducted an evaluation of the doctoral programs of Florida State University. His study was a follow-up of previous study by the same institution and surveyed all doctoral students of the years 1952 to 1970. Armstrong (1974) at the University of Utah investigated doctoral training for college and university administrators in general; and with specific attention to the Ph.D. degree program at the University of Utah. His conclusions were based on a survey of the literature; catalogs and printed materials from other institutions; and a survey of 1973-74 students and all graduates of the 60 University of Utah. From these findings an evaluation of the program at the University of Utah was made, with recommendations for improving the areas of program, admissions, and financial assistance to graduate students. Graduates of the Higher Education degree program were surveyed by Randolph Manning in 1974 to document the establishment and development of the graduate degree program at George Washington University. Vandermeulen (1975) surveyed 135 graduates of thirty-five doctoral programs in Higher Education across the nation. His conclusions were mainly sociologically and psychologically oriented. He did, however, conclude there exists a definite lack of empirical research con- cerning doctoral preparation in Higher Education. Dressel and Mayhew (1974) combined their intuitive insights for the examination of Higher Education as a field of study. Every aspect of Higher Education was examined; the history and literature; a complete picture of various programs being offered; the components of programs; and problems, issues, and recommendations for Higher Education as it develops as a field of study. As of this date, this study represents the most encom- passing and comprehensive attempt to define the parameters of Higher Education as a discipline; and has proven an invaluable guide to this dissertation. 61 Summary The explorations and writings into the assessment of quality in graduate education has revealed concern that the quality of graduate study may not be advancing as rapidly as desired; some would say it is deteriorating. The focus and intensity of evaluation of graduate edu- cation has varied according to the concerns of the assessor. Two important reasons for concern about quality in graduate education run throughout the literature. First, the previous rapid growth in numbers of students and the present competition for students has exacted a heavy toll in impaired quality. The lecture, textbook, frequent examinations, grades, and all the paraphernalia of the undergraduate class has crept into graduate study. The second reason for concern is the recent pro- liferation of graduate study into hundreds of institutions which are often indifferently qualified in terms of faculty, equipment, library holdings, or scholarly tradition to offer graduate programs. In addition to the two problems confronting all graduate education, Higher Education faces the long and sometimes painful problem of establishing itself as an important field of academic study. Research has detailed that Higher Education has achieved significant proportions in current enrollments, 62 in awarded degrees, program offerings, in concentrations afforded students, and in numbers of faculty. Yet wide variations in program content and purpose exist on both the national and local perspective. CHAPTER III METHODOLOGY Introduction The identification of characteristics related to quality in graduate programs in Higher Education at Big Ten institutions is the centrality of this study. There— fore, the study is designed to provide information, based on the extent of agreement among department chairmen and recognized authorities, about doctoral program character— istics most important to judgments about quality. The study, employing a modified Delphi Method, is designed as a three-stage mail survey to the eight institutions of the Big Ten who offer graduate programs in Higher Education; and to seven recognized authorities in the field. The first opinionnaire asked each partici- pant to rank various characteristics about quality in order of importance. The second opinionnaire contained all the characteristics of the first and a modal ranking for each. The participants again ranked the character- istics and provided reasons why their rankings differed from the modal rankings. The third and final stage of the 63 64 survey contained only the top five characteristics and the opinions given. The participants were again asked to rank the characteristics in order of importance. Frequency tabulations were compiled on the importance of each characteristic. In addition, mean rankings were computed so that program characteristics could be arranged in order of their rated importance. In this chapter, the four steps by which this study progressed are delineated. These four steps include identifying and selecting the sample of this study, and the instruments used to gather information, as well as the methods of information collection and analyses. Procedure Step One Before proceeding with the investigation, a thorough search was made of the literature to determine that this research was not a duplication of the efforts of others. Also, the literature in the field of graduate study, and Higher Education specifically, was reviewed to determine the history of graduate education and Higher Education, its scope, present status, and possible future. Step Two A search of catalogs of Big Ten institutions revealed that all the institutions with the exception 65 of two offer the doctoral degree in Higher Education and therefore were included in this study. Institutions in the sample are listed in Appendix A (N=8). It was assumed that the department chairmen at these eight institutions would reflect, at least in part, the different philosophies and opinions about quality that might be associated with them. Also, the search of the literature revealed that in almost no instance of survey research about a specific discipline were the department chairmen the exclusive source of information at a given institution. Thus the names of these indi- viduals were gathered from the catalogs or from persons who have an active association with the various depart- ments. The authorities were selected by the author from people currently active in the American Association for Higher Education, Association of Professors of Higher Education, or who have achieved national recognition for their investigation of and writings about Higher Education. The names of these experts participating can be found in Appendix B (N=7). Though the total sample was drawn to indicate as much diversity as possible in a panel of fifteen persons, the results may not accurately reflect the opinions of other department chairmen in other types of Higher Edu- cation programs, nor the opinions of all the acknowledged 66 experts in the field. Also, the advantages of a rela- tively small group for an intensive study inevitably are balanced by the disadvantages of small numbers for interpretation and generalization of the results. These limitations of the study should be kept in mind throughout the remainder of this report. Step Three As a third step, an opinionnaire was devised to gather the necessary information of the first stage of the survey. Program characteristics developed by the author through a review of the literature were listed under five major headings. The headings are: Students, Personnel, Program, Finance, and Facilities. Program characteristics listed under each heading on the first opinionnaire are as follows: A. Students--these characteristics include those concerned with the academic ability of entering students, the congruence of student career interest with program purpose, the level of dissertations, the sense of com- munity among students, the positions occupied by graduates, and the placement of recent graduates. B. Personnel--included in this section are char— acteristics concerned with the academic preparation of the faculty, the research and publication activity of each faculty member, faculty experience in their area of 67 expertise, involvement of the faculty in departmental affairs, evaluation of teaching effectiveness, the utili- zation of interested faculty outside the department, and teaching loads. C. Program——these characteristics included those concerned with the types of degrees offered, program pur- pose and plan, flexibility of requirements, core require- ments, admissions policies, competencies expected of graduates, courses related to specialty of the faculty, enrollment size, leadership quality of the chairmen, and cognate area relationships. D. Finances--this section contained character- istics concerned with institutional support, external support, financial assistance to students, and ratios of budget allocations. E. Facilities--characteristics included in this section are concerned with library holdings, essential equipment, support services, computer facilities, instructional space, and faculty office space. The initial form of the instrument was reviewed by the Chairman of the author's Guidance Committee. A pilot run was conducted with the Chairman of the Depart- ment of Administration and Higher Education at Michigan State University. The revised form of the opinionnaire (Appendix C) evolved from suggestions of these two reviewers. 68 Panel members were asked to rank the various characteristics in order of importance for judging quality in doctoral programs in Higher Education. Ratings were made with number one being most important. Respon- dents were also invited to comment on individual items, and to add items if they felt important characteristics had been omitted. The second opinionnaire (Appendix D), built on the results of the first one, contained all the charac- teristics under the five headings in the order of their rated importance to quality judgment. In addition, the first responses were included for each individual panelist. Participants were instructed to note the modal rankings and then rank the characteristics again in order of importance. If their responses differed from the modal responses, they were to indicate, if possible, one or two reasons for their choices. The third and final opinionnaire (Appendix E) was constructed based on the second. Modal responses were again indicated. This stage also contained the minority opinions provided by some of the participants. These minority opinions were listed under two headings, one for those participants contending a goal should be "more important" than the first modal rating, and the other for those participants contending a goal should be "less important" than the first modal rating. In addition, the 69 second responses were included for each individual panelist. Participants were instructed to observe their second rankings, the modal responses, and any minority opinions provided. Then each panelist was asked to indicate for the last time his ranking of cri- teria for evaluation and return the opinionnaire. Step Four The fourth step was concerned with the information gathering by means of opinionnaires. The first stage opinionnaires were mailed January 16, 1976, to the selected samples, together with cover letters (Appendix C) and a self-addressed, stamped return envelope. The opinionnaire was also accompanied by a personally addressed and typed cover letter signed by the Chairman of the Department of Administration and Higher Education at Michigan State University. Thirteen completed responses were received by the cut-off date of January 26, 1976, for a total response rate of 87 percent. Table 6 (Appendix F) summarizes the responses to the first opinionnaire. All seven authorities responded, while institution participation was limited to six of the possible eight. The tabulated results of the first opinionnaire may be found in Table 1. The second Opinionnaire and cover letter (Appen— dix D) was mailed on February 4, 1976, with a cut—off 70 date of February 16, 1976, for tabulation. Fourteen completed opinionnaires were received for a total response rate of 93 percent. Table 6 (Appendix F) summarizes the responses to the second opinionnaire. All seven authorities responded again, while institutional response rose to seven of the possible eight. The tabulated results of the second opinionnaire may be found in Table 2. The third and final opinionnaire and cover letter (Appendix E) were mailed March 12, 1976, with a cut-off date of March 22, 1976, for tabulation. Twelve completed opinionnaires had been returned by that date and two more opinionnaires came in after that date for a total response rate of 93 percent. Table 6 (Appendix F) summarizes the responses to the final opinionnaire. Six of the seven authorities participated, while all eight institutions responded. The tabulated results of the third opinionnaire may be found in Table 3. Because of the relatively small number of total participants, all three stages of the opinionnaires were tabulated by hand for frequency distribution and mean and modal ratings on the importance of various program characteristics in evaluating quality. This procedure served two purposes: (1) it provided summary tabulations 71 for use in construction of the second and third stages of the opinionnaires immediately after the cut-off dates; and (2) it provided detailed tabulations for use in the analysis and presentation of the collected information. Summary The methodology of this survey was simple, yet as the response rates indicate, it was highly successful. The first step, to determine if similar research had been conducted in this particular area, proved most rewarding. The selection of the sample of participants was the second step. The third step was devoted to construction, testing, and refining of the first stage of the information gathering instrument. This first stage instrument pro- vided the basis for both the second and third stage opinionnaires. The final step was the actual execution of the survey. By means of the techniques outlined above, detailed information was gathered, organized, and developed to reveal the nature and dynamic of charac- teristics for assessing quality in graduate Higher Edu- cation programs. CHAPTER IV ANALYSIS OF INFORMATION Introduction This study is designed to provide information, based on the extent of agreement among department chair- men and recognized authorities, about characteristics most important to judgments of quality in doctoral pro- grams in Higher Education at Big Ten institutions. Utilizing a modified Delphi Method, the study is executed through a three-stage mail survey to the eight institutions of the Big Ten who offer graduate programs in Higher Education, and to seven recognized authorities in the field. The first stage of this survey asked the participants to rank various characteristics about quality in order of importance. The second stage pro- vided the participants with a modal response to each item; asked that they rank the items again; and also provide reasons why their second ranking differed from the modal response. The final stage provided modal rankings based on the second opinionnaire, and minority 72 73 opinions when given. The participants again ranked the characteristics in order of importance to judgment about quality. This chapter reports the information obtained from each of the three surveys. Detailed tabulations of each questionnaire are presented. Table 2 summarizes the results of stage two; and the stage-three opinion- naire results are presented in Table 3. In addition, the effectiveness of the Delphi Method in bringing about convergence of opinion is discussed. Finally, Table 5 summarizes the characteristics selected as the best cri- teria for evaluating graduate programs in Higher Education in the order of their endorsement by the participants in this study. The remainder of this chapter discusses the results of each questionnaire, the success of the Delphi Method, and the summary listing of characteristics that could be used in assessing quality in doctoral programs in Higher Education. Analysis First Opinionnaire Introduction. From detailed results of the first opinionnaire (Table 1), it is apparent that both groups of participants-~department chairmen and experts--agreed about the importance of some kinds of characteristics for 74 the assessment of quality and disagreed about the impor- tance of other characteristics. In addition, there appears to be more agreement among the participants within groups than agreement between the two groups. Though the main purpose of this first opinion- naire was to eliminate some characteristics, the results warrant further attention along two lines: the distri- bution of preferred or first-choice characteristics, and the spread in scores from the highest ranked to the lowest ranked characteristic. Though the "score" or mean ratings provide a convenient index for the identification of the most important characteristics, the frequency tabulation of ratings give a better indication of the degree of con- sensus among the participants about the importance of each characteristic. The remainder of this section briefly discusses each category of the opinionnaire and the ratings of the characteristics. Students. Examples or apparent consensus of difference of opinion can be found in the section dealing with students. The mean ratings or scores for each characteristic do not reflect agreement on which item should be first or second; the two items that receive the highest frequency tabulations from both groups are the same. For example, the characteristic concerning the high congruence of student career interest with 75 program purpose and emphasis received a mean score of 1.6 from chairmen and 2.6 from the authorities, yet this item was preferred as the number one characteristic by six of the thirteen participants (Table 1). The par- ticipants did not, however, agree to the same extent on any of the remaining items in this category. Personnel. The category where the greatest dif- ference of opinions occur contains characteristics about faculty or personnel (Table 1). The difference is evi- denced by the lack of agreement both among members of each group of participants as well as between the two groups. The characteristic which was rated first by chairmen received a score of only 2.5 and was preferred as number one just twice. The item rated first by the authorities scored 3.0 and was selected as number one by just two judges. The item ranked third by the authorities was first on the chairmen's list, whereas the item ranked least important by chairmen was the second choice of the authorities. This disagreement over personnel character- istics to be used in judging quality of graduate programs persisted throughout the second and third stages of this survey. Program. Different program measures, practices, and policies at the institutions of the participants did not appear to influence very greatly the importance 76 v cofipmospm mumspmum Hflmcu on ugm>maou wauomnfio mgofluwmom CH pmomad mums mummm moucu ummH ecu CH mpcmflmflomu mmnmmo m mugopsum an UmMMSm ma moflufl>fluom maflnznuuos CH ucoEm>Ho>gw was wuflgDEEoo mo mmgmm 4 m ngEo>chom pampsum mo oaumflumuomumco ma mcoHumu nummmflp mo mmmgmsgwgs was mocmaamoxm mo Hm>oa amen g m pamwu on» as codename“ cam aficmnopmma mo mCOHu uflmom mmdooo 30g moungpmuw H mammnmEm can mmomnsm Emumoum Baas ummumugfl Homumo unopgum mo wogosumgoo swan 8 ma whose UmHHmm muoom xcmm UmHHmm muoom xcmm UWMMMM loud loud ouoom xcmm mofiumflnmuomumnu I I Amaucv Anlgv mmwuwnosus< Amlgv ngHflch mucopgommmm HH¢ mBZMQDBm mzo mwdam .mMHGZZOHZHmO “m2¢mwomm .Q.mm m0 NBHAfiDO mmE abomd meszUQDh mom H mqm<8 mBZMQDBm m0 mUHBmHmmBUHo>cH mo omummp noes m ma whose consumes tam mgfimfl>pm .mggcomou HOm wEHp oumngmom How gumocoo pomamou mpmoH mgflsomme goflumsHm>w mumsomnm can unmosum an pmgHEMmumo mm swan mH mmogm>wuommmm meanomma moflumflnouomumnu Amzzommmm .Oum .dmo .942 .mmw mo mmuoom an pmgfleumuop ma Emmmonm map mgflumucm mugmpzum mo mpflaflnm owEmUmom Hmumgom one common Imum pounmm loud pmuumm whoom xgmm mum OHOOW XCMM OHOUm MCMM Amaugv Lance mmwuflnonus< muemecoammm Ham longs cmaugmau moflumfiuouomumno mBZmQDBm UODGHuGOUIIH mqmdfi 78 HMCOHDMC oCm HmCoHumH .mumum .HMCoHuCuHDmCH ”moH>Hmm on wuHComm on» mCOEm quEuHEEoo mcouum m mumeo muons muHMHoomm mo moons mumHHmonmmm CH Hm>mH wuHmHm>HCC Ho wmmHHoo oCH Hm COHumuumHCHepm Ho mCHCommu CH pooCoHuomxm mum mHmQEmE wuHCowm HHC cmuHHHus mum CoHumospm HmCmHm CH woummnmuCH mum on? CoHuCuHumCH map CH muonsmmHm wuHComm Emumoum on» on huHm>0H 6C8 Mom EmMHm ICCuCo .mHCmHoommH Scum Iona Cqu COHuommeumm manHme muHComm one muH>Huom Coummmou CH UmmmmCm mum muHsomm HHC omunmm imnm oHoom xcmm lence mmHuHuoausa pounmm Imum muoom mem powwow Imum oHoom mem Amqu CmEHHMCU AmHnCV mquCCommmm HHC moHumHHouomnmnu Amzzommmm emscHucoouuH mqmge 79 H v 0 m v muHsomm on» mo mCH ICHoHu muHoHoomm oCm Scum loud onu mo momomusm onu ou ouoHHmoummm ohm mooCoH luomxo oopoHoH oCm omHsoo mcooC uCooCHm HouoH>HoCH Home on uCoHon Imam mpCoEoHHsgoH Eoumoum «0 suHHHQonHm m mH muons Con oCo omomusm Eoumoum mo huHuoHo o mH oHoCs mUflfimHHO#OMHM£U Edmwomm ooumoo .D.Um Ho .Q.Cm oCu UHOC mmoum UHEoo loom ocu mo mHoQEoE HHC whom» ooHCp Doom oCu CH CoHu IMOHHQSQ an pouComonmoH Coon o>oC muHsoow HHC powwow loam ouoom mem powwow powwow IOHQ OHOUW Mflmm lmhm OHOOW MGM“ lence mmHuHuonusa AmHqu Amqu CoEHHsCO muCopCommom HH¢ mOHumHHouooHoCu AHZZOmmmm noncHucoouuH mqmoum con IHooo UCm mHCmuoUmoH mo muHHmsg CmHC m mH ouoCs mouooCoum oco mouspmooum on no uooHo mum moHoHHom ConmHEoC mHmoQ uoHCmou m Co poCHmuCHoE oum mEmumoum oHMCmoo CuHs momCoco IuouCH pCo mmHCmCoHuoHom muoumuumHCHEpo uo muoCooou .muonouoomou on on mCHuomoum mouosomum mo oouoomxo moHuHHng oCo moHoCouomEoo oonHoomm mo muHuMHo o mH ouoCs pouuom Ioum ouoom xCom wouuom Ioum ouoom xCom wouuom Ioum ouoom xcom Asucv mouuuuoausa Amqu CoEuHoCO Amuncv muCooCommom HHC mOHfimHHOHUMHMQU Edmwomm UwfldHuGOUIIH mqmdfi 81 ooCoumHmmo HoHoCoCHm mo Eu0m oEom mCH>Hooou ouo ouCoosum mo suHuosmE C muoonoum UCo muCoum poonoum nouoomou mo Euom onu CH mumeo Eoumoud onu uOH Duommsm HoCuouxm sodum Hmuouooo uou CoHuoooHHo muHmuo> IHCC Houou onu op COHH IMOOHHM uomosn Eoumoum mo 0Humu o>HuHmom m mH muons omomusm Emum loud on oumsgooo UoEooo mH uuommsm HoCoHusuHumCH moHumHuouoouan mmUZ¢ZHm muCoUCum mo uonECC mmuoH o mHHouCo Eoumoum ons oouuom Ioum ouoom xCom oouuom noum ouoom nCom oouuom Ioum ouoom xCom lungs mmuuunoausa louse cmauumao Amuncv muCopCommom HHC moHumHuouoouonu 24mmomw coccuucoouuu muons 82 Eoumoum onu mo mUooC on» uom ouosgooo ouo moHuHHHoom uousmfiou Eoum Ioum onu uOH muHComm msm uom oommm nouoomou OCm monmo oumsvopm mH ouons Eoumoum onu uom uCop Isum msm uom moomm HMCoHu IosuumCH ouosgooo mH ouons Eoumoum on» mo moooC on» Home mHoumsg loom mooH>uom uuommsm uComoum ouo uCouuomEH uo HMHuCommo pouopHmCoo uCoEmHCvo oCo moHuHHHoom mmCHpHon huounHH qu>oHou mo woodwovo Co mH muons nouuom Ioum ouoom mem pouuom Ioum ouoom xCom oouuom Ioum ouoom xCom lsuce mmuuuuonus< Amqu CoEuHonU annCv muCooCommom HH< moHumHuouoouonO mmHBHAHuflm oosCHpCoollH mamas 83 ratings of scores assigned characteristics in this cate- gory. Table 1 indicates an agreement on clarity of program and plan as the most important criteria in doc- toral program evaluation. This item received the best score from both groups of panelists, yet was selected only twice as first choice of chairmen, while six of the seven authorities preferred it as first choice. Finances. Table 1 reveals agreement by all par- ticipants both in scores and preference that institu- tional support and the ratio of budget allocation to total institutional allocations for doctoral study to be the most important financial characteristics used in evaluating graduate programs. Facilities. Similar examples of agreement both in score and preferences are found in the tables on Facilities (Table 1). Summary In general, characteristics with a score of 3.0 or below were considered to have been endorsed by the panel. However, the author decided to include all characteristics on the second opinionnaire. In addition, a few characteristics were restructured based on comments from the respondents and observed over-lap. Each par- ticipant was encouraged to include any criteria that were felt to have been omitted. When additional items were 84 suggested, these were reviewed by the author for possible overlap or similarity with others and added when appro- priate. Second Opinionnaire Introduction. The second stage opinionnaire for this survey was based on the first stage and contained all the characteristics listed under the five headings. The characteristics were arranged in order of their rated importance to quality judgment. In addition, characteristics that were suggested by the panelists were included after the review by the author. The first response for each item was included for each individual panelist. Frequency tabulation and mean rankings of the second stage opinionnaire are reported in Table 2. As with stage one, the mean ratings are useful as an index for the identification of the most important char- acteristics, but the frequency distribution of ratings give a better indication about the degree of consensus among participants. In addition to the frequency tabulations and mean ratings, the analysis of stage two reports the minority reasons provided by respondents when their second ranking differed from the modal ranking. These reasons are listed under two headings--one for those participants 85 contending a characteristic should be “more important" that the first modal ranking, the other for those par- ticipants contending a characteristic should be "less important" than the first modal ranking. This section will discuss the results of each category, the summary listing of characteristics, and the minority responses provided by the participants. Students. As in the analysis of stage one, examples of apparent consensus or difference of opinions can be found in this section (Table 2). Both groups of participants agree on the same characteristic being pre- ferred as number one. Congruence of student career interest with program purpose and emphasis was chosen number one by five participants from each group, for a consensus preference of ten out of fourteen panelists. Chairmen and authorities were in agreement on the first three items in this category. The scores of the fourth, fifth, and sixth items were 3.3, 4.2, and 5.5 respec- tively. The spread between the scores for the fifth and sixth items was sufficient to justify the elimination of the sixth item from the third stage opinionnaire. This item was concerned with the general academic ability of entering students as determined by a standardized test or previous grade point average. While this item received a total score of 4.5 on the first survey, it dropped to a score of 5.5 for the second stage. 86 OH m.H H UCoEo>oHnom uCopCum mo UHHmHuou Iomuono mH mCoHuouuomme mo mmoCngHCC UCo ooCoH uHooxo mo Ho>oH ann C muCoUCpm an Couonm mH moHuH>Huom oHHnsnuuos CH uCoEo>Ho>CH UCo muHCCEEoo mo omCom C CHon onu CH oOCoCHuCH CCo mHnmuouooH mo mCoHu IHmom mmsooo 30C mouospouw mHmmnmEo pCo omomusm Eoumoum nuHs umououCH uoouoo uCopCum mo ooCo ICumCoo ann o mH ouons wouuom Ioum ouoom xCom pouuow Ioum ouoom xCom wouuom Ioum ouoom xCom Aanv moHuHuonuCC Asqu CoEuHmno neHqu muCoUCommom HHC moHumHuouoouonO mBZMQDEm 038 mwdfim .MMHCZZOHZHQO umEmeomm .Q.mm m0 MBHACDO mmB Ebomd mBZMZODDh mom mBZHQDBm ho mUHBmHmWBUCMCmU N Wflmdfi 87 nouoomou pCo mCHmH>Co .mCHnooou uom oEHu ouosgopo uOm CuooCoo uoonou mUMOH mCHnooos CoHumsHo>o opospoum CCo uCooCum an UoCHEuouop on ann mH mmoCo>Huoommo mCHnomos moHumHuouoouonU qmzzommmm .ouo .Cmu .sCS .55 mo mouoom an ooCHEuouoU mH Emumoum on» mCHuouCo muCopCum mo suHHHnm UHEopooo HmuoCom ons COHumosuo ouospoum uHonu ou UCo>oHou mHuoouHU mCOHuHmom CH Uooon ouo3 muoos oounu umoH on» CH muConHoou ooumoo pouuom pouuow Ioum ouoom xCom loum Couuom ouoom xCom Ioum ouoom xCom 11‘ anqu Asucv moHuHuonusC mucoecommom HHC Asqu CoEuHonU moHumHuouooumnu MBZWQDBm eoseuuaoonnm mamas 88 o.m m wuHMHoomm mo moouo ouoHumoummo CH Ho>oH muHmuo>HCC uo omoHHoo onu no COHuouumHCHEUo uo mCHnooou CH CooCoHuomxo oum muonEoE suHCoom HHC Eoumoum on» o» wuHo>0H UCm uom EmmHanuCo .mHnmuopmoH Eoumoum nuHs COHuoommHDMm manano suusomu one CoNHHHuC mum CoHumo ICUm uoanm CH CoumououCH mum on3 COHuCuHumCH onu CH ouonzomHo muHCoou muH>Huoo nouoomou CH pomomCo ouo suHComm HHC muHmmmm Eoumoum HouCoE quomoo CH mmoum OHEop noon on» sn uCoEo>Ho>CH mo ooumop ann m mH muons Couuom toum ouoom mem Couuom uoum ouoom 3Com Couuom Ioum ouoom xCom Abuse mmuuuuonusC inane cmauumno anuCV muCoUCommom HHC mUHfimHHOMOMHOSU szzommmm COSCHHGOUIIN qudfi 89 OH NH N.H H mpooC uCoUCum HCCCH>HCCH Home on uCoHoHu Imam muCoEouHCgou Emumoum mo muHHHonon o mH ouons Con pCo omomusm Emumoum mo muHuoHo o mH muons mOHumHuouomuan ZCCOOCC ooumop .o.pm uo .C.nm on» pHon mmoum UHEop loom onu mo muonEoE HHC muoos oounu umom onu CH COHumo IHHnCm an pouComoumou Coon o>on muHCoom HHC HMCOHHMC CCo HoConou .ououm .HoCOHuCuHumCH ”ooH>uom ou suHCoom onu mCoEm “CoEHHEEoo mCouHm o mumeo ouons pouuom Ioum ouoom xCom pouuou noum ouoom xCom lungs mausuuoausa lungs cmeuumau pouuom Ioum ouoom mem anqu muCoCCommom HHC mOfiHMflHOUOMHMSU Amzzommmm GOSCHHGOUIIN mqmdfi 90 CoEuHono uCoEuuomop on» an Uo©H>oum COHmHooC CCo mHnmuopooH mo auHHng ann o mH muons mpuopCoum UCm mouCpoooum on no umoHo ouo moHoHHom ConmHECC mHmmn uoHCmou C CO UoCHouCHoE mum mEoumoum ouoCmoo nuHS momCmno IuouCH UCC mmHnmCoHuoHom muououumHCHEpm uo muonooou .muonoumomou on 0» mCHummoum mouospoum mo pouoomxo moHuHHmsg pCo moHOCouomEoo ponHoomm mo suHuoHo o mH muons huHCoom onu mo mCH ICHmuu suHoHoomm pCo Scum Ioum on» mo momomusm onu on ouoHumoummo ouo mooCoH Iuomxo pouoHou UCo omusou pouuom Ioum ouoom xCom pouuom Ioum couuow ouoom xCom Ioum ouoom MCom lsucv mouuuuonusa anqu AhnCv COEHHCCU muCoUCOQmom HHC mOHumHHGUUMHMHAU SCMUQmm GUCCHHCOUIIN wands 91 u H.m N o o.m m u m.~ m Henna umuouooe uom COHuoooHHo suHmuo> IHCC Houou on» 0» COHu IMUOHHo uomusn Eoumoum mo 0Huou o>HuHmom o mH ouons m H.H H s o.H H mH H.H H omomusm Eoum Ioum ou ouosgowm poEoop mH uuommsm HCCOHuCpHumCH moHumHuouomuonU mMOZCth o o.OH oH o n.m 0H 0 v.m 0H muCoUCum mo uonECC omuoH o mHHouCo Eoumoum ons o o.m m H H.m m H m.m m wouommo mum mooumov .Q.©m UCC .o.nm map nuom o H.s m o m.m s o s.m m muCoCCum HHo u0m uCoE IouHCUou ouoo o mH muons pouuou Couuou pouuom noum ouoom nCom Ioum ouoom xCdm loum ouoom xCom moHumHuouomuonO Aanv moHuHuonuCC Aanv CoEuHonU neHqu ZCCGQmm muCopCommom HHC eoseHucooaum mqus 92 v.H H uCommum mum quuuomEH uo HoHuCommm CouoCHmCoo uCoEmHCwm CCM moHuHHHoom Eoumoum onu mo mpmoC onu mummE sHmumsv loco mm0H>uom uom uuommsm mmCHUHon muounHH UCo>onu mo somsvopo Co mH muons moHumHumuooumnO mmHBHHHUCW mCCCm COHuCuHumCH Eoum oHnoHHm>o mmHnmuCoumHmmm mCHnooou CCo noumommm ooCoumHmmm HoHOCoCHm mo Euom mEOm mCH>Hmomu mum mquCCum mo suHuOnoE C muomuuCoo oCo muCoum uomnoum nouoomou mo Euom onu CH mumem Eoumoum on» uom uuommnm HCCuouxm pouumu Imum ouoom xCom lungs mmuuuuoausa Couuom Ioum ouoom mem pouuow Ioum muoom xCom lungs casuumsu AanCv muCooCommmm HHC moHumHumuoouonU mmUZCZHm ©05CHHCOUIIN mHMCB 93 o o.m m o m.s m o m.s m smumoua on» uou unusomu msm uom ooomm noumomou pCm oonmo oungoUo mH muons o «.4 m o o.m m o m.v m emumoud any uou name ICum msm uom oommm HoCoHu IosuumCH mumsvopm mH muons o o.e s o u.s s o ~.e v smumoud mnu mo mUomC on» uom mumsgopm mum mmHuHHHomm umHCmEou pouuom nouuou pmuuou Ioum ouoom xCom Ioum ouoom xCom Ioum ouoom xCom moHumHuouomuonO Aanv moHuHuonuCC AsuCV CoEuHmnO anqu mmHsHHHOCm muCopCommmm HHC UODCHHCOOIIN WHmCB 94 Items 2, 3, and 4 drew minority opinions from the respondents. Item 2 suggests that one measure of quality can be in terms of graduates occupying positions of leadership in the field. While this characteristic received an average rating of second, minority opinions of those who felt it should be "less important" can be_ summarized as: more often a result of selection rather than quality of educational preparation. Item 3, dealing with the importance of involvement of students in worthwhile activities to develop a feeling of departmental community, drew minority opinions in support of both higher and lower ratings. Those par- ticipants who suggest this item to be “less important" than third contend: students are expected to work together, but this is not necessary in order to have a quality program. The viewpoint in support of higher rating sug- gests this to be: a more contemporary measure (of program quality) than the number of graduates in leadership positions. High quality of dissertation by graduate stu- dents, Item 4, drew diverse opinions as to importance as a measure of program quality. This item was rated number one by one participant consistently, who felt it to be the single best clue to program quality. The opposite opinion was shared by several panelists who rated the 95 item low and felt the dissertation to be only a learning experience, and does not necessarily indicate the potential for attaining a leadership position. Personnel. This section containing criteria about personnel again proved to be the area of least agreement among participants (Table 2). Yet some con- vergence of opinions can be observed. Item 1, high teaching effectiveness as determined by student and graduate evaluation, was chosen first by six of the participants. Reasons given by those participants who felt this item should be "less important" are summarized as: used by almost everyone, but with little confidence in its accuracy. Perhaps the fact that institutions use this method of evaluating faculty was the chief reason the item was ranked first. Teaching loads and concern for adequate time for teaching, advising, and research was ranked second by the participants. This characteristic was considered essential, but incidental to program quality by those panelists who ranked it lower than second. There was strong support for a high degree of involvement by the faculty in departmental program affairs as a good measure of quality. Although this 96 item was rated number three, several participants indi- cated it to be first order of business in creating an excellent program. The item, "all faculty are engaged in research activity" was ranked fourth by both groups of panelists. The minority Opinions which supported a "less important" rating are summarized as: some faculty are synthesizers, not researchers; and research has little relation to teaching effectiveness. The utilization of faculty elsewhere in the insti- tution who are interested in Higher Education as a cri- teria for quality assessment received the most diverse ratings of any item in this category. Chairmen scored it 6.9 for a ranking of eighth, while the authorities scored it 3.1 for a second place ranking. One authority rated the item as number one, and those who felt it "more important" contend that "outsiders" help avoid thinking of Higher Education as a closed discipline, and rather as one that is inclusive of the total role of the institution and participation should express this fact and nourish the program. The results of this stage indicated that four of the ten original characteristics should be eliminated from the third opinionnaire. 97 Program. Table 2 summarizes the results of the rankings in the category on Program. Clarity of program purpose and plan as an assessment of quality in graduate education was selected as number one by twelve of the fourteen participants in this stage of the study. One chairman ranked offering both Ph.D. and Ed.D. degrees as number one in assessing quality, while one authority felt that a flexibility of prOgram requirements would best reflect quality. It should be noted that the latter of these two criteria, program flexibility, averaged a ranking of second by all the respondents, while the dual degree (Ph.D. and Ed.D.) offering averaged ninth and was not included on the final opinionnaire. There was general agreement on the rank order for the characteristics of: program flexibility; courses being appropriate to the program and to the expertise of the faculty; the existence of clarity of competencies and qualities expected of graduates; and the maintenance of regular contacts with cognate programs. These were ranked second, third, fourth, and fifth, respectively, as criteria which should be included in assessing quality in graduate programs in Higher Edu- cation. The remaining five characteristics in this cate- gory received very 1ow ratings and were excluded from the final stage opinionnaire. 98 Finances. There was agreement on the ranking of the characteristics in this category. Thirteen of the fourteen participants ranked "institutional support is deemed adequate to program purpose," as number one (Table 2); and "the existence of a positive ratio of budget allocation to the total university allocation for doctoral study" as the second most important criteria. A minority opinion suggested the second charac- teristic might be "less important" because it is not necessary for programs in Higher Education to be as expensive as those of other areas. Financial assistance to a majority of students was considered to be "more important" than fourth place because it is essential to recruiting strong students. Facilities. In Table 2 are listed the rankings and scores of criteria in this category. The importance of adequate library holdings to program quality was expressed by ten of the participants. Adequate support services and having essential facilities and equipment ranked a close second and third, respectively. The panel tended to give slightly larger scores to the remaining two items. However, the dif- ferences between scores of items in this category did not warrant elimination of any from the third and final Opinionnaire. 99 Summary The second opinionnaire was built on the results of the first. It contained all the characteristics of the first one and ranked them in order of their importance to quality judgment. In addition, the first responses were included for each individual. The purposes of the second stage opinionnaire were: (1) to have the par- ticipants note the modal rankings and then rank the char— acteristics in order of importance; and (2) to elicit, if possible, reasons why the panel members' second rank- ings differed from the modal rankings. A second modal ranking for each item was computed from the mean score and frequency tabulation of this stage of the survey. These criteria eliminated one of the six characteristics originally listed under "Students," four of the ten characteristics listed under "Personnel," five of the ten characteristics under "Program," three of the eight characteristics listed under "Finances" and one of the six characteristics listed under "Facilities." In general, characteristics omitted from the third opinionnaire are the ones toward the end of each table in Table 2. The minority opinions were summarized and listed under two headings--one for those participants contending a criteria should be ”more important" than the first 100 modal ranking and the other for those participants con- tending a criteria should be "less important" than the first modal ranking. This distribution along modal rankings and the minority opinions served as the basis for the third and final opinionnaire. Third Opinionnaire Introduction. The third and final opinionnaire (Appendix E) was constructed from the information obtained from the second opinionnaire. Modal responses were indi- cated. This stage also contained the minority opinions provided by some of the participants. In addition, the second stage responses were included for individual panelists. Participants were instructed to observe their second rankings, the modal responses, and any minority opinions provided. Then each panelist was asked to indicate for the last time his rating of cri- teria important in assessing graduate programs in Higher Education. Table 3 contains the results of this third and final stage of the survey. The mean ratings serve as a convenient index for the identification and ranking of the most important characteristics; but the frequency distribution of first choice ratings provide a better indication of the degree of con- sensus among participants about the importance of 101 each characteristic. These indices serve as the basis for the list of criteria for evaluating quality in Ph.D. programs in Higher Education. The listings in Table 3 should be viewed with this factor in mind. There are a few more general comments about the ratings in this stage that do seem appropriate at this time. First, there is generally good agreement between mean ratings. Preferences sometimes scatter over several items, particularly if a number of characteristics were suggested for a given category, but generally 70 to 80 per- cent of the first and second preferences also received a mean rating of 2.5 or lower. Occasionally a character- istic was preferred by a number of participants even though it was not considered a particularly good index, such as the characteristic in the Personnel category in Table 3 (concern for teaching loads). This item was preferred by four respondents but had a mean score of 3.1. This incongruence of mean score and preference is unusual. In general, the concurrence of mean scores and preference choices is an encouraging indication that these criteria are acceptable to both chairmen and authorities and therefore are likely to be useful in the assessment of program quality. Second, for most categories there were one or two or three characteristics that were definitely rated higher and preferred more often by all the respondents, 102 HH v.H H CoHuooCUm mumspmum uHmnu ou qu>oHou mHuomqu mCOHuHmom CH poomHm ouo3 muoom oounu ummH onu CH mHConHoou ooumoo muCopCum sn pouonm mH moHUH>Huom oHHnsnuu03 CH quEo>Ho>CH UCC huHCCEEoo mo omCom C quEo>mHnoo uCoCCum mo UHDmHuouoouono mH mCoHumu Iuomme mo mmmCngHCC pCo ooCoHHooxo mo Ho>mH ann C pHon onu CH ooCoCHHCH UCm mHnmumpomH mo mCOHu IHmom hmsooo 30C moumsnouo mHmonmEm UCm mmmomusm Emumoum nuHB ummuouCH uoouoo HCopCum mo oOCoCuOCoo ann o mH muons Couuom ouoom mem pouuom Imum toum ouoom nCom AmuCV mmHuHuonuCC AmnCv CoEuHonU pouuow Imum ouoom nCmm anqu muCoUCommom HHC mOHumHuouooumnO mezmabem mmmmB MUCBm .mmHCZZOHZHmO «mZmeomm .Q.mm m0 wsHHCDO mmB BDOQC mBszOQDb mom mBZMQDBm m0 mUHBmHmmBUCmCmU m MHmCB 103 w.m v Eoumoum mnu ou suHm%0H pCo uom EmonCnuCo .mHnmuopooH Emumoum nuH3 CoHuoommHuom manano muHCoom mns wuH>Huoo nouoomou CH pomomCm mum muHsoom HHC ooNHHHuC mum CoHumo Ispm uoanm CH CosmououCH mum on3 CoHusuHumCH onu CH muonsomHo >UHComm muHonm Eoumoum HouCoE Ipuomon CH mmmum UHEop loom onu an uCoEo>Ho>CH mo ooumop ann m mH muons nouoomou pCo mCHmH>Uo .mCHnooou uom oEHu mumsgopo u0m CumoCoo uoonou mCMOH mCHnomos COHuoCHm>o oumspmum UCm uCoCCum an poCHEuoumC mo ann mH mmoCo>Huoommm mCHnooos Couuom Imum ouoom nCmm nouuow Imum ouoom nCmm nouuom noum muoom mem nouns mmuuuuonpsC AmuCV CmEuHonU AVHHCV muCopCommom HHC wOHHwHHOHUMHMEU Hmzzommmm eonauueoonum mCmCs 104 m.m v N.H H mHmmn uoHCmou C CO UoCHmuCHmE mum mEoumoum ouoCmoo nuHB mmmCono IuouCH pCo mmHnmCoHumHom muououumHCHapm uo muonooou .muonoumomou on on mCHuomoum mouospoum Ho oouommxo moHuHHng oCo moHOCouomEoo ponHoomm mo suHumHo m mH muons suHCoom mnu mo mCH ICHmuu suHmHommm pCm Eoum Ioum onu mo momomusm on» on ouoHumoummm mum mooCoH Iuomxo poumHou va omuCoo mnooC uCoCCum HoCoH>HpCH Home on uCoHon Iow muCoEouHCwou Eoumoum uo suuuunuxouo m mu muons CCHC pCo omomusm Eoumoum mo suHumHo m mH muons Couuom Ioum muoom xCom pouuom Ioum ouoom xCom pmuuom Ioum muoom nCmm noncv moHpHuonusC AmnCV CoEuHonO AVHHCV muCoUCommmm HHC wUHHmHHOHUMHMfiHU Emeomm noncHucoouum mCmCs 105 mUCCm CoHuCuHumCH Eoum oHnoHHo>o mmHnmnCoumHmmm mCHnomou pCo noumommm ooCmumHmmo HoHOCoCHm mo EuOH oEOm mCH>Hooou mum mquUCum mo muHuofloE C mucouuCoo pCo muCoum poowoum noumommu mo Euom mnu CH mumem Emumoum mnn uom uuommsm HoCuouxm spawn Hmuouoov uOH CoHuoo IoHHo Homnsn Eoumoum mo 0Huou o>HHHmom m mH muons omomusm Emum loud on ouosgopo noEoop mH uuommsm HoCOHuCuHumCH oouuom Ioum ouoom xCom wouumw Ioum wouuom ouoom nCoC Ioum ouoom mem longs mmuuuuosusa l HvHflGv HmICV CoEuHan mHCmUCommom HHC moHumHuouoouan mmUZCZHm UmSCHHCOUIIm MHmCB 106 o m.s s o m.s m o s.s m emumoum on» uou acme ICum msm umm mommm HCCoHu IosuumCH mumsvopm mH muons o m.s s o m.m a o u.s s amumoud mnu mo mpooC onu uom oumswmpm mum mmHuHHHoom uoquEou m m.H H o m.m m m v.~ m uComoum mum quuuomEH uo HMHqummm poumpHmCoo pCmEmHCgm pCm mmHuHHHooh o H.~ m m o.m N m m.m m smumoud on» no memo: mnu mummE sHmungoUm mmoH>uom uom uuommsm v H.u m m m.u u m m.u u mmcueuon sumunuu ucm>mumu mo sooswmpm Cm mH muons pmuuom Umuuom Umuumm ImHm mHoom xcmm lmHm wHOOm Mdmm lwum muoom Mddm mOHHmHHQHUMHan I I AVHuCV Amuse mmHuHuonusC Amuse cmeuumno mmusuquoCC muCooCommom HHC UOSCHHCOUIIM mHmCB 107 while there are some characteristics that were considered less adequate by an equal number. Consensus was great enough so that in most categories a clear line can be drawn between the criteria that are endorsed by the panelists and those that could be inadequate or contro- versial. Third, the importance of including the minority responses is difficult to assess. For example, a par- ticipant who ranked an item low on the second opinionnaire reads the summary of opinions included in the third opinionnaire and notes that some participants thought the item should be more important than the modal response. This could contradict his own reason for rating it of less importance on the second stage and cause him to rank it higher on the third stage. Whether or not this occurred is almost impossible to determine. Yet, obtain- ing the minority responses from the participants proved of considerable value to the author in providing infor- mation upon which to construct the final list of criteria, as well as affording some insights into the field of Higher Education provided by this wide range of par— ticipants. Convergence: The Oracle Speaks As for the effectiveness of the Delphi Method in bringing about convergence of opinion, the results reveal considerable success among'participants. 108 While only one category achieved convergence, convergence was occurring in each of the five categories. Convergence began to occur with the second stage opinionnaire, which reported the modal rankings of the first opinionnaire, and continued through the third stage. Table 4 shows that, with a few exceptions, the most significant convergence occurred on the third and final opinionnaire. This stage reported the modal rank- ings for the second stage and individual minority opinions provided by the participants. TABLE 4 CONVERGENCE OF OPINION ON CHARACTERISTICS CHOSEN "NUMBER 1" Stage One Stage Two Stage Three Category Total Total Total Total Total Total Number Chosen Number Chosen Number Chosen of Number of Number of Number Items 1 Items 1 Items 1 Students 6 4 6 3 5 3 Personnel 10 6 10 5 6 4 Program 10 7 10 3 5 2 Finances 4 2 5 2 5 1 Facilities 6 4 6 3 5 3 The difference in stages where convergence became significant apparently resulted from the type of judgment involved. To rate characteristics not directly involving kabple becomes essentially a matter of factual judgment, 109 but to rate characteristics in terms of people is a value judgment. The results indicate that the reasons for divergence, as summarized on the third opinionnaire, were perhaps more influential in regard to value judgments than in regard to factual matters. It should be noted that movement toward consensus does not necessarily mean that the process changed atti- tudes among the participants. This may or may not be so. It is possible that opinions changed as a result of feed- back which added dimensions not previously considered by some participants. That is, increased understanding of a particular statement may have caused some participants to change their ratings even though their basic attitudes remained constant. Why the changes occurred cannot be stated with any degree of certainty. That they did occur, however, is indisputable. The different groups came much closer to agreement on what characteristic in each cate- gory should be considered number one as the survey pro- gressed. Summary List of Criteria for Evaluating Graduate Programs in Higher Education The thrust of this survey was to identify cri- teria for evaluating graduate programs in Higher Education at Big Ten institutions. Table 5 lists those criteria determined necessary for the assessment of Ph.D. programs in order of their rated importance. Criteria have been 110 TABLE 5 SUMMARY LIST OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION Category Criteria Students There is a high congruence of student career interest with program purposes and emphasis Graduates now occupy positions of leader— ship and influence in the field A high level of excellence and uniqueness of dissertations is characteristic of student achievement Personnel Teaching effectiveness is high as deter- mined by student and graduate evaluation Teaching loads reflect concern for ade— quate time for teaching, advising and research Program There is a clarity of program purpose and plan There is a flexibility of program requirements sufficient to meet indi- vidual needs Course and related experiences are appropriate to the purposes of the pro- gram and specialty training of the faculty Finances Institutional support is deemed adequate to program purpose There is a positive ratio of program budget allocation for doctoral study Facilities There is an adequacy of relevant library holdings Support for services adequately meets the needs of the program Facilities and equipment considered essential or important are present 111 drawn from the summary of stage three of the survey and are limited to those items with a rating of 3.1 or lower. Criteria are grouped under the same five headings used on the survey instruments: Students, Personnel, Program, Finances, and Facilities. Though probably somewhat too long and detailed, this summary list of criteria provides an important first step in efforts to establish a systematic procedure for evaluating graduate programs of study in Higher Education. Summary The purpose of this study was to identify criteria for evaluating graduate programs in Higher Education at Big Ten institutions. Information from which the criteria would be identified was gathered by a three-stage mail survey. The results of each of the three opinionnaires used in the survey have been summarized, reported in apprOpriate tables, and interpreted. In addition, the effectiveness of the Delphi Method, as used in this sur- vey, to bring about convergence of opinion was reviewed. Finally, a summary listing of criteria for assessing graduate programs in Higher Education was compiled from the information obtained in this survey. The listing of criteria for use in evaluation is not proposed as a definitive view of quality in graduate programs in Higher Education, but rather as a first 112 winnowing of the multitude of criteria and measures that might be considered in the assessment of such a complex and varied educational effort. The list as it stands now would profit from further review by experts in the field of Higher Edu- cation, and probably could be simplified somewhat. How- ever, even in its present form it represents a better picture of graduate program assessment criteria that would be acceptable to a cross-section of department chairmen and experts than has been available so far. As such, it could be useful to the Higher Education Departments in the Big Ten institutions, as well as to others who are interested in furthering the systematic development of procedures to assess quality in graduate programs in Higher Education. CHAPTER V SUMMARY Introduction A desire for highest quality seemingly needs no defense, regardless of the product or craft involved. Yet, the consideration of quality in any setting seems to provoke controversy. Perhaps a part of the contro- versy regarding quality traces to the ambiguities within the concept itself. Little is known about how to assess quality, or even what to assess. Yet today, assessment of quality is among the most complex issues facing institutions of higher edu- cation. Despite the controversy which sometimes accom- panies assessment, systematic evaluation of all graduate programs is becoming increasingly important and useful. Doctoral education appears to have entered a period of readjustment and reassessment. One major reason for this review of priorities is the serious financial stress experienced by the higher education community. In addition, the demand for doctorates in the job market has largely been satisfied. And finally, a general 113 114 concern has been growing over how well universities have actually been serving individuals and the nation. Of course, assessment is necessary for positive reasons as well. The expansion of knowledge continues unabated and the modes of transmitting and utilizing knowledge are accelerating without pause. Given the rate of change in the environment, regular assessment of the effectiveness of research and teaching at the graduate level are necessary to forestall obsolescence and irrelevance. Lastly, quality assessment possesses an internal virtue of no small consequence. The assessment process has salutory results. As new ideas emerge and better procedures are introduced, concerned assessment questions long-standing assumptions and generates a climate for' healthy growth and development. For these and perhaps other concerns, regular assessment, no matter how agonizing at times, should be fundamental, especially in graduate education. Higher Education as an integral part of American higher education faces the current critical questions put to all graduate study. It is out of these considerations and questions concerning the status and quality of graduate education and Higher Education in particular that this study developed. 115 The remainder of this chapter is divided into three sections. The first will be devoted to a review of the organization and procedures of the study. Secondly, the author will present observations and cone clusions. And finally, recommendations will be offered for consideration. Overview of Study The objective of this investigation was to identify criteria for evaluating graduate programs in Higher Education at Big Ten institutions. To achieve this objective, information was gathered from two sources: (1) chairmen of departments of Higher Education at Big Ten institutions; and (2) recognized authorities in the field of Higher Education. The first chapter develops the basic assumptions and operating principles for the remainder of this study. Basic assumptions need to be clear before proceeding, although the process of identification may be time- consuming. To skip over or ignore this task may lead to compounded problems later, since everything that eventually becomes operational should relate to basic assumptions. That assessment of quality in graduate education is necessary and possible and that Higher Education can be considered a field of study, therefore subject to assessment, are the underlying assumptions upon which 116 the investigation was based. To substantiate such assumptions, a literature review was completed and was reported in Chapter II. Recapitulating some of the information derived from the review, graduate education, in spite of criticisms (some deserved), has lived through a number of phases in responding to educational and societal pressures, and should continue to evolve in the next decade in a socially desirable fashion. Information gathered from writers and researchers revealed that the focus and intensity of evaluation of graduate education has varied according to the concerns of the assessor. The analysis exhibited two distinct concerns: (1) the rapid growth in enrollment and the resulting impairing of quality, and (2) the proliferation of graduate study into institutions often indifferently qualified to offer graduate programs. Examination of the literature also left little doubt about the proportions Higher Education study has reached. As a means of examining all facets of the higher education enterprise, Higher Education serves an integral function in the research process. Unfortunately, variations in program content and purpose continue to cloud both the national and local perspective, the , analysis of the literature revealed that a single pat- tern for program does not exist. Rather, there are vari- ations by region, within regions, and among the sub-fields. 117 Chapter III details the methods and procedures followed during this study. The focus of the investiga- tion was the chairmen of the eight departments of Higher Education in the Big Ten, and the seven recognized author- ities in the field of Higher Education. The research design included the chairmen and authorities in order to determine the extent of agreement between and among the participants about criteria for evaluating Higher Education doctoral study. The heart of the study, however, is the fourth chapter. Information from the participants is tabulated, examined, and analyzed. Movement toward consensus of agreement on criteria for assessing quality is observed. As the survey progressed, the two groups of participants-— chairmen and authorities--came much closer to agreement on which characteristics should be considered most impor- tant in assessing quality. A final summary listing of criteria for assessing doctoral programs in Higher Edu- cation was compiled from the information in Chapter IV. The fifth and final chapter summarizes the study, offers some conclusions, and suggests recommendations for con- sideration. Conclusions It is somewhat difficult to separate those sub- jective conclusions which one has gathered as a result of such a study from those objective conclusions which 118 are supported by the information. Objectivity becomes a large problem as one sorts the subtle feelings gained through reading the literature, through impressions of knowing some of the participants, and from being part of one of the doctoral programs being examined. In hopes of preserving some separation of fact from feeling, con- clusions offered should be viewed from three perspectives: those conclusions supported by the information of the survey, those conclusions gathered through research of the literature, and those conclusions arrived at by the author as a result of conducting the study. Methodology The published research on doctoral education is useful. The first step in any investigation is to under- take a review of the related literature. The research available on graduate education can be divided into three broad categories: general findings covering many areas of graduate education; requirements for doctoral degrees; and the status of graduate students. In reviewing the literature, it is important to distinguish between those studies which substantiate their findings with data from degree recipients; those which report on data obtained from individuals who are still in the degree process; those which deal with manpower needs; and those which report anecdotal criticism of graduate education. 119 The Delphi Method is helpful as part of a depart- ment's continuing process of evaluation. Areas of greatest agreement or disagreement are identified, and analysis of the group-by-group ratings reveal which constituencies are most dissatisfied. Furthermore, analysis of the ratings of each statement, within broad categories, pro- vides greater insight into the nature of the agreement or disagreement. Clearly, knowledge of this kind is invalu- able to those responsible for planning a department's future as well as providing day-to-day direction. An opinion survey, such as this study, provides more up-to-date criteria for assessing quality than do the traditional objective measures. Departments can live on their reputations much more easily and for longer periods of time than can most business firms. A depart- ment may continue to attract students, to retain an expert or two, and to raise faculty salaries, and yet be declining in quality either relatively or absolutely. The reverse is true--a department which is making rapid strides may find its reputation lagging by a decade or two. Graduate Education There is widespread concern for systematic evalu- ation of all graduate education. A characteristic mode of graduate education evolved early and has persisted; yet this same mode has been continually subjected to 120 evaluation and assessment. As early as 1925, professors were asked to rate the quality of graduate departments, and similar ratings have been collected periodically ever since. Three major studies of quality in graduate edu- cation have been made during the last forty years. More recently, many agencies and organizations have been making their own evaluations. Graduate educators and researchers should rethink the relationships between disciplines, program purposes, and proqram assessment. Research data indicates that it is no longer true, if it ever was, that all doctoral programs are designed to train research scholars or even teacher-scholars. The old model of research eminence is insufficient for judgments about academic excellence. The nine hundred year history of the university and the roughly century old history of American graduate programs have both been marked by surges and retreats. They have both been marked by complex readjustments, usually long after the need for readjustment had become clear. The several years ahead are going to require many such adjustments, many new arrangements, many new alignments, and much change, but the complexity of graduate education in general has gotten so great that we are in no position to sit back and let them proceed at their historically slow pace. 121 Higher Education Higher education as a field of graduate study is feasible and defensible. One of the assumptions of this investigation is that Higher Education is considered a field of study and therefore subject to assessment. Higher Education as a degree-granting program has reached significant dimensions in offerings, in specialties pro- vided, in degrees available, in faculty, in degrees already awarded, and in enrollment. Regardless of the skepticism expressed in many quarters, doctoral programs in Higher Education are defensible and feasible, albeit requiring hard work and adequate expenditure of funds. A clarity of program purpose and plan is the major concern in evaluating doctoral programs in Higher Education. This concern for the nature of the study of Higher Education is reflected by the numerous investi- gations into program content and course offerings. As the information from this study suggests, chairmen and authorities agree that clarity in purpose, goals, and objectives is one of the most prevalent concerns in Higher Education today. One reason for the concern over program clarity is the manner in which the program requirements are perceived. For instance, the partici- pants agreed that there must be a flexibility of program requirements sufficient to meet the needs of each student. On the other hand, courses should be appropriate to the 122 specialty training of the faculty. This sample of experts agreed that opportunities for specialization need to exist within the parameters of clearly thought-out objectives to be achieved by degree candidates. A philosophy which says programs should be different for each student but keep within certain patterns has ramifications extending into the many facets of communicating program results. Student satisfaction with degree programs is continuing to be important as a measure of qualigy. An evergrowing quantity of research into Higher Education has utilized data from students and degree recipients. Many such studies have focused on satisfaction with the degree program and on occupation status. Most of the par- ticipants in this study thought a high congruence of student career interest with program purpose and emphasis was of first importance in assessing in terms of students. The success in placement of graduates and their rise to positions of leadership has become an important measure of qualigy. The study of Higher Education appears to be increasingly popular, and institutions appear to experience no great difficulty in attracting applicants. Yet the employment market is less stable. How each department responds to these criteria will be indigenous to that department, but the criteria should be considered carefully. 123 The scholarly and research competency_of faculty members should be of the highest quality. Clearly the participants agree that if doctoral programs in Higher Education are not to be considered inferior intellec- tually, an adequate number of faculty members who have demonstrated both research and scholarly competence should be present. A graduate proqram in Higher Education must have faculty members who have demonstrated competency and competence in directing doctoral level research. It would be in the interest of all departments, and of Higher Education in general, to insure that disser- tations produced under their auspices are of the highest quality. While some departments appear to have established a policy and/or tradition of placing high value upon dis- sertations, others show little interest and consider the dissertation only a learning experience. Implications of these diverse opinions regarding dissertation quality have a wide-ranging impact for Higher Education within the graduate education community. For a rapidly expand— ing field, the value of the dissertation content may be quite transitory and may not be amenable to further exploitation. However, for many segments of the academic community, the quality of dissertation work is an unquestionable criterion and offers the best clue to whether a program has attained status as a scholarly field. 124 The adequacy of relevant library holdings is a good indicator of program quality. A majority of both chairmen and authorities agree that the library holdings and facilities meeting the needs of program participants is critical to quality assessment. This "library resource index" has consistently been used in evaluating the quality of graduate education, and is considered one of the best nonhuman factors closely related to assessing quality. Adequate financial support is essential to achiev- ing and maintaining quality in a Higher Education program. Participants unanimously agree that the institution should support with institutional funds a core faculty which can personify and give cohesion to a program. In addition, they endorsed the notion that while external funding is considered important in certain areas of research and service, reliance on it is deemed unwise. A decision to support a quality program in Higher Education would appear to be expensive, but without sound funding the program would likely remain ephemeral. Summary For many investigators, the examination of quality has a variety of aspects and ramifications. It is clear that assessment of quality in graduate education should be sensitive to the diversity of professional and social roles for which doctoral students are preparing. Within 125 this framework, specific criteria for evaluating graduate programs in Higher Education are difficult to determine. Some information derived from this investigation suggests that the criteria for assessment must be responsive to the mix of objectives of a program being evaluated. As the objectives of programs vary in relative importance, so too must the criteria used in judging success in meet- ing the defined objectives. The catalogue of criteria developed here is useful to indicate the parameters for evaluating graduate pro- grams in Higher Education. Beyond this, any greater specificity that is developed should reflect a particular program in a given department. The present study does not pretend to reflect all aspects of evaluating quality or its absence, but it provides an approximate view by fifteen people repre- senting a cross-section of the field. Recommendations Having developed and executed a survey, a final task remains. A social scientist who concentrates on a subject night and day for several months incurs an obligation, if he believes the topic worthwhile, to develop recommendations for improvement of the system studied and the method of study utilized. What contributions have been made to the history of Higher Education? By whom and for what purposes has 126 this history been made? These questions suggest an intensive study into the history of Higher Education is needed. Related to this matter of history and image is the problem which all students and departments of Higher Education face: the definition of the study of Higher Education. Once the parameters of that domain have been established, a department will better be able to inter- pret to its institution and to the larger public the nature of the professional study of Higher Education and the particular role accepted by the department. What are the objectives and learning experiences of doctoral programs in Higher Education? If Higher Education is a specialized field of study, specific objectives, related courses, and learning experiences should be developed. Every Higher Education department or program should undertake its own evaluation. Such a study should not limit itself just to surveys of graduates or current students. Consensus of opinions concerning quality should be obtained from the faculty members and the department chairmen. Other groups, such as faculty members, students, and professional organizations might have similar or dif— ferent opinions about the criteria for judging quality. Consensus across groups as well as among groups should be determined. 127 Discussion of the results of such a study of graduate Higher Education quality by a group of experts in the field would be useful. A special group might be gathered to discuss the results, or such discussions might be initiated at scheduled professional or associ- ation meetings. The summary of criteria could be used as a frame- work for a model to assess quality in graduate Higher Education programs which could be tested empirically. Two recommendations that go beyond the scope or intent of this study are offered for consideration. A clearing house should be established for the dissemination of information about dissertations, mono- graphs, and other research being conducted by faculty and students at Big Ten institutions. Such a service would prove a valuable resource to faculty and students and would not be too overburdensome a task to be carried out by institutions on a regular or rotational basis. The quality of students enrolling in graduate programs in Higher Education in the 19703 by comparison with the 19603 is another subject in need of research. Most studies have changes in numbers and distribution of graduate students, but do not assess any shifts in stu- dent quality that may be occurring. While one might hope that the process of contraction experienced in recent years has not reduced the enrollment of the most 128 able students, we cannot assume this to be the case. A study focused on enrollment trends by selected measures of student quality would be most valuable. Once the criteria for evaluation have been deter- mined, the process of implementation requires certain considerations. In implementing such an evaluative pro- cess, the department or evaluator should ask participants to respond in three ways to each possible measure: 1. Rate its adequacy or appropriateness as an indi- cator of the quality of the listed proqram char- acteristic, using a weighted scale from "very good" to "inadequate"; 2. Indicate one measure for each characteristic preferred by the respondent; 3. Indicate whether the information is currently available, not available, or whether availability varies from area to area. In addition, respondents should be asked to determine the interdependency of criteria. For example, is institutional support being deemed adequate to program purpose dependent upon there being a clarity of program purpose and plan articulated by the department? Also, participants should assess the relationship between program clarity and the congruence of student interest 129 with that program purpose. This example illustrates the necessity for evaluators to consider the criteria as a whole rather than five separate categories. EPILOGUE Often the results of a study such as this one tend to be "writ in stone." The author is not so presumptuous as to assume these pages will be so honored, yet cri— teria for assessing quality can become a locked system and fail in the purpose for which they are intended. The following essay attributed to Francis Bacon is offered to those who would develop and use criteria in the assessment of quality in graduate education: In the year of our Lord 1432, there arose a grievous quarrel among the brethren over the number of teeth in the mouth of a horse. For thirteen days the dis- putation raged without ceasing. All the ancient books and chronicles were fetched out, and wonderful and ponderous erudition, such as was never before heard of in this region, was made manifest. At the beginning of the fourteenth day, a youthful friar of goodly bearing asked his learned superiors for permission to add a word, and straightaway, to the wonderment of the disputants, whose deep wisdom he sore vexed, he beseeched them to unbend in a manner coarse and unheard-of, and to look in the open mouth of a horse and find the answer to their questionings. At this, their dignity being grievously hurt, they waxed exceedingly wroth; and, joining in a mighty uproar, they flew upon him and smote him hip and thigh, and cast him out forthwith. For, said they, surely Satan hath tempted this bold neophyte to declare unholy and unheard-of ways of finding truth contrary to the teachings of the fathers. After many days of grievous strife, the dove of peace sat on the assembly, and they as one man, declaring the problem to be an everlasting mystery because of a grievous dearth of historical and theological evidence thereof, so ordered the same writ down. 130 APPENDICES APPENDIX A PARTICIPATING INSTITUTIONS APPENDIX A PARTICIPATING INSTITUTIONS University of Illinois Urbana-Champaign Indiana University Bloomington University of Iowa Iowa City Michigan State University East Lansing The University of Michigan Ann Arbor University of Minnesota Minneapolis The Ohio State University Columbus University of Wisconsin-Madison Madison 131 APPENDIX B PARTICIPATING AUTHORITIES APPENDIX B PARTICIPATING AUTHORITIES Dr. K. Patricia Cross Visiting University Professor Office of Vice-President for Academic Affairs University of Nebraska Dr. Paul L. Dressel Assistant Provost Michigan State University Dr. Lyman A. Glenny Center for Research and Development in Higher Education University of California-Berkeley Dr. Fred F. Harcleroad Committee on Higher Education University of Arizona Dr. Algo Henderson Center for Research and Development in Higher Education University of California-Berkeley Dr. Lewis B. Mayhew School of Education Stanford University Dr. Dyckman Vermilye Executive Director American Association for Higher Education 132 APPENDIX C COVER LETTERS AND FIRST OPINIONNAIRE MICHIGAN STATE UNIVERSITY ..___o COUJ‘CI‘I OF EDUCATION l~’\Sl' [ANSINO ° Mlt‘HIUAN ° 4882-! DEPARTMENT OF ADMINISI'RNI'ION AN!) IIIGIIIR H)U(‘All()N liRICKSON "All APPENDIX C COVER LETTERS AND FIRST OPINIONNAIRE January 16, 1976 Dr. Fred F. Harcleroad American College Testing Program P.0. Box 168 iowa City, Iowa 52240 Dear Dr. Harcleroad: High quality is the goal of every Ph.D. program. Yet, little is known about how to assess quality or even what to assess. In this study, an attempt will be made to narrow this knowledge gap by identifying characteristics related to Ph.D. program quality. Surely this information will be of interest to members of the Higher Education community, given current pressures for program evaluation at the graduate level. I encourage your participation. Yours truly, Van C. Johnson, Chairman Department of Administration and Higher Education VCJ/am 133 MICHIGAN STATE UNIVERSITY 134 ._._.---—.-— ....._ -.—.____.._._____. ._——.-..— (LOI ”'16!" ()I‘ HNM'A'IIUN I‘ASI IANSIN‘. ' MK MILAN ' “WM IN‘I‘AR'IMI‘NI‘ Ul' AIDMINLS'I'RAIIUN ANI) lll(.lll'R ||)l|(.Al'l()N I‘RICKSON “All January 16, 1976 Dr. Fred F. Harcleroad American College Testing Program P.0. Box 168 Iowa City, Iowa 52240 Dear Dr. Harcleroad: Would you believe just 20 minutes ?!?3 That is all; just 20 minutes of your time to participate in a graduate degree program study in the field of Higher Education. It will take 2 minutes to read this cover letter and approximately 15 minutes to complete the questionnaire. This study, as part of a dissertation, is an attempt to identify characteristics related to quality in doctoral programs in the field of Higher Education at Big Ten Institutions. The study is designed to provide information, based on the extent of agreement among department chairmen and recognized authorities, about doctoral program characteristics most important to judgments about quality. The study, employing a modified Delphi Method, is designed as a three-stage mail survey. The mailing list consists of Higher Education Department Chairmen at Big Ten Institutions offering the program and seven recognized authorities in the field. The first questionnaire, enclosed, groups characteristics of Higher Education programs under five major headings: Students, Personnel, Program, Finance and Facilities. Each participant is asked to rank each characteristic in order of importance. Respondents are invited to add items they feel important which have been omitted. Program characteristics listed under each heading were identified through a review of Higher Education, graduate education and evaluation literature, and discussion with others knowledgeable about graduate education. Helpful materials include: Barak, 1973; Berelson, 1960; Blackburn & Lingenfelter, 1972; Brown, 1970; Cartter, 1966; Clark, 1974; Dressel & Mayhew, 1974; Ewing, 1963; Harcleroad, 1972; Heiss, 1970; and Higgins, 1968. 135 Dr. Fred F. Harcleroad January 16, 1976 Page 2 The second questionnaire will be built on the re3u1ts of the first one, arranging characteristics under each of the five headings in the order of their rated importance to quality and eliminating some of the lower-rated characteristics. The third and final questionnaire will be a further refinement of the second using the-same procedures. The success of this project depends upon your participation since only a total of 15 participants are involved. Take the final minute to return the first questionnaire in the enclosed stamped, self-addressed envelope. Please return by January 26, 1976. Thank you for your time and cooperation. Sincerely, Albert Lynd 136 THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS PURPOSE This study will attempt to identify characteristics related to quality in doctoral programs in Higher Education. The study is designed to provide information, based on the extent of agree— ment among department chairmen and acknowledged authorities, about doctoral program characteristics most important to judgments about quality. Within each of the characteristics in important. Please you feel should be INSTRUCTIONS five catagories presented, please rank the order of importance, with #1 being most add any additional statements or characteristics considered. RETURN By: January 26, 1976 To: Albert Lynd 40l-4D Erickson Hall Michigan State University East Lansing, Michigan 48824 Stamped, self-addressed envelope enclosed PLEASE COMPLETE FOR 2nd AND 3rd STAGE FOLLOW-UP Name: Institution: Address: 137 THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS In your Opinion, how adequate or appropriate is each character- istic as an indicator of a program's quality? Please make your ratings by writing a number in the blank to the left of each item, with #1 as most important. A. STUDENTS The general academic ability of students entering the program is determined by scores of GRE, MAT, etc. There is a high congruence of student career interest with program purpose and emphasis. A high level of excellence and uniqueness of dissertations is characteristic of student achievement. A sense of community and involvement in worthwhile activities is shared by students. Graduates now occupy positions of leadership and influence in the field. Degree recipients in the last three years were placed in positions directly relevant to their graduate education. Other: Other: 138 PERSONNEL All members of the academic staff hold the Ph.D. or Ed.D. degree. A11 faculty are engaged in research activity. All faculty have been represented by publication in the past three years. All faculty members are experienced in teaching at the college or university level in appropriate areas of Speciality. 'Ihere is a high degree of involvement by the academic staff in departmental program affairs. The faculty exhibits satisfaction with program leader- ship, enthusiasm for and loyalty to the program. Teaching effectiveness is high as determined by student and graduate evaluation. There exists a strong committment among the faculty to service: institutional, state, regional and national. Faculty elsewhere in the institution who are interested in Higher Education are utilized. Teaching loads reflect concern for adequate time for teaching, advising and research. Other: Other: 139 PROGRAM Both the Ph.D. and Ed.D. degrees are offered. There is a clarity of program purpose and plan. There is a Flexibility of program requirements sufficient to meet individual student needs. There is a clarity of specified competencies and qualities expected of graduates preparing to be researchers, teachers or administrators. There is a core requirement for all students. Course and related experiences are appropriate to the purposes of the program and speciality training of the faculty. Admission policies are clear as to procedures and standards. The program enrolls a large number of students. There is a high quality of leadership and decision provided by the department chairman. Relationships and interchanges with cOgnate programs are maintained on a regular basis. Other: Other: U. 140 FINANCE «nu—~— Institutional support is deemed adequate to program purpose. External support for the program exists in the form of research project grants and contracts. A majority of students are receiving some form of financial assistance. There is a positive ratio of program budget allocation to the total university allocation for doctoral study. Other: Other: FACILITIES There is an adequacy of relevant library holdings. Facilities and equipment considered essential or important are present. Support services adequately meet the needs of the program. Computer facilities are adequate for the needs of the program. There is adequate instructional space per FTE student for the program. There is adequate office and research space per FTfi faculty for the other. Other: Other: APPENDIX D SECOND OPINIONNAIRE APPENDIX D SECOND OPINIONNAIRE THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS PURPOSE This study will attempt to identify characteristics related to quality in doctoral programs in Higher Education. The study is designed to provide information, based on the extent of agree- ment among department chairmen and acknowledged authorities, about doctoral program characteristics most important to judgements about quality. INSTRUCTIONS FOR STAGE TWO Please do not be concerned with your first rankings. Note the modal ranking for the characteristics within each of the five categories. Please rank the characteristics again, in order of importance, with #1 being most important. If your second ranking differs from the modal ranking, please indicate, if possible, one or two reasons for your choice. RETURN By: February 16, 1976 TO: Albert Lynd 401-4D Erickson Hall Michigan State University East Lansing, Michigan 48824 Stamped, self-addressed envelope enclosed Name: Institution: Address: 141 142 THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS Please do not be concerned with your first rankings. Note the modal ranking for the characteristics within each of the five categories. Please rank the characteristics again, in order of importance, with #1 being most important. If your second ranking differs from the modal ranking, please indicate, if possible, one or two reasons for your choice. A. STUDENTS There is a high congruence of student career 5 interest with program purpose and emphasis. a Graduates now occupy positions of leadership and influence in the field. A sense of community and involvement in worthwhile activities is shared by students. A high level of excellence and uniqueness of dissertations is characteristic of student achievement. i Degree recipients in the last three years were placed in positions directly relevant to their e graduate education. The general academic ability of students entering the program is determined by scores of ORE, MAT, GPA, etc. Other: Other: ‘mcog B., 143 PERSONNEL Teaching effectiveness is high as determined by student and graduate evaluation. Teaching loads reflect concern for adequate time for teaching, advising and research. There is a high degree of involvement by the academic staff in departmental program affairs. All faculty are engaged in research activity. The faculty exhibits satisfaction with program leadership, enthusiasm for and loyalty to the program. Faculty elsewhere in the institution who are interested in Higher Education are utilized. There exists a strong committment among the faculty to service: institutional, state, regional and national. All faculty members are experienced in teaching or administration at the college or university level in appropriate areas of specialty. All faculty have been represented by publication in the past three years. All members of the academic staff hold the Ph.D. or Ed.D. degree. Other: Other: 144 C . PROGRAM There is a clarity of program purpose and plan. There is a flexibility of program requirements sufficient to meet individual student needs. Course and related experiences are appropriate to the purposes of the program and speciality training of the faculty. There is a clarity of specified competencies and qualities expected of graduates preparing to be researchers, teachers or administrators. Relationships and interchanges with cognate programs are maintained on a regular basis. Admission policies are clear as to procedures and standards. There is a high quality of leadership and decision provided by the department chairman. There is a core requirement for all students. Both the Ph.D. and Ed.D. degrees are offered. The program enrolls a large number of students. Other: Other: D. 145 FINANCES Institutional support is deemed adequate to program purpose. There is a positive ratio of program budget allocation to the total university allocation for doctoral study. External support for the program exists in the form of research project grants and contracts. A majority of students are receiving some form of financial assistance. Research and teaching assistantships available from institution funds. A high proportion of the budget is from hard, not soft, money. Students work with faculty research projects financed by soft money. Funds are available for visiting scholars. Other: Other: 146 E. FACILITIES There is an adequacy of relevant library holdings. Facilities and equipment considered essential or important are present. Support for services adequately meet the needs of the program. There is adequate instructional space per FTE student for the program. There is adequate office and research space per FTE faculty for the program. Computer facilities are adequate for the needs of the program. Other: Other: APPENDIX E THIRD OPINIONNAIRE APPENDIX E THIRD OPINIONNAIRE THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS PURPOSE This study will attempt to identify characteristics related to quality in doctoral programs in Higher Education. The study is designed to provide information, based on the extent of agree— ment among department chairmen and acknowledged authorities, about doctoral program characteristics most important to judgements about quality. INSTRUCTIONS FOR STAGE THREE This third and final stage includes your second stage rankings and a new modal ranking for the characteristics within each of the five categories. Reasons for differing from the first modal responses are summarized. These reasons are listed under two headings, one for those contending a goal should be "more important" than the first modal rating, the other for those contending a goal should be "less important" than the first modal rating. Please glance at your second rating, the modal ranking and the reasons some participants did not agree, then rank for the last time the characteristics in order of importance, with #1 being most important. RETURN By: March 22, 1976 To: Albert Lynd 401-4D Erickson Hall Michigan State University East Lansing, Michigan 48824 Stamped, self-addressed envelope enclosed Name: Institution: Address: 147 .‘- u . u. . 148 THE IDENTIFICATION OF CRITERIA FOR EVALUATING GRADUATE PROGRAMS IN HIGHER EDUCATION AT BIG TEN INSTITUTIONS This third and final stage includes your second stage rankings and a new modal ranking for the characteristics within each of the five categories. Reasons for differing from the first modal responses are summarized. These reasons are listed under two headings, one for those contending a goal should be "more important" than the first modal rating, the other for those contending a goal should be "less important" than the first modal rating. Please glance at your second rating, the modal ranking and the reasons some participants did not agree, then rank for the last time the characteristics in order of importance, with #1 being most important. A. STUDENTS 2nd 3rd r—I W603 There is a high congruence of student career interest with program purposes and emphasis. 2 Graduates now occupy positions of leadership and influence in the field. Less Important: more often a result of selection than educational quality. 3 A sense of community and involvement in worthwhile activities is shared by students. More Important: 3 more contemporary measure than the number of graduates in leadership positions. Less Important: students are expected to work together but not necessary to program quality. 4 A high level of excellence and uniqueness of dissertations is characteristic of student achievement. More Important: single best clue to program quality and affects future hirings and promotions. Less Important: development of outstanding research skiIls is not necessary to attain a leadership position. S Degree recipients in the last three years were placed ‘ ———_— F_- o o o o 0 1n p051tions directly relevant to their graduate education. B. PERSONNEL 149 H P>fl 3:51. Teaching effectiveness is high as determined by student and graduate evaluation. Less Important: used by almost everyone but with little confidence in its accuracy. Teaching loads reflect concern for adequate time for teaching, advising and research. Less Important: essential but incidental. There is a high degree of involvement by the academic staff in departmental program affairs. More Important: the first order of business is creafing an excellent program. All faculty are engaged in research activity. Less Important: some faculty are synthesizers not researchers; has little relation to teaching effectiveness. The faculty exhibits satisfaction with program leadership, enthusiasm for and loyalty to the program. Faculty elsewhere in the institution who are interested in Higher Education are utilized. More Important: should utilize "outsiders" to avoid’thinking of Higher Education as a closed discipline. C. 2nd PROGRAM N l... r>coz 3:4 150 There is a clarity of program purpose and plan. There is a flexibility of program requirements sufficient to meet individual student needs. Course and related experiences are appropriate to the purposes of the program and speciality training of the faculty. There is a clarity of specified competencies and qualities expected of graduates preparing to be researchers, teachers or administrators. Less Important: difficult to know what the competencies are. Relationships and interchanges with cognate programs are maintained on a regular basis. More Important: important to prevent Higher Education from becoming provincial. D. 2nd FINANCES 151 H F>UOZ 32a E. 2nd Institutional annmrt is deemed adequate to program purpose. There is a positive ratio of program budget allocation to the total university allocation for doctoral study. Less Important: not necessary for Higher Education dectoral programs to be as expensive as those in other areas. External support for the program exists in the form of research project grants and contracts. A majority of students are receiving some form of financial assistance. More Important: essential to recruit strong students. Research and teaching assistantships available from institution funds. FACILITIES M L“>DO H 3:2 There is an adequacy of relevant library holdings. Less Important: good "off-campus" students can be resourceffiI in locating materials. Support for services adequately meet the needs of the program. Facilities and equipment considered essential or important are present. Computer facilities are adequate for the needs of the program. There is adequate instructional space per FTE student for the program. APPENDIX F RESPONSE BY PARTICIPANTS APPENDIX F RESPONSE BY PARTICIPANTS TABLE 6 RESPONSE BY PARTICIPANTS Participants First Stage Second Stage Third Stage Institutions 6 7 8 Authorities 7 7 6 Total 13 14 14 Percentage 87 93 93 152 APPENDIX G COMMENTS APPENDIX G COMMENTS As part of the second stage of this study, par- ticipants were called to rank characteristics in order of importance. If their responses differed from the consensus responses, they were asked to indicate, if possible, their reasons. These minority opinions were listed under two headings--one for those participants contending a characteristic should be "more important" than the modal rating, and the other for those participants contending a characteristic should be "less important" than the modal rating. In addition to these requested responses, the participants added their comments about characteristics and, in some instances, commented on each other's minority opinions. Providing supporting evidence that the Delphi Method can bring forth honest comments about the topic, the following are direct quotes from the participants: 153 A. B. Students Characteristic: Characteristic: Personnel Characteristic: 154 Graduates now occupy positions of leadership and influence in the field. Comment: The time frame makes this a question— able criterion. People out ten years may be in strong positions but the quality of the program may have changed markedly. A sense of community and involvement in worthwhile activities is shared by students. Comment: This statement is ambiguous. If it means campus activities, the criterion is not so important at the graduate level. If it relates to overall pur— pose in life and commitments to society, then it is of high importance. Faculty elsewhere in the institution who are interested in Higher Education are utilized. Comment: Should utilize outsiders to avoid thinking of Higher Education as a closed discipline. Comment on comment: Higher Education is inclusive of the total role of the college or uni- versity, and the participation should express this fact and nourish the program. C. D. Program Characteristic: Characteristic: Finances Characteristic: 155 There is flexibility of program requirements sufficient to meet individual student needs. Comment: Sounds well, but can lead to omission or avoidance of essential elements in preparation for educational leadership; apt to lead toward emphasis on occupational training, not appropriate at the doctoral level. There is a clarity of specified competencies and qualities expected of graduates preparing to be researchers, teachers or adminis- trators. Opinion: Less important. Difficult to know what the compe- tencies are. Comment on the opinion: 1. However they must be described in part. 2. We had better find out! A majority of students are receiving some form of financial assistance. Opinion: More important. Essential to recruit strong students. Comment on the opinion: Probably a realistic statement but why it becomes more important is hard for me to see. 156 E. Facilities Comments concerning all characteristics: 1. I still do not feel these can be ranked. They are all important. 2. I can't believe I ranked these in this manner. One participant expressed his appreciation for the opportunity to participate in this study: "This was an interesting project. Thanks for the opportunity to participate." SELECTED BIBLIOGRAPHY SELECTED BIBLIOGRAPHY Alciatore, Robert T., and Eckert, Ruth. Minnesota Ph.D.‘s Evaluate Their Training. Minneapolis: University of Minnesota Press, 1968. An Annotated Bibliography on Graduate Education, 1971- I972. Washington, D.C.: National Boardgon Graduate Education, 1972. Balderston, Frederick E. Managing Today's University. San Francisco: Jossey-Bass, 1974. Barak, Robert J. Graduate Study Programs in the Field of Higher Education. Washington, D.C.: American AssoCiation fer Higher Education, 1973, ERIC 085044. Barzun, Jacques. Graduate Study at Columbia. New York: Columbia University Press, 1958. Ben-David, Joseph. American Higher Education: Directions Old and New. New York: McGraw-Hill, 1972. Berelson, Bernard. Graduate Education in the United States. New York: McGraw-Hill, Co., 1960. Bissell, Claude Thomas. The Strength of the Universipy. Toronto: University ofTToronto Press, 1968. Blackman, Robert T., and Lingenfelter, Paul E. Assessing Quality in Doctoral Programs: Criteria and Correlates of Evaluation. Ann Arbor: Center for the Study of Higher Education, The University of Michigan, February, 1973, ERIC 078728. Blessing, James H. An Annotated Bibliography on Graduate Education. Washington, D.C.: National Academy of Seience, July, 1971, ERIC 058830. . Graduate Education, An Annotated Bibliography. Washington, D.C.: Office of Education, 1961, ERIC 095728. 157 158 Boyer, Calvin J. The Doctoral Dissertation as an Infor— mation Source. Metuchen, N.J.: The Scarecrow Press, Inc., 1973. Bowker, Albert. "Quality and Quantity in Higher Edu- cation." Journal of the American Statistical Association 60 TMarch 19657. Breneman, David W. Graduate School Adjustments to the "New Depressidn1r in Higher Education. Washington, D.C.: National Board on Graduate Education, 1975, ERIC 101643. . The Ph.D. Degree at Berkeley: Interviews, Placement, anleecommendatidhs. Berkeley: Ufiiversity of California, 1971. Brice, Bert Charles. "The Doctoral Program in Higher Education at North Texas State University: An Appraisal." Ph.D. dissertation, North Texas State University, 1970. Brown, D. G. "A Scheme for Measuring the Outputs of Higher Education." In The Outputs of Higher Education: Their Identification, Measurement and EvaIuation. Boulder, Cdiorado: Western Interstate Commission for Higher Education, July, 1970. Burnett, Collins W. "Higher Education as a Specialized Field of Study." Journal of Research and Development 6 (1973): 4-15. Calvert, Jack G.; Pitts, James N., Jr.; and Dorton, George H. Graduate School in the Sciences: Entrance, Survival, and Careers. New York: Wiley-Interscience, 1972. Carmichael, Oliver C. Graduate Education: A Critique and a Program. New York: Harper and Brothers, 1961. Carr, William D. "Doctoral-Level Graduates with Higher Education as a Field of Study." In Higher Edu- cation: A Developing Field of Study. Edited by Fred J. Harcleroad. Iowa City: American College Testing Program, 1974. Cartter, Alan M. An Assessment of Quality in Graduate Education: A Comparative Study of Graduate Departments in Twenty-Nine Academic Disciplines. Washington, D.C.: AmerICan Council on Education, 1966. 159 Cartter, Allan M. Ph.D.‘s and the Academic Labor Market. New York: McGraw-Hill, 1976. Clark, Mary Jo. The Assessment of Quality in Ph.D. Programs: A Preliminary Report on Judgments by Graduate Deans. Princeton, N.J.: EducationaIi Testing Service, October, 1974. Crane, Diana. "Scientists at Major and Minor Universities: A Study of Productivity and Recognition." American Sociological Review 30 (October 1965). Currie, Andrew C. "An Investigation and Identification of Higher Education as a Graduate Field of Study and Research.” Ph.D. dissertation, Ohio State University, 1968. DeMott, Benjamin. "Reforming Graduate Education." Change 6 (February 1974): 25-29. Dibden, Arthur J. "A Department of Higher Education: Problems and Prospects." Educational Record 46 (1965). Dressel, Paul L., Johnson, F. C., and Marcus, P. M. The Confidence Crisis. San Francisco: Jossey- Bass, 1970. Dressel, Paul L., and Mayhew, Lewis B. Higher Education as a Field of Study. San Francisco: Jossey- Bass, Co., 1974. Elder, J. P. A Criticism of the Graduate School of Arts and Sciences at Harvard University and Radcliffe College. Cambridge: Harvard Ufiiversity Press, Eshelman, James N., ed. New Directions in Graduate Education. Proceedings of the SevenEh Annual Meeting of the Council of Graduate Schools. Washington, D.C.: Council of Graduate Schools, 1967. Ewing, John C. "The Development and Current Status of Higher Education as a Field of Graduate Study and Research in American Universities." Ph.D. dissertation, Florida State University, 1963. , and Stickler, W. Hugh. "Progress in the Develop— ment of Higher Education as a Field of Professional Graduate Study and Research." The Journal of Teacher Education 15 (1964). 160 Fleming, Robbin W. Meeting the Needs of Doctoral Edu- cation in New York. Report of the Commission on Doctoral Education in New York. Albany: New York State Board of Regents, 1973, ERIC 083936. Glass, John F., and Glass, Judith. "Improving Graduate Education." Educational Forum 32 (May 1968): 439-46. Gottlieb, David. "Process of Socialization in American Graduate Schools." Social Forces 40 (December 1961). Graduate Education--Purposes, Problems, and Potentials. Washington, D.C.: National Board on Graduate Education, 1972. Greenburg, Daniel S. "Top Graduate Schools Holding Their Own." Change 6 (September 1974): 47-48. Harcleroad, Fred F., ed. Higher Education: A Developing Field of Study. Iowa City, Iowa: The American College Testing Program, 1974. Harvey, James. The Student in Graduate School. Washing- ton, D.C.: American Association for Higher Education, January, 1972. Hatch, Winslow. "What Standards Do We Raise?" New Dimensions in Higher Education. WashingEdH, D.C.: United States Department of Health Edu- cation and Welfare, 1964. Heckman, Dale, and Martin, Warren Bryan. Inventory of Current Research on Postsecondary Education. New York: McGraw-Hill, Co., 1972. Heiss, Ann M. "Berkley Doctoral Students Appraise Their Academic Programs." The Educational Record, Winter, 1967. . Challenges to Graduate Schools. San Francisco: Jossey-Bass, 1970. Henderson, Algo D. The Innovative Spirit. San Francisco: Jossey-Bass, 1970. , and Henderson, Jean G. Higher Education in America. San Francisco: Jossey-Bass, 1971. 161 Henry, David D. Challenges Past, Challenges Present. San Francisco: Jossey-Bass, 1975. Higgins, A. Stephen. "The Rating of Selected Fields of Doctoral Study in the Graduate Schools of Edu- cation: An Opinion Survey." Ph.D. dissertation, Columbia University, 1968. . "Reflections of Quality. A Longitudinal Assessment of Doctoral Programs in Education." Ph.D. dissertation, New York University, 1968. Higher Education as a Field of Study. Washington, D.C.: Association of Professors Of Higher Education, March, 1972, ERIC 076110. Hollis, Ernest V. Toward Improving the Ph.D. Programs. Washington, D.C.: American Coundii on EducaEion, 1945. Hughes, Raymond M. "A Report of a Committee of the American Council on Education." The Educational Record, April, 1934. . A Study of the Graduate School in America. Oxford, Ohio: Miami University,ii925. Johnson, D. Gale. A Rating of Graduate Programs. Washington, D.C.: American Council on Education, 1970. Jordan, Robert T. "Library Characteristics of Colleges Ranking High in Academic Excellence." Colle e and Research Libraries 24 (September 19635. Kellams, Samuel E. Research Studies on Higher Education: A Content Analysis. Charlottesville: Center for Higher Education, University of Virginia, 1974. Keniston, Hayward J. Graduate Study and Research in the Arts and Sciences at Ehe University of Pennsylvafiia. Philadelphia: University of Pennsyivania Press, 1959. Kerr, Clark. The Uses of the University. Cambridge, Mass.: Harvard University Press, 1963. Kidd, Charles V. "Graduate Education: The New Debate." Change 6 (May 1974): 43-50. 162 Margulies, Rebecca 2., and Blau, Peter M. "The Pecking Order of the E1ite--America's Leading Profes— sional Schools." Change 5 (November 1973): 21-27. Mayhew, Lewis B. The Literature of Higher Education 1971. San Francisco: Jossey-Bass, 1971. , and Ford, Patrick J. Reform in Graduate and Professional Education. San Francisco: Jossey- Bass, 1974. Margolis, J. "Citation Indexing and Evaluation of Scientific Papers.” Science 155 (March 10, 1967). Meeting the Needs of Doctoral Education. A Statement oflPolicy and Proposed Action by the Regents of the UniVersity of the State ofiNew York. Albany: New YorEiState Department of Educatidn, August, 1973. Millman, Stephen D., and Toombs, William. The Quality of Graduate Studies: Pennsylvania and Selected States. Center for the Study of Higher Education, The Pennsylvania State University, 1972. More than Survival. San Francisco: Jossey-Bass, 1975. Ortega, y Gasset J. Mission of the University. Prince- ton: Princeton Uhiversity Press,*1944. Palinack, Robert S., et a1. Survey of Requirements for a Doctoral Program in the Field’of Higher Edu- cation. Syracuse: Syracuse University, May, I970. Perkins, Dexter, and Snell, John C. The Evaluation of Historians in the United States. New York: McGraw—Hill, 1962. Peterson, Marvin W. "Form, Function, and Strategic Issues in the Study of Higher Education." Journal of Research and Development in Education 6 (Winter 1973): 16-29. The Purposes and the Performance of Higher Education in the United States. New York: McGraw-Hill Book Company,ii973. Rogers, James F. Higher Education as a Field of Study at the Doctoral Level. Washington, D.C.: American Association of Higher Education, National Edu- cation Association, 1969. 163 Roose, Kenneth D., and Andersen, Charles J. A Rating of Graduate Programs. Washington, D.C.: American Council on Education, 1970. Ross, Naomi. "Characteristics of Several Current Doc- toral Programs and Members of APHE." In Higher Education: A Developing Field of Study. Edited by Fred J. Harcleroad. Iowa City: American College Testing Program, 1974. Sagan, Edgar L. "The Emergence of Higher Education as a Field of Study." Columbus, Ohio: The Ohio State University, 1968. Spurr, Stephen H. Academic Degree Structures Innovative Approaches. New York: McGraw-Hill, Co., 1970. Stallings, William M., and Singhal, Sushila. "Some Observations on the Relationships Between Research Productivity and Student Evaluation of Courses and Teaching." American Sociologist 5 (1970). Storr, Richard J. The Beginning of the Future. New York: McGraw—Hill, 1973. A Study of Florida State University Doctoral Graduates andiTheir Reactions to the Doctoral Program at the University. Tallahassee: Florida State University Press, 1957. Survey of Changes in Graduate Programs in Higher Edu- cation. Wadhington, D.C.: American Council on Education, June, 1972, ERIC 071608. The Survey of Graduate Education at Stanford. Stanford University, June, 1972, ERIC 063854. Travelstead, Will W. "An Appraisal of Degree Programs of Academic Administrators in Higher Education." Ph.D. dissertation, Southern Illinois University, 1974. Tucker, Allan; Gottlieb, David; and Pease, John. Factors Related to Attrition Among Doctoral Students. East LanSing: Michigan State Uni- ver31ty, 1964. Uhl, Norman P. Identifying College Goals the Delphi Way. Durham, N.C.: National Laboratory for Higher Education, 1970. 164 Vandermullen, Darrell S. "The Motives, Perceptions and Aspirations of Recent Graduates of Doctoral Programs in Higher Education." Ph.D. dissertation, Southern Illinois University, 1974. Watters, Everett, ed. Graduate Education Today. Washing- ton, D.C.: American Council on Education, 1965. Whaley, W. Gordon, ed. Association of Graduate School Proceedings, 1966-1969. Austin, Texas: The University of Texas at Austin, 1969. , ed. In These Times. Austin, Texas: The Uni- versity of Texas at Austin, 1971. Wilson, Kenneth M. Of Time and the Doctorate. Atlanta: Southern Regional Education Board, I965. Young, Burus Byron. "The Rise and Development of Instructional Courses in Higher Education." Ph.D. dissertation, Stanford University, 1952. "llllll‘ll'llll‘lllI