- ‘ ‘ ' .‘ .r'i" Z ‘ . : 1 ‘0 .- ‘. . . ‘1. 0...? . .- o _L ‘l " ‘: ‘l ‘ T. . ‘ .,' I . "190"" J . ; L ’l‘ I, . A """’f’” ., i ' . l. p, | , I . s ‘ r 5 H H/ '1’ 1‘ ‘ , F u- I i - ‘ . . ._ C ‘l t‘ 4‘ q I | ‘ ‘. H ' 7 .w'm‘ ,f’ I” in": or" , :' 1 .I l ,o ' u" u If 1 ".0 ‘ 0‘ 'J! r" n' "ma” “0’ ' v 5... ,1“ - . ‘ “O. u 4‘ .' 0". ‘ I . ’ . -¢ 4 I ’1: ”3 “I“. 1 1’ ,O'H'H’ W V l ".h"..' ' . n l ‘ g). ' """luw'. ‘ '1‘ ‘ ' 3 . o x I w I \ ‘ i J .p > I l 1 - )1 c L. .h‘r . ‘ . "'- ' ‘,'"'5~"~'-‘“ ' "‘"L’N‘F'"‘;r2u\'.. .2 ‘ “ v; . II mum; mun Lil! M w mm W! M ll OVERDUE FINES ARE 25¢ PER DAY PER ITEM Return to Book drop to remove this checkout from your record. © Copyright'by MARIA BEATRIZ MOREIRA LUCE 1979 11 AN ASSESSMENT OF CRITERIA TO EVALUATE THE GRADUATE EDUCATION PROGRAM IN BRAZIL By Maria Beatriz Moreira Luce A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY College of Education International and Comparative Education 1979 ABSTRACT AN ASSESSMENT OF CRITERIA TO EVALUATE THE GRADUATE EDUCATION PROGRAM IN BRAZIL By Maria Beatriz Moreira Luce This study sought to identify criteria adequate for the evaluation of graduate programs in Brazil. A survey was the means for collecting the ratings and rankings given by faculty members at selected Brazilian graduate programs. A questionnaire using Likert- type and ranking items asked the importance attributed by each respondent to each of the l09 items listed. The data analysis reported in this dissertation indicates that the most highly rated criteria and indicators were: (l) Library: current periodicals; (2) Facilities: classrooms and laboratories; (3) Library: books and monographs; (4) Academic Environment: discus- sion, investigation, and expression; and (5) Facilities: research space and equipment. The study presents the means and standard deviations obtained for each indicator and also includes some figures obtained for a relational analysis. This dissertation was developed to provide useful information to educational planners, policy makers, administrators, and evaluators Maria Beatriz Moreira Luce involved in Brazilian higher education or comparative studies. It is suggested that additional investigations concentrate on more specific and in-depth analysis and interpretation of the policy- making processes, i.e., on the study of social facts or organiza- tional and academic variables in their relationships with aspects of the educational system. The appendices section includes a facsimile of the question- naire and additional data. ACKNOWLEDGMENTS During the development of this study, many individuals and organizations provided support and assistance to the writer. Unfor- tunately, it is possible to cite here only those making the most significant contributions. Dr. David K. Heenan was an academic advisor during my gradu- ate education and the chairperson of my doctoral guidance committee. I am extremely indebted to him for his continuing assistance and incentive; he is a very special example of professional dedication. Dr. Margaret F. Lorimer, a member of the dissertation guid- ance committee, was instrumental in the design and review of this study. I am grateful to her for providing valuable and experienced guidance. Drs. Ted w. Ward, Walter F. Johnson, and Donald A. Taylor, who also served on the guidance committee, contributed with many important suggestions. My appreciation for their time, concern, and support. Financial support for my doctoral program and the survey was provided by CAPES (Coordination for Training Higher Education Personnel, at the Brazilian Ministry of Education and Culture). I especially thank Dr. Darcy Closs, Director, for his interest and personal contri- butions. At the Universidade Federal do Rio Grande do Sul, Dr. Juracy C. Marques, Coordinator of the Graduate Program in Education, and iii Professor Gilberto M. Medeiros, Director of the College of Education, offered support for the field work. All the faculty members responding to the questionnaire, the Pro. Rectors interviewed, and the coordinators of the selected graduate programs made possible the completion of the study. My sincere appreciation for their contribution. At Michigan State University, many professors assisted in many ways. The Office of Research Consultants, at the College of Education, facilitated the data organization and analyses. Particu- lar appreciation goes to Swatna for her competent assistance. Mrs. Susan Cooley assisted with typing and final arrange- ments. I am indebted to her for her cooperation. Fernando Bins Luce was my companion and most valuable sup- porter. His encouragement and continuing expertise and assistance at all stages of research deserve my special mention. iv TABLE OF CONTENTS Page LIST OF FIGURES ........................ viii Chapter I. INTRODUCTION ..................... l The Research Focus ................. 3 Purpose and Method ................. 5 Limitations ..................... 7 Procedures ..................... 8 Overview ...................... 9 II. LITERATURE REVIEW ................... ll Evaluation ..................... l2 Evaluation Defined ................ l2 Evaluation Methodology .............. l3 Evaluation Models ................. 15 Evaluation Criteria ................. l8 Academic Evaluation ................. 2l Graduate Evaluation Criteria ........... 23 III. RESEARCH PROCEDURES .................. 48 Research Approach .................. 49 Survey Design .................... 50 Research Variables ................. 51 The DeveTOpment of Instruments ........... 52 Designing the Questionnaire ............ 53 Planning the Interviews .............. 56 Pilot Tests .................... 58 Selection of Subjects ................ 60 Population .................... 60 Sampling ..................... 62 Interviewees ................... 63 Data-Collection Procedures ............. 64 Soliciting Through the Questionnaire ........ 64 Interviewing .................... 65 Preparation of Survey Data ............. 66 Organizing the Survey Data File .......... 67 Chapter Page Methodological Difficulties ............. 71 Data Analysis .................... 73 Descriptive Analysis ............... 74 Relational Analysis ................ 75 IV. SURVEY RESULTS .................... 77 Respondents' Characteristics ............ 78 Academic Position ................. 78 Academic Degrees ................. 78 Professional Meetings ............... 79 Publications ................... 80 Perceptions of Change ............... 8O Perceived Value of Research ............ 8l Academic Activities--Actual and Ideal ....... 81 Evaluation Criteria Assessment ........... 82 Criterion One ................... 83 Criterion Two ................... 84 - Criterion Three .................. 85 Criterion Four .................. 86 Criterion Five .................. 87 Criterion Six ................... 88 Criterion Seven .................. 89 Criterion Eight .................. 9O Criterion Nine .................. 9T Criterion Ten ................... 92 Criterion Eleven ................. 93 Criterion Twelve ................. 94 Criterion Thirteen ................ 95 Criterion Fourteen ................ 96 Criterion Fifteen ................. 97 Exploring Relationships ............... 98 Respondents and Rankings ............. 98 Relationships Among Indicators .......... 99 Overview ...................... lOO Important Criteria and Indicators ......... lOl V. REVIEW AND PERSPECTIVE ................ lO4 Reviewing the Findings ............... lO4 The Respondents .................. l05 The Evaluation Criteria .............. lO7 Integrative Summary ................. 114 A Methodological Perspective ............ l16 vi Chapter Page APPENDICES .......................... IZT A. QUESTIONNAIRE, LETTER TO COORDINATORS, LETTER TO THE SUBJECTS ................... 122 B. CODING PLAN ...................... 141 C. QUESTIONNAIRES DELIVERED AND RETURNED ......... 143 D. TABLES USED FOR QUESTIONS IN PART I OF THE QUESTIONNAIRE .................... I45 BIBLIOGRAPHY ......................... 150 vii LIST OF FIGURES Figure Page l. Variables Related to Quality in Graduate Education Programs ................. 25 2. Data Matrix ..................... 67 viii CHAPTER I INTRODUCTION Graduate studies are a recent development in Brazilian higher education. Academic programs, leading to master's and doctoral degrees, were established in the early l9605. Prior to this period there were limited opportunities to earn a doctoral degree, usually under the tutorial system. Since then, there has been a period of continued growth and consolidation of efforts. Not only do the number of programs, students, and faculty demonstrate this expansion but also the amount of support stemming from private and governmen- tal sectors. In l965, graduate education was formally and officially instituted through an act of the Federal Council on Education which defined its structure and functions (Brasil, l965, 1973). Subse- quently, several legislative and administrative measures were intro- duced to enhance the quality and to expand the number of graduate programs and degrees. Included among these were requirements for admission and promotion of university faculty members, accreditation criteria, and financial support. Recent data show that the majority of graduate programs in Brazil are at the master's level and that most are not fully accred- ited (CAPES, l976). Figures also point out a considerable growth 1 in graduate study opportunities and reflect a vigorous response to increasing social and economic demands. To discipline the proliferation and diversification observed in graduate programs, the Federal Government created, in 1975, the National Council on Graduate Studies (Sousa, l975). Its tasks are to formulate a national policy for graduate education, to establish priority objectives and goals, and to propose an integrated budget according to national development planning. This Council produced the National Plan for Graduate Studies (Brasil, l975) which declared that the training of faculty, research- ers, and high-level professionals should be the primary goal of gradu- ate education. Also, based on a diagnosis of the situation of graduate studies in Brazil, the Plan described three major directives: first, institutionalization of graduate education as a regular and finan— cially stable activity of university programs; second, elevation of efficiency and quality; and, third, planned expansion of study areas and regional capacity. These directives evolved from the nature of problems shaping the development of graduate studies and should guide the action of agencies supporting education and research. In addition to the centralized efforts led by the National Council on Graduate Studies and by the agencies represented in this body,1 other institutions also foster the growth of graduate educa- tion in Brazil. 1Ministry of Education and Culture, Planning Secretariat at the President's Office, Federal Council on Education, Department of University Affairs, Coordination for Training Higher Education Per- sonnel (CAPES), National Council for Scientific and Technological The universities which offer graduate studies have, in gen- eral, sought to further the improvement of their programs. Special emphasis has been devoted to student and faculty personnel policies, to the betterment of facilities, and to administrative collaboration. In most cases, a pro-rectorship of graduate studies and research has been designated and some graduate programs have received planning assistance. The latter called for a definition of scope and functions, a review of admission and graduation processes, and planning for higher efficiency and efficacy in teaching and research. Furthermore, Government and other educational and business institutions interested in human and research outputs from graduate programs have provided a supportive but demanding pressure for quan- tity and quality. Cooperative efforts have been initiated in some fields of graduate studies for planning common goals. The Research Focus The quantitative expansion of graduate education, observed in figures for programs, enrollment, faculty, and financial resources (CAPES, T976), presents a serious challenge for an accountable per- formance to both supporting agencies and the institutions themselves. Thus, the need for a systematic appraisal of operational conditions and quality of outcomes arises. The Federal Council on Education and funding agencies have each develOped some capability for estimating the worth of their Development (CNPq), National Fund for Scientific and Technological Development, and National Bank for Economic Development. accreditation or assistance. Also, some of the universities are developing competence for institutional monitoring of their systems. They are aiming at more accountability to objectives and clientele; and, therefore, to enhance the competitive ability to receive support either from within the organization, or from sponsoring institutions and the community at large. It is generally acknowledged that efforts toward evaluation of Brazilian graduate programs are still at an embryonic stage. Even at a broader level, in the international perspective, a literature search indicated the need for appropriate and more elaborated tech- nology. This study is intended to contribute to the development of research and evaluation studies in Brazilian graduate education. A basic assumption is that an effective and comprehensive evaluation should be the basis for rational judgment in decision-making situa- tions and that the criteria used in the evaluation must be relevant for the specific situation. Therefore, research involving the insti- tutions and persons influencing and affected by the development of a program is required to gather germane information that will help to determine the set of criteria necessary for evaluation or decision making. The primary postulate is that as perceived needs and goals vary, the values identified with quality and productivity also change. That is, although a common set of criteria may be used for evaluation of most graduate programs, the importance or value attributed to each indicator is unique to a given academic community or to a situation.1 Historical, philosophical, social, economic, and political factors determine the values used for planning and evaluating an educational endeavor. The focus of this research is on the first step in the develop- ment of an evaluation process; that is, the identification of criteria for the actual evaluation. The major question addressed is: What are the criteria considered appropriate to evaluate graduate programs, in Brazil? Secondarily, the investigation was aimed at learning the role of several academic and attitudinal characteristics in the values dis- played by the group selected for the study. Criteria developed from the recommendations of various sources were presented to a group of faculty members currently related to graduate programs. Several indicators associated with quality and efficiency of graduate programs were rated and ranked by the respon- dents. Purpose and Method An assessment of criteria suitable for evaluating graduate studies may serve many purposes. One purpose of this study is to enable and stimulate institutions toward formative evaluation and self-study. It may, also, prove useful for planning any comparisons, trend analyses, and appraisals undertaken by outside examiners. The study provides information that can facilitate the iden- tification of criteria capable of holding a program accountable to 1Criterion and indicator are further defined on pages 18 and 19. its own objectives or of estimating its adequacy. It provides to researchers and policy makers a suggestion of program characteris- tics validated by the faculty who participated in the survey. But it does not advocate any arbitrary set of criteria to be met, nor does it gather the opinion of an exclusive source. It merely col- lects and analyzes reactions from a relatively broad segment of faculty members in the graduate education system. The importance attributed to the evaluation criteria and their indicators or other data aggregated through the research instru- ment may be used by a faculty or evaluator to develop the set of criteria most appropriate to the specific situation. The number, nature, and hierarchy of criteria adopted for a given program review should be exclusive and consistent with the purposes for which they are employed. However, the knowledge of level of agreement on a particular indicator, across the study population and for the groups repre- sented, may aid developments in the areas of external evaluation and accreditation. Special benefits are offered to policy making in graduate education. An interest in history and comparative education determined the observation of some international concepts and practices. In addition to testing relationships between academic positions, activities and attitudes of the faculty involved, some attention was given to distinguishing traditional European, American, or any values which may be associated with distinctive characteristics of higher education in other systems. Within the technical scope, the assembled information may provide a base for further research on specific criteria for differ- ent fields and levels, and permit the elaboration of evaluation instruments. There are elements for guidelines, priorities, monitors, or measurements for educational planning, administration, and evalua— tion. Limitations The study and adoption of evaluation criteria are often conceptually and politically difficult tasks. Each society or nation has unique needs and strategies for the solution of its problems. Brazilian graduate education has been influenced by foreign forces through cultural, intellectual, political, and economic factors. One of the restrictions present in this study is not only the particular background of each respondent but also the researcher's values. Although not currently involved in any Brazilian program and seeking impartiality and-objectivity, the researcher may have a cul- tural and professional bias. The researcher is a Brazilian who is pursuing an advanced degree in the United States, specializing in Education. Possibly, a further limitation is the quality of the instru- ment, which was developed exclusively for this investigation from predominantly American literature. While it seems that the evalua- tion process should be the same, there could be some cultural dif- ferences. Restrictions also had to be imposed on the number of mate- rials, subjects, and institutions covered. Economics and time were factors which, likewise, limited the extent of the study. Procedures After determining the need and scope of the study, two basic methodological phases were identified. The first encompassed the gathering of fundamental information and the development of research procedures and instruments. In the second phase, survey and analy- sis of the participants' responses led to the assessment of the cri- teria. A review of the literature and consultation with some Michigan State University specialists provided the base for the construction of research instruments. A self-administered questionnaire requested faculty to respond to the importance attributed to indicators listed; and an interview schedule was prepared for policy makers. The ques- tionnaire was revised after a pilot study with twelve individuals. The second phase began with the preparation for the field work and included data collection and analysis. Five major univer- sities in Brazil were selected. Faculty members from randomly chosen programs, within those universities, received the questionnaire. In addition, the highest administrator in graduate studies at the same universities, the available program coordinators, and officials at the Ministry of Education and Culture were interviewed. Data analyzed in the study were derived primarily from the questionnaire. The interviews and publications collected, although a source of valuable information, were used as explanatory or inter- pretative content-~following the adopted design. Questionnaire marks received statistical treatment through nonparametric techniques. The research findings were organized and summarized to per- mit direct answer to the stated problem. A speculative comment on causal or intriguing relationships is appended, in an effort to stimu- late further investigation. Scope and design limit the attainment and generalizations from this study, yet should not belittle its validity for future advancements. Models, instruments, and proce- dures for planning and evaluation in graduate Brazilian education or comparative studies may result from this dissertation. Overview In seeking the criteria considered apprOpriate to evaluate graduate programs in Brazil, four more segments evolve and constitute this dissertation. Each of them is organized as a chapter, numbered from two to five. The second chapter, a review of selected literature, provides for an identification of concepts and functions of graduate education and the basic issues in evaluation. From this review, the indicators used in the questionnaire were originated. Research design and procedures are detailed in the third chapter. The investigation approach, the development of instruments, the selection of subjects and materials, and the data-collection and analysis procedures are described. lO In the fourth chapter, the information gathered through the questionnaires is disclosed and analyzed. A few tables and excerpts aid the description. Finally, an assessment of values attributed to selected criteria and indicators leads the writer to offer some comments about the findings and observations about the graduate education sys- tem in Brazil, and to recommend a methodological perspective. This constitutes the fifth chapter. A bibliography and an appendices section follow the text, as supporting documents. CHAPTER II LITERATURE REVIEW This assessment of criteria to evaluate Brazilian graduate programs required a comprehensive review of the literature. Although the responses provided by the faculty currently involved in graduate education constituted the core of the investigation, it was necessary to gather a host of information as background reference. The issues and aspects selected for inclusion in this chapter were critical to the concept of the study and the develop- ment of research design and instruments. These notes, however, rep- resent only the most useful references. The sequence of this chapter reflects the issues approached from the conceptual to the actualization phases. In the first section, evaluation is defined and its methods and models reviewed. Evolving from those, the theme of evaluation criteria and their use provide elements for the second section. The third and last section examines academic evaluation, eSpecially of graduate education, and reviews the most prominent criteria, or areas of concern, and indicators used or suggested in the evaluation of graduate programs. ll 12 Evaluation Despite a strong theoretical debate as to whether academic program evaluation is or is not "research," there is an increasing demand for planning and development of evaluation in educational programs. In order to better organize the basic framework for this investigation, a review of major issues on evaluation follows. Evaluation Defined Attempts to define evaluation reflect concern with the desira- bility or value of programs. Greenberg (1968) characterizes evaluation as "the procedure by which programs are studied to ascertain their effectiveness in the fulfillment of goals" “3.260). Brooks (1965) complements this view, listing objectives an evaluation should seek to measure: (a) the extent to which the program achieves its goal, (b) the rela- tive impact of key program variables, and (c) the role of the program as contrasted to external variables (p. 34). These authors emphasize the information-seeking aspect of evaluation as does Schuman (1967) when defining evaluation as “the determination of the results . . . attained by some activity . . . designed to accomplish some valued goal or objective" (pp. 31-32). Another perspective is offered by Dressel (1976), who presents evaluation as "both a judgment on the worth or impact of a program, procedure, or individual, and the process whereby that judgment is made" (p. 1). This approach intimately links evaluation and decision making, bringing the concept a step beyond the informational point. 13 Scriven (1967), likewise, emphasizes the judgmental dimen- sion, introducing evaluation as a "methodological activity which combines performance data with a goal scale" (pp. 40-41). This study, acknowledging evaluation as the means of assess- ing the effectiveness and efficiency of a program, is designed to provide support for the evaluation decision-making process and, moreso, to evaluations aiming at effective decisions. Evaluation Methodology Evaluations may take several different approaches. Stake (1967) described informal evaluation as relying on casual observation, implicit goals, intuitive norms, and subjective judgment. He characterizes this method as of variable quality-- sometimes penetrating and insightful, sometimes superficial and dis- torted. Recipients of services, journalists, practitioners, and policy makers are mostly those who are typically charged to use this impressionistic or informal evaluation. An alternative approach is to evaluate through planned research procedures or formal methods. Within this category, a dis- tinction is made between methods emphasizing input§_and those stress- ing outputs. Educational programs evaluated by teacher qualifications and the ratio of library books to students are examples of judgments based on input indicators. Explicit checklists and formulas are often used, and data are commonly obtained through on-site inspection. The program accounting approach to evaluation also observes input of efforts, focusing on the maintenance and quantitative 14 analysis of program activity records. The extent or number of student- contact hours is a characteristic indicator used in this method. This approach poses little attention on outputs or effects but, because it can provide information on the ability to establish con- tacts and the cost of program-client contacts, it is useful for determining the administrative viability of a program (Caro, 1971). Evaluative research emphasizes outputs or effects and a con- cern with the application of the scientific method. Suchman (1969) stresses the difference between evaluation, a "general social process of making judgments of worth regardless of the basis for such judg- ments," and evaluative research, the "use of the scientific method for collecting data concerning the degree to which some specified activity achieves some desired effect" (p. 15). Hyman and Wright (1967) defend evaluation based on "methods that yield evidence that is objective, systematic, and comprehen- sive" (p. 742). Although evaluative research is strongly associated with outputs, because the bases or criteria for judgment are generally connected to the goals of a program, it should not imply a lack of concern for input and process variables. Scriven's (1967) suggestion of a "mediated evaluation" combines input and output variables. The formal or planned research procedure is the preferred method for evaluation for decision making. And the evaluative research approach has a record of still-modest but promising possibility. It has been increasingly mandated and incorporated for both large and small human-service programs, and particularly to education. 15 Evaluation Models Although the possibilities for application of research methodology, and confidence in its usefulness, are a present and pressing problem for social scientists, innumerable efforts toward enlarged feasibility and quality in research for educational evalua- tion have produced several positions or models. Some of them are widely recognized and, if not completely adequate for a specific problem or program, are helpful for stimulating a unique and tailored design. The measurement model emphasizes the use of tests, norms, and standards while avoiding dealing with variables which are not opera- tionally definable and objectively measurable. It is valued because reliability, validity, and objectivity are present, but it requires a very restricted scope and, therefore, has a limited application in extensive and influential decision making. The effectiveness model seeks to assess the attainment of a program's objectives. It concentrates on results achieved, or on the measurement of the degree of success or failure encountered by a program in reaching predetermined objectives. An assumption of this model is that once objectives can be defined, the appropriate method- ology and criteria for assessing the program can be selected. This model has been described in the literature and it encompasses many characteristics of classical research designs. It is very useful for diagnosis and program-improvement efforts, but is only suitable for well-planned programs and for outcomes evaluation. 16 The summary judgment model requires minimal data collection and emphasizes generalizations by experts or professionals. Often this procedure provides for outsiders to visit, observe, and inter- view program personnel. Despite shortcomings overcome by evaluators' experience, the model permits excessive subjectivity and occasional factors to play a major role. The system model, as described by Etzioni (1960), is con— cerned with establishing a working model of a social unit which is capable of achieving a goal. It credits an organization with at least four important survival functions: (a) the achievement of its goals, (b) the effective coordination of organizational subunits, (c) the acquisition and maintenance of necessary resources, and (d) the adaptation of the organization to the environment and to its own internal demands. It, therefore, attributes importance to many overlooked organizational characteristics which are functional and increase final effectiveness. This model stresses evaluation as an ongoing and continuous process, well suited for decision making. In contrast to the goal-attainment-oriented models, the sys- tem design establishes the degree to which an organization realizes its goals under a given set of conditions. Optimum, then, is the key word; a balanced distribution of resources among all organizational objectives is sought, as Opposed to maximal satisfaction of any one goal. The ideal model is unique, but potentially the most effec- tive and comprehensive would probably require: (a) a priori con- sideration of evidence, having program objectives in mind, i.e., a 17 planning stage when criteria for success or failure are explicitly stated; (b) collection and analysis of information during the course of the program in order to clarify objectives or later alter pro- cesses and, thus, provide for continuous feedback and discussion; (c) a posteriori consideration of evidence to assess results and to explain why the desired outcomes were achieved or not; and (d) con- tinual review through recycling as evidence of effectiveness accu- mulates. The above-mentioned sequence is proposed for many evaluation projects and specifically advocated by Dressel (1976) when the impli- cation that "evaluation is the collection and interpretation, through systematic and formal means, of relevant information which serves as the basis for rational judgment in decision situations" is explained (p. 9). Furthermore, this concept and process particularly suit academic institutions. Dressel emphasizes that this type of evalua- tion provides a model for decision making which students and society should emulate, capturing the essence of education, while encouraging confidence and understanding of a university's procedures and pro- cesses. The survey, which is the core of this dissertation, can pro- vide information for evaluation based in any of the models. However, the system model's requirements and the advocated procedural sequence were of particular concern. 18 Evaluation Criteria Further elaboration on the concept and functions of cri- teria in evaluative research and decision making is useful for under- standing the development and the need or value of this study. A criterion is generally defined as a standard on which a judgment or decision is based. It is something established by authority, custom, or consent as a rule for the measure of quantity, weight, extent, value, or quality (Good, 1959). The identification of criteria is a fundamental and required step in the judgmental process. The criteria orient the gathering of judgmental data. Stake (1970) shows particular concern with the processing of judgmental data in educational evaluation. The author supports the study of alternative procedures for collecting or analyzing judg— ment data, something educators have neither explored nor communi- cated extensively. Stake (1970) also identifies three methods of gathering judgment data: (a) a collection of judgmental responses with a proto- col from a group of individuals and aggregation of responses to des- cribe the value commitments of that group; (b) employment of experts to make personal observations of events or processes and then to describe the value commitments apparent in them; and (c) employment of experts to analyze documents and to reconstitute the value commit- ments made in creating the program. This study was designed following the first method; it sought judgmental responses of faculty involved in graduate education and 19 aggregated their responses to describe the value commitments of the group. The survey results point out commitments about the criteria appropriate for evaluation of graduate programs in Brazil. Criteria are instrumental at all phases of the evaluation process. In the first, the descriptive phase, a program's character- istics, its functions, and other distinctive aspects are observed. Each of these aspects or relationships may be called an indicator, i.e., the unit used for observation. Indicators are registered through the collection of frequencies or relative attributes observed in the program. The quantification or categorization of indicators consti- tutes the measurement phase. Data collected may be organized by each individual indicator, any aggregates, or other measures, such as ratios or means. It is the measurement step which provides the evaluator with a "picture" of the program. Proceeding toward the evaluation, an assessment phase will require comparison of the data gathered with any relative or abso- lute standards selected for the evaluation. This is the moment immediately before the judgment; it is when the evaluator determines whether or not each standard is met. An assessment using absolute standards uses the indicators and criteria considered appropriate, acceptable, or meritorious, by the decision makers in the evaluation process. There are multiple sets of criteria by which a program can be judged in an absolute 20 sense, because there are numerous reference groups or points of view. Relative comparison is accomplished through the use of indicators for which there are similar observations taken from other programs. The last and most characteristic step in the judgmental process is the decision on which set of standards is valid or useful for the situation. As Stake (1967) points out, judging is assign- ing a weight, an importance, to each set of standards. It is hardly a judgmental matter to determine whether one program betters another with regard to a single characteristic. There are many program characteristics and they are not equally important. The evaluator must decide which indicators to use and/or which programs to com- pare. This process constitutes the judgmental phase. From relative and/or absolute judgments an overall composite rating merit is reached. Qualifying statements usually add insight and prepare for an educational decision concerning the program. Cronbach (1963) and Scriven (1967) disagree on the use of relative and absolute evaluations. Cronbach opposes comparisons and supports in-depth "case studies," with extensive measurement and thorough description. Scriven, however, indicates that educators always want an assessment of quality and that the best way to answer is by direct comparison. This study evolved from the identification of a broad range of aspects which were considered as possible indicators for an assessment of graduate programs. These indicators were grouped by 21 labeled criteria. Neither indicators nor criteria determine any quantity of the attributes and are applicable to both absolute and relative comparisons. The criteria and indicators used in this inves- tigation are specifically oriented toward academic evaluation. Academic Evaluation Although many of the statements collected under this t0pic may be generalized for other kinds of educational institutions, the focus is on the identification of major issues in the evaluation of higher education. Evaluations and value commitments are present in all phases of higher education. Universities are multifunctional and very diver- sified organizations which benefit from every effort toward syste- matic and explicit evaluation. Institutions of higher education have special obligations to students and society, i.e., effectiveness and accountability. They assemble a major part of the individuals most privileged by social wealth and most expected to return those benefits in terms of community services. Moreover, the central goal of education is evaluation, that is, to promote the student's ability to make wise judgments (Dressel, 1976). Quality seems to be a traditionally mystical word for edu- cational institutions. All institutions stress "quality" as a basic goal reflected in their programs. It has been only in most recent years that the above-mentioned concepts of accountability and effec- tiveness reached the almost untouchable world of higher education. 22 Any consideration of what constitutes quality in a setting is bound to provoke controversy. Cartter (1966) notes that "in an operational sense, quality is someone's subjective assessment, for there is no way of objectively measuring what is in essence an attribute of value" (p. 4). Evaluative judgments of relative or absolute quality in program are often severely criticized. Perhaps quality is not the best word or concept after all; it is just the traditional one. The issue would probably be better posed when the main question is how efficient is the institution in identifying the abilities of its students and how effective it is in providing them with the appropriate curriculum. Blackburn and Lingenfelter (1973) proposed the use of excellence, as a dimension of quality, charging the evaluator with the task of selecting the constellation of "excellent" traits neces- sary for "quality." These authors insist that quality and excellence exist, that some programs or products have more of it than others, and that quality and excellence do matter in important ways. Academic evaluation is, therefore, often intimately linked to quality and excellence assessments. As a result, new forms of evaluation for higher education have evolved in response to realities and needs of the institutions. They stress self-study, colleague consultation, realistic planning, and at least minimal feedback from analysis of the outcomes. Actions have to be guided by an accurate perception of institutional strengths, weaknesses, and social contri- butions. Limited resources must be invested where they produce the 23 best returns, regardless of the developmental or financial situation of a nation. Good graduate education must have the highest-attainable quality in its program. It is not only where technological and social innovations are intensely transmitted and utilized but where they are, in large measure, generated. Sound research and teaching at the graduate level are critical to the understanding of the human situation and the rapidly changing environment. Quality assessments have particularly salutary consequences in graduate programs. New ideas emerge, better practices are intro- duced, and they generate a climate for healthy growth and development. Yet, two major problems confront the assessment of excellence and the evaluation of graduate programs: (a) identifying the appropriate indicators and criteria for excellence, and (b) determining the sati- isfaction levels or quantifying the criteria so as to permit compari- sons and a final judgment (Blackburn & Lingenfelter, 1973). Graduate Evaluation Criteria A review of studies and critiques of graduate education reveals some of the essential characteristics which become criteria for evaluation. Some objective characteristics related to excel- lence may be identified by examining reputational studies' "correlates of quality." Another way of selecting criteria is through a syste- matic examination of important characteristics. This investigation was guided by the system approach to evaluation and the organization. The model adopted for searching 24 and arranging the selected criteria departs from input tow rd output variables, examining objectives, curriculum, evaluation process, faculty, students, and organizational facilities and characteris- tics. Figure l on the next page illustrates this point. In the following subsections, issues are presented support— ing the criteria and indicators used in the survey questionnaire. Criterion One: Program Objectives As earlier mentioned, the identification or elaboration of objectives is a necessary initial step in program development and evaluation. Many authors provide support and suggestions for the inclusion of this criterion; Tyler's (1949) contribution deserves special mention. Diverse views on the objectives of higher education are likely to exist within a faculty and among institutions. Even narrow- ing the focus to graduate education, resistance and controversy may be encountered and stem from several factors. Basic philosophical differences in regard to the nature and the role of education and dif- ferent perceptions of societal needs account for most disagreement. In addition, terminology and skepticism about the usefulness of this objectives-identification task bring about many objections. Despite the diversity of sources from which graduate education objectives can be inferred or drawn, there are a few roles or general goals considered pertinent to graduate programs. A fundamental role of graduate education in Brazil is offi- cially stated in several Government documents and summarized in the 25 .msmcmoca cowumozom mumaomcm cw agw_m:c o» owum_mc mmpamwcm>uu.p mczmwg mmpamwcm> aw \ QQKKQ .A “.ch0; w>cu U wEmUwU< @— cowum~wcmmco Eocmoca .m— mopuw>wuo< .macoczum .m mocmccm>ow owsmumo< .n _ mwcocPcumu Pmcuaam moocaomom pmwocmcwm zumcavo mmosnweou< Faucmsga mmuanwcuu< .mucouaum mpmmuzouocu xupaomu mmh<=n§o meow: su_>_ou=uoea sumscuu< supsumc .0 mucoamo_ucma cowumspm>m moo; cowumapm>u cowumnpcmmco Espsowcczu mo>wooonao Emcmoca I—Nmfi' have u i? say» mmmoocm .vp .Np .pp .m .m 26 publication "New Perspectives for the System of Higher Education" (CAPES, l976b). Graduate education is charged with "increasing strategic importance with repercussions of a multiple dimension in the national process." Further, graduate studies are also defined by the National Plans for Development (PND I and PND II, 1971 & 1974). They state a philosophy which aims toward planned national develop- ment according to a societal model, capable of transforming and reforming the country's economic, social, and political institutions and respecting the human values and identity of the Brazilian culture. The formally presented goals of graduate programs, following the National Plan for Graduate Studies, are: (l) to train professors for university teaching in order to meet the quantitative expansion of learning and to raise its quality; (2) to train researchers for scientific work directed toward creating nuclei and centers which will meet the program and regional needs of society; and (3) to prepare high level professionals, according to the demand of the labor market, for private and public institutions (Brasil, 1975). Over and above, credit must also be given to objectives bestowed on higher education institutions: (1) diffuse and expand the knowledge and culture of society; (2) utilize the resources and instruments of teaching and research for a real change in material and cultural conditions of a society geared up for its social and economic development; and (3) prepare, train, and qualify human resources on a high level, both in volume and diversity, and adequate for a produc- tive national system and the university's own educational system (Brasil, 1975). These official goals, therefore, direct the graduate pro- grams to general objectives which emphasize the specific discipline's contribution to the social and economic development, the role of the professional, the importance of research activities, and professional 27 competence. In addition, cognitive domain objectives are usually clearly stated for graduate programs. The most commonly mentioned relate to knowledge of subject matter content (methods and theory) and to intellectual abilities or skills, such as critical thinking and decision making. Criterion Two: Curriculum Organization The curriculum brings together pe0p1e (students, faculty, administrators, staff) and the environment to engage in educational processes designed to facilitate the attainment of educational and social goals, not only for students but also for faculty members and the institution (Dressel, 1976). Educational processes may take place anywhere, e.g., in school, at home, at work, or in leisure time. They are determined by the people, the environment, the materials and equipment, and the nature and range of interactions among them. Learning, the ultimate goal, is an internal process influenced by these educational factors. Some psychologists defend the idea that learning experiences (and even objectives) should be adapted to individual differences in learning style and interests. Curriculum theorists add that the particular discipline or field of study characteristics are also important determinants of educational practices. A graduate curric- ulum organization requires, because of its specific objectives and learners' characteristics, special attention toward a flexible but demandinghprogram through which students' needs and talents are maximized. 28 One of the instructional features most peculiar to graduate programs is independent study. This type of offering allows the stu- dent to earn credit for work in a given area and under the supervision of an appointed instructor, but through an individualized educational process. Content and objectives may be uniquely planned and/or the sequence, duration, and rate of progress set by the learner. The utmost individualized experience at this level is the research requirement. The majority, if not all, of the Brazilian masters' and doctoral programs nmndate the presentation of a thesis or dis- sertation.1 Comprehensive examinations, even if an official test is prepared for a large group of candidates, require also individualized preparation and evaluation. Inasmuch as the individualization aspect, the offering and requirement of a basic core of courses, tends to be a characteristic of graduate curricula, Brazilian master's degree programs are cen- tered on a relatively large number of courses and seminars which stu- dents are expected to attend for a specific major. Doctoral-level curricula seem to require proportionally less mandatory credits and more individualized study. A valuable complement to the core of required courses is the credits students may earn in courses and seminars designated as optional or electives. If taken within the student's major field, they usually serve for increased depth. However, many of these credits are often acquired in other departments or even schools and 1Based on a review of "By Laws" and official documents from the Brazilian Government about graduate programs. 29 contribute to larger breadth of learning experience, another desir- able feature in graduate programs. Besides planning for a basic curriculum organization, the faculty and administrators in charge of master's and doctoral programs often face less-patterned problems. Decisions have to be made con- cerning credits incoming students have earned at other institutions or programs of the same level, or about acknowledging a candidate's proficiency on required course content without duplication of effort. In addition to the above-mentioned structural aspects, a curriculum's organization should attempt to foster adequacy between objectives and content of a particular course or seminar and its teaching methods. A review of instructional practices and the student's periodic evaluation should provide the means for assessing this point. Many graduate programs present the students other require- ments which are designed to facilitate the achievement of proposed. objectives. Among the most common are the compulsory attendance on a full-time basis (residency) during all or most of the program and the participation in research activities during this period. Both these requirements are stressed at the doctoral level. Criterion Three: Curriculum- Evaluation Foci Curriculum evaluation is a major part in any institutional self-study or planned inquiry on the effectiveness of'a graduate program. It includes a thorough examination of all the factors which determine students' learning, as promoted by the faculty and 3O facilitated through coordinated staff, environment, materials, and activities. No matter what model or method is used for evaluation, as previously reviewed, there are two basic foci a curriculum evalua- tion can approach. First is the question of its adeguacy or the suitability of its goals and objectives for fulfilling social needs. This is largely a point of assessing a program's external validity. Second is the question of its congruence or its achievement of pro- posed goals and objectives, a matter of internal consistency. Criterion Four: Evaluation Participants Increasingly, professional evaluators have been sought to plan and help conduct evaluations. The degree to which an evaluator, or the persons designated to carry the task on, will be engaged is influenced by time, money, and expertise, among other factors. There are, undoubtedly, unique responsibilities and activities which require special care and competency (Worthen & Sanders, 1973). It is generally agreed that evaluation must become an accepted practice among a program's faculty so that decision making and change may take place. Faculty members find evaluation valuable if it leads to satisfaction, to suggestions for improvement, or to reward. Sup- port for evaluation depends also on the preparation and effort an activity requires, and on the material rewards and prestige the evalua- tion brings (Dressel, 1976, p. 335). Likewise, any other persons who are directly or indirectly involved in a particular graduate program tend to benefit from their input. 31 Consensus among evaluators and educational leaders suggests that a variety of sources be consulted in an evaluation study. A program's faculty must be involved in most, if not all, parts of curriculum and total program evaluation. And their partici- pation can take many forms. The simplest and usual is through peer or reputational evaluation. The program's coordinator, i.e., its leader who is respon- sible for administering pe0ple, resources, rewards, and punishments, should also participate. Without the coordinator's support, no evalua- tion can succeed. At another level, administrators who coordinate the programs, to whom the program is directly accountable, and peers in the university structure should also be provided with means to offer input on relevant issues. Usually, the most-sought opinion, in traditional graduate program evaluation, is that of other specialists in the field who work at other institutions. Their best contribution is to assess the program's reputation or to give value to items such as academic productivity or special awards, through the "peers" appraisal. Examples of these contributions are set by accreditation reviews organized by professional associations. In Brazil, the scarcity of experts in some areas has also brought the opinion of foreign scholars into practice, usually to complement the national panel. An obvious means of evaluating an enterprise of any kind is to use client satisfaction ratings and suggestions. The consumers of graduate education are primarily the students and the employers of graduates. Both these groups have been used in evaluative studies. 32 Student opinion surveys are reported by Florida State Univer- sity (1957), Heiss (1970), and Harvey (1972), among others. The student judgment on the quality of instruction provided and the sug- gestions and complaints assure a researcher about the value of their appraisal. Former students, either dropouts or graduates, also offer a different and important perspective. Berelson's (1960) and Alciatore and Eckert's (1968) studies are examples of profitable inquiries into the value of a program and possible reforms. The other major clients of graduate programs are employers of alumni. Since the objectives of graduate studies, as defined by the National Plan for Graduate Studies (Brasil, 1975), are largely concentrated on the preparation of teachers, the opinions of college administrators which have recruited a program's graduates must be endorsed. McGrath (1961) is an example. Other kinds of employers, to be determined by the particular program characteristics, should also be included. Criterion Five: Academic Credentials Some studies have used academic degrees and awards as a major index of faculty's quality. Given the special circumstances of Brazilian graduate education, the mere possession of advanced degrees by individual faculty members may be an appropriate indicator. CAPES' (1976) study considered separately the master's and doctoral degrees exhibited by the faculty as an indicator of faculty quality. 33 Considering the value of broad and diverse background educa- tion, programs have been urged to avoid "institutional inbreeding" when hiring faculty. Therefore, the degree of diversity of the edu- cational background of the faculty may be included as an indicator of faculty quality. Bowker (1965) examined the number of pyppg§_won by a faculty as an index of quality. This is an indicator directly related to the quality of one's contribution to a specific field or societal need and which could play a useful role in evaluating graduate pro- grams, mostly when excellence is being considered. Faculty development is an issue receiving increasing support by the agencies monitoring higher education and by faculty members themselves. Traditionally, development programs have helped faculty members to upgrade and update their knowledge of the academic speciali- zations; new concepts call for attention also to instructional roles. Therefore, attempts to assess the competence and efficacy of the faculty should comprehend abilities in all activity areas they perform (Miller, 1972). An additional point to be reviewed is the copgruence of faculty academic credentials or professional expertise to the program's objectives. A professor's potential contribution to the specific areas of teaching and research must be assessed, not merely on the basis of an advanced degree and scholarly production. This aspect is of particular importance in the Brazilian case, because the rush to develop graduate programs provoked the hiring of instructors from 34 adjacent fields and to jeopardizing the program's and the students' objectives. Criterion Six: Faculty Activities and Productivity As a result of traditional functions and contemporary pres- sures, university faculty today have many roles. They must be involved in teaching, research, and, sometimes, in community services and academic governance and administration. The allocation of their time and attention to these activities is influenced by their prefer- ence and the university's internal and external demands. There is evidence that some faculty members consider this situation as a dilemma, with conflicting ambivalent values at play. Professors engage in many activities, for which they have no specific preparation. The range of their interest in, capability in, and commitment to those kinds of assignments varies enormously (Dressel, 1976). The most common activity performed by faculty members is teaching: instruction of undergraduates, instruction of graduate students, course and curriculum development, preparation of instruc- tional materials, evaluation and grading of student progress, advis- ing students, and (hopefully) promoting opportunities for feedback on the quality of their work. Another area in which professors engage is professional service, both internal (for example, supporting another department's needs in one's specialty) and external (for example, participating in professional associations or extension services, consulting, etc.). 35 Some faculty activities are centered on research, creativity, and scholarly production of a pure or practical nature. Considering that graduate education has as a major objec- tive the production of new knowledge, scholarly output stands as the most widely publicized criterion of excellence. Indices of scholarly productivity have been developed for both departments and indi- viduals. They usually assign importance to books, articles, reports, patents, and other products associated to these; a productivity score is obtained by the aggregated credits in each category. Examples of productivity indices are shown in the work of Crane (1965), Pelz and Andrews (1966), Stallings and Singhall (1970), among others. There are studies which offer support for output as a measure of reputation. Berelson (1960), Cartter (1966), and Crane (1965) found a relationship between the most prolific departments and those receiving top ranking. Possibly a more valid measure of scholarly productivity is the citation index used by Clark (1956), Bayer and Folger (1966), Cole and Cole (1967), Creager (1967), and Myers (1970). The index calculates the number of citations in scholarly publications. Added value may be given for a citation older than ten or fifteen years, based on the assumption that durability is an indicator of extra quality. The citation criterion seems to be more appropriate for natural sciences than other disciplines, because of traditional research procedures. Margolis (1967) and Smith and Fiedler (1971) discuss the advantages and disadvantages of the citation index in detail. 36 A controversy also exists between the value given to each category of scholarly products: how to compare books, articles, and patents, if not the publication in which they appear. An additional complication is posed to less-developed countries when publication in national journals is compared with publication in internationally recognized periodicals. Criterion Seven: Academic Governance Faculty members are often called upon, or seek, to partici- pate in governance at departmental, college, or university levels. These activities are peculiar to the academic organization and help to attain and preserve an environment conducive to the fulfillment of the universities' roles. The degree of participation in the many academic governance and administrative tasks varies greatly, according to personal and institutional factors. Faculty participate in recruiting, selection, and promotion of their peers. Although these activities are ultimately a respon— sibility of administrators, faculty members customarily have an input in the processes. Brazilian public institutions and some pri- vate universities admit and promote their faculty through open selec- tion procedures. Positions are advertised and candidates are examined by the specific department. Recruitment is not very active, but is becoming important in some fields. Senior faculty members usually lead this sector, with provisions for juniors' contributions. 37 Selection of incoming graduate students is also a respon- sibility shared by a program's faculty; however, undergraduate admis- sions are handled mainly by a central administrative unit. Permanent faculty also participate in university policy making and academic administration. Every segment of the faculty is repre- sented in executive or consultive bodies, along the organizational structure. The representation, activeness, and power they enjoy is a matter of very interesting and controversial studies. Faculty members also may contribute in the management of a department or school. Although not a general practice, federal uni- versities in Brazil have rotating systems working for every adminis- trative position and, therefore, faculty manage the academic side of the institutions. The allocation of financial and human resources is, however, a subject of increasing interest to most faculty. Pro- fessors bring in resources through their productivity (either student credit hours generated or research money) and receive privileges in the distribution of teaching assignments. Underlying these activi- ties is the evaluation of peer's effectiveness, by formal and informal means. Discussions of faculty activity and the evaluation of their participation frequently revolve around teaching, and this seems to be the most difficult and least-accepted evaluation tapic. Teaching and advising often have contrasting definitions and expectations, providing soft ground for research and judgment. Research, public service, and participation in academic governance, however, are objective and tangible bringing greater recognition than teaching. 38 Thus, evaluation of faculty activity may use any of the above-mentioned indicators but is still quite limited, Sporadic, and inadequate. Criterion Eight: Student Attributes Because each faculty tends to select the best possible stu- dents and good students are attracted to programs with a reputation for quality, the quality of students enrolled in a graduate program has been a widely used measure in the evaluation of graduate programs. When comparisons between programs are sought, the student's ppgpr; graduate records, standardized tests, and the receipt of student- attached fellowshipp or awards may be used (Cartter, 1966; Bowker, 1965; Perkins & Snell, 1962). In general, the elements requested for the screening of graduate students should permit an assessment of the student quality. Other measures also valuable for the selection process are: letters of recommendation from former professors or employer, the student's professional or academic experience, and a statement of academic and professional objectives presented by the candidate. Major schools' catalogs offer sufficient evidence in this area. In Brazil, some programs have experimented with a collective selection process by which candidates are tested in academic abilities for the specific field and other measures. A combined index, which also includes the student's program preference, leads to a ranking of candidates selected. The results of such examination would seem to provide an estimate of the students' quality which could be used for program evaluation. 39 A not-so-frequently feasible measure is an interview of prospective students with the program's faculty. Criterion Nine: Student Activities Historically, students have experienced peculiar accommoda- tion problems in their relationship with colleges and universities. The academic organization, with its structure and processes, and the curriculum and instruction have been the subject of much criticism, praise, and many reactions which denote cognitive and affective interactions. Katz and associates (1968), Sanford (1962), and Schwab (1969) offer significant contributions Unthe understanding of these aspects, although not specifically about graduate students. Seeking to provide an educational experience responsive to students' needs and to the complexities of human problems in the mod- ern world, graduate programs have increasingly called for more active participation of students in their own education. Curriculum and instruction, as well as the whole academic governance system, have been revised to include student representation. In Brazilian public universities, students are represented in all academic decision-making bodies. The evaluation of students' access and participation in academic life is, therefore, an important sector of program review. Among the aspects to be assessed, the availability of publications on students' rights and responsibilities and the norms for academic admission and progress stand out. Also, graduate students are expected to have equal means of rppresentation and participation in the academic decisions. 40 One of the indicators most used in the evaluation of gradu- ate programs is the number of scholarships and fellowships obtained by students' merits. Useful for comparisons or criterion-referenced measurements are the awards given to individuals in national and regional competition. But another positive aspect is the ability of a program to offer to the best candidates some financial rewards because of its potential to attract good students. Additional effort at program levels should include the fpgjlj: _tig§ necessary for individual and group study and, whenever possible, paid opportunities for research and teaching assistance provided. Within this topic a double-check is possible. Not only the professors' advising activities should be evaluated, but also the students' rights and access to quality academic and professional guidance. Criterion Ten: Physical Facilities Although no systematic studies of physical facilities for graduate programs could be located, an evaluation should consider the existence or adequacy of laboratories, office space, computer capabilities, seminar rooms, instructional materials, and other neces- sary facilities. A probable reason for the lack of specific studies is that facilities are usually shared by graduate and undergraduate programs. However, since the facilities required differ very much among disciplines, specialists should be utilized for their evaluation. Because of the unique characteristics of development that many Brazilian graduate programs are undergoing, it would seem appropriate to elaborate further. 41 In the past, higher education faculty members worked at universities on a part-time basis, with a strong emphasis on teach- ing. The advent of graduate education brought about a serious prob- lem not only for acquiring advanced laboratories and adequate libraries but also office space and support services which were not available before. Therefore, physical facilities must be an important item in the evaluation agenda, demanding a careful assessment of needs and possibilities. Departmental reviews suggest that all physical facilities be examined for their adequacy and a comparison made with facilities provided for other programs. The acceptability of the facilities in relation to the purposes and both present and future plans should also complement the appraisal of traditional physical attributes such as light, temperature, and space. Utilization statistics are useful administrative information (Dressel, 1971). Criterion Eleven: Library An obvious and most important facility required for all graduate programs is an adequate library. Several studies identify library as a significant criterion of excellence (Jordan, 1963; Cartter, 1966; Perkins & Snell, 1962). If the library is to be used as an index of a program's quality, a detailed analysis must be undertaken. Reference materials, a strong collection of periodicals in pertinent subjects, and a spe- cific number of volumes are necessary. Although these are basic requirements, the administration of the library is equally important. 42 Updated catalogs, interlibrary loans, library personnel, and §pppg_ for study must be adequate. Moreover, the acquisitions plan and budget should be reviewed. Brazilian university libraries have faced some problems with acquisitions of books and of complete periodical collections. The distance from publishing centers and the fact that most current pub- lications are in foreign languages are some of the factors accounting for fewer readers' requests and limited purchasing by the librarians. Furthermore, many Brazilian institituons offering graduate programs are relatively new. Building and updating a library is an extremely demanding task. Criterion Twelve: Special Facilities Several references emphasize the importance of physical facilities and the library for graduate programs, as reviewed. How- ever, depending on a program's objectives or subject, additional attention must be given to other types of facilities. In the United States, professional associations and accredit- ing institutions for professional fields have particularly stressed the importance of quality and accessible local agencies where the faculty and students observe or practice. In Brazil, the existence and quality of such agencies are limited. The major aspect for a program assessment is the adequacy of community and university agencies where clinical or instrumental resources needed for the planned curriculum will be available. This includes pupils, patients, machines, and agricultural land, or other program needs. 43 Other services which facilitate faculty productivity are clerical and administrative support, specialized physical and human research support, and data-processipg systems for professors' and students' use. Criterion Thirteen: Program Organization Any evaluation, but particularly self—studies, must devote thorough and extensive attention to a program's organizational aspects. A program's structures and functions have to be analyzed in conjpnction with the purposes._objectives. and functions of the university as a whole, and to the specific mission of that unit. Higher education institutions are very peculiar organizations. Leadership, structure, and power are interestingly established, as Richman and Farmer (1974) report. Overall productivity and excel- lence depend greatly on the harmony and efficiehpy enjoyed by the various subunits. Graduate education seems to be especially criti- cal for healthy administrative solutions and faculty morale. Dressel and associates' (1971) findings and Parsons and Platt's (1973) "the core sector of the University" provide some evidence and analysis, respectively. An evaluation of Brazilian graduate programs must approach this topic with increased effort, because at most universities the graduate programs are a parallel or superior unit, different from a department or a college. Many graduate programs originated through external of interdepartmental action, have separate budgets, and some 44 of their heads are more powerful than department chairmen or deans. Faculty members belong to departments but the graduate programs are responsible for activities which consume the major part of their work schedule and which provide a large "complementary" part of their salaries. The details of these arrangements deserve further study with objectives other than this investigation. The relationships of the graduate program with the central administration or other university units should be carefully scruti- nized. Another area of concern which contributes to productivity is the organizational provision for faculty and student participation in the administration. Among the stipulations facilitating this process, there are: (a) faculty and student representation in program and uni- versity academic decision-making bodies; (b) ppblishedppolicies, norms, and rights for faculty members; (c) pprticjpation of faculty of all categories, according to their position and reSponsibilities; and (d) written minutes or communications on administrative and academic meetings and decisions. Criterion Fourteen: Financial Resources A good deal of effort in the area of higher education adminis- tration has been devoted to financing and budgeting. The Ministry of Education and Culture and special task forces in various Brazilian universities have reviewed the general literature and sought to intro- duce some of the managerial concepts developed in government and busi- ness to program administration. 1 45 Graduate education has received increasing support from sev- eral public agencies as a result of official commitments, such as the earlier-mentioned National Development Plans. Growing complexity, greater expenses, and demanding external forces are leading the universities to new decision-making structures, specialized planning offices, and progressive use of data-processing and management information systems. Despite many controversies about the advantages and cost of these innovations, some basic indicators for adequate financial allocation remain unsettled. While graduate programs compete for resources, education and research economic returns are still being debated (U.S. Government, 1969) and evidence of accountability of graduate programs is diffi- cult to obtain. Four basic areas of indicators for financial resources allo- cation may be identified: (a) competition between programs_yithip the university; (b) competition between similar programs for funding at national and regional levels; (c) competition between teaching, research, and extension activities; and (d) competition at the national level, between the various teaching and research disciplines or fields of study. Involved in these alternatives are reputation, merit evalua- tions, number of faculty in each category, and many personal values. The resource allocation is largely a political process where pre- cisely objective evaluations are difficult and often objectionable. However, a program's evaluation should review budget and expenditures of past years and future plan. General procedures 46 suggested are a comparison between sums requested, sums allotted, and actual expenditures. Dressel et a1. (1971) value a department's bud- get adequacy and the process by which the funds are administered, but emphasize that the department's success in obtaining funds, fellow- ships, and research grants is another indicator of its quality and the effectiveness of its management. Criterion Fifteen: Academic Environment Ashby (1974) emphasized the importance of the moral aspects of the academic profession: scholarship (with reverence to truth and humility), equality for any scholar, and internationalism (reverence toward friend or enemy engaged in the advancement of knowledge). These may be considered as the guidelines for academic freedom. Academic freedom is certainly one of the most universally prized and controversial values in the university world. Ross (1976) traces some of its roots and current issues in the North American scene, distinguishing between freedom in teaching and research and freedom on campus or community. Academic freedom may be associated with several different indicators, when appraisal is sought, but its importance for high academic morale and as an incentive to produc- tivity is yet to be contested. A graduate program's stimulating environment is often one in which professor and students enjoy many opportunities to engage in discussions, inquiry, and free expression. It is organized to permit faculty and students to voice to their administration their opinions 47 and appreciations; and where formal and informal evaluation criteria andyprocesses receive the participation of leaders and peers. Communications with external agencies in the community and a certain degree of autonomy are also associated with a desirable aca- demic environment (Dressel et al., 1971). CHAPTER III RESEARCH PROCEDURES Quality and excellence are two of the words most often used to express the goal of graduate education. However, institutions have not had uniform success in attaining these attributes (Berelson, 1960). The faculty is reputed to be the mainstay of a graduate program, its members' productivity and group morale being important factors leading to excellence. A corollary observation is that faculty behavior is related to institutional quality (Wilson, 1942; Parsons & Platt, 1968). Moreover, reputation and evaluation are issues of critical significance for professors, who are the subjects of this survey. This study, seeking to identify criteria acceptable to Brazilian professors for the evaluation of graduate programs, utilized a selected group of faculty members. Their opinions, expressed through the ratings and ranking of a given set of indicators, were assumed to reflect not only the criteria acceptable but their willingness to support evaluation. The assessment of criteria for graduate education excellence proceeded from the deve10pment of an instrument to the collection and interpretation of faculty values. 48 49 The primary objectives set for the investigation were (a) to provide an overall picture of the criteria perceived as relevant for Brazilian graduate programs, (b) to observe differences or patterns of values among groups exhibiting various faculty characteristics, (c) to infer factors determining the differences (if there were sig- nificant ones), and (d) to contribute to more rational and empirical evaluations of graduate programs' effectiveness. Research Approach The approach used is basically a survey through which the researcher aimed at description and analysis of limited relationships. The major procedural stages through which the study evolved were as follows: a. deciding the goals of the study and the primary research areas; reviewing the relevant literature and discussion with Brazilian and American experts; designing the study and establishing the major research procedures; proposing the study to a sponsoring agency and revising objectives according to some suggestions; designing the research instruments; testing and revising the research instruments and tech- niques; selecting the subjects to be covered by interviews and questionnaires; collecting data: interviewing or distributing question- naires and securing their return; processing the data: coding questionnaire responses and preparing for computer inputing or transcribing the taped interviews; 50 j. analyzing the information through statistical methods; k. assembling the results and interpreting the statistical analysis; and 1. relating the findings to other research, drawing conclu- sions and making recommendations. Conducting the survey, besides requiring a certain amount of technical knowledge which the investigator was still acquiring during the course of the study, was a prolonged exercise. From the decision to examine which criteria were most valued by Brazilian graduate pro- fessors to the writing of this dissertation, a number of subtle and not-so-subtle changes took place. The process of carrying on a research study and the major steps and decisions undertaken during the investigation are described in this chapter. Survey Design The survey design resulted from the consideration of ques- tions led by a somewhat reverse order of research steps. First, it was determined what kinds of outcomes were wanted and what informa- tion would facilitate arriving at those judgments. Then, the follow- ing aspects were considered: which variables should be measured, what kinds of relationships would be investigated, what kind of sample or groups would be used, and how the subjects should be questioned. The process of survey design aimed at precision, logic- tightness, and efficiency but was bound by a need for relative flexi- bility in field work and economic constraints. The basic research question was determined to be: What are the criteria considered appropriate for the evaluation of graduate 51 programs in Brazil? The desired outcome was the identification of the characteristics regarded as most important in quality graduate programs. Or, in other words, what are the recognized correlates of excellence? A descriptive, enumerative survey enables counting how many members of a population or sample attributed a determined value to each selected program characteristic. An analytical, relational survey permits the exploration of relationships between particular variables. This study was designed with descriptive and analytical phases. Besides the enumerative objective, the researcher aimed at finding associations and tentative explanations between respondents' characteristics and the criteria estimation. However, no prediction was to be attempted, allowing for a relaxation of representativeness. Research Variables The addition of analytical objectives warranted the identi- fication of two kinds of variables. Independent variables (also called explanatory or experimen- tal) are the factors associated with the “opinions" collected. The study was set up so that these factors could be varied systematically and that they could be observed working both in isolation and in various combinations. All respondents' characteristics selected for use in this study were considered as independent variables in the first analysis of this survey. Dependent variables are the results or effects observed. They were measured through two different scales and tested for statistical 52 significance. The numerous criteria or indicators of program quality and efficiency were the established dependent variables. Besides, these, and in order to approximate the "other things being equal" condition, controlled variables must be acknowledged. The criteria used for sampling (e.g., excluding professors on leave or not currently involved in graduate programs, and randomization) aimed at the elimination of some variables as a source of variation in the study. To observe or reduce the influence of some uncontrolled variables, statistical techniques were helpful. However, this inves- tigation, as many surveys, has some shortcomings in terms of dealing successfully with controlled and uncontrolled variables. The research objectives and design are affected by these shortcomings, as well as some instrument limitations. The Develppment of Instruments After deciding on fundamental research questions and design, procedures were established which would make reaching the objectives possible. One or more instruments would be necessary to measure independent and dependent variables. Having a large sample of graduate faculty members and motivated by the desire to avoid some intervening factors led to the choice of a questionnaire as the major instrument of data collection. A self-administered questionnaire made it possible to cover a larger number of subjects at lower cost than interviews. Training field workers, travel, and maintenance expenses would be much more expensive than the preparation of materials, printing, addressing, 53 and mailing. Also, the number of subjects who would be available for interviews at a limited predetermined time seemed to be less than the number likely to respond to a questionnaire returned by mail but hand delivered with a stamped self-addressed envelope for return. Designing the Questionnaire The first decision on questionnaire design concerned question sequences and the order of question sequences. Factual questions would precede attitudinal ones; thus, respondents' characteristics were sought before their opinions or valuation. This should lead the subjects througheasier personal questions about their academic status before demanding more intense participation. To avoid unneeded effort, the questionnaires were individually coded by the researcher in reference to the respondent's university, graduate programs, and his/her status, i.e., program coordinator or regular faculty member. (See first page of model questionnaire in Appendix A.) Other independent variables included in the factual group were: academic ranking, year and institution of undergraduate and graduate degrees, number of professional meetings attended in 1977 and 1978 combined, presentation of any professional paper at those meetings, and the number of articles, books, and monographs produced in 1977 and 1978 combined. Whenever a limited number of options was available for a certain variable, the requested information was to be checked; otherwise, responses were open-ended (page two of model questionnaire). 54 Some questions, still within the respondents' characteristics or independent variables group, but demanding attitudinal response, were also included in the first part of the questionnaire. They solicited the subjects' opinions about the changes-~faculty, library, finances, and students' qualifications--observed in graduate programs within their institutions. Further, their reaction was requested to the value attributed to research activities in Brazilian universities in general, and particularly by the graduate program in which they teach. These were closed questions of a five-point Likert-type (Likert, 1932) scale response (page three of questionnaire). The last section in the instrument's Part I was related to percentage of time spent in the six major academic activities and the ideal schedule. Part II of the questionnaire was designed to collect the subject's attitudes concerning the dependent variables. Specifically, the introductory question read: What importance would you attribute to the following indicators, ifyypu were to evaluate a Brazilian graduate program in your specialty field? The order given was: "Please, consider all the items presented, and circle your choice. Afterwards, select the most important and, if wanted, add any sugges- tions." The items featured in Part II represent the criteria or indi- cators used in evaluation measurement. They were displayed by groups which correspond to the sections in the review of the literature. The number of indicators presented for each criterion is as follows: 55 1. Program 0bjectives--7 2. Curriculum Organization—-ll 3. Curriculum Evaluation Foci--2 4. Evaluation Participants--9 5. Academic Credentials--5 6. Faculty Activity and Productivity--l3 7. Academic Governance--6 I 8. Student Attributes--8 9. Student Activities-—8 10. Physical Facilities--7 11. Library--8 12. Special Facilities-~8 13. Program Organization--8 14. Financial Resources--4 15. Academic Environment--5 The fifteen categories totalled 109 indicators. This large number of items prompted the decision to present an overall instruc- tion and, therefore, bring a higher degree of uniformity and reduce the reading time. However, to enhance the possibilities of identi- fying significantly preferred criteria and to introduce alternative stimuli, this part was to feature both closed and open types of questions. The rptipg, or giving a numerical value to some kind of judgment, was one of the closed techniques selected. Subjects were requested to rate the importance given to each indicator on a five- point scale, ranging from "one" to "five." 56 Ratings, although seemingly the most appropriate technique for this assessment of subjective importance, may be of doubtful validity and suffer from unreliability. Ratings may be easily influ- enced by variables that neither the subject nor the investigator is aware of and bring in the halo effect. To counteract some of these undesirable effects and other response errors, ratings were seconded by ranking. Ranking meant indicating the most important items of each category. Subjects had to choose which were the first, second, and third most valued indi- cators within each criterion. Rankings tell nothing about the differences between ranks; i.e., the distance or interval between ranks may vary greatly. But suitable statistical devices compensate for this shortcoming and the results are useful for the objectives of the study. Each criterion category, after soliciting rating and ranking, provided for an open reaction. The intention was to collect any other alternative indicators which were not listed under the category, or to allow free suggestions to any question features. Summary com- ments and additional suggestions were encouraged at the end of the questionnaire. Planning the Interviews Despite comprehensive literature reviews and several inter- views with American scholars and Brazilian faculty members who were acquainted with graduate studies in Brazil, the investigator sought 57 to enhance her possibilities of better understanding that aspect of higher education. Therefore, a series of formal interviews was planned with leaders at selected institutions of the graduate education system. These interviews enabled personal contact and visits provid- ing for "in loco" view of some programs. Interviews offer great flexibility in eliciting information but are time consuming, expen- sive, and rarely uniform. Also, respondents can not enjoy anonymity, as they can in questionnaires, and feel less free to express views they fear might be disapproved (Selltiz, 1959). Therefore, the deci- sion was made that the investigator would attempt to interview policy makers and high-level university administrators. The elements assembled through these interviews were to serve only as complemen- tary and illustrative for the research analysis. Interviews were planned to be flexible and with open-ended questions. They were "focused interviews" (Merton, Fiske, & Kendall, 1956). The earlier review of the literature and establishing the research objectives helped in setting forth the major areas of inquiry and the questions outlined in the interview guide. The interview guide was planned to: (a) focus attention on salient points in the study, (b) secure comparable data in different interviews, and (c) gather an approximate range of items to support the questionnaires' data analysis. In sum, the guide only indicated topics for further conversation, because the researcher was interested in the meaning of concepts and procedures valued by the interviewees and their effects in the institutions. 58 The following are focal points developed for the interview guide: a. Is the institution, to any degree, involved in the evalua- tion of graduate education? b. What are the most important objectives of graduate education? c. What is the role of research in your institution? d. What are the major features in terms of graduate cur- riculum organization? e. What are the dynamics of academic governance and program deve10pment? Who participates in what? f. How are faculty members selected and evaluated? 9. How are students selected and evaluated? h. What are the most valued facilities for graduate programs? i. How are graduate programs funded? Any additions to or subtractions from this guide were in order, at the researcher-interviewer's discretion. Pilot Tests Almost every aspect of a survey research can be subjected to pilot work. Many ideas and decisions were submitted to various critics at different project stages. However, the questionnaire which was designed specially for the investigation required closer evaluation. The first drafts of the instrument were tested exploratorily. Each criterion category and its list of possible items demanded lengthy discussions with professors and colleagues with interest and expertise. Of particular help were lecturers and some participants of the several short courses held by Michigan State University for 59 Brazilian officials under the PEAS Program (Programa de Ensino Agricola Superior). Members of the dissertation guidance committee aided with question design and basic wording of the items. The formal review of the instrument was done with cooperation from several colleagues, professors at the Universidade Federal do Rio Grande do Sul. The approved questionnaire draft was translated and submitted to a dozen faculty members who, generally, had the same professional characteristics as the survey subjects. Their responses, suggestions, and critiques were then reviewed and accepted by a smaller group of colleagues who were familiar with the research problem and method. Question wording was the basic problem. But other difficul- ties mentioned were: the length of the questionnaire, the time required for making rating and ranking decisions, and the physical appearance of the instrument. The researcher analyzed the possibility of dismembering the Part II items and sending different content for different subjects. Response errors and sampling considerations led to rejecting the alternative; it was felt that the length would not be a major factor in nonresponse. During this preparatory stage, two graduate program coordi- nators were interviewed. Their answers and further suggestions assisted in developing interview skills and in writing the letters that would accompany the questionnaires. 60 Time and cost limited the choices of equipment and materials used; therefore, none of the physical aspects of the questionnaire could be tested. Selection of Subjects As previously stated, a basic assumption in this study is that effective and comprehensive evaluation requires not only quality research but also the participation of the widest range of groups or persons influencing and affected by the program with which they are particularly concerned. An assessment of criteria for the evaluation of graduate studies ideally should encompass the voice of faculty, students, administrators, research contractors, alumni, employers of graduates, and any people who collaborate with or are serviced by a particular program. This was far too excessive for the present undertaking. Thus, it was determined that this investigation would focus primarily on the faculty of Brazilian graduate programs. Further, a sample of faculty members would become potential respondents to the questionnaire. Population Considering a population as the aggregate of all the cases that conform to a designated set of specifications (Selltiz, 1959), the stugy's population consists of faculty members listed at graduate programs, in 1978, of five selected major Brazilian universities: Universidade do 550 Paulo, Universidade Federal do Rio de Janeiro, 61 Universidade Federal do Rio Grande do Sul, Universidade Estadual de Campinas, and Universidade de Brasilia. As of December 1975 (CAPES, l976a), there were forty-eight institutions of higher education offering graduate studies at the master's and doctoral levels. They are located in various geographi- cal regions and, therefore, it was necessary to limit the number of institutions providing elements for the survey sample. The basic criterion for the selection of universities to be included was the size of graduate studies effort, as indicated by numbers of programs offered. According to this criterion, the ten largest institutions are: Universidade do Sao Paulo (USP) Universidade Federal do Rio de Janeiro (UFRJ) Universidade Federal de Minas Gerais (UFMG) Universidade Federal do Rio Grande do Sul (UFRGS) Universidade Estadual de Campinas (UNICAMP) Pontificia Universidade Cat61ica do Rio de Janeirot(PUC/RJ) Universidade Federal’de Pernambuco (UFPE) Universidade de Brasilia (UnB) Escola Paulista de Medicina (EPM) Universidade Federal Fluminense (UFF) OKOCDVOlUl-DUON-d u—l The need to curtail traveling time and expenses determined that the study would involve only five of the largest institutions. Among those selected there were two state universities (Universidade de Sao Paulo and Universidade Estadual de Campinas) and three federal institutions. The federal institutions belong to the official system of higher education and, therefore, are bound by rules and regula- tions for personnel, academic structure, and governance dictated by centralized governmental agencies such as the Ministry of Education and Culture and the Department of Public Service Personnel. 62 Instead of taking the five highest-ranking universities, the investigator opted to include the Universidade de Brasilia. This institution is a federal university but with characteristics some- what different than, for example, Universidade Federal do Rio Grande do Sul and Universidade Federal de Minas Gerais. It was founded as a university, following the regional development plan, and from the beginning had planned higher education with research and graduate studies. The other universities originated from a consortium of sev- eral traditional public colleges, had difficulties in dealing with the concept of university, and graduate programs were initiated more as an extra feature in their institution. The inclusion of Universidade de Brasilia brought variety to the selected group but led to the exclusion of Universidade Federal de Minas Gerais. This institution and the Universidade Federal do Rio Grande do Sul are both universities of similar size and organiza- tional structure, although evidently many particular characteristics are present. But Universidade Federal do Rio Grande do Sul is the researcher's employer; thus many chores were facilitated at home. Sampling Considering the large number of faculty members who were to receive the survey questionnaire, a sample of graduate programs was taken. The ppjt_used for sampling was the program itself. There were two major reasons for sampling programs rather than individual pro— fessors. 63 First, one of the questions being sought was whether the academic position or the administrative position of coordinator had any effect on the responses. Second, from the official publications and the available information, it was impossible to know which faculty actually were involved in the programs at the time. By delivering instruments to each coordinator and asking him/her to forward them to the professors increased the likelihood of reaching every element of the sample. Following this rationale, a list of all graduate programs at the five selected universities was compiled from the 1978 catalogs. Then, eight programs were drawn from each institution. This produced one-phase element sampling through unstratified and random selection. Probability sampling was used to enhance accuracy and confi- dence. Every program had a specified nonzero chance of entering the sample. The sampling process indicated the forty programs that were asked to cooperate in the research. Interviewees The interviewees sought for the study were high academic officers from the universities participating in the study. Since the objective of the interviews was primarily to aid the interpreta- tion of information collected through the questionnaire, the faculty member in charge of research and graduate education was requested to collaborate. Also, a national perspective was attempted. Officials from some agencies represented in the National Council on Graduate Studies (CNPG) would represent the policy makers' view. 64 Data-Collection Procedures Although the distribution of questionnaires and the inter- views occurred in the same period, each required a particular pro- cedure. Soliciting Through the Questionnaires The questionnaires were accompanied by a cover letter of presentation which was addressed to each individual. In that letter, the investigator introduced herself and the research objectives, and solicited the professor's cooperation. It also provided a suggested date of return. This letter was duplicated on official Universidade Federal do Rio Grande do Sul--Faculdade de EducacSo (College of Educa- tion) paper and hand addressed and signed by the researcher. A stamped, self-addressed envelope was attached to the instru- ment. In addition, the front page of each questionnaire also had the investigator's name and address. With some variations, the general procedure for delivering the questionnaires was as follows. After seeking permission and orientation at the central office for graduate studies of each uni- versity, the investigator visited the coordinator of the selected graduate programs. In the majority of cases they were personally requested to collaborate by forwarding to the professors the ques- tionnaire and envelope package and by answering a questionnaire them- selves. When they were not available, the office supervisor permitted an updating of the mailing list and received the packages for dis- tribution. The coordinators' questionnaires were then accompanied 65 by a letter requesting special attention. In some instances, the investigator had the opportunity to meet a few faculty members and encourage their response. One exception to the procedure occurred at the Universidade Federal do Rio de Janeiro. As advised by the central graduate studies office, the university mail was used to distribute the questionnaires. A large package of questionnaires and attachments was sent to each program coordinator. The letters were addressed according to an available faculty list. The Universidade Federal do Rio de Janeiro is an institution that still has a significant number of programs scattered over Rio de Janeiro's metropolitan area. Reaching the eight programs in three days was impossible. One week after the instruments were distributed, the researcher sent a personal letter to each program coordinator outside her uni- versity. The remainder were reached by phone calls. The intention was to thank them for the assistance and to request additional incentive for faculty response. Questionnaire and letter facsimiles are in Appendix A. Interviewing Interview appointments were arranged directly by the inves- tigator with the office of the persons requested. The interview guide was used in a loose manner and all but one were taped, at the inter- viewee's discretion. Afterwards, the conversations were transcribed for easier utilization. 66 According to the time available, the meetings averaged an hour and one-half. The general attitude displayed was of extreme cordiality and interest in the research issue. Some interviews covered every aspect listed and facilitated detailed discussion of problems and values in Brazilian graduate edu- cation. Others were less thorough but each one provided a fruitful product. No formal validation of the interviews was deemed neces- sary. Their objectives and style coupled with the status of the interviewees sufficed. Preparation of Survey Data As the questionnaires were returned, it was necessary to decide a means of transforming the raw survey data into a meaning- ful set of results. The methods of analysis were determined jointly by the nature of the survey variables and the survey objectives. This investigation had objectives ranging from a simple des- cription of the population to an exploration of a complex set of relationships between the population variables and the set of depen- dent variables. The state of the knowledge in the particular field did not permit advancing stipulated relationships. Consequently, the use of descriptive and inductive data-analysis procedures was more promising than the formulation of straight testable hypotheses. O'Muircheartaigh and Payne (1977) present a description and evaluation of statistical methods of data analysis which provided guidance for the decisions on preparation and processing of the survey 67 data. An account of major procedures undertaken and a brief rationale, when applicable, follows. Organizing the Survey Data File The Data Matrix The rectangular data matrix, giving the values of the vari— ables for each case, indicates a hierarchical structure with several levels of interest. Figure 2 is an illustration. Data were collected from each faculty member, which, besides exhibiting many academic and attitudinal attributes, may be gathered by program, knowledge area, and university. 0 *— University éééééz~wm TI. LP W In! ..——-Faculty Member Figure 2.--Data matrix. Intermediate subgroups, represented as additional levels within the basic structure, are simple to identify. An example is the classi- fication of professors by academic rank, degrees obtained, or publish- ing productivity. In fact, the hierarchical structure may be comple- mented by a tree diagram which permits to each survey subject a display of several concomitant characteristics. 68 Data File Features The data collected show special features which dictate how they are handled in both file construction and analysis. Sjgp: The survey questionnaires were sent to 455 cases and 146 were returned. Each subject established values for 173 variables. The data of each case were arranged over three punched cards for a total of 438 ordered cards in file. Subgroups: The cases in the data matrix were identified by university, program, and the faculty condition (coordinator or regular faculty member). This allowed the analyses of all cases and separate cases for each category or combination. The data matrix is illustra- tive. Missing data: Inevitably, there are missing data because questionnaires were incomplete, not returned, or presented a problem during the data processing. This situation is described in a later section. Data validation: Due to time, expenses, and consequently design limitations, quality control of the raw data was rather low. However, the respondents' characteristics and the type of survey instrument were factors diminishing the need for checking misinter- pretations and miSplaced marks. Careful control was exercised in coding and punching. Processing the Survey Files Data file features determined the procedures for processing and analysis of data files. A sequence of activities was devised and is described below. 69 Step one: The major process was converting the data from questionnaires into base data file in machine-readable form. Primary mpgjp_were used; i.e., the questionnaire data were transferred to coding sheets which facilitated card punching. Card punching was preferred over the secondary media because they are universally available and all computer packages can handle data on cards. These were relevant considerations since the first analyses, used in this dissertation, were done with Michigan State University computers and further work is planned in Brazil. Recording data on cards required the observation of some basic rules: (a) avoiding nonnumerical codes, (b) avoiding multi- punched codes, (c) assigning a code to every value of a variable, and (d) using a fixed-column format. Each questionnaire, it should be remembered, had a coded number which facilitated the identification of each case. Additional identification was included to indicate the sequence of cards for each case. These numbers permitted not only sorting of cases and cards but simple aggregation of cases. Card layout is shown in Appendix 8. Step two: This involved data quality control through the following activities: (a) coding check as coding is carried out, (6) card punching verification, (c) card reading through comparison of coding sheets and computer reading, and (d) complete file check by comparing the identification number of every card with matching questionnaire code. 70 Step three: Although the sampling process was quite simple and the survey design did not include transformations, weighting of cases and variables, or the addition of new data, special decisions were required to prepare the data file for statistical analysis. One of them was establishing criteria to deal with missing data. Reviewipg the Data File As previously stated, the survey conditions and objectives helped to determine the methods of statistical analysis. The observations collected ranged from objective subject characteristics to attitudes expressed through ratings and rankings. Scales of Measurement Measurement of any variable is based in the concept of an "admissible transformation” and follows two basic conditions: (a) each object or individual is either equivalent or nonequivalent to a category, and (b) the equivalence relationship is symmetric and transitive (Coombs, 1953). According to the nature of each variable observed, different scales of measurement are used. It is important to identify which scales apply because they determine the apprOpriate method of analysis. The nominal scale was used for the identification of univer- sity, graduate program, coordinator or regular faculty, academic rank, year and institution of degrees, and presentation of papers at meet- ings. This scale admits only frequency counts, calculation of mode, and the use of chi-square or coefficients of contingency. 71 The ordinal scale was suitable for the measurement of number of publications, meetings attended, hours spent for typical academic activities, faculty opinion about change, the value of research in universities and their particular program, and the ranking attributed to some variables in each category. Nonparametric and ranking methods such as Spearman's Rho and Kendall's Tau are adequate. The interval scale allowed the measurement of the ratings given to each dependent variable or evaluation indicator. Since a unit of measurement was defined and the numerical values suggest equal intervals between the scale positions, more stringent statisti- cal methods can be used. Means, variances, correlations, and all parametric techniques are applicable. After understanding the measurement scales, a review of the survey design led to the evaluation of methodological problems actually encountered as a preparation for the data analysis. Methodological Difficulties Although some of these difficulties or special features were foreseen and dealt with in previous sections, the major ones should be closely analyzed. Nonresponse was the most frequent and disturbing problem. Respondents were not under any obligation to cooperate. Only one attempt to collect the information was directly made. Absolute con- trol over the delivery of questionnaires was impossible. The instru- ments were given to the coordinator's office of each program; beyond that, the researcher does not know how and when they were forwarded 72 to the faculty members. Besides, there was one coordinator who formally declined participation and one who declared the question- naires were not delivered. These two programs were withdrawn from the sample. Another source of nonreturn might be the conscious individual refusal. Many surely did not bother to read or to return the ques- tionnaire. One professor, for example, replied that it was a waste of his time and that he did not believe in any social survey results. Another, whom the investigator had the opportunity to meet somewhere else, said that despite being listed in the faculty, he had not served any functions in the program during the last two years and that he did not consider himself its member. These are just examples of this problem. Something defined informally by some colleagues as "cultural" difficulty to reply on time to any incoming mail and research requests should also be mentioned, but with a cautious tone. This expression is related to the lack of correspondence reply habits or cooperation spirit. There were responses lost due to administrative problems. Some professors may have been on sabbatical or sick leaves and their names would still be in the official listings. One program with approximately a dozen faculty members returned two questionnaires, say- ing that the addresses were abroad for the term and, then, their names were withdrawn from the count. 73 Other grrprg might be present in the survey's results due to the quality of the questionnaire, the execution of the field work, or other causes. The questionnaire directions and questions may not be under- standable to the subject and lead to either a nonresponse or to an inaccurate response. The second problem is of impossible detection through this research's design. The first would require detailed analysis of response bias, response variances, or other error measure- ments. Neither investigation is within the study's scope. Response errors cannot be reduced or eliminated simply by increasing the number of observations; they apply equally to census and sample surveys (Kish, 1965). Decreasing any errors' rate demands a special study of the questionnaire and the survey design which may be initiated through material collected in this project but requires lengthy and technical expertise. A final sourcecrflnuch anguish and frustration was the loss of some collected questionnaires which were seriously damanged in a machine failure at a U.S. Mail distribution center. The question- naires had been inventoried in Brazil but were being taken to the United States for coding and processing. Appendix C presents a summary of questionnaires delivered and returned. Data Analysis The research objectives and design and the nature of data collected determined the data-analysis procedures. Two basic 74 data-analysis phases (a descriptive and an analytical) evolved from the information gathered. Descriptive Analysis The objective of this phase was to organize an overall pic- ture of the respondents' characteristics and the importance they attributed to the evaluation criteria or their indicators. Although guiding research questions could be identified, during the research's planning stage, the analysis follows the values given to the list of respondents' characteristics and the evaluation indicators. The respondents' characteristics of the survey included: Subject Identification: 1. University Academic Data: 2. Program 3. Program coordinator or faculty member . Academic position Bachelor's degree year Bachelor's degree institution . Master's degree year . Master's degree institution . Doctoral degree year Doctoral degree institution . Any other advanced degree . National meetings . International meetings . Paper presentation . Articles . Books . Monographs Perception About Change: 18. Faculty quality 19. Library 20. Finances 21. Student quality Perception About Research Value: 22. Brazilian universities 23. Individual program 75 Work Schedule: 24. Actual undergraduates 25. Actual graduates 26. Actual research 27. Actual extention 28. Actual administration 29. Actual outside 30. Ideal undergraduates 31. Ideal graduates 32. Ideal research 33. Ideal extension 34. Ideal administration 35. Ideal outside The results obtained for these variables were organized through frequency distributions and modes for every variable. A second section, also within the descriptive phase, analyzed the dependent variables. The ratings of importance provided for the listed evaluation indicators were studied through frequency distribu- tions, means, and standard deviations. The rankings within each criterion were determined by the mode and relative frequency of responses. Some comments or suggestions added at each criterion's open- end were reported following the quantitative descriptive analysis. Relational Analysis This second analytical phase permitted the examination of relationships between selected independent and dependent variables. The purpose of such analysis was to contribute to decisions on the identification of evaluation criteria for specific situations. It could indicate the existence of any differences or patterns of values among faculty groups, or associations between the indicators. Since the scope of the dissertation's analysis had to be limited, the researcher identified the following respondents' 76 characteristics to be cross-tabulated with the selected evaluation criteria indicators: coordinator or regular faculty member academic rank origin of master's degree origin of doctoral degree presentation of papers publication of articles 'th Q0 (TO) The dependent variables selected for this relational analysis were the highest-ranking indicators of each criterion set, excluding the evaluation foci (number three) which had only two alternative indicators. In the first relational analysis, the selected independent and dependent variables were cross-tabulated. Frequency distribu- tions of cases according to the classificatory criteria and chi- squares allowed the identification of significant relationships. Besides this analysis, the first-ranking indicators of each criterion were cross-tabulated among themselves and the significance of their relationships assessed. CHAPTER IV SURVEY RESULTS In this chapter the data collected through the survey's ques- tionnaire are organized and analyzed in order to answer the proposed research problem. The results described herein represent a substan- tial but not a complete review of the outcomes. They were selected on the basis of consistency with the research objectives and the need to reduce the numerous possible findings for the purpose of this dis- sertation. Data treatment involved basic descriptive statistical techniques and restricted relational analysis. The chapter comprises four major sections. The first focuses on the research subjects' characteristics, including some attitudinal and academic information. The second section analyzes the data on dependent variables and weighs the importance attributed by the faculty members to each evaluation criterion and indicator. The third section reports the results of an exploration into a limited number of relationships among some independent variables and the indicator most frequently placed in the first-ranking position of each criterion area. In the last section, there is an analytical summary of findings reported in the preceding sections. ‘77 78 Respondents' Characteristics As mentioned before, the investigation involved faculty members from sampled graduate programs at five major Brazilian uni- versities. The number of respondents to the questionnaire totals 146 professors. Among them, there were fourteen holding administra- tive positions as the coordinators of their graduate programs. Each respondent was asked to indicate his/her major academic characteristics and accomplishments and his/her attitude concerning change and research. The responses are described below. Academic Position Although institutions may vary in the subdivisions used and the criteria for placing faculty in the academic ranks, they all fall into four major traditional categories. Most of the graduate faculty members held positions of asso- ciate professor (35.6%) or assistant professor (34.2%). Considerably fewer were full professors (18.5%). The remaining probably were visit- ing scholars or temporarily assigned faculty. No instructors were identified. Academic Degrees The majority of survey subjects earned a bachelor's degree in Brazil (72.1%); in fact, 45.6% graduated from the same institution where they currently work. Fifty-four percent of the respondents graduated between 1960 and 1969, and 24.8% during the 19505. The remainder were almost evenly divided between before 1950 and after 1970. 79 Master's degrees had been earned by 57.7% of the group; only 25.9% of them received these diplomas from the same institution where they presently teach. The largest number (39.5%) were granted by North-American institutions. Others were from Brazilian Universities (14.8%), European universities (13.6%), and other institutions (6.2%). Again, it was during the 19605 that most degrees were received (48.8%) but 41.7% were granted between 1970 and 1974. Doctoral-level degrees were held by 67.8% of the faculty mem- bers. Among them, one-third were awarded by universities in the United States, one-fourth by the Brazilian institution where they work, 21% by European universities, and 14% by Brazilian institutions other than the faculty members' employer. The rest were conferred by universities in other countries in Latin America and in Asia. Doc- toral degrees were earned more recently: 68.7% after 1960 and 26.3% in the 19505. Approximately one-fifth of the subjects were awarded the title of Livre Docente. This degree is granted by Brazilian univer- sities recognizing a faculty member's scholarly competence, and requiring, among other criteria, the presentation of a thesis. Livre Docente diplomas were traditionally the highest academic degree granted in Brazil, but are being discontinued by educational legisla- tion, as master's and doctoral programs become more available. Professional Meetings During 1977 and 1978, 78.1% of the subjects attended national professional meetings; of those, 39.5% participated in only one event, 80 29.8% in two, and 14.9% in three. In addition, half of the faculty attended international meetings and, of those, the majority (68.5%) went to one international meeting. The majority of faculty members (76.5%) also indicated that they presented papers at those meetings. Publications Twenty-one and nine-tenths percent of the respondents did not publish any articles during 1977 and 1978, 15.1% published one, 18.5% two, 11% three, and the remainder four or more. Close to one-fifth of the faculty members produced pppk§5 the majority of those (69.0%) had one published during 1977 and 1978. Monographs were published by 26.7% of the professors, with 59.0% of them having produced one volume and the others from one to six. Perceptions of Change Faculty members who had been involved in the program at least since 1976 were questioned about the perceived changes in their institutions. The aspects presented were faculty quality, library quality, financial resources, and student quality. Their attitudes could be expressed as: much better, better, little or no change, worse, and no opinion responses. The highest frequency responses were: (a) for faculty quality, better; (b) for library quality, better; (c) for financial resources, little or no change; and (d) for student quality, little or no change. 81 Opinions about change ranged for all four aspects from "worse" to "much better." A favorable tendency, i.e., "better" and "much better" responses, outnumbered "worse" and "little or no change" for faculty and library qualities; but a less favorable Opinion was found for financial resources and student quality, where "worse" and "little or no change," together, were predominant. Perceived Value of Research Faculty members were also asked about their attitude concern- ing the emphasis placed on research activities. Overwhelmingly, the subjects disagreed with the statement that "research activities have received excessive value, in contrast to teaching, at Brazilian universities" (62%), The largest group pre- ferred the "disagree" rather than the "strongly disagree" alternative (3 %). However, when the same statement focused on the graduate pro- gram they belong to, professors strongly disagreed about excessive emphasis on research (39%). Academic Activities--Actua1 and Ideal In actual practice, most faculty members divided their work schedule primarily between teaching and research. Concerning teach- ing, graduate students required more time than undergraduates. Involvement in academic administration appeared as the fourth-ranking demand for time. Extension and community services were not done at all by the majority of the respondents. Also, approximately one-third 82 of the professors engaged in regular professional activities outside the university. Concerning the same subdivision of the work schedule for an "ideal" relative allocation of time, 12.8% of the faculty members said they did not want any involvement in undergraduate teaching and advising. But the majority felt they should devote up to 25% of their hours to that responsibility. Graduate students should account for 26 to 45% of the work schedule; research should consume the same or more hours. Extension and administration should require some, but little, time for approximately 60% of the professors. The same pro- portion would rather not work outside the university. The tables showing this section's data are in Appendix 0. Evaluation Criteria Assessment The survey subjects provided factors for an assessment of the proposed evaluation criteria. Faculty members returning the questionnaires rated each criterion's set of indicators and ranked the most important indicator within the criterion. The analysis of results is reported in the summary table organized for each criterion, by indicator, using the questionnaire in translation. Means and standard defiations were calculated from the frequency distribution of ratings. Also, the mode indicator for first-, second-, and third-ranking importance is identified in most criterion sets. In addition, a selective account of pertinent com- ments collected at the open-end of each criterion concludes this descriptive analysis of the dependent variables' values. 83 Criterion One A graduate program's objectives should emphasize: Mgpp__ 3.0. a. contribution of the subject area to the social and economic development. 4.219 .983 b. the role of the area's professional. 3.341 1.149 c. the importance of research activities. 4.096 .950 d. the domain of subject matter content. 4.029 1.025 e. critical thinking. 4.547 .857 f. using processes and techniques associated with professional competence. 3.970 1.014 g. decision-making. 3.801 1.141 The highest-rated objective emphasis was critical thinking, which was also the indicator with the least dispersion of frequencies. However, when taking the options as a set, the contribution of the subject area to the social and economic development ranked highest, with 38.7% of the subjects' support. Second ranking was equally divided between critical thinking and the importance of research activities, with 23.9% of the faculty indicating each one. Third ranked was using processes and techniques associated with pro- fessional competence, but the frequencies were scattered and only 19.1% agreed. Many comments or suggestions were offered concerning the objectives a graduate program should emphasize. Among these, prepara- tion for teaching career and the development of national technologies (applied research) stood out. It was also observed that most of the objectives were interdependent and intersectioned. 84 Criterion Two A graduate program's curriculum should include: Mean 5.0. a. learning experiences flexible enough to permit the students a development according to their individual needs and talents. 4.051 1.116 b. credit for independent or individualized studies. 3.401 1.263 c. elective or optional courses within the major area. 3.971 1.046 d. elective or optional courses in other areas. 3.277 1.241 e. a core of required courses. 3.964 1.358 f. opportunities to transfer some credits of same level from other institutions. 3.467 1.207 9. opportunities to wave prerequisites by examination. 2.920 1.362 h. teaching methods adequate to content and objectives. 4.125 1.036 i. periodical evaluations of student's academic progress. 3.787 1.151 j. mandatory full—time attendance, at least during part of the program. 3.584 1.365 k. research experience required during the program. 4.190 1.115 The means of ratings indicate that mandatory research experi- ence was the most valued single indicator of criterion two. The ranking frequencies pointed out flexible learning experiences as the highest priority for 33.6% of the faculty members. Second ranking was the research experience item with 19.3% response. Least dispersion occurred in the teaching_methods indicator and the most with respect to mandatory attendance. This "curriculum organization" criterion brought the largest number of comments and suggestions. Remarks stressed the complexity 85 of graduate programs' requirements and that a balance between the various indicators was a desirable feature. Integration among courses and their objectives, the need to monitor time requirements for effi- ciency, and the difficulties of implementing "flexibility" were also mentioned. Some respondents expressed a foreboding concern for the teaching methodology indicator. Criterion Three A formal curriculum evaluation should consider if the program's: Mean 5.0. a. phi1050phy and objectives answer to societal needs. 4.421 .915 b. objectives are being attained. 4.647 .654 This criterion concerning evaluation foci presented only two indicators, which are the basic evaluation approaches. The assessment of program's objectives attainment was favored by the ratings' fre- quencies and showed more concentration of opinions. No ranking was solicited because of the reduced number of opinions. Few faculty members reacted to this criterion, most display- ing lack of understanding or skepticism about how societal needs are determined and about a "typical bureaucratic" question. 86 Criterion Four Participants in a graduate program's evaluation should be: Mgpp__ 5.0. a. the program's faculty members. 4.617 .842 b. the program coordinator (administrator). 4.326 .945 c. other university administrators related to the program. 3.128 1.206 d. subject area specialists from other institutions. 4.000 1.194 e. foreign specialists. 2.766 1.268 f. presently enrolled students. 3.409 1.380 g. program alumni. 3.507 1.222 h. temporary or permanent drop-outs. 1.954 1.174 i. program clients: alumni employers, research consumers, etc. 3.203 1.335 The means of ratings and the ranking of indicators coincide. Most important is the participation of the prpgram's faculty, followed by the coordinator in charge, and by the judgment of ppecialists. The level of agreement, observed through lower deviations from the means, is also higher for those indicators. Many observations were collected in the open—ended part. They ranged from support to "no value" for administrators, alumni, and clients. They stressed the point that just being a foreigner does not necessarily qualify one as a specialist; he/she should be acquainted with Brazilian reality and be invited only when there are no nationals available. Other suggestions included evaluation specialist to par- ticipate in design and analysis of data, and an outstanding scholar from another area to bring a broad perspective into the program's significance assessment. 87 Criterion Five The facultyfs academic and professional qualifications should be evaluated hy: Mean S.D. a. graduate degrees in their specialty teaching area. 4.058 1.062 b. professional, teaching and research activities compatible with the program's objectives. 4.814 .426 c. the degree of diversification between the institutions where the faculty members earned their degrees. 2.920 1.219 d. distinctions received by the faculty in their professional area. 3.324 1.167 e. faculty participation in professional development activities. 4.364 .907 This criterion's indicators also provided for an importance hierarchy which coincides with the rankings indicated. Both systems point out that faculty's activities were the most-valued indicator of academic qualification. Second ranking was professional develppment and third, although the questionnaire failed to ask specifically, was the graduate degrees held by the professors. Open-end contributions focus on the difficulties faced by Brazilian universities to achieve any diversification of academic origin among the faculty degrees and the need to value scientific production. Furthermore, there were comments on the subjectiveness or partiality of distinctions awarded and the uneven quality of con- tinuing education programs. 88 Criterion Six Faculty activity which should be valued: Mean S.D. a. undergraduate teaching. 3.920 1.138 b. teaching graduate courses or seminars. 4.514 .747 c. advising graduate students. 3.927 1.062 d. thesis and/or dissertation guidance. 4.612 .665 e. curriculum planning and evaluation. 3.464 1.160 f. opportunizing students' evaluation of their teaching. 3.511 1.227 9. research production or other activities which foster the knowledge development. 4.777 .482 h. publications of books and monographs in their subject area. 4.130 .935 i. publication of articles or reviews in national journals. 4.174 .958 j. publication of articles in international journals. 4.237 .929 k. participation in community services (extension). 3.312 1.158 1. presentation of papers in professional meetings. 3.949 1.137 m. association in professional organizations. 3.000 1.350 The first-ranking (with 28.4% of response) and highest-rated indicator of faculty activity was research production. This item also divided the second-ranking highest frequency with graduate teaching, each receiving 19.3% of responses. However, the second-highest rating indicated thesis/dissertation guidance. The standard deviation fig- ures show a considerably higher level of concentration in the ratings for the three highest-rated indicators. 89 Faculty members responding to the open-end portions said that it is difficult to decide the order of importance among the indicators and that it might depend on the objectives of the evalua- tion. Some stated that there is hardly any distinction between the kinds of publications. Additional suggestions included the capacity to attract research funds and the ability professors have to communi- cate with the external environment. Criterion Seven The faculty members should participate in: a. faculty recruitment, selection, and promotion. b. evaluation of peers' productivity. c. recruitment and selection of students. d. formulation of institutional policies and administration. e. faculty load distribution. f. the program's financial resources allocation. Mean 4.187 3.307 4.252 3.942 3.775 3.761 5.0. .937 1.337 .993 1.214 1.153 1.156 The average rating of indicators shows that faculty partici- pation in recruitment and selection of students was the most-valued activity in academic governance and administration. Faculty recruit- ment, selection and promotion decisions came second, following the rates. However, the ranking procedure determines that 37.0% of the respondents agreed on the formulation of policies and administration as the most important activity. Second ranking was the recruitment 90 and selection of students (29.3%) and third ranking was the partici- pation in financial resources allocation (24% of responses). Professors' comments to this criterion seem to support the participation of faculty in any and all academic decisions, although some voiced the opinion that they should not decide "alone" and that some of the indicators are the responsibility of administrators. Criterion Eight The selection of students should be based on: Mean 5.0. a. past school records. 3.511 1.225 b. recommendations from former teachers. 3.237 1.198 c. present or past employers' recommendations 2.978 1.129 d. former professional experience. 3.770 1.014 e. statement of professional and academic objectives. 3.881 1.100 f. entrance examination for the specific subject area. 4.217 .957 g. standardized tests of intellectual ability. 3.097 1.326 h. the recommendations of the program's faculty, after interview. 4.212 1.010 The means of ratings and the ranking frequencies identify entrance examinations as the most important information for the selection of students; 36.2% of the respondents expressed this rank- ing position. The second and third rankings were won by the rggpmr mendations of the faculty, after interview, which received close to one-fourth of the frequencies for each of both positions. The stan- dard deviation also indicates the least dispersion of values for the highest-ranking alternative. 91 Reactions to this criterion ranged from extreme rejection Of standardized tests tO the use Of a wide variety Of informational items. A professor proposed that admissions should be at the academic advisor's discretion; and some respondents preferred careful scrutiny Of all data, mostly Of the letters Of recommendation. Another sug- gestion was toward planned recruitment in the programs which tradi- tionally provide good candidates. Criterion Nine Students should have access to: Mean S.D. a. published rights and responsibilities before their university. 4.181 1.160 b. formal rules for curriculum, admissions, academic progress, graduation, and drop outs. 4.388 1.032 c. representation in the academic decision- making process. 3.854 1.306 d. participation in curriculum planning and evaluation. 3.314 1.363 e. Opportunities for participation in research or administrative activities with pay. 3.686 1.188 f. Opportunities to earn merit scholarships. 4.553 .806 9. academic advising and professional guidance. 4.393 .957 h. space and material resources for study. 4.550 .842 The ratings mean and the ranking frequencies indicated that merit scholarships were considered the highest priority in terms Of students' rights or activities; 21.4% of the respondents declared that indicator in the first rank. Second-ranking indicator was also the same alternative, with 23.7% Of the professors pointing that out. 92 The third-ranked highest frequency was achieved by space and materials, where 22.8% of the marks were counted. Expressions in the Open-end part demonstrate support for student benefits and rights. Some stated that the students' partici- pation in academic decisions should be limited, and that some cleri- cal Or other activities not directly related to the student's academic Objectives might be disruptive to the study's progress. Criterion Ten A graduate program should have: Mean S.D. a. private or semi-private offices for faculty and technical personnel. 4.567 .936 b. Offices for the coordinators. 4.307 1.199 c. space for support personnel. 4.293 .963 d. adequate classrooms, meeting rooms and laboratories. 4.871 .413 e. research space and equipment. 4.837 .472 f. didactic space and equipment. 4.707 .569 9. space for recreational or extra-curricular activities. 3.081 1.276 Ratings and means coincided to point out that the most-valued physical facility for a graduate program was space for teaching; 38.6% of the faculty members indicated this as the first-ranking indicator. Second was §pace and equipment for research, with 36.0% Of the fre- quencies, and third ranking was didactic support, which added to 32.4% of the ranking marks. Contributions from the respondents suggested that not only the existence Of good and appropriate facilities is important but 93 that the efficient use Of them is crucial. Other alternatives ranged from individual space for students to each professor sharing the Office with the advisees. Criterion Eleven The library should have: Mean S.D. a. current collections of periodicals in the program's subject area. 4.943 .233 b. bOOks and monographs in the specialty area and related subjects. 4.857 .408 c. reference materials as: abstracts, catalogs, and bibliographies. 4.729 .666 d. collective catalogs. 4.341 .900 e. interlibrary loans. 4.294 .912 f. personnel for readers' assistance. 4.036 1.049 9. space for individual and group work. 4.079 1.057 h. acquisitions plan and budget. 4.610 .690 The library criterion ratings correspond to the ranking results. Therefore, periodical collections was the most important indicator for the faculty members; second ranking was the books and monographs, and third the reference materials. These rankings achieved 78.1%, 52.6%, and 32.8% Of the responses, respectively. The standard deviations also indicate concentration Of those rating frequencies. Among the comments, there were emphases for more use Of library facilities and for longer hours Of services. One professor argued that no space for groups sessions is necessary; yet another claimed that the "technical and cultural level" Of the librarians needed upgrading. 94 Criterion Twelve Other resources that a graduate program should have: Mean S.D. a. accredited institutions and places for use and training of students and faculty (hospital, school, . . .). 4.296 1.015 b. adequate number Of clinical and instrumental resources. 4.301 .985 c. clerical and administrative support. 4.254 .855 d. didactic resources. 4.409 .753 e. human and physical resources for research. 4.752 .511 f. contracted institutions for external activities. 3.840 .918 9. data processing services for faculty use. 4.029 1.003 h. data processing services for student use. 3.912 1.025 Ratings and ranking frequencies also coincided identifying the most important indicators Of auxiliary resources. The first rank- ing was human and physical research resources, which comprised one- third Of the responses; the second ranking was adeguate number Of clinical and instrumental resources with one-fourth Of the responses; and third ranking was the same indicator ranked in first place. The standard deviation Of the first—ranking indicator showed the least dispersion. Faculty members Observed that this particular criterion's values depend greatly on the nature of the program's subject matter. Support for the availability Of data-processing services for student use varied. An additional suggestion was that counseling services should be provided for students, faculty, administrators, and employees. 95 Criterion Thirteen A graduate program's organization should have: Mean S.D. a. the philosophy and Objectives compatible with the university's program. 4.170 1.026 b. the structure and functioning compatible with the university's program. 3.962 1.066 c. a functional relationship with university's central administration. 3.827 1.098 d. relationships with administrative and support units. 3.993 .922 e. faculty and student representatives in the university collective decision bodies. 4.368 .857 f. organizational rules and policies for faculty rights and responsibilities. 4.173 .933 g. the participation Of all faculty members in the several activities, according tO their rank and responsibilities. 4.321 .978 h. the minutes Of faculty and administrative decisions current and available. 3.967 1.090 The means Of ratings indicate that the most valued indicators were: student and faculty representation in the university, wide participation of the faculty in program's activities, and program's pylayg, However, the ranking, considering all the indicators as a set, pointed out that compatible philosophy and Objectives was first with 38.9% Of the responses. Second was compatible structure and functioning with 24.4% Of the responses; and third was wide partici- pation of faculty in program's activities, which received 21.4% Of the responses. The comments Offered by the respondents ranged from complete disregard for organizational characteristics ("bureaucracy and 96 politicking") to statements about the need to have compatible and well-planned organization in order to achieve the proposed Objectives of graduate education. One faculty member included a built-in evalua- tion and feedback unit in the program's organization. Criterion Fourteen A program's financial resources should be: Mean S.D. a. compatible and proportional to the university's financial resources. 3.576 1.371 b. compatible and proportional to resources available tO similar national programs. 3.985 1.215 c. proportionally distributed among teaching, research, and extension, according to the faculty activities. 4.180 1.104 d. proportional to the importance of the subject area for the country's social and economic development. 4.263 1.153 This criterion's indicators were also rated and ranked in corresponding order Of importance. The first ranking was resources proportional to the socio—economical development needs with 34.6% Of response; the second was allocation according to faculty activities load with 43.0% of frequencies; and third ranking was proportional to resources available for similar programs with 33.1% of the responses. Faculty members pointed out that the allocation of financial resources is a complex problem; that it is difficult to assess rela- tive importance; that there is a National Plan of Graduate Studies which regulated much Of the resources; and that finances have tO be at least compatible with the Objectives set for the program. 97 Criterion Fifteen TO stimulate the academic and scientific production, a graduate prpgram should: Mean S.D. a. permit tO students and faculty discussion, investigation, and expression. 4.839 .458 b. inform the criteria used for the evaluation Of students and faculty. 4.067 1.066 c. Offer channels for receiving and appraising Opinions and grievances from faculty and students. 4.199 .842 d. involve other community institutions in the educational process. 3.612 1.223 e. ask faculty and students to contribute for the periodical evaluation Of the program and its administration. 4.163 .971 The first-ranked and highest-rated indicator of this criterion concerning organization environment and morale was the freedom Of discussion, investigation and expression. This was supported in 75.3% Of the responses. Second ranked was information Of evaluative criteria supported in 30.6% Of the responses. The third-ranked indicator was the conduct Of Opinions and grievances, with 38.2% Of the respondents indicating this choice. Among the comments from the faculty members, there were remarks about the need to implement the suggestions and to acknowledge griev- ances and about the extent Of freedom to express ("in general or in scientific matters?"). One professor said that, although very sig- nificant, the indicators presented did not enhance the scientific production--referring to the criterion's question. 98 Exploring Relationships In addition to the main descriptive interest, the survey permitted the exploration Of relationships between variables. Two basic kinds Of data association were examined. The first maintained the previously established independent and dependent variables; the second sought interrelationships among selected variables. Respondents and Rankings As described in Chapter III, the researcher selected six variables related to the respondents' descriptors: (a) coordinator or regular faculty member, (b) academic position, (c) origin of master's degree, (d) origin Of doctoral degree, (e) presentation of papers, and (f) publication Of articles. These data were cross- tabulated with the three highest-ranking indicators Of each criterion, which are identified earlier in this chapter. Contingency tables were organized through SPSS's subprogram CROSSTABS. The chi-square test was used to determine whether a system- matic relationship existed between each independent variable and the three highest-ranking dependent variables Of each criterion set. The statistical significance test pointed out the probability or likeli— hood that an Observed relationship could have happened. Significant relationships at the .05 level were found between: a. the faculty's academic position and the rankings given to the library indicators. The raw chi-square was 22.96610, with nine degrees Of freedom, indicating a systematic relationship between the variables; the contingency coefficient was .37199, indicating the 99 strength of the relationship. It was Observed that the respondents who fell in the "other" academic position (visiting professors?) presented a response pattern different than the subjects Of the other three categories. b. the presentation of papers at professional meetings and the ranking given to indicators of academic governance. The raw chi- square was 8.25015, with three degrees of freedom, indicating the statistical dependence; the contingency coefficient was .24083, indi- cating the strength Of the relationship. Subjects who presented papers seemed to have more diversified perceptions Of the indicators' rankings. At the .09 level, significant relationships were seen between: (a) coordination Of graduate program and the value attributed to dif- ferent program Objectives, and to indicators Of academic environment; (b) academic rank and curriculum organization features and financial resources allocation indicators; and (c) publication Of articles and student attributes and library items. Relationships Among Indicators In order to seek some possible structure or interdependence among the dependent variables, a cross-tabulation Of some indicators was developed. The number Of variables was limited; the first-ranking indicator in each criterion set was isolated from the others. There- fore, two dimensions were given. Significant relationships at the .05 level were found between: 100 a. faculty research activity and classroomp, meeting rooms, and laboratories. The corrected chi-square was 4.51893 with one degree of freedom, indicating the statistical dependence. The asso- ciation between the variables was measured through Phi (.13257) and Gamma (-l.OOOOO). The latter indicates that discordant pairs predomi- nate; i.e., subjects choosing indicator research activity_are not likely to choose classrooms, meeting rooms and laboratories. b. merit scholarship for students and current collections Of periodicals at the library. The corrected chi-square was 4.03579 with one degree Of freedom. Phi measure Of association was .18925 and Gamma .49384 indicating that concordant pairs predominate. Overview An overview of the data gathered from the respondents indi- cates that the subjects hold relatively low-seniority positions, but all academic ranks are represented in those graduate programs, except for the instructors who are not formally in the graduate-level track. The dates for undergraduate degrees suggest that these faculty members are relatively young. Over two-thirds graduated less than twenty years ago. Graduate studies and degrees held do not date back for a long period for most professors. In fact, many completed either a master's or a doctoral degree in this decade, and many others have not received their highest graduate degree yet. While a majority of the subjects were educated in Brazil at the undergraduate level, most Of the professors attended foreign institutions for their graduate studies. 101 Most faculty members are involved in professional activities which include attending national meetings, the presentation of papers, and the publication Of articles. Attending international meetings and publishing books and monographs are less prevalent. Their attitudes concerning graduate programs show that most believe positive change has occurred in terms Of faculty and library quality, but that student quality and financial resources could be improved. Also, they would like to see more emphasis on research activities. The professors' work schedule is occupied predominantly by teaching graduate students and research or publications. However, they would appreciate less undergraduate-related responsibilities, considerably more time for research, a little more time for extension, and a little less for administration. Important Criteria and Indicators The assessment of ratings and ranking of importance given to the evaluation indicators shows the following: The most frequent mode Of ratings was five, which appears to indicate that the items submitted through the questionnaire were, in general, viewed as very important. Interestingly, the means Of ratings were not very good pre- dictors Of the rankings; i.e., in six out Of the fifteen criteria sets the indicator with the highest average ratings was not ranked first most frequently. 102 However, the means were closer than the median in anticipat- ing the preferred indicator. Agreement on the first-ranked indicators was higher than for the items ranked second and third; i.e., the percentage Of coinciding responses was higher. The indicators most highly rated across the fifteen criteria were: Library: current periodicals (mean: 4.943). Facilities: classrooms and laboratories (mean: 4.871). Library: books and monographs (mean: 4.857). Academic environment: discussion, investigation and expression (mean: 4.839). th—J 5. Facilities: research space and equipment (mean: 4.837). The indicators which individually had the highest agreement Of ranking responses were: 1. Library: current periodicals (78.1%). 2. Academic environment: discussion, investigation and expression (75.3%). 3. Evaluation participants: faculty members (60.3%). Academic credentials: academic activities compatible with program's Objectives (57.0%). 5. Library: books and monographs (52.6%). These two sets Of data seem to indicate the issues which are foremost in the minds of faculty members. As a set, the indicators Of evaluation foci and library achieved the highest means of ratings, 4.534 and 4.486, respectively. These data may be interpreted as criteria which are highly valued by the respondents. Average standard deviations pointed out financial 103 resources as the criterion with highest dispersion Of ratings (1.210) and library with the lowest (.739). These Observations stressed the overall value attributed tO the indicators and the level Of agreement among the responses. It appears that they are a direct reflection of the faculty attitudes, but caution is advised because they may be related to the number and quality Of alternative indicators submitted. CHAPTER V REVIEW AND PERSPECTIVE The faculty responses to the survey questionnaire provided data that can be put together in many different ways and submitted to numerous multivariate analyses. Some Of these possibilities were developed in the preceding pages. This chapter contains a summary Of those Observations in the fourth chapter and an attempt to interpret them using information and insights largely acquired during the visits and interviews. The first section includes a review of the ratings and rank- ing Of indicators, leading to inferences and a commentary on the per- ceived attitudes Of the faculty members. Additional suggestions pre- sented at the end Of the returned questionnaires are also considered. Following is a synthesis Of the findings, organizing recom- mended criteria and indicators for the evaluation Of graduate programs in Brazil. A methodological perspective on planning and evaluation research is proposed in the third section, which ends with an overview Of the investigator's Objectives and approach to this study. Reviewing the Findings This section is subdivided in two parts, which correspond to Part I and Part II Of the questionnaire. 104 105 The Respondents Understanding and interpreting the data on the importance attributed to the criteria and indicators for the evaluation Of Brazilian graduate programs might be better comprehended by a review Of the information collected about the respondents. It is generally acknowledged that good graduate education requires a quality faculty with research competence and basic facili- ties such as adequate libraries, laboratories, and equipment. Inven- tories made by the Coordination for Training Higher Education Per- sonnel (CAPES) and the National Council for Scientific andTechnOlogical Training (CNPq) revealed that the number Of available faculty members with appropriate degrees and experience is far below the desirable level (CAPES, l976c). The studied sample seems to correspond closely to the total Brazilian graduate faculty population in terms Of age, tenure, and academic status. A review Of their academic and scientific activities indi— cates that there are relative differences among the respondents' productivity, based on the number of papers presented and publications. The information collected does not support any specific evaluative comments on faculty productivity, but the policy makers interviewed expressed the need tO develop further incentives and means for increased high-quality creative work. CAPES (1977), Jacob (1978), and other references including very recent newspaper interviews with national experts in scientific administration point out the need tO upgrade the quality Of the faculty as the mainstay Of good graduate programs. 106 The Opinions expressed by the faculty concerning the importance attributed tO research activities are consonant with reflections from the interviewees. Respondents disagreed on the statement that exces- sive support is given to research, in contrast to teaching; policy makers acknowledged the need for increasing support for research. However, it was warned, teaching could also be improved, or receive greater support. In fact, most graduate administrators and faculty interviewed were worried about the lack Of incentives for quality teaching. An example Of institutional action toward this point is the involvement Of universities in the "Institutional Program for Faculty Training" (PIC’). ‘E> Perceptions about change in the quality Of faculty, students, and library and the available financial resources also coincide with statements collected in the field visits. There is concern about the rapid and uncontrolled expansion of undergraduate and graduate pro- grams, mostly at the master's level (CAPES, 1976c, 1977). This development, despiteincreasing financial support which provided for the betterment Of facilities and training Of faculty, led to a relaxa- tion of entrance requirements, high student/professor ratios, and a general symptom of inefficiency. Data on faculty actual and ideal work assignments showed that most respondents would like some modifications in the distribution Of their work load. It was apparent that they would rather concentrate more on academically recognized activities, i.e., research and working with graduate students. These data also indicate coherence with per- ceptions of value given to research and the probability of success in 107 the achievement Of the policy makers' Objectives for upgrading the graduate faculty. The Evaluation Criteria The ratings and rankings given to each of the indicators listed in the questionnaire and the elements from interviews and materials collected during the field work suggest some attitudes and trends which lend themselves to certain judgments and interpretations. Program Opjectives The rating and ranking responses pointed out the high values Of importance for the indicators related to the learning processes most Often associated with quality education: research activities, critical thinking, and knowledge Of subject matter. These are uni- versal values. The distinction awarded to social and economic rele- vance is a reflection Of the needs and expressed Objectives of a developing nation. The indicators most closely related to actualization or competences were less valued, even though highly rated. Professional performance, use of processes and techniques, and decision-making ability were seen in this dimension. This Observation seems to concur with stated concerns about the quality of the graduates produced, specifically on the discrep- ancy between the proposed or formal goals and the intermediate Objec- tives or means. As described in Chapter I, graduate studies in Brazil aim equally at the development Of scientists, professors for the expanding undergraduate population, and the training of high-level 108 specialists. From the few specific mentions and the rating received by the "role Of the area professional," it seems that the latter two Objectives actually are emphasized less. In fact, an examination Of curriculum requirements in several programs showed minimal attention to practicums, pedagogical instruction, or the use Of educational technologies. However, the programs clearly declared the training Of future instructors as a primary objective. Curriculum Organization The Brazilian educational system, in general, has failed to encourage the students to do independent study, develop habits Of intellectual rigor, and to utilize scientific investigation (CAPES, 1977). The proportional ratings attributed to credit for independent studies, elective or Optional courses, transfer Of credits and credit by examination, coupled with the actually rare Offerings on this line, may be interpreted as evidence for the above judgment. However, in broad terms, the flexibility and adequacy to meet individual needs and talents was ranked most important. And the research requirement, which is possibly the most valuable individualized learning experience, was rated highest. This attitudinal pattern elicits a diagnosis of ambivalency or dichotomy between an ideal value (flexibility/individualization) and the means to achieve that (independent studies, etc.). Program Evaluation The issue Of evaluation in Brazilian higher education insti- tutions is still incipient, as are the experiences Of self-study and 109 program reviews. Within the formal realm, accreditation inspections are probably the major evaluative process which most Of the programs have undergone. However, many official sources have voiced interest' about new forms Of quality and adequacy assessments for graduate programs. An evaluation focused on efficiency and congruence seems to be more readily accepted by the faculty than a design to check the external validity Of a graduate program. A possible justification for this rating is the existence Of two basic and conflicting posi- tions, one which sees graduate education and research as geared toward the advancement Of science and the training Of scientists and another which emphasizes a technological approach as adequate for developing countries, i.e., doing applied research and training needed manpower. The respondents pointed out that the program's faculty and its coordinator should be consulted first in any program appraisal. Second, the traditional subject matter specialists, with understand- ing Of national characteristics, should be listened to. In addition, alumni, students, other university personnel, clients, and drop-outs should be involved. It seems that in the hierarchy Of importance attributed to the potential contributors Of program evaluation, decision-making and implementation sources in curriculum modifications should be at the top. The selection of alumni over current students may be related to the status and perspective conferred to the first or may be related to a faculty protective behavior, because enrolled students are more closely involved. The low importance given tO former 110 students who abandoned is interesting; the indicator received the lowest overall rating mean. Indeed, drop-outs could be a valuable source of information on the strengths and weaknesses of a program. Faculty Quality There is a striking coincidence among the most-valued aca- demic credentials and the attributes actually exhibited by the Brazilian faculty, as indicated in the "respondents' characteristics" Of this study or in general statements of Observation. The profes- sors are stronger in terms Of professional and academic experience than in earned advanced degrees. Also, the mobility of Brazilian scientists is much lower than their European or American counterparts, bringing excessive "inbreeding." TO diminish the effects Of this situation, increasing continuing education and post-doctoral fellow- ships have been made available by the Brazilian Government. A similar pattern is evident in the choice Of academic activi- ties. The respondents had indicated before their desire tO devote more time to research and publications and to work with graduate stu- dents. In fact, they valued these indicators the most. Activities which Offer fewer Opportunities for recognition, such as curriculum planning and improvement of teaching ability or extension and profes- sional service, were less important. Among the indicators Of faculty participation in academic decisions, the respondents declared the recruitment and the selection Of students and faculty most important. These are decisions which directly permit a quality control on the program's inputs. Following, 111 indicators which measure the involvement of faculty in the educa- tional and organizational process were rated. It should be noted that the professors ranked first participation in policy and administrative decisions. Peer evaluation, necessary for faculty promotion, was rated lowest--an Observation dissonant from the highest-rated indi- cators. Student Quality Professors responding to the questionnaire indirectly expressed dissatisfaction with the quality Of students currently enrolled, saying that the present students are inferior to the preceding student groups. This is an attitude also Observed during the interviews with adminis- trators. The importance assigned tO this problem suggests the prO- fessors' willingness tO seek alternative criteria in the recruitment Of most-able students. Entrance examinations and personal interviews (which were ranked highest) are not used frequently. In fact, many programs have little means of control over student selection; there are many scholarships Offered to the prospective students who teach in small colleges and the employer's recommendation (the least-valued indicator) is a powerful factor in the admission process. Research and technologies in the areas of admissions, school- records evaluation, and testing are not well developed in Brazil; this poses additional complications in the use Of those traditional indi- cators Of students' quality. 112 The ratings given to indicators Of student activities show an awareness Of the means to attract good candidates. Scholarships and adequate facilities are, likewise, among the priorities suggested by CAPES (1977) tO upgrade the quality Of graduate students. Indicators related tO students' interactions in the insti— tution, which also may be seen as a factor toward better education because they facilitate the relationships between students and the university, rated lower. The lowest recommendation was given tO the students' participation in curriculum planning and evaluation, sug- gesting that the respondents have low regard for the contributions Of the students or have a paternalistic attitude which gives the students a limited bill Of rights and responsibilities but. want limited further involvement and changes Of student roles. Program Facilities The highest-ranked facilities indicators relate directly to teaching and research. Following, importance was assigned to the facilities of private and semi-private use (Offices, space for indi- vidualized study). Significantly high ratings were given to the library criterion as a set and to each Of its indicators. The faculty members were con- cerned with both the library holdings and with its efficient Operation. It should be mentioned that libraries and facilities have also received special support from specific governmental and institutional programs. Among the special facilities indicators, the ones related to research and teaching were considered most important, consonant with already-commented-on attitudes. 113 Program Organization and Evaluation Among the indicators Of program organization, those desig- nated as most important were associated with the role faculty members play in the program and university. Less emphasis was given to the more formal or structural aspects, such as the relationships with superior administration and the communications on decisions. This pattern and the importance attributed to indicators Of a healthy academic environment suggest that Brazilian faculty members share in the international postulates of university: academic free- dom, autonomy, and the collective decision-making process (Ben-David, 1977; Ross, 1976; Brubacher, 1977). It is interesting to point out that the least-valued indicator is the involvement with other community agencies, which is consistent with the low priority given to extension activities. Financial Resources The indicator chosen as most important for financial resources allocation criterion was the proportionality Of the subject and rele- vance for social and economic development. It indicates a concern for adequacy and is consonant with the value attributed tO graduate educa- tion Objectives and evaluation foci. Second was the importance Of internal interests, something already hinted by the ratings given to indicators of evaluation par- ticipants. Indicators which require comparisons between graduate programs within or outside the university were less valued. 114 The analysis presented in the fourth chapter and the commen- tary herein provided are based only on a first and broad review Of the data gathered. Further uses Of the information are on demand, but beyond the scope of this dissertation. Integrative Summary In answering the basic research question: What are the criteria considered appropriate to evaluate graduate programs in Brazil? the faculty responding to the questionnaire rated highest some criteria and indicators. The table on the following page groups the criteria in six major content categories, according to the average ratings attributed. The highest rated indicators of each category are listed with the mean Of ratings, also quoted in Chapter IV. The ratings provided for all indicators presented in the survey instrument support the following Observations: a. there is a high degree Of homogeneity among the faculty, as measured by the dispersion Of their ratings; and b. there is an oaverall agreement on the value Of the indi- cators submitted to the faculty appreciation, as indi- cated by the most frequent mode of ratings (five). The policy makers interviewed stressed the importance and interest the issue and technologies Of evaluation is receiving at this moment, in Brazil. Some institutional formal programs to develop capacity for self-study are supported, most at the initial stage Of gathering basic management information. The policy makers emphasized 115 Summary List Of Criteria and Indicators for Evaluating Brazilian Graduate Programs Criterion Indicator Mean Facilities 1. Library collection of periodicals 4.943 2. Classrooms, meeting rooms and laboratories 4.871 3. Library books and monograph holdings 4.857 4. Space and equipment for research 4.837 Academic 1. Environment which permits discussion, Organization investigation and expression 4.839 2. Representation Of program's faculty and students in university governance 4.368 3. Participation of all faculty members in the several programs' activities 4.321 4. Availability Of channels for receiving and appraising opinions and grievances from students and faculty 4.199 Faculty 1. Professional, teaching and research activities compatible with program's Objectives 4.814 2. Research production and other activities which foster the knowledge development 4.777 3. Thesis and/or dissertation guidance activities 4.612 4. Teaching graduate courses and seminars 4.514 Program 1. Attainment of program's Objectives 4.647 2. Program's evaluation by the faculty 4.617 3. Critical thinking as an Objective 4.547 4. Philosophy and Objectives according to societal needs 4.421 Students 1. Opportunities tO earn merit scholarships 4.553 2. Availability of space and material resources for studying 4.550 3. Academic advising and professional guidance 4.393 4. Curriculum, admission, academic progress and drop-out norms 4.388 Finances 1. Budget proportional to the importance of the subject area for the country's social and economic needs 4.263 2. Budget allocations according tO teach- ing, research, and extension faculty loads 4.180 116 their compromise toward improving faculty qualification and the attainment Of internationally acceptable graduate education standards. The values which the faculty members support are not unique to the Brazilian environment. The general literature reviewed in Chapter 11 allows for this inference. What is probably distinctive is the overall hierarchy suggested by the ratings. Library holdings, facilities, and academic freedom were rated higher than the creden- tials of faculty as researchers and teachers, or the student's attain- ment, placed highest in the work of Berelson (1960), Cartter (1966), and Crane (1965), among others. A Methodological Perspective This study was developed to provide useful information to educational planners, policy makers, administrators, and evaluators involved in Brazilian higher education. It was focused on the pro- fessors' attitudes concerning graduate programs. The questionnaires' rating and ranking responses were inter- preted as a result Of the interchanges between the individuals and their environment. An underlying assumption was that the response meanings are bound to context and biography, although not many statis- tically significant relationships between the first-ranked indicators and the faculty in different categories could be established. The findings should stimulate further research because this was an initial survey tO serve the search for an adequate approach to describing, interpreting, and deciding on Brazilian graduate education. 117 Therefore, it is suggested that additional investigations concentrate on the more specific and in—depth analysis and inter- pretation Of the policy-making processes, i.e., on the study Of social facts or organizational and academic variables in their relationship with aspects of the educational system. A starting point of a planned research program, following this dissertation, may include: a. analysis of relationships between the independent vari- ables and the other indicators not yet cross-tabulated; b. analysis Of relationships among the 109 dependent variables (indicators); and c. analysis of findings according to university and program subject area. These sets of analyses would use only the information already col- lected through this survey. Additional studies could focus on: d. variances Of emphasis on the indicators appropriate for each different subject area or graduate level (master's or doctoral); e. replication Of this survey with students, alumni, and other clients as respondents; f. development of instruments and research procedures tO collect data on the appropriate indicators; and g. collection Of data, following the validation Of indi- cators by the particular faculty, to support assessment and evaluation Of selected program. 118 These studies would use the results discussed in this dissertation and approximate the Operational and decision-making levels. Also needed are "qualitative studies" Of the criteria not only preferred but suitable for evaluating Brazilian graduate pro- grams. As examples, are suggested: h. assessment Of means and ends Of graduate programs, emphasizing the goals, objectives, and Operational tar- gets as paired tO the respective policies, incentives, and administrative decisions or monitors used; i. assessment of outcomes and effects expected from both the program's Objectives and means; and j. assessment of changes in the faculty's (and other groups involved in evaluation research) attitudes during and after exposure to evaluation themes and alternative indi- cators. These studies Stem from the need for in-depth approaches and because the relationship between Objectives, means, and outcomes is, at least, theoretically clear. The suggestions for further research, presented in this sec- tion, should be considered as an invitation to inquiry rather than a follow-up Of conclusions. The development Of new and better ways to evaluate requires breadth and depth studies, carried through longi- tudinal and occasional efforts, and involving many investigators or an interdisciplinary team. Critical aspects to be analyzed and taken into account are the relationships between a program's faculty and its students or 119 clients, the program's organizational structure and reward system, and the relationships between the program's faculty and students and the social-economic environment. The investigator remains interested in further exploring the relationships between the faculty's international connections and the values or indicators Sponsored, although these first cross-tabulations have not shown any significant differences due to regional foreign affiliations. At this time, when there is a need for more precise and sophis- ticated planning, management, and evaluation Of graduate education, the research methods and analysis procedures (including statistics) seem Of great importance. Thus, the tools of research and, more impor- tantly, the conception underlying research, become part Of the profes- sional armament Of planners and evaluators. Distinguishing between findings that are statistically significant and those that are educa- tionally significant is their task. The findings analyzed in this dissertation were intentionally limited by scope and design; however, they should provide both researchers and decision makers stimuli for searching appropriate means of evaluating and bringing about the development Of quality graduate education in Brazil. The methodological perspective exposed in the dissertation and further described in this section is based on two primary con- cerns: (a) heuristic in the sense of making subjects and others aware Of the need for creating means that attempt to comprehensively gather and analyze the Opinions about a broad range Of items Of 120 persons affecting and affected by the educational program(s) on target; and (b) exploratory in the sense of demonstrating a possible mode Of doing research that conforms tO a philosophical-professional position and hOping to encourage others along this line. APPENDICES 121 APPENDIX A QUESTIONNAIRE LETTER TO COORDINATORS LETTER TO THE SUBJECTS Note: A translation Of each question and indicators, listed in Part II Of the questionnaire, is presented in Chapter IV. 122 APPENDIX A QUESTIONNAIRE LETTER TO COORDINATORS LETTER TO THE SUBJECTS Para uso do pesquisador Cddigo da area: InformacOes desejadas pelo pesquisador: I. Dados pessoais do respondente II. Valores atribuidos a indicadores de qualidade de programas de pés-graduacao Por favor, responda a este instrumento: Devolver para: Profa. Maria Beatriz Luce UFRGS - Faculdade de Educacao Av: Paulo Gama, s/n? 79 andar 90.000 - Porto Alegre, RS. 123 124 Dados Biograficos e Académicos Assinale o namero apropriado ou responda sobre a linha. Posigao Academica Professor Titular Professor Adjunto Professor Assistente Aux. de Ens. ou Instrutor de-DU'I Outro (especifique) Ano e instituigao onde Obteve diplomas: Ano Instituigao Graduacao Mestrado Doutorado Outro: Encontros profissionais em 1977 e 1978. Encontros nacionais: Encontros internacionais: Apresentou nele(s) algum trabalho: Sim 1 N50 2 NOmerO de publicacOes em 1977 e 1978. Artigos: Livros: Monografias: 125 Se vocé jé era um membro deste corpo docente em 1976, faca um circulo no nfimero que corresponde de melhor modo a sua apre- ciacao das mudangas ocorridas em sua area, dentro de sua ins- tituigfio, com replacfio a: . Pouce ou MuitO . Sem Melhor nenhuma Pior . .~ Melhor mudanga Opiniao . qualidade dos docentes 4 3 2 l 0 . recursos e materiais da biblioteca 4 3 2 1 O . recursos financeiros par para 0 programa de p65- 4 3 2 l 0 graduacao . qualidade dos alunos 4 3 2 l 0 Como vocé reage a seguinte afirmacfio? Atividades de pesquisa tem recebido demasiado valor, em contraste com ensino, nas universidades brasileiras. Concordo plenamente 4 Concordo com reservas 3 Discordo com reservas 2 Discordo plenamente 1 0 N50 tenho Opiniao Atividades de pesquisa tem recebido demasiado valor, em contraste com ensino, no programa de p65-graduacfio que leciono. Concordo plenamente 4 Concordo com reservas 3 Discordo com reservas 2 Discordo plenamente l 0 N50 tenho Opiniao 126 Qual O percentual de seu horério de trabalho dedicado a: Ensino e alunos de graduagSO Ensino e alunos de pés-graduacao Pesquisa e publicagOes Servicos a comunidade Administracfio academica Atividades profissionais fora da instituicao Atual 100 100 127 II. Valores atribuidos a indicadores de qualidade de programas de pOs-graduagao. Qual a importancia que vocé atribui aos indicadores abaixo relacionados, se tiver de avaliar um programa de pOs-graduacao brasileiro em sua area de especializagao? Por favor, considere todos os itens que apresentamos, fazendO um circulo n0 valor escolhido. ApOs, selecione os mais impor- tantes e, se desejar, acrescente sugestOes. 128 05 Objetivos d0 programa de p65-graduac30 IMPORTANCIA devem enfatizar: MA MI XI NI MA MA a. a contribuicSO da area de conhecimento para 0 desenvolvimento social e econ6mico 5 4 3 2 l b. O papel desempenhado pelo profissional da area 5 4 3 2 1 c. a importancia das atividades de pesquisa 5 4 3 2 l d. O dominio dO conteGdO da matéria 5 4 3 2 1 e. a habilidade de pensar criticamente 5 4 3 2 l f. a habilidade de usar processos e técnicas associados a competéncia profissional 5 4 3 2 l g. a habilidade de tomar decisOes 5 4 3 2 1 Por ordem de importancia, voce classificaria: l9 lugar + 1:] 29 lugar + 1:] (coloque a letra correspondente) 39 lugar + D SugestOes: O curriculo do programa de p65-graduacao deve incluir: a. experiéncias de aprendizagem que sejam suficiente- mente flexiveis para permitir aos alunos um desen- volvimento de acordo com as necessidades e talentos individuais 5 4 3 2 b. crédito para estudos independentes ou individualizados 5 4 3 2 c. disciplinas eletivas ou Optativas dentro da area de concentrach 5 4 3 2 d. disciplinas eletivas ou Optativas noutras areas 5 4 3 2 e. um conjunto de disciplinas Obrigatérias 5 4 3 2 f. Oportunidades para a tranferéncia de alguns créditos de mesmo nivel Obtidos noutras instituicfies 5 4 3 2 g. Oportunidades para a dispensa de pré-requisitos mediante exames 5 4 3 2 u—J—l 129 IMPORTANCIA MA MI XI NI MA MA h. metodologia de instrucao adequada aos Objectivos e conteados 5 4 3 2 l i. availiacOes periOdicas do progresso academico dos alunos 5 4 3 2 1 j. Obrigatoriedade da frequencia em horario integral durante, pelo menos, parte do programa 5 4 3 2 l k. exigéncia de experiencia em trabalho de pesquisa durante O programa 5 4 3 2 1 Por ordem de importancia, voce classificaria: l9 lugar+ [:1 29 lugar + 1:] (coloque a letra correspondente) 30 lugar + Cl SugestOes: Uma avaliacao sistematica d0 curriculo deve considerar: a. se a filosofia e Objetivos dO programa respondem a necessidades da sociedade 5 4 3 2 l b. 5e 05 Objetivos d0 programa estao sendO atingidos 5 4 3 2 1 Da avaliacao de um programa de pés-graduacfio, devem participar: a. docentes do programa 5 4 3 2 1 b. cordenador (administrador) do programa 5 4 3 2 l c. outros administradores universitarios aos quais o programa esta ligado 5 4 3 2 l d. especialistas da area de conhecimento mas pertencentes -a outras instituicOes 5 4 3 2 l 130 IMPORTANCIA MA MI XI NI MA MA e. especialistas estrangeiros 5 4 3 2 1 f. alunos presentemente matriculados n0 programa 5 4 3 2 l g. egressos titulados pelo programa 5 4 3 2 l h. evadidos temporarios ou permanentes 5 4 3 2 l i. clientes do programa, comO: empregadores de formados 0u consumidores de pesquisas 5 4 3 2 1 Por ordem de importancia, vocé classificaria: 19 lugar + [:3 29 lugar + [Z] (coloque a letra correspondente) 3? lugar + [j SugestOes: A qualificacao academica e profissional do corpo docente deve ser avaliada por: a. diplomas de pés-graduacao nas areas que especifica- mente lecionam 5 4 3 2 l b. ativadades profissionais, de ensino e de pesquisa compativeis com 05 Objetivos do programa 5 4 3 2 l c. grau de diversificacfio das instituicOes onde os docentes realizaram sua formacao 5 4 3 2 l d. distincao recebidas pelos docentes em suas areas de atuacao 5 4 3 2 l e. participacao dos docentes em atividades que 1hes pro- porcionem atualizacao e crescimento profissional 5 4 3 2 1 Por ordem de importancia, vocé classificaria: 19 lugar-+ E] 20 lugar + [:1 (coloque a letra correspondente) 131 SugestOes: IMPORTANCIA MA MI x1 NI MA MA Ativadades dos docentes que devem ser valorizadas: ‘thOU'Q’ SugestOes: . ensino de graduacao . ensino de disciplinas e/ou seminarios de pés-graduacao . aconselhamento académico de alunos em p65-gradua950 . orientacao de dissertacOes e/ou teses dos alunos . planejamento e evaliagao do curriculo . Oferecimento de Oportunidades para alunos avaliarem 0 0101010101 bhh-D-b wwwww NNNNN —l—l—l—l-—l ensino por eles ministrado 5 4 3 2 l . ativadades de pesquisa ou outras que auxiliem O avanco do conhecimento em sua area 5 4 3 2 l . publicacao de livros e monografias em sua especialidade 5 4 3 2 1 . publicacfio de artigos e resenhas bibliograficas em periOdicos nacionais 4 3 l . publicacao de artigos em periOdicos internacionais 4 3 2 l . participacao em servicos a comunidade (extensao) 4 3 2 l . apresentagao de trabalhos em reuniOes profissionais (congressos, seminarios, etc.) 5 4 3 2 l . associacao em entidades profissionais 4 3 2 1 Por ordem de importfincia, vocé classificaria: 1? lugar-+ [] 20 lugar + [:] (coloque a letra correspondente) 132 IMPORTANCIA MA MI 05 docentes devem participar de: XI NI MA MA a. recrutamento, selecao e promocao de docentes 5 4 3 2 l b. avaliacao da produtivadade de seus colegas docentes 5 4 3 2 l c. recrutamento e selecfio dos alunos 5 4 3 2 l d. formulacao de politicas e administracao da instituicao 5 4 3 2 l e. distribuigao de encargos dos docentes 5 4 3 2 l f. alocacao de recursos financeiros do programa 5 4 3 2 1 Por ordem de importancia, vocé classificaria: 19 lugar-+ [] 29 lugar-+ [:3 (coloque a letra correspondente) 39 lugar-+ [j SugestOes: A selecao dos alunos deve ser baseada em: a. registros escolares anteriores 5 4 3 2 l b. recomendacOes de professores anteriores 5 4 3 2 1 c. recomendacéo de seu empregador atual ou anterior 5 4 3 2 l d. experiéncia profissional anterior 5 4 3 2 l e. apresentacao de exposicao de Objetivos profissionais e académicos 5 4 3 2 1 f. exames de selecao especificos para a area de conhecimento 5 4 3 2 l g. exames de habilidade intelectual padronizados 5 4 3 2 l h. recomendacoes de docentes do programa, apOs entrevista 5 4 3 2 l 133 IMPORTANCIA MA MI XI NI MA MA Por ordem de importancia, vocé classificaria: l9 lugar + [:1 29 lugar-+ [:1 (coloque a letra correspondente) 3? lugar + D SugestOes: Os estudantes devem ter acesso a: a. publicacao sobre direitos e responsabilidades dos alunos junto a universidade 5 l b. normas curriculares, de admissao, progresso académico, retencfio, desligamento e conclusfio do curso l c. representacfio no processo de decisOes académicas l d. participacao no planejamento e valaiacao de curriculo l e. Oportunidades para participacfio remunerada em atividades de pesquisa e administracao auxiliar 5 l f. Oportunidades para Obtencfio de bolsas de estudo por mérito 1 g. aconselhamento académico e orientacéo profissional 1 h. recursos materiais e espaco para estudar 1 Por ordem de importancia, vocé classificaria: l? lugar + [:] 29 lugar + [:] (coloque a letra correspondente) SugestOes: 3? lugar-+ [:j 134 IMPORTANCIA MA MI 0 programa de pés-graduacao deve dispor de: XI NI MA MA a. gabinetes privativos ou semi-privativos para professores e técnicos 5 4 3 2 l b. gabinetes para coordenadores 5 4 3 2 l c. ambiente para pessoal de apoiO 5 4 3 2 1 d. salas de aula, reuniOes e laboratOrios adequados 5 4 3 2 l e. espaco e equipamento para pesquisa 5 4 3 2 l f. espaco e equipamento didatico 5 4 3 2 1 g. espaco para ativades recreativas e estra-curriculares 5 4 3 2 1 Por ordem de importSncia, vocé classificaria: 19 lugar +-[:] 2? lugar-+ [:1 39 lugar + [_] SugestOes: (coloque a letra correspondente) A biblioteca deve ter: a. colec§0 de periOdicos na area de conhecimento espec1fica e atualizada 5 b. libros e monografia nos assuntos especificos e correlatos 5 c. materiais de referéncia comO: abstracts, indices e bibliografias . catélogos coletivos . servicos de comutacéo bibliografica pessoal técnico para assisténcia a0 leitor areas para trabalho individual ou em pequenO grupo . planO de aquisicOes e orcamento planejado 340-th 010101010101 h-b-b-h-b-b wwwwww NNNNNN —-l-—.l.—l—-l-—l—l 135 IMPORTANCIA MA MI XI NI MA MA Por ordem de importfincia vocé classificaria: l9 1ugar-+ [j 29 lugar-+ [:] (coloque a letra correspondente) O 3. IUQOY‘ + [:l SugestOes: Outros recursos que 0 programa pOs-graduacao deve ter: a. servicos ou locais para uso e treinamento de profes- sores e alunos (ex: hospital, escola, etc.) creden- ciados por instituigOes competentes 5 4 3 2 l b. recursos clinicos ou instrumentais adequados a0 nOmerO de alunos 5 4 3 2 1 c. servicos de datilografia e apOiO administrativo 5 4 3 2 l d. recursos didaticos 5 4 3 2 l e. recursos fisicos e humanos para pesquisa 5 4 3 2 l f. instituicOes conveniadas e credenciadas para atividades externas 5 4 3 2 1 g. servico de processamento de dados para uso dos professores 4 3 2 l h. servico de processamento de dados para uso dos alunos 4 3 1 Por ordem de importfincia, vocé classificaria: 1C.) lugar -> [:1 29 lugar + [Z] (coloque a letra correspondente) 3C.) lugar + [:1 SugestOes: 136 IMPORTANCIA MA MI XI NI 0 programa deve se caracterizar por: MA MA a. filosofia e Objetivos compativeis com 05 da universidade 5 4 3 2 l b. organizacéo de acordo com estrutura e funcionamento da instituicao 5 4 3 2 l c. relacionamento funcional com a administracéo central da universidade d. relacionamento com unidades academicas administra- tivas e de apOiO 5 4 3 2 l e. representagfio decente e discente nos Orgaos colegiados colegiados da instituicao 5 4 3 2 l f. politicas e normas de organizacao, direitos e responsabilidades dos docentes 5 4 3 2 l- g. participacao de todos os docentes nas varias atividades, de acordo com sua posicao e responsabilidades 5 4 3 2 1 h. registros das decisOes do corpo docente a adminis- tracao atualizadas e disponiveis 5 4 3 2 1 Por ordem de importfincia, voce classificaria: l9 lugar + [:1 29 lugar + [:1 (coloque a letra correspondente) 0 3. lugar-+ [] SugestOes: Os recursos financeiros d0 programa devem ser: a. compativeis e proporcionais aos recursos da universidade 5 4 3 2 l b. compativeis e proporcionais aos recursos disponiveis para programas similares n0 pais 5 4 3 2 l 137 IMPORTANCIA MA MI XI NI c. distribuidos proporcionalmente entre ensino, MA MA pesquisa e extensao, de acordo com a dedicacao dos docentes a estas areas 5 4 3 2 l d. proporcionais a importancia da area de ensino e ( pesquisa n0 desenvolvimento sOciO-econOmico do pais 5 4 3 2 1 Por ordem de importancia, vocé classificaria: l9 lugar-+ [] 29 lugar + [:j (coloque a letra correspondente) 39 lugar -> 1:] SugestOes: Para estimular a produgao clientifica e académica O programa de pés-graduacao deve: a. permitir a alunos e professores a discussao, investigacfio e expressao 5 4 3 2 l b. informar quais os critérios usados para avaliacao d0 desempenho de alunos e professores 5 4 3 2 l c. Oferecer canais para recepgao e consideracao de OpiniOes e reclamacfies de professores e alunos 5 4 3 2 l d. envolver outras agéncias da comunidade para . participarem do processo educacional 5 4 3 2 1 e. solicitar a participacfio de alunos e professores na avaliacfio periOdica da organizacéo e administracao do programa 5 4 3 2 1 Por ordem de importancia, vocé classificaria: 19 lugar + I: 2? lugar +4 [:1 (coloque a letra correspondente) 39 lugar + [:1 138 Sugestoes: Comentarios ou sugestoes adicionais $50 apreciados: 139 # MINISTERIO DA EDUCAcAo E CULTURA vmvsnsmaua FEDERAL DO 1110 GRANDB no sun. FACULDAD: DB RDUCACAO MS. I! NS-Elllllul [I [WI Ilmo Sr. Prof. M.D. Universidade de Prezado Professor: A criagéo e a expansio de vasto nfimero de cursos de poo-gradug 950 em nosso pais, durante os filtimos anos, originam a necessidade de tecnologias que facilitem o planejamento, a administragao e a avg liagao de tais programas. Elemento fundamental para o desenvolvimen- to de modelos e técnicas que auxiliem tais atividades é a identificg 950 de critérios e indicadores considerados apropriados para avaliar cursos de pos-graduagao. Visando a reunir tais informagoes estou realizando uma pesqui- sa com amostra de ambito nacional. O estudo sera apresentado 5 Michigan State University como tese para obtengfio do grau de Ph.D. e 05 resultados estarao também 5 disposiqao de interessados no tema. Sua participagao, através de resposta ao questionirio em ane - xo, é de extrema importancia considerando-se o nfimero de sujeitos sg lecionados dentre os administradores de pSs-graduagio. Outrossim , apreciaria sua colaboragao para solicitar aos professores e alunos , cujos nomes estao abaixo relacionados, que também colaborem. Por favor, retorne o instrumento antes ou dia 33 g; setembro usando o envelope previamente enderegado e selado que anexo. Antecipadamente, agradego sua atengao. Maria Beatriz M. Luce Professor-Assistente 140 mmsrtmo DA touucto t CULTURA vmvsasxmos FEDERAL no mo camps DO sun. vacuum: m: noucacxo III” I! '58-le [I [DWI Ilmo. Sr. Prof. Prezado Professor: 0 estabelecimento e a expansao de cursos de pés-graduagao em nosso pais, durante a presente década, originam a necessidade de desenvolver tecnologias que facilitem o planejamento, administra - 930 e avaliagao de tais programas. Com 0 prop53ito de identificar quais os critérios e indicadg res considerados apropriados para a avaliagao de cursos de p5s-grg duagao, solicito a sua participagéo através do preenchimento do questionério em anexo. Esta pesquisa 5 parte fundamental da tese que pretendo apre- sentar na Michigan State University visando a completar o programa de doutorado. As informagfies que conto reunir deverio ser potenci- almente fiteis para o desenvolvimento de modelos e técnicas de ava- liagao e estarao 5 disposigao de interessados. Sua participagao E de extrema importancia, devido a sua posi 950 e nfimero de sujeitos visados nesta investigagéo. Por favor, d3 volva o questionario antes an die 32 gg setembro usando o enve10pe previamente enderegado e selado que anexo. Antecipadamente, agradego sua atengfio. Maria Beatriz M. Luce Professor-Assistente. APPENDIX B CODING PLAN 14] APPENDIX B CODING PLAN l--University 2- 4- 3--Program 5--Faculty number 6--Academic rank 7- l4- 17- 20- l3--Academic degrees 16--Professional meetings l9--Academic production 23--Change question 24--Research in university 25--Research in area 26- 38- 45- 48- 59- 65- 67- 76- 79- 37--Schedule 44--Criterion 47--Criterion 58--Criterion 64--Criterion 66--Criterion 75--Criterion 78--Criterion 83--Criterion 84- 85—-Criterion 86- 98--Criterion 99-l02--Criterion l03-108--Criterion l09-lll--Criterion ll2-ll9--Criterion l20-l22--Criterion 123-l30-~Criterion l31-l33--Criterion l34—l40--Criterion l4l-l43--Criterion l44-lSl--Criterion 152-l54--Criterion l55—162--Criterion l63-l65--Criterion lG6-l73--Criterion l74-l76--Criterion l77-lBO--Criterion lBl-l83--Criterion 184-188--Criterion l89-l9l--Criterion one ratings one rankings two ratings two rankings three ratings four ratings four rankings five ratings five rankings six ratings six rankings seven ratings seven rankings eight ratings eight rankings nine ratings nine rankings ten ratings ten rankings eleven ratings eleven rankings twelve ratings twelve rankings thirteen ratings thirteen rankings fourteen ratings fourteen rankings fifteen ratings fifteen rankings I42 APPENDIX C QUESTIONNAIRES DELIVERED AND RETURNED 143 «Db-b-h-b-bbb wwwwwwww NNNNNNNN -J-'-l-‘—-'-‘-'-' OOOOOOOO APPENDIX C QUESTIONNAIRES DELIVERED AND RETURNED Delivered Returned 7 0 20 7 (coordinator did not receive) (cooperation declined) 10 —ul 000090 HOWU'IUWNN—l aooommonm OOOCDOUTOO \IOOOOOOO (distant campus) d ._a 0) —-l .5 lmmwoo 455 -—l h 0‘ —l b .b APPENDIX D TABLES USED FOR QUESTIONS IN PART I OF THE QUESTIONNAIRE 145 APPENDIX D TABLES USED FOR QUESTIONS IN PART I OF THE QUESTIONNAIRE 1. Academic Rank . . Absolute Relative P°S1t1°n Frequency Frequency Other 14 9.6 Instructor Assistant professor 50 34.2 Associate professor 52 35.6 Full professor 27 18.5 Missing 3 2.1 Total 146 100.0 146 147 o.oop map o.oo_ me— o.oop map Punch m.~m Ne m.e¢ mo m.o op xcmpm w.e n ¢.m m m.mp mp cmnuo I; 8 mg : 9m 2 883 o.~m mm m._~ mm m.¢ N wpm~mm manpoma< m>wumpwm mpspomn< m>mumpmm muapomn< cowumoog mwcmmo Pagouuoo mwcmmo m.cwummz mmcmmo m.co~m;umm cwcgmm mam: mmmcmmo sows: pm cowpsuwumcH .HHH o.oo_ may o.oo_ we, o.oo_ may pmuo» ~.~m Ne m.~¢ No N.@ m gem—m —.N m n. P m.op mp mgommnlmemp e._ N _.~ m m.mm em mmmpuommp m.~. om ~.mm Fe m.om en mmmpuommp ~.om mm o.¢~ mm m.w mp «mmP1onF m.m~ mm N.N e n. F gmuemumump zucmacwcd aucmzcmca zucmacmcu aucmacmed mucmsomcd zucmacmcu m>wumpmm mu:_omn< m>wpm_mm mu=FOmn< m>wum_mm muzpomn< cmm> wwsmwo pmcoyuos . wmcmmo m.gmummz mmcmmo m.copm;umm 1'1 1".IIC 'Iln‘ll 11.! 'I’v ’. Ii'l'. 1,1...IrIII A, l 1 1 I 1‘ 1.1". v.Iil1|T..1l..l1ll. '1'..- n 1 -v I, I ' Ill‘l‘l- 1“ umccmm mmmcmmo we me> .HH 148 IV. Number of Professional Meetings Attended (1977-1978) Number of Frequency Frequency of Meetings Nationally Internationally 1 45 50 2 34 21 3 17 l 4 9 1 5 5 6 1 7 l 8 l 9 or more 1 .. Blank 32 73 Total 146 146 V. Number of Publications. PUUT122tggns PEETTLAXng Pfiglglfiing Pfiglglfiing Articles Books Monographs l 22 20 23 2 27 4 10 3 16 5 3 4 14 l 5 ll 1 6 1 7 8 9 or more .. .. Blank 32 117 107 Total 146 146 146 149 VI. Perception of Change (frequencies) Faculty Library Financial Student Quality Quality Resources Quality Much better l4 16 12 4 Better 57 59 36 29 Little or no change 47 44 51 69 Horse 4 8 20 25 No opinion 7 2 8 2 Blank l7 l7 l9 17 Total 146 146 146 146 VII. Value of Research: "Research activities have received excessive value, in contrast to teaching. . ." (frequencies) In Brazil In Program Strongly agree 5 5 Agree 37 33 Disagree 48 45 Strongly disagree 40 55 No opinion 13 4 B1ank 3 4 Total 146 146 BIBLIOGRAPHY 150 BIBLIOGRAPHY Alciatore, Robert, and Eckert, Ruth E. Minnesota Ph.D.'s Evaluate Their Training: A Study of the Relationship of Various Ph.D. Programs to Later Career Service and Satisfaction. Minneapo- lis: University of Minnesota, 1968. Ashby, Eric. Adapting Universities to a Technological Society. San Francisco: Jossey-Bass, 1974. Bayer, A. E., and Folger, John K. "Some Correlates of a Citation Measure of Productivity in Science." Sociology of Education 39 (1966): 381-90. Ben-David, Joseph. Centers of Learning: Britain, France, Germany, United States: An Essay. New York: McGraw-Hill, 1977. Berelson, Bernard. Graduate Education in the United States. New York: McGraw-Hill, 1960. Blackburn, Robert T., and Lingenfelter, Paul E. Assessing Quality in Doctoral Programs: Criteria and Correlates of Excellence. Ann Arbor: Center for the Study of Higher Education, The Uni- versity of Michigan, 1973. Bowker, Albert. "Quality and Quantity in Higher Education." Journal of the American Statistical Association 60 (1965): 1-15. Brasil. "Definicao dos Cursos de Pés-Graduacao." Documenta 44 (1965): 67-68. "Parecer n9 977/65." Documenta 156 (1973): 472. Plano Nacional de Desenvolvimento I. Brasilia, 1971. Plano Nacional de Desenvolvimento II. Brasilia, 1974. , Conselho Nacional de Pos-Graduacio. Plano Nacional de P65- Graduagao. Brasilia, 1975. Brooks, M. "The Community Action Program as a Settin for Applied Research." Journal of Social Issues 21 (1965): 29-40. 151 152 Brubacher, John Seiler. 0n the Phi1050phy of Higher Education. San Francisco: Jossey-Bass, 1977. CAPES. Situagao da Pés-Graduacao: Brasil-75. Brasilia, 1976a. (Ministry of Education and Culture, Department of University Affairs, Coordination for Training Higher Education Personnel). New Perspectives for the System of Higher Education. Brasilia, 1976b. . Sistemas de Credenciamento dos Cursos de Pas-Graduagao Estados Unidos. Brasilia, 1976c. Os Graus de Mestre e Doutor nas Instituigaes de Ensino Norte-Americanas. Brasilia, 1977. Caro, Francis G. "Issues in the Evaluation of Social Programs." Review of Educational Research 41(2) (1971): 87-114. Cartter, Allan M. An Assessment of anlity in Graduate Education. Washington, D.C.: American Council on Education, 1966. Clark, Kenneth E. America's Psychologists: A Survey of a Growing Profession. Washington, D.C.: American Psychological Asso- ciation, 1956. Cole, Stepehn, and Cole, Jonathan R. "Scientific Output and Recog- nition: A Study in the Operation of the Reward System in Science." American Sociological Review 32 (1967): 377-90. Coombs, C. H. "Theory and Methods of Social Measurement." In Research Methods in the Behavioral Sciences. Edited by L. Festinger and D. Katz. New York: Dryden, 1953. Crane, Diana. "Scientists at Major and Minor Universities: A Study of Productivity and Recognition.‘I American Sociological Review 30 (1965): 699-714. Creager, J. A. The Use of Publication Citations in Educational Research. Washington, D.C.: American Council on Education, 1967. Cronbach, L. J. "Course Improvement Through Evaluation." Teachers' College Record 64 (1963): 672-83. Dressel, Paul L. Handbook of Academic Evaluation. San Francisco: Jossey-Bass, 1976. ; Johnson, F. Craig; and Marcus, Philip M. The Confidence Crisis. San Francisco: Jossey-Bass, 1971. 153 Etzioni, A. "Two Approaches to Organizational Analysis: A Critique End a Suggestion." Administrative Science anrterly 5 (1960): 57-78. Florida State University. A Study of Florida State University Doc- toral Graduates and Their Reactions to the Doctoral Program at the University. Tallahassee: Florida State University, 1957. Greenberg, B. G. "Evaluation of Social Programs." Review of the International Statistical Institute 36 (1968)?'260-77. Harvey, James. The Student in Graduate School. Washington, D.C.: American Association for Higher Education, 1972. Heiss, Ann M. Challenges to Graduate Schools. San Francisco: Jossey-Bass, 1970. Hyman, H., and Wright, C. "Evaluating Social Action Programs." In The Uses of Sociology. Edited by P. Lazarfeld, W. Sewell, and H. Wilensky. New York: Basic Books, 1967. Jacob, Gerhard. "A Pesquisa em Fisica no Brasil durante os Proximos Anos." Boletim Informativo da Sociedad Brasileira de Fisica 9(1) (1978). Jordan, Robert T. "Library Characteristics of Colleges Ranking High in Academic Excellence." College and Research Libraries 24 (1963): 369-76. Katz, Joseph, et al. No Time for Youth. San Francisco: Jossey- Bass, 1968. Kish, Leslie. Survey Sampligg, New York: John Wiley & Sons, 1965. Likert, Rensis. "A Technique for the Measurement of Attitudes." Archives of Psycholggy 140 (1932). Margolis, J. "Citation Indexing and Evaluation of Scientific Papers." Science 155 (1967): 1213-19. McGrath, Earl J. The Quantity and Quality of College Teachers. New York: Teachers College, Columbia University, 1961. Merton, Robert King; Fiske, Marjorie; and Kendall, Patricia L. :flmg Focused Interview: A Manual of Problems and Procedures. Glencoe, 111.: Free Press, 1956. Miller, Richard I. Evaluating Faculty Performance. San Francisco: Jossey-Bass, 1972. 154 Myers, C. R. "Journal Citations and Scientific Eminence in Con- temporary Psychology.“ American Psychologist 25 (1970): 1041-48. O'Muircheataigh, Colm A., and Payne, Clive. The Analysis of Survey Data. New York: Wiley, 1977. Parsons, Talcott, and Platt, Gerald. The American Academic Profes- sion: A Pilot Study. National Science Foundation, Grant GS51 3, W68. The American University. Cambridge, Mass.: Harvard University Press, 1973. Pelz, Donald C., and Andrews, Frank M. Scientists in Organizations. New York: John Wiley, 1966. Perkings, Dexter, and Snell, John L. The Education of Historians in the United States. New York: McGraw-Hill, 1962. Richman, Barry M., and Farmer, Richard N. Leadership, Goals, and Power in Higher Education. San Francisco: Jossey-Bass, 1974. Ross, Murray G. The University: The Anatomy of Academe. New York: McGraw-Hill, 1976. Sanford, Nevitt. The American College. New York: John Wiley and Sons, 1962. Schwab, Joseph J. College Curriculum and Student Protest. Chicago: University of Chicago Press, 1969. Scriben, M. "The Methodology of Evaluation." AERA Monograph Series on Curriculum Evaluation, No. 1. Chicago: Rand McNally, 1967. Selltiz, Claire, et a1. Research Methods in Social Relations. New York: Holt, Rinehart and Winston, 1959. Smith, Richard, and Fiedler, Fred E. "The Measurement of Scholarly Work: A Critical Review of the Literature." Educational Record 52 (1971): 225-32. Sousa, Edson Machado de. "A Pos-Graduacfio na Universidade Brasileira, in Conselho de Reitores das Universidades Brasileiras." Pos- Graduagio (l976). Stake, Robert E. "The Countenance of Educational Evaluation." Teachers College Record 68 (1967): 523-40. 155 Stallings, William M., and Singhal, Sushila. "Some Observations on the Relationship Between Research Productivity and Student Evaluation of Courses and Teaching." American Sociologist 5 (1970): 141-43. Suchman, E. "Evaluating Educational Programs: A Symposium." Ihg_ Urban Review 3(4) (1969): 15-17. . Evaluative Research. New York: Russell Sage Foundation, 1967. Tyler, Ralph W. Principles of Curriculum and Instruction. Chicago: University of Chicago Press, 1949. U.S. Government. The Economics and Financing of Higher Education in the United States. Washington, D.C.: Government Printing Office, 1969. Webster's New Collegiate Dictionary. Springfield, Mass.: G & C Merriam Co., 1976. Wilson, Logan. Academic Man. New York: Oxford University Press, 1942. Worthen, B. R., and Sanders, J. R., eds. Educational Evaluation: Theory and Practice. Worthington, Ohio: Jones, 1973. 689 to 6161 ‘62 aunr 9““1 JOSSQJOId Jofew \4‘014 ‘-\‘ Qflfi flfifiak \‘e ‘ Ki. -\ u u; 99189;) UGLneanpa 'o'ud .10} siu9u191ynb91 913 JO 1u9wng1n} SplBMOl p91doooe U99q seq aon1 PJLBJON ziuieag eguew Aq p9w9391d 'IIZVUB NI NWDOUd nonvonoa Elli/DOW!) 3H1. BlVD'IVAI-l 0.1. VIU3lIUD :10 lNElNSSI-ISSV NV P9119119 SISQIII am 12111 5311190 01 8! Sm "11TII'IIJTIIIIIIIIT