A STUDY QF THE EVALUATNE PRACYICES EN MANAGEMENT EDEJCAHON AND DEVELOPMENT PROGRAMS EN SELECTED UNITED STATES COMPANIES Thesis gar Hm Dogma of Ed. D. MICHIGAN STAN“: UNIVERSITY Carl I. Shafer 1961 This is to certifg that the h dd A Study Of Tl'neeflgvgluaft ve Practices In Management Education and Development Programs In Selected United States Companies presented by Carl l. Shafer has been accepted towards fulfillment of the requirements for Ed.D. degree in Education Major professor Date August 10, 1961 0-169 LIBRARY Michigan State University 4.--vr—... .—.-_~ the I andc' ABSTRACT A STUDY OF THE EVALUATIVE PRACTICES IN MANAGEMENT EDUCATION AND DEVELOPMENT PROGRAMS IN SELECTED UNITED STATES COMPANIES by Carl I. Shafer The purpose was to study the theoretical aspects and the practices of evaluation of formal management education and development programs in industry. Procedures The theoretical aspects of educational evaluation were Synthesized from the literature of education, psychol- ogy and business. Major published evaluative research was reviewed and analyzed. A survey of evaluative practices was performed by sending a detailed questionnaire to 158 large companies. One hundred eighteen or 74.6 per cent of the questionnaires were returned which revealed the cur- rent practices of evaluation of management training pro- grams in the selected companies. Major Findings 1. Slightly over half of the training departments of the companies represented in the survey spend 1-5 per cent of their time on evaluation of their management training programs. About one-fifth spend less than one per cent 2. 5. 4. 5. Carl I. Shafer and one-fifth spend 6-10 per cent of their time on evaluation. Rarely does a training department spend more than ten per cent of its time on evaluation. Nearly all companies that responded spend five per cent or less of their training budget on evaluation. Indications were that a static position will be main- tained and perhaps even a slight increase in evaluative effort will be made in the future. Many companies that reported the higher amounts of time and budget for evaluation were also those which planned to increase their evaluation activities in the future. Evaluation effort, in terms of time and budget, ap- peared to be greater in companies where tap management stresses the evaluation of management training pro- grams as compared to those where top management shows little or no interest in evaluation of management training. There was a tendency for evaluation effort, in terms of time and budget, to be greater in companies with the larger management training staffs (over five persons) and where the training department has been in operation longer (over ten years). SO. av f1: inn in; abl tee ati is Carl I o Shafer 6. Generally in the companies responding, there seemed little or no relationship between the effort, in terms of time and budget spent on evaluation of management training activities, and: a. the level of management training; b. who writes or deve10ps the programs; c. who leads or conducts the pregrams; d. where the training program is given or e. the type of company (major product). 7. By far the major deterrent to effective evaluation in the opinion of the respondentcompanies was that evalu- ation research techniques were difficult to apply in productive or Operating situations. There was also some indication that the research techniques which are available are inadequate and are too likely to be in- fluenced by variables and contaminating factors in an industrial situation. Generally there appeared to be interest in evaluation; finances and time were avail- able for evaluation; there is knowledge of the research techniques available; top management stresses evalu- ation and is interested in the findings when evaluation is performed. 8. As revealed by the survey, questionnaires of various de- sign and purpose were the most often used method of 10. (D a: f e t}: of co fe 9. 10. Carl I. Shafer evaluation. Questionnaires designed to assess the feelings or like of a program and the benefits of a program were the type most frequently used. No companies offered revolutionary evaluative ideas or research methods other than.variations of those found in the literature. Few published studies could be located which attempted to give an industry-wide picture of evaluation rac- tiggg. None were comprehensive in the sense of cover- ing adequately the methods or effort in the area of evaluation. But all seemed to be using inadequate data and publicizing that they found a general lack of ef- fective evaluation of management training programs. 0n the contrary, this more comprehensive study, in terms of evaluation practices and effort, showed the selected companies are engaged, at least to some extent, in ef- fective evaluation activities and are somewhat cognizant of what effective evaluation includes. A STUDY OF THE EVALUATIVE PRACTICES IN MANAGEMENT EDUCATION AND DEVELOPMENT PROGRAMS IN SELECTED UNITED STATES COMPANIES By 4‘ 0 Carl I? Shafer A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF EDUCATION College of Education 1961 sert Prof majo the his disc h C01 ale inE to} the ti: DIN A O ' .A Z :.../ i" "l/ x.“ 53/2Qfiét? ACKNOWLEDGEMENTS Sincere appreciation is expressed to a number of in- dividuals for their contributions helping to make this dis- sertation a reality. Special appreciation is given to Professor Lawrence Borosage, the writer's doctoral program major adviser and dissertation chairman. His inspiration, the sharing of his time and experience, wise counsel, and his encouragement of individual scholarly pursuit made the dissertation a most interesting and meaningful project. Other doctoral program committee members who gave liberally of their time were: Professors George Brandon and William Harley, Education; Einar Hardin, Economics; Rolland Simonds and Dalton McFarland, General business. The author is grateful for their constructive suggestions as the dissertation came to fruition. Further appreciation is expressed to the author's colleagues for their suggestions and practical counsel; also to those who participated in the Juryman group preced- ing the survey of companies; and to the industrial educa- tors who completed the survey-questionnaire indicating their company activity in evaluation of management educa- tion and development programs. Acknowledgement by name is precluded, but the debt is real. As usual, my wife Carol gave liberally of her time and comments and proved indefatigable in repeated typings ii of‘ we the as for res1 of the manuscript. Her interest and willing helpfulness were a constant source of encouragement. The contribution of these individuals made writing the dissertation a satisfying experience. While the author gratefully acknowledges the assistance, full responsibility for the research and the views expressed must inevitably rest with the author. Carl I. Shafer iii CHA III. Iv, CHAPTER I. II. III. IV. TABLE OF CONTENTS NATURE OF TE SNDY O O O O O O O O O O O 0 Setting of the Study. . . . . . . Delimitation of the Study . . . . Need and Importance of the Study. Definition of Terms . . . . . . . Investigational Procedures. . . Preview of Succeeding Chapters. Bibliography. 0 o o c o o o o 0 THE THEORETICAL ASPECTS OF EVALUATION History of Educational Evaluation Purposes of Evaluation. . . . Some Principles of Evaluation . Steps in the Evaluation Process Focus of Evaluation . . . . . . Methods of Evaluation Summary . . . . . . . Bibliography. . . . . REVIEW OF EVALUATIVE STUDIES IN LITERA- TUBE-00000000000000.0000 O O O O O O O O O O O O O O O I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O C C O O O O O C O O O O O O O O C O O O O O E Industry-Wide Studies of Specific Evalu- ation Prac'ticeao c c o o o c o o 0 Evaluation of Human Relations Pregrams. . . Evaluation of Management Education and De- velopment Programs for Industry Given by Universities. . . . . . Other Attempts to Evaluate Management Edu- cation and Development Programs in Industry . . . . . . . . . . . . . . . . summary 0 O O O O O O O O O O O O O O O 0 Bibliography:............... THE SURVEY AND FINDINGS . . . . . . . . . . Development of the Questionnaire. Selection of Companies. . . . . . Distribution of Questionnaire . . Number of Responses . . . . . . . Distribution of Responses . . Nature of Management Training Activities in the Selected Companies . . . . Industry Groups -- Time and Budget Spent on Evaluation 0 o e o 0 Size of Staff -- Time and Budget Spent on Evaluation 0 o o e o o o o 9 c o o o o 0 iv 0 O O O O O O O O O O O O O O O O O O O O 0 PAGE mmew H 10 11 15 16 21 23 26 27 32 34 55 55 38 39 41 43 46 #7 49 52 56 59 CHAJ IV. V. PAGE THE SURVEY AND FINDINGS (Continued) Years of Operation -- Time and Budget Spent on Evaluation. . . . . . . . . . . 62 Tap Management Stress on Evaluation -- Time and Budget Spent on Evaluation. . . 63 Company Plans for Future Evaluation Effort. 65 Deterrents to Effective Evaluation. . . . . 68 Focus Of Evaluation 0 o o o o o o o o o e 71 Methods and Systems of Evaluation . . . . . 76 Summary of Findings . . . . . . . . . . . . 9O Bibliography...0000000000000 95 CONCLUSIONS AND RECOMMENDATIONS . . . o . o 97 ConCluBions o o o o c o o o c o o o 98 Recommendations Concerning Evaluative methOdsoooooo no... 106 Recommendations If There Is to be Greater Effort in Evaluation . . . . . . . . 109 Recommendations on the Purpose of ENalu- ation. . . . . . . . . . . . . 111 Recommendations for Further Study . . . . . 112 APPENDICES. . . . . . . . . . . . . . . . . 115 TABLE 2. 3. 10. ll, 13. 14. 15. 16, TABLE I. 2. 3. 4. 5. 9. 10. ll. 12. 15. 14. 15. 16. LIST OF TABLES Distribution.of Responses by Industry Groups . . . . . . . . . . . . . . . . Distribution of Responses by Size of Company................ Average Per Cent of Time Management Training Departments Engage in Various Levels of Training . . . . . . . . . . Writing and Development of Management Training Programs. 0 o e o o o o o o 0 Instruction of Management.Training Pro- gramsoooooooooooeeooeo Location of Management Training Programs. Industry Group and Per Cent of Time Spent on Evaluation. . . . . . . . . . . . . Industry Group and Per Cent of Budget Spent on Evaluation. . . . . . . . . . Training Staff and Per Cent of Time Spent onEV’aluation............. Training Staff and Per Cent of Budget Spent on Evaluation. . . . . . . . . . Years of Operation and Per Cent of Time Spent on Evaluation. . . . . . . . . . Years of Operation and Per Cent of Budget Spent on Evaluation. . . . . . . . . . Tcp Management Stress on Evaluation and Per Cent of Time Spent on Evaluation . Tap Management Stress on Evaluation and Per Cent of Budget Spent on Evaluation Future Effort and Time to be Spent on Evaluation.............. Greatest Deterrent to Effective Evalua- tion of Management Training. . . . . . vi PAGE 51 52 53 57 58 61 62 63 65 66 69 TABLE 17. 18. 19. 20. PAGE Focus of Evaluation in Companies' Example of Their Use of Evaluation Methods. . . . 75 Methods of Evaluation in Companies' Example of Their Use of Evaluation. . . . 77 Systems of Evaluation . . . . . . . . . . 81 Planning for Evaluation . . . . . . . . . 89 vii APPEN II III IV V] LIST OF APPENDICES APPENDIX PAGE I. Overall Objectives of the Survey. . . . 116 II. List of Jurymen That Were Sent Ques— tionnaires and Objectives of Survey for Appraisal and Comment. . . . . . 118 III. Questionnaire -- Survey of Evaluation of Supervisory, Management or Execu- tive Training Programs . . . . . . . 120 IV. Cover Letter to Questionnaire . . . . . 124 V. Follow~Up Letter to Questionnaire . . . 125 VI. Names of Formal Management Education and DeveIOpment Programs Cited by the Companies as Their Best Example of Use of Evaluation Methods . . . . 126 viii ation devel this synth and t is re tices selec Surve PPOgI CHAPTER I NATURE OF THE STUDY The theoretical aspects and the practices of evalu— ation actually used in formal management education and development programs in industry are the subject matter of this study. The theoretical aspects of evaluation are synthesized from the literature of education, psychology and business. The major published research on evaluation is reviewed and analyzed. A survey of evaluation prac- tices in.management education and deve10pment programs in selected companies in the United States is reported. The survey reveals current practices of evaluation in.these programs. Setting of the Study The demand for qualified managers at all levels of industry has increased over the last twenty years. The era when an individual could deve10p slowly by experience is rapidly disappearing. Today, as industry grows and be— comes more complex, the education of the manager needs to be broader and the pace of deveIOpment quickened. This education and development, or lack of it in the managerial force (10:1 geria aimed In-pl grams deve] since in t2 ins 1 mal ] surv Deep PSpo Ment Vari force, can be a factor of business success or failure (10:l,‘ 4:2). To prepare the individual to make effective mana- gerial decisions, industry has supported many activities aimed at the education and deve10pment of the manager. In-plant training, university or association sponsored pro- grams, formal off—the-job training and informal self- develOpment activities have received the greatest emphasis since World War II (10:1). In 1960 the number of managers in the United States enrolled in formal management train- ing programs exceeded 500,000. The 1948 total in all for- mal programs was ten thousand (1:1). Several large companies, it was revealed in the survey which is a part of this study, employ over twenty peOple as full-time management trainers. One corporation reports current yearly expenditures of four million dol- lars on training. Interestingly enough, smaller organiza- tions appear as successful as the larger ones in imple- menting broad, all inclusive development programs for the various levels of their managements (7:7-8). Ten years ago the American Management Association, a management training institution and disseminator of man- agement information, had a staff of forty people; now it ‘10:1. This method of giving bibliographic infor- mation.will be used throughout the study. The number be- fore the colon denotes the source in the bibliography at the end of the chapter. The number after the colon indi- cates the page in that source. con tic act. impa effe spon ing a pa; fere: has 400. Ten years ago AMA held twelve conferences or seminars a year, and now the number has increased to 1,000 a year. The American Society of Training Directors has grown in membership from 100, when organized in 1945, to considerably over 5,000 members currently-~another indica- tion of the increased emphasis of management training activity (6:12, 2:54). Basically, management education and development imparts the knowledge, skills and attitudes required for effective performance on present jobs or for higher re- sponsibilities. Therefore, any activity aimed at promot- ing growth and improvement in this area can be considered a.part of management development (10:1). Courses, con- ferences and meetings of various types and having varied objectives, however, seem to be the major method, other than on-the-job training, to impart or generate this growth and improvement (7:21). In summary, then, there is a great demand for ef- fective managers to run companies successfully, and there are indications of increased management training to meet this demand. Courses or conferences are usually relied upon as the primary means of formal management education and development. But how are these pregrams evaluated? What are the major evaluation experiments in this area? What are the general practices of evaluation in management training programs in industry today? These basic questions and researched explanations are the subject of this study. ing 1 exple steac dust: ative princ to ha as be. ture u grams. 800pe, VIGWed Delimitation of the Study This study is not an exhaustive investigation deal- ing with all aspects of evaluation. No attempt is made to explore every theoretical ramification of evaluation. In- stead, a background and framework are presented which in- dustrial educators may use as guiding principles in evalu- ative attempts within their companies. The theoretical principles concerning evaluation which are presented appear to have stood the test of time or are generally recognized as being authoritative by specialists in the literature. Many evaluative studies are presented in the litera- ture which discusses particular management training pro- grams. Only those writings which appeared noteworthy in scope, pioneering or unique in.method, however, are re- viewed in this study. While the phrase, “management education and devel- opment," in its broadest sense may include many activities, in this study it is delimited to include only formal pro- grams, courses or conferences which attempt to promote managerial growth and improvement. The evaluative prac- tices in the management education and deveIOpment programs are confined by the survey to those found in 118 large com- panies in the United States. This representation, however, probably reflects much of the major activity in evaluation of management training programs in this country. study of tr ation intert the s: condu< Beyond the limitations of time and space of this study are two broad‘related areas: first, the methodology of training needs analysis; and second, personnel evalu- ation and its many ramifications., It is hoped that others interested in these related problems or in tepics outside the sc0pe of discussion of evaluation in this study might conduct the additional needed research. Need and Importance of the Study The need and importance of the research undertaken and described in this study were derived from several sources. One primary source was the literature of or re- lated to industrial training. An increasing number of articles discussing evaluation seem to be appearing each year in the journals of training, psychology and business. Each of these articles usually attempts to shed light upon some facet of evaluation. There has been, however, no single comprehensive piece of research which pulls to- gether and analyzes the many important articles and their ideas about evaluation in management training. Frequently an author challenges the training pro- fession to exert greater efforts in the area of evaluation. Exemplary of this challenge is that found in Kirkpatrick's (8:52,34) “The Most Neglected Responsibilities of the Training Department." In this article Kirkpatrick lists, in his Opinion, three main neglects, one of which is: "To an r_.._.. eval sult trai to freq cond ness ties. funct impli much of 30 SESts BTGat 0mmen Socie 0f hi 285 p ConcSr 161113 b the tr traini tion a, Ofn evaluate training programs and make use of evaluation re- sults." Kirkpatrick emphasizes that while results of training evaluation.cannot be transferred from one company to another, evaluation techniques and procedures can be frequently transferred and adapted. Reeve (ll), Belman and Bliek (3:51) and Mead (9:65) conducted surveys by mail of training directors in busiw ness and industry concerning their duties and responsibili- ties. Each study showed that evaluation was an important function of the director or training department, but the implication was that work in the area of evaluation left much to be desired. After.Reeve's (ll) comprehensive study of 308 training people and their responsibilities, he sug- gests that "methods of evaluation" is an area that has great need of research. Another motivation for the present study was a rec- ommendation by the 1950 Research Committee of the American Society of Training Directors. This five-man group composed of high level industrial and university educators surveyed 283 peOple by questionnaire in industrial firms, business concerns and in colleges. This Committee searched for prob— lems broad in scOpe and interest and worthy of research in the training field. Of importance was that "evaluation of training programs" received the second highest considera- tion and was preceded only by the related research problem of "supervisory training." (13:2,4) pert atiox shee1 pendj authc publi naire 7 Also, an endeavor was made to ascertain current ex- pert opinion on the need for a survey of training evalu- ation which was to be included as part of this study. A sheet describing the "Overall Objectives of Survey," Ap- pendix I, was sent to twenty-two peOple believed to be authorities in evaluation because of their experience or publications in this field. A proposed survey question— naire was also included for their appraisal and comment. Seventeen of the twenty-two selected jurymen, Ap- pendix II, responded with proposed revisions to the ques- tionnaire or personal letters of suggestion; however, only six returned the "Overall Objectives of the Survey," Ap- pendix I. In every returned letter though, a ”Yes" was given to the question, "Do you believe there is a need for such a survey?" One juryman from a university also empha- sized, "I feel the project is highly worthwhile . . ." Later, as the questionnaires were sent to the indus- trial concerns participating in the survey, several encour— aging letters were received. One letter from the head of the management deve10pment research staff in a large cor- poration stated in part, "The topic you have selected for study is indeed a most important one." Further evidence of the need and importance of the survey may be indicated, in some measure perhaps, by the fact that 101 of the 118 persons responding to the questionnaires requested a sum- mary cOpy of the findings of the survey. and C intex persc evalu ally be cl: becaus Countm light given Meanin Finally, it was believed the research undertaken and described in this study would be generally useful and interesting to university and industrial educators and personnel administrators,since the study seeks to uncover evaluation principles and practices which are not gener- ally known. Definition of Terms Although most of the terminology in this study will be clear to the reader, several of the terms need defining because of varying connotations in various parts of the country and in different institutions or industries. In light of this, the following stipulative definitions are given so that the reader might readily understand the meaning intended by the writer. 1. "Evaluation" in this study is used primarily in its broadest sense, that is, any reasonable assessment of "To what extent was accomplished that which was attempted?" Evaluation.may make use of measurement but is not limited to it (6:1, 12:21, 15:1-2). The words: theory, practices, ideas, principles, methods and purposes have their usual definitions but are frequently pre- faced with the term "evaluation“ describing a method concerning evaluation, a practice concern— ing evaluation and the like. 2. 5. 4. 5. "Education" is used in its general sense, and also as it pertains to the acquisition of man- agement information and principles through formal class or conference groups (15:9). "Development" is the intellectual growth and emotional maturation of the individual in mana- gerial competence (15:9). Education and develop- ment are used in the study to denote the combined intended result of company sponsored formal man- agement conferences or programs as Opposed to in- formal or self-development activities and job development prescribed by an individual's super- visor. "Management training" in this study refers to any level of education and development given to individuals that direct peOple or operations in industry. Ianagement training, a commonly ac- cepted term in industry, is used interchangeably with management education and development. Three management levels are discussed: "Top management" is the executive management which has overall responsibility for policy making. "Hid— dle management" has administrative responsibility, that is, implementing, interpreting and carrying out company policy. “Foreman or other first eral lines and t 10 level management" is supervisory management which directs the activities of the employee work group (15:9). Investigational Procedures After the study area was selected and outlined, sev- eral lines of inquiry were pursued-~the first and second lines of attack were conducted somewhat simultaneously, and the third was built upon the first two. The first line of attack examined documentary re- search concerning the theoretical principles of evaluation. Literature from education, psychology and the business area was minutely investigated and analyzed. An attempt was made to synthesize a reliable body of authoritative infor- mation concerning the theoretical aspects of evaluation. This served as the background for the review of research and the later survey of evaluation in management education and develOpment programs. The second line of attack was also directed to docu- mentary research. Studies were reviewed which in any man- ner surveyed or partially surveyed evaluation practices. Also, experiments which discussed evaluative attempts with management training Programs in specific companies were consulted. ' Finally, in the third line of attack, a question- naire was determined to be the most effective method of gath tice A qu up b sale the over sent sugge Appen data cuSse< TheOI‘e hiStO] and p Ll prOCeg theSiz perime ture . praCt: ation 11 gathering the data which would reveal the evaluative prac- tices of management training in.the selected companies. A questionnaire was designed and a mailing procedure set up believed capable of achieving maximum returns from the selected companies. Pilot questionnaires were sent to the jurymen group, Appendix II, for their appraisal. The overall objectives of the survey, Appendix I, were also ‘ sent to the jurymen with provision for their comments and suggestions for improvement. The revised questionnaire, Appendix III, later mailed to 148 companies, supplied the data upon which the survey findings are based. Preview of Succeeding Chapters The findings of the three lines of inquiry are dis- cussed in logical order in the study. Chapter II, "The Theoretical Aspects of Evaluation," begins with a brief history of evaluation. This is followed by the purposes and principles of evaluation, the steps in an evaluative process, and the focus and methods of evaluation as syn- thesized from the literature. Chapter III reviews the evaluation studies or ex- periments in management training as reported in the litera- ture. Several industry-wide studies of specific evaluation practices are discussed. The major attempts with evalu- ation of "human relations" management training programs pose of ma Opera budge the s a num pleme to ev rents' ter n« with 4 12 are reviewed, and a number of studies having unique pur- pose, sc0pe or evaluative method are also presented. Chapter IV describes the survey findings. The level of management training activity, size of staff, years of operation and the amount of effort in terms of time and budget spent on evaluation are given. Chapter IV reveals the stress which tap management places upon evaluation and a number of other factors which may hinder or help the im- plementation of evaluation. Future effort to be devoted to evaluation was determined along with the greatest deter- rents to evaluation in these companies. Lastly, the chap- ter notes the focus of evaluation in these companies along with the methods and systems used in their evaluation. The major findings are summarized at the end of Chapters II, III and IV. In Chapter V these diverse find- ings are then brought together into a number of conclusions and recommendations which have been reached in a study of this purpose and sc0pe. III II I II Tunclnld mil; VAL Q~ W&. B E t p filldlqlankq Cb ~\~ .l .54 . t C O Svl ‘8 w. I. Zn . . . A K .0 LT... 4 5 6 n/. so. l. 2. 3. 4. 5. ,7- 8. 10. ll. 13 Bibliography Appley, Lawrence A., "Manager Training in Proper Per- spective." Management News, Vol. XXXIII, No. 4, April, 1960. PP 0 1‘7. Belman, Harry S., "A Survey of the Status and Function of Training Departments." Journal of the American Society Lf Training Directors, Vol.-EII, No. 8, August, 1 5 . PP. 34-36. Belman, Harry W. and Bliek, John E., "The Head of the Training Function." Journal Lf the American Societ 9; Training Directors, Vol. XIII, _No. 2, February, 19 9. pp 0 42-53 0 Bliss, Gordon "M., "The Training Director's Wider Re- sponsibility." Journal Lf the American Societ g; Trainigg Directors, Vol. XIV, No. 5. March, 0. p. 2. Borosage, Lawrence, "Some Considerations Regarding Evaluation of Training." East Lansing, Michigan, Na- tional Project in Agriculture Communications, 1958. p. 6. Cunningham, J. W., "Enlargement and Professionalization of the Training Function." Journal Lf the American Societ Lf Training Directors, Vol. XIV— No. 6, June, 1965 pp. 11—14. , , Current Practices Lg the Development g£_Man- agement Personnel. New York, American Management Association, 195 . Kirkpatrick, Donald L., "The Most Neglected Responsi- bilities of the Training Department." Journal Lf the American Societ of Training Directors, Vol. XIII, No. 4, Iprii, I95 0 PP0729550 Mead, George E., ”The Pre-Employment Education of Train- ing Personnel." Lafayette, Indiana, Unpublished Masters Thesis Submitted at Purdue University, 1954. , Program Leader's Manual, General Motors Ap- proach 32 Management DeveIOpmeng. Flint, Michigan, Management—Training Department, General Motors Insti- tute, 1960. Reeve, David F., "A Survey of the Duties and Responsi— bilities of Training Personnel in Business and Industry." The Journal of Industrial Training, Vol. VII, No. 5, September-October, 1955. 12. 15. 14. 12. 13. 14. 15. 14 Hammers, H. H. and Gage, N. L., Educational Measure- ment and Evaluation. New York, Harper and Brothers, I§ES. pp?“£§v7"636. Research Committee, American Society of Training Dir- ectors, ”Research Needed in Industrial Training." Journal of Industrial Training, Vol. IV, No. 2, March- Ipril, l§§0. pp. 2-18. Schwartz, Alfred and Tiedeman, Stuart 0., Evaluatin Student Progress in the Secondary School. New YOrk, Longmans, Green and Co., 1957. pp. xi, 454. Zerfoss, Lester F., "Progress in Management DevelOp- ment." Journal Lf the American Society Lf Trainin Directors, Vol. XI, No. 6, November-December,1 1§57. pp. 4-17. size lite: appl: ment that CHAPTER II THE THEORETICAL ASPECTS OF EVALUATION In this chapter an attempt will be made to synthe- size some of the theoretical aspects of evaluation from the literature of education, psychology and business which is applicable to industrial management education and develop- ment programs. The writings concerning evaluation reveal that educators appear to be more proficient with evaluation theory and practice, as related to the tOpic of this study, than any other group. A perusal of the Journal of EQEEE? tional Research shows that nearly every area of public and private education is concerned with evaluation. Educators in business and industry have also become increasingly aware of the need for evaluation. Articles pertaining to this t0pic have become more frequent in re- cent years in the Journal of the American Society of Train: $28 Directors. Much of the literature on evaluation, however, describes confessional accounts or experiments that were performed attempting to evaluate a course or program. A smaller number of articles gave insight into the tepic of this chapter--the theoretical aspects of evaluation. In this chapter, it is desirable to make a brief review of evaluation history so that evaluation will be 1.5 more eval synt ful the atio: plice pros, disti. tice < grams Shoot ment k ligencl t° the COHC ep 16 more understandable in its modern context.‘ The history of evaluation is followed by some principles of evaluation as synthesized from the literature. A series of steps, use- ful to obtain effective evaluation, are set forth. Lastly, the chapter is concluded by outlining the focuses of evalu— ation and some of the methods of evaluation which are ap- plicable to industrial management education and development programs. History of Educational Evaluation Evaluation of management training has no history distinct from educational evaluation in general. The prac- tice of evaluation as applied to management training pro- grams appears from the literature to be only a late off- shoot of the educational evaluation movement. This move- ment had its beginnings about eighty years ago with intel- ligence and psychoIOgical tests. The early tests gave way to the concept of measurement and finally to the broader concept of evaluation. Schwartz and Tiedeman (1826:?) recall that testing in America actually had its beginning in England, France and Germany. Notable early contributors were England's Sir Francis Galton, who develOped a wide array of statistical techniques; Alfred Binet working in France, who published a scale for measuring intelligence; and the German Wilhelm 1? Wundt, who formed the first experimental laboratory in psychology. Travers (20:5-5) gives credit to Joseph Mayer Rice for the transmission of the earlier concepts of testing to America. Rice, an M.D. from Columbia University, after practicing medicine in New Yerk, traveled to Eur0pe in the 1880's to study pedagogy and psychology at Jena and Leipzig for two years. On his return to the United States, and filled with the zeal of educational reform which he had found in the centers of learning in EurOpe, he went on an inspection of the schools of the U. S. in 1892. Rice visited with more than 1200 teachers in their classrooms in the east and middle west and also visited some twenty institutions for the training of teachers. Travers (20:5- 5) reports that Rice's studies were an outstanding piece of research and had far reaching implications, although at the time they seemed to have no observable effect on educa- tion in America. The extension of public school education stimulated psychologists and educators to take a further look at the problems of testing and measurement in American education. Edward L. Thorndike developed standardized tests and scales for measuring achievement. Lewis Terman adapted the Binet intelligence test for use in.the U. S. Rudolph Pinter and Donald Paterson brought out non-language intelligence tests, and Arthur Otis contributed to the development of the first grO‘ gan were clas tere "Dur mode were move} most 18 group of intelligence tests during World War I (8:6-7). Between 1910 and 1950 fashionable school systems be- gan using batteries of standardized tests, although many were poorly constructed and not particularly useful to the classroom teacher (18:6-7). Micheels (15:1) termed the in- terest in testing "a rash-like outburst." He also said, "During this time tests, like some of the wonder drugs of modern medicine, were overused. In many instances they were also misused. Some of the apostles of the testing movement were certain that objective tests were the key to most of the educational problems then existing." But an inevitable reaction set in during the 1950's. One reason was the renewed attention given to the objectives of modern education. As educators strove to achieve greater effectiveness in evaluation, it became obvious that formal tests alone were of limited use in appraising certain be- havior changes. Under the leadership of Eurich, Raths, Tyler and Wrightstone, a movement deve10ped to broaden the concept of measurement to include such attributes as atti- tudes, interests, ideals, ways of thinking, work habits and personal and social adaptability (7:428, 15:2). The 1930 to 1940 period might be called the "Rise of Evaluation" era as Opposed to the emphasis on measure- ment which preceded it. Many new batteries of tests came out. Personality tests such as the Rorschach and other projective tests appeared, interest inventories were devised l9 and anecdotal records were introduced as an evaluation technique. Textbooks before the 1950's were designed to orient the students of education in the scientific method of meas- urement and usually had only the word measurement in the title. After 1950, book titles including the notions of measurement and evaluation became more frequent. The term evaluation gained ever increasing acceptance as a result of Tyler's writings coupled with the revolt against the traditional curriculum. There were serious attempts to find ways not only to test tangible skills and knowledge but also the less tangible objectives of a modern educational program (25:6—7, 16:21). The 1940 to 1950 decade marked the refinement of the techniques generated during the 1950's. ‘The Eight Year Study (19) of thirty high schools published in 1942 illus- trates the broad scape of a modern evaluation program. Here Tyler and others attempted to make evaluation opera- tional. Teachers, administrators and research people en- gaged in evaluating the effectiveness of educational pro- grams had to develop their own instruments as they sought to evaluate study skills, critical thinking, appreciation and interests. The study tended to show that testing specialists had too long been concerned with the knowledge aspects of education at the expense of the intangible out- comes of the educative process. 20 Even as WOrld War I contributed to testing, so World war II created the conditions for further evolvement of the evaluation.movement. The problem of how to train and use effectively the millions of men and women in and out of the armed service provided possibilities of unlimited experi- mentation (18:7-8). The last decade of work in evaluation, and the re- sulting principles and methods are partially exemplified in the discussion in the remainder of this and the next chapter. This brief review of evaluation history will serve as a contrast showing the advancing thoughts in evalu- ation and why certain evaluative concepts and principles are presently stressed. Evaluation, then, is the evolu- tion and product of about eighty years of research and ex- perience by educators and psychologists. Purposes of Evaluation A number of authors (5:548-50, 6:1, 7:285, 17:205, 18:2) generally agree that the pagig purpose of evaluation should be to achieve effectiveness in education. The basis for determining the effectiveness of education should be the extent educational objectives are realized. Borosage (5:1) enlarges upon the basic purpose of evaluation and sets forth several purposes as related to a training program: 1. 2. 3. 4. 5. 6. 7. 21 The chief purpose of evaluation is to validate the total approach to training that is used in the organization. The second purpose is to determine whether con- tent in a training program is functional. .The third purpose is to determine needed modi- fications in instructional method. The fourth purpose is to provide greater psychological security and morals to the staff responsible for training both individually and collectively. The fifth purpose is to provide information basic to effective guidance in an individual develoPment program. Only as we appraise indi— vidual achievement are we in a position to plan additional improvement. The sixth purpose of evaluation is to provide a sound basis for public relations. The seventh purpose is to examine the extent to which financial resources have been used ef- fectively. Some Principles of Evaluation Principles of evaluation might well be considered the pertinent, yet general framework of thoughts or ideas guiding the evaluator. Tyler (22:495-97) states the prinp ciple that any evaluation of educational achievement should not neglect the inter-relationship of the student's infor- mation, skills, ways of thinking, attitudes and interests. Instead the evaluation should make an effort to relate these factors. According to Tyler, humans act in a fairly unified fashion; hence in any given situation, information is not usually separated from skills. 22 Harris (7:58) believes that evaluation offers the greatest potential benefit if it is conducted over a period of time and is a continuous and built-in part of the total training process. Belman and Remmers (1:51-52) tend to agree but with this reservation: "It is difficult to evalu- ate long-range training programs." They suggest evaluation may be more workable if attempted in lesser units coincid- ing with the units of an educational program. Also, pro- vision should be made for evaluation during the planning stages of training programs. Harris (7:58) further submits that evaluation should be concerned with results rather than effort expended, i.e., evaluation of a particular teaching technique should be appraised in light of the changes in student behavior or knowledge. Borosage (5:4) and Harris (7:58) agree that self- appraisal is usually better than evaluation by outsiders, although a combination of both is frequently even more de- sirable. They also concur that an evaluation.project to be most effective should allow participation by all indi- viduals involved. The factors of self-appraisal and par- ticipation, they believe, help achieve greater motivation to improve the education program. . Belman and Remmers (1:51-52) indicate the establish- ment of control groups is a desirable evaluation technique. The evaluator then.has a basis for comparison. This tech- nique, however, does not assure that variables and 25 contamination factors will be eliminated and these factors still must be isolated and taken into consideration. A last evaluation principle according to Belman and Remmers (1:51-52) is "Results of the evaluation should be expressed in terms that are understandable to those in— volved." While mathematical terms or the language of sta- tistics often express most accurately the results, common sense dictates that the interpretation of the data should be made in the terminology of the organization and under- standable to those who are to use the results of the evalu- ation. Steps in the Evaluation Process A number of authors (5:1, 7:482, 22:497-501, 4:2, 15:97, 16:50, 25:26, 2:25, 6:15) list what they term as steps or guides in the evaluative process. While there is some variance in details, the writers from diverse fields of education agree on the content of a series of steps. A synthesis of these recommended steps from the various sources related to industrial management education and development follows: 1. OBJECTIVES. First is the establishment of the purposes or objectives which the education is to accomplish. In industry, the education and 2. 5. 4. 24 training may have been or will be based on a needs survey. Thus in this step, the task would be to re-define needs in terms of ob- jectives. IDENTIFICATION. The second step suggests the identification of situations where the changes in behavior may be noted in the day-to-day work situation. This may include the selec- tion of available tests or measures or a test situation apprOpriate for the major objectives as outlined in step one. APPLICATION. Included in this step is the trial and refinement of the most promising methods or instruments for obtaining and ap- praising the evidence regarding each objective. The refinement may include the evaluation of the evaluative device or method itself in terms of how well it serves its purpose. RESULTS. The fourth step prOposes to analyze, interpret and use the results of evaluation. The data have their complete meaning only after they have been interpreted and related to the purpose and content of the program. Moreover, the results of the evaluation should be in a 25 form that can be interpreted by the intended reader. 5. IMPROVEMENT. Finally, evaluation is intended to be an integral and continual part of the educational process. The results of evaluation normally would result in modification and im- provement in the educational program.' While five steps in the evaluative process have been listed here, the contributors are not in agreement as to the number of steps involved. Agreement, though, is not necessary. More important is the implication that an orderly process should be followed using sound evaluative thinking. The exact number of steps should be flexible enough to meet the demands of the program to be evaluated. A brief elaboration of step one, Objectives, seems necessary. The literature is replete with guidance on set- ting objectives or goals in education and their importance to effective evaluation. Tschudin, Belcher, and Nedelsky (21:110) explain that objectives can be classified or separated into two components: "behavior" and "content." Behavioral objectives refer to the thinking, feeling or acting involved. Content refers to the subject area which is also inherent in the objectives. The major problem in setting objectives of an edu- cation program, however, appears to be the clarification of the objectives in such terms so that it is later 26 possible to determine whether or not the objectives have been achieved. Werkers in evaluation find that a statement of objectives in some form is readily available, yet often these objectives do not provide a suitable basis for evalu- ation because the objectives are stated in very general terms. The difficult task, then, is to translate broad, vague objectives into more specific results (7:485, 5:549, 8:15, 20:10). Rivlin (17:207) asks, "How can we measure progress toward a goal unless we know what the goals are?" Focus of Evaluation The focus of evaluation is, in general, ghap the re- searcher evaluates. Kirkpatrick (9:5, 10:21, 11:15, 12:28), Besco, Tiffin and King (2:18), Messer (14:56-57), Borosage (4:2-6) and Korb (15:19-104) designate a variety of focuses. The authors emphasize that several of these focuses should be considered for comprehensive, effective evaluation: 1. The ADMINISTRATIVE ARRANGEMENT under which this training pragram was given 2. The GOALS AND OBJECTIVES of the training program 3. The METHOD OF PRESENTATION of the training pro- gram (conference techniques, conference leaders) 4. The PARTICIPANTS' PROGRESS during the training program 5. The FEELING 0F PARTICIPANTS about the training program 6. The LEARNING OF PARTICIPANTS as a result of the the training program (certain.management prin- ciples, supervisory knowledge) 27 7. The BEHAVIOR OF PARTICIPANTS as a result of the training program (different actions back on the job resulting from the pregram) 8. The RESULTS, EFFECT, IMPACT on the group, inter- group relationships or organization as a result of the training program. Kirkpatrick places the last four listed focuses of evaluation in ascending order of difficulty. He believes number five, "the feeling of participants about the train- ing program," is the least difficult evaluation and most often performed; number six, "the learning of participants," less attempted; number seven, "the behavior of partici- pants," and number eight, "results, effect, impact on the organization," the least attempted, yet most important. Methods of Evaluation The methods of evaluation are the "how to" or the techniques that may be used in the evaluation of management education and development programs. A sampling of the techniques suggested by Kirkpatrick (9:5-9, 10:21-26, 11:15-18, 12:28-52), Besco, Tiffin and King (2:15-27), Messer (14:44-64), Borosage (4:2-6) and Korb (15:99-104) reveals several possible evaluation techniques for manage- ment education and development programs. The following list of techniques was included as part of the survey ques- tionnaire, Appendix III. A discussion of the systems of 28 evaluation and the interrelationship of these techniques in practice is included in Chapter IV of this study. 1. 2. 3. 4. 6. 7. 8. 9. 10. 11. 12, Course material information tests: a. Before program b. During program c. Directly after program d. Sometime after program Attendance at pragram Amount of participation by participants Interviews with, or questionnaires to: a. Participants b. Participants' supervisors c. Participants' employees d. Participants' peers Check of users' attitude toward service or product rendered by participants' department Use of participant questionnaires on: a. Changed attitudes b. Feelings about program c. Benefits from program Control and experimental groupings Reports by outside observers Direct observation of participants' actions on job after program Organizational attitude or communications sur- veys Use of records on absenteeism, material waste, turnover and/or grievances in the participants' work group Significant increases in "performance review" ratings of the participants by their super- visors 29 Summary 1. Historically, Schwartz and Tiedeman (18:6-7) mark the testing-measurement-eva1uation movement as being about eighty years old. Early pioneers in the movement were England's Sir Francis Galton developing statistical concepts, France's Binet with scales for measuring in— telligence, Germany‘s Wundt forming the first experi- mental laboratory in psychology and America's Rice who transmitted EurOpe's pedagogy and psychology to this continent around the turn of the century. The testing- measurement-evaluation movement grew in this country with the work of Thorndike (standardized achievement tests), Terman (adapted Binet's intelligence test for the U. 8.), Pinter and Paterson (non-language intelli- gence tests) and Otis (intelligence tests during World War I). The 1950's saw efforts to broaden the idea of testing-measurement and add the concept of evaluation. Eurich, Raths, Tyler and Wrightstone were leaders in this 1950-1940 "Rise of Evaluation" era. The 1940-1950 decade marked the refinement of the techniques founded during the 1950's. The summary thoughts that follow briefly set forth the theoretical aspects of evaluation as it has evolved in the 1950's and in particular how evaluation might be applied to management education and development programs today. 50 2. A number of authors (5:548-50, 7:285, 6:1, 17:205, 18:2) generally agree that the purpo§e of evaluation is to achieve effectiveness in education. The basis for determining the effectiveness should be the extent edu- cational objectives are realized. 5. The principles of evaluation are many and varied. Prin- ciples might well be considered as the pertinent, yet general framework of thoughts or ideas guiding the evalu- ator. These principles appear important for the indus- trial educator: a. Evaluation offers the greatest potential bene- fit if conducted over a period of time and a built-in part of the total training process. b. Evaluation should be concerned with results rather than effort spent in conducting the program. c. Programs with specific objectives can most easily be evaluated but variables which may influence results should be isolated and taken into consideration (7:58, 1:51-52). 4. A synthesis of recommended steps for evaluation from various sources (5:1, 7:482, 22:497-501, 4:2, 15:97, 16:50, 25:26, 2:25, 6:15) indicates an orderly yet flexible process that should be used in evaluative attempts: a. OBJECTIVES the education is to accomplish should be formulated or re-defined. A clari- fication of objectives in terms of measurable change is frequently necessary. b. IDENTIFICATION of a "measure" or "test situ- ation" where changes may be noted is suggested. Ce 6. 5. Lastly, ation. 51 APPLICATION and/or refinement of the "measure" or evaluative device may be necessary. RESULTS are then to be analyzed, interpreted and put in usable form. IMPROVEMENT or modification of the educational program should then be instigated. the chapter reviews the various focuses of evalu- ! It was suggested that several focuses be con- 3; sidered for comprehensive, effective evaluation. Finally, the research methods useful to evaluate management edu- 1 cation and development programs were outlined from.the é literature. These focuses and methods of evaluation are reflected in parts of the survey-questionnaire, Ap- pendix III, and were used to help obtain the data for Chapter IV. 1.. .2. 5. 4. 5. 6. 7. s. 9. 10. ll. 52 Bibliography Belman, Harry S. and Remmers, H. H., "Evaluating the Results of Training." Journal p£_ppp,gpfi§%pap Societ pf Trainin Directorg, VoI. XII, No. 5, , 9 . pp. 28-52. Besco, Robert, Tiffin, Joseph and King, Donald 0., "Evaluation Techniques for Management Development Pro- grams." Journal of ppg American So at p; Trai n Directors, Vol. III, No. II, Oct., I959. pp. - . Borosage, Lawrence, "PrOgress Report: Michigan Voca- tional Education Evaluation Project--Quarterly Report." East Lansing, Michigan, Mimeographed paper, June 50, 1959. Borosage, Lawrence, "Some Considerations Regarding Evaluation of Training." East Lansing, Michigan Na- tiogal Project in Agricultural Communications, 1558. p. . Egner, R. E., "Changing Concepts in Evaluation." North Central Association Quarterly, Vol. XXIV, No. 4, AprII, 1955. pp. 548-5 . Evaluapion pp Eypepgion. Washington' D. C. U. S. fiepartment of Agricu ture, Federal Extension Ser- vice, June, 1956. pp. iv, 84. Harris, Chester W., Editor, and others, Enc clo edia 8&0 Educational Research. New York, The MachIIan 50., l . Pp e nix, ism. Herzog, Elizabeth, Some Guide Lines for Ev luati e Re- search. Washington, D. . . . apartment of EeaIth, Education and Welfare, 1959. p. 117. Kirkpatrick, Donald L., "Techniques for Evaluating Train- ing Programs, Part I Reaction." Journal f the erican Society 9; Traini Director , VoIT_IIII, 0. 11, Nov., 0 Pp. 3" e Kirkpatrick, Donald L., "Techniques for Evaluating Train- ing Pragrams, Part II Learning. Journal of the American Society 2; Training Direcpors, Volt-XIIIT'N3.-I2, Eec., 0 DP 0 1-2 0 ' Kirkpatrick, Donald L., "Techniques for Evaluating Train- ing Programs, Part III Behavior." Jourpslzp£.the Ameri- can Sopiety p; Traini Directo s, o .' V, No. 1, Jan., 0 pp. 13"]. o 12. 15. 14. 15. 16. 17. 18. 19. 20. 21. 22. 25. 33 Kirkpatrick, Donald L., "Techniques for Evaluating Training Programs, Part IV Results." Journal of the American Societ g; Traini Directors, VoI. 117' N0. W, eb., 1730—19 . pp. '52 75253 . ' Korb, L. David, Traini the Su ervisor. Washington, D. C., U. S. Civ 1 erVice CofiiIssIon, 1956. pp. v, 125. Messer Elizabeth F. Assessi d Re ortin Traini Needs and Pro ress. ’Washing on,§%7 5., U. 3. CIvII oEEI ServicE—C ssion. 1956. pp. v1, 77. Micheels, William J., "A Broad Concept of Evaluation." Unpublished, mimeographed copy of speech. p. 6. Remmers, H. H. and Gage, N. L., Education 1 Measureme t and Evaluation. New Y0rk, Harper an r0 ers, . pp. xiv, 650. Rivlin, H. N., "Changing Concepts of Evaluation." Edu- cational Forum, Vol. XX, N0. 2, Jan., 1956. PP. 205- Schwartz, Alfred and Tiedeman, Stuart 0., Evaluating ew or , Student Progress in the Secondar School. Longmans, Green aid 50., I957. pp. x1, 454. Smith, Eugene R., Tyler, Ralph W. and others, Afippgis- Egg %%%tfi:pgrt13§2Stuggnt Pro§res§.. New York, arper Travers, Robert M. W., Educ tional Mea urement. New York, The Macmillan Co.m.m Tschudin, Mary S., Belcher, Helen C. and Nedelsky, Leo, Evaluation 3p Basic Nursipg Education. New York, G. P. Putnam's Sons, 1958. pp. xvi, . Tyler, Ralph W., "General Statement on Evaluation." Journal pf Educ tion 1 Research, Vol. XXV, N0. 7, March, 1942. pp. 4 -501. wrightstone, J. Wayne, Justman, Joseph and Robbins, Irving, Evaluation in Modern Educatio . New Yerk, American Eook 50., I956. pp. xI, EEI. CHAPTER III REVIEW OF EVALUATIVE STUDIES IN THE LITERATURE The purpose of this chapter is to review briefly the evaluative methods used in the studies of management educa-' tion and development programs as reported in the literature. Articles in journals and magazines, reports in pamphlet form, as well as portions of texts from the fields of edu- cation, psychology, and the general area of business were minutely investigated for evaluative research.having to do with, or related to, management training. Although the literature contains a surprisingly large number of evaluative studies, a review of all of the writings or even every detail of those desirable for inclu- sion would demand much more space than is allowable in a study of this scope. Thus only those writings which ap- peared of consequence in scope, pioneering or unique in method have been included. For convenience, the studies reviewed have been categorized according to their purpose of inquiry or type of program evaluated in the following areas: that of in- dustry-wide studies of specific evaluation practices; hu- man relations programs; university programs given for in- dustrial managers; and a general category of located studies. 54 35 Industry-Wide Studies of Specific Evaluation Practices Few studies were found which gave an industry-wide picture of evaluation practices. None were comprehensive in the sense of covering adequately specific evaluation practices and most revealed only miscellaneous data and indictments about the lack of evaluation. A Bureau of National Affairs (5) questionnaire found only one-fourth of the companies surveyed using spe- cific training evaluation techniques, in most cases some form of an attitude survey. An American Management Asso- ciation (16) study of company opinion about the effective- ness of their management education.pr0grams was based on evaluation methods the companies admittedly considered in- adequate. Mahler (15) found that about one in ten compan- ies used any systematic research e. find out what training was necessary, only one in forty studied the merits of their training methods, and the big majority of the survey indicated they used somebody's Opinion as to the effective- ness of their training. Evaluation of Human Relations Programs The evaluation of human relations management pro- grams apparently is a favorite with industrial trainers. 56 Several varied attempts are reported in the literature. The foremost study located, in terms of elaborate research procedures used and most often cited by others in the lit- erature, was the Ohio State-International Harvester Com- pany study by Fleishman, Harris and Burtt (4). They used a lSO-item questionnaire designed to measure leadership attitudes in foreman. The questionnaire was given before, immediately after and sometime after the training program to the control and experimental groups. The Bell Telephone Company study by Stroud (24) was similar to the Ohio State study in the research techniques used and in the attempts made to ascertain improved perform- ance and organizational effectiveness resulting from the training. But both studies agreed about the inadequacy of the self-rating type of test directly after a pr0gram in determining increased performance and effectiveness.' In the Ohio State study (4) four external organiza- tional criteria (absenteeism, accidents, grievances and labor turnover) were used as a measure of improved organi- zational effectiveness. A Detroit Edison study reported by Seashore (20), on the other hand, used a morale survey with extensive feedback to trainees, their supervisors and other employees, as a measure of organizational effective- ness. Kirkpatrick (8) tested with comment sheets his hu- man relations training pr0gram for foreman to reveal their 57 feelings about the program. Also, pre- and post-test scores on a human relations test were used to ascertain increased knowledge from the program, and correlation of the change in these test scores with job performance back at the trainee's company was performed--but with apparently incon- clusive relationships. Soik's (21, 22) study at the Allen Bradley Company was patterned after Kirkpatrick's with the addition of group participation as an evaluative indicator. To evaluate the effectiveness of training adminis- tration and training methods and to determine future train- ing needs, the Proctor and Gamble Company (20) employed a systematic and intensive interview program with foreman candidates and with the candidates' supervisors. Lawshe, Bolda and Brune (5, 9) used verbal responses to case problem situations after human relations training and correlated these responses with the judgment of so- called expert responses in order to test increased human relations knowledge. .' Lastly, in an evaluative attempt with a human re- lations program, Osterberg and Lindbom (14) used a delayed questionnaire (three years after program) sent to program participants to determine any continuing behavioral or or- ganizational change resulting from the training. 58 Evaluation of Management Education and Deve10pment Programs for Industry Given by Universities Only one piece of research was located which at- tempted to give an overall picture of evaluation of the university conducted programs for management. The Opinion Research Corporation (6) report reveals that forty per cent (the largest percentage indicated) of the companies sur- veyed evaluate the worth of the university courses for their managers by observing the managers“ performance back on the job after the program. The Bell Telephone System appears to have obtained the most publicity in the utilization of university pro- grams and also in the evaluation of their programs. One study at the University of Pennsylvania, reported by Viteles (26), evaluates a humanistic studies program for third and higher level supervisors. An extensive battery of tests, given on a before and after basis to control and experimental groupings, was used to determine increased intellectual background and modified attitudes, interests and values. The author of the Pennsylvania study is care- ful to emphasize that the research indicates only immediate outcomes. Another Bell study (2) on a program at Dartmouth and Williams Colleges, similar in research design, also in- cluded the use of tests plus personal interviews with par- ticipants one year after the program to ascertain longer range effects. 59' Trice (25) reports a unique evaluative experiment at Cornell where an initial and repeat study, and a study five months later were used to determine change in attitudes toward conference leading as a result of training. The evaluative instrument was a 100-item card sort on behavior compatible and incompatible with the management function of conference leading. Other Attempts to Evaluate Management Education and Deve10pment Programs in Industry Several other evaluative attempts with management education and development programs, other than university conducted or human relations programs, were noted in the literature. Three studies, two from the General Electric Company (12, 25) and the other at the B. F. Goodrich Com- pany (7), reported attempts to measure changed behavior or improved performance as a result of management training. In one case an intricate composite index of various person- nel statistics was used as a basis of evaluation. The second attempt utilized control and experimental groupings and series of questionnaires one year after the program to the pr0gram participants, their peers, those they super- vised and the participants' supervisors. The Goodrich evaluators also used questionnaires but only those ratings by superiors. In addition, a series of participant atti- tude and achievement tests was employed in the Goodrich study. ‘Zleffptag- . 4O Savitt (18, 19) went into an involved experiment to measure increased knowledge, as a result of management training, and than interestingly correlated the change of before and after test scores with the participants' age, formal education, supervisory experience and tests of mental ability. The Monsanto Chemical Company (1) held a series of plant problem conferences which were also considered train- ing ground for the participants. Evaluation techniques in- cluded the superiors' rating of participants' performance on the job on a before and after basis, a participant atti-l tude survey, and attendance at the voluntary conferences. Rich's study (17) at the Sharp and Dohme Company provided an interesting adaptation of the sociometric tech- nique where conferees rated each other. Through a system of confidential interviews, in which the results were re- vealed to each participant individually and through con- tinued inter-action in conferences, the leader attempted to improve inter-personal relations of the group inside and outside the conference room. Improvement resulting from the conferences was noted on repeated sociometric ratings during and at the end of the program. Meadow and Parnes (11) report the only experiment located which strives to measure results of a creative problem-solving course made popular by Osborn (l5) and his brainstorming procedure. Experimental and control groups 41 were employed with a battery of nine test measures includ- ing tests of creative ability, apperception and intelli- gence. Lastly, the General Motors - AC Spark Plug Division study (10) was the only piece of research available which reported an attempt to evaluate comprehensively a number of management courses given over several years on a plant- ‘wide basis. Structured personal interviews were exten- sively used with the participants of the various programs and were the evaluative technique in this case. Summary In the studies reviewed in the literature concern- ing the evaluation of management education and development programs, several evaluation methods and research designs ‘were presented. The following appear as usable evaluation methods: 1. Tests designed to measure the course or content material imparted as a result of the program, given before, during, immediately after and some- time after the pr0gram 2. External organizational criteria, e.g., absentee- ism, accidents, grievances, labor turnover, as a measure of improved organizational effectiveness as a result of the management training pr0gram 5. Morale, organizational attitude or communications type of surveys to determine improvement in the participants' work groups 4. 5. 6. 7. 8. 90 10. 42 Comment questionnaires of various designs after the program to ascertain the participants' like of or benefit from a program Group participation and/or the attendance at a series of conferences which are held on a volun- tary basis Systematic interviews with program participants and/or their supervisors on the merits of the program Systematic obServation of the performance of the management person back on the job after the program Tests of various design and purpose given on a g before and after basis to control and experi- ; mental groupings v A composite index of selected personnel statis- tics with an index movement that would corres- pond with the desired behavior change or im- proved performance of the managerial group as a result of their training Control and experimental groupings along with a series of related performance inquiry type ques- tionnaires, one year after the program to the participants and/or their peers, those they su- pervised and the participants' supervisors . l. 2. 5. 4. 5. 6. .7. 8. 9. 10. 45 Bibliography Allen, Louis A., "Evaluating a Management Deve10pment Program." Management Record, Vol. XVI, No. 7, July, 1954‘. Ppe 2 - . Andrews, Kenneth R., "Is Management Training Effective? 2. Measurement, Objectives and Policy.” Harvard Busi- ness6Review, Vol. XXXVII, No. 2, March-AprII, I957. pp. 5-72. Bolda, Robert A. and Lawshe, C. H., "The Use of Train- ing Case Responses in Management Training Evaluation." Educational app Ps cholo ical Measurement, Vol. XIX, No. 4, Winter, 1 5 . , Fleishman, Edwin A., Harris, Edwin F. and Burtt, Harold E., Leadership and Supervision ip_Industpy: 5p Evalu- ation of a Su erviso Traini Pro ram. Columbus, 195 OHIO: 5313 State niversity, . pp. xii, 110. Foreman Trainipg. Washington D. C. The Bureau of NatIonal Affa rs, Inc., January, 1952. p. 17. , GettipglReal Value from Exe utive Development Pro rams pp Universities. OpInion esearc orporation, I955. pp. 357-35 appendix. Goodacra, Daniel M. III, "The Experimental Evaluation of Management Training: Principles and Practice." Personnel, Vol. XXXIII, No. 6, May, 1959. pp. 554-558. Kirkpatrick, Donald L., "Evaluating Human Relations. Programs for Industrial Foreman and Supervisors." Madison, Wisconsin, Unpublished Doctoral Thesis Sub- mitted at the University of Wisconsin, 1954. p. 144. Lawshe, C. R., Bolda, Robert A. and Brune, R. L., "Stu- dies in Management Training Evaluation: II The Effects of Exposures to Role Playing." Journal of A lied Ps cholo , Vol. ILIII, NO. 5, October. I95 . pp. 287- 25%. , "Management Development Program." Flint, fiIchigan, Education and Training Department, AC Spark Plug Division of General Motors Corp., January, 1956. Unpublished mimeograph paper. pp. 14, appendices. .7 Irw- --.--|I~z'v' 11. 12. 15. 14. 15. 16. 17. 18. 19. 20. 21. 22. 44 Meadow, Arnold and Parnes, Sidney J., "Evaluation of Training in Creative Problem Solving." Jour a q£.A - I%%2%APB cholo , V01. ILIII, No. 5, June, . pp. Merrihue, Willard V. and Katzell, Raymond A., ”ERI - Yardstick of Employee Relations." H ard Business Re- view, V01. XXXIII, No. 6, Nov.-Dec._%5§', . _""pp. 91:99? Osborn, Alex F., A lied Ima ination, grinciples and Procedures 2; CraEII93—EpIEE;E§T—Efiaw or , harlas ScrIbner's Sons, I957. pp. ii, 579. Osterberg, Wesley and Lindbom, Theodore, ”Evaluating Human Relations Training for Supervisors." pgvanced Mppagement, Vol. XVIII, No. 9, Sept., 1955. pp. 28. , "Psychologists Give Management Quiz on Train- ing Methods." Busipesp Week, Aug. 16, 1952. pp. 106- 107 o ' "Research and Information - Evaluating the ResuIts of Management Development Activities." Manage- ment News, V01. XXVI, No. 5, April, 1955. p. 8. Rich, Joseph M., ”Measuring Supervisory Training: The Sociometric Approach." Perso el, Vol. XXIX, No. 1, Jan.-Feb., 1952. pp. 78-84. Savitt, Morris A., ”Is Management Training WOrthwhile?" Personnel, Vol. XXXIV, N0. 2, Sept.-Oct., 1957. pp. Savitt, Morris A., "The Retention of Management Train- ing." Journal 2; the American Societ p; Traigfing DirOCtOI‘S, 5° 0 11:1, figs 5, 361)., 90 ppe " 30 Seashore, Stanley E., The Trai 2; Le or for E;- fective Human Relations. er s, rance, n ted—National EducationaI Scientific and Cultural Organization, 1957. p. 45. Soik, Nile, "An Evaluation of a Human Relations Train- ing Program." Journal 9; the American Societ of Training Directors, 70 . XXII, No. 5, Marci, I958. pp. 54-49. Soik, Nile, "An Evaluation of a Human Relations Train- ing Program, A Follow Up To." JOpppal O' the American Society 9; Training Directors, 0 . , No. 7, JuIy, l 0 PPO 31" 50 25. 24. 25. 26. 45 Sorensen, Olav, "The Observed Changes Enquiry." Crotonville, New York, The General Electric 00., 1958. Unpublished mimeograph study. pp. 48, appendices. Stroud, Peggy V., "Evaluating a Human Relations Train- ing Program." Personnel, V01. XXXVI, No. 6, N0v.-Dac., 1959 0 pp 0 52-65 0 Trice, Harrison M., "A Methodology for Evaluating Con- ference Leadership Training." Industrial and Labor Re- lations Research, Vol. V, No. 2, a , 59. pp. 2-57 Viteles, Morris 8., ”Human Relations and the Humanities in the Education of Business Leaders: Evaluation of a PrOgram of Humanistic Studies for Executives." Person- ppl Ps cholo , Vol. XII, No. 1, Spring, 1959. pp. I-28. ‘. 1,“ r .. CHAPTER IV THE SURVEY AND FINDINGS In.this chapter a survey of the evaluation prac- tices in management education and development programs in selected United States companies will be described and the survey findings discussed. The purpose of the survey was twofold: first, to determine the evaluative practices and second, the factors and relationships that perhaps hinder or help the implementation of evaluation. The impetus for the survey and the benefits that might be derived were as follows: 1. The more that is known about evaluation in practice, the better industrial educators armed with theory should be able to suggest, revise and use sound and practical evaluation pro- cedures to promote more effective education. 2. The results should provide a comprehensive, up-to-date source of information on evaluation practices for those who plan and administer management training activities. 5. The knowledge of evaluation activities should throw light on existing conditions that may need change and improvement or that otherwise may go unnoticed. 4. The results can be used for comparison.with earlier or later studies, thus identifying trends and also providing a sound basis for action. ' 46 5.. \_‘a '.. ’l . ‘ ‘ 47 Development of the Questionnaire The questionnaire method was used to collect the survey data. Several authorities (5, 7, 8, 9, 15, 17, 20, 21) were consulted for the design and use of a question- naire to secure the data. Necessarily, the final revision of a questionnaire becomes somewhat of a compromise between the desire of the researcher to secure ample, accurate data and a questionnaire design which he believes the re- spondent will consider and return. The items for the questionnaire were derived mainly from the discussions of Chapter II, ”The Theoretical Aspects of Evaluation," and Chapter III, "Review of the Evaluative Studies in the Literature." The questions were designed for ease of response, and the questionnaire was divided into four parts for the respondents' convenience: Part I - YOur Company and Management Training Part II - Evaluation and YOur Comp Part III - Purposes and Methods of YOur Evaluation Part IV - YOur Company and You A preliminary form of the questionnaire with indi- vidually typed letters was sent to twenty-two people be- lieved to be authorities in evaluation because of their experience or publications in the evaluation field. This pilot group or jury was asked to appraise and comment upon thequestionnaire. Those jurymen holding positions in in- dustry were also asked to respond to the individual ques- tions in the instrument indicating their company use of 48 evaluation. The jurymen are listed in Appendix II. From the suggestions given by the jurymen, the revised four- page questionnaire, Appendix III, was evolved. Selection of Companies The selection of companies and persons who ulti- mately comprised the mailing list was taken from.the 1260 pgerioan Society 9; Training Directors' Directory of 5,095 members. It was believed these people had the interest and ‘were most likely to be aware of the evaluation.practices in.their company. In order to acquire a representative listing of industry and provide a mailing that was soon- omically practical in time and cost, only aircraft and mis- sile manufacturers, automotive manufacturers, chemical, plastic or drug manufacturing firms, electrical and elec- tronics companies, metal and steel manufacturers, paper producers, public utilities, rubber manufacturers and tax- tile production firms were included. This list produced 259 companies. The person who appeared to have the highest ranking title in the Directory for a company was selected to represent that company. The 259 companies were then compared with Fortune Magazine's 1960 listing of the 500 largest American indus- trial concerns and fifty largest utilities, size being measured in terms of dollar sales. This comparison 49 resulted in a list of 182 companies by excluding those not found on Fortppe's list. Further refinement to 158 com- panies was made with the decision that only one letter be sent to the central office of a large corporation rather than to several of its divisions or plants. For example, a questionnaire was sent to American Telephone and Tale- graph rather than each of the Bell Systems. Of these 158 companies and individuals' names, ten were selected to be part of the pilot or jury group. Thus the final mailing Vii‘ié’i ' list included 148 names as selected and refined from the Aperican Society pf Trainipg Directors' Directory. Distribution of Questionnaire Several efforts were made to induce response from those firms asked to participate in the study. An automa- tically typed letter, Appendix IV, with individually typed name and two questionnaires were sent to each of the se- lected companies. The extra OOpy of the questionnaire was enclosed for the respondent's file and reference. After two and one-half weeks or when the first wave of returns began to taper off, eighty-seven follow-up let- ters, Appendix V, were sent to the non-respondents. An- other copy of the questionnaire was included at this time. In both the first and second mailings, the Offer of a sum- :mery OOpy of the results of the study was used as further inducement. 50 Number of Responses One hundred nine survey-questionnaires out of 148 ‘were returned from the regular group, and nine out of the ten were returned from the industrialists represented in the pilot or jurymen group. Fifteen companies returned questionnaires with additional letters of explanation or ..~ ‘1 . attached cOpies of evaluative experiments which included their research techniques. This combined return rate of 74.7 per cent compares very favorably with other question- naire studies sent to similar people in business organiza- tions (1, 5, 6, 18, 19). Distribution of Responses As mentioned earlier in this chapter, the mailing list was structured and refined to include only the very large companies. Table 1 indicates the number of responses from the selected industrial groups. 51 Table 1 Distribution of Responses by Industry Groups W "w“? Number of . Per Cent of Industry Group Responses Total Aircraft and missiles 6 5.1% Automotive 15 11.0 Chemicals, plastics, drugs 15 12.7 Electrical and electronics 18 15.2 Metal and steel manufacture 17 14.4 Paper 6 5.1 Petroleum 14 11.9 Public Utility 4 5.4 Rubber 2 1.7 Textiles 2 1.7 General manufacturing 21 17.8 Total 118 100.0% An endeavor was made to prevent the respondents from generalizing their answers to a larger population than represented. The person answering was asked to list the number of employees for the chosen representation of his answers. Table 2 enumerates the size of industry by em- ployees and the organizational representation. Even though the largest firms (by sales) were contacted, the great ma- jority of representation is from firms or parts of firms ‘with 25,000 or less employees, and 41.5 per cent of the respondents represent organizations or parts of organiza- tions of less than 10,000 employees. 52 Table 2 Distribution of Responses by Size of Company Response Generally Representative of: Number of No. of % of Employees Single Div., Corp. Re- Total Plant, Part as a sponses Non- of a Whole by Size Affiliate Corp. of a Corp. Less than 10,000 7 21 21 49 41.5% 10,000 - 25,000 12 27 59 55.0 25,000 - 50,000 2 12 14 11.9 50,000 - 100,000 9 9 7.7 100,000 - 200,000 4 4 5.4 200,000 and over 5 5 2.5 Total 7 55 76 118 100.0% Nature of Management Training Activities in the Selected Companies Although the primary concern of the survey was to secure information concerning the practice of evaluation, it is useful to note the nature of the management training programs and the relationship, if any, to evaluation. A question was included in the survey which requested each company to indicate, if possible, the approximate time it ‘Was engaged in each of the designated levels of management training . Table 5 gives a summary picture of the extent of the training at the various levels of management in the h' | 55 Table 5 Average Per Cent‘ of Time Management Training Departments Engage in Various Levels of Training W Level of Training Per Cent of Time Pre-supervisory training 10.0% Foreman or other lst level 40.2 Middle management training 51.5 Top management training 8.6 Other 9-9 Total 100.0% Based on survey question No. 5. responding companies. It could be expected that the major- ity of time would be spent in foreman or other first level training and also at the middle management level. Natur- ally supervision is more numerous at these levels. Also, there would be less training at the pre-supervisory level and at the tOp management level, because there are fewer persons in preparation for supervision and in tOp manage- ment in proportion to the central two levels. Other train- ing, 9.9 per cent, includes incidental programs the management training departments are called upon to give, e.g., special courses for engineers, technical pe0p1e and secretarial programs. Individual analysis of questions NO. 5 and No. 14 on the returned questionnaires gave no support to the thought that firms heavily engaged in, for example, fore- man or other first level training spend more effort in 54 evaluation of these programs than those heavily engaged in middle management training. Evaluation efforts were some- what in proportion to the training at the other levels also. Tables 4, 5 and 6 record the company activity with respect to the writing and development of training programs, the people or organizations conducting these programs and the location of the management courses or programs. Table 4 Writing and Development of Management Training Programs . No. of Times Per Cent of Written or Developed by Mentioned Total Mentions Outside consultants 59 21.0% College or university peOple 59 21.0 Own staff 95 51.0 Other 15 7.0 Total 186 100.0% Table 5 Instruction of Management Training Programs . 1. -...W-—._.. . No. of Times Per Cent of Conducted by Mentioned Total Mentions Outside consultants 55 16.9% College or university people 51 24.7 Own staff 92 44.4 Other 29 14.0 Total 207 100.0% 55 Table 6 Location of Management Training Programs No. of Times Per Cent of Conducted at Mentioned Total Mentions A college or university 48 25.0% American Management Associ- ation 47 22.5 Own plant or within company 97 46.4 Other 17 8.1 Total 209 100.0% It is shown that a surprisingly large percentage of companies mention that a portion of their management train- ing is either written or developed by outsiders, conducted by outsiders and done outside the plant or company. The other categories showed a number of the training programs written or developed by people from special fields or functions, sometimes supervisory or technical personnel outside the training department, yet within the company. These same people sometimes conduct the programs. Additional mentions were vendors and programs conducted by local management clubs. Management club sites, local hotels, resort type or special conference sites owned or rented, in addition to professional society and other training organi- zation facilities, were mentioned as other places where management training programs are Conducted. - am" we: ‘ “ 56 Because of the limitations in wording of questions 4, 5 and 6, it was not possible to perform an overall com- parison of programs develOped or conducted by outsiders, or programs given outside the company, and the amount of evaluation in relation to these factors. It was found only that most companies relied on a number of sources in ‘ rm" programing their training. Individual analysis of the ques- tionnaires, however, seemed to indicate that the responding industries do not spend more effort, in terms of time or Vat)“: ‘1'“... ' money, in evaluating programs developed or conducted by outsiders or in evaluating programs given outside the com- pany as cOmpared to the work done by their own staff. Industry Groups - Time and Budget Spent on Evaluation The type and size of companies reporting and the nature of their management training activities is useful for a preliminary understanding of the selected companies. But a more primary concern of the survey was to ascertain the present effort put forth by these companies toward the evaluation of management training. It was believed this evaluative effort could be partially determined by asking (a) a question pertaining to the per cent of time and (b) a question concerning the per cent of budget spent on evaluation in the management training department of each 57 contacted company. While these questions were closely a1- 1ied and one would expect the response to be somewhat re- lated, it was hoped that together the questions would secure more complete data. Table 7, than, points out the effort on evaluation in terms Of training department time by the industry grouping. Table 7 Industry Group and Per Cent of Time Spent on Evaluation Less Industry Group None Than 1-5%' 6-10% Over 1% 10% Aircraft and missiles l l 1 1 Automotive 2 5 1 2 Chemicals, plastics, drugs 2 10 Electrical and electronics 4 10 l 1 Metal and steel manufacture 5 5 4 1 Paper 4 1 Petroleum 4 4 2 Public Utility 5 1 Rubber 1 1 Textiles 2 General manufacturing 2 12 5 1 Total (95)‘ 1 18 57 16 5 *————t Number responding to the time spent on evaluation question, No. 7. From the data in Table 7, it appears that most com- panies spend some time, if even less than one per cent, on evaluation. Slightly more than half, or fifty-seven out of the ninety-five companies responding to this question, ‘3.)5“ 58 spend approximately 1-5 per cent of the management training department's time on evaluation of formal management edu- cation and development programs. Little or no relationship, however, appears to exist between the type of industry and the amount of time spent on training evaluation. The only exception might be the metal and steel manufacturing group where four out of the thirteen companies spend 6-10 per cent, and one company in this group spends over ten per cent of its time on evalu- ation. But there is no readily apparent reason for this one possible exception. Table 8 details the other related question concern- ing the per cent of the training department's budget spent on evaluation of its activities. Table 8 Industry Group and Per Cent of Budget Spent on Evaluation W Less Industry Group None Than 1-5% 6-10% Over 1% 10% Aircraft and missiles l 2 1 Automotive 1 4 2 Chemicals, plastics, drugs 7 5 Electrical and electronics 7 8 1 Metal and steel manufacture l 7 5 1 Paper 5 Petroleum 7 2 Public Utility 1 1 1 1 Rubber 1 l Textiles l 1 General manufacturing 1 6 9 1 1 Total (95)’ 6 42 59 4 2 Number responding to the budget spent on evaluation question, No. 8. 59 In this case the majority do indicate they spent a part of their budget, how ever small, on evaluation. Nearly half indicate that it is less than one per cent. In the matter of the evaluation budget, there seems no relationship whatsoever between proportion of budget spent and a particular industry group. One aircraft and missile manufacturer justified their one per cent or less on evaluation in their budget by saying that most evaluation was useless in their opinion and "performance of peOple in present jobs" and "availability of people for promotion" were their main tools of evaluation. One of the questionnaire responses might be cited which reveals the dollars that can be spent on evaluation, even though a small part of the management training bud- get. A large electrical and electronics firm replied that they currently spend $40,000,000 yearly on education and training and indicated they spend over ten per cent of their budget on evaluation. Therefore, in this instance, a conservative estimate would be about $400,000 devoted to evaluation activities. Size of Staff - Time and Budget Spent on Evaluation In Tables 9 and 10, the time spent on evaluation and the budget spent on evaluation questions are compared with 3"“, muse- . - ‘- 60 the size of training staff. The total companies in each per cent column of Tables 9 and 10 would necessarily be the same as in the two previous tables. Table 9 Training Staff and Per Cent of Time Spent on Evaluation Less Total Size of None Than l-5%1 6-10% Over by Training Staff 1% 10%; Size None 1-5 peOple l 17 4O 8 2 68 6-10 1 10 5 14 11-20 5 l 4 21-50 2 2 Over 50 2 4 1 7 Total by Per Cent 1 18 57 16 5 95 FT It is readily evident in Table 9 that the majority (sixty-eight) of the firms responding in the survey have one to five peOple on their training staff. Forty or nearly half of all companies represented fall into the 1-5 per cent of time spent on evaluation with 1-5 peOple on their respective staffs. There appears to be a tendency of increased evalu- ation effort, insofar as time spent, as the size of staff grows. Nearly all firms with over five training people spend at least one per cent of the department's time on evaluation, and there is a larger proportion from each Vial-I. ' 61 size grouping in the six per cent or over categories as the size of the staff increases. Additionally, the data revealed in Table 10 picture a tendency for the larger training staffs to spend a greater prOportion of their budget for evaluation when compared with the smaller staffs. About two-thirds of the companies which have six people or over on the training staff spend one per cent or more of the budget on evaluation. Con- versely, more than half the companies which have five or less on the training staff say they spend less than one per cent of their budget on evaluation. Six companies in this group spend none of their budget on evaluation. Table 10 Training Staff and Per Cent of Budget Spent on Evaluation ' Less Size of None Than 1-5% 6-10% Over Training Staff 1% 10% None 1-5 peOple 6 54 28 l 1 6-10 5 6 1 11-20 1 5 21-50 1 1 Over 50 1 1 1 Total by Per Cent 6 42 59 4 ‘2 62 Years of Operation - Time and Budget spent on Evaluation In still another attempt to reveal some of the fac- tors influencing the use Of evaluation, it was thought that perhaps experience or years of Operation of the management training department may have some bearing upon the evalu- ative effort as revealed by the time and budget questions. As Tables 11 and 12 show, there appears to be a relation- ship between years of Operation and time or money spent on evaluation of training. Table 11 Years of Operation and Per Cent of Time Spent on Evaluation W Less Total Number of Years None Than 1-5% 6-10% Over by 1% 10% Years Less than 1 year 1 1 1-5 years 1 4 l5 2 20 6-10 7 25 4 1 57 11-15 5 9 6 1 19 16-20 2 5 2 9 Over 20 years 1 5 2 l 9 Total by Per Cent 1 18 57 16 5 95 Twelve of the nineteen training departments which spend six per cent or over of their time on evaluation have been in training over ten years. 65 Table 12, which follows, shows that five of the six training departments which spend six per cent or over of their budget on evaluation have been in training over ten years. Table 12 Years of Operation and Per Cent of Budget Spent on Evaluation a . Less Number of Years None Than 1-5% 6-10%» Over 1% 10% Less than 1 year 1-5 years 1 ll 8 6-10 4 14 18 1 11-15 1 8 9 2 16-20 5 2 1 Over 20 years 4 2 l 1 Total 6 42 59 4 2 TOp Management Stress on Evaluation-Time and Budget Spent on Evaluation In this final set of tables the effort, in terms of time and budget spent on evaluation, is compared with the extent top management stresses evaluation of management training programs. As Table 15 discloses, there does seem to be a tendency to Spend more time on evaluation as top management emphasizes evaluation and results of training programs. It should be noted that about two-thirds of the re- spondents report that their top management stresses 1'.“ v a Table 15 TOp Management Stress on Evaluation and Per Cent of Time Spent on Evaluation TOp Management Less Total Stresses None Than 1-5% 6-10% Over by Evaluation 1% 10% Extent To no extent 1 5 1 1 6 To little extent , 7 l5 5 25 To some extent 7 57 10 - 54 To a great extent 1 4 2 5 10 Total by Per Cent 1 18 57 16 5 95 evaluation to some extent or to a great extent. The other one-third say that none or little stress is placed on evalu- ation of their programs by tOp management. Fifteen out of the nineteen which spend six per cent or over time on evalu- ation have a top management which emphasizes evaluation to some extent or to a great extent, and the three companies which Spend over ten per cent of their training depart- ment's time on evaluation all have evaluation stressed to a great extent by their top management. In Table 14 there is less tendency to spend money on evaluation in relationship to the stress placed on evaluation by tOp management. But there does, neverthe- less, appear some relationship between budget and the stress placed on evaluation by the companies in general. The six training departments which spend six per cent or over of their budget on evaluation have top managements \iia ‘ 65 which place importance on evaluation either to some extent or to a great extent. The two training departments which spend over ten per cent of their budget on evaluation point out their top management stresses evaluation to a great ex- tent. Table 14 Top Management Stress on Evaluation and Per Cent of Budget Spent on Evaluation TOp Management Less Stresses None Than 1-5% 6-10% Over Evaluation 1% 10% To no extent '2 2 2 To little extent 2 l5 8 To some extent 2 24 24 2 To a great extent 1 5 2 2 Total by Per Cent 6 42 59 4 2 Company Plans for Future Evaluation Effort No studies were located in the review of literature which attempted to determine specifically the level of management training evaluation effort in time or budget. The data from the previous tables do, however, give some indication of current effort in training evaluation. In brief review: 1. Slightly over half the companies spend 1-5 per cent of their time on evaluation of their man- agement training courses or programs. About 66 one-fifth say they spend less than one per cent and another one-fifth indicate they spend 6-10 per cent of their time on evaluation. 2. Nearly all companies, with a few exceptions, re- veal that they spend five per cent or less of their budget on evaluation of their programs. Since no studies could be found which previously in- vestigated the evaluation effort in management training, it is not possible to conclude whether the training profession is devoting more effort to evaluation than in the past. It can only be surmised that as the training profession has grown over the years, more evaluation theory and tech- nique have been developed and used. The survey, however, endeavored to ascertain if training peOple were satisfied with their present effort and if in the future they planned to give greater impor- tance to and spend more time on the evaluation of their training activities. Table 15 presents their answers to this question. Table 15 Future Effort and Time to be Spent on Evaluation ' Number of Per Cent of Future Effort and Time Mentions Total Mentions Less 2 ' I 1.9% About the same 46 43.0 More 44 41.1 A great deal more 15 14.0 Total 107 100.0% I Ski-M" 67 From the data collected, 41.1 per cent indicate that they are going to spend more time or give greater effort to evaluation activities and fourteen per cant say they will spend a great deal more time. Thus 55.1 per cent reveal that they are going to spend more or a great deal more time on evaluation. Forty-three per cant say they are llmql' going to spend about the same amount of time on evaluation. From the companies responding, then, it appears that at least a static position will be maintained and perhaps' V; V e -A~. there will be even a slight increase in evaluative effort ’ in the future. It is noteworthy that thirty of the fifty-nine com- panies who said they are going to do more or a great deal more in evaluation are already spending 1-5 per cent of their time on evaluation. Ten of the fifty-nine are de- voting 6-10 per cent of their time, and another three are already spending over ten per cent of their time on evalu- ation. So those who are spending less than the median time on evaluation are not necessarily those which will exert greater effort and spend time on evaluation in the future. Many of those companies devoting the greater effort in time and budget to evaluation already will actually increase their evaluation activities in the future. Another relationship found was.that twelve of the fifteen respondents who said they were going to do a great deal more in the future with evaluation also had their tOp management stressing evaluation to some extent or to a great extent. It is quite possible then, that top manage- ment stress upon evaluation does result in somewhat greater intentions of future effort and time to be spent on evalu- ation of management training activities. Deterrents to Effective Evaluation In view of the current effort that is being put forth in evaluation of management education and develop- ment programs, and in light of future plans to have per- haps a slight increase in this area of training activity, it is interesting to note the respondents' answers to what they believe is generally the greatest deterrent to effective evaluation of management training in their com- pany. Each respondent was requested to check what he thought in his experience was the one greatest deterrent. Their answers are recorded in Table 16. Since several were unable to choose just one item, their two responses were then included. This would not, however, appreciably change the relative relationship of the data in the table. The data seem to show that by far the major deter- rent to effective evaluation in the opinion of the re- spondents is that evaluation research techniques are dif- ficult to apply in productive or operating situations. M7 Table 16 69 Greatest Deterrent to Effective Evaluation of Management Training m Greatest Deterrent to No. of Per Cent of Effective Evaluation Mentions Total Mentions Lack of time 21 16.5% 5 Lack of finances 4 5.0 5. Lack of interest 6.5 Lack of knowledge of evaluation research techniques 14 11.2 _1 Evaluation research techniques V difficult to apply in productive or Operating situations 55 41.7 Top management does not stress evaluation of management train- ing 10 7.9 Top management not interested in findings when evaluation.per- formed 2 1.6 Other 15 11.8 Total 127 100.0% All other deterrents fall behind in importance to this de- terrent as indicated by the training people included in this survey. Looking at the data in a reverse way, it seems that generally there is interest in evaluation; there are fi- nances and time; there is knOwledge of evaluation research techniques, at least in the eyes of the respondents; top management stresses evaluation and is interested in the 70 findings when evaluation is performed. None of these ap- pear to be outstanding deterrents when compared with the difficulty of applying evaluation research techniques to productive or Operating situations. The two companies, a metal firm and a petroleum company, which checked "tOp management not interested in findings when evaluation 1.35.10” performed," also indicated tOp management did not stress evaluation but that they were going to give more effort to evaluation in the future. This question on deterrents to evaluation evoked more written comment than any other in the survey. One person listed as a deterrent to effective evaluation the lack of demonstrable value of many of the supposed tech- niques of evaluation. Several others went on to indicate their belief that there were few techniques which can ac- curately measure the non-tangible nature of the results of a management training program. Still others said it was difficult to isolate the effects of training in a total organizational picture. Another maintained that variables which are difficult to control make scientifically valid evaluation impossible. Most of these comments, however, were from companies that indicated they did perform some evaluation even with these inadequacies. Two similar comments came from a public utility and a general manufacturing firm listed as deterrents to effective evaluation in their companies. Both comments 71 ring with a seeming futility of evaluation and no evalu- ation attempts were reported. We have read and studied almost everything avail- . able on the subject, and techniques even closely applicable are not good enough to justify time and effort. We simply do not know of any technique whereby we can find an accurate evaluation at a reason- able cost. So, at least for the present, our evaluation is based pretty much on 'seat of the pants' judgment. A final comment on deterrents to evaluation came from a respondent who said, Too many people in personnel and training sec- tions are afraid of what an effective evaluation would show. Focus of Evaluation At this point in the building of the questionnaire it became necessary to strike a compromise. There would probably be great amounts of data that possibly could be collected on the varied attempts of evaluation on a myriad of management training courses or programs in the selected companies. But in the interest of time on the part of the respondent, the questionnaire was tailored to secure data on a particular program that was representative in each company. In this case, the respondents were asked to list one formal course or program which was evaluated and was the best example of their company's use of evaluation ' 1“}?- 4319' 72 methods. Then the respondents were asked in related ques- tions to describe the focus of their evaluation and also the evaluative techniques used for this particular train- ing program. The result of such an inquiry somewhat precludes that the total picture reported of industry management " .‘ 994°C“! training evaluation will be the best picture available. But while individual training departments may not always focus their evaluation so broadly or utilize such elabor- ate or extensive evaluative techniques, the data reveal 9 what has been done or perhaps could be done again. Thus the data seem indicative Of the currently available and usable evaluative techniques. The complete list of programs or courses which were referred to by the companies as their best example of the use of evaluation methods is included in Appendix VI. Generally three types of prOgrams are most often mentioned: (a) courses having management in their title, (b) that of supervisor and (c) work simplification or methods improve- ment. Most of these programs were given at the foreman, first and/or middle management level, with no seeming con- centration of evaluation of programs given in any one grouping. It should be noted that the "human relations" type of course most often found evaluated in the review of lit- erature, Chapter III, is only mentioned specifically three 75 times by company respondents, and this is not a major men- tion in light of the many other more frequently mentioned courses that were evaluated. "Human relations," however, could very well have been a sub-tapic of several of the more general management courses listed. Twenty-nine companies or about one-fourth stated they have no best or worthy example of their evaluation of a prOgram. Since the questionnaire did not probe speci- fically as to the "why" in these cases, some of the reasons can only be surmised. Several did indicate that their top management did not stress evaluation and two mentioned the lack of adequate techniques. One said the cost of accurate evaluation was prohibitive. Several stated that since they had no formal management training courses or programs, they could perform no evaluation. But even though twenty-nine companies reported no best example of their evaluation, most indicated some time and budget spent on evaluation. So the conclusion must not be drawn that one-quarter of the companies do not perform evaluation, only that from the way the question was worded, one-quarter of the respondents deemed no evaluation worthy of reporting at this time. With these varied types of management training pro- grams given as best example of the company's evaluation, the next related question had to do with the focus of evalu- ation in each instance. Here the question was structured to include those focuses of evaluation as suggested .2!» '.. M 74 by Kirkpatrick (10:5, 11:21, 12:15, 15:28), Besco, Tiffin and King (5:18), Messer (16:56-57), Borosage (4:2-6) and Korb (14:99-104) as outlined in Chapter II on the theoreti- cal aspects of evaluation. In other words, if these focuses are deemed impor- tant by those who set forth the principles of effective evaluation, it should be revealing to see the focuses of evaluation as reported by the companies. It should be kept in mind that a respondent could check one or more statements of focus of evaluation, and nearly all indus- tries indicated a multi-focus approach in their evaluation. Table 17 indicates the response under each of the items listed. The highest number of mentions, the feeling of par- ticipants about the training program, tends to agree with Kirkpatrick's (10:4) thinking that reaction or feeling about a training program is most easy to measure, and nearly all training directors do it. Kirkpatrick (15:28) also believes that feeling, learning, behavior and results, the last four focuses of evaluation in Table 17, have an ascending order of difficulty. While the difficulty of evaluation may be greater in the latter focuses listed, it appears that there are still many attempts in these areas of evaluation as shown by the returns. It may be that in- dustry practices more difficult evaluation than is indi- cated by the published articles in professional journals. Table 17 75 Focus of Evaluation in Companies' Example of Their Use of Evaluation Methods W Per Cent Focus of Evaluation No. of of Total Mentions Mentioned ADMINISTRATIVE ARRANGEMENT under which the training program given 50 8.2% GOALS AND OBJECTIVES of the training program 44 12.1 METHOD OF PRESENTATION of the training program (conference techniques, con- ference leaders, etc.) 50 15.7 PARTICIPANTS' PROGRESS during the training prOgram 58 10.4 FEELING OF PARTICIPANTS about the training program 62 17.0 LEARNING OF PARTICIPANTS as a result of the training program (certain management principles, supervisory knowledge, etc.) ' 55 15.1 BEHAVIOR OF PARTICIPANTS as a result of the training prOgram (different actions back on the job resulting from the program) 45 12.5 RESULTS, EFFECT, IMPACT on the group, intergroup relationships or organi- zation as a result of the training program 41 11.2 Total 100.0% 565 In theory, Borosage (4:2) states, "Evaluative con- siderations involve a variety of facets each providing evi- dence to determine the extent to which a training program ~ IJNW 76 is meeting its goals. All of these must be considered rather than a fragmented approach." In practice, it ap- pears from the data resulting in Table 17 that training people are generally doing a comprehensive job of evalu- ation and probing its various "facets" by a multi-focus ap- proach; An analysis of individual company returns on this question seems to indicate small industries, in size of employment or training staff, or industries in the man- agement training field only a few years, are as active in a multi-focus approach to evaluation as are the larger in- dustries or those having management training over a longer period of time. Methods and Systems of Evaluation .In Chapter II, the writings of a number of authors (5, 4, 10, ll, l2, 15, 14, 16) suggested a variety of methods for evaluative assessment. These methods were combined into a workable question and their use asked of the selected companies for this study. Here, also, the respondents were requested to check those methods of evaluation which were used in their best example of evalu- ation of a management training program. Attempts were made in the construction of the question to give the re- spondent recognizable methods of evaluation yet combining those methods which seemed related. Table 18 reveals the total number of responses per method of evaluation. Table 18 77 Methods of Evaluation in Companies' Example of Their Use of Evaluation Per Cent No. of of total Method of Evaluation Responses Responses Course material information tests: Before program 25 4.%% During program 21 4.5 Directly after program 52 6.6 Sometime after program 7 1.4 Attendance at program 50 6.2 Amount of participation by partici- pants 40 802 Interviews with, or questionnaires to: Participants 65 12.9 Participants' supervisors 54 7.0 Participants' employees 7 1.4 Participants' peers 4 .8 Check of users' attitude toward service or product rendered by participants' department ‘ 5 .6 Use of participant questionnaires on: Changed attitudes 18 5.7 Feelings about program 49 10.0 Benefits from program 45 9.2 Control and experimental groupings 10 2.0 Reports by outside observers 20 4.1 Direct observation of participants' actions on job after program 50 6.2 Organization attitude or communica- tions surveys 7 1.4 78 Table 18--Continued m Per Cent No. of of Total Method of Evaluation Responses Responses Use of records on absenteeism, ma- terial waste, turnover, grievances increased production, cost reduc- tion, or safety in the participants' work group , 17 5.5 Significant increases in "perform- ance review" ratings of the par- ticipants by their supervisors 14 2.9 Others 14 2.9 Total 488 100.0% Several possible observations can be drawn from the data. First of all, there appears a relationship between the high 12.9 per cent on interviews with, or question- naires to participants and that of the use of question- naires to ascertain the feelings about or benefits from a program, 10.0 and 9.2 per cent respectively. Course material information tests seem well used in comparison to the other possible evaluation methods. A sizeable number use tests before, during and after their programs. A smaller percentage, however, test their train- ees sometime after the program to determine lasting effects of their training. The method of testing sometime after the program may be most meaningful in terms of real re- sults of a program. Nevertheless, as is emphasized by "Wm” 79 Table 16, this may be a research technique which is diffi- cult to apply in productive or operating situations in in- dustry. A check of users' attitude toward service or pro- duct rendered by the participant's department is perhaps a good measure of effectiveness of a program according to the principles set forth in Chapter II. On the other hand, the method is least used by the respondents. This, too, may be due to indications that as an evaluation technique, it is perhaps difficult to apply in an operating situation. Control and experimental grouping, often found in the review of literature as part of an evaluation design, is one of the lesser used techniques by the bulk of prac- titioners responding in this study. The difficulty of use of this evaluative technique in productive situations, how- ever, was generally recognized throughout the review of literature. The methods of evaluation given under the "other” category included: promotions as a result of the train- ing; discussions with instructors or conference leaders; one chemical company has set up a tOp management evalu- ation committee; and a metal firm gives actual on-the- job problems which incorporate the training material given during their management course. Several companies specifically mention results in terms of dollars saved as a method of evaluation of their programs. Two companies - ' 3351“!” stress what they term a "climate of acceptance" by top management, that is, through informal interviews or chance meetings with top management, feedback is given on the believed results of the training programs. One of the respondents says that in his situation, "Oral feedback from tOp management is of more value than sta- . tistical analysis." Another says, "Basically, I feel if the responsible management now requesting the program feels the needs he wanted met are met, the program was success- ful." Now, to carry the analysis of evaluative methods a step further, an attempt was made to discbver what systems of evaluation might be in use in the various companies. Table 19 was developed to ascertain which evaluative methods from Table 18 might be used singularly or in com- bination to assess the various focuses of evaluation as described in Table 17. Any observations drawn from the data of Table 19 should be prefaced by two limitations. First, many of the respondent companies indicated a multi-focus approach to evaluation and also gave a sizeable number of evaluation methods to assess the focuses of their evaluation. It was not possible, in all cases, to match focus with method or determine the system used. In these instances only the determinable data were included in the table. Secondly, the data of Table 19 give only a summary picture. The data ' 5,-. 0-“, 81 a xx -n4 thzfm ma 0 mm ma OH #H HH ma N\ usespasnod .mpqsm llfloapnsn he consumes 9050 loan 90 cowbeom unseen ouspfipps .oeoms Ho uoono mason .mpflsmaowpnsm moo Iboaase .mpssaaOHuasm shoes» Inseam .opasmfiospusm - eeseaaosunsm “an mosasqdoaumesu 90 .mpr meowbnopsH mpqsmfiowpnsm hp soapsmaoapnsm no undoad asnwonm as eonsunoppd summons nouns oaapoaom asamonm norms hapoenan asnwoam wqwnsn assmonm onomom «memo» mosses lacuna Hesnopss omnsoo posgfiH .rooess .mpasmom spasm Ifioapnsm no Hoabsmoq spasm lacunae HO wstHso sandman IHpHsm HO wswaoom dowpsdasbm no msoom mmosmo .mpqsm neoeree seeps» Isomong Ho coupe—m mam poms nomads ¥ . dds—6 sosussasas so scenes dowpssasbm Ho maopmhm ¢H cane? 82 NH HH 32.x .. . OH ma om macaw M903 .opnsm awospasm on» as smooch romeo so .aoaposdon pmoo .qosposuonn commences . .eoossboflnm .aoboqnnp .opmsw Hsdnopsa .amwoop lemmas no monooon Ho omb cheeses odomedowssaaoo no odouwpps dowpsuansmno aanmonm nouns now no escapes .mpqsmHOflpHsm Ho soapsbnompo pooafln mno>uem Inc crampso hp monomom mwowmdosm aspnosasemxo one Hospaoo summons Eon“ mpwmonom summonm props mwnfiaomm repossess somseso “do commandos» Imofiw pnsmaowpnsm no em: posmaH .poosss .mpasmom spasm twowpswm Ho H0fi>wnmm spasm twowpasm no masonsoa mpqsmwo Ifipnsm Ho nonwonm .munsm msaaoos scanned scares Iqomonm Ho scares mobflp noenoo use masow uses nomqsnnq .sessq soapsdasbm no osoom doquHGOOIImd manna soarssaebe so posses 85 in... I .. :ougi OH muomflb IQTQSm Hflmap hp apnea IHOHprm on» mo mmnflpmh zsoflbon ecumenounomz as mmsosoqfl undefimfismflm mused mpsmm mpqsmflo nowpmp mobs» pommaH IflOHpsmm IHOflpHmm lapswm mmenwohm Isomonm loonno puma .poommm Ho mo mo .mpqsm no use Immqesu4 sowpmdam>m mo posses .mpadmom 90H>snom quQHMmq wsHHoom IHoapHsm douse: meow .qfiad4 sowpwsHm>m Ho mooom coranHOOIImH magma 84 take on greater meaning in light Of some of the individual questionnaire returns and attached letters of explanation. With this preface the following observations can be made from Table 19, from the questionnaires and related infor- mation supplied by several of the companies. It is apparent from Table 19 that course material information tests are the most useful, in the eyes of the respondents, to measure the participants' progress during the training program, and to measure the learning of par- ticipants on a before, after and sometime after the pro- gram basis. The learning of participants as a result of the training program is also frequently evaluated by means of interviews or questionnaires to the participants, but less to the participants' supervisor--a method which may have more meaning than its use would seem to indicate. It became evident that when evaluation of learning ‘was the objective of the evaluation, frequently the program title indicated the course was more concerned with princi- ‘ples than being phiIOSOphical. Also, most of the attempts to use control and experimental groupings apparently are ‘performed when the evaluation involves the learning of participants. There seems to be reliance upon the attendance at a program, the amount of participation, and interviews or questionnaires as evaluative techniques when evaluating the method of presentation. 0n the other hand, an ' - TERI." 85 evaluation of the administrative arrangement or the goals and objectives of a training program appears less attempted, at least through the formal evaluation methods as listed. The companies reported that evaluation in these areas is more often done by informal means than the formal ones as specified in the survey-questionnaire. Insofar as the feeling of participants about a pro- gram, there is some indication that a number of companies believe that participation by the conferees is at least ' one expression of this. Of course the high figure in Table 19 is the thirty companies which used a questionnaire to determine feelings about the program. This is a rela- tively inexpensive, easy to administer measurement with limited, yet valuable, information according to Kirkpatrick (10). To ascertain the behavior of participants or the re- sults and effect on the organization, (both focuses may be. closely related), a variety of methods have been used by the respondent companies. According to Kirkpatrick (l2, 15), these are the more difficult areas to evaluate; how- ever, there were several reported attempts with methods of potential validity. The observation of participants' action on the job after the program, organizational attitude sur- veys, the use of department personnel or production statis- tics and the use of "performance review" ratings were men- tioned by a few of the companies as brought out in Table 19. 86 Probably the most elaborate example of an evalu— ation program measuring the results or effect upon the organization was turned in by a large, but decentralized chemical company. In this case the plant manager at one of their locations, with about 400 men in his management group, set forth seven organization goals for improved operations. It was determined that training in problem solving and communications would help reach their goals. iEight line people were selected to carry out the training as develOped by the plant manager, his staff and the edu- cation staff. Briefly these were some of the goals and reported :results. The evaluation method was simple personnel and 3pr0duction statistics. 1. Reduced lost time injuries--accident frequency rate reduced from 1.7 to O in four years. Minor injuries reduced in the same period from 1,200 to 485. 2. Improved quality of product--rejects reduced by twenty-seven per cent. 5. Improved cost performance--operating costs down ten per cent. It was made evident by the respondent that the training, while perhaps not entirely responsible for all of the improvement in Operations, at least played a major role helping the organization to reach its goals. Several other firms reported various statistics as a technique of evaluation for impact upon the organization. Five companies that held a methods improvement program used 87 their records of dollars saved as a result of improvements as the basis for their evaluation. An unusual statistic for evaluation was used by a petroleum company. After a course in "discussion leading" they actually out down the number of plant meetings. A metal and steel manufacturing firm structured, ‘with the participants' supervisors, on-the-job problems to discover the behavioral change and increased learning re- sulting from a "management principles" course. These ,problems included the principles of the conference session ‘which could be applied directly under the eyes of the su- ,pervisor. A furniture manufacturing firm performed a simi- lar on-the-job evaluation using the principles of a quality control training program as applied to day-to-day Opera- tional quality problems. Another petroleum firm put on a course entitled "counseling" for their first through middle management levels. Their evaluation involved one focus--that of be- havior of participants as a result of the training program. Their basic evaluation technique was indication of any in- creases in "performance review ratings.” Their choice of supporting techniques, however, indicated substantial ef- fort and thought. They not only used before and after rat- 11133 but control and experimental groupings as well to help validate any changes that might result from the training IHPOSrame -' ~ swam 88 A number of companies returned examples of their questionnaires. One electrical and electronics firm ex- hibited a questionnaire for assessing progress during a training program. Most of the questionnaires returned as evaluation examples, however, had roughly three parts or levels: 1. Questions about the method of presentation 2. Questions concerned with the feeling of partici- pants about the training program, e.g., ques- tions regarding their preference of parts Of a program 5. Questions about change in behavior resulting from the program Although the questions in the third category are the opinions of the participants and perhaps biased, they are still revealing to the evaluator. In the eyes of the com- ;panies these questions are better than no evaluation at all. The same electrical and electronics firm questioned, "Has this program helped you do a better job in your present po- sition? If so, how?" A chemical plant used, "Has the pro- gram helped you in carrying out your present or future job duties? In what way?" A utility asked, "Of the following subjects, how effectively have you been able to apply them to your work?" and "Of the following subjects, to what de- gree has each been of assistance to you in dealing with your fellow employees?" Finally, Table 20 is the last table of this chapter and presents a matter directly related to the evaluative method and system. It will be recalled from Chapter II ' ‘N 'ua' 89 that Belman and Remmers (2:51-52) suggest that provision for evaluation should be made during the planning stages of training prOgrams. They cautioned that evaluation should not be thought of as an appendage at the end of a program. The question asked by the present researcher sought to gain an indication of industry practice in this matter. Table 20' Planning for Evaluation Planning and No. of Per Cent of Evaluation Performed Mentions Total Mentions Before a program starts 55 f 26.8%7 During a program 49 25.9 Directly after the prOgram 55 26.8 Sometime after the program 44 21.5 Other 2 1.0 Total 205 , 100.0% Since there were 118 respondents to the survey, the data suggest that about half the practitioners are aware that effective evaluation is a continuous process. These companies indicate they do plan for and perform evaluation of some focus and with some method regularly before, during, directly after and some time after their management train- ing programs. The two checking the other category men- tioned they used a once-a-year, overall method of evalu- ation. 90 Summary of Findings 1. A questionnaire survey was sent to 158 companies to as- certain the practice of evaluation in management educa- tion and develOpment programs. One hundred eighteen or 74.7 per cent of the questionnaires were returned. The survey probably is representative of the major evalu- ation activities in management training in large United States companies. The 118 firms which responded to this survey spend the majority of their time (71.5 per cent) engaged in fore— man or other first and middle level management training prOgrams. An overall picture reveals that much of the management training at these companies is done by their own staff. The rest of the management training is de- velOped and programed by outside consultants or college and university peOple and frequently is conducted away from the plant or outside the company. About two—thirds of the companies responding have 1-5 peOple on their management training staff. Years of operation vary greatly. Approximately one-fifth have operated a management training department for 1-5 years, twomfifths of the companies indicate a manage- ment training department for 6-10 years, another one— fifth for 11-15 years, one-tenth for 16—20 years and one-tenth have been in management training over twenty years. 91 5. Slightly over half the training departments spend 1-5 4. 5. per cent of their ting on evaluation of their manage- ment training courses or programs. About one-fifth in- dicate they spend less than one per cent and another one—fifth indicate they spend 6-10 per cent of their time on evaluation. Nearly all companies responding, with a few exceptions, reveal that they spend five per cent or less of their budget on evaluation of their management training programs. About two~thirds of the respondents report their tap management stresses evaluation to some extent or to a great extent. The other one-third report little or no stress by top management on evaluation of their programs. Generally in the companies responding, there appears little or no relationship between the effort, in terms of time and budget spent on evaluation of management training activities, and: a. the level of management training; b. who writes or develops the programs; c. who leads or conducts the training; d. where the training prOgram is given or e. the type of company (major product). on the other hand, there seems to be a tendency for evaluation effort, in terms of time and budget, to be greater in companies with the larger management train- ing staffs (over five persons) and where the training ‘ ‘.. “syn 5 ”y 6. 92 departments have been in operation longer (over ten years). The survey also reveals a tendency for evalu- ation effort to be greater in companies where top man- agement stresses or places importance on the evaluation of management training programs as compared to those where top management shows little or no interest in evaluation of management training. From the companies responding, indications are that at least a static position will be maintained and perhaps a slight increase in evaluative effort will be made in the future. Many of the companies reporting the higher amounts of time and budget for evaluation plan also to increase their evaluation activities in the future. Furthermore, top management stress upon evaluation ap- pears to result in greater intentions of future effort and time to be spent on evaluation. By far the major deterrent to effective evaluation in the opinion of the companies participating in the sur- vey is that evaluation research techniques are diffi— cult to apply in productive or operating situations. Generally it seems there is interest in evaluation, finances and time are available, there is knowledge of evaluation research techniques in the opinion of the respondents, and t0p management stresses evaluation and is interested in the findings when evaluation is performed. ._ 4.5".“ - _ r 95 8. The most frequent Egggg of evaluation, the question— naires revealed, is to ascertain the feeling of parti- cipants about a training program, that is, the partici- pants' liking or impression of the program. Most com- panies indicate, however, that they are doing a compre- hensive job of evaluation by probing its various "facets" by a multi-focus approach. 9. The survey disclosed that the companies use a variety of methods for evaluation assessment. Frequent use is made of interviews with, or questionnaires sent to par- ticipants in a training program which attempt to ascer- tain their feelings about or believed benefits from a course or program. Course material information tests before, during and after programs are frequently used also. A small percentage, however, test their trainees some time after the prOgram. Attendance at programs, amount of participation, interviews with, or questionnaires sent to participants' supervisors and direct observation of participants' actions on the job after the prOgram are frequently used methods of evaluation also. No companies offered revolutionary evaluative ideas or research methods other than variations of that found in the theoretical aspects of evaluation chapter and the review of the literature in Chapters II and III. 933.3,“, 94 10. Lastly, the findings of the survey indicate that about half the management training practitioners in the in- dustries reporting are aware that effective evaluation is a continuous process,as the writings of the authori- ties in the theoretical chapters suggest it should be. These companies report that they do plan for and per- form evaluation of some focus and with some method regularly before, during, directly after and some time after their management education and develOpment pro- grams. ‘ ' ‘_ ‘.mvviw l. 2. . 3. 4. 5. 6. 7. 10. ll. 95 Bibliography Belman, Harry S. and Bliek, John E., ”Training Depart- ments in Business, Industry and Government," Journal of the American Societ; 9; Training Directorg, Vol. fiII, NO. g, septo, V 8’ pp. 45- O ‘ Belman, Harry S. and Remmers, H. 3., "Evaluating the Results of Training," Journal of the American Societ . gg Trainin Directors, Vol. XII: No. 5, May, 1953, pp. _5§:____E "‘“‘"‘“‘ ‘ . Besco, Robert, Tiffin, Joseph and King, Donald 0., "Evaluation Techniques for Management Deve10pment Pro- grams." Journal of the Amerio Societ g; Traini Directors, Vol. XII, No. ll, Uct., 1959. pp. 13-23. Borosage, Lawrence, "Some Considerations Regarding Evaluation of Training." East Lansing, Michigan, Na- tiogal Project in Agriculture Communications, 1958. p. . Clark, Harold F. and Sloan, Harold S., Classrooms ;g the Factories. Rutherford, N. J., Fairleigh Dickinson University, 1958. pp. xiii, 159. , Current Practices i; thg Develoament 9; Management Personnel, New Yerk, American anagement ssoc ation, T955, p. 55. , "Evaluation of a Training Program in Action," Personnel. Vol. XXIV, No. 5, March, 1948. pp. 572-577. Festinger, Leon and Katz, Daniel, Research Methods in the Behavioral Sciences. New Ybrk, The Dryden Press, 1955. pp. xi, 660. Hillway, Tyrus, Introductign t9 Research. Boston, Mif- flin Co., 1956. pp. xi, 284. Kirkpatrick, Donald L., "Techniques for Evaluating Training Programs, Part I Reaction." Journal of fig; American Societ of Trainin Directors, Vol. XIII, No. ll, Nov., 1 59T' pp. 5-3. Kirkpatrick, Donald L., "Techniques for Evaluating Training Programs, Part II Learning." Journal ofthg American Societ of Training Directorg, Vol. XIII, fi—‘o. T_2, Dec., 59'.” pp. 21-26. 12. 15. l4. 15. 16. 17. 18. 19. 20. 21. 96 Kirkpatrick, Donald L., "Techniques for EValuating Training Programs, Part III Behavior." Journal 9; the American Society of Training Directors Vol. XIV No. l 38.11., I960. pp. I?“ e ’ ’ ’ Kirkpatrick, Donald L., "Techniques for Evaluating Training PrOgrams, Part IV Results." Journal of the American Segietg g; Trainin Directors, Vol. XIV, no. fig Eebqq 0 PP. - 2e Korb, L. David, Traini the Su ervisor. Washington, D. C., U. S. Civi ervice Comm 38 on, 1956. pp. v, 125. Lundberg, George A., Social Research, A Study in Methods of Gatheri Data. New York, Longmans, Green and Co.,'I . pp. xx, 426. Messer Elizabeth F. Assessing and Reporting Traini Needs and Pro re . ’Wash ngton, D. ., . . Civil Service Commission, 1956. pp. vi, 77. Parten Mildred Surve 8 Polls and Samples. New York Harper’and Bros: 1950. ’pp. iii, 62 . ’ "Psychologists Give Management-a Quiz on Taginggg’Methods." Business Week, Aug, 16, 1952. pp. Reeve, David F., "A Survey of the Duties and Responsi- bilities of Training Personnel in Business and Industry." The Journal of Industrial Trainin Vol. VII No. 5 §3Pte" 0th, E5 0 ’ , ’ Rummel, J. Francis, Ag Introduction 32 Research Pro- cedures in Education. New York, Harper and Bros., 1958. PP! XV:- 91150 ZBisel, Hans, Say it With Figures. New York, Harper &. Bros. 1957. p0 2570 ‘ ‘ 3431997 CHAPTER V CONCLUSIONS AND RECOMMENDATIONS This study reported the theoretical aspects and the practices of evaluation actually used in formal management education and development programs in industry. In Chapter I the need and importance of the study were outlined as derived from several sources. Selected theoretical aspects of evaluation were synthesized from the literature of edu- cation, psychology and business in Chapter II. The chap- ter began with a brief history of evaluation and was fol- lowed by the purposes and principles of evaluation, the steps in an evaluative process and the methods of evalu- ation as suggested in the literature. In Chapter III, the studies or experiments in the literature discussing the evaluation of management train- ing were reviewed. Chapter IV covered the questionnaire- survey's findings of evaluative practices in management education and development programs in 118 selected United States companies. The levels of management training ac- tivity, size of staff, years of Operation and the amount of effort, in terms of time and budget spent on evaluation, were given. Also revealed in Chapter IV was the degree of stress which tOp management places upon evaluation and a number of other factors which were revealed as hindering or helping the implementation of evaluation. Future effort 97 98 to be devoted to evaluation was determined along with the greatest deterrents to evaluation in these industries.’ Lastly, the chapter noted the focuses of evaluation in the selected companies along with the methods used in their evaluations. A summary at the end of each of the chapters high- lighted the major points and findings of the research. The main task of this chapter, then, is to cast these di- verse findings into several major conclusions and recom- mendations which have been generated in a study of this purpose and sc0pe. Conclusions 1. Generally the authorities,in the literature of evalu- ation, agree that the basic purpose of evaluation should be to achieve effectiveness in education. 2. The following principles, or guide-lines, should be con- sidered by the evaluator of industrial management train- ing programs: a. Evaluation usually offers the greatest potential benefit if it is conducted over a period of time and as a built-in part of the total training process. b. Evaluation should be more concerned with the re- sults of programs rather than effort expended in conducting the programs. c. Programs with Specific objectives can most easily be evaluated. 99 d. Variables Which may influence the results of evaluation should be isolated and taken into consideration. 5. Evaluation of management training programs should be performed in an orderly, yet flexible process. These steps are recommended: a. b. c. d. 6. Define the problem to be studied or evaluated and formulate the objectives the education is to accomplish. Identify a measurable or test situation where changes may be noted. Apply the measure or evaluative method. Analyze and interpret the results. Improve or modify the management training pro- gram. 4. A comprehensive evaluation of a management training pro- gram is encouraged. Evaluation attempts in several of these areas will provide evidence upon which improve- ments or greater effectiveness of programs might be pachieved: a. b. Go d. The method of presentation of the training pro- gram, e.g., evaluation of conference techniques, conference leaders The participants' progress during the training program The feeling of participants about the training program The learning of participants as a result of the training program, e.g., certain management prin- ciples, supervisory knowledge The behavior of participants as a result of the training Program, e.g., different actions back on the job resulting from the program - a "it! “4:." 100 f. The results, effect or impact on the group, inter-group relationships or organization as a result of the training program 5. As indicated by the companies responding to the survey, 6. 7. (c) "The feeling of participants about the training pro- gram" is the focus of evaluation which is most often at- tempted by these companies; however, there were numerous other evaluative attempts with other focuses also in- dicated. It appears that industry is doing a somewhat comprehensive job of evaluation by frequently using a multi-focus approach. Small companies, in size or train- ing staff, or companies in the training field only a few years, are as active in a multi-focus approach to evalu- ation as are the larger companies or those having man- agement training over a longer period of time. Management training practitioners in about half the com- panies reporting are aware that effective evaluation is a continuous process as the authorities in the earlier theoretical chapters suggest it should be. The compan— ies report that they do plan for and perform evaluation of some focus and with some method before, during, di- rectly after and some time after their management edu- cation and development programs. In the experiments in the literature concerning the evaluation of management education and development pro- grams, several evaluation methods appear workable with, . is: war 101 of course, the inherent limitations various situations may present. In most cases, the evaluator should seek to isolate all contaminating factors which may influ- ence the result of using any of the following methods. b. C. d. f. 80 h. 1. Some form of test, designed to measure the course or content material to be imparted as a result of the prOgram, given before, during, immediately after and some time after the program. (However, there is some indication in the reported studies on human relations programs that the self-rating type of test given directly after a program has serious inadequacies in determining increased managerial performance or effectiveness as a re- sult of the training.) External organizational criteria, e.g., absentee- ism, accidents, grievances, labor turnover, as a measure of improved organizational effectiveness as a result of the management training program Morale, organizational attitude or communications type of surveys to determine improvement in the participants' work groups “Comment" questionnaires of various designs after the program to ascertain the participants' like of or benefit from a program Group participation and/or the attendance at a series of conferences which are held on a volun- tary basis Systematic interviews with program participants and/or their supervisors on the merits of the pro- gram Systematic observation of the performance of the management person back on the job after the pro- gram Tests of various design and purpose given on a before and after basis, administered to control and experimental groupings A composite index of selected personnel statistics, with an index movement, that would correspond with the desired behavior change or improved performance of the managerial group as a result of their train- ins 102 j. Control and experimental groupings along with a series of related performance inquiry type ques- tionnaires, one year after the program to the participants and/or their peers, those they su- pervised and the participants' supervisors 8. As revealed by the survey, questionnaires of various design and purpose are the most often used method of evaluation. Questionnaires designed to assess the feelings or like of a prOgram and the benefits of a program are the type most frequently used. Most ques- tionnaires are given to the participants or managers in the program. Somewhat less common is the question~ naire filled out by the managers' supervisors, and few questionnaires are used to question the managers' em- ployees or the managers' peers. 9. Course material information tests are also frequently used on a before, during and directly after the program basis but rarely used some time after the program to determine the lasting effects, if any, of the training program. 10. Group participation and/or attendance at a voluntary ‘series of conferences is often used as an evaluative measure 0 ll. Systematic observation of participants actions on the job after the program is also frequently used as an evaluative method by the companies represented in the survey. 105 12. A number of the other evaluation methods and research techniques expounded in.the literature are infrequently used, as revealed by the survey. They are: a. Check of users"attitude toward service or pro- duct rendered by participants' departments b. Control and experimental groupings c. Morale, organizational attitude or communica- tions surveys in the managers' work groups. d. Use of records on absenteeism, accidents, grievances, labor turnover and the like in the participants' departments 15. No companies offered revolutionary evaluative ideas or research methods other than variations of those found in the literature. 14. Few published studies could be located which attempted to give an industry-wide picture of evaluation,ppgpr piggp. None were comprehensive in the sense of cover- ing adequately the methods or effort in the area of evaluation. But all seemed to be using inadequate data and publicizing that they found a general lack of ef- fective evaluation of management training programs. 0n the contrary, this more comprehensive study, in terms of evaluation practices and effort, shows indus- try is engaged, at least to some extent, in effective evaluation activities and is somewhat cognizant of what effective evaluation includes. ‘ mm v 104 15. Slightly over half of the training departments of the companies represented in the survey spend 1-5 per cent of their pipp_on evaluation of their management training courses or programs. About one-fifth say they spend less than one per cent and another one fifth say they spend 6-10 per cent of their time on evaluation. Rarely does a training department spend more than ten.per cent of its time on evaluation. 16. Nearly all companies responding spend five per cent or less of their budget on evaluation of their manage- ment training prOgrams. 1?. Indications are that at least a gtatic position will be maintained and perhaps even a plight incregsg in evaluative effort will be made in the future. Many companies reporting the higher amounts of time and budget for evaluation plan also to increase their evaluation activities in the future. 18. Tap management stress upon evaluation activities ap- pears to result in greater intentions of future effort and time to be spent on evaluation. 19. About two-thirds of the respondents in the survey re- port their top management stresses evaluation to some ,extent or to a great extent. The other one—third say that there is no stress or little stress by top manage- ment on evaluation of their programs. Evaluation 20. 21. 22. 105 effort, in terms of time and budget, appears to be greater in companies where top management stresses or places importance on the evaluation of management training prOgrams, as compared to those where tap management shows little or no interest in management training evaluation. There is a tendency for evaluation effort, in terms of time and budget, to be greater in companies with ,the larger management training staffs, that is over five persons, and where the training department has been in operation longer, that is over ten years. Generally in the companies responding there seems to be little or no relationship between the effort, in terms of time and budget spent on evaluation of manage- ment training activities, and; A a. the level of management training; b. who writes or develOps the programs; c. who leads or conducts the programs; d. where the training program is given or e. the type of company (major product). Lastly, it can be concluded that by far the major deterrent to effective evaluation in the opinion of the respondent companies is that evaluation research techniques are difficult to apply in productive or operating situations. There is also some indication 106 that the research techniques which are available are inadequate and are too likely to be influenced by variables and contaminating factors in an industrial situation. Generally there appears to be interest in evaluation, finances and time are available for evalu- ation, there is knowledge of the research techniques available, and t0p management stresses evaluation and is interested in the findings when evaluation is per- formed. Recommendations The author wishes to point out the distinction be- tween the "Conclusions" just discussed and the "Recommenda- tions" which are made in this section. The distinction is that the conclusions are derived exclusively from the data revealed by the research while the recommendations do not necessarily grow out of conclusive data. The recommenda- tions, and in some cases implications, have as their genesis the revealed data but may also be influenced by the author's philosophy, observations and experience con- ducting industrial management education and develOpment programs. Recommendations Concerning Evaluative Methods The frequency of usage of an evaluative method ap- pears directly related to the difficulty of administering 107 that method in productive or operating situations. Tests and questionnaires of various designs and purposes, which were reported most frequently used, usually require less effort to prepare and administer in industrial situations. There is, however, some indication in the literature of the inadequacy of tests and questionnaires as a measure of pro- gram results, especially the self-rating type used directly after a program. It seems, then, if the test or questionnaire method of evaluation is to be continued because of its relative ease of administration, greater attempts should be made to assure that the method measures adequately the program re- sults. Several suggestions could be made in this direction: a. Strive for greater use of experimental and con- trol groupings. b. Give the tests or questionnaires on a before and after basis, but most important some time after the prOgram to ascertain any lasting results of the program. 0. Questionnaires given to the participants' super- visors, the participants' employees or the par- ticipants' peers may reveal some interesting results. Additionally, the evaluator should make greater at— tempts to isolate variables and contamination factors which may affect the result of his evaluation. Furthermore, if the evaluative technique is apt to be riddled with variables and contamination from outside factors that are not likely to be able to be controlled, then it would be best to .{V 108 abandon that method rather than pursuing evaluation for the sake of the research exercise. The evidence from the study does not reveal, nor was it the purpose of the study to determine, the most effec- tive evaluative method. It does not seem possible to deter- mine or recommend the one most effective method applicable in.many situations. One program and situation may vary greatly from the next. Rather, it seems best to use that evaluative method which best assesses, with the least amount of bias, the results of a particular program in a particular situation. Moreover, several or a combination of methods, sometimes objective and sometimes subjective, should be used to evaluate in a multi-focus approach. It is recalled that there were a number of peOple in the survey who voiced dissatisfaction with the present research and evaluation methods available. There does ap- pear to be room for improvement in many of the techniques or at least in the sound application of some of them. Naturally, an ingenious new technique would be welcomed. On the other hand, it is wholesome that devices are being employed, however inaccurate, because rough evaluation may be better than none at all. Probably the steel rule was used before the micrometer and contributed to the develop- ment of the micrometer. In like manner, evaluation tech- niques may be borrowed, adapted and are often times refined into workable, more precise evaluation tools. 109 Beyond specific evaluation methods, a few companies revealed what might be termed a "climate of acceptance” on the part of top management. Perhaps this nclimate of ac- ceptance" could very well be cultivated through an involve— ment process. One chemical company indicated they insti- tuted an evaluation committee of tap management peOple. The committee in this case designed and implemented the evaluation devices with the help of the training depart- ment. This appears to be a very practical, yet effective, plan to get evaluation.performed.with the plus factor of built in involvement of higher management. Recpmmendations If There Is to be Greater Effort in Evaluation As shown by the survey, evaluation effort by manage- ment training departments in general is somewhat related to the stress top managements place upon evaluation activi- ties. It seems, then, any indictment of training peOple concerning a lack of evaluation perhaps should also be an indictment of their top managements. Furthermore, if tap managements want evaluation of their companies' management training activities, then by stressing more evaluation their training people will probably respond. Likewise, if tap management is concerned.with effectiveness in evalu- ation and demands it, the training departments will probably respond with effective evaluation. it.” 5.; 59" : L. 110 This related implication might be ventured. Arti- cles in journals and texts continue to be published giving accounts of evaluation experiments and describing evalu- ation research methods. As important as they are to in- creasing the training person's research background, the general status of evaluation and the amount of evaluation in industry will not significantly increase until top management requires greater activity in this area by its management training department staff. Again recalling the relationship between evalu- ation effort, in terms of time and budget, and those indus- tries having the larger management training staffs (over five peOple), top management apparently allows sufficient manpower to perform the evaluation in these cases. There- fore, if tOp management really desires effective evaluation, it is necessary to allow the manpower to perform the re- search. One, two or more people in a training department expected to devote the majority of their time to conducting management training conferences will, after having performed the attendant duties of a conference leader, have little time for effective evaluation research. TOp management, if it deems effective evaluation de- sirable with its management training activities, would be wise to assign one man, who has background in educational and psychological research, at least part-time to evaluation activities. The evaluation, in some instances, might also .‘~ : i». 111 be performed by someone on a nearby university staff. The research specialist, however, should be brought into the program at its inception in order not to handicap the per- formance of complete and effective evaluative research. Recommendations on the Purpose of Evaluation The political implications within an organization shadowing evaluation research should not be overlooked. Indicative of such a situation is a respondent who stated in the survey, "Too many peOple in personnel and training sections are afraid of what an effective evaluation would show." From the author's personal experience, this state- ment would fit more situations and programs than would be readily admitted. Beyond the major indicated deterrent to evaluation--that evaluation research techniques are diffi- cult to apply in productive or operating situations--there is a withdrawn feeling on the part of training people that if effective evaluation is performed, the management train- ing department may be researched out of a job. On the other hand, if the basic purpose of evaluatkul is as it should be--to achieve effectiveness in education-- and if this is communicated to all concerned, there should be no fear of What the evaluation reveals. Evaluation set up merely to justify the program's effectiveness or the continuance of the management training activity is tread- ing dangerous ground. But most important, the evaluation hi :3}. ' 3' .' .‘1 112 such cases would have missed the basic purpose of evalu- ations-the achievement of effectiveness in education. Recommendations for Further Studz Several attendant problems presented themselves, either during or as a result of this investigation, and in the author's thinking are worthy of further study. Cer- tainly these are not the only problems confronting those concerned with evaluation. It is believed, though, any revealing facts or principles of action regarding these problems would be welcomed by students of evaluation as well as those who direct management education and develop- ment activities and those who perform the evaluation of the management training courses or programs. 1. Cost of Evaluation: The present study was con- cerned in this respect with the per cent of a training department's time and budget spent on evaluation as compared to a number of other factors. Little or no work has been done on the cost of evaluation, that is, are the research results really worth the effort and the dollars spent? Studies relative to the cost of one evaluation method as compared to another in a similar situation may prove useful. Some methods of evaluation may be very effective; however, the cost of application may make them prohibi- tive to use in certain situations. 23”.. ? t «'7’ 2. 5. 4. 115 Establishment of Criteria fer Evaluation: Some work was noted in this area: but further research should be undertaken as to usable, practical criteria, i.e., plus or minus measurable fac- tors in the manager's work group or the organi- zation as a result of the training program. Also, successful means of eliminating contamination which affect the selected criteria should be de- termined. This is a very difficult area and re- search producing even a few guide lines would be helpful. Program Results in.Terms of Dollars Saved: In the eyes of many theorists and practitioners as well, the basic purpose of evaluation is to achieve effectiveness in education. Nevertheless, cost-conscious tap management would probably be extremely interested in indications of dollars saved, costs reduced and other criteria which may be converted to a dollar figure. Few experiments were noted in this also very difficult area. Uses of Specific Evaluative Methods: The pre- sent study reported a number of evaluative methods and some of their uses in different com- panies with various management education and develOpment programs. Beyond this, extensive '31:“?! 114 research on a particular evaluative method, its limitations and possible applications in indus- trial management training situations may make more usable the present methods. A compilation of the findings in several of these evaluative methods areas would be highly desirable. 5. Collection of Usable Evaluation Devices: In the present study a partial attempt was made to se- cure forms, sheets, questionnaires and similar evaluation devices used successfully by compan- ies. The response was negligible. Another study with its primary purpose the acquiring of such devices may provide a collection of helpful idea material for those about to embark on evalu- ation of their programs. While the results ob- tained in using such evaluation devices would not be transferable, the devices themselves might very well be revised and adapted for use in an- other evaluation project. Careful studies with any of these problems or parts of these problems, even if they fail to solve all aspects of the problem, would undoubtedly lead to some valuable conclusions and guide-lines in the implementation of evalu- ation for management education and develOpment programs. APPENDICES 115 A. B. C. 116 APPENDIX I OVERALL OBJECTIVES OF SURVEY The survey will attempt to ose new or additional ideas, techniques, methods, explanat ons, reasons concerning evaluation of management training programs in industry that have not been uncovered in the literature and to see how data revealed compares with the literature. l. The more that is known about evaluation in practice, the better industrial educators armed with theory can suggest, revise and use sound and practical evaluation procedures. 8. The results will provide an up—to-date source of infor- mation on evaluation practices for those who plan and administer management training activities. The survey will attempt to reveal the spatus of managew ment education and development evaluat on activities in a major segment of U. S. industry. 1. This should throw light on existing conditions that may need change and improvement, that may otherwise go unnoticed. 2. The results can be used for comparison with other studies, thus identifying trends and also providing a sound basis for action. . The survey will be searching for the levels of evaluation, ractices used, not used, factors and relationships that perhaps Hinder or help the implementation of evaluation. 1. While simple statistical calculations may be necessary to arrive at partial bases for conclusions, the survey is more concerned with the "what, " the "how' 3" and the "wpy' s" of the evaluation in practice. 2. Several authors suggest that the returns on mail ques- tionnaires are inversely proportional to their length and intricacy. Also that the most effective question- naire attempts to tackle one subject in a brief manner, with a mippmm of effort by the respondent. MAILINGS: Suggestions by you and others from this pilot survey will be incorporated into a revised instrument and sent to 158 major industries and utilities selected from the 1960 American Society of Training Directors' member— ship listing. ;a . .37 117 Do you believe there is a neeg for such a survey? Yes No . Comments: (pencil is O.K.) Does it appear that the valugg or outcomes described might be derived with the use of the questionnaire? Yes , No . Comments: Do you suggest any other values or outcomes? Other com- ments: . JV APPENDIX II 118 LIST OF JURYMEN THAT WERE SENT QUESTIONNAIRE AND OBJECTIVES OF SURVEY FOR APPRAISAL AND COMMENT INDIVIDUAL Belman, Harry S. Bright, William E., Jr. Burr, Elbert W. Crissey, Dr. Orlo L. Duenweg, Louis Fleishman, Dr. Edwin A. Form, Dr. William Goodacre, Dr. Daniel M. Guerin, Quintin W. King, Dr. Donald C. Kirkpatrick, Dr. Donald ORGANIZATION Chairman and Professor Industrial Education Purdue University Manager, Employee Development & Training Pure Oil COmpany Manager, Personnel Dev. Monsanto Chemical Company Chairman, Personnel Evaluation Services General Motors Institute Director, Training and Communication Detroit Edison Company Professor, Industrial Administration & Psychology Yale University Associate Director, Labor and Industrial Relations Center Michigan State University Industrial Psychologist B. F. Goodrich Company Chief, Regional Training Branch, Internal Revenue Service Professor, Industrial Psychology Purdue University Advanced Management Deve10pment Administrator International Mineral and Chemicals Corporation LOCATION LaFayette, Indiana Chicago, Illinois St. Louis, Missouri Flint, Michigan Detroit, Michigan New Haven, Connecticut East Lansing, Michigan Akron, Ohio Chicago Illinois LaFayette, Indiana Skokie, Illinois INDIVIDUAL Lawshe , Dr. C H. Seashore, Dr. Stanley MacAndrews, Carl G. Marcus, Leonard Pearson, Archie A. Sahrbeck, Charles, Jr. Sorensen, Olav Steinmetz, Cloyd S. Tiffin, ,. Dr.'Joseph Whitlock, Dr. Gerald wrightnour, William F. ORGANIZATION Director, University Extension Purdue University Program Director Survey Research Center University of Michigan Manager, Training Division E. I. duPont Company Director, Management Information Service American Management Assoc. Manager, Training Department Ford Motor Company Manager, 'Management Training General Motors Institute Personnel Development and Education Relations Service General Electric Company Director, Sales Training Reynolds Metals Company Professor, Industrial Psychology Purdue University Professor, College of Business Administration University of Tennessee Director of Management Deve10pment U. S. Rubber Company 119 LOCATION. LaFayette,. q Indiana Ann Arbor, Michigan Wilmington, Delaware New York City Detroit, Michigan Flint, Michigan Ossining, New York Richmond, Virginia LaFayette Indiana Knoxville, Tennessee New York City ' 1' an.” APPENDIX III SURVEY OF EVALUATION ll2O OF SUPERVISORY, MANAGEMENT OR EXECUTIVE TRAINING PROGRAMS In order to play the game right -- so we're both talking about the same thing -- keep these ground rules in mind as you make your check marks: 1) 2) 3) We're primarily concerned in this questionnaire with your company's EVALUATION of formal management courses, conferences or programs. We are not concerned with the subject matter of these training activities. We' re talking about any level of MANAGEMENT training (supervisory, foremen, executive too) but not apprentice, sales, technical training, and the like. By "EVALUATION" we mean any reasonable assessment of "To what extent did we accomplish what we tried to do?" PART I - YOUR COMPANY AND MANAGEMENT TRAINING Size of full time training staff engaged in management training? (Consider two half-time people as one full-time, etc.) None -5 people -10 11-20 21-30 ___0ver 30 lmll Number of years management training department or an organ- ized management training func- tion has been in operation? _Less than 1 year —1- 5 years :6 1o ___11- -15 ____16-2o ___Over 20 years Indicate approximate percentage of management training depart- ment's time regularly engaged in Pre- supervisory training Foreman or other lst level Middle management training pr management training Other dim... A. Management training programs written or developed by {Check as many as necessary please) _Outside consultants :College or university people :Own staff '__;Other Management training programs conducted py_ (EEEEE‘EE many as necessary please) _Outside consultants :College or university people :Own staff ___0ther Management training programs conducted at (Check as many as necessary please) .A college or university :American Management Assoc. :Own plant or within company :Other "13‘” 10. .121 PART II - EVALUATION AND YOUR COMPANY Approximate percentage of train- 11. ing department's time spent on evaluation of management train- ing courses or programs? (Rough estimate 13 O.K.) . ' ane :::less than 1% Approximate percentage of train- 12. ing department's budget spent on evaluation of management train- ing courses or programs? (Rough estimate is O.K.) ane In the future, to what extent is your training department plan- ning to give importance to and spend time on evaluation of management training activities? .__;Less ___fibout the same More :::A.great deal more Does your company regularly plan for and perform management training evaluation? (Check as many as necessary please) ___pefore a program starts During the program Directly after the program Sometime after the program Other To what extent does your top management stress or place im- portance on evaluation of management training programs? (Check one please) ___To no extent ___Tb little extent ___To some extent ___Tb a great extent What would you say is generally the greatest deterrent to effective evaluation of manage- ' ment training in your company? (Check the one greatest please) __;Lack of time ___Lack of finances ‘___Lack of interest ___lack of knowledge of evalua- tion research techniques ___Evaluation research tech- niques difficult to apply in productive or operating situations Top management does not stress evaluation of manage- ment training Top management not inter- ested in findings when evaluation performed ___Other 13. 15. 16. 122? PART III - PURPOSES AND METHODS OF YOUR EVALUATION List the name of a management 1h. This program (in No. 13) training course or program which given to was evaluated and is the best ___Pre-supervisory level example of your company's use ___Poremen or other lst level of evaluation methods. (If no ___yuddle management ' evaluation was performed, write ___pr management "none" and move on to PART IV ___Other please.) (Nine or title of course or program evaluated) uh ' in?" What was the FOCUS or PURPOSE of your evaluation in this case? (No. 13) Did you)attempt to find out something about (Check as many as necessary please. mes=““””“ '___The ADMINISTRATIVE ARRANGEMENT under which this training program was given _The GOALS AND OBJECTIVES of the training program :‘me METHOD OF PRESENTATION of the training program (conference techniques, conference leaders, etc.) _The PARTICIPANTS' PROGRESS during the training program :The FEELING OF PARTICIPANTS about the training program —The LEARNING OF PARTICIPANTS as a result of the training program _(certain management principles, supervisory knowledge, etc.) ___The BEHAVIOR OF PARTICIPANTS as a result of the training program (different actions back on the Job resulting from the program) _The RESULTS, .EEFECT, IMPACT on the group, intergroup relationships _or organization as a result of the training program. _Other g Your evaluation of this program (No. 13) was based on (Check as many as necessary please) a. Course material information e. ___Check of consumers' atti- tests: tude toward service or pro- _Before program duct rendered by participants' :During program department :Directly after program ‘ :Sometime after program f. USe of participant questionnaires on: b. ___Attendance at program _Changed attitudes :Feelings about program c. _Amount of participation :Benefits from program "by participants g. _Control and experimental d. Interviews with, or question- *groupings ~ naires to: _Participants h. ___Reports by outside observers :Participants' supervisors :Participants' employees 1. ___Direct observation of partici- :Participants' peers pants' actions on Job after program .LC.) (no. 16 continued) J. Organizational attitude or 1. '___Significant increases in '_-communications surveys "performance review" ratings of the participants by their k. use of records on absentee- supervisors —_—ism, material waste, turn- over, grievances, etc. in m. ___Others or remarks the participant's work group PART IV - YOUR COMPANY AND YOU 3 17. Are your reaponses on this survey 18. Maaor product or function of nerall representative of (Check your company? one pIease) ‘ Aircraft and missiles _Yburs single plant which is a ___Automotive ? ~non-affiliate of a corporation ___Chemicals, plastics, drugs ’ ___Tbur division which is part_ ___Electrical and electronics of a larger corporation ____Metals and steel manufacture ___Ibur corporation as a whole ___Paper ___Petroleum Which has: (No. of employees:) ___PUblic Utility ___Rubber _Less than 10, 000 employees ___Tbxtiles :10, 000 - 25, 000 ___General manufacturing :,25 000 - 50, 000 Other :50, 000 - 100, 000 :100, 000- 200, 000 :200, 000 and over OPTIONAL INPOWMNTION I would like a summary copy of the results of this study. Yes___, No . Your Name Title Company Address City and State Thank you very much for your interest, your time and cooperation. It is indeed appreciated! Please return your questionnaire in the addressed, stamped envelope which is enclosed for your convenience. WE WOULD APPRECIATE ANY DUPLICATED COPIES OF YOUR EVALUATION EXPERIMENTS OR TECHNIQUES THAT YOU MAI HAVE AND WISH TO SHARE WITH US. l£¥+ APPENDIX IV MICHIGAN STATE UNIVERSITY EAST LANSING COLLEGE OF EDUCATION You are no doubt well aware of a much bandied-about topic in the management training field today -- that of the evaluation of training. As part of a larger study on training evaluative practices in U.S. industry, we are seek- ing to find out Just WHAT is going on in management training evaluation. In order to determine the scope, some of the techniques, the interest and effectiveness of training evaluation as used by industry today, we are calling on people like yourself to help provide a composite picture of the evaluation of management training. You are the only person in your company receiving a request for this information. Ybu (or one of your designated staff well versed in your company's current evaluation practice) can be of great assistance by filling out the enclosed, quickly checked survey-questionnaire and returning it at your earliest con- venience. We believe you will find it a relatively simple questionnaire, self-explanatory, requiring very little time and a few check marks to complete. we hope that the information requested is not considered of confidential nature. In any event, your answers will not be cited by company name. Also the questionnaire may be returned anonymously, or you may omit a question if you so choose. We are primarily interested in totals, relation- ships and overall analysis of the data returned. Please feel free to make any additional comments that you wish. we desire as complete a picture of management training evaluation in U.S. industry as possible. If you have duplicated copies of your evaluation experi- ments or techniques, and wish to share copies, we would very much appre— ciate that also. Enclosed is an extra copy of the survey for your files. And should you desire a summary of our larger study on management training evaluation which will include the results of this survey, please check in that space provided on your returned questionnaire. Thank you for your help in this matter. Your prompt consideration, time and contribution is truly appreciated! Cordially yours, Carl Shafer Industrial Evaluation Project 305 College of Education 4"} 125 APPENDIX V MICHIGAN STATE UNIVERSITY EAST LANSING COLLEGE OF EDUCATION As part of a study on evaluation practices of management training activities, a questionnaire as enclosed was sent to you recently. A similar request was sent to 150 of the country's largest firms. Many companies have responded but we would also like to have your firm represented in our study. Since you are the only person we have contacted in your company, may we emphasize our dependence upon your response. We believe you will find our questionnaire can be completed rather quickly .with a few check marks. Yet at the same time you will be helping us with the data needed for the completion of our study. Information revealed will not be cited by corporate or individual names in any publications resulting from this study. We do appreciate your prompt consideration of this matter. It is our desire to send you a final copy of the report if you will so indicate on page four of the questionnaire. We believe this report will prove to be an interesting document. If your return questionnaire is already on the way to us, may this letter be another thank you. Cordially yours, Carl Shafer Industrial Evaluation Project 305 College of Education Enclosures 126 APPENDIX VI NAMES OF FORMAL MANAGEMENT EDUCATION AND DEVELOPMENT PROGRAMS CITED BY THE COMPANIES AS THEIR BEST EXAMPLE OF USE OF EVALUATION METHODS MANAGEMENT IN TITLE Basic Management Course PHF Basic Management Orientation F-M Basic Management for New Supervisors F Management Institute M Manggement Trainee Program New Manager Training F The Job of Managing F-M Fundamentals of Management M Basics of Job Management F-M Management Study Program PbF-M Notre Dame Middle Management Course M Advgnced Management Training Advanced Management Program M One Week Management Institute M-T Pittsburg University Manage- ment Program Management Development Pro- gram M Management Deve10pment Pro- gram F-M-T Advanced Management Program M Works Management Control Training Course F-M Management Information Meeting F-M Business Management Seminar M-T Professional Business Management F-M-T LEADERSHIP Leadership Training F .Leadership Training F-M-T SUPERVISOR IN TITLE Responsibilities of a Super- visor F-M Supervisory Training Program F-M Supervisory Training Program F Supervisory Practices F-M Supervisory Development .Course Supervisory Development Course F-M Engineering Supervisory De- velopment Program M Supervisor Orientation Pro— gram F Supervisory Indoctrination PBM FOREMEN Foreman Conference Program F Basic Foreman Training F Foreman Refresher Program F-M HUMAN RELATIONS Human Relations in Manage- ment F Human Relations F-M Understanding Human Rela- tions F-M ECONOMICS Basic Economics F—M Company Basic Economics F-M AMA - Atlantic Profit Planned Management M-T Operation Bootstrap F—M-T CONFERENCE LEADERSHIP Advanced Conference Leader- ship M Discussion Leading F-M-T COUNSELING Counseling F-M Conducting the Job Perform- ance Interview M-T METHODS Job Methods Improvement F Work Simplification Work Simplification F-M Work Simplification - Methods Improvement F-M—T 127 COLLEGE GRADUATE College Graduate Program Basic Orientation for Cola lege Trainee College Graduate Management Training SAFETY Your Safety Responsibili- ties P—F-M Safety and Accident Preven- tion F-M-T COMMUNICATION Reading Improvement F-M—T Public Speaking F-M—T Communications: Principles and Methods PbF-M-T OTHERS Job Training F New Engineering Section Head Deve10pment Program F Personnel Placement and Development PhF-M-T Board and Pulp Mill Start-Up F-M Organization Development in a Headquarters M-T "Sky Top" (Annual Operating Meeting) M Incentive Administration F-M Interdepartmental Indoctri- nation Course M Imagination and Your Job M Quality Control F Cost Control F What're We Doing M How Our Labor Management Agreement Works F-M—T Attitude Survey Feedback F-M—T National Training Labora- tories Sensitivity Train- ing M-T Standard Practices PbF-M-T Production Assistant PrOgram P Applied Accounting F-M The Human Enterprise M BBW’U - given at Pre-SUPERVISORY LEVEL - given at FOREMAN OR OTHER FIRST LEVEL - given at MIDDLE MANAGEMENT LEVEL - given at TOP MANAGEMENT LEVEL I v ., .~ 'n ‘ V—- .0 :_ Rx m. 3...: '—/ “-r 1.. 3. 4:- ‘IV‘I' “eh:- ., ”I. i ' .fin ‘ r.‘ r, C II, hhc' Idea 1' .01», {I-disw J gap-14‘ 1": ‘~ ‘ v 1.... L“ ‘..-5... r... L Phi «I . '1': - 5 18% :. ‘= 3. N .~‘ '- “at: & 54:? "‘6 W 5 r :‘r— ~ MICHIGAN STATE UNIVERSITY LIBRARIES IIHIHIHIHI IIIIHINIHI||IINIINIHWHIIIIHI1W! ”IN 3 1293 03146 0375