AUDIENCE EFFECTIVENESS EVALUATION PROGRAM FOR AIR FORCE MOTION PICTURE AND TELEVISION PRODUCTIONS Thesis for the Degree of M. A. MICHIGAN STATE UNIVERSITY JOHN BRANDON BARNES 1970 IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII LIBRARY LI , ’ IWUmIIvcrsiStyto ABSTRACT AUDIENCE EFFECTIVENESS EVALUATION PROGRAM FOR AIR FORCE MOTION PICTURE AND TELEVISION PRODUCTIONS By John Brandon Barnes The problem which this thesis attacks is the lack of an adequate audience effectiveness evaluation program for the films and video tapes produced by the motion picture and television indus - tries. Special emphasis is placed on the products produced by the United States Air Force. The importance of film and video audience effectiveness evaluation is established by pointing out the relationship of the problem to the communication process, by the use of a small sample survey of independent and institutional educational film producers, by personal correspondence, and by taped interviews with persons active in the motion picture and television fields. Past studies relating to military film production as a poten- tial resource for the military and the civilian researcher are noted. John Brandon Barnes An overview of the extent of the present Air Force audience effectiveness evaluation program is presented. Conclusions reflect that their present program is not adequate. Many of the available methods for measuring audience effectiveness are identified with a discussion of the advantages and disadvantages of each. The questionnaire method, which was found to be most appropriate for Air Force use, was selected. The details of the study that was designed to test the questionnaire method on a current Air Force film are included. The results provide a good example of the type of information that can be obtained from employ- ing such an evaluation method. In the final recommendation for implementation of a work- able audience effectiveness evaluation program for the Air Force' s Aerospace Audio-Visual Center, six areas are considered. They are: proper working relationships for the researcher, producer, and management; management' 5 responsibilities toward the researcher; research and its location in the organization; the researcher' s qualifications; logistical framework of the operation; and identifying the functions of the research and evaluation program. AUDIENCE EFFECTIVENESS EVALUATION PROGRAM FOR AIR FORCE MOTION PICTURE AND TELEVISION PRODUCTIONS By John Brandon Barnes A THE SIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of MASTER OF ARTS Department of Television and Radio 1970 Accepted by the faculty of the Department of Television and Radio, College of Communication Arts, Michigan State University, in partial fulfillment of the requirements for the Master of Arts degree. U313 E SMEZ/ Director of The sis PREFACE The subject of this work lies in the broad scope of communication research and specifically is directed toward the area of audience effectiveness evaluation for motion picture and television products. The comments, conclusions, and recommen- dations, although applicable to the motion picture and television industries, are often directly intended for the United States Air Force and the agency responsible for producing its film and Video products, the Aerospace Audio—Visual Service. The emphasis in the direction of the Air Force stems from the fact that the author is a professional Air Force officer and will be actively engaged in the television and motion picture fields. It is necessary that the reader bear in mind that this is a critical work based upon the desire to strive for production of a higher standard of films and video tapes through audience effective- ness evaluation. It should not be taken as personal criticism of management, procedures, or policies now in effect. However, the proposal to implement such a program will, by necessity, take a practical and realistic approach, as opposed to what may be the ii ' The recommendations are based on the assessment "ideal operation. ' and analysis of the present situation by the author. It reflects infor- mation received through personal observation, interviews with personnel at both the Aerospace Audio—Visual Service (AAVS) Head- quarters and at the AAVS Center at Norton Air Force Base, California, as well as with persons in related fields. It draws on much of the empirical material in research and evaluation that is available. The soundness of this proposal lies in recognizing what has been done, what is being done, and what can be done for the United States Air Force in the area of audience effectiveness evaluation. In addition to the scholarly objectives to which this work aspires, there are some personal goals that this research will hope- fully accomplish. As Lewis A. Dexter, former public opinion analyst for the Democratic National Committee, appropriately said, In the early stages of learning, men are content to have ideas. At a certain point, they learn how not to do things--and since the methodology of warning and inhibition is relatively easy to learn and to communicate while the methodology of creativity and construction needs an extra dash of imagination and insight beyond what can be taught, it is possible that the second and third generation of scholars in many fields get less excitement out of it than did the founders. 1Lewis A. Dexter and David M. White, eds. , People, Society, and Mass Communications (New York: Free Press, 1964), p. 501. iii It is hoped that the ideas expressed here will provide that ' As was pointed out, this work may not "extra dash of imagination. ' excite many of the second or third generation scholars, but hopefully it will excite some current generation management in the motion picture and television industries. There are many to whom I am indebted for assistance, sug- gestions, and technical data used in connection with the preparation of this work. In addition to all the Faculty members who I had for instruction and who guided much of my thinking, I would like to express my appreciation to Dr. J. David Lewis who initially assisted me with establishing a firm foundation in the "Proposal"; to Dr. Thomas F. Baldwin for his valuable consultation on the study design and questionnaire; and to Dr. Robert Schlater, my thesis advisor, who patiently directed my efforts from producing a Military Staff Study toward the completion of a scholarly Thesis. I want to thank Major Jack H. Oswald, Chief Internal Media Branch, Office of Information, Secretary of the Air Force, for his personal interest and encouragement, as well as for providing related administrative support for parts of this work. He was responsible for providing many documents and films, and making arrangements for transpor- tation which enabled Visual observation of the Air Force facilities and interview s. iv A great deal of credit for the final completion of this work should go to my wife, who not only applied her ability as a proofreader to the task but managed the affairs of four active children while their father completed the research, compilation, and writing of this endeavor. TABLE OF CONTENTS CHAPTER Page I. INTRODUCTION . . . . . . . . . . . . . . . . 1 Statement of the Problem 1 Definition of Terms . . . . . . . . . 2 The Importance of Film and Video Audience Effectiveness Evaluation . . . . . 3 Past Studies Relating to Military Film Production . . . . . . . . . . . 14 Overview and Extent of Present Air Force Audience Effectiveness Evaluation Program . . . . . . . . 23 Conclusions on the State of the Present Air Force Audience Effectiveness Evaluation Program . . . . . . . . . . . 28 II. METHODS AND TECHNIQUES OF AUDIENCE EFFECTIVENESS EVALUATION . . . ._ . . . 40 Mechanical . . . . . . . . . . . . . 40 The Lazarsfeld —Stanton Program Analyzer . . . . . . . . . . . . . 40 The Cirlin Reactograph . . . . . . . . . . 43 The Hopkins Electric Televoting Machine . . . . . . . . . . . . . 44 The Schwerin System . . . . . . . . . . . 45 Physiographs . . . . . . . . . . . . . . 47 The Dynascope . . . . . . . . . . . . . . 51 Infra -Red Film . . . . . . . . . . . . . . 52 Distractor . . . . . . . . . . . . . . . . 53 Nonmechanical . . . . . . . . . 58 Thurstone' 3 Method Of Equal- Appearing Intervals . . . . . . . 60 Likert' 3 Method of Summated Ratings . . . 62 vi CHAPTER Page Guttman' s Scalogram . . . . . . . . . . 65 Osgood' 8 Semantic Differential . . . . . . 67 III. SELECTION AND TEST OF A METHOD FOR AN AUDIENCE EFFECTIVENESS EVALUATION PROGRAM . . . . . . . . . . 71 The Practical Application of the Questionnaire Method . . . . . . . . . . 78 The Air Force Audience . . . . . . . . . . . 81 The Questionnaire . . . . . . . . . . . 82 Applying Values for the Attitude Measurement . . . . . . . . . . . 86 Results, Recommendations and Conclusions . . . . . . . . . . . . . . . 89 IV. RECOMMENDED IMPLEMENTATION PROCEDURE FOR ESTABLISHING A WORKABLE AUDIENCE EFFECTIVENESS EVALUATION PROGRAM AT THE AEROSPACE AUDIO-VISUAL CENTER . . . . 100 Establishing the Proper Working Atmosphere . . . . . . . . . . . 102 Management' 8 Responsibilities . . . . . . . . 106 Position and Its Location in the Organizational Structure . . . . . . . . . 108 Qualifications of the Researcher . . . . . . . 111 Logistic and Operational Framework of the Program . . . . . . . . . . 115 Research Functions and Expected Benefits for the Aerospace Audio— Visual Service . . . . . . . . . . . . . . 118 APPENDIX A: QUESTIONNAIRE . . . . . . . . . . . . . 121 SOURCES CONSULTED . . . . . . . . . . . . . . . . . . 140 Vii CHAPTER I INTRODUC TION Statement of the Problem The problem which stimulated the interest and research in the fields of motion picture and television is basically that these industries do not have an adequate evaluation program to measure the effectiveness of their film and Video products on their audiences. The objective of this thesis is to stimulate interest and generate further study by management in the film and television industries by: 1. Justifying a need for an audience effectiveness evaluation program for motion picture and video products. 2. Identifying methods and techniques of film and video evaluation which are presently available. 3. Selection of a method which is compatible with an organi - zation' s resources and testing it to illustrate the type of information that can be made available through the use of various question techniques in a questionnaire format. 4. Propose a practical and workable organization with which management and specifically the United States Air Force' 3 Aerospace Audio-Visual Service can implement an Audience Effectiveness Evaluation Program. The basic purposes of this program are: to relate the pro- ducer of film and video tapes to the intended objectives of the product and to the audience of that product; to establish procedures for audience feedback and evaluation; and to apply the evaluation infor- mation to improve the communication process through film and video production techniques. Definition of Terms In View of the nature of this research project, which attunes itself closely to the use of certain words which are part of the pro- fessional vocabulary of the motion picture, television, and military worker, it is necessary that several terms be defined. "Film and television products" --are the finished films or video tapes which are ready for distribution or testing. ”Aerospace Audio -Visual Service" --applies to the Head- quarters for the unit that is responsible for the production .of all the Air Force' 3 film and video tape products. "Aerospace Audio—Visual Center'' --is the actual facility that produces the films and video tapes. Both the Headquarters and the Center are located on Norton Air Force Base, San Bernardino, California. ”AAVS" --is the abbreviation for the Aerospace Audio-Visual Service and the Center. In this study, unless specifically designated, little distinction is made between the Headquarters and the Center. "Methods and techniques"-—are terms often used inter- changeably by various researchers when referring to specific ”modus operanti” in the evaluation areas. For the purposes of this paper, the word "method" will be applied to the broader scope of research operations; whereas the word ”techniques” will differentiate the various research operations concerned withimethod. The Importance of Film and Video Audience Effectiveness Evaluation The proper management of any organization charged with the responsibility for producing a product calls for considerable knowledge about its consumer or public. The production of films and video tapes is no exception. Evaluation of the impact and effectiveness of films and video tapes on the audience should be an integral part of the management process. It would be improper for the reader to infer from the "Statement of the Problem” that the importance .of audience effectiveness evaluation is not recognized by the management and producers involved with the film and television industries. The results of this research work support a very high awareness of the problem, especially by those active in the fields. Occasionally a decision by a producer based largely upon guesswork will succeed, perhaps even brilliantly. However, long term decision making by those in the production of film and television products will be good and successful only if based upon relevant facts about the values, the interests, and the knowledge of the audience; facts about the success in communicating information and ideas; and facts about the way these products affect people from one situation to another. The nature and the number of facts that are needed will vary. Belson warns, "to act without them is to invite at least some. degree of failure. ”2 Dr. David H. Curl, Pro- fessor of Education, Western Michigan University, supports this notion. "In the absence of facts, we base our communication efforts upon reasonable intuition and hunches well tempered by bitter expe - "3 It seems there is a relationship 0f knowing these rience. . . . facts to the communication process. If we are to understand the responsibilities of those con- nected with the film and television industry toward the task of 2William A. Belson, The Impact of Television (Connecticut: Archon Books, 1968), p. 1. 3David H. Curl, "Why Do Films Fail?" Photo Methods for Industry, Vol. 9, November 1966, pp. 67 -68. audience effectiveness evaluation, it appears that the logical questions to ask are: Where do we start? and with whom? In the communica- tion process one might look at the standard communication formula-- ”Who says what to whom, when, and how, with what effect?" At first, because the subject matter is primarily with film and video tape, it may seem that the concern should be with only certain elements of this formula: 1. e. , "who”--being the communicator or film maker, "what"--being the message to be communicated, and "to whom"-- being the audience. There could be some cause for alarm that pro— ducers often fail to consider the "to whom" aspect of the communica- tion formula. Raymond Bauer says: A revision is gradually taking place with respect to the classic formula of communications research: "Who says what to whom with what effect?" Originally, communications research implicitly gave the major initiative to the communicator. The main question asked was: What sort of communications and media are most effective and under what circumstances? It would be incorrect to say that the audience was ignored, but it was certainly accorded very little initiative. Audiences have, however, proved intractable. They make their own decisions as to whether to listen or not to listen. Even when they listen, the communication may have no effect, or it may boomerang. Increasingly, researchers have had to shift their attention to the audience itself to find out what sorts of people they are dealing with under what circumstances. 4Raymond A. Bauer, "The Communication and the Audience, " in People, Society, and Mass Communications, ed. by Lewis A. Dexter and David M. White (New YOrk: Free Press, 1964), p. 127. It would appear that if the awareness of the audience is becoming that important to the researcher, perhaps it should become equally important to the film and television producer. If it is assumed that the producer is responsible for creating the stimuli (film or video tape) with the purpose of adjusting or steering behavior in a particular direction, it would stand to reason that there must be some relationship or "mutual understanding" between the persons who produce the stimuli and the persons who respond, although it necessarily need not be considered an automatic one. If a situation existed in which a producer makes a film or video tape without in some way determining the attitudes and back- ground of the audience, any general or specific assumptions he may make about this audience could be correct or completely false. Thus it would appear that the producer is taking a "big gamble" on the success of his approach in communicating and meeting his objectives. It is suggested that most film -making organizations, and particularly the United States Air Force, could not afford to take a chance on succeeding at those low odds. Perhaps this line of reasoning supports the belief that a critical part of the problem, which this study approaches, lies in identifying a method to learn about audience attitudes. If the objec - tive of the film maker is to produce films and video tapes which effect attitude change and result in some type of behavioral adjustment, then perhaps a few assumptions about audience behavior and what causes the change will assist the reader in better grasping this com- plex area of human communication. Phillip Davison, former editor of Public Opinion Quarterly, lists these assumptions: 1. That all human actions and reactions, including changes in attitude and knowledge, are in some way directed toward the satisfaction of wants and needs. 2. That man' s wants and needs are dependent for their satis - faction on his environment. 3. That human attention is highly selective. From birth people learn that satisfaction of their needs is dependent more on certain aspects of their environment than on other aspects. They, therefore, focus their attention on these aspects. 4. That people gradually accumulate and carry around with them a substantial quantity of information about those aspects of the environment that are important to them. Therefore, in order to increase the possibility of achieving a ”mutual understanding" between the producer and his audience, the producer should be cognizant of the needs of the audience, the audience' 3 environment, and the prevailing attitudes about his needs and environment before establishing his production approach to a film or video tape. Without this type of information, it would appear the prob- lem of producing an effective film or video tape is magnified. To 5Phillip W. Davison, "On Effects of Communication," Ibid., pp. 73-75. measure the type of information needed would require some method of testing prior to the production of the vehicle. The problem of inadequate evaluation could be further compounded once the vehicle is produced, because with or without an audience pretest, there appears to be no adequate method to determine if the film or video tape was effective in meeting its short and long range objectives. (There is, also, as part of the overall problem, some question about the adequacies of the objectives for these products. The sub- ject of ”objectives" is discussed in more detail on page 32 of this work. ) The importance of audience effectiveness evaluation has been recognized by many prominent persons in commercial and edu- cational films and television, as well as by the Air Force manage- ment responsible for and concerned with the production of film and television products. Almost unanimous agreement as to the impor- tance of audience evaluation is reflected by those actively involved in the industry. They also tend to agree that it is not being effectively accomplished. To support this contention a small sample of independent and institutional educational film producers was surveyed to find out how they determined the criteria for their films, and how they tested the audience effectiveness. This survey is in no way intended to be representative of all film producers. The small size of the sample merely indicates some idea of the possible, opinions and what the general attitude of the profession may be. The total sample was 76. This sample was divided into four categories. In Category A, 34 were randomly selected from 2, 640 educational film producers listed in the National Information Center for Educational Media' 5 Film Index. In Category B, 19 producers were selected randomly from a list of 198 producers which were designated specifically as providing service in film productions. Category C was a personal selection by the author of 10 film asso- ciations and foundations whose purpose was to actively promote and encourage film development in their interest area, i. e. , American Film Association and Farm Film Association. Category D was a personal selection by the author of 13 large companies which have displayed a greater ”professionalism” toward educational films by preparing teachers' guides with objectives, etc. Such firms are represented by Encyclopedia Britannica, Coronet Instructional Films , and McGraw -Hill Text Film Division. A simple four ques- tion document was designed for the survey. The results were that 37 per cent of the 76 answered. How- ever, only 18 (or 24 per cent) of those actually indicated that audience evaluation applied to their operation. Therefore the results are based on 24 per cent of the original 76. 10 The four questions and the results of the responses are as follows: 1. DO YOU BELIEVE AUDIENCE EFFECTIVENESS EVALU- ATION IS IMPORTANT? IS IT NECESSARY? 94 per cent said yes it is important. 6 per cent said no. 67 per cent said yes it is necessary. 22 per cent said no. 11 per cent had no answer. 2. HOW IS THE TYPE OF MATERIALS, PRODUCTION TECHNIQUES, AND GENERAL APPROACH TO MAKING A FILM OR VIDEO TAPE DETERMINED? The answers were equally divided into three categories: a. Individual producer' 3 intuition and experience. b. Board or committee. c. Audience and sponsor' s acceptance. 3. WHAT CRITERIA DO YOU PRESENTLY USE IN DETER— MINING THE EFFECTIVENESS OF YOUR FILM AND VIDEO PRODUCTS? 22 per cent use audience reaction (method not indicated). 22 per cent use a preconceived or set formula. 17 per cent use sale of products. 39 per cent failed to comment. 11 4. DO YOU USE ANY TYPE OF AUDIENCE OR USER ”FEEDBACK" IN EVALUATING YOUR PRODUCTS? IF SO, WOULD YOU EXPLAIN? 83 per cent have no type of feedback. 17 per cent use various methods. Some use a postcard or single sheet evalu- ation form. One uses results on exams. Walt Disney Studios uses Nielsen Television ratings for their television productions. Another firm enters their films in "Film Festivals” to get a com- parative evaluation. What this informal survey indicates is that the producers think audience evaluation is important; but they are not doing it. Of these, a lesser number think that it is necessary. There are some that believe it serves no purpose (at least for them). Although there was no probe for the reason why producers did not conduct audience evaluation, there was an indication that cost was a factor and that there existed an attitude of ”why should they spend the money if the consumer wasn' t interested? ” It appears that often the producer or committee sets up the production approach without any user feedback. If the product sells, it is assumed to be a success. Less than 20 per cent of the producers from all categories made any effective effort to see if their products were achieving their objectives. Management as well as producers, who are heavily com — mitted to the film and television industries, are well aware of the 12 importance of an active audience effectiveness evaluation program and some of its problems, as is indicated by the following comments. Coronet Instructional Films has been a leading producer of educa- tional films for over thirty years. It is probably the largest civilian producer of educational and instructional films. John E. Creeden, Manager of Education Services for Coronet, I says: Audience effectiveness evaluation is indeed very important and probably an absolute necessity. In Coronet' s case, our audience is primarily, if not completely, students that we are attempting to reach, to instruct, and to inform. Our films are not intended as entertainment, but are produced to provide learning and instruction about a given subject or a certain aspect of the sub- ject. Like all educational materials, they must be evaluated in terms of their effectiveness to do the job for which they have been intended. 6 Shelton K. Stern, Editorial Director, Text-film Division, McGraw- Hill Book Company, adds: An audience effectiveness evaluation is extremely important to us. We must be concerned with the success of the production techniques we use in the various subject matter areas. The results of such an evaluation can indicate to us strategy for development of future productions. Although audience effective— ness evaluation is important and is necessary, it obviously can- not be budgeted for each individual production. Edward J. Pfister, Assistant to the Executive Director, National Instructional Television Center, expresses concern: 6Letter from John Creeden, Chicago, Illinois, January 29, 1970. 7Letter from Shelton Stern, New York, New York, Feb- ruary 4, 1970. 13 Audience effectiveness evaluation is both important and necessary. As a matter of fact, it is important enough to be the key to eventual acceptance or rejection of television in education, in my view. Education more and more will be asked to justify its costs. And television, if it wants grow- ing acceptance from education, will have to justify itself, especially in terms of effectiveness. Ronald E. Sutton, Manager, Education Department, The American Film Institute, comments on some of the problems: I do believe audience effectiveness evaluation is important and necessary. However, when one attempts to answer the question ”how it can be done, ” I find myself with many more queries than answers. There are so many variables in the situation. The present research in the field, as you are undoubtedly aware, tends to be far from the main stream of what I feel is audience response due to the difficulty of controls and the setting of evaluative objectives. 9 Steve Knudsen, President of the Board of Trustees, University Film Foundation, supports the cause for continuing efforts: I remain a firm believer in the need for continuing efforts in this area of determining audience effectiveness. Robert W. Gibeaut, Post Production Manager, Walt Disney Produc- tions, reflects some far ~sightedness on the part of the Disney Organization: 8Letter from Edward Pfister, Bloomington, Indiana, Jan- uary 29, 1970. 9Letter from Ronald Sutton, Washington, D. C. , February 3, 1970. 10 Letter from Steve Knudsen, Ames, Iowa, January 30, 1970. 14 We do believe that audience evaluation of our theatrical product is very important. We began shortly after World War II on an audience evaluation program. Dr. Hideya Kumata, Director of the International Communication Institute, Michigan State University, sums up the importance of evaluation and research: It is the relationship of evaluation to practice which should merit serious attention. We need pioneers in any new field. There is a danger, however, that the energies and talents of these pioneers may be dissipated on numerous byways which will not produce fruit- ful results. It is here that research may perform one of its functions. Research can help to specify some of the avenues of most profitable effectiveness. Moreover, it can weigh the relative merits of several contemplated approaches to a prob- lem, and aid in eliminating the more inefficient and uneconomi - cal ways. 12 Past Studies Relating to Military Film Production For the past twenty -eight years there have been periodic attempts to master the problem of film evaluation within the military services. The actual number of studies is not really known since they have often been buried in the archives under obscure and apparently unrelated titles. However, several locations which have 11Letter from Robert W. Gibeaut, Burbank, California, February 2, 1970. Hideya Kumata , An Inventory of Instructional Television Research (Ann Arbor: Educational Television and Radio Center, 1956), p. 1. 15 some of these documents on file are listed as a matter of interest as a source for further research. Presently the Office of Information Retrieval and Evaluation, Information for Armed Forces, Department of Defense, is working on a project to locate every available study dealing with evaluation of military films. That office presently has all the known studies from 1958 to the present. It is hoped that all previous studies can be located and copies obtained for use by re- searchers. Many of the studies by Carpenter, Altus, Hoban and van Ormer, Hovland, Lumsdaine, and Sheffield have been published and have been the source of much valuable information on the effects of films on attitude change and training. Several brief summaries of unpublished studies conducted by and for the military services have been included here to illustrate the variety of research that has been done in this area of film audi - ence evaluation. One organization which was responsible for a great number of reports was The Bureau of Social Science Research, Inc. Under contract to the Directorate for Armed Forces Infor- mation and Education, Department of Defense, the Bureau of Social Science Research conducted a program for approximately two years (July 1962 through June 1964) to evaluate the effective- ness of troop information materials. The program . . . [was] designed to meet the following requirements: 1. development of data on the extent to which the broad themes of the AFIE Program are effectively expressed and reflected in specific products; 16 2. accumulation of information on the social and psychological characteristics of the audience; 3. investigation of the mechanics of the transmission belt through which products are disseminated; 4. experimental research on the effectiveness of given products, and on the cumulative effectiveness of the ongoing program, together with collateral research on the factors that condition effectiveness; 5. continuous and systematic feedback from the analysis of data obtained through the foregoing procedures to the design and production of each subsequent product. One such report was written by the Bureau of Social Science Research on the film The Third Challenge by Thomas Lorimer. Summary: . . . 186 enlisted men in SAC, stationed at Carswell Air Force Base, Texas, were shown this film . . . and given a fifteen question tOpical questionnaire before and after the showing. Some findings . . . 58% rated it as above average or way above average. Only 2% rated it below average. 66% gained information from the film. . . . More than half of the persons who did not know what an "Insurgent" was before the film, answered wrong after it. Experience has shown that Sharp rises in level of informa- tion do not necessarily portend parallel effects on opinions and attitudes of viewers. . . . Although the film did not affect the viewer' s inclination to serve in antiguerrilla units --it produced no deterrent effect. 3Bureau of Social Science Research Inc. , A Program of Research on the Effectiveness of Troop Information Materials (August, 1962), cited by Jack H. Oswald, "A Survey of Evaluation Programs on the Effectiveness of Air Force Films" (unpublished research paper, University of Southern California, 1964), p. 24. 4Bureau of Social Science Research Inc. , Report No. 8: The Third Challenge (June, 1963), ibid., p. 29. 17 A little different approach was taken in Report 13 on Cumulative Effects of Three Major Emphasis Films on Attitudes of Basic Trainees. This study was conducted to determine the cumulative effects of three films used in the Armed Forces Information and Education Major Emphasis Program. The films were The Road to the Wall, Freedom and You, and The Third Challenge. This research measured the effect of these films on the attitudes of samples of Army trainees. Of secondary interest were the changes in attitudes which took place as a result of four weeks of basic training plus the effects of the use of one film as against the use of three films. Some of the conclusions were: that the attitudes of those trainees tested by questionnaire after viewing the three films were consistent with the overt intent of the films. Another group that was exposed to a questionnaire m to seeing the films decreased sig- nificantly in certain areas, and actually did worse on an information quiz about the film The Road to the Wall. The effect of the one film, Freedom and You, in which the subjects were exposed to the questionnaire before seeing the film, was generally in an undesirable direction, that is, in a direction opposite from what the film apparently intended. They were less 18 anti -Russian. This was a boomerang effect, considering the unflattering portrayal of the Russians in the film. These trainees also became much less extreme in their condemnation ofCommu- . 15 msm. A study by the United States Training Device Center, Human Engineering Department, points out some unique and diversified subjects that can be approached through the use of film and research. Problem: The Quartermaster Food and Container Institute of the Army requested the Special Devi ces Center to investigate the problem of unfavorable attitudes of soldiers toward Army food and how these attitudes might be changed through use of films. The basic field ration--C ration--was selected as the particu- lar food item that meets with the most wide -spread disapproval. In this study several film variables were investigated which were thought to be related to the effectiveness of films in re- structuring trainee' s attitudes. These variables concerned the prestige of the main character in the film, the degree to which the main character is identified and characterized in the film and the relationship of the main character to the trainee. Results: To produce an effective attitude -changing film this research indicates that: 1. The main character should be a prestigeful figure rather than an anonymous narrator. 2. The main character should be well characterized and identified in the film, and 5Bureau of Social Science Research, Report of the Bureau, Cumulative Effects of Three Major Emphasis Films on Attitudes of Basic Trainees (Washington, D.C.: n.p., 1964), pp. 38-48. 19 3. The main character should have a high degree of relationship to the audience. Recommendation: Films produced with the objective of changing attitudes should incorporate the above findings. One of the earliest indications of a film study appeared in an article in the monthly digest of the War Department' 3 pamphlet entitled What the Soldier Thinks, December 1943. It mentioned a study on the effects of film on attitudes. The article describes an experimental study conducted among trainees at a replacement training center. The conclusion was that "effectiveness can be stepped up materially if an Officer will introduce the film with a short talk. "17 Three groups were used in the study: A control group, a group that saw the film only, and a group that saw the film and heard the talk. A questionnaire was administered afterwards. The results were that those trainees in the control group scored an average of 20; those who saw only the film scored 29; and those who saw the film and heard the Officer talk scored 35. 16U. S. Department of the Navy, The Relative Effectiveness of Several Film Variables in Modifying Attitudes: A Study of the Application of Films for Influencing the Acceptability of Foods (Port Washington: Office of Naval Research, NAVTRADECEN-269- 7-60, 1956), p. 1. 17U. S. War Department, ”Effects of Film on Attitudes, " What the Soldier Thinks, Morale Services Division, Army Services Forces (Washington, D.C.: n.p., 1943). 20 These studies are certainly only a few among the many that have over the years produced a great amount of information. Generally the researcher' 5 recommendations for the film makers are only inferred, very rarely specified. "Classical research studies . . . [are] for the most part merely descriptive of how people reacted to certain films, with little attempt to help us reason why people reacted as they did, or to compare different methods of - 8 presentation for effectiveness, v.1 It would appear that perhaps an important reason for review- ing these studies would be: to learn what has been done; to keep from repeating the same tests; and to use the information as a building tool. Particular studies could be developed that are pertinent to the problems at hand. It would seem research often gets "bogged down" because it has spent a great amount of its time in duplicating studies to get basic information. Recognition of this fact is evident by the far -sighted researchers who have compiled such cumulations of past research and studies in the field of television and mass communica- tions. One such work is a unique bibliography of more than 1700 studies of the mass media. It was compiled and annotated by 18Curl, "Why Do Films Fail?" p. 68. 21 Professor F. E. Barcus of Boston University, who conducted a content analysis of all available content analysis studies from 1900 to 1958. Dr. Barcus classified more than a thousand items by sub- ject which included television and film. 19 Another specialized source is An Inventory of Instructional Television Research by Dr. Hideya Kumata. It is composed of seventy-one research projects dealing with the effectiveness of instructional television. Reid and Macheman, under contract with the Office of Education, U. S. Department of Health, Education, and Welfare, authored a document entitled Research in Instructional Television and Film, which consists of 333 summaries of studies concerned with instruc— 21 tional television and film. Chu and Schramm did a broad survey of research on instructional television entitled Learning from Tele — 2 vision: What the Research Says. 2 One of the finest resources for 19David M. White, "A Critique of Bibliographic Matter in Mass Communications," People, Society, and Mass Communica- tions, p. 585. 2 0Kumata, An Inventory of Instructional Television Research. 21U. S. Department of Health, Education, and Welfare, Research in Instructional Television and Film (Washington, D. C. : U. S. Government Printing Office, 1967). 2Godwin C. Chu and Wilbur Schramm, Learning from Television: What the Research Says (Washington, D. C. : National Association of Educational Broadcasters, 1967). 22 television and film research is the Educational Resources Information Center (ERIC). ERIC is a nationwide information network for acquiring, selecting, abstracting, indexing, storing, retrieving, and disseminating the most significant and timely educational research reports and projects. Stanford University is the clearing house for educational media and technology. Probably one of the most extensive instructional film research compilation efforts was the series of Technical Reports prepared by the Instructional Film Research Program at Pennsyl- vania State College. They were jointly sponsored by the Department of the Navy and theDepartment of the Army. The studies were compiled into three documents: Instructional Film Research 1918- 195023 and Instructional Film Research Reports, Volumes I and II. 24 These two volumes include sixty -one reports along with analyses and conclusions on various phases of film research. 23U. S. Department of the Navy and U. S. Department of the Army, Instructional Film Research 1918- 1950, by Charles F. Hoban and E. G. van Ormer, Technical Report No. SDC 269-7 -19 (Port Washington: Office of Naval Research, 1951). 24U. S. Department of the Navy and U. S. Department of the Army Co-operating, Instructional Film Research Reports, Vols. land 11, by C. R. Carpenter and L. P. Greenhill, Technical Reports SDC 269-7 -1 through 61 (Port Washington: Office of Naval Research, 1953 and 1956). 23 It would certainly seem that the knowledge that can be gained from those who have gone before and the use of their data would justify a "survey of the literature" as a valuable asset to the film and television researcher. Overview and Extent of Present Air Force Audience Effectiveness Evaluation Program A Special Preparedness Subcommittee from the Committee on Armed Services, United States Senate, charged the Department of Defense To provide a system whereby higher headquarters can evaluate whether troop information materials have the desired effectiveness, how extensively they are used, what areas need improvement, whether there is adequate understanding of the objectives of the program, and whether an individual commander is giving adequate emphasis to the training. The term troop information materials has been interpreted by the Office of Information for the Armed Services to include film and video tapes prepared and contracted by the Armed Services for showing to the troops. As a result, a staff position entitled Chief of Information Retrieval and Evaluation was established to organize and administer such a program. 25U. S. Congress, Senate, Committee on Armed Services, Military Cold War Education and Speech Review Policies Report, 87th Cong., 2nd sess., 1962, p. 7. 24 The Department of the Air Force designates the Aerospace Audio-Visual Service (AAVS), a service under the Military Airlift Command, as the single manager for producing, procuring, dis— tributing, and preserving motion pictures to meet Air Force needs. Air Force Regulation charges the Aerospace Audio-Visual Service with the responsibility to "evaluate each motion picture production requirement and each script; to conduct evaluation of motion pictures after their use by intended audiences; to determine effectiveness of each film in terms of original purpose and presentation technique; and to apply data gained to improve the quality of film productions. "26 The following is an overview of the Aerospace Audio-Visual Service and its mission: The Air Force program holds the Aerospace Audio-Visual Service responsible for providing photographic and Television Services in response to Headquarters United States Air Force requirements. More than 2, 500 personnel, 35 per cent of whom are civilian, support world wide Operations in seven basic areas: (1) documentation of significant Air Force actions, (2) instrumentation photography of missile and space activities and weapons effects, (3) motion picture production, (4) motion picture and still archival depositories, (5) film distribution, (6) television utilization, and (7) training. Although the Aerospace Audio-Visual Service began to con- solidate production, distribution, and depository functions with its recently moved Headquarters, at Norton Air Force Base, 26U. S. Department of the Air Force, Programing and Producing Motion Pictures, Air Force Regulation No. 95- 14, Washington, D.C. (May 12, 1967), par. 5, p. 3. 25 San Bernardino, California, in the fall of 1968 one production unit was at Orlando Air Force Base, Florida; a second produc- tion unit and film depository were at Wright —Patterson Air Force Base, Ohio; the 1352nd Photographic Group produced films at Lookout Mountain Air Force Station, Los Angeles, California; and the Air Force Film Library Center at St. Louis, Missouri, supported continental United States requirements and five regional world wide libraries. In addition, missile activi- ties at Vandenburg Air Force Base, California, were covered by the 1369th Photographic Squadron; 600th Photographic Squadron provided documentation in Vietnam and Thailand; a series of detachments and operating locations supported other Air Force activities throughout the United States, Europe; and two mobile television units were assigned to the Air Force Logistic Command. 27 The specific mission of AAVS is to perform the following services for the Air Force: Audio-visual services and products to meet the require- ments of Hq USAF, the major commands, and separate operating agencies. Instrumentation and technical photography in support of Air Force research and development activities. Armament recording photography services in support of tactical operations. Administrative and technical management of audio-visual activities or services for, or in support of the major com- mands. Technical advice and assistance on the operation and man- agement of authorized photographic and television facilities in the Air Force. 2 27Jay M. Sedlik, "Diagnostic Pretesting of the United States Air Force Instructional Motion Picture Theory and a Pro- posed Model" (unpublished Ph. D. dissertation, University of Southern California, 1969), pp. 3-4. 28U. S. Department of the Air Force, Organizations and Functions, Headquarters Aerospace Audio-Visual Service Manual 23-1, Norton Air Force Base, California (September 26, 1969), p. 1. 26 The Air Force in fiscal year 1968 (July 1967 - 1968) produced 605 reels of motion pictures at a cost of nearly $4 million. More than four -fifths of this film was done by Air Force military and civilian personnel. Approximately ninety reels were completed by commercial contract at a cost of $710, 000. The budget for all photographic activities was $12. 8 million. This includes the processing of 50 million feet of film (16mm and 35mm) and the distribution of 385, 000 prints. 29 The Air Force has 302, 000 films on inventory; 80, 000 to 100, 000 of these are in the film library at Norton Air Force Base, California. The remaining films are being used by Air Force units world wide, and in support of Air National Guard units, Reserve Officer Training Corps (ROTC), Civil Air Patrol, Air Reserve units, etc. ‘ The process of requesting and producing a final film or video tape from AAVS is a long and complex procedure. The pro- cess usually starts with an idea, at a local Air Force base or Major Command Headquarters, that a film is needed to accomplish a cer- tain task, such as instructing, documenting, etc. An official Motion Picture Production Request (Air Force Form #411), which specifies objectives and justification, is submitted through the higher echelon 29Sedlik, "Diagnostic Pretesting of the United States Air Force Instructional Motion Picture Theory and a Proposed Model, ” p. 207. 27 of command to the Aerospace Audio-Visual Service Headquarters for evaluation. If it is approved, it is assigned to a production schedule. A pre -production meeting is held with the producer, a technical advisor, and a Major Command representative from the command that originated the request. At this time it is determined what type of presentation would best be suited for the project-- television, film, or perhaps slides. If a film is selected, it is scripted. The script is sent to the Command for clearance; then it is forwarded to a Headquarters Air Force staff agency for approval; and then, depending on the subject matter, to a staff agency in the Department of Defense. If all levels of command give their approval, the production proceeds with the producer securing his talent, taking his shots on location, etc. The final film is evaluated by the Tech- nical Advisor, who has approving authority by the Command for the technical aspects only. The producer evaluates and approves the production. The film is printed and sent up the ladder for final approval. It is then copied into the number of prints determined necessary and distributed by the Film Library. Included with some of the films is an Air Force evaluation form (Air Force Form #482), which is used as a guide for a film evaluation. The purpose of this form is to solicit information that will be valuable in planning new productions and effective use of funds and resources. This form is 28 filled out; and the information and the film are returned for redistribution, or the form is returned and the film is kept by the using agency for a long term requirement. This procedure in most respects is not unlike that to which the commercial film companies are exposed. If a school or large corporation had a requirement for a film, it would submit a request. A pre -production meeting would be held. The script and test film would probably require approval from top management. Occasionally a similar attempt is made by the commercial film producer to evaluate his product. Conclusions on the State of the Present Air Force Audience Effectiveness Evaluation Program The above summary is perhaps oversimplified, and this procedure may appear to be very logical and sound; but the total time required for this process to take place is between two and three years. Colonel Gallerani, Commander, Aerospace Audio- Visual Service, indicated: To proceed with the paperwork through the channels, to get it [film request] approved and back, takes about a year and a half or two years. 30Colonel Altero Gallerani, United States Air Force, personal interview at the Headquarters Aerospace Audio -Visual Service, Norton Air Force Base, California, March 25, 1970. 29 Colonel Alexander, Commander of the AAVS Center, adds: The average production timerequired for a motion picture of average length [fifteen to twenty minutes], average com- plexity, is six months. It is conceivable that because of the recent consolidation of resources and the programmed potential of television taping, this time could be reduced. There are often problems which affect the schedule outside the production and process structure. For example, if the film is an instructional one, such as the use of a new weapon system, the production may be ready to shoot before the final model of the weapon is ready--hence a delay. If the process proceeds normally and the film is scripted, then disapproved at that point, it would seem that a great deal of money has already been wasted. If the produc— tion is completed and the final print is sent up the chain of command to be approved, and it is not approved, a greater amount of time and money has been wasted. If it passes all personal evaluations, it is sent to the field for use, to hopefully achieve whatever the basic pur- pose was two or three years previously. The evaluation form that accompanies the film is certainly not an accurate indication as to the effectiveness. . 1Colonel Richard Alexander, United States Air Force, personal interview at the Aerospace Audio-Visual Service Center, Norton Air Force Base, California, March 25, 1970. 30 The following examples are taken from an actual evaluation form, cited by Captain Jay Sedlik, 32 which solicits information in response to such inquiries as: HOW DOES THE EFFECTIVENESS OF THIS FILM COMPARE WITH THAT OF PRECEDING FILMS IN THIS SERIES? ( ) MORE EFFECTIVE ( ) LESS EFFECTIVE ( ) ABOUT THE SAME The question implies that the audience has seen previous films of the series. What if they had not, or if this was shown for the first time to a group of basic trainees new to the service? What is meant by effec- tiveness? Is the purpose of the film to teach, entertain, make the audience aware, or go out and talk up the program? In the final anal- ysis, if it is discovered through the responses that the film is about the same as the previous ones, what does that reveal? Maybe they were all bad, or all good. If the results reveal that the film was less effective, so what? It has already been produced and the money spent. DO YOU THINK THE FILM IS MORE EFFECTIVE AS AN ”OPEN-END" ( ) OR WITH THE PROJECTED CONCLUSION FOLLOWING DISCUSSION ( )? 32Sedlik, "Diagnostic Pretesting of the United States Air Force Instructional Motion Picture Theory and a Proposed Model, " p. 126. 31 A film on Moral Leadership is generally shown to all Air Force personnel. Some of them would know what "open-end" meant, and some probably would not. Unless it was pointed out in the film, how would the audience know if the film would be more effective with a projected conclusion or an open-end? HOW MANY PEOPLE IN YOUR AUDIENCE ? This question, if answered by every person filling out the question- naire, would reveal many different answers. If the questionnaire is being filled out by one individual, perhaps the Chaplain, who normally conducts programs using this type of film, he will most likely not be representative of the troops that will be seeing this film en masse. He perhaps can evaluate the technical accuracy of the message, but probably not the effectiveness of the film on the audiences. If this form is originally intended to be answered by only one person, the responses to the first two questions would be greatly influenced or biased because of the greater awareness of the message on moral leadership that the Chaplain bring with him when he views the film than that of the normal audience. The only real apparent value of this type of questionnaire is that it will provide utilization figures on how many showings the film made and how many people were in the audience, not how many 32 watched it or were affected by it. The question needs to be asked, what is meant by effectiveness? The word effectiveness is mentioned throughout the evaluation questions without any definition. Effective- ness can mean the achievement of whatever objectives were set out as the purpose of the film. One of the areas of concern is that often the objectives are so poorly thought out, defined or written that those who have to interpret them are led down the wrong path. Although it is not the intent of this work to be a study on objectives, it is pertinent to look at them in enough detail to make the reader aware of this potential problem area. John B. Gilpin, Research Associate, Earlham College, says: Everybody talks about defining educational objectives, but almost nobody does anything about it. Books on education often stress objectives; "how to" papers on programming list "defin- ing objectives" as a first point; and training materials such as films and film strips often contain a description of the "objec- tives. ” But how often are educational units, whether large or small, prepared in response to the questions: 1. What is it that we must teach? 2 . How will we know when we have taught it? . 3. What materials and procedures will work best to teach what we wish to teach? Not only must these questions be answered to instruct effectively, but the order in which they are answered is impor- tant. The first question must be answered before the other two. . The probable reason that objectives are usually stated poorly is that few people know how to proceed. This is not 33 surprising, because little has been written on the preparation of objectives. One approach to constructing effective objectives is taken by Dr. Robert F. Mager. Dr. Mager defines objectives as an intended behavior or change on the part of the learner. He indicates basically that not only does one have to describe and measure the intended change, but one must go a step further and describe how the indi - vidual will demonstrate the change at the time one' s influence over him ends. He uses the words "terminal behavior" to define this con- dition. He indicates that an effective objective must communicate intent. This can be accomplished by the use of words open to fewer interpretations. For example, ”What do you mean whan you say you want a learner to 'know ' something? Do you mean you want him to be able to recite, to solve, or to construct?” Little has been com- municated because to "know" can mean many things. Then how does one go about describing the intended behavior of the learner? First, identify the terminal behavior by name; you can specify the kind of behavior that will be accepted as evidence that the learner has achieved the objective. Second, try to define the desired behavior further by describing the important conditions under which the behavior will be expected to occur. Third, specify the criteria of acceptable performance by 33Robert F. Mager, Preparing Instructional Objectives (Palo Alto: Fearon Publishers, 1962), forward, quoting John B. Gilpin, supra. 34 describing how well the learner must perform to be considered acceptable. 34 If it is assumed that objectives are important and need to be considered, it may be easier to understand that because of the involve- ment of many different people at many various levels in a film' 8 production, the objectives are often lost. These objectives should be carefully defined at every step of the process from the initial point when the originator specifies them on the official Motion Picture Production Request to the person who watches the finished product. If they are poorly worded and overgeneralized in the begin— ning, and they are not redefined at least in the pre -production meeting, they will become increasingly ambiguous at each stage of the film' s development, thereby greatly increasing the Chances that the finished product will be ineffective. In the final analysis, if the film is truly ineffective in achieving its intended objectives, the time and loss of productivity of all those who saw the film has probably been wasted. This could mean thousands of accumulated man—hours. Dr. Kanner, Chief, Audio-Visual Applications Office, Depart- ment of the Army, says: "Suppose the film turns out not to 34Mager, Preparing Instructional Objectives, p. 12. 35 be very good; we've lost $50, 000 to $60, 000. "35 (Major Archer, AAVS, Chief of Production, estimates costs at $15, 000 to $20, 000 for the average film. )36 If the films are not effective, the additional expense of distribution is wasted. Captain Paula Kerns, Chief of the Film Library and Distribution Branch, says: "One important thing is not the amount of money spent in a given year on films, but the amount of money, time, [people], equipment, and postal services to handle and send these ineffective films to the field. ”37 Once these films arrive, some may be rejected and not used. Those that are shown may still not be effective. The cost process of returning them is repeated plus the additional expense of storing them. It is conceivable that an ineffective film may be shown to servicemen for several years without an indication that it is not accomplishing its goals. Even more alarming, it may be having a boomerang or opposite effect than originally intended. A good example of such a situation occurred in 1941 as a result of a series 35 Dr. Joseph Kanner, personal interview, Washington, D.C., March 19, 1970. 36Major Norma A. Archer, United States Air Force, personal interview held at the Aerospace Audio-Visual Center, Norton Air Force Base, California, March 25, 1970. 37 Captain Paula V. Kerns, United States Air Force, personal interview held at the Aerospace Audio-Visual Center, Norton Air Force Base, California, March 24, 1970. 36 called Battle of Britain. It was produced by a noted Hollywood producer. The purpose centered around the belief that the new Army trainee did not know much about his foe, the Germans, and that he had no idea of how ruthless and cruel they were. The objec- tive of the film was to instill a "fighting mad" attitudein the troops against the terrible Nazis. A superb series of twelve films was produced showing the Nazis running over Poland, defeating France, efficiently using their air power, mechanized forces, _et_a_l_. The film was being shown to the new trainees as a part of their indoctri - nation. However, when an effectiveness evaluation was conducted on several groups before and after seeing the film, some startling results were discovered. The study was designed to measure the trainee' s attitude about the Germans. The findings indicated a "boomerang" effect. Instead of wanting to go out and defeat the Germans, the trainees had more fear of the Germans. They thought that if the Germans were so strong, the United States was fighting a losing war and should call it quits. 3 It would seem that the possibility of this type of circum- stance, or even one of lesser consequence, existing within the film- making organization would be of great concern to all staff levels involved with the production and distribution of such a product. 38 . . Dr. Kanner, personal 1nterv1ew. 37 Certainly the success of a commercial film enterprise depends on the effectiveness of the product, which is often reflected in sales and profit. However, since the government is not in the business of making a profit, effectiveness must be measured in other ways. Perhaps the posing of several questions to management would emphasize the extent of the present Air Force system in evaluating its film and video tape products. How does the Air Force know the films and video tapes are effective? What is the adequacy of the evaluation method the Aerospace Audio-Visual Service is now using? Can the Air Force afford to not know how effectively its money is being spent in the area of film and television? The possibility of negative answers to these probing questions would merely indicate that the Air Force may not have an adequate audience effectiveness evaluation program. The Aerospace Audio -Visual Service recently became aware of this and is investigating methods to develop such a program within the AAVS Center. Colonel Bjoring, Deputy Chief for Opera- tions, Aerospace Audio -Visual Service, indicated: We at AAVS have recently been directed by Headquarters United States Air Force to develop a capability for diagnostic pre and post testing of Air Force Audio-Visual products. 39 3 9Letter from Colonel Donell E. Bjoring, United States Air Force, February 10, 1970. 38 The Commander of the AAVS Center, Colonel Alexander, points out: The present plan is for the Audio-Visual Center to establish some kind of pretesting and posttesting in order to evaluate the potential and actual effectiveness of the films and television tapes which we make. The problem of conducting effective evaluation appears complex. The expense seems large. The qualified personnel to perform such a task appear to be few in number. This state of affairs is not unique with the military, but applies as well to civilian institutions and commercial film -making companies. There is no question that the many Air Force levels of management dealing with audio —visual services are aware of this problem. Colonel Leo Beinhorn, Office of Information, Secretary of the Air Force, states: Communications research is extremely important. I think we all know that. If you start with the premise of having clear- cut objectives, add to it a series of results, and then test and interpret—-the results should indicate something. And frequently it does. You don' t keep testing until you get the results that you want-~nor do you blindly continue to produce until you either force it down the throats of your audience or have them totally reject it. We respond accordingly--and hopefully we' ve achieved a measure of understanding. I can't emphasize too much the need to gain an appreciation for the people we' re talking to, and the way that we' re talking. OColonel Alexander, personal interview, AAVS Center, March 25, 1970. 41Colonel Leo Beinhorn, USAF, Chief Internal Information Division, in a speech for the Production and Planning Conference, Headquarters Aerospace Audio -Visual Service, January 26, 1970. 39 The problem, as originally stated, is that the present system for audience evaluation of Air Force films and video tapes is not an adequate one. CHAPTER II METHODS AND TECHNIQUES OF AUDIENCE EFFECTIVENESS EVALUATION There are a considerable number of methods and techniques available to the researcher to measure the impact or effectiveness of a film or video tape on its audience. Each one has a certain unique feature which sets it apart from the others. These methods are divided into two categories: mechanical and nonmechanical. Those classified as mechanical involve the use of some type of device or machine. Those classified as nonmechanical are generally pencil and paper techniques and rely heavily on the human admin- istrator. Me chani cal The Lazarsfeld -Stanton Program Analyzer The Lazarsfeld- Stanton program analyzer, originally con- structed to test radio programs, is a device which enables a subject 40 41 to record his reaction to a film as he watches it. 42 The reactions are recorded in terms of Like, Dislike, or Indifference. The viewer holds a green button in one hand and a red button in the other. If he likes a particular part of the motion picture, he presses the green button and keeps it pressed down for as long as his approval continues. If he dislikes what he sees and hears, he uses the red button in a similar fashion. If he feels indifferent, he does not press either button. The buttons are connected electrically to a battery of pens, resting on a continuously moving paper tape. When a button is pressed, the corresponding pen is jogged down an eighth of an inch perpendicular to the tape. The pen remains in this position until the button is released. Timing pens serve to synchro- nize reactions with the program. The result is a continuous record, accurately timed, of the listener' s feelings during each second of the broadcast, or the audience' 5 reaction to a motion picture. Responses are thus caught at the time they are experienced, and possible errors in retrospective reporting of reactions are reduced to a minimum. The following steps are usually involved in testing a film in this way: 42Paul Lazarsfeld and Frank Stanton, eds. , Radio Research (New York: Duell, Sloan and Pearce, 1943). 42 The content of the film is analyzed to determine the types of responses likely to be elicited. (This list of potentially significant elements also serves as an interview guide.) The film is then shown to representative segments of the intended audience (in groups of ten, since the program analyzer at present normally accommodates only this number) who record their immediate reactions. Questionnaires are given to each subject before and after the film is seen. These questionnaires provide information about the respondent and serve as one measure of the educational or attitudinal effects of the film. These steps are frequently accompanied by group inter- views to determine the detailed nature of the subject' S reaction. The interviews are recorded and the final analysis is then based on three sets of data: program analyzer' s graphs, questionnaires, and transcripts of the interviews. The total number of Likes and Dislikes is usually presented graphically in the form of a profile. It is thus possible to determine at a glance where the peaks and valleys of audience reactions are. The focused interviews following the film can then be directed to determine what in these scenes was particularly liked or disliked. 43 The Ci rlin Rea ctograph Basically the Reactograph, as developed by Bernard D. Cirlin, is identical to the Lazarsfeld- Stanton program analyzer. Its technical development, however, differs from the original device. The recorder unit of the Reactograph consists of a Chemi - cally treated moving tape against which rest fifty electrical styli. Two buttons are attached to each stylus by a pair of wires. As the respondents View a motion picture, they press the red button to express their dislike and the green button to express their approval. As they press the button an electrical current passes through the stylus and burns a mark into the chemically treated paper. If they press the Dislike button, a dotted line appears on the paper. If they press the Like button a solid line appears. If they have no reaction, no mark is shown. Along the left hand margin of the tape the time intervals are marked so that it can be seen which individual has expressed approval or disapproval for every second of the motion picture. The recorder unit can be attached to a totalizer. The totalizer shows in graphic form the total number of people express- ing likes and dislikes for each two -second interval. The test 43Leo A. Handel, Hollywood Looks at Its Audience (Urbana: University of Illinois Press, 1950), p. 49. 44 produces two detailed records. One is a chart showing a like and dislike profile; the other is the recorder tape which shows how each individual reacted during the picture. In addition to getting a reaction graph for the total group, a subgroup totalizer may be used. This subgroup totalizer would produce separate like and dislike profiles for men and women, people of different age groups, or for any other breakdown of the audience that may be desired. The Hopkins Electric Televoting Machine This device is used by Audience Research, Inc. , in con- nection with its Preview Jury system. 44 The picture is screened before a representative audience. In addition to the basic break- downs such as sex, age, and income, the test audience is divided into two predetermined groups representing the ratio which wants and which does not want to see the picture in question. This ratio is determined in previous surveys using the title, the principal cast, and a bare outline of the story as a guide for the interviews. Every previewer keeps a dial in his hand which enables him to record five degrees of response to the scenes as they pass in succession on the screen. The following dial positions are provided. 44Ibid., pp. 49-50. 45 Like Very Much Like Neutral Dull Very Dull The reaction recording machine consists of a compactly built mechanism similar in appearance to a slightly oversized portable typewriter. The machine reels off a chart -line on a white tape about five inches wide, recording the mean average reaction of the assembled observers to each scene in the motion picture being tested. The Hopkins device can compute the cumulative reaction of a hundred viewers at a time. The previewers are given questionnaires to fill out before and after the preview. The answers to the questions furnish infor- mation amplifying the group information expressed in the profile curve produced by the Televoter. The Schwerin System The Schwerin method or "number cueing" method is related in principle to the preceding procedures of testing films while the . . . 45 . screening 13 In progress. SchwerIn, however, does not use any mechanical devices to obtain the "profile" of a picture. He relies 45W. R. Weaver, "Studios Use Audience Research to Learn What Pleases Customers, " Motion Picture Herald (July 20, 1946). 46 on questionnaires filled in by the participants at certain intervals while the picture is being shown. Schwerin developed this technique in the testing of radio and television programs. A sample audience which numbers approximately four hundred is roughly preselected on the basis of income, education, sex, age, and occupation. At the theater, each person in the audience completes a series of questionnaires which contain infor- mation concerning his movie-going habits, his general likes and dislikes by type of motion picture entertainment, and his specific attitude toward and familiarity with the type of picture being tested. A test director indoctrinates the audience about the method of indicating reactions during the session. During the screening, large numbers are flashed on a separate screen at intervals as the picture is being run. These numbers are the respondent' s "cues, " and correspond to numbers printed on the questionnaire. Opposite each of these printed numbers are three boxes labeled "good, " I H H H ”fair,’ or "poor" (or "interesting, mildly interesting, or "not interesting, " depending on the type of picture being tested). When a number is flashed, the respondent checks his reaction to that portion of the film which he has seen since the last number flashed. After the picture has ended, the test director leads an audience discussion period. Members of the audience are invited to 47 express their opinions on specific portions of the picture they have just seen, from whatever point of View they desire. These comments, which may range from criticisms of acting, casting, production values, direction, music, etc. , to strong endorsements of those and other points, are recorded. Schwerin then matches his sample to the universe. In the process, many of the questionnaires will be discarded to bring respondents into the proper ratio. The final sample will include 250 to 350 completed questionnaires and will be a true sample of the particular universe being investigated by the test. The cued reac- tions of this matched sample are reduced to a profile of the whole audience' 3 reaction to the film. Physiographs The term ”physiograph” is used to refer to any instrument that measures and records one or more physiological responses. These responses are sometimes useful in interpreting process response to communication, and so their instrumentation is impor- tant. The physical responses with which audience analysts are concerned are those that can be measured peripherally--that is, 46Jeffery J. Auer, An Introduction to Research in Speech (New York: Harper 8: Brothers, 1959), passim. 48 without penetrating into or under the skin. A surprising variety of such processes are available. They include the psychogalvanic skin response (GSR), heart rate, blood pressure, Skin temperature, capillary dilation, the palmar sweat response (PSR), gross patterns of activity in the brain, breathing, and covert muscular tensions. The skin offers some resistance to the passage of a mild electrical current. Under conditions of arousal or stress, this resistance drops markedly. This response of the skin to certain stimuli is what is called the GSR. By passing a current through the skin and recording its resistance continuously on a strip of graph paper (called a graphic level recorder), we may subsequently corre- late the dips in skin resistance with elements of the message being presented at the time. The heart responds to environmental stimuli by increasing or decreasing its pumping rate and by increasing or decreasing the strength of its contractions. The rate in particular may be measured from an electrocardiogram (EKG), which records the electrical potentials associated with heart action. It may also be recorded by placing a microphone against the chest and recording the amplified sound of the heartbeat. Blood pressure, which results partly from changes in heart action and partly from other factors in the circu- latory system, usually is measured in a way that is familiar to most 49 of us from our occasional visits to the doctor. Transduced to an electrical signal and amplified, the measurement can be recorded continuously on a graphic level recorder. As we know from such experiences as blushing, blanching, hot flashes, and cold sweats, the surface temperature of the skin responds to extreme emotional stimulation. A thermistor, which is a sort of electronic thermometer, may be fixed on the surface of the skin to measure the rather minute temperature differentials that accompany emotional arousal or depression. The resulting skin temperatures may be read from a meter or recorded permanently and continuously on a stripchart recorder. Probably closely associated with skin temperature, though by no means inflexibly tied to it, is the action of the capillaries. These dilate and contract in response to a great range of stimulation (for example, cigarette smoking usually causes them to contract) including emotionally arousing stimuli. Their dilation may be measured by an electroplethysmograph--a device that uses a photo- electric cell to measure the amount of light passed through the skin from a source of known standard intensity. The amount of light transmitted varies with capillary dilation. The palms of the hands and soles of the feet perspire in response to emotional stimulation. Through the centuries, sweaty 50 palms have been taken as a sign of anxiety, fear, or great anticipation. In recent times it has become possible to quantify the amount of palmar sweating in several ways and to use the resulting measure as an index of emotional arousal. Perhaps the most satisfactory method of measuring palmar sweating is to seal a small, airtight capsule on the palm and to pass dry air through the capsule and over a sensitive electronic humidity sensor. As the palms sweat, they release moisture into the dry air, and so the humidity of the air reaching the electronic sensor is a direct and continuous measure of palmar sweating. Patterns of electrochemical activity in the brain may be sensed by electrodes placed on the surface of the scalp. Very useful for diagnosing brain malfunction, these measurements have in recent years been thought to hold possibilities for identifying char- acteristic patterns attributable to subjective states. The work on ' as electroencephalograms (or EEG' s) are called, ”brain waves, ' is still incomplete. Little is known about the interpretation of EEG patterns produced by auditors while listening to spoken messages. Breathing, which is deep, slow, and regular during sleep, may become shallow, rapid, or irregular during aroused states. By means of an air-filled tube tied around the chest, it is possible to measure changes in breathing. A device for measuring respiration 51 is called a pneumograph, and its mechanical pressure readings can be transduced into electrical signals and recorded on a strip- chart recorder. Finally, small muscle contractions characterize many states of attention and arousal. These may be measured by record- ing the electrical potentials that all muscles create when they contract. A small electrode is attached to the surface of the skin, which records the small currents in the muscle immediately below. These tiny potentials can be amplified and recorded in the same manner as other physiological data. These measurements may be taken separately or several of them may be taken simultaneously and recorded on a single, wide strip of chart paper. An instrument which makes recordings of several physiological responses simultaneously is called a polygraph. The original polygraph was the lie detector, but more sophisticated polygraphs are constantly under development for research into human behavior, including response to communication. Such instruments are useful for measuring process responses. As we have seen, there is still much to learn about the interpretation of the data they yield. The Dynascope The Dynascope is a device that takes pictures at frequent intervals of people who are watching a television set and of the picture 52 shown on the set at the time the photograph of the Viewer is taken. 47 From these pictures it is possible to obtain some information as to how attentive or inattentive the audience was. The Dynascope has been used only experimentally and is subject to some drawbacks. If pictures are taken at intervals of longer than five seconds, something may be missed. Lighting in the television room has to be excellent. And there is a question as to whether the use of this device conditions television viewers. How— ever, the inventor has evidence that it does not. He reports that its presence has not prevented members of some cooperating house- holds from making love in the television room. This device appears to be too expensive and impractical for use in a classroom. Infra -Red Film One method still being experimented with involves the photographing of the audience from a concealed camera using infra- red film. The audience does not know the photograph has been made. Since it is timed to a particular second, the director and producer can get a true picture of the audience' 3 interest. Dr. Kanner used this technique in 1951. 7National Industrial Conference Board, Evaluating Media: Studies in Business Policy, No. 121 (New York, 1966), p. 64. 53 We put people in a van, viewing the film and using infra -red lighting, so the students were not aware. We photographed them for a motion picture record of facial expressions while they were watching the film and then ran both simultaneously. You had the film that the students were watching on the right and you had the fihn of the students watching it on the left. We tried to correlate; that is, if we saw the students falling off, we'd wonder how much they would learn. It was fairly close. 4 Distractor A device called a "distractor" is used by Dr. Edward L. Palmer, the research director for television' 5 "Sesame Street" workshop. It is simply a slide screen placed next to the television screen of a video tape playback unit on which slides are now and then flashed in an effort to distract the viewer from the "Sesame Street" showing. How frequently the attempt is successfulis an indication about the effectiveness of the "lesson" being televised. 49 Another technique being tested is the use of concealed micro- phones to pick up conversations of the audience leaving the theater after the preview. In one such test six concealed mikes were used, and due to the slow flow of the audience from the auditorium and lobby, it was possible to pick up uninhibited reactions to the picture. 8 4 Dr. Kanner, personal interview, Washington, D.C. , March 17, 1970. 49Children' s Television Workshop, "Children Help Share Their Own TV Series, " News Release, National Educational Tele— vision (NET), n.d. 54 The major advantage of these devices is that they locate the sequence which gives rise to favorable or unfavorable responses more reliably than an interview alone, which has to depend on the respondent' s recollection of what has caused his reaction. This enables a producer to study in detail important episodes of the picture evaluated by standards of judgment that closely approximate those of the average movie -goer. It makes it possible to determine audience reactions to a particularly crucial statement, the recogni - tion of a subtle pun, the reception of an especially expensive set, or other such matters. One of the most important difficulties lies in the different meanings whi ch may be expressed in the subject' 3 pressing buttons or turning a dial. Difficulties of interpretation will arise, for instance, when characters of the picture are intentionally designed to create a negative reaction from the viewer. This negative reaction to the "bad man” of the picture may register as a dislike on the tape of a program analyzer, or "neutral" on the televoting machine, even though the interest in the sequence may be actually very high. This limitation may be overcome by intensive interview- ing. The difficulty of interpreting correctly the meaning of dislike reactions may also be reduced in advance by means of detailed instructions. 55 It is believed by some critics of these mechanical devices that members of test audiences may forget to actuate the apparatus when they become especially interested in the picture. Tests con- ducted by Bernard D. Cirlin and other experiments, however, have shown that after a conditioning period of about three minutes, the activation of the like and dislike buttons as used for the Lazarsfeld- Stanton analyzer and the Cirlin Reactograph is done unconsciously and automatically. The subject' 3 actions may be compared to those of the driver of an automobile where the application of the brakes or gearshifting is done without a conscious effort. There is a further purely mechanical difference between the analyzer used by Audience Research, Inc. , and the other devices. The release of the buttons of the program analyzer results in an automatic return to the neutral position. If, therefore, a respondent forgets to actuate the reaction mechanism, he registers indifference. The dial used by Audience Research, Inc. , does not revert auto- matically to neutral; and a respondent who forgets to indicate his reactions may have left the dial in a positive or negative position and in this manner distorts the results. In the case of the program analyzer, distortion would come about only when there is a question as to whether the respondent' 3 interest is of a positve or negative nature. 56 All program analyzers are based on the original device invented by Paul Lazarsfeld and Frank Stanton. The Lazarsfeld- Stanton program analyzer' 5 capacity is at present low, limited to ten persons. The individual reactions are not combined automati - cally but must be summed up in a time —consuming clerical operation. Cirlin and Audience Research, Inc. , streamlined the basic device and made it more practi cal for the actual commercial use in the motion picture field. The device used by Audience Research, Inc. , differs in some aspects from the Lazarsfeld-Stanton and Cirlin analyzers. The latter permit the recording of the individual reactions of every respondent. This method makes it possible to determine the actual number of persons who express like or dislike at a given time, and also provides permanent individual records to be used as bases for interviewing later. Since the device used by Audience Research, Inc. , does not record the individual reactions, the interpretation of results is apt to be less clear. A plus ten reaction, for example, may mean that only a few viewers expressed an opinion and like the specific sequence of the picture; or it may be interpreted to indicate that all persons registered a reaction, and that more liked than disliked the scene. Moreover, there is no guide provided for subsequent interviews with the individual respondents. 57 Confusions introduced into a test by the more complicated button operation required to register five possible reactions may interfere with listening. While there can be no doubt as to the desirability of a graded reaction, it is also important that the psycho- logical nature of the listening situation should approximate as closely as possible the ease of listening at home. A simplification of registering devices seems particularly important for the testing of motion pictures. A motion picture test takes a considerable amount of time in the first place, a fact that increases the desirability of a simple registering mechanism that can be operated almost automatically. Also, fihns require the visual attention of the audience; and it is important not to disrupt the concentration of the respondent by a device that might distract his eyes from the screen. There are two basic criteria by which the worth of any measuring instrument can be determined -- reliability and validity. Knowing the reliability of the measuring device enables its user to know to what extent the results are consistent. Validity shows whether the device is measuring what it purports to measure. One type of validity test is made by setting up two test groups. The technique of testing the same program on different 50Handel, Hollywood Looks at Its Audience, pp. 56-57. 58 audiences can also serve to indicate what size of sample is needed to produce a stable program profile. Validity may be determined by comparing the results of the measuring device with results obtained by different methods of established validity. By comparing the evaluation of a film by the listener, as expressed on a questionnaire, to the second -by—second record of the program analyzer, it is possible to determine to some extent the validity of these second -by -second responses. The reactions of the audience obtained in this manner are especially significant if they are compared with the results of surveys covering other motion pictures. Nonmechani cal The mail questionnaire method is, in principle, very similar to the personal interview approach. Instead of being inter- viewed, the preview patrons are handed questionnaires with the request to take them home, fill them in within a few days, and mail them back to the research office. The underlying purpose of this technique is to have some time elapse between the respondent' 8 exposure to the film and his expression of appreciation or criticism. 51Theodore M. Newcomb, Ralph H. Turner, and Philip E. Converse, Social Psychology (New York: Holt, Rinehart, and Winston, Inc., 1966), pp. 510-515. 59 This method provides for answers to specific questions and, therefore, draws specific reactions and permits an intelligent, detailed tabulation. Also, it indicates the classification data (age, sex, occupation, etc.) of the respondent and in this manner makes it possible to determine whether or not a representative sample has been obtained. Several different paper —and —pencil tests have been developed to measure attitudes. Of these tests, four have been fairly highly refined and have been used most extensively. These major techniques are: Thurstone' 3 method of equal ~appearing intervals, Likert' 8 method of summated ratings, Guttman' s scalogram, and Osgood' s semantic differential. Each of these techniques to be discussed makes different assumptions about the nature of the test items that are used and the kind of information they provide about a person' s attitudes. How- ever, there are certain basic assumptions which are common to all of these methods. First of all, it is assumed that subjective attitudes can be measured by a quantitative technique, so that each person' 3 opinion can be represented by some numerical score. Secondly, all of these methods assume that a particular test item has the same meaning for all respondents, and thus a given response will be scored identically for everyone making it. Such assumptions may not always 60 be justified, but as yet no measurement technique has been developed which does not include them. Thurstone' s Method of Equal -Appearing Intervals The first major technique of attitude measurement was developed by Thurstone, in 1929, in his study of attitudes toward religion. 52 The scale which he constructed introduced the metric to an area of research where it had never been used before. Thurstone assumed that one could obtain statements of opinion about a particular issue and could order them according to a dimension of expressed favorableness-unfavorableness towards the issue. Furthermore, the ordering of these statements could be such that there appeared to be an equal distance between adjacent statements on the continuum. Because of the latter assumption, one can make judgments about the degree of discrepancy between different people' 8 attitudes. Thurstone also assumed that the statements are uncorre- lated and that each statement has a position which is independent of the others. That is, acceptance of one statement does not necessarily imply the acceptance of any others. A Thurstone scale is made up of about twenty independent statements of opinion about a particular issue. Each statement has 521bid., pp. 520-523. 61 a numerical scale value determined by its average judged position on the continuum. A person' s attitude on the issue is measured by asking him to check those statements with which he agrees. His score is the mean scale value of those items which he checked. An example of a shortened version of such a scale follows: Trait: Attitude toward Open Housing Scale Value Statement Least Favorable Most Favorable A person should refuse to rent to anyone he doesn' t like. Federal laws enforcing open housing should apply only to public housing, not to private neighbor- hoods. Local governments should pub— licly urge people to engage in fair housing practices. Only in extreme cases of dis- crimination in housing should there be some sort of legal intervention. A person must rent to the first eligible applicant, regardless of race, color, or creed. The hallmark of a Thurstone scale is that the intervals between the statements are approximately equal. This property of the scale is achieved by the method in which it is constructed. The 62 first step is to collect a large number of opinion statements about some particular issue. Any statements which are confusing, ambiguous, double -barreled, or likely to be approved by individuals with opposed attitudes are immediately discarded. Each of the remaining statements is then sorted into one of eleven categories by a group of judges, according to the degree of favorableness or unfavorableness toward the issue expressed by the statement, regardless of the judges' own attitudes. These categories thus make up a scale which ranges from very favorable, through neutral, to extremely unfavorable opinions about the issue. By tabulating the ratings of all the judges, it is possible to calculate both the numeri - cal scale position of each statement (its average scale value), as well as the extent to which the judges agreed in its placement (its spread of ratings). The statements which are selected for use on the final scale are those whi ch have high interjudge agreement and which fall at relatively equally-spaced intervals along the continuum. A subject' s attitude on the particular issue is then derived from his responses to this final set of scale items. Likert' 8 Method of Summated Ratings One of the practical drawbacks of the Thurstone scale is that its construction is extremely laborious and time -consuming. 63 To cope with this problem, Likert developed a different technique which could produce an equally reliable attitude scale with relative ease. 53 The Likert scale is made up of a series of opinion state- ments about some issue. However, in contrast to the Thurstone scale, a person' s attitude is measured by asking him to indicate the extent of his agreement or disagreement with each item. This is done by having the person rate each item on a five -point scale of response (strongly agree, agree, undecided, disagree, strongly disagree). A person' s attitude score is the sum of his individual ratings. An example of a single scale item is the following: A. ”People should be allowed to move into any neighborhood they choose. " Rating Value Response 1 a) Strongly Agree 2 b) Agree 3 c) Undecided 4 d) Disagree 5 e) Strongly Disagree Likert assumes that each statement that is used in the scale is a linear function of the same attitude dimension. This assumption is the basis for the operation of adding up a person' 8 individual scores (or summating his ratings, to put it more 53w... pp. 496-498. 64 formally) to obtain his final score. A further implication is that the items in a scale must be highly correlated with a common attribute and thus with each other, as opposed to Thurstone' 5 distinct and independent items. It is important to note that at no point does Likert assume equal intervals between scale values. For example, it is quite possible that the difference between "agree" and "strongly agree" is much larger than the difference between "agree" and ”undecided. " This means that a Likert scale can provide information on the ordering of people' 8 attitudes on a continuum, but it is unable to indicate how close or how far apart different attitudes might be. Likert' 5 method of scale construction is similar to Thur- stone' 3 in the initial collecting and editing of a variety of opinion statements. The remaining statements are then rated by a sample group of subjects on the five —point response scale in terms of their own opinions about the statements. This is in contrast to the Thur- stone technique, where the ratings are made by trained judges and based not on personal opinions but on some relatively objective evaluation of where the statements fall on a continuum. The final Likert scale is composed of those items which best differentiate between sample subjects with the highest and lowest total scores. 65 Guttman' s Scalogram A third scaling technique is based on the assumption that a single, unidimensional trait can be measured by a set of statements which are ordered along a continuum of "difficulty of acceptance. " That is, the statements range from those which are easy for most people to accept to those which few persons would endorse. Such scale items are cumulative, since the acceptance of one item implies that the person accepts all those of lesser magnitude (those less dif- ficult to accept). To the extent that this is true, one can predict a person' s attitude towards other statements on the basis of knowing the most difficult item be will accept. On the following page is an example of how such a scale might appear. In order to obtain a scale which represents a single dimen- sion, Guttman presents sample subjects with an initial set of items and records the extent to which they respond to the items with specified answer patterns. 54 These patterns, which are referred to as scale types, follow a certain step -like order. The subject may either accept none of the items in the set (score 0), accept item A only (score 1), accept items A and B only (score 2), accept items A, B, and C only (score 3), etc. If the subject gives a nonscale response 541pm., pp. 506-507. 66 pattern (e. g. , accepts item C only and not those of lesser magnitude), it is assumed that he has made one or more response errors. By analyzing the number of response errors made, Guttman is able to determine the degree to which the initial set of items reflects a uni - dimensional attribute (that is, the extent to which they are "scalable"). The final scale is obtained by eliminating poor items and retesting sample subjects until a scalable set of items has been developed. Trait: Attitude toward Open Housing A cceptabi li ty Statement Least Difficult to Accept A. Generally speaking, people should be able to live anywhere they want. B. Real estate agencies should not discriminate against minority groups. C. The city should actively support the idea of open housing. D. There should be a local review board which would pass on cases of extreme discrimination in housing. E. There should be federal laws to Most Difficult to Accept ' enforce open housing. A person' s attitude is then measured by having him check all the statements on the scale which are acceptable to him. His 67 score is that of the appropriate scale type or (if he has given a nonscale response pattern) that of the scale type closest to his response. As the latter scoring procedure implies, it is almost impossible to develop a perfect unidimensional scale. This may be because people are actually responding not on the single dimension hypothesized, but rather on a different one, or on multiple dimen- sions. Osgood' 3 Semantic Differential The three methods just described attempt to measure attitudes by having people indicate the extent of their agreement with various opinion statements. In contrast to this approach, Osgood has studied attitudes by focusing on the meaning that people give to a word or concept. 55 Underlying this technique is the basic assump- tion of a hypothetical semantic space of an unknown number of dimensions, in which the meaning of any word or concept can be represented as a particular point. Osgood' s procedure is to have people judge a particular concept on a set of semantic scales. These scales are defined by verbal opposites with a mid -point of neutrality and are usually composed of seven discrimination steps. For example, a particular person' 5 meaning of the concept 551bid., pp. 528-531. 68 "integration" is measured by his ratings of it on a set of semantic scales. good __ __ __ bad strong weak fast slow active passive and so on. An analysis of the ratings collected by this method may reveal the particular dimensions which people use to qualify their experience, the types of concepts that are regarded as similar or different in meaning, and the intensity of the meaning given to a particular concept. Osgood' s own research has indicated that there are three dominant, independent dimensions which people use in judging concepts. He refers to these dimensions as the evaluative factor (e. g. , good -bad), the potency factor (e. g. , strong -weak), and the activity factor (e. g. , active—passive). Although this method can provide a lot of information about a concept, it is not exactly clear how the concept' 3 meaning for a person is related to opinion state- ments he would make about it. These four questionnaires may be used in an interview method. An advantage of the interview method is, first of all, that 69 it is possible to control the sample by a selection from the audience. Secondly, all questionnaires are filled in completely and correctly by professional interviewers. The results of the survey, moreover, are available a few hours after the questioning is terminated. This method has one drawback. It is evident from the results of such surveys that many of the audiences will be inclined to talk favorably about a film if asked to evaluate it immediately after the exposure. They lack the necessary time perspective to judge it properly. There is the added possibility that they will give answers which they think will please the interviewers. The drawback of the mail questionnaire method lies in the difficulty of controlling the size and composition of the sample in as satisfactory a manner as when personal interviews are conducted in the theater. This method is also slower, as it is necessary to wait at least a week until the bulk of the questionnaires are returned to conclude the tabulation. Such a delay may be too great for the motion picture time table. Also, it has been observed that some questions, especially the open -answer ones, frequently are not filled in. Here, too, we meet the problem that better —educated per- sons are more apt to fill in the questionnaires, and that persons who liked the picture arealso more likely to return the ballot than those who did not care for it. 70 There is one way to overcome the main weaknesses of both the interviewing and mail questionnaire method. This is to obtain the names, addresses, and classification data of persons who attended the film and conduct the actual interview one or two days later at the home of the respondent. This is a rather costly and slow method, but the results are superior to either of the other tech- niques. The researcher is undoubtedly familiar with most, if not all, of these various methods and techniques of measuring audience effectiveness. This group of approaches was compiled to provide a basis for the selection of a method that is compatible with an organization' s resources and to give the reader the opportunity to gain a broader perspective into the range of methods and techniques available. CHAPTER III SELECTION AND TEST OF A METHOD FOR AN AUDIENCE EFFECTIVENESS EVALUATION PROGRAM Of all the evaluation methods that are available, there seems to be little guidance available to recommend any particular method as being most effective in a given situation. It was pre — viously pointed out that there are certain advantages and disadvan- tages to each procedure. In order to establish any recommended technique it seems necessary to apply it against certain other variables which extend beyond those that are purely research oriented. The availability of money, manpower, time, and use of technical facilities are of prime importance and must be considered when developing a research and evaluation program. The available resources which a commercial or educational institution can rely on have, of course, a great bearing on the extent of the program. With an abundance of resources the ideal situation would perhaps be to have ready access to all the methods and techniques, applying each 71 72 to the specific problem at hand. This certainly is not necessary nor desirable, since many of the methods and techniques often pro- vide duplicate data. But prudent use of available resources, regard- less of how meager they may be, would certainly provide some degree of better evaluation than if none were applied. The procedure in adopting a specified technique may be best illustrated by using the Air Force' s film—making organization and applying the three variables of money, manpower, and physical facilities against various evaluation alternatives. Normally money would appear to be the most influential consideration. For the civilian organizations, directly or indirectly, money controls all the other variables. For the Air Force it is certainly important, but dollars and cents are often an indirect application. The people in the service are already on the payroll. The facilities are in being. It would seem that the main considera— tion really is allocation and alignment of present resources. A direct financial consideration would be applied to a new investment in Special equipment, or for hiring of persons not already on the Air Force payroll. In looking at the Aerospace Audio—Visual Services situation, there appears to be no adequate evaluation program. Since the results of such a program are unproven, a high expenditure for 73 special equipment and research consultants appears unlikely. Colonel Gallerani indicated: We [AAVS] are going to experience more cuts [budget] like the total Air Force. Salaries are going up to such a point that you almost can' t afford them. Eighty per cent of our budget is in salaries. 56 A financial investment in equipment for a research program which is not presently in being would seem a valid point for management to decide against a huge money outlay. However, once the program has proven its worth, then one could expect a more extensive con- tribution for expansion and more in depth research. There may be those critics who will say "the results will more than pay for the investment. " Then there are those from management who may raise the question "Will it?" It appears the responsibility hes with ”sell" the program. Management' s main reserva- the researcher to tion rests in the caution that accompanies a period of "tight monies. " The Air Force' 3 budget was sharply reduced for fiscal year 1970. Like most Air Force units, the Aerospace Audio-Visual Service felt the pressure to do the same job with less funds. As Colonel Alexan- der put it: You have an answer [for the evaluation program] from the times, and the times are now that we [AAVS] cannot hope to get additive manpower spaces or dollars in a program like this. My view now is that we are going to have to do it within our existing resources.57 [Underscoring mine. ] 56Colonel Gallerani, personal interview, AAVS, Norton Air Force Base, California, March 25, 1970. 7Colonel Alexander, personal interview. 74 It would appear logical then to select a method that did not require a great amount of expensive equipment and personnel outside of I "existing resources.‘ This comment leads to the second criteria, which involves that of manpower. Two aspects of this consideration must be mentioned here: 1) The number of personnel needed to staff a research operation, and 2) the expertise of those personnel required to meet the research and evaluation needs. The expertise or ”know -how” for this particu- lar type of evaluation appears to be rare in the United States, let alone within the military services. Several factors may point to this condition. One is the apparent scarcity of evaluation research literature in the fields of motion picture and television. This is especially obvious when compared to the vast numbers of films and television products produced by the industry on an annual basis. There are 575 television stations58 and over 2600 indepen- dent and institutional film producers of educational films, plus the unknown number of commercial film producers, each producing an unknown number of products. Another point to support the 8National Industrial Conference Board, Evaluating Media, 75 problem of finding qualified people is the logical observation made by Belson about the similar state of affairs in Britain. One reason for the shortage of people qualified to conduct television research is that very few organizations teach practical courses in methods of social research. In the length of one' 5 experience in research, many tricks of the trade are learned, but there is no satisfactory substitute for basic train- ing in what these methods are or how to use them. The third variable in considering the selection has to do with the technical facilities to accomplish the task, i. e. , accessi- bility for computer time for scoring and decoding data, resources for selecting sample audiences, facilities and opportunity for the audience to be tested. This area is one which (if the program is given reasonable support) would give the least trouble. Due to the nature and size of the Air Force, these types of facilities are avail- able. Regardless of the technique selected, these particular resources could be used to fill whatever requirements were gener- ated by a research study. In applying the three variables against each of the audience evaluation methods under consideration, the mechanical devices were all eliminated because: 1) They would require an initial outlay of money that was not already budgeted for and not available. 2) The technical personnel to administer and maintain them would increase the cost 59Belson, The Impact of Television, p. 362. 76 and personnel needs. 3) As has been mentioned, the data obtained through the use of mechanical equipment are often limited, and can be gained from other less expensive techniques. 4) They are often not mobil because of the intricate wiring and size. The audience must be brought to the machine facility. 5) The state of development of these mechanical techniques has not reached a point where a large number of people can be tested at one time. Many are limited to ten or twenty persons at a sitting. Of the nonmechanical techniques under consideration, the personal interview as a primary technique was eliminated because of the amount of time required and the number Of interviewers needed to get a true sample of the audience. There is also the problem of training the interviewer. However, this method is con— sidered by some authorities in the research field to be one of the most productive and most reliable. For example, if you sought to uncover the influences behind a city council decision, depth interviews might be most fruitful. Here you want to get the people with influence and not a random assortment of the general public. In intensive interviews, you seek to explore the content of a person' s mind and, if possible, the preconscious or subconscious motivations for his actions. Survey research taps information only about an individual' s conscious motivation. 60Charles H. Backstrum and Gerald D. Hursh, Survey Research (Chicago: Northwestern University Press, 1963), p. 9. 77 The Schwerin system does not appear practical in its true form because of the requirement for the test director to interview and conduct group discussions about the film. This task cannot normally. be delegated and requires a special talent. This means the test director would have to travel to where the sample audience is. There are certain aspects of the program that could be adapted, such as the flashing numbers on the screen. It has the advantage of obtaining an instant evaluation as the film is in progress, plus the benefits of a postfilm questionnaire. The method selected here is the pre- and posttest ques- tionnaire method. This procedure appears as productive in provid- ing information as the others. It also seems to provide a great deal of flexibility. The questionnaire to be of full value needs to be designed specifically for each particular research film and Video tape project. Dr. Kanner, who has been working in the audio—Visual field with the military services since 1950 as a producer, researcher, and a manger, states: . because of the vast number of films put out by the military services, and the length of time to produce them [two or three years], that evaluating each one would only delay them. The logical approach is then to just take a few and apply to the others what we learn from them. Doesn' t work! All the films I've worked with, even though the subject matter may be similar, have their own specific problems. . Every film requires its own evaluation. 61Dr. Kanner, personal interview, Washington, D.C. , March 17, 1970. 78 Since the questionnaire method of evaluation has been selected, there are at least four techniques one can use. Any one of these are simple to administer. They can be administered to large numbers of people at any location with the minimum amount of expense. Their main drawback is the expertise needed to design them. The value of the questionnaire and the ways it can be used most effectively are discussed in some detail in the following section. The Practical Application of the Questionnaire Method The purpose of the following study is twofold. The primary pur- pose is to provide an example of the types of information that can be made available through the use of the questionnaire method of measuring attitudes. The secondary purpose is to provide some raw data that can be used by the Air Force in evaluating the audience effectiveness of the Air Force Now #7 film. The Air Force Now is one of a monthly series of Commander' s Call films. The objective of the Air Force Now is to provide the Air Force worldwide Commander' s Call program with a monthly film covering a wide variety of Air Force activities. The current general objective of Commander' 5 Call is as follows: Commander' 8 Call is designed to provide a regular face - to-face communications interchange between the Commander 79 and the members of his unit. Its purpose is to encourage a better understanding of the individual' 8 relationship to his job, unit, community, and country. 62 The contents of the Air Force Now should fulfill this objec- tive. It should contain a cross section of Air Force events and efforts of Air Force people on the job around the world. The format is designed to remain flexible for special or priority projects as required. Where possible, a feature approach will be used--with its news peg being an event of major interest. Overall objectives of the Air Force Now film series are: ° To inform members of the U. S. Air Force about opera- tional missions of the Air Force. ° To interpret, by pictorial means, Air Force information objectives. ° To report events of significance to the Air Force. ° To motivate Air Force personnel toward appreciation of the Air Force and positive consideration of its career possi - bilities. ° To accomplish the above through interesting, dynamic and contemporary film techniques. A summary of the Air Force Now #7 that was used as a test vehi cle follow 5: 2 Internal Information Division, Secretary of the Air Force, Office of Information, "Concepts and Analysis of The Air Force Now" (Washington, D.C. , n.d.), p. 1. 63Ibid. 80 Running Time: 15:33 OBJECTIVES of this film are to inform and motivate the mem— bers of the Air Force about Air Force missions, activities, significant events and people on the job around the world accomplished through interesting, dynamic and contemporary film techniques. SUMMARY: WRIGHT - PATTERSON AFB, OHIO The only glass blower in the Air Force is a Russian immi - grant working for a laboratory at Wright -Patterson AFB. Examples of his work and how it fits into the Air Force mission are illustrated. EGLIN AFB, FLORIDA Lt. General B. 0. Davis, retiring after 27 years of service, talks about his career and discusses the role of new ideas and young servicemen in the Air Force of today and tomorrow. NELLIS AFB, NEVADA A personality sketch covering the life of a young Air Force couple in the Las Vegas community and their duties at Nellis. The husband is an F-111 mechanic and his wife an admin specialist. MACDILL AFB, FLORIDA An Air Force Sgt at MacDill AFB has what is believed to be the largest collection of Air Force patches and emblems in existence. The origin and acceptance of the patch are explained. NHA TRANG AB, VIETNAM A RVNAF pilot training school graduation and footage of Vietnamese air operations indirectly emphasizes the progress of the Vietnamese Air Force and their ability to take over an increasing combat role. PENTAGON, WASH. , D. C. Artist renditions of the Air Force Medal of Honor recipients are displayed in the Pentagon. A visual tribute to all the win- ners with narration conveying just what they stand for. 81 The Air Force Audience The audience which the test vehicle must be designed for is described by Colonel Beinhorn: The average Air Force officer is less than 33 years old; yet 64 per cent are less than 35. One out of five is over 40. For airmen nearly 55 per cent fall below the age of 25; yet 94 per cent are under 40. Less than 5 per cent of our airmen have less than a high school education and about 36 per cent have some college training. Eighty per cent of our officer force have at least a Bachelor' 8 degree and nearly one out of every five have a Master' s degree or more. It' s all part of the new look that the Air Force is taking on and it' s changing. The young have gotten younger and the old stayed with us and got older. Now, more than ever before, there is a greater separation of age, and it' 8 pointed up every time we present the facts. We' re in the middle of a changing Air Force --generation gap, changing society, the whole Age of Aquarius bit. In order to treat the generation gap properly, we' ve got to bridge the gap in both directions. While the effort to appeal to the younger force should be a strong consideration, the entire Air Force Family should be our target objective. While the senior members of the Air Force grew up with Glenn Miller, the younger audience cut their teeth on The Beatles. Our young people see and hear far more than their older associates. They are television' 3 chil- dren; those who have grown up with Captain Kangaroo, the information eXplosion of the 605; the immediacy of Telstar; and the realities of life and death. They have shared their early adult years with the agony of assassination; the excitement of travel to and walking on the moon; and the intimate world of color television. Today' 8 Air Force has been schooled in the world of advanced audio-Visual technology. Their eyes have adapted to 64Colonel Leo Beinhorn, in a speech January 26, 1970. 82 the modern film makers like Fellini and television programs varying from ”The Laugh -In” format to Network Documentary Reports. Current statistics reflect that the average high school graduate has watched 15, 000 hours of television and 500 feature films prior to graduation while spending only 11, 000 hours in the classroom. For each book the average college student reads, he sees 20 movies. Most of our younger airmen and officers have known only Vietnam as a significant war of their times. Most weren' t born by the close of World War II and to them America' 5 past international role of leader is only history. They have seen the role of government and military change to one of only passing popularity and many are not motivated toward a career in the Air Force. In order to effectively communicate we must use NOW language: the language of the times. At the same time, we must talk with sincerity and be certain that we are not missing a healthy portion of the audience --their older asso- ciates. Many times it is visually impossible to communicate to everyone in the same manner. If this proves to be the case, then the final film should have a proportionate message for both audiences. Like any mass media, The Air Force Now faces the problem of communicating to all members of its audience. Trying too hard to talk to a young audience using young tech- niques can result in a false, labored approach. It can also mean loss of the entire audience. 65 The Questionnaire To accomplish the purpose of the study it was necessary to design an appropriate questionnaire. The film was Viewed. From this observation, plus a knowledge of the objectives for the entire series as well as the producer's objectives for each of the six seg- ments, questions were designed. The four interview techniques 5Internal Information Division, "Concepts and Analysis of The Air Force Now, " pp. 3-4. 83 mentioned earlier in this paper are Thurstone' s method of equal- appearing intervals, Likert' 3 method of summated ratings, Guttman' s scalogram, and Osgood' s semantic differential. Of these the two most heavily used were Osgood' s and Likert' 3 techniques. The reasons for selecting these two were based on the degree of difficulty involved in the question design versus the number of questions that had to be designed for a significant test. Both Thurstone' s and Guttman' 3 tests require an assumption that statements about a par- ticular issue could be obtained and that an ordering of these state- ments according to a positive or negative dimension could be made. This was not possible for the author to do under the existing condi - tions, which were due largely to the unknown nature of the audience, and because the film had not been released prior to the test. For the make -up of the questionnaire the reader is referred to Appendix A. The questionnaire consists of 78 structured questions. Of these 78 questions, there are six 21 -word semantic differential word pairings, one 18-word scaled response (similar to the semantic differential), 22 Likert method questions, 47 multiple choice, and I recall ordering question. The total number of tested responses is 232, 9 of which are demographic in nature and will not be counted. Two basic areas were placed under consideration when designing the questionnaire. One was that the purposes of the study 84 design were to obtain information regarding the four objectives of the film series. The other area was to test if each segment of the film met the requirements set forth by the producers. The film objectives are: 1. To inform the audience of Air Force operational missions. 2. To motivate the audience toward a career in the Air Force. 3. To gain appreciation of the Air Force. 4. To make the fihn presentation dynamic, interesting, and contemporary. The questions were written in order that one response would be applicable to one of the four objectives: 26 responses were keyed to Information. 61 responses were keyed to Motivation. 41 responses were keyed to appreciation of the Air Force. 95 responses were keyed to judging the merits of the film production. In order to measure the attitudes of the audience toward the Objectives for each one of the six segments, questions were designed and related to those objectives. The segment objectives, as related by the producer, are: Glassblower--to illustrate an unusual talent required by the Air Force. Relates the beauty of this unusual mission requirement. Gen. Davis --tells the audience that the Air Force is cog- nizant of the fact that there have been racial problems and that 85 the Air Force has, is, and will continue to do something positive about it. Also, General Davis as a man is illustrated. Married Couple --Young couples can get along well in the Air Force, maintain secure jobs, and have a good time. They are much like civilians and probably have more opportunities and job security than their civilian counterparts. The antithesis of Army. Patches--unusual hobby. Has no particular significance other than to illustrate a man' 8 personal thing. Does show a little on Air Force history. An interesting sidelight. RVNAF Pilot Training-~RVNAF are an up and coming organization. They are building progress "towards controlling their own military/ country. " Medal of Honor—-A strictly nostalgic reminder to the audience that the Air Force has a proud heritage. The AF has a tradition of heroism and service to their country for which all airmen can be proud. The heros of our service. 66 Consideration was given to the order and format of the questionnaire. In View of the length and possible fatiguing effect on the respondents, the decision was made to compartmentalize the questions by segment, rather than mix them, or place all of one type in one section. Because of mechanical scoring, special answer sheets were used. Instructions were provided on the semantic dif- ferential questions to allow the respondent the opportunity to mark on the questionnaire, and later transfer the answer to the official answer sheet. This eliminated constant back and forth Visual irrita- tion but increased the chance of error. The open -end recall ordering queStion was deliberately placed at the beginning. As noted in the instructions, it would be picked up prior to the respondent getting 66Letter from Major Jack H. Oswald, April 23, 1970. 86 the remainder of the questionnaire. This was for the obvious purpose of not allowing the structured questions to influence the respondent' s "recall" capability. Applying Values for the Attitude Measurement A value scale was established for each type of question in order that value could be assigned to each answer. This made it possible to sum up all the answers and apply the totals to an intensity scale of attitudes. Value Scale For the semantic differential and 7 -point scale: . 6 . 5 . 4 . 3 . 2 . 1 0 Positive A B C D E F G Negative . G F E D C B A For the Likert 5 -point scale: POSItive . 6 .45 . 3 . 15 0 Negative (Strongly —— —-— —-——— —— —- (Strongly A B C D E . Agree) D1sagree) For the 5 - and 4 -point multiple choice scale: 5 -point scale -— same as the Likert value 87 . 6 . 4 . 2 0 Positive Negative 4 -point scale - - For the 3 -point scale: . . . 6 . 3 0 . Pos1t1ve Yes No Opinion No Negat1ve These three types of point scales, the 5, 4, and 3, must be individually scored. According to the value, the answer implies a positive or negative connotation. Example Attitude Scale = Value X Number of Sample Value Number Value for the Question Response Maximum Positive Range - . 6 X Number of Sample Lowest or Most Negative Range 0 X Number of Sample . 3 X Number of Sample Mean or No Attitude or No Opinion Of the 223 responses that are included in the questionnaire, 183 of those are of a specific attitude nature. The others are designed to solicit factual information. The total sum of the 183 "value numbers" for each attitude response is divided by the number of questions in the questionnaire (183). The answer, which is the average or mean 88 score, is applied against the Attitude Scale of maximum and minimum values. This figure reflects the intensity of the overall attitude of the audience toward the film in relation to the scale. Example If the sample audience were 200 and the question was number 69 (7 -point scale): Number of Sample Value 50 answered D (.3) = .3 X 50 = 15.0 100 answered F (.1) = .1 X 100 = . 10.0 25 answered A (.6) = .6 X 25 = 15. 0 25 answered B (.5) = .5 X 25 = 12.5 52. 5 Value Number for Question 69 If this value number is added with other value numbers for other questions: Question Number Value Number 10 = 28. 5 13 = 40. 5 69 = 52. 5 80 = 25. 0 etc. = ---- etc. = ---- Total 183 divided into Total = Mean Number or Average, e.g. , 58. 5 89 The 58.5 is applied against the Attitude Scale = Value X Number of Sample Positive ———120 100 80 1631(3) 40 20 0 Negative Opinion (58.5) This is a broad example to explain the concept of applying values to questions. Careful selection of certain key responses can demographically be broken down into specific areas. Even certain attitude responses can reveal a great wealth of information. A pro- file using the semantic differential could be plotted for each of the segments. This would provide a comparison on each of the 21 —word pairings. Applying the value of each question or response to the attitude scale, one can determine the attitude toward each segment, each question; and it can be correlated with race, age group, rank, etc. It is recommended that a great deal of time be spent with the data. Many productive conclusions can be drawn from this type of information. Results, Recommendations and Conclusions A pretest of the questionnaire was administered April 29th to 60 junior and senior members of the Air Force ROTC at Michigan 90 State University. The fihn was introduced by the ROTC instructor as a regular Commander' 5 Call film just received by the detachment. There was no indication that the audience would be asked to fill out a questionnaire until the film was over. After the film the administra- tor was introduced. He explained that the test was to serve as the only input of information from a civilian oriented audience, as opposed to one that was actively engaged in the Air Force. It was believed that if the questionnaire was introduced as a pretest, it would be taken too lightly. Conditions for the pretest were acceptable, but three factors were evident that may have had some influence on the test results. During the showing of the film, the room was almost uncomfortably warm, the window blinds which were drawn to block out the light would occasionally flap from the open windows, letting in a flood of light, and the sound came from the back of the room while the screen was in the front. The following results are a few examples of the type of infor- mation that can be made available through the use of a questionnaire method. Each answer is indicated by a number value. It is to be applied to the Attitude Scale: Most positive fl— —— —— TI? — — ——(—)— Most negative opinion 91 Attitudes expressed toward the four objectives of the film series. --As was indicated previously, there were many questions structured for each objective. The following represent only a few which were coded. Number of Average Questions Number Objectives Coded Value To inform about Air Force opera- tional missions. 10 24.4 To motivate toward a career in the Air Force. 36 21. 8 To gain apprecia - tion of the Air . Force. 23 24. 0 To make the film presentation dynamic, interesting, and contemporary. 43 2 1. 7 Since the ”mean" on the Attitude Scale is 18, these results reflect a positive attitude, the degree of which is measured by the intensity indicated toward the positive end of the scale. Attitudes expressed toward the producer' 5 objectives for each segment. -- One question was structured specifically to match each of the producer' 8 objectives. Normally several such questions for each segment would be designed for this purpose. 92 Value Question Segment Number 53 Glassblower 29.4 85 General Davis 28.0 115 Married Couple 17. 9 146 Patches/Emblems 22.4 Part II 29 Vietnamese Air Force 24.5 54 Medal of Honor 28.8 Average number value 25. 1 In this area the value numbers can be expected to be higher because the questions are aimed at the core of the filrn' s objectives. It is evident that each of the producer' 5 objectives had a positive effect, except for the segment on the married couple. The specific break - down of the answers for that question (115) was: Strongly Agree ( 5) Agree (21) No Opinion ( 8) Disagree (22) Strongly Disagree ( 4) The objective was basically to show that "young couples get along well in the Air Force. They are much like civilians and probably have more opportunities and job security than their civilian counter- ' However, the effect that actually was received by the parts. ' audience indicated that the film emphasized that the Air Force is interesting for a happily married couple, and the point of better job 93 security and opportunities was missed. Question 114 was worded to show that it is possible to be happily married in the Air Force. It received a number value of 25.7. Question 116 emphasized that the Air Force is interesting for a young married couple and received a number value of 29.0. Since the indicators are very positive in nature, these positive attitudes certainly did not hurt the success of the fihn, but they do indicate that it missed a part of the producer' 5 objective. Semantic differential as an attitude indicator. --A more detailed breakdown of specific attitudes toward three of the segments and the film in general are represented in the table on the following page. Since this sample was a pretest, certain words appear which solicit very little or no response. These are most likely poor word selections or words that have a different meaning to the author than to the audience. These words should be eliminated or substitutions made. Determining adequate word pairings should be a part of the information gained from a careful investigation of the target audience. The pretest will indicate if the pairings are appropriate. The word pairing "fast-slow" indicated little response. It was intended to reflect the pace of the film and should have been labeled "fast pace - slow pace. " The words active and inactive, in the general evaluation 94 Segment 52 EN 5: 353 a: z? 55 Baé O N D S], III 3 < a: m U '0'; ,4 m k, I}, g I: ‘8 Additional Words Word Pairings El g g g 3 ‘3 :I If] in Used for. Questions a?) Iran 3 <8 Ema, onEntIre Film 2: m = 5 a .. 5 §G ”0 23 0’ Number Value Friendly -Unfriendly 26 25 23 26 Strong -Weak 2 2 24 2 5 22 Comfortable -Uncomfortable 22 22 24 2 6 Refreshing -Du11 1 9 2 1 2 1 22 Good taste -Poor taste 22 23 24 . 19 Active -Ina ctive Valuable -Worthless 2 l 2 3 22 2 3 Patriotic -Unpatriotic 2 1 22 27 25 Stimulating -Boring 1 9 2 1 2 3 22 Easy to understand -Hard to understand 20 25 25 2 6 Pleasant -Unplea sant Useful for me -Useless for me 21 20 22 22 Happy -Sad Informative -Uninformative 2 2 2 3 2 6 2 2 Low pressure -High pressure 2 1 19 14 22 Sociable -Unsociable Modern -Old fashioned 25 25 20 2 1 Important -Unimportant Proud -Ashamed 2 1 24 2 6 25 Emotional -Unemoti onal 1 8 2 1 25 20 Reliable -Unreliable 2 1 2 3 2 3 Satisfying -Unsati sfying 20 20 2 3 2 1 Personal -Impersonal 22 24 20 22 Believable -Unbelievable 2 3 2 5 2 6 Unusual -Commonplace 22 22 2 1 Fast —Slow 1 9 17 17 Average Number Value 20. 2 22. 3 22. 22.7 Attitude Scale Most Positive 36 30 24 :18 12 0 Most Negative 0 Opinion 95 section, did not solicit much of a response. Certain words will be more applicable to certain segments, but a standard list should be applied to all segments to enable a comparative measurement. In using the Recall Rank and Order technique, in which the respondent is asked to list the Various segments of the film as best he can, the assumption must be made that the most prominent item will be the first one the respondent lists. The second item listed will be the next most impressive, and so forth. Also, in recording and interpreting these results, some provision must be made for the logical behavior of listing the items in sequence. In the next table, the left hand side includes the segments in proper sequence. The top labels the columns as to the position in which the respondents placed the segment. The normal sequence listing is indicated by an asterisk. Any deviation from that position would be significant, especially if it were toward the first position. In the table on the following page, note the high recall indi- cated for the married couple (56) and the high ranking of second position. This would indicate that it had made an impression on the audience. However, the results in the objective study indicate little recognition of the producer' 5 Objectives. The semantic differential scale reflects only a 20. 2 number value. The indication for the 96 Segments In the Order Mentioned in Order Total of Viewing lst 2nd 3rd 4th 5th 6th Responses Glass Blower 19* 1 3 4 7 2 36 Lt. General Davis 19 10* 8 3 7 0 47 Married Couple 9 28 17* 2. 0 0 56 Patch Collection 4 3 9 8* 4 5 33 Vietnam Air Force 1 8 9 16 11* 1 46 Medal of Honor 4 6 9 14 12 6* 51 researcher is that more probing questions should be included to gain some answers to this dilemma. Number of Segments Recalled by the Respondents 1 recalled two segments 8 recalled three segments 6 recalled four segments 28 recalled five segments 13 recalled six segments Note that almost half of the respondents missed one segment. This may be a clue that the film should be limited to only five segments. However, much additional research is needed to validate that assump- tion. 97 Multiple choice questions. --Included in this questionnaire design is a group of multiple choice questions which do not solicit an attitude response, but one of knowledge. The purpose of these questions is twofold. One purpose is to indicate very simply if the audience was paying attention to the film. If these ”attention indicator" questions reveal that no one was paying attention, it might explain the reason for the entire film not being successful. Question 91 asked for the rank of General Davis. Fifty -three of the sixty responses were correct. Question 58 in Part II asked for the word that appeared on the Medal of Honor. Thirty -five of the respondents marked ' which was correct. Fourteen guessed it was ”honor" ”valor, ' because of the title of the medal. These examples illustrate a high level of attention toward the film. The second purpose is to ascertain if any learning took place. For example, in the segment on the married couple, part of the producer' s objective was to reveal job opportunities. This was part of the overall objective of career motivation and appreciation of the Air Force. One of the techniques used is to identify a desirable Air Force installation near a resort attraction and to identify it with the Airmen in the film. The situation in the film segment revealed an Airman First Class at Nellis Air Force Base near Las Vegas working as a flight line mechanic. The results of questions 117, 118, and 98 119 were: 55 respondents remembered the Airman' 3 rank, 60 respondents recalled his job, and 45 respondents recalled the name of the installation. This was a good indication that these specific points got across. The test designer must be careful that the word- ing of the question or the choice of possible answers are such that the right answer is not too obvious. For example, in question 118 (which all the respondents answered correctly) the words "flight line maintenance" may have been too general in view of the emphasis placed on it in the film. A better wording of the question may have been: What specific job did he do on the flight line? Conclusions and recommendations. -—In developing the test questionnaire, it is recommended that considerations be given to: 1. Balancing the number of questions and the types of questions directed to specific objectives. 2. Obtaining semantic differential word pairings from investi - gation of target audiences, to insure that the "language" is accurate. 3. Conducting an interview and discussion with the pretest audience about specific feelings concerning various aspects of the film. 4. Working closely with the computer programmer in order that certain program scoring techniques can be built into the questionnaire design. 99 5. Devoting a great amount of study and attention to interpreting the results and developing conclusions. A larger sample of representative audiences would be more meaningful. However, it should be noted that generally the. ratings will gravitate around the center area of the Attitude Scale. If there is a greater deviation toward the positive or negative poles for any particular question, then a special investigation of the production technique or script should be made to determine the cause. Certain- ly if it is toward the positive end, it possibly can be used again; or if it is toward the negative end, it can be avoided in future films. With some basic experience and several studies, a standard can be established within which the film must fall to be acceptable. The goal, of course, would be the "most positive" results, but this is unlikely to happen. Therefore, a reasonable standard should be developed which would take into consideration the cost, time, objec- tives, and other variables affecting the quality of the final output. The data from the survey should be interpreted immediately. Many are the punched cards, full of data, lying unanalyzed and unreported in the files of surveyors who, rather than labor over what they have found, have rushed out to gather more data. CHAPTER IV RECOMMENDED IMPLEMENTATION PROCEDURE FOR ESTABLISHING A WORKABLE AUDIENCE EFFECTIVENESS EVALUATION PROGRAM AT THE AEROSPACE AUDIO-VISUAL CENTER It has been established that officials at the Aerospace Audio-Visual Service, Norton Air Force Base, California, are interested in the problems involved in developing a research opera - tion for film and video tape audience effectiveness evaluation, and that additional manpower for this project is unlikely. It further appears that if this program is to start, some impetus must come from within the "existing resources. " It would seem the urgency of starting such a program would be of importance to the management of the Aerospace Audio—Visual Service, especially since Headquarters Air Force has directed AAVS to set up a study group to look into the feasibility of incorporating the "Diagnostic Pretesting of the United States Air Force Instruc- tional Motion Picture" proposed by Captain Jay Sedlik. This study 100 101 group appears to be proceding slowly because it has run into the problems that plague all the Air Force at present: money and man- power. Major Archer, Chief of Production at the AAVS Center, says: "With neither the staff nor the finances to proceed, from the outset it (the study group) has been started on a smaller scale to get a feeling for what we need. "67 Mr. Don Chapman, a member of the Center staff, adds: We are just beginning and are feeling our way here. There is no real authority at this moment for setting up such an organi - zation. . . . There is a short supply of money and many other factors contingent upon producing a better product. 68 Therefore, the ideal situation of the permanent research staff with its own resources and extensive capability cannot be the approach of this proposal. A "practical" approach, although some- what less than ideal, may have a greater chance of success in getting started. ”Getting started" is important in order to get the planning and impetus of the program under way. If, on the heels of Captain Sedlik' s proposal and the Air Force' 3 directive, a workable program can be adopted, the chance of it gaining the necessary support is favorable. 7‘Norma Archer, Major USAF, personal interview. 8 6 Don H. Chapman, personal interview held at Aerospace Audio-Visual Center, Norton Air Force Base, California, March 24, 1970. 102 There are six areas that must be considered in establishing an effective evaluation program at the Aerospace Audio -Visual Center. The six areas are: To understand and establish the proper atmosphere in which the researcher, producer, and management can operate. To look at management' 3 responsibilities toward the researcher. To define the position and its location in the organizational structure. To describe the qualifications for the position of researcher. To look at the logistical framework of the operation, to include mission objectives, office supplies, travel and mis - cellaneous eXpenses, printing and computer requirements. To identify the functions, information generated, and bene- fits for the producer, management, and AAVS provided by the research program .: Establishing the Proper Working Atmosphere It would appear, because of the uniqueness of the creative aspects of the production business and the methodical approach of the researcher, that one important factor should be the relationship between management, producer, and researcher. If these three 103 cannot operate in a positive and constructive atmosphere, no matter what type or quality program is implemented, it probably will not be successful. It is safe to say that, because of the different attitudes, the creative and production staff are not going to fully welcome such a program unless it is properly instigated and inte- grated into the system. When it comes to the probing and evaluating research such as has been suggested in this paper, there is frequently a reaction on the part of the production or creative staff to resist it or to simply ignore it. All production people, researchers, and probably most management personnel are aware that this situation exists. It is important that an attempt be made to understand and resolve the conflict. Dr. William A. Belson, the holder of the first Gold Medal of the Market Research Society and the Thomson Gold Medal for Media Research in Britain, and widely published in the areas of television, other maSS media, and research methods, says: p. One of the more potent reasons for attitudes of this kind is that television [and film] production at its best depends upon the flair of the artist, of the producer, of the creative director. Understandably, .many of these people feel that to be con- strained by requirements developed by some statistically- minded researcher would stifle the creative urge and make a dull thing of output. However right or wrong creative staff may be in this belief, the fact remains that the researcher who presents a technical report which appears to impose require— ments upon creative staff, is liable to have his work ignored. Over and above this, the artistic mind and the research mind 104 are frequently far apart and they may on occasions be out of sympathy with each other. Where this is so, the apparent intrusion of research into the artistic domain can be sharply resented. However, this is by no means the only . . . [problem of producer/researcher relationship] . Many an experienced and practical producer is likely to take the line that he knows the business of production better than the research man whom he sees as edging into . . . [his area] . Even if he sees the research man in his proper role, namely as someone who can provide him with information relevant to the decisions he has to make, the producer is usually a very busy man who has to meet deadlines day by day. Accordingly he will not be able to give anything but the minimum of time to working out what the research man' 5 report actually means or how to implement a suggestion made by someone not fully conversant with television production. In the meantime, he plays his hunch-~which may well be right and even brilliantly so--but which occasionally may be disastrously wrong. However experienced he may be, the decision -maker needs facts: the problem is how to present them to him in a form which he is able and willing to use in his own rather special circumstances. As unfortunate as anything else is the remoteness of the average research man from the production scene. His basic training and his way of thought are likely to be very different from those of production staff. Yet all too often he makes no effort to bridge the gap or to familiarize himself with the problems and the realities of production. Accordingly, the scene is set against his seeing problems in production terms or reporting his results in production terms: he does not know enough about the production of television material to be of much use. This in turn means that production staff do not become favourably aware of what research might do for them. [These three reasons are not the only factors that must be considered when adopting a detailed research program. ] . Each is an important reason however, and fortunately the .same course of remedial action is appropriate with respect to each of them. This course of action is to bring the researcher into close and continuing contact with the production team. If this is done with reasonable tact and good sense and if the researcher 105 works at it, several things are very likely to follow. In the first place, the researcher will probably learn enough about program production to be able to discern and tackle relevant problems and to provide usable results. In all likelihood, he will come to think in production terms. Secondly, production staff will get to know him at a personal level and will learn at first hand what he can possibly do for them. If he concentrates first upon problems about which production staff are personally aware and about which they would like help, acceptance of him and his discipline will occur much more quickly than it would were he to try to impose his own ideas of what is needed. His integration into the production scene may well change his earlier views in any case. This integration will of course involve the researcher in spending a lot of time round and about the studios, and in listening to and mixing with production staff at all levels. He will have to avoid anything like mystique and jargon. These are usually the props of people who have not yet discovered the simplicity of their own craft and they can be disastrous for the acceptance of research. He will have to let production staff see how he goes about his work -—not in anything remotely- like lecture form, but quite informally and as opportunity permits. This involvement of production staff should include occasional attendance at group testing operations, hearing tape recordings of intensive interviews, being shown or told where the informants come from, along with the simple steps that are taken to ensure that these informants are a proper cross -section of the audience. It helps too if producers are free to look in on the analysis of viewer opinions as this analysis proceeds. There are obvious limits to the amount of this inVolvement which should be attempted and there are obvious dangers to be avoided. What the researcher has to do is to convey to pro— ducers the simple commonsense of what he is doing and at the same time to trigger off in the producer an awareness of the possible further uses to him of this new tool. What the researcher must avoid at all times is any sort of blind protec— tiveness of research as such. In particular, he must be willing to admit its weaknesses and its limits and above all he must never defend bad research: there is a real danger that such overselling and overprotectiveness will lead to his being left alone in his importance. With the integration of research thinking into the produc- tion department--and that is what is required--two more things should tend to happen. Findings from program research are 106 likely to become integrated into creative thinking so that the lessons of this research will operate not as controls but as a way of thinking. Secondly, when the results of some enquiry are given to a busy producer, they will in all likelihood be results of work which that producer initiated in order to make a particular decision. He is hardly likely to feel he is too busy to give the results his time. For the sort of relationship outlined here, the researcher ought really to . . . [work with] the production department on a continuing basis and his office should be on production prem- ises. . . . [His ultimate responsibility would be to the Commander of the AAVS Center. ] Nonetheless his contact with . the production department to which he is assigned must be a direct and working contact: anything less than this would be too remote to achieve a real integration of program research into production. 69 Management' 8 Re sponsibi litie s Management' 8 responsibilities toward this integration are most important. From personal experience the author has witnessed the creation of a new position or function based on the rare skills of an individual. Much too often the one person who has the expertise and motivation to meet the requirement becomes so involved in rou- tine and other administrative problems that he is no longer effective in using his specialized skills. Belson warns: Only if . - . [the researcher] is relatively free from the machinery of a continuous routine will . . . [he] have that necessary time and flexibility for detecting the changing prob- lems of program production, and for initiating appropriate research action. Moreover, he must constantly be on his guard against being taken over by some new routine of his own making. 9 Belson, The Impact of Television, pp. 352 -361. 107 A new routine develops all too easily. It may consist of a regularised program-testing service, of the continuing study of reaction to certain kinds of program material, of the institution of quite frequent survey work to chart relevant characteristics and attitudes of viewers. If there is a case for running such work on a continuing basis, then the routine parts of that work should be contracted out. Even for intermittent use of such procedures, it is desirable to contract out if at all pos - sible because, once the researcher sets up the necessary machinery for carrying out a useful research operation, there is a danger that he will go on with the running of it because this is usually more settled and is easier than the exacting task of vigilant, ranging enquiry. Perhaps, too, the attractions of empire building will contribute to his hanging on to it. On the other hand, there is an obvious limit to the degree to which the researcher should stay free of routine or of the grind of detailed research. He should, for instance, . . . [go out to the field to conduct eXploratory interviews, primarily to stay in touch with viewers, but also to pick up leads as to the inadequacy or strengths of a particular program]. He must, if such leads seem to warrant systematic investigation, give his full time and attention to the design and to the supervision of the neces- sary research. He must work with good speed. And no doubt he will for a time be closely occupied with the writing of the report and with its presentation. Unavoidably, he will be involved in further detail and rou— tine whenever group testing of programs is done. Thus he will be the one to collaborate with a producer in deciding what will be tested; he will have to plan and to organise the content analy- sis of viewers' reactions and to make systematic checks on the work of his analysts; he will have to do a certain amount of the content analysis himself. Then again he would be required to meet regularly (as well as in the course of day -to -day work) with producers and management and to stay closely in touch with all developments in . . . [organizational] planning and policy. ~ There would, then, be various things to be done on a regular basis. There would also be frequent occasions when close and sustained attention to the detail of specific enquiries would be called for. However, this situation is very different from that in which the researcher has become immersed in one or perhaps several procedures for the regular collection of specific information. Valuable as the regular provision of this 108 information may be, the immersed researcher inevitably loses contact with the full range of the . . . [production] problems . -—and, of course, with that wider range of research methods suited to the solution of such problems. Sometimes the contracting out of a job which is time con- suming or of a routine character will be frustrated by the absence of outside facilities for doing such work. This is the more likely to be the case with a method which is fairly new. In such circumstances the researcher will have to do or to con- trol the job himself. If for a sound reason this job was set going on a regular basis, then the researcher who intends to stay in touch with the full range of his . . . [organization' 5] research needs should make every effort to find or to train others to take over the routine of such an operation, preferably freeing him- self of its operation altogether. New methods do not flourish when all our attention is being given to the maintenance of several specific research procedures. . . . Vigilant program research requires that the researcher should know when and what and to whom to delegate and that he should know when to free himself of an established procedure altogether. These thoughts of Belson' s, when applied to the structure and situation at the Aerospace Audio-Visual Center, reflect an accu- rate appraisal Of the attitudes and present a realistic approach for developing the proper operating environment. Position and Its Location in the Organizational Structure The next step in developing this proposal is to consider the existing criteria which will affect the implementation of the program. Money and personnel are correlated to each other in that they repre- sent a fixed resource allocated to Headquarters AAVS and the Center 7OIbid. 109 to carry out their mission. In View of the current economic shortage it is impractical to consider that additional manpower and financial assistance would be allocated to AAVS to establish a special research function--regardless of the justification and apparent savings. Therefore, the only apparent alternative is to establish the function from resources already available to AAVS. It would seem, if there is to be a program at all, the mini- mum number from which this research function can be started is one qualified person to spearhead and develop the program. If this assumption is correct, there would then be a need to create a posi— tion which could be entitled ”Special Assistant to the Commander for ' This individual would Audio—Visual Research and Evaluation. ' function as was previously pointed out under the section on manage - ment' 8 responsibilities. To provide appropriate support for this position and to maintain the routine research operations, the Film Library Center under the Division of Utilization would be tasked. The Film Library is presently charged with an evaluation responsi - bility, and this proposal would require a little extension of that existing function. 71 Although this Division is charged with the responsibility, it is not presently geared for the type of ultimate 71U. S. Department of the Air Force, Air Force Film Library Service, Air Force Regulation 95 - 11 (Washington, D. C. , February 6, 1967), par. 4f, p. 2. {I‘ll 'EI‘II‘ 110 program that is suggested. However, for an initial phase, it is conceivable that some realignment of present position functions and tasks by the Director of the division would allow whatever logistical and office support needed to be available. One of the obvious draw- backs to this structure, but certainly not an insurmountable one, is that it requires a careful understanding between the Special Assistant and the Director of Utilization as to a functional working agreement between the Special Assistant and the people assigned to provide the logistical support for the research and evaluations. The Director of Utilization must be informed what his people are doing to meet that responsibility, which comes under his direction. These people are working for the Director of Utilization, but must be responsive to the needs of the Special Assistant for research and evaluation. To operate under these circumstances (which have been generated by the lack of additional manpower), it would seem that a mutual or directed agreement as to proper working relationships would be required. It further appears that to make the operation function smoothly with the least amount of ”bogging down, " the communica- tion channel between those providing the support and the Special Assistant be direct. The Director of Utilization should not have a need nor responsibility to be directly involved in the operation of the research. This notion should be clearly understood by the - III .III lllllll I l.[I.-l ' [I it'll I E II I '1' In! In" UIII .‘Il 111 Commander of the Center, so the Director of Utilization does not feel he is being held responsible for anything other than the quality of work his people in the Film Library Center are providing in sup- port of the research function. Therefore, the situation which is created is basically that personnel are technically working for one manager and responsive to another. This would appear to be in violation of certain management and organizational principles. How - ever, if the situation is approached properly, with clearly defined procedures, a workable solution can be negotiated to overcome this possible Objection. Such organizational structures exist throughout the Air Force today. This type of operation would parallel a tenant unit on a base in which the Base Commander is responsible for pro- viding necessary logistical support without having direct responsi - bility for the tenant' 5 mission. Qualifications of the Researcher The fourth area of consideration deals with the qualifications of the "Special Assistant to the Commander for Audio-Visual Research and Evaluation. " Ideally this position should be filled by an Officer or Civilian Government employee with training in methods at a practical level. He should be versed in scientific method in the special setting of social research, methods of problem isolation, techniques for drawing representative samples of the population; 112 various interviewing techniques, methods for formulating questions and for pretesting them, construction of attitude scales, certain techniques of the psychological laboratory, the different techniques for analyzing and systematizing the information gathered, use of computer facilities, use of statistical and other indices of the mean- ingfulness of patterns found in the analyzed information, and the writing of systematic and readable reports. Obviously a general background in social psychology, particularly the psychology of per- ception, learning, and memory, is highly desirable. In addition, this person must not only know what the different methods are, but when and how to use them. Certainly a knowledge of film and pro- duction would be helpful. The number of qualifications ideally needed perhaps represent an "academic overkill" of the requirements. The hope of identifying such a person is unfortunately rather small, especially when money and manpower restrictions will dictate that the field be narrowed to the Air Force, and probably to the Aero- space Audio -Visual Command. The solution to this dilemma is to 1) find the best possible qualified person--one who meets as many of the recommended qualifications as possible, 2) take advantage of all those specialists within the Air Force who possess some of the required skills, 3) search out and use all other military and civilian resources for consultations and information, and 4) develop Jul-i J‘tlilil I III lllll III II I! III 113 a working library of current professional periodicals and books in the research area. If the research position can be filled with a person who has some of the background and is highly motivated to search out the needed expertise, it should be sufficient to get the program started. It is important to identify the many sources already in existence that have the type of attitude profiles on audi - ences that the producer needs, without going to the expense and time of duplicating data. As an example of how the resourceful researcher can use information already available, the following illustration is offered: The Directorate of Data Systems and Statistics, Comptrol- ler of the Air Force Headquarters, Washington, D. C. , periodically collects personnel data on a variety of subjects. Its sample is 5 per cent of the Air Force Airmen, and 10 per cent of the Officers. An effective use of this resource is to get on the distribution list for copies of the Sample Survey Questionnaire. This provides the researcher with a catalogue of all the information that is available. He can pick special questions which would provide certain types of attitude information, certain demographics about special audiences, and opinion about related subjects to be used in a projected film or video tape product. Samples of the questions are: Which of the following alternatives best describes your attitude toward the present job you hold in the Air Force? 114 What percentage of time do you spend in direct supervision of other personnel? How satisfied are you with your present Air Force specialty? Which one of the following topics would you most like to know more about at this particular time of your career? [More than 15 alternatives were available for selection. ]7 Answers to these types of questions can be obtained by requesting a print out from the Director of Data Systems and Statistics. It would appear that the answers to this type of infor- mation from some 50, 000 people would be invaluable to the producer in determining what subjects to program for the Air Force Now film series or for using specific approaches to training films. Another resource the researcher might use, although the exact type of ”audience profile" data available is not known to the author, is the Office of Personnel Training Requirements at Head- quarters United States Air Force, in Washington, D. C. , and the Human Resources Organization at Air Training Command' 3 Lackland Air Force Base, San Antonio, Texas. There are several periodically published military reports which would be of value: "The USAF Per- sonnel Report, Characteristics and Attitudes from Sample Surveys 72U. S. Department of the Air Force, ”Airman Sample Survey 70-2, " Data Services Center (Washington, D. C. : n. p. , November 1969). 115 World Wide, " and the Department of Defense' 8 ”Profile of the U. S. Serviceman. " In addition, there are those film research studies and com- pilations which have been previously cited in this work in Chapter I. Also, a systematic survey of the current literature, professional periodicals and journals, and personal contact with professionals in the field will reveal current developments in the research area. These are but a few of the opportunities the researcher has at his disposal to effectively start an evaluation program. It would. seem the effectiveness of the research and evaluation program will hinge on the researcher' 3 background, and he should compensate in his weak areas by augmenting from the many resources available. Logistic and Operational Framework of the Program The fifth area of consideration deals with the very neces- sary requirement of administrative support, to include such items as office supplies, travel funding, printing requirements for the questionnaire, etc. , and access to computer facilities. It would appear that securing the necessary office supplies, printing, and other miscellaneous requirements of this nature could be taken care of without incurring a great expense from "in house" resources, that is, from funds currently being used to Operate the AAVS Center. I J ‘In 1‘ l. I. I! IZIU 1". 4' 116 Some funds should be programed by the Commander to allow the Special Assistant to initially get out and "beat the bushes” for useful contacts and material that will benefit the program, and also, to occasionally administer on -the-spot study programs in the field. Some funding should be programed for purchasing relevant literature and subscribing to periodicals and journals. All this appears to be minor compared to the overall AAVS budget, and it probably would not amount to more than several thousand dollars a year. Although there does not appear to be any set standard for establishing budgetary guidelines for research and evaluation, John L. Burns expresses his opinion for the financing of research in the education field, and he cites an example of an industrial research budget. He says: To finance such a program of research and development for education, each state and locality should set aside 2% of its education budget each year. On the basis of an aggregate edu— cation budget of $20 billion, this would amount to some $400 million. This may seem tiny in comparison with the sums spent by industry and defense. But it is enormous in contrast to the $20 million, about one -tenth of one per cent of our all school budget, now being spent on educational research. There is an urgent need for a full scale program of research and development in education comparable to the R & D programs now under way in defense and industry. Today, industrial research is a $12 billion operation. The ”industry of discovery" as one economist calls it. "In my own field of electronics, there is an axiom to the effect that-—either you get into research, or 117 you get out of business. " Many industrial corporations set aside 5 to 10 per cent of their annual budget for research. If AAVS set aside only one -half of one per cent of their $12. 8 million photographic budget, it would amount to some $64, 000 --the cost of one or two films. Within the logistical and operational framework are certain mission objectives which need to be established for the researcher and his ”staff. " The mission objectives that Captain Sedlik put forth in a proposal to Headquarters United States Air Force concerning the research branch concept are well worth mentioning here. He thought that, in addition to implementing the Diagnostic Pretesting Program, this research staff would be responsible: 1. To report to the Commander Aerospace Audio-Visual Service, his staff, and subordinate units on significant current and past research and theory in motion picture, television, edu- cation, instructional technology psychology, etc. , which bear on the scripting, production, and use of film and television media for instruction. 2. To assist requestors in the validation of released films by empirical tryout on large groups under strict experimental conditions insuring reliability and validity. 3. To conduct special studies, e. g. , on evaluative instru- ments and procedures on film, cataloguing, and distribution, etc. ‘ 4. To evaluate novel applications of film and television. 73John L. Burns, "The Challenge of Quality in Education, ” Problems and Controversies in Television and Radio, edited by Harry J. Skornia and Jack W. Kitson (Palo Alto: Pacific Books, 1968), pp. 184 -185. 118 5. To arrange for regular seminars in fihn, television, and related disciplines for the benefit of production and super- visory personnel. 6. To advise on effective presentations for conferences, briefings, and speeches. 7. To publish scholarly papers and actively participate in professional organizations. 74 These objectives are worthwhile and should be incorporated in the eventual structure of the research function. However, in the initial effort, and since my recommendation essentially involves only . one man, many of these objectives may need to be modified to fall within those limited capabilities of the position. I would propose that items 5, 3, 2, and 4 be established in that priority to initially get the program started, and that the others be augmented as the capability and function broadens in scope. Research Functions and Expected Benefits for the Aerospace Audio-Visual Service With Captain Sedlik' s objectives as a goal, a functional model has been developed which describes the tasks performed by the research operation, the information generated for the producer, and the use of this information by management for the end benefit of the Aerospace Audio-Visual Service. 74Jay M. Sedlik, "A Proposal for the Diagnostic Pretesting of the U. S. Air Force Instructional Motion Picture" (unpublished paper, United States Air Force Academy, 1969), p. 5. ' E I ll‘ Jill‘- 41 l I i I l l I .I l I III III II. . IIIIII III (III. . I I ll. ‘ l 119 .monmwvnm m5 an «0500.3 05 3 2:5an uvxogd “Em ogzuosom ecu—260.5 33050 am 2:9: :35 o Aswan—no.5 m: .«o «moo 05 3.52% 93 moi”— -omnno “59:55 m: 398 :25 :03? «360.5 23m; -333 2508.3 3?: d o .30:qu .«0 coflsfluumau may. o .305qu m0 mansumsom EB .9555 605259300 ask a .muosuoua 0&8QO Ho mcwcmwmmu new mcgflnm o .30:qu m><< no“ 29395. 98 mSwuwouQ mMCmu -wc3 no «cmfinofigon o .coflan -256 5 383.38 a5 3.5982 .moosuoun 2 35098 mo «5585 05 new ammooun “832:5me Quad: v29 .Awga no»: no 0.3.3“. umEQfima . .m .3 30:60.5 63.22 03» mo wcnsuonom w>nmcuc3m mo “08.3 5 monoummfiu 9E. .mmmcmsfiomto 8:8:me 0260.:— 3 nofiwmg «a 35 3369a Ho mug on? .33me mo meg 53.36 «5360.3 35 3360.3 mo avg can. .mcommvH 3.3m: umugoun 35 “man Ba 5 3260.5 .3283 no 308mm 25. 6303500 3 coroncoun 93me vmmamgo m5 ESE Hg 325.3 05 no 3:08on wmofi 3 953223 m Ema—hm 2.3. 6083 3 “360.3 05 303.3 3 moamwvaw uflaouua m5 .«0 moflmfmuowuwso 0.3. 696an neon 033 can “53.an mp 53 35 308%. Mo Show 23. .mocowusw Hmflcouoa v5 “0 mofimEouompEo on? .3269:— nwsoufi communes :mon 23: £0.53 muaoocoo watchamg .mo>fiomFo «one bo>flowtm 3 50:00.3 5 0323.805 up 68 “.95 398.30 uflsobna 0:9. 0 O .mmwooua 85323:. $5 mo 3:53 12.8QO 0 $3an moflmzmaomuaco 8:393 595 056098 no flaw mo mankind. o .mcoflgwnfioo m>bm¢po£m 5 3350.3 988 no 03» mo mcoflmgmkm o .333 @9297 was 98 «Sudanese mo $555 69830 2:539.» can coflmfiuofih mama—60.5 5 36.05 988 no OB» no 3083 .«o 292me 2523800 0 5260.3 3m: 05 mo 25500 05 3 QEmcoumeu 5 3035.3 25393 mo coflmgwkw o .maoauoum 3:332: no mmEBm con—ugh REF c 60.53 -uouomumgo @2833? 96 mo £92.34 o .m>no&um :mon 35: 3260.5 “an? 30>?” 0» 3033.3 mo mcoflggwsm o .aofiflnmoa twang mo monmflmaomumno 85:95 .«0 mowufim o .mconosuoua m><< mo 3thqu E350 0 6593.2» uoaauofim 31$ 53. £32 2:52 ”:32 unavocoo vmfifimv no «cosh—3969 o oo «tam 13m; . 033w moamouo< new muuocwm «cofiomgz new mcobocfim wcacflm and nobody. magammm madam. v.8 noosoounm ho.“ vouwumcwo :ofiafiuofl: 5am— mgnhg can .wcflwooq .wfisugaomwm "macro—=5 23m AWQOE HZMSH>M~EU¢ mogmmmm A Does this film make you proud of your association with the Air Force? A. Yes B. No C. No opinion If yes, because: A. The type of hard workers that are in the Air Force. B. The big job that has to be done and we are doing it. C. The Air Force is an effective organization and I am a member of it. D. The Air Force is my way of serving my country. If no, because: This film does not apply to me. Air Force people are dumb. I am just putting in my time. I don't believe in the mission of the Air Force. This film isn't typical of the Air Force. tTJUOtfiID What effect did this film have on your attitude about the Air Force? A. Positive B. No effect C. Negative 17. Please indicate your agreement/di agreement with the following statement by placing an X on the scale where it best represents your opinion. (Be sure the answer is traanerred to the answer sheet.) "The Air Force Now is a film series about the Air Force and Air Force people which communicate: by u ing a combination of stories about Air Force happenings and modern film treatment to tell the .s. ,' -. ” moo: lt‘t. Disagree : : : : : : Agree A B C D E F G How did you feel when the film ended? Friendly : : : : : : Unfriendly A B C D If ‘F G Inactive : : : : : : Active A B ’7 ID I? ‘F G Comfortable : : : : : : Uncomfortable A B C D ‘E F G Weak : : : : : : Strong A B C D ‘E F G Refreshed : : : Dull A B C D E F G Worthless : : Valuable A B C 4D ET F” G Pleasant : : : : : : Unpleasant A B C J E F ‘GI Bored : : : : : : Stimulated A B C D7 ‘E Fr ‘G Happy : : : Sad A B C D E F G Unsociable ' : : Sociable A B C D E F G Informed : : Uninformed A B C D E F I? Unemotional : Emotional A B C I7 E’ ‘F ‘G Satisfied : Unsatisfied A B C D E7 FI’ G Impersonal - Personal A B C D E7 ‘F G Patriotic Unpatriotic A B C ‘D E’ F’ G Unimportant : : Important A B C D If F‘ G’ Proud ‘ ' : Ashamed A B C D E F‘ G This section is directed toward the technical production and presentation of the film. 18. 19. 20. 21. 22. Use the following scale to answer questions 35-52. Circle the letter in the answer which reflects your opinion about the degree of quality for each item. A. Completely unsatisfactory B. Poor C. Below average D. Average E. Above average F. Well above average G. Completely outstanding 35 - 36. 37. 38. 39- 1+0. 1+1. 1.3. an. 1.5. 1+6. 1+7. 1+8. 1.9. 50. 51. 52. 53- 51+. 5.5- 56. 57. Color ' Camera movement Framing of scenes Quality of sound reproduction Use of sound effects Choice of camera shots Pacing and timing Length of segments Organization of material Appropriate appearance of talent Dress of talent Intelligibility of speech Sincerity of talent Enthusiasm of talent Personality of talent Believability of talent Dramatic interest Humor used A A B B C C D D E E F F G G 35- 36. 37. 38. 39- 1+0. Al. he. 2.3. 14h. 2+5. 1+6. 1+7. 118. 1+9. 50. 51. 52. Now that you have finished this segment of the survey be certain you transfer your answers to the answer sheet The purpose of the unit on the glassblower was: To point out an unusual skill in the Air Force. To use special camera and photography effects. To show the importance of glassblowing in the Air Force. Used for general interest. Strongly Agree Agree A B A B A B A B No Opinion Disagree C C D D Strongly Disagree Which of the following did you most like, enjoy, or find interesting about the glassblower segment? A. Information about glassblowing B. Knowledge that such a job existed in the Air Force C. Watching the Air Force employee make glass tubing 59- 6o. 61. 62. Which of the following did you most like, enjoy or find interesting about the glassblower segment? A. Music B. Color C. Photography How long do you think the segment on the glassblower lasted? A. 1 minute or less B. Between 1 and 2 minutes C. Between 2 and 3 minutes D. Between 3 and h minutes E. Between h and 5 minutes Where would this man most likely be employed? A. Air Force Systems Command at Wright-Patterson AFB B. Avionics Laboratory at Wright-Patterson AFB C. Air Force Logist.cs Command at Wright—Patterson AFB D. Aerospace Laboratory, Houston, Texas What did you think of the music that accompanied the segment on the glassblower? A. It was groovy B. It did not fit C. It was very appropriate D. Don't remember it ’E. Don't care How do you think the music related to the segment on the glassblower? . Don't remember the music . It added to the meaning of the story . It detracted from the meaning of the story . It neither added to nor subtracted from the meaning of the story UOwID 8h. 85. 86. The following is a list of words which reflect feelings about the glassblower segment of the film you saw. Circle the letter on the scale where it best represents your feeling. WHEN YOU HAVE COMPLETED THIS SECTION TRANSFER YOUR ANSWERS TO THE ANSWER SHEET. Friendly A B C E F Unfriendly 63. Strong A B C E F Weak 6h. Uncomfortable A B C E F Comfortable 65. Dull A B C B F Refreshing 66. Good taste A B C E F Poor taste 67. Worthless A B C E F Valuable 68. Patriotic A B C E F Unpatriotic 69. Boring A B C E F Stimulating TO. Hard to Easy to Tl. understand A B C E F understand Useless Useful 72. for me A B C E F for me Informative A B C E F Uninfonnative 73. High pressure A B C E F Low pressure 7h. Modern A B C E F Old fashioned 75. Proud A B C E F Ashamed 76. Unemotional A B C E F Emotional 77. Reliable A B C E F Unreliable 78. Unsatisfying A B C E F Satisfying 79. Impersonal A B C E F Personal 80. Believable A B C E F Unbelievable 81. Unusual A B C E F Commonplace 82. Fast A B C E F Slow 83. The purpose of the interview with General Benjamin 0. Davis was: Strongly No Strongly Agree Agree Opinion Disagree Disagree To announce his retirement. A B C D E To show that the blacks could have a future in the Air Force. A B C D E His message was directed to all the Air Force personnel, not just the blacks. A B C D E 87. General Davis was asked what the most significant event in his career was. Which answer comes the closest? . To show that the black man could be a success in the Air Force. To become a General. To be Vice Commander of US STRIKE Command. . To retire. To show the effectiveness of blacks in the service. MUCUJ3> c8. What was the reason for the informality of the scene showing General Davis and the interviewer walking along the water? It was an appropriate setting because the discussion was informal. It got away from a military setting. General Davis was already retired when that portion was shot. It was a Clever setting used in current television shows. . I have no idea. MUOUJID 89. How long do you think the segment covering General Davis in the film lasted? A. 1 minute or less B. Between 1 and 2 minutes C. Between 2 and 3 minutes D. Between 3 and h minutes E. Between h and 5 minutes 90. What do you think about the music used in the segment of the film concerning 8 General Davis? Don't remember it It didn't fit It was very appropriate It was groovy . Don't care meow». 91. What rank was General Davis? Brigadier General Major General Lieutenant General . General (four stars) UOU'JiD -8— 92 . 93- 9h. 95. 96 . 97. 98. '99. 100. 101. 102. 103. 10h. 105. 106. 107. 108. 109. 110. 111. 112. The following is a list of words which reflect feelings about General Davis segment of the film you saw. Circle the letter on the scale where it best represents your feeling. WHEN YOU HAVE COMPLETED THIS SECTION TRANSFER YOUR ANSWERS TO THE ANSWER SHEET. Friendly A B C Unfriendly 92. Strong A B C Weak 93- Uncomfortable A B C Comfortable 9h. Dull A B C Refreshing 95- Good taste A B C Poor taste 96. Worthless A B C Valuable 97. Patriotic A B C Unpatriotic 98. Boring A B C Stimulating 99. Hard to Easy to 100. understand A B C Understand Useless for Useful for 101. me A B C me Informative A B C Uninformative 102. High pressure A B C Low pressure 103. Modern A B C Old fashioned 10h. Proud A B C Ashamed lOS. Unemotional A B C Emotional 106. Reliable A B C Unreliable lO7. Unsatisfying A B C Satisfying 108. Impersonal A B C Personal 109. Believable A B C Unbelievable llO. Unusual A B C Commonplace 111. Fast A B C Slow ll2. 113. llh. 115. 116 . 117. 118 . 119. 120. 121. What was the purpose of the unit on the young married couple? Strongly No Agree Agree Opinion Disagree To show the problems of a young married couple in the Air Force. A B C D To show that it is possible to be happily married in the Air Force. A B C D To show that the benefits for the young married couple in the Air Force is better than in civilian life. A B C D To show that life in the Air Force can be interesting especially for a young married couple. A B C D What was the rank of the husband? A. Airman Basic B. Airman 1st Class C. Sergeant D. Staff Sergeant E. Tech Sergeant What job did he have? Auto mechanic Clerk Flight line maintenance Worked in the Commissary . A driver WUOWID Where was this couple stationed? . Scott AFB Randolph AFB Nellis AFB . Edwards AFB . Norton AFB txibotfilb Do you believe the couple in this unit were actually in the Air Force? A. Yes B. No (If N9 - What made you think they were not?) Circle the answer that most applies. A. The location was unbelievable B. They looked too much like actors C. The apartment was not real enough D. The dialogue was not natural E. Too good to be true F. Not possible for his rank -10- Strongly Disagree —‘7 -.—._——-— — H_-—,___—_____————— §_-__.__M——-_—___‘fl_—___—I—__-_ 122. How long do you think the segment on the Air Force couple lasted? A. 1 minute or less B. Between 1 and 2 minutes C. Between 2 and 3 minutes D. Between 3 and h minutes E. Between 4 and 5 minutes 123. What do you think of the music used with the segment on the Air Force couple? A. It did not fit B. It was groovy C. Don’t care D Do not remember E. It was very appropriate 12h. How do you think the music related to the segment on the Air Force couple? A. Don‘t remember the music B. It added to the meaning of the story C. It detracted from the meaning of the story D. It neither added to nor detracted from the meaning of the story -11- 125. 135. 136. 137. 138. 139. 1&0. 1&1. 1&2. 1&3. 1&&. 1&5. 1&6. 1&7. 1&8. 1&9. The following is a list of words which reflect feelings about the Air Force Couple segment of the film you saw. Circle the letter on the scale where it best represents your feeling. WHEN YOU HAVE COMPLETED THIS SECTION TRANSFER YOUR ANSWERS TO THE ANSWER SHEET. Friendly A B c F G Unfriendly Strong A B C F G Weak Uncomfortable A B C F G Comfortable Dull A B C F G Refreshing Good ta to A B C F G Poor taste Worthless A B C F G Valuable Patriotic A B C F G Unpatriotic Boring A B C F G Stimulating Hard to Easy to understand A B C F G understand Useless Useful for me A B C F G for me Informative A B G F G Uninformative High pressure A B C F G Low pressure Modern A B C F G Old fashioned Proud A B C F G Ashamed Unemotional A B C F G Emotional Reliable A B C F G Unreliable Unsatsifying A B C F G Satisfying Impersonal A B C F G Personal Believable A B C F G Unbelievable Unusual A B C F G Commonplace Fast A B C F G Slow What do you think was the purpose of the segment in the film about the sergeant's collection of unit emblems and patches? Strongly No Agree Agree Opinion Disagree To show an interesting hobby A B C D To point out some of the past traditions of the Air Force A B C D To introduce the sergeant as a typical NCO with an interesting sideline A B C D To show your unit's emblem (patch) A B C D -12- 135. 136. 137. 138. 139. 1&0. 1&1. 1&2. 1&3. 1&&. 1&5. Strongly Disagree E START YOUR SECOND ANSWER SHEET. LABEL IT ANSWER SHEET #2. How long do you think this segment lasted? A. B. C. D. E. What do radowr» 1 minute or less Between 1 and 2 minutes Between 2 and 3 minutes Between 3 and & minutes Between & and 5 minutes -Would you like to have seen more of this subject? Yes No No Opinion you think of the music used to accompany this unit of the film? It was very appropriate It did not fit Do not remember It was groovy Don't care How do you think the music related to the segment? wow» Don't remember the music It added to the meaning ofthe story It detracted from the meaning of the story It neither added to nor detracted from the meaning of the story When was the first patch used? UOWP 1918 191& 1917 1939 -13- The following is a list of words which reflect feelings about the Patch Collection segment of the film you saw. Circle the letter on the scale where it best represents your feeling. WHEN YOU HAVE COMPLETED THIS SECTION TRANSFER YOUR ANSWER: TO THE ANSWER SHEET. 16. 17. 18. 19. 20. 21. 22. 23. 25. A- —.—.-.—,A——___—— ———..-—..—-—~.———— — Friendly A B C E G Unfriendly 6, Strong A B C E G Weak 7. Uncomfortable A B C E G Comfortable 8, Dull A B C E G Refreshing 9, Good taste A B C E G Poor taste 10, Worthless A B C E G Valuable 11, Patriotic A B C E G Unpatriotic 12, Boring A B C E G Stimulating 13, Hard to Easy to 1&, understand A B C E G understand Useless Useful l5. for me A B C E G for me Informative A B C E G Uninformative 16. High pressure A B C E G Low pressure 17. Modern A B C E G Old fashioned 18. Proud A B C E G Ashamed l9. Unemotional A B C E G Emotional 20. Reliable A B C E G Unreliable 2l. Unsatisfying A B C E G Satisfying 22. Impersonal A B C E G Personal 23. Believable A B C E G Unbelievable 2&. Unusual A B C E G Commonplace 25. Fast A : B : C : D : E : F : G Slow 26. What was the purpose of the sequence concerning the Vietnamese Air Force? Strongly NO Strongly Agree Agree Opinion Disagree Disagree It pointed out the rapid Vietnamization of the Air Force A B C D E It was to let the audience know the Vietnamese were making a contribution A B C D E It showed that the Vietnamese are coming closer to con- trolling their own military , force A B C D E -1h- How long do you think the segment on the VNAF lasted? A. l minute or less B. Between 1 and 2 minutes C. Between 2 and 3 minutes D. Between 3 and A minutes E. Between A and 5 minutes What do you think of the music used in segment concerning the Vietnamese Air Force? It was very appropriate It did not fit Do not remember It was groovy . Don't care MUOWID How do you think the music related to segment about the Vietnamese Air Force? A. Don't remember the music B. It added to the meaning of the story C. It detracted from the meaning of the story D. It neither added to nor detracted from the meaning of the story -15- The following is a list of wor s which refl7ct feelings about the Vietnamese . Air Force segment of the film you saw. Circle the letter on the scale where it best represents your feeling. WHEN YOU HAVE CCMPLETED THIS S TTION TRANSFER YOUR ANSWERS TO THE ANSWER SHEET. &3. &&. &5. &6. &T. &9. SO. 52. 53- 5&. 55- 56. 57. Friendly A B C D E F G Unfriendly 33. Strong A B C D E F G Weak 3A. Uncomfortable A B C D E F G Comfortable 35. Dull A B c D E F G Refreshing 36. Good tagte A B C D E F G Poor taste 3?. Worthless A B C D E F G Valuable 38. Patriotic A B C D E F G Unpatriotic 39 Boring A B C D E F G Stimulating #0. Hard to Easy to &l. understand A B C D E F G understand Useless Useful &2 for me A B C D E F G for me Informative A B c D E F G Uninformative &3. High pressure A B C D E F G Low pressure &&. Modern A B c D E F G Old fashioned &5. Proud A B c D E F G Ashamed &6. Unemotional A B C D E F G Emotional &7. Reliable A B C D E F G Unreliable &8. Unsatisfying A B C D E F G Satisfying &9. Impersonal A B C D E F G Personal 50. Believable A B C D E F G Unbelievable 51. Unusual A B C D E E G Commonplace 52. Fast A B C D E E G Slow 53. What was the purpose of the Medal of Honor segment of this film? Strongly Strongly Agree Agree Opinion Disagree Disagree To instill pride in the Armed Forces A C D To show the display in the Pentagon A C D To pay honor to that that received the Medal of Honor A C D E To explain what the Medal of Honor is A C D E -16- _‘-—._._-—__- _—‘ ___..._ .—___— —.—-—-—.—-—- _'__——_ 58. 59- 60. 61. 62. What word appears on the Medal of Honor? U0w3> Valor Gallantry Heroism Honor How many Medals of Honor were awarded to airmen in World War 11? (Army Air Corps) mtjowib o u l2 38 26 How long do you think the segment about the Medal of Honor lasted? tdtjriujp 1 minute or less Between 1 and 2 minutes Between 2 and 3 minutes Between 3 and A minutes Between A and 5 minutes What did you think of the music used to accompany the segment concerning the Medal of Honor? A. B. C. I).~ E. It was groovy It didn't fit Don't remember it It was very appropriate Don't care How do you think the music related to the segment concerning the Medal of Honor? Don't remember the music It added to the meaning of the story It detracted from the meaning of the story It neither added to nor detracted from the meaning of the story -17- The following is a list of words which reflect feelings about the Medal of Honor segment of the film you saw. Circle the letter on the scale where it best represents your feeling. WHEN YOU HAVE COMPLETED THIS SECTION TRANSFER YOUR ANSWER? TO THE ANSWER SHEET. Friendly A B c D E F G Unfriendly 63 Strong A D C D E F G Weak 6D. Uncomfortable A B C D E F G Comfortable 65 Dull A D C D E F G Refreshing 66. Good taste A B C D E E G Poor taste 6? Worthless A D C D E F G Valuable 65. Patriotic A B C D E F G Unpatriotic 69. Boring A B C D E F G Stimulating 70. Hard to Easy to Tl. understand A D C D E F G understand Useless for Useful 72. me A B C D E F G for me Informative A B C D E F G Uninformative 73. High pressure A B C D E F G Low pressure 7A- Modern A B C D E F G Old fashioned 75. Proud A B C D E F G Ashamed 76. Unemotional A B C D E F G Emotional 77. Reliable A B C D E F G Unreliable 78. Un atisfying A B C D E F G Satisfying 79. Impersonal A B C D E F G Personal 80. Believable A B C D E F G Unbelievable 81. Unusual A B C D E F G Commonplace 82- Fact A B C D E F G Slow 83. -18- SOURCES CONSULTED SOURCES CONSULTED Abelson, H. I., and Karlins, M. Persuasion: How Opinions and Attitudes Are Changed. New York: Springer, 1959. Alexander, Richard L. , Colonel, United States Air Force. Personal interview, Aerospace Audio-Visual Service Center, Norton Air Force Base, California, March 25, 1970. Archer, Norma A. , Major, United States Air Force. Personal interview, Aerospace Audio-Visual Service Center, March 25, 1970. Auer, J. Jeffery. An Introduction to Research in Speech. New York: Harper 8: Brothers, 1959. Backstrum, Charles H. , and Hursh, Gerald D. Survey Research. Chicago: Northwestern University Press, 1963. Beinhorn, Leo, Colonel, United States Air Force. Speech presented for the Aerospace Audio-Visual Production and Planning Con- ference, Norton Air Force Base, California, January 26, 1970. Belson, William A. The Impact of Television. Connecticut: Archon Books, 1968. Bjoring, Donell E. , Colonel, United States Air Force. Personal let- ter, Norton Air Force Base, California, February 10, 1970. Bloom, B. , ed. Taxonomy of Educational Objectives-~Handbook I: Cognitive Domain. New York: David McKay Company Inc. , 1966. Bureau of Social Science Research. Report of the Bureau. Cumulative Effects of Three Major Emphasis Films on Attitudes of Basic Trainees. Washington, D. C. , 1964. 140 141 Burns, John L. "The Challenge of Quality in Education. " Problems and Controversies in Television and Radio. Edited by Harry J. Skornia and Jack W. Kitson. Palo Alto: Pacific Books, 1968. Campbell, D. T., and Stanley, J. C. Experimental and Quasi- experimental Designs for Research. Chicago: Rand McNally, 1963. Chapman, Don H. Personal interview, Aerospace Audio -Visual Service Center, Norton Air Force Base, California, March 24, 1970. Children' 8 Television Workshop. ”Children Help Share Their Own TV Series. ” News Release, National Educational Television (NET), n. d. Chu, Godwin C. , and Schramm, Wilbur. Learning from Television: What the Research Says. Washington, D. C. : National Association of Educational Broadcasters, 1967. Creeden, John E. Personal letter, Chicago, Illinois, January 29, 1970. Curl, David H. "Why Do Films Fail?" Photo Methods for Industry. Vol. 9, November 1966, pp. 66—68. Deutschmann, Paul J. ”Measurement in Communication Research. " Introduction to Mass Communications Research. Edited by Ralph O. Nofziger and David M. White. Baton Rouge: Louisiana State University Press, 1958. Dexter, Lewis A. , and White David M. , eds. People, Society, and Mass Communications. New York: Free Press, 1964. Ewing, T. A. ”A Study of Certain Factors Involved in the Study of Changes of Opinion." J. Soc. Psychol. , XVI (1942), 63 —88. Gallerani, Altero, Colonel, United States Air Force. Personal interview, Aerospace Audio -Visual Service Center, Norton Air Force Base, California, March 25, 1970. Gibeaut, Robert W. Personal letter, Burbank, California, Feb- ruary 2, 1970. 142 Handel, Leo A. Hollywood Looks at Its Audience. Urbana: The University of Illinois Press, 1950. Internal Information Division, Secretary of the Air Force, Office of Information. "Concepts and Analysis of the Air Force Now. " Staff paper. Washington, D. C. , n. d. (Typewritten) Kanner, Joseph H. , Dr. , Chief, Audio —Visual Applications Office, Department of the Army. Personal interview, Office of Assistant Chief of Staff for Communicationa/E1ectronics, Washington, D.C. , March 19, 1970. Katz, D. "The Functional Approach to the Study of Attitudes. " Publ. Opinion Quart. , XXIV (1960), 163 -204. Kerns, Paula V. , Captain, United States Air Force. Personal interview, Aerospace Audio—Visual Service Center, Norton Air Force Base, California, March 24, 1970. Kiesler, C. A., Collins, B. E., and Miller, N. Attitude Change: A Critical Analysis of Theoretical Approaches. New York: Wiley, 1969. Knudsen, Steve. Personal letter, Ames, Iowa, January 30, 1970. Kumata, Hideya. An Inventory of Instructional Television Research. Ann Arbor: Educational Television and Radio Center, 1956. Lazarsfeld, P. F., Berelson, Bernard, and Gaudet, H. The People' 8 Choice. New York: Columbia University Press, 1948. Lazarsfeld, Paul, and Stanton, Frank, eds. Radio Research. New York: Duell, Sloan and Pearce, 1943. Mager, Robert F. Preparing Instructional Objectives. California: Fearon Publishers, 1962. McGuire, W. J. "The Nature of Attitudes and Attitude Change. " Handbook of Social Psychology. Edited by G. Lindzey and E. Aronson. Reading, Mass: Addison-Wesley, 1969. Vol. III, 136—314. 143 National Industrial Conference Board. Evaluating Media: Studies in Business Policy No. 121. New York: n.p. , 1966. Newcomb, Theodore M , Turner, Ralph H., and Converse, Philip E. Social Psychology. New York: Holt, Rinehart, and Win- ston, Inc., 1966. Oswald, Jack H. , Major, United States Air Force. "A Survey of Evaluation Programs on the Effectiveness of Air Force Films. " Unpublished research paper, University of Southern California, 1964. Oswald, Jack H. , Major, United States Air Force. Personal letter, Washington, D.C., April 23, 1970. Pfister, Edward J. Personal letter, Bloomington, Indiana, January 29, 1970. Sedlik, Jay M. "Diagnostic Pretesting of the United States Air Force Instructional Motion Picture Theory and a Proposed Model. " Unpublished Ph.D. dissertation, University of Southern California, 1969. Sedlik, Jay M. , Captain, United States Air Force. "A Proposal for the Diagnostic Pretesting of the U. S. Air Force Instruc- tional Motion Picture. " Unpublished paper, United States Air Force Academy, California, 1969. (Typewritten) Stear, Shelton K. Personal letter, New York, New York, Feb- ruary 4, 1970. Sutton, Ronald E. Personal letter, Washington, D. C. , February 3, 1970. U. S. Congress. Senate. Committee on Armed Services. Military Cold War Education and Speech Review Policies. Report, Special Preparedness Subcommittee. 87th Cong. , 2nd Sess. , 1962. U. S. Department of Health, Education, and Welfare. Research in Instructional Television and Film, by J. C. Reid and D. W. Macheman. Office of Education. Washington, D.C.: Government Printing Office, 1967. 144 . Department of the Air Force. "Airman Sample Survey 70-2. " Washington, D. C. : Data Services Center, November 1969. . Department of the Air Force. Programing and Producing Motion Pictures. Air Force Regulation 95 -14, Washing- ton, D.C. , 1967. . Department of the Air Force. Air Force Film Library Series. Air Force Regulation 95 -11, Washington, D. C. , February 6, 1967. . Department of the Air Force, Headquarters Aerospace Audio-Visual Service. Organizations and Functions. Aerospace Audio -Visual Service Manual 23 -1, Norton Air Force Base, California, 1969. . Department of the Navy. The Relative Effectiveness of Several Film Variables in Modifying Attitudes: A Study of the Application of Films for Influencing the Acceptability of Foods. Port Washington: Office of Naval Research, 1956. . Department of the Navy and U. S. Department of the Army, Co-operating. Instructional Film Research 1918-1950, by Charles F. Hoban and E. G. van Ormer. Technical Report No. SDC 269-7 -19. Port Washington: Office of Naval Research, 1951. . Department of the Navy and U. S. Department of the Army, Co-operating. Instructional Film Research Reports, by C. R. Carpenter and L. P. Greenhill. Technical Reports SDC 269—7 -1 through 61. Port Washington: Office of Naval Research, 1953 and 1956. Vol. I and Vol. II. . War Department. "Effects of Film on Attitudes. " What the Soldier Thinks. Morale Services Division, Army Services Forces. Washington, D.C.: n.p., 1943. Weaver, W. R. "Studios Use Audience Research to Learn What Pleases Customers. " Motion Picture Herald, July 20, 1946. N STRTE UN V. LIB HICHIGQ I llllllllllllll H llllllllllll £3112 RnRIEs WIN" (0 9310423138