THE ISSUE OF PEER REVIEW: A CASE STUDY OF THE AGENCY -T0 - INDIVIDUAL APPROACH TO FEDERAL FUNDING 0F SCIENTIFIC RESEARCH Dissertation for the Degree-of Ph. D. MICHIGAN STATE UNIVERSITY EDMUND F. VANDETIE 1977 J'JZ’J! :1". ; I -6- ”‘41:? r'" " 1' n Pas; W _ ' ‘ A , LI. .. I Michib. -;c :1...,, Univcrsi .y This is to certify that the thesis entitled THE ISSUE OF PEER REVIEW: A CASE STUDY OF THE AGENCY-TO-INDIVIDUAL APPROACH TO FEDERAL FUNDING OF SCIENTIFIC RESEARCH presented by EDMUND F. VANDETTE has been accepted towards fulfillment of the requirements for PH.D. degreeinHIGHER EDUCATION & ADMINISTRATION viz/W M Major professor Date March 15. 1977 0-7639 ABSTRACT THE ISSUE OF PEER REVIEW: A CASE STUDY OF THE AGENCY-TO-INDIVIDUAL APPROACH TO FEDERAL FUNDING OF SCIENTIFIC RESEARCH by Edmund F. Vandette The federal government historically has chosen to support scientific research in a number of areas through two basic mechanisms. One is the agency-to-institution approach wherein the federal agency concerned provides a research grant directly to a selected institution. The second, which has become more prevalent since World War II, is the agency- to-individual approach wherein the federal agency concerned awards a research grant directly to the individual involved. Therefore, the decision regarding who does the research and tumv it is done is not decentralized, but, to a large extent, lodged.in the federal agency which thus retains the ultimate decision-making power rather than the institution having such authority as in the first basic mechanism. .A prime characteristic of the agency-to-individual approach is peer review, the underlying assumption of which is that proposals should be judged by people who have the substantive expertise needed to make such a judgment. Yet in recent years there has been increasing criticism directed at this approach. Sm and its implication for the workability of the peer This study's focus was the nature of that critici Edmund F. Vandette review system. The model for this study was the National Science Foundation because either individual or peer panel review is utilized by the Foundation in its evaluation of the vast bulk of formal requests for funds. Moreover, the process employed by NSF was made particularly visible in a series of oversight hearings held in the summer of 1975 by the Congres- sional Subcommittee on Science, Research, and Technology. In addition to testimony from these hearings and re- commendations made by the Subcommittee based on this testi— mony, this study utilized the historical method to determine past trends in policies and practices relative to the award- ing of research grants. This researcher also employed the interview technique in gathering relevant data on a personal basis from many sources in government and educational insti- tutions and within the National Science Foundation itself. The study was conducted in order to answer questions ;pegarding whether or not the peer review system (1) provides for the advance of science in the most effective way possi- blew (2) is fair and impartial and subject to political in— fluence and geographical favoritism; (3) is economically feasjlile; (4) promotes "grantsmanship" and is too secretive. Based on the anwers to these questions gained through the research methodology described above, this study con- (3 tjaat there is confidence in the peer review system, clude that indeed, data indicate that agency staff could not per- I form as well without the system intact. In addition, though Edmund F. Vandette a lack of confidence in the willingness of the system to support innovative research was expressed, consensus was that no method superior to peer review has been found for judging the scientific competence of prOposals. Regarding secrecy, favoritism, and "cronyism," this study found that less candid reviews would be made if signed verbatim re- views were available to applicant and public; that the ex- istenceiof "cronyism" is possible, though it had not been witnessed by those consulted by this researcher; and that geographic and economic distribution of grants appears nor- mal. Finally, the research suggests a need for further study of the overall effect of the competition promoted by peer review on university communities. THE ISSUE OF PEER REVIEW: A CASE STUDY OF THE AGENCY-TO-INDIVIDUAL APPROACH TO FEDERAL FUNDING OF SCIENTIFIC RESEARCH by =L (HI: ‘ 'L Edmund F. Vandette A DISSERTATION submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Administration and Higher Education 1977 ACKNOWLEDGMENTS The author is indebted to many people, more than can be acknowledged by name. He is espeCially indebted to Drs. Sweetland, Featherstone, Anderson, and Weinheimer, persons of the highest order to whom deep gratitude is extended. Dr. William Sweetland and Dr. Richard Featherstone deserve a special note of appreciation. They were the di- rectors of this research and more--teachers and friends, men in whom there was no inconsistency: total educators. Also thanks to Dr. Philip Kearney for his constant encouragement and his permission to utilize his valuable time and pertinent suggestions. And a special thank you, too, to the interviewees who were kind and considerate enough to share their insights into and knowledge of this topic. To Anne Cauley goes appreciation for her editorial assistance and great patience. ‘ And, finally, the most heartfelt gratitude of all is towed to members of my family: to my father, Bernard, and :mother, Elsie, who simply gave a lifetime of self-sacrifice, love, and encouragement; and especially to my wife, June, and our sons, Edmund and Robert, whose love, patience, and unselfishness made this endeavor possible and worthwhile. ii TABLE OF CONTENTS LIST OF FIGURES . . . . . . . . . . . . . . . . . . CHAPTER I INTRODUCTION AND STATEMENT OF THE PROBLEM. . Introduction . . . . . . . . . . . . . . . . The Problem . . . . . . . . . . . . . . . . Background of the Problem . . . . . . . . . The Origin of the Agency-to-Institution Approach . . . . . . . . . . . . . . . . The Origin of the Agency-to-Individual Approach . . . . . . . . . . . . . . . . The Variations of the Agency-to-Individ- ual Approach . . . . . . . . . . . . . . Invitation for Bid . . . . . . . . . . Request for Proposals . . . . . . . . Grant Announcement . . . . . . . . . . Unsolicited Proposal . . . . . . . . . Sole Source Contract . . . . . . . . . Significance of the Study . . . . . . . . . Study Design and Research Questions . . . . Limitations of the Study . . . . . . . . . . Outline of the Study . . . . . . . . . . . . II AN HISTORICAL REVIEW OF FEDERAL RESEARCH FUNDING: FROM LAND GRANT COLLEGES TO THE NATIONAL SCIENCE FOUNDATION . . . . . . . . Introduction . . . . . . . . . . . . . . The Agricultural Experiment Station: The Development of the Agency-to-Institution Approach . . . . . . . . . . . . . . . . . . The German Experiment Station Movement. . The Hatch Experiment Act . . . . . . . . New Developments in the Twentieth Century . . . . . . . . . . . . . . . . . The Need for Wartime "Hardware": The Devel- opment of the Agency-to-Individual Approach . . . . . . . . . . . . . . . . . . The Forerunners . . . . . . . . . . . . . The Bush Report . . . . . . . . . . . The Office of Naval Research . . . . . iii Page vi DUMP I—‘ 10 ll 11 13 14 15 16 18 19 20 22 22 23 24 25 28 29 31 31 33 CHAPTER Page The National Science Foundation . . . . . . 35 NSF Purposes and Organizational Policy . 35 Procedures for Grant Allocation . . . . 37 The Role of the Reviewer . . . . . . . . 43 Economic Aspects of Peer Review . . . . 44 Evaluations of the Agency-to-Individual Approach Employed by NSF . . . . . . . . 45 III DESIGN OF THE STUDY AND RESEARCH METHOD- OLOGY . . . . . . . . . . . . . . . . . . . 47 Design of the Study . . . . . . . . . . . . 47 Research Methodology . . . . . . . . . . . 51 Documentary Data . ._. . . . . . . . . . 51 Interview Data . . . . . . . . . . . . . 51 IV STUDY FINDINGS . . . . . . . . . . . . . . 56 Introduction . . . . . . . . . . . . . . . 56 Provision for the Advancement of Science . 57 The Brookings Institute Seminar . . . . 58 Further System Drawbacks . . . . . . . . 65 The Dilemma of the Innovator . . . . . . 66 NSF Support of Quality Research . . . . 68 The Peer Review System: A Fair Approach? . 69 Reviewer Bias and "Cronyism" . . . . . . 69 Political Influence . . . . . . . . . . 77 Geographic Distribution of Grants . . . 78 Industrial Research Opinion Poll . . . . 98 The Economic Aspects of Peer Review . . . . 99 Economic Arguments Against . . . . . . . 100 Economic Arguments In Favor . . . . . . 102 Grantsmanship and the Issue of Openness . . 103 Grantsmanship . . . . . . . . . . . . . 104 System Openness . . . . . . . . . . . . 105 V CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE RESEARCH o o o o o o o o o o o o o o o o o 117 Conclusions . . . . . . . . . . . . . . . . 118 Peer Review and Scientific Excellence . 118 Peer Review and the Question of Fairness . . . . . . . . . . . . . . . . 121 Peer Review and the Cost Factor . . . . 126 Peer Review and Confidentiality . . . . 127 summary 0 O C O O O O O O O O O O O O O 129 Recommendations for Future Research . . . . 130 APPENDICES Appendix A: Complete Listing of Interviewees . . . . . . . . . . . . . . . 134 iv Page Appendix B: Industrial Research Opinion Poll . . . . . . . . . . . . . . . . . . . . 136 SELECTED BIBLIOGRAPHY . . . . . . . . . . . . . . . 141 LIST OF FIGURES Figure 1 Flow Chart of the Movement of a Research Pro- posal Through the Foundation . . . . . . . . . 2 Scientific Research Project Support Dollars for Each State Plotted against State Popula- tion 0 O O O O O O O I O O O O I O O O O O O O 3 Scientific Research Project Support Dollars for Each State Plotted against NAS and NAE memers O O C O O O O O O O O O O O O I O O O 4 Scientific Research Project Support Dollars for Each State Plotted against Publications in Leading Scientific Journals . . . . . . . . 5 Scientific Research Project Support Dollars for Each State Plotted against NSF Fellowship ReCipients O O O O O O O O O O O O C O O O O O 6 NSF Reviews from Each State Plotted against Population . . . . . . . . . . . . . . . . . . 7 NSF Reviews from Each State Plotted against NAS and NAE Members 0 I O O O O O O O O O O O 8 NSF Reviews from Each State Plotted against Publications in Leading Scientific Journals 9 NSF Reviews from Each State Plotted against NSF Fellowship Recipients . . . . . . . . . . vi Page 40 82 84 86 88 90 92 93 94 CHAPTER I INTRODUCTION AND STATEMENT OF THE PROBLEM INTRODUCTION The federal government historically has chosen to support scientific research in a number of areas through two basic mechanisms. One mechanism, which had its or- igin in the agricultural land grant system, initiated by the Morrill and Hatch Acts in the mid-18003, is the agency- to-institution approach wherein the federal agency con- cerned provides a research grant directly to a selected institution. Because the institution makes the majority of the decisions on the specific areas of research to be pursued, the approach typifies the decentralized research effort. The prime focus of the federal agency is simply to supply the necessary resources without giving much more than general direction as to the type of research that is going to be done and the manner in which it will be carried out. The second basic mechanism, which has become more prevalent since World War II, is the agency-to-individual approach wherein the federal agency concerned awards a .research grant directly to the individualinyolvedd This approach has a number of variations, but is best charac- terized by the fact that the federal agency indicates that it wants certain research undertaken and then awards grants directly to the individual concerned with the expec- tation that he or she will complete the research. In other words, the decisions regarding who does the research and how it is done are not decentralized but, to a large ex- tent, lodged in the agency. The federal agency thus re- tains the ultimate decision making power rather than the institution having such authority as in the first basic mechanism. THE PROBLEM In recent years there has been been increasing crit- icism directed at the agency-to-individual approach when that approach involves, as one of its prime characteristics, a review of individual project proposals by peer groups. A "peer review system" is here defined as a method of evaluating a specialized creation--such as a proposal to perform scientific research--which involves having a group of people knowledgeable in the area of specialization eval- uate the creation. The experts are called "peers," the term loosely deriving from the relationship between the proposer and the evaluators.l Thus, the underlying 1National Science Foundation Peer Review, Vol. I., A Report of the Subcommittee on Science, Research, and Tech- nology of the Committee on Science and Technology. U.S. House of Representatives, Ninety-Fourth Congress, Second Session (Washington, D.C.: U.S. Government Printing Office, 1976), p. 13. assumption of such group review is that proposals should be judged by people who have the substantive expertise needed to judge the proposal. The present investigation is concerned with that variation of the second basic mechanism described, namely the agency-to- individual approach involving peer review as an essen- tial element, and with the various criticisms which have been aimed at its operation (hereinafter agency-to-indi- ‘vidual approach and peer review system will be used inter- changeably). BACKGROUND OF THE PROBLEM The Origin of the Agency-to-Institution Approach Federal involvement in higher education is not a new phenomenon. Federal lands were set aside for public education in the Northwest Ordinance of 1787.2 Further, the federal government was basically responsible for the creation of America's unique system of land-grant colleges and universities when, through the passage of the Morrill Act of 1862, it granted substantial amounts of land to the states on the condition that they sell the land to endow at least one college in each state.3 The Morrill Act did not specify that these were to be "public" colleges, 2Christian K. Arnold, "Higher Education: Fourth Branch of Government?" Saturday Review (January 1964), p. 61. 3 Ibid. but the establishment of the institutions under the pro- visions of the Act helped to crystallize the concept of public higher education in the United States. The Act did require that these institutions be under the juris- diction of the state legislature and did define, in general terms, the types of programs, students, and oversight that were to be involved. In 1953 the National Manpower Council could report that "the most important single govern- ment step in connection with the training of scientific and professional manpower" was the Morrill Act.4 As an extension of the philosophy and principles of the Morrill Act, the Hatch Act in 1887 directed the estab- 1ishment of tax-supported institutions for agricultural research, and its comprehensive phraseology permitted a broad.range of agricultural station functions. The pre- amble:nmrked out the wide limits of authorized activity, frtnn "acquiring and diffusing . . . practical information" to "scientific investigation and experiment respecting the principles and applications of agricultural science."S With equally generous phrasing, Section 5 instructed the governing board of each experiment station to use the Hatch funds for "paying the necessary expenses of conducting investigations and experiments and printing and distributing 4Ibid. 5H. C. Knoblauch, E. M. Law, and W. P. Meyer, State Agricultural Experiment Stations (Washington, D.C.: United States Department of Agriculture, 1962), p. 81. the results."6 Thus, the law, which allowed each station to shape its_own definition of research activity, approved both "pure" science and "applied" science; yet it did not prescribe the features which research activity must possess in order to be classified as "scientific" or "applied" and, therefore, to qualify for federal support. Moreover, the lack of specificity in the statute let the stations specialize, if they chose, in providing only known infor— mation to the farmer rather than in discovering new know- ledge. The broad latitude of the Hatch Act became the key- stone of its popularity. It won the support of those people who saw in it a means for stimulating scientific effort toward new discovery. It also gained cooperation from an increasing number of peOple who regarded it not only as an incentive toward new discovery, but also as an immediate and practical method for wide application of scientific knowledge to farming.7 Under the Act, funds appropriated annually by the Congress are distributed among the states by a formula that takes into account such factors as the size of the rural population in each of the several states and plans for regional, interstate research. Programs supported by these funds are initiated by plans outlined within the 61bid. 71bid. institutions under the supervision of the directors of the experiment stations and of the extension programs. These directors are solely responsible for the adminis- tration and guidance of the programs, but both past per- formance and future plans are reviewed and evaluated annually by the Secretary of Agriculture and by the Con- gress during appropriation hearings. The institutions have a wide degree of freedom in in the use of these funds (a clear example of the agency- to-institution approach at work). Experiment stations, for instance, may undertake basic research, as in biolog- ical science, or applied research. The directors may--in fact, are encouraged to--pool their resources with other experiment stations to attack regional problems; and the funds may be used for research carried out in c00peration with industrial organizations, state and county govern- ments, and foundations. To a considerable degree, federal funds can also be used for overhead expenses. They can even be utilized to pay the institutionhs contribution to retirement sys- tems, social security, and group insurance programs. They may be used to pay the necessary expenses of administra- tive planning and direction and, for research programs, they may be employed in the construction of buildings, the buying of lands, and the purchase of fixed equipment. The basic legislation for these Hatch Act programs is "open-ended," and the continuity provided through the .f ' ——. vi.v A .,__C..—.—‘ +___—_ V4 annual appropriations makes it possible for the university officials to plan their programs years in advance. Act- ually, the law authorizes the experiment stations to anticipate appropriations in order to contract for equip- ment essential for carrying out the research programs. The research effort operationalized through this system is totally integrated with the instructional programs of the institutions involved and has strengthened, rather than weakened, them. In fact, the agricultural experiment stations, under the law, must be departments of the univer- sities in which they are located.8 The Origin of the Agency-to-Individual Approach The type of research that grew up during World War II, which was "new" only in its expanded degree of federal involvement, did not lend itself to the agency-toéinstitu- tion type of relationship promoted by the Hatch Act en- couraging agricultural research. The government needed specific answers to specific questions, and the answers were often in the form of "hardware."9 To get these answers, the federal agency that needed them simply went out and purchased the services of the person or organiza- tion the agency felt most likely to do the best possible job. Unlike the system established under the Hatch Act, 8Harold Orlans, Science Policy and the University (Washington, D.C.: The Brookings Institution, 1968), p. 90. 9The term "hardware" is here used to mean tools, machinery, etc., the production of which is in high demand during wartime. there was little pretense in the arrangements for providing general and continuing support of research; the government was simply purchasing short-term services it needed and which it could not itself provide. This approach led, at one extreme, to the establishment of laboratories essen- tially owned by the government but operated by universities, such as the wartime Harvard Underwater Sound Laboratory, and, at the other extreme, to the temporary employment of a university scientist by a governmental agency. Federal involvement in research did not stop at the end of World War II, but rather grew and flourished. War- time experiences, in fact, demonstrated and accelerated the growth of our national dependence on the extension and application of scientific knowledge as a necessary condi- tion for the continued welfare of our people, the preser- vation of our national integrity, and even the maintenance of human dignity and freedom. However, the nature of the research needed did change. The need became generalized, and long-term goals replaced short-term objectives. "Hardware" became less important than broad-based explorations of fundamental phenomena. The "production" of engineers and scientists rated equal Priority with the employment of those already available. During the war the United States had engaged in a "frantic and rather ruthless exploitation" (said J. Robert Oppen- heimer) of the basic knowledge accumulated in the years before the war. Now it needed to engage in a "sober modest attempt to penetrate the unknown," to replenish and increase our stock of basic knowledge.10 Some people foresaw this change, and legislation was passed to provide mechanisms for coping with it. The Na- tional Institutes of Health were established with broad responsibilities for research relating to health; in 1946 the Atomic Energy Commission was given research responsibil- ities involving nuclear processes. Then, four years later, the National Science Foundation was created to "promote the progress of science." Even for the more narrowly "mission-oriented" agencies, research tended to become broader, more basic, less "hardware"-ish.ll The posture of the government became more that of a patron of research than that of a purchaser of services. In this emerging system the university itself was largely bypassed. Individual members of faculties began to submit proposals for research to the appropriate agency, often after formal discussions with staff personnel at the agency. These prOposals were reviewed by advisory boards composed of eminent scientists (often the elder statesmen in the field), and, on the basis of their recommendations, grants were made or contracts let. The basic relationship established, then, was an agency-individual one, in contrast to the agency-university relationship established with 10Arnold, p. 62. 11Orlans, p. 93. 10 agricultural research and extension. The Variations of the Agency-to-Individual Approach Under the agency-to-individual approach federal agencies use five basic means to support research activi- ties: invitation for bid, request for proposal, grant announcement, unsolicited proposal, and sole source con- tract. Within these means there are specific elements of the peer review system which can be operative alone or in combination. These elements include: 1. staff judgment - there are two extremes here: the heavy reliance on staff to determine how a proposal will be reviewed, select reviewers, use reviewers as only advisors, and make the decision on funding; or the minimal use of staff for only clerical functions; 2. individual peer review - occurs when an expert outside the staff evaluates a proposal without consulting other evaluators; 3. panel peer review - occurs when a group of ex- perts meets together to evaluate a proposal; the panel may be a standing or an ad hoc group; 4. site visit - takes place when a team composed of agency staff and/or outside experts visitse1scientific facility when the quality of that facility is a determin- ant in the funding of the proposal; 5. policy board - usually a standing group with rotating membership that considers especially crucial 11 questions regarding a proposal's appropriateness for a particular agency; 6. consultation with other funding agencies - is necessary when a proposal requests partial funding from I 12 several agenc1es. Invitation for Bid The most directive type of procurement is one whereby the agency specifies and justifies some clear need and gives the detailed specifications of the outcome it wants. In such cases the agency itself issues its developed specifications htas full detail as possible in 13 what is called an Invitation for Bid. The bid acceptance consists of nothing more than the offerer signing his name to the solicitation, thereby indicating that he will comply with the specifications, and giving a price; indeed, the only consideration in this kind of procurement is price-- the low bidder gets the contract. Request for Proposals The second type of procurement, and one which is used very extensively at the National Institute of Educa- tion, is called the Request for Proposals. In this case the requirement again is clear and indeed there may be an indication by the agency of what the end product should 12NSF Peer Review Vol. I, pp. l3, 14. 13Tom Clements, Personal Interview, National Insti- tute of Education, Washington, D.C;,‘December 1975.‘ 12 look like; but the requirement is in an area where there is sufficient ambiguity so that alternative paths to attaining that requirement may be used. There is a little bit more Opportunity for offerer initiative and creativity here. In this second type of agency procurement, the pro- posals that come in go through a two-stage review. The first review is one of the technical merits of the pro- posals, with no idea whatsoever as to the price on any of them; at this stage a competitive range is established. Next,those proposals which do not merit further consider- ation are identified and discarded with a justification as to the reasons for their rejection. This process is usually conducted by staff but with the cognizance of the Director. Following staff review and evaluation, the second stage usually uses outside field reviewers who establish the competitive range of proposals which may not be per- fect but in which there is enough merit to consider negotiation. These reviewers then quantify the resultant scores to the greatest extent possible. For each proposal they develop a schedule of negotiable items indicating those technical issues which have to be addressed in any revision of the proposal in order to raise it to what they consider the acceptable level. At that time, then, the bids are open, the cost proposals made, and, here again, the outside reviewers identify those which are l3 completely unreasonable and those which are in range. They then associate the price bid with the technical pro- posal to make sure that they are appropriately supportive of each other and identify places where the agency thinks there are again negotiation issues. The contract officer (who may act by himself at this point, though frequently, in technical matters, a program person may be involved) will present this schedule of negotiable items to each bidder who is within the com- petitive range. The bidders then usually have about two weeks to think through their approach and costs and give the agency what is called the best and final offer. The staff again does a technical review and scales the revised work plans according to their technical merits, again with- out any examination of price. If the staff then finds two or three which are good enough, the contract goes to the low bidder. Grant Announcement In the third method by which federal agencies can support research the agency has some kind of a broad area of interest or concern but wants to encourage the best ideas in the field; an example might be a program of re- search on reading comprehension. Rather than issuing a Request for Proposals,then, a Grant Announcement is issued. If there is any way in which the competition is going to be limited, for instance to non-profit corporations, then l4 formal rule making is required as well as the Grant An- nouncement itself. The Grant Announcement simply specifies a broad area of interest by defining the limits of that area, presents the criteria and review procedures, and gives the date for proposal submission. Then, usually a fairly pure peer review process prevails here and the end result of their approval is not a contract but a grant. This method has a tendency to encourage not only initia- tive at the application end by the individual scholar, but it also provides him with a good deal more latitude in terms of the way he operates (a quality unique to the rendering of a grant rather than a contract). UnSolicited Proposal The fourth way in which procurements are engaged by federal agencies is via the Unsolicited Proposal. In fact, it is a matter of HEW policy that agencies are always open to Unsolicited Proposals which means that anybody with an idea can send that idea in any form with a price attached to it and that proposal gets attention. What happens on a formal basis is that about every three months an agency has a small panel of internal people examine the group of unsolicited proposals that has been submitted to make sure thattfluaproposals are indeed ger- maine to what the specific agency is about and that there is sufficient information on the topics to do a fair and honest review job. A selected proposal goes to a staff 15 office where it undergoes staff review. If the proposal has a good deal of promise or if the staff feels it does not have the competence to judge it, external reviewers are used as well, not as a panel, but as individual re- viewers, and they provide individual Opinions, without any consensus, about the proposal. The staff then pro— cesses the range of comments and comes to a determination to fund or not fund the proposal. Significant to note here is that an Unsolicited Proposal is a reward for originality. In essence, then, what an agency is doing is reinforcing creativity with an unsolicited award. Sole Source Contract There is one last method of research solicitation which is a kind of a variation of the first two directive types discussed above, and that is the Sole Source Con- tract, which is usually avoided unless it is the only alternative. In the Federal Procurement Regulations there are some thirteen to fourteen conditions which govern when competitiveness should be relinquished for this type of procurement. These conditions include national ex- igencies, for example. ~An agency takes the position that anytime there is going to be a sole source procurement, the staff must first of all do an honest and energetic job of seeking sources of performance. If it finds that there is only one performer who can do the job, it then, before any involvement with that performer at all, must attain approval from the highest agency levels first, 16 that the work to be performed is important enough to take this step; second, that this performer has the qualifica- tions that are needed to do the work; and, third, that there is documentation that only this performer has the competency to do the work. Only when these three condi- tions are met and acknowledged by the agency's Deputy Director or Director will the agency use the Sole Source Contract . SIGNIFICANCE OF THE STUDY The underlying assumption of the peer review system (a system which is an integral part of the five types of agency-to-individual research support methods just out- lined) is that proposals should be judged by people who have the substantive expertise which is needed for such judgment, peOple who are the peers of the eminent scholars in the field. Indeed, as early as 1963, Philip H. Abel- son had this to say: "The influence of the peer review system or any other method of support potentially affects all researchers in the United States."14 Yet in 1975 Rep. R. E. Bauman of Maryland denounced the system in bitter terms on the floor of the House by saying: I suggest that there is a need for revision of the basic system by which . . . research grants are made. They are handed out in an unregulated and secretive manner known as the "peer review system." This system allows cronies to get together and finance their pet projects, where grant application 14"Proliferation of Bureaucracy)" Science 163 (Feb- ruary 1969). 17 writing has become an art and where many people are not devoting themselves to basic research needs but rather to feathering their own nests. Such criticism led this researcher to an investiga- tion of the peer review system in an effort to determine both its strengths and its weaknesses--and the possibil- ities for change. This investigation began with a review of current articles on the subject from research journals and newspapers, the majority of which were mostly critical. Also observed was the fact that these criticisms were not isolated to educators or research specialists at univer- sities but included business and Congress itself. Occa- sionally embodied in the criticism was reference to past governmental techniques of funding research and suggestions that they were more equitable and produced better results than current modes. The reliability of such criticism is not to be accepted blindly. Rather this researcher chose to examine the peer review system in federal funding of research himself. This examination, based on information from the literature and experts' observations of the process at work, can then lead to a greater understanding of peer review and thus put the system into proper perspective for evaluation. 15U.S. Congress, Rep. R. E. Baumann denounced the peer review system on the floor of the House. Congres- sional Record, 24 June 1975, p. H6015. 18 STUDY DESIGN AND RESEARCH QUESTIONS The model for this study was the National Science Foundation (NSF) because either individual or peer panel review is utilized by the Foundation in its evaluation of the vast bulk of formal requests for research funds. Moreover, the process employed by NSF was made particularly visible in a series of oversight hearings held in the summer of 1975 by the Congressional Subcommittee on Science, Re- search, and Technology. In addition to testimony from these hearings and recommendations made by the Subcommittee based on this testimony, this study utilized the historical method to determine past trends in policies and practices relative to the awarding of research grants. Moreover, this re- searcher employed the interview technique in gathering relevant data on a personal basis from many sources in government and educational institutions and within the Foundation itself. The results of a poll conducted by Industrial Research on peer review were also used as sup- portive data for the conclusions of this study. Finally, telephone contacts were made with various individuals and universities when expediency demanded information from them relative to the focus of this study. The study was conducted in order to answer the fol- lowing critical questions regarding the peer review system employed in the agency-to-individual approach to federal funding of scientific research: 19 (1) Does the agency-to-individual approach, which employs peer review, provide for the advance of science in the most effective way possible? (2) In terms of the samples involved, is the agency- to-individual approach, with its operative system of peer review, a fair and impartial approach? Is it subject to political influence, "cronyism," geographical/institutional favoritism? (3) What are the economic arguments for and against the peer review system? (4) Is there opinion to substantiate the charge that the agency-to-individual approach promotes "grantsman- ship" at the expense of scientific excellence? What kinds of grants are being made? What types of research are being supported? Is the system too secretive? LIMITATIONS OF THE STUDY The first major limitation of the study resulted from the investigator's decision to restrict himself to the examination of the second general category of federal funding for research, the agency-to-individual approach, ‘with its peer review system, and the operation of that approach within NSF. This limitation, of course, was introduced purposely to make the study manageable. The decision proceeded from the writer's early realization that to study the problem more comprehensively would be an overwhelming task. 20 A second limitation flowed from the first. Because of ease of access, NSF was chosen for specific study in- stead of all federal agencies which have monies for the support of individual researchers. Thus, this is an ex- ploratory case study since at this point a scientific sample of all agencies is not available. The researcher of course realizes that generalizations from this limited study cannot be applied to all of the agencies supporting research in the federal government. A third limitation resulted from the inherent biases which the subjects interviewed possess and which are pre- sent, to some degree, in the documents investigated. A fourth limitation grew out of a certain amount of reticence on the part of those participating in the en— tire investigation. .Because of this situation, certain key actors (although surprisingly only a very small number) would not submit to personal interviews; nor was the writer able to gain free access to all of the documentary evidence extant. This limitation hopefully may be removed for future research once some of this information is revealed and discussed. OUTLINE OF THE STUDY Following this introductory chapter, Chapter II of this study will review the literature relevant to federal funding of research, in general, and to the growth and development of the agency-to-individual approach to such 21 funding which utilizes the peer review system, in par- ticular. This review will encompass an historical over- view of the movement from the agency-to-institution approach to the agency-to-individual approach. The second chapter will also describe the National Science Foundation (the model for this study) and define its peer review process. Chapter III will present the design of the study and describe the methodology used to attain data relevant to the research questions. Chapter IV will reveal the findings of this researcher with regards to the research questions posed.concerningthe effectiveness and impar- tiality of the peer review system. And Chapter V will offer suggestions for change in the system, if research indicates such are warranted, and recommendations for areas of future research on this topic. CHAPTER II AN HISTORICAL REVIEW OF FEDERAL RESEARCH FUNDING: FROM LAND GRANT COLLEGES TO THE NATIONAL SCIENCE FOUNDATION INTRODUCTION Any review of literature in the area of federal funding of research reveals a pattern in the development of a government philosophy regarding such funding. That pattern is a clear reflection of this nation's growing technology, a growth which demanded from science and government that research keep pace with the changing needs of the American populace. Thus, one finds that the early land grant colleges and agricultural experiment stations were funded on an agency-to-institution basis because the government believed it could best encourage and monitor the agricultural research needed by a farming oriented nation through a partnership with the nation's centers of learning. That belief began to alter with the shift in our country's needs brought on by a wartime econ- omy, a shift which signaled a greater emphasis on industry precipitated by the demands of the war machine and Amer- ica's growing defense technology. Now the federal govern- ment saw its role in research funding going beyond its 22 23 partnership with universities; it needed, instead, to deal directly with those researchers who could develop "hardware" to meet its particular defense specifications. At the close of the War President Franklin D. Roose- velt was anxious to continue the scientific advances be- gun in the name of defense, to perpetuate the federal government's actual encouragement, through funding, of research efforts by universities, laboratories,and indiv- idual scientists. As a result of his desire, Vannevar Bush, Director of the wartime Office of Scientific Re- search and Development, was commissioned to report on how Roosevelt's dreams could be made reality. Bush's history- making document, Science - The Endless Frontier, laid the groundwork for a whole new philosophy for government funding of basic research. And the National Science Foundation, established in 1950 as a result of Bush's recommendations, made that philosophy operative as its peer review system began the task of stimulating individ- ual scientists to contribute their knowledge to a tech- nologically changing America. THE AGRICULTURAL EXPERIMENT STATION: THE DEVELOPMENT OF THE AGENCY- TO-INSTITUTION APPROACH The best example of institutional grant development and procedure is contained in Knoblauch, Law, and Meyer,16 whose purpose in their work is to document and evaluate 16p. 3. 24 administrative policy developments in the operations of the State Agricultural experiment stations. It is a history of cooperative research procedure and policy de- velopment under the dually administered system of publicly supported agricultural research of the State Experiment Stations and the U.S. Department of Agriculture. Although intended primarily to aid agriculture, this development combining laboratory research and classroom training be- came the model now used in most areas of preparation for a career in science and technology; it also set in place a unique pattern that many still regard as the model for effective State-Federal cooperation in scientific endeavor. The German Experiment Station Movement A pioneer in this endeavor was Samuel W. Johnson who spent a lifetime searching for the most effective way to institutionalize research in agriculture. He per- sonally observed the German experiment station movement in its infancy and became America's first advocate for a similar movement in the United States, noting that: There have been lately established in Germany, especially in Saxony, a number of so-called experiment stations, or experimental farms, with laboratories in connection, for the ex- clusive object of promoting scientific agri- culture. These are intended to make science practical, and practice scientific, and no agency can be desired better adapted for these important purposes.17 17S. W. Johnson, On the Relations that Exist Between Science and Agriculture (New York: State Agricultural Society Trans., I855), 15: 73-80. 25 Johnson also proposed that American farmers ignore their slow-moving state legislatures. He urged each county to recruit a large membership with a minimum of 2,000 dues-payers, incorporate itself as a "Scientific Institution," adopt a constitution, and elect officers, a board of directors, and "one or more professors of agricultural science"; he also urged the construction of a building containing lecture rooms, a library, and, in particular, an analytical chemistry laboratory.18 In 1854 Johnson travelled to Moeckern in Leipzig, Germany to visit a new institution which its founders called an Agricultural Experiment Station. He later translated into English the statutory provision authoriz- ing the new institution to conduct scientific research in agriculture. The station concentrated in one location all necessary facilities for the conduct of research in the laboratory and in the field. Thus, the station was directed to concentrate on research as a profession. The Hatch Experiment Act In the United States the movement to institution- alize experimentation on a formal and practical basis at federal expense gathered momentum from 1855-1861. Johnson and his advocates made their influence felt in the dis- cussion and policy formulation that took place between 1862 when the Land Grant Act was passed and 1887 when the 131bid.. 26 Hatch Experiment Station Act became law. The Land Grant Act of 1862 established the United States Department of Agriculture and helped give the De- partment a scientific reputation. However, the Act did not require the Commissioner to be a scientist, nor did it designate scientific research as his sole duty.19 In fact, not until the late 18803 was there crystalliza- tion of any clear departmental research policy based on long continued experiments.20 The basic legislation authorizing the establishment of federal grant payments to agricultural experiment sta- tions came with the First Morrill Act of 1862, "an Act donating public lands to the several States and Territor- ies which may provide colleges for the benefit of agricul- ture and the mechanic arts." This Act provided the first federal, legal authority under which the cooperative features of today's nationwide agricultural research sys- tem were to develop. Slow progress in developing the Agricultural Depart- ment as a national experiment station served as an incen- tive in the states to go ahead with state stations. As A.C. True has noted: . . . up to 1872 only 6 colleges in the country taught either chemistry or physics by the laboratory 19Knoblauch, Law, and Meyer, p. 25. 20E. D. Ross, The U.S. Department of Agriculture During the Commissionership (Washington, D.C.: Agricultur- al History, 1946), pp. 129-143. 27 method. The early use of the method, thus, was scattered and purely experimental. It remained for the land-grant colleges to assist in a development basis of necessity rather than design.21 A station founding decade followed from 1877-1887. The winning of federal support for the Land-Grant college system came when the Hatch Act became law in 1887. This Act produced the land-grant college agriculture ex- periment stations and, in the process, set a precedent for federal-state cooperation in agricultural research.22 The preamble to the Hatch Act marked out the widely spaced limits of authorized activity which ranged from: acquiring and diffusing . . . practical information to scientific investigation and experiment respec- ting the principles and applications of agricultural science.23 Furthermore, the Act had broad latitude which became the keystone of its popularity as well as a mark of its fu- ture weakness. As Knoblauch has noted: Could not the trustees, often more zealous for teach- ing than for research, apply the funds to college instruction at the expense of the station interests envisioned by career directors and scientists?24 Also, the generous inclusion of the word "diffusion" in the Hatch Act placed the interest of research at a serious disadvantage. The states most in need of a concentration on research lacked either the resources to accommodate the 21Knoblauch, Law, and Meyer, p. 3. 221bid., p. 52. 23Ibid., p. 90. 24Ibid. 28 extension movement, or the willingness to preserve the Hatch income as a trust fund for research.25 New Developments in the Twentieth Century Therefore, led by Dr. A. C. True, Director of the Office of Experiment Stations, a movement for a new fed- eral endowment authorizing and directing only original research emerged as the necessary solution to the problem. In a scholarly statement Needs of the Station: U.S. Depart- ment Agricultural Office Stations Annual Report 1903, True declared that the stations could not produce, without an immediate and substantial increase in revenue, the scien- tific discoveries basic to a steadily improving technology in agriculture. He noted: Under present conditions it is useless to expect that . . . our stations will discover many of the principles on which permanent improvement of our agriculture must rest, but unless the way is open for them to seriously attack these fundamental problems their future work will necessarily be comparatively fragmentary and inconclusive. They may, as in the past, obtain many results which can be usefully applied in practice but they will not be able to furnish solid foundations for the en- largement of our agricultural industries.26 What was being requested,then, was a second federal donation based on the precedent of the Hatch Act, a request which was fulfilled with the passage of the Adams Act in 1906. This Act firmly established the principle in Amer- ican governmental policy that federal aid shall join with 25Ibid., p. 95. 26Ibid., p. 100. 29 state aid for the purpose of subsidizing scientific re- search in the state stations.27 The Purnel Act of 1925 and the Bankhead-Jones Act of 1935 added to an even more complete endowment of the 28 Agricultural Experiment Stations. And legislation in 1946 and 1955 further extended research in the land grant colleges.29 This development of agency-to-institution funding thus set a pattern for federal and state and university relations and became the model for research grants referred to as "agency—to—institution" grants. THE NEED FOR WARTIME "HARDWARE": THE DEVELOPMENT OF THE AGENCY- TO-INDIVIDUAL APPROACH The relationship between government and the scien- tific community was changing as a result of new demands foisted upon research by the nation's defense needs in World War II. Over 100 years earlier, the National Acad- emy of Sciences had been created to serve as a link be- tween science and the special needs of the government during the Civil War. The Academy's first report dealt with a naval subject: the intriguing and practical prob- lems of calibrating compasses aboard ships equipped with iron smokestacks. Since that time, however, the technical problems had become more complicated and so had the 271bid., p. 107. 281bid., p. 108. 291bid., p. 119. 30 . . 30 relations between government and sc1ence. World War I led to the formation of the National Research Council to enable the Academy to make more effectively the contributions which a considerably matured structure of science could offer to a new set of wartime needs. And World War II generated additional organiza- tions, such as the National Defense Research Committee and.the Office of Scientific Research and Development,as part of the Executive Branch of the government itself. In the course of those wartime years the basic sciences and the universities were mobilized in strikingly effec- tive ways to support wartime objectives.31 And though the methods by which these scientific resources were put to 'work were informal, by the end of World War II there were quite a number of responsible people who understood the essential national needs of a continuing connection be- tween the government and basic research, a connection to be forged by expanding government support of scientific research. The agricultural research needs attended to by the experiment stations established under the Hatch Act (and supported by government funds paid to the institutions connected to these stations) were now joined by the coun- try's growing need for a different kind of science and 3°Ibid., p. 119. 31F. Joachim Weyl, ed., Research in the Service of National Purpose (Washington, D.C.: U.S. Government Printing Office, 1966), p. 126. 31 technology to serve industry, medicine,and education. The Forerunners The Bush Report In March 17, 1944 President Franklin D. Roosevelt sent a letter to Dr. Vannevar Bush, Director of the Office of Scientific Research and Development, requesting that he submit to the President his recommendations with regards to the following areas: 1. Diffusion of knowledge, consistent with military security, to world nations regarding American contribu- tions to science made during the war effort. 2. Organization of a program to further research into disease. 3. Government aid to research activities conducted by public and private organizations. 4. Proposal for a program for discovering and developing scientific talent in American youth.32 Responding to the President's demand that the in- formation, techniques, and research experience developed by the Office of Scientific Research and Development and by scientists in universities and industry be used in the days of peace for "improvement of the national health, creation of new enterprises bringing new jobs, and the betterment of the national standard of living,33 Dr. Bush 32Vannevar Bush, Science - The Endless Frontier (Wash- ington, D.C.: National Science Foundation, 1945), pp. 3-4. 33 Ibid., p. 19. 32 charged that, indeed, the federal government had no real national policy for science, nor a government body with responsibility for formulating or executing a national science policy.34 And to correct these deficiencies, Bush presented a blueprint for implementation of such a policy and such a body, Science - The Endless Frontier. In that report Dr. Bush viewed the public and private colleges and the endowed research institutes as the cen- ters of basic research that must furnish both the new scientific knowledge and the trained research worker. And Bush saw five fundamentals that must underline govern- ment support with respect to the organization of the admin- istration of scientific activities within the government: (1) Whatever the extent of support may be, there must be stability of funds over a period of years so that long range programs may be undertaken. (2) The agency to administer such funds should be composed of citizens selected only on the basis of their interest in and capacity to promote the working of the agency. They would be persons of broad interest in and understanding of the peculi- arities of scientific research and education. (3) The agency should promote research through contracts or grants to organizations outside the Federal Government. It should not operate any laboratories of its own. (4) Support of basic research in the public and private colleges, universities, and research institutes must leave the internal control of policy, personnel and the method and scope of the research to the institutions themselves. (5) While assuring complete independence and freedom for the nature, scope, and methodology of research carried on in the institutions receiving public funds, and while retaining discretion in the allocation of funds among such institutions, the 34Ibid., p. 34. 33 Foundation proposed herein must be responsible to the President and Congress. Only through such responsibility can we maintain the proper relation- ship between science and other aspects of a demo- cratic system. The usual controls of audits, reports, budgeting, and the like, should, of course, apply to the administrative and fiscal operations of the Foundation, subject, however, to such adjust— ments in procedure as are necessary to meet the special requirements of research.3 The final chapter of the report recommended the establishment of a National Research Foundation, con- ceived as the principal means for carrying out the other major recommendations contained in_the report. The Bush Report thus provided the blueprint for the future. The fundamental premise was that only the system- atic pursuit of scientific research on a broad front, with the financial support of the federal government, would ensure the choice our nation would require to pursue an effective course of development along the lines which history had prescribed. And although the agency-to- institution approach still held sway, the groundwork had been laid in this expanding funding philosophy for a change in that approach to meet new needs. The Office of Naval Research The Bush Report persuasively induced the Congress to make unprecedented appropriations for the support of research, and the Office of Naval Research resulted from the first successful Congressional action implementing 35Ibid., p. 33. 34 the Bush Report. The Office of Naval Research was organ- ized to act on the principle that the resources of scien- tific research and invention are the mainspring for sustaining technological innovation. This concept is written in Public Law 588 of the 79th Congress, 1946, which provides that the Office of Naval Research will: plan, foster, and encourage scientific research in recognition of its paramount importance as re- lated to the maintenance of future naval power and the preservation of national security.36 Indeed, the Office of Naval Research, as an outgrowth of Bush's emphases in his historic report, was to be the very model for federal support of research. According to one distinguished scientist: "The creation of ONR, the selection of its goals and ideals, and the skill and efficiency with which these were carried out in the intervening years have served as a beacon to show the Government and the scientists of the United States how to cooperate and keep world leadership in scientific research and development here in this countryT37 For the Navy ONR was responsible for the identifica- tion and endorsement of the significance of basic research as an important part to the solution of the fast unfolding developmental problems associated with modern sea-power. In drawing from the lessons learned in naval warfare and in recruiting an able group of civilian scientists and engineers to work with young naval officers to extend these lessons into further research, the ONR has furnished 36Weyl, p. 55. 37Ibid., pp. v, vi. 35 within the Navy a focal point for leadership and coordin- ation of the fundamental research and development essential to unified progress.38 Alan P. Waterman perhaps best summarizes the role the ONR played in the development of changing federal government philosophy regarding funding of research, a change which would lead to the agency-to- individual approach utilized by the National Science Foundation: . . . in mutual consultation with academic and in- dustrial scientists and administrators the ONR evolved policies and procedures . . . which pion- eered the way for increasing participation of the Federal Government in a comprehensive program of scientific research throughout the country.39 THE NATIONAL SCIENCE FOUNDATION The scope of the program to which Waterman refers was broadened in 1950 with the establishment by an act of Congress of the National Science Foundation. Recall that in his report to President Roosevelt, Dr. Vannevar Bush had suggested just such an organization to insure the continuance of the scientific advances of wartime. NSF Purposes and Organizational Policy The purposes of the National Science Foundation (NSF) are to: increase the nation's base of scientific knowledge and strengthen its ability to conduct scientific research; 38Dr. Alan P. Waterman, "Pioneering in Federal Support of Basic Research," in Weyl, p. 8. 39Ibid., p. 9. 36 encourage research in areas that can lead to improvements in economic growth, energy supply and use, productivity, and environmental quality; promote international coopera- tion through science; and develop and help implement science education programs that can better prepare the 40 nation for meeting the challenges of the decades ahead. Reorganization Plan No. l of 1973, effective July 1, 1973, transferred to the Director of NSF the functions of the Office of Science and Technology which was abolished by the reorganization plan. The Foundation consists of the National Science Board of twenty-four members, and a Director, all appointed by the President with the advice and consent of the Senate. The National Science Founda- tion Act also provides for a Deputy Director and four Assistant Directors, appointed by the President and sub- ject to Senate confirmation. The Director is the chief executive officer of the Foundation and serves ex officio as a member of the Board and as chairman of its Executive Committee. The National Science Foundation initiates and sup- ports fundamental and applied research in all the scien- tific disciplines. This support is made through grants, contracts, and cooperative agreements awarded to universi- ties, nonprofit, and other research organizations. Most of this research is directed to unresolved scientific ques- tions concerning fundamental life processes, natural laws 400.8. Government Manual, p. 544. 37 and phenomena, fundamental processes influencing man's environment and the forces impacting on man as a member of society as well as on the behavior of his society. Additional research is focused on societal problems of national importance and is directed toward contributing to the knowledge required for their practical solution. The Foundation also supports major national and inter- national science programs including the U.S. Antarctic Research Program, International Decade of Ocean Explora- tion, Ocean Sediment Coring Program, and others. And among the centers supported by the Foundation are: the Kitt Peak National Observatory, the Cerro Tololo Inter— American Observatory, the National Radio Astronomy Obser- vatory, the National Astronomy and Ionosphere Center, and the National Center for Atmospheric Research.41 Procedures for Grant Allocation The National Science Foundation employs either in- dividual or peer panel review to evaluate the majority of formal requests for funds.42 A more detailed summary of the Foundation's procedures for awarding individual grants is described below. This summary encompasses only the major aspects of NSF funding procedures, aspects 41Bush, p. xxi. 42These methods are described in "Peer Review and Proposal Evaluation," an NSF study dated June 1975 which is included in the peer review hearings record. 38 which will serve later in this paper as a basis for in- terpretation and analysis of the workability of the peer review system. The Foundation's methods for selecting the recip- ient of an award begin when it is decided what form an award will take. For the most part, the award is a grant to or contract with an institution (which assumes financial responsibility) for its use in support of a project directed by a principal investigator (who assumes responsibility for the work). Two other types of awards include one for education (e.g., NSF Graduate Fellowships), which involves an institution and a person, but no project, and one made to an institution to improve itself. The Foundation does not normally make: 1) awards to individuals in the absence of an institution to sponsor the individual; 2) research grants to institutions for a particular scientist, not specifying that work in a particular field be done.43 In deciding award recipients, NSF acts on applica- tions which may have been unsolicited or which may have been solicited and received by the Foundation in response to its request for proposals for research projects in a specified broad field or to its request for proposals to carry out a project already defined. 43NSF Peer Review Vol. I, p. 15. 39 National Science Foundation staff play a large role in most review processes in the stage when decisions are made on an application for funds. In fact, when small amounts are involved in an individual award (the case for foreign travel awards) staff judgment alone may de- termine the decision. Basic scientific research receives over half the funds of NSF (more than $300 million annually); most basic research awards fall into the $10,000 to $100,000 range.44 Review procedures for these awards vary; staff judg- ment is used along with individual peer review (both these methods play a role in evaluating all basic research applications), while peer panels are utilized in fewer than half of NSF basic research programs. The following flow chart and list of processes in Figure 1 illustrate the procedures utilized in the evalua- tion of an application by the former Directorate for Re- search, procedures which are still basically similar though NSF has reorganized. For applied research NSF annual expenditures in recent years have reached $70 million with most applied research grants falling within the range of $40,000 to $400,000 (an average of two to three times the size of basic research grants).45 In awarding these funds, the 44Ibid. 451bid.,p. 17. National Science I , I I Board H1“ ' I I ‘ I | Assistant DirectorL I I for Research l$——>1 I I . l I I I Division . 12 ; I Director I T «E I S 11 I . I I I Section I I Head I I r I 5 10 I I .. I, L | I +7¢+ rogram + __7 Other NSF Officer ‘3 Other I Components NI . . Agencies I J: I I 88 7e 9b 8b — I I Central Grants 4_ Processing' ,3 2 98 Office l 7" | I ——————————— Pd:dp——-I————— ———————— l V 15 Mail +——' Panel Reviewers Reviewers ‘I I' II It Principal Investigator Figure 1. Flow Chart of the Movement of a Research Proposal Through the Foundation* * Source: National Science Foundation Peer Review, Vol. I., A Report offthe Subcommittee on Science and Tech- nologyr U.S. House of Representatives, Ninety-Fourth Con- gress, Second Session (Washington, D.C.: U.S. Government Printing Office, 1976), p. 16. 41 Notes: 1. Principal Investigator (P.I.) conceives research plan. ‘ 2. P.I. may hold preliminary discussions with Pro- gram Director (P.D.) and/or submit preliminary proposal for evaluation. 3. Final proposal is prepared by P.I., approved by Institution and sent to NSF. 4. Central Processing assigns proposal to a Division. 5. Division Director assigns proposal to a Section. 6. Section Head assigns proposal to a Program. 7a. Program Director chooses reviewers and sends out proposal for independent mail review. 7b. Program Director may choose panel members and send them copies of proposal in preparation for panel meeting. 7c. Program Director may discuss proposal with an- other agency. 7d. Program Director may discuss proposal with other components of NSF. 7e. Program Director may make site visit (or site visiting team may be appointed and report back to P.D.). 8a. Independent mail reviewers evaluate proposal and return signed, written reviews. 8b. Panel members discuss proposal and indicate rating. 9a. Program Director may decline proposal but suggest some major modification that would make it more acceptable or may suggest that a new proposal may be written. 9b. Program Director may decide to recommend funding and negotiates revised budget with P.I. 10. P.D. recommends funding of revised amount. 11. -Section Head reviews recommendation, approves and transmits, or rejects. 42 12. Division Director reviews recommendation, approves and transmits, or rejects. 13. Assistant Director for Research may review recommendation. 14. If grant is large enough, National Science Board must review. 15. Grant is made to Institution, which disburses funds to P.I.‘s project. Foundation selects projects that will bring benefits soon after completion. Thus, the Research Applications Direc- torate decides onla specific benefit and then solicits projects designed to achieve the known goal (a program solicitation); or the Foundation may issue a "request for proposal" wherein the project needed to attain the out- lined benefit is described beforehand. Then, too, NSF must consider unsolicited preposals for applied research. To adequately screen these three types of proposals, then (program solicitation, request for proposal, and unsoli- cited proposal--allow .m.D ".U.o .coumcflnmmzv .mbma Mash .mo>wumucmmou loom mo Omsom .m.D .hmoHocsooe pom noumomom .Oocommm do mouumfieopmsm may onomoo mode rumom .mmcwnmom unoflmuo>o Hmeoomm .Sme>om Home coflumocoom OOCOflOm Hocowumz “mousom awn—253:... 2: 22:29.5.— sswm1 . ss~s_ . sss~_., Asses seem sass .s _ . _ a . .1. q .honflusz. . u . .x. . .25 d. 8 3s . .5... ._.... u. . . . , . ..ssss_ an ..H mm. . . . . H s . . :82 m d. , a 8 MW . ..ssssm Mm .85 m mm . . ”:5... ea . 1mm so... . _ :5: es... .- 1 8:8 SE 2: 4 E: as . . . ssssm 83 alone, and any state that falls below receives too few dollars. As one can see, most of the states cluster about the line, indicating that the distribution of NSF funds is closely correlated with the population of the state. However, there are three special cases--Massachusetts, New York,and California--all falling well above the line. These three states receive proportionately more funds than other states when judged by population.116 Atkinson concluded that if NSF's Objective was sim- ply to assign its research funds to each state by popula- tion, the task would be much easier. NSF's objective, however, is to seek out and support the most innovative and promising research with the only constraint being to avoid ppdpg concentration of funds. In this regard, it is obvious that some states have a greater concentration Of scientific talent than others.117 Figure 3 presents the distribution of research funds as a function of the number of members of the National Academy of Sciences and the National Academy of Engineer- ing on a state-by-state basis. In this graph Massachu- setts has 236 members of the Academies and, as in the previous graph, receives approximately $30 million in research.funds. The straight line on the graph is com- parable to the one on the preceding graph. If research 116Ibid. 1171bid. 84 mumpsmz maz pap maz umcammm pmuuoam mumum some now mHmHHoo uuommsm pomfloum nowmomom owmeuGOAOm .m ousmHm .m ousmfim mom "condom 5.352 mg azc mcz can an . emu :5 E o. 88— 8 Q E “d m .188 m m w as. 258 m . m s m . 0 was” 2,: 8:3 8 as: so: .. . $25 or: d ”:5 mac 0 as. . . . ._.. -_. .. 88m 85 dollars were assigned to each state proportionate to its number of members in the Academies, then all points would fall on the straight line. Examining this graph, note that most states are fairly close to the line except for Massachusetts and California, which are dramatically below. By this measure, these are states with a great deal of scientific talent which receive far fewer dollars than would be warranted on the basis of Academy members.118 Atkinson said: "Obviously, we do not want to rely on Academy membership as our only measure of scientific talent."119 Figure 4 shows the distribution of research dollars as a function Of the number of scientific publications produced in each state. Publications were calculated by counting the number Of articles appearing in leading scientific journals over a period of several years; articles were assigned to states in terms of the residence of the authors. In examining Fugure 4, it is evident that New York and California are above the line, indicating that they receive more dollars than expected if funds were distrubuted proportionate to journal publications. Mass- achusetts, however, falls close to the line as do the other states. This figure suggests that the distribution of NSF funds correlates rather closely with the source of scientific llerid. llglbid. 86 mHMCHDOH Oemepcowow mceommq cw cheumoeaosm uncemmm omuuoam oumum sumo How mnmaaoo uuommsm uOOnOHm noncommm Oemeucowom .e owomwm .m musmem mom "mouoom 3.22932: .8 33:5 33 82 I cm: on» a: c: «320 '1..."| ill' I u 88— 8ch (SGNVSDOHI NI) SUVTIOG HOUVESHU 88m. . . SE 2.: .. » =- as: so: .. :83 was; 2: a as... as o I3“. _ s88 87 publications.120 The horizontal axis on Figure 5 represents the num- ber of individuals receiving NSF fellowships in 1974 who chose to go to universities within each state. Califor- nia and Massachusetts fall far below the line, indicating that they receive less money than they should based on this measure of scientific excellence. On the other hand, New York falls above the line, which means that it gets a disproportionately large share of research funds when compared to the number Of NSF fellows choosing to attend New York'universities.121 Atkinson concluded that the picture that emerges is fairly clear. In general, the distribution of research funds is closely correlated with state characteristics, such as population, income tax revenues, doctoral scien- tists in the labor force, and so forth. There are three states--California, Massachusetts, and New York--which receive more NSF dollars than they should based on these criteria. However, when various measures of scientific excellence are examined, it is clear that these states are receiving fewer dollars than they qualify for based on their scientific merit; Obviously, then, NSF's dis- tribution Of funds turns out to be something Of a com- promise between a state's population and its collection 12°Ibid. 121Ibid. 88 magmadaomm danmeoaamm mmz nmaammm pmuuoam oumum zoom wow mnmaaoo uwommsm uOOmoum nonmomom Oemeucmwom .m whomem .m whomem mom "condom ”Ema—um:— mEmzojum ...mz can . as" sNN . as" as .. a . _ . . . c ‘ _. ..m was. . . . I ”was; ea 4 88— m . 35m mzc o H w H sessN MW m w ..m. . as” m 6.2: m S m mm 253 asses o :30 89 Of scientific talent. NSF has no precise formula for making this compromise; rather the various forces Oper- ating on NSF have defined its policy. Whether this policy is correct may well be judged differently by different individuals.122 In Figure 6 the number Of NSF reviews are plotted on the vertical axis and state population is plotted on the horizontal axis. The straight line on the graph in- dicates how reviews should be distributed if they were solicited from a state proportionate to that state's pop- ulation. Except for a few Outliers, most Of the states cluster closely about the line, indicating that the dis- tribution of NSF reviews is closely correlated with state population. However, the District Of Columbia and Mary- land provide considerably more reviews than one would expect given their populations. The same Observation applies to California and Massachusetts, whereas Ohio is disproportionately low.123 Atkinson then said that the question is whether a few select states tend to monopolize the review pro- cess. In order to answer this question, it will be useful to consider the source of NSF reviews in relation to the previously mentioned measures of scientific excellence.124 122Ibid. 1231bid. 124Ibid. 90 coflumasmom unnamed wouuon oumum comm Eoum m3ow>mm mmz .m owsmflm .N whomwm mom "mousom . _ 185.28.: 2:22.512 . 8a: 582...! ...... :82--I.---éSemi 83 82 . . a _ . .. . oo o o to ««4 o O O . O. 2.8 51 o. . 0. an: - 88 3: O o . . ; .2. - 83 C .. as: .. . . um..- 88 .2 . . as: ea: .. 1 e82 . . . _ as; e: 4 =5 :8 o .5 ‘ w _. . . . .. as: mamas—30 uaawnu' 91 Figure 7 plots NSF reviews as a function of the number of National Academy members in the state. The straight line has the same meaning as in previous graphs. Once again, most states cluster about the straight line, except that the District Of Columbia falls well above, whereas California and Massachusetts are dramatically low. By this measure, the District is again overrepre- sented in the review process, but the Opposite is the case for California and Massachusetts.125 Figure 8 presents a similar plot for the distri- bution of NSF reviews as a function of the scientific publications produced in each state; and Figure 9 is a plot for NSF fellowship recipients. These last four figures indicate that the geogra- phic distribution Of NSF reviewers closely approximates both population and various measures of scientific excel- lence for most states. Those states that fall above the line in one graph fall below the line in another, depend- ing on which measure is considered. There are only three points that are.Outliers in all four figures. The Dis- trict of Columbia on all measures provides far more re- views than are to be expected. California and Massachu- setts provide too many reviews based on population, but far too few reviews based on any of the measures of scientific excellence. The high concentration of reviews 1251bid. 92 mama—am: M42 pom mmz umcwmmm oouuoam oumum comm Eowm m3ow>om mmz .h ousmwm .m ousmam mom “mousom mama—52 22 9.2 22. cam can sMN sm_ as , a $25 2: . 221:8: .. ”1:: SE « 8cm 35» was 0 M 88 m H O 0 as: J . n 88 W. :82 45 . sssm_ 93' mHmcusOh mewucoeom mcwomoq ca mcowumoeaosm umcemmm woquHm ououm some Eowm m3me>om mmz .m whomem .m whomwm mom "condom ., 3222.52.15 gums—.2, as s: a: as s: a: s . NM £2 .. as .m. .. 88 m n m n . .. as m dd . m . Ham ea I 252 gem—wee: I . £25 2t 4 ~25 uzc o cess.. 94 mucmwmwoom mesm3oHHom mmz umcwmm< wouuon Oumum comm Eoum m3mw>om mmz .m oudmem .m whomem mom "wowsom 3558“: 2533.5... ...mz an as on s: 8 ”fig...“ .. . a... . o was; 2: 4 .28 o c . 8% E5 mzc o .550 do! g . o o ._.: om: so: M s .. 88 m . . n a“: m a .. as m a... W . I 252 S - . . s82 95 from the District reflects the fact that NSF coordinates with other federal agencies on many proposals; this is particularly the case in the RANN program but is also true in other areas such as in the national programs in oceanography and the atmospheric sciences.126 Atkinson said that, nevertheless, the data indicate that Washington-based scientists may be overrepresented in the NSF review process. Further analyses need to be undertaken to determine whether this is the state of affairs and should be corrected. The same comments do not apply to California and Massachu- setts; in these two cases, NSF procedures strike a balance between selecting reviewers on the basis of a state's population and its scientific expertise. If you favor either side of this balance you will be dissatisfied with NSF procedures, but I know of no way to resolve the issue. The choice is a matter of judgment and the question is whether reasonable people agree with NSF's trade—off between population size and scientific excellence. A very large segment of the scientific community is involved in the NSF review process. Despite this large involvement, one can ask whether certain universities and certain areas of the country are given a favored status in the competition for research funds. "The Report gives information on this question; we also have additional data that will help you to make a judgment. Unfortunately, I do not have time to review all of the data today; there are, however, a few results that I want to comment on," 126Ibid. 1271b1d. 96 said Atkinson.128 Consider, first, information comparing university departments ranked in the top 20 by the American Council on Education with departments ranked below the top 20. There is no question that an applica- tion from a scientist employed by one of the top 20 departments has a higher probability of NSF funding than an application from some other univer- sity. In the chemistry section, for example, the overall approval rate is 35%. However, the approval rate is 53% for applications from chemistry departments ranked in the top 20, and only 30% for departments below the top 20. This is not an unfair situation; departments are ranked in the top 20 because they have, on the average, a more innovative and.distin- guished faculty.12 ' The more important question to Atkinson, however, concerns possible relationships between the scientist who submits a proposal and the scientists who review that prOposal. The NSF data clearly indicate that proposals submitted by scientists from the top 20 departments have the same distribution of reviewers as proposals from other schools; the assignment of reviewers in terms of the eminence of the university with which they are affiliated is not statistically correlated with the em- inence of the school from which the proposal originates. A closely allied issue concerns the correlations between the geographic locations of reviewers and the scientist submitting the proposal. But when data is analyzed on a regional basis, again there is no' statistically 1281bid. lngbid. 97 significant relation between the location of the applicant and the locations of reviewers. Atkinson maintained that these two observations are particularly important and mean that the review process is not biased by the distinc- tion of the institution submitting the proposal or by the region of the country in which the institution is located.130 Despite Atkinson's testimony, however, Dr. Doris Wilsdorf, University of Virginia, claimed that her data showed that: "clearly, indications are that geograph- ical concentration of funding rather than dispersal of "131 She based this claim on the funding has taken place. Citation Index in which the names of all people whose work has been referenced in thousands of journals world- wide are given, together with the names of the people that have cited them and the journals in which they have been cited. Thus, one can make a count of how often the use of the research output of any one author was made.132 Dr. Wilsdorf further maintained that a person who is largely engaged in teaching rather than research is handicapped in seeking funds from NSF and that there is a direct relationship between the number of references in publicationsauuithe success of an individual or group 13°Ibid. 131NSF Peer Review, Special Hearings, p. 730. 1321bid., p. 706. 98 receiving positive or affirmative peer review comments . . 133 that lead to successful appl1cations. Industrial Research Opinion Poll As a pertinent conclusion to the findings regarding the second research question on the fairness of the peer review system, an opinion poll of scientists themselves and their perceptions of the peer review process provides some valuable insights. Such a poll was conducted in 1975 by Industrial Research Magazine. 134 According to Robert R. Jones, the editor, a major control feature in this nation's research efforts is failing to do its job, and the world is poorer for this breakdown. The control feature is peer review, a profession-guiding procedure that promises to apply the vest of disinterested wiSdom to research project funding and, ergo, to individual scientists' tenure, promotions, and salary increases. [Yet] while there are strong indications that four out of every five scientists who have knowledge of the peer review system favor it in principle only one in eleven believes that the system as it operates today is fair and reasonable. The other 91% believe that this important cog in our research machine has broken down, and they have called for modifications to make it acceptable to them.135 In order to obtain specific opinions of scientists regarding peer review, Industrial Research Magazine, via an Opinion poll, surveyed and received responses from 1,093 persons who work as scientists or engineers in the research and development industry. Almost 900 of these 133Ibid., p. 708. 1 134"Much Of U.S. Research Controlled by Faulty Mech- anism," Industrial Research (September 1975). PP. 87-91. l351bid., p. 87. 99 respondents indicated that they believe peer review fails if its goal is to assure equitable distribution of research 136 Another interesting finding relates to this funds. paper's earlier consideration of peer review's promotion of scientific advancement in terms of the system's encour-* agement of innovation: 87 percent of those responding answered "yes" to the question: "Are funds denied indi- viduals challenging reviewers?"137 It is important to note here that although the opinion poll results provide interesting correlations with some of the findings of this study, the polls themselves can be viewed as suspect because of relatively small response rates and because the criterion fixrselection of potential respondents (that the respondent received the free magazine Industrial Research) was not scientifically oriented toward obtaining a true cross-section of the scientific community. A copy of the IR questionnaire including response results can be found in Appendix B. THE ECONOMIC ASPECTS OF PEER REVIEW The question of economics surfaced again and again in the interviews conducted for this study of peer review. Although there seemed to be no denial that the agency-to- individual approach to federal funding entails greater 136Ibid. 137NSF Peer Review, Special Hearings, p. 234. 100 expense than the agency-to-institution approach, there was disagreement regarding the necessity of such expense. Detractors of the system claimed that additional costs were exhorbitant and unnecessary, while the system's backers justified the costs as necessary for the opera- tion of an open and fair system which involves the pub- lic sector in the person of the scientific expert serving as a reviewer of grant applications and the scientific researcher applying for the grant. Economic Arguments Against Perhaps one of the strongest economic arguments against the peer review system, or the agency-to-individual approach, is the high cost factor: peer review is an expensive system to Operate. Indeed, the costs of main- taining the system mount steadily with the multiplication of bureau and agency staffs and interlocking advisory boards, increasing treks to Washington by increasing num- bers of university staff members, overlapping agency and university administrative and accounting services, the development of large (but still inadequate) mailing lists in order to announce an ever increasing number of separate programs, and even the establishment of perman- ent university liaison offices within the nations' capitol-- costs absent or greatly reduced with the older agency-to institution approach.138 138Dr. Christian Arnold, Personal Interview. 101 Dr. Christian Arnold presented the hypothetical situation of the young scientist who acquires a research grant and six weeks into his research discovers that it has no chance of success. Note that there are many positive points which have motivated his application for a grant in addition to a belief in his own research aims, points such as more prestige for his university, promo- tional possibilities, monetary reward, etc. Can those motivational aspects be put aside when the research which has been funded "turns sour." Dr. Arnold said no: There is no incentive for him to stop his work. Neither the scientist nor the university would have anything to gain from stopping that two-year project or whatever short of its termination. How- ever, if this was the university's money there would be all sorts of incentives to stop the pro- ject when it was found that it was no longer going to be productive and let the scientist get on with what he thinks is more productive. He then went on to maintain that, in comparison with institutional funding, it is perhaps twenty to fifty times as expensive to operate a national project grant review system. In the first place you cut down with an institu- tional support program. You have practically no administrative overhead at the national level. Your awards to an institutional program would have to be settled on some kind of a formula basis. All you need really to do is to have some control to make certain the funds are spent for what they are de- signed tO be spent for, fund accountability. At the local level then you would necessarily have some PR too, but you wouldn't have travel expenses you have at the national level. You wouldn't need 1391bid. 102 as many copies of the proposal for one thing. It would be just less paperwork, fewer people involved, less travel involved.14 Economic Arguments In Favor Dr. Emerson Elliott from the National Institute of Education counters arguments like those of Arnold's by insisting that the cost of Operating the peer review system can be justified on the basis that involving the public sector in the process does lead to additional ex- pected expenses. Many people don't realize, he claimed "that you do have a lot of requirements in the govern- ment for notification, publication, announcements in the federal register, comment and review periods, open re- view system, open meetings, the Federal Advisory panel approvals..."141 And all these requirements cost money to fulfill. Dr. Charles Haughey agreed with Dr. Elliott. "It seemed to me that that kind of thing is the price we have to pay for opening creative research possibilities to everyone in the U.S. The notion really is that we are opening the coffers in support of any good idea coming "142 He from anywhere, the university or elsewhere. agreed that, under contracted research, compliance re- porting could be required and termination could be 140Ibid. 141Personal Interview, Washington: D.C., January 1976. 142Personal Interview. 103 arranged for given certain contingencies met or not met; thus, the system could be managed more efficiently. How- ever, such an approach would limit the spontaneity of the research sponsored through a grant-in-aid. Furthermore, he insisted, "we are trying to buy the interest of that total person [applying for the grant] so that his teaching and his service as well as his scholarship are to some 143 The university extent affected by that grant-in—aid." contract approach, though perhaps more economically effi- cient, does not permit the same type of access to the in- dividual scientist applying for the research grant. Dr. Heath supported this notion when he claimed that if a person was a true researcher and if he "survived" the panel review process and won a grant, then he would have the expertise to be flexible enough in his research approach to avoid the type of pitfall to which Arnold refers when he charges that a scientist on a grant could deliberately continue a dead-end project for the sake of maintaining funding and obligations.144 GRANTSMANSHIP AND THE ISSUE OF OPENNESS During the Congressional Oversight Hearings on the National Science Foundation's peer review system two of the most significant issues to be considered were those of grantsmanship (a funding award made on the basis of 143Ibid. 144Personal Interview. 104 proposal writing rather than proposal content) and the actual openness of the system's Operation. Congressmen wanted to know what kinds of grants were being made by NSF, what research emphases were being supported. And how were these emphases affected by the system's insis- tence on the confidentiality of peer review results and on the necessity for the proposer to be able to "Write well"? Grantsmanship Rep. Robert Bauman expressed a strong interest in the oversight committee exercising its obligation to look closely at NSF activities, especially at the research projects it was supporting. He then went on to cite a case which indicated a reason for concern. This information was gleaned from the daily NSF notices sent to our congressional Offices: $9,600 in March of this year, for a biography of Isaac Newton, given to Indiana University. Would you believe there are 145 biographies of Isaac Newton in the Library of Congress at this very moment? Now, I am sure Isaac Newton was a very great man. I am sure he was greater than the gentleman from Maryland; perhaps he even approached the greatness of the distinguished chairman of the subcommittee (Mr. Boland) who has been so able in presenting this bill today. But how many times does the apple have to fall?145 Bauman went on to cite yet another case of ques- tionable research: NSF's grant of $36,500 to the 145Congressional Record,June 24, 1975, p. H6015. 105 University of Michigan for a study of the "Evolution of Song Learning of Parasitic Finches." I have a lot of finches in my district. I am sure they would be interested in this type of spending. At a dollar a finch they could all ben- efit... I suggest there is a need for revision of the basic system by which these research grants are made. They are handed out in an unregulated and secretive manner known as the "peer review system." This system allows cronies to get together and finance their pet projects, where grant application writing has become an art and where many people are not devoting themselves to basic research needs but, rather, to feathering their own nests, just as the finches whose songbird qualities are being studied under one of these grants.146 Bauman's charges are serious ones. Indeed, if the emphasis is on grantsmanship as an art rather than the significance of an applicant's research objectives, how can the public expect that worthwhile projects are being funded, that the best scientific ideas are being utilized? According to Dr. Rustum Roy: Today [that] the total proposal-aim--peer review system is approaching a dangerous level of inef- ficiency. We waste more time using up the best brains of the country in writing their own pro- posals. That is the major part of the ineffici- ency of the process: the writing and review of dozens of proposals--which is a totally nonproduc- tive, bureaucratic process. This is an enormous drain on American scientific manpower. Meanwhile, the great American invention, of the multiple sources of funding,which was the truly American innovation, that, I think, is endangered. Most of the money, especially for new work, tends to end up in a smaller and smaller number of agencies, principally ERDA and NSF.147 146Ibid. 147NSF Peer Review, Special Hearings, pp. 754, 755. 106 Are some people getting funded on the skill of the presentation of the idea rather than on the merit of the idea itself? Dr. Heath from NMU admits that there has to be skill in presentation for an applicant to be considered for a grant, but he argues that merit of the idea is the over-riding concern. Yet when asked if he believed a Columbus or a Kepler idea would get funded today, he indicated that, yes, it would. "I think if somebody got a really good idea and if he can write--but if he can't write, he can't get it funded."148 Does the wild idea, the really wild idea, out there get a chance? He responded: I think so. I've worked on the atom bomb. That was a wild idea and that certainly was funded. Now it may have been an administrative decision at the time but Bush, Einstein, etc., were really the peers in that instance. As a matter of fact, while I was in it I was wondering how they dared reach the decision. But really we were looking for new knowledge--we had an inkling on the way to go and we went there successfully. 49 NIE's Dr. Thomas Clements acknowledged the exis- tence of grantsmanship in the peer review system when he admitted that it is always possible that someone is more eloquent and rhetorically skillful than another, that some people write better or know more about the federal game than others. But he qualified his admission: I suppose their batting average would be somewhat higher. I do not think, and maybe I am not the 148Personal Interview. 1491bid. 107 best source on this because afterall, I am a Fed, but I really do not think that anybody is going to con my staff with unalloyed grantsmanship because my people do have substantive competence, in the area of their working, and they are not going to be conned by a bunch of glittering generalities. When NSF's Dr. John Maccini was asked what his organ- ization was doing about the existence of grantsmanship, he referred to the seven criteria used by reviewers in evalua- ting an application, as listed in NSF's Guide for Preparation of Proposals. And the very use of such criteria would seem to prevent a proposal from being accepted based only on its written quality, though Maccini added, "Now this doesn't negate the problem that you have got to write a good propo- sal...you have got to say it well."151 System Openness The issue of whether or not the peer review system is working to the disadvantage of worthwhile research by funding "well written project proposals" rather than substantive research efforts bears directly on the con- cern regarding the openness of the system. Do secrecy and confidentiality promote bias, favoritism, insubstan- tial research? Many of peer review's critics say yes. The first witness to testify before the oversight hear— ings committee, Rep. John B. Conlan from Oregon, levelled a harsh attack on the system: Here isanramazing system, gentlemen, where indi- vidual program managers are given carte blanche 150Personal Interview. 151Personal Interview. 108 authority to select peer reviewers who will be used to evaluate proposals. ...Unfortunately, under NSF's current management practices, they have a completely arbitrary system that is closed and unaccountable to the scientific community and to the Congress. It is common knowledge in the science community that NSF program managers can get whatever answer they want out of the peer review system to justify their decision to reject or fund a particular pro- posal. ...I know from studying material provided to me by NSF that this is an "Old Boy's System," where pro- gram managers rely on trusted friends in the aca- demic community to review their proposals. These friends recommend their friends as reviewers. ...It is an incestuous "buddy system" that frequently stifles new ideas and scientific breakthroughs, while carving up the multimillion dollar Federal re- search and education pie in a monopoly game of grants- manship. Conlan then went on to cite as an example of the system's weaknesses with regards to grantsmanship and openness the NSF funding, with more than $3.3 million, of the Individualized Science Instructional System--ISIS-- a curriculum project at Florida State University}53 He indicated that the program includes several explicit sex education courses and referred to one of the courses as being "so hot that it is unavailable for public or con- gressional review."154 I say this because one Of the sex mini-courses now being pilOt tested called "Human Reproduction," itself achieves a new height in science porno 152NSF Peer Review, SPeCj-a1 Hearings, pp. 4' 5' 153Ibid., p. 6. 154Ibid., p. 7. 109 literature. There is some material, including prurient questions aimed at mixed classes of 10th grade boys and girls, that I cannot bring myself to quote here in this public hearing. I will simply provide a typical "True-False" ques- tion here in my text so that you can see my point. This question in the Teacher's Guide for "Human Reproduction" is asked following one of the mini- courses many how-it's-done sex lessons: True or False. A man with a large, flaccid penis will have the same size erection as a man with a small penis.155 In favor of a public policy which would make reviews' and reviewers' names available to principal investigators and to the Congress, Conlan concluded by accusing NSF of being arrogant in its refusal to be more Open in its process. "The examples I have cited, are therefore, part of an overall pattern which, when taken together, consti- tute an indictment of a sick system that cries out for reform. The answer, gentlemen, is total openness in the system."156 The ”sickness" to which Conlan referred gains new dimensions in an article by Edward Roeder entitled "The Consulting Con Game: There are reportedly a lot of honest consultants, and this writer knows a few. But most, who oper- ate behind the scenes outside the bureaucratic safeguards and delays of the system, receive too little public scrutiny. They can turn out to be someone like E. Howard Hunt, a former White House consultant and $100-a-day project director for an Office of Education contract at the time he lssIbid. 1561bid., p. 11. 110 engineered the Watergate break-in. They can be beneficiaries of absurd wastes Of the taxpayers' money, such as the $23,000 HEW research study en- titled, "An Evaluation and Parameterization of Stability and Safety Performance of Two and Three- wheeled Vehicular Toys for Riding? (The study's chief conclusions? Tikes fall off trikes because they either (a) lose their balance, or (b) collide with objects.) Or they can simply be clever men who invent fraud- ulent schemes for their own profit. What follows is one such case, the con game of a trio of consultants who latched onto the Emergency School Assistance Program (ESAP), part of HEW's Office of Education. Aside from collecting fat consulting fees from ESAP, these men were able to award sizable contracts to phony grantees created by themselves or their friends, and to require legitimate grantees--usually small, black, Southern universities--to kick back a portion of their fed- eral funds to bogus consulting firms in which they had an interest. 57 When one acknowledges the legitimacy of these com- plaints, one must then confront a new question: How far can Openness in the decision-making process co-exist with effective evaluation of award applications? With regards to NSF at the oversight hearings, the agreement was over- whelming that increased openness per se is desirable. To what extent was hotly debated with choices ranging from the most secret to the most open, though no one ascribed to the extreme position that applicants should know noth- ing about who reviewers were or what they said.158 But there were four distinct positions on the degree of open- ness which should exist: 157New Times (November 1974): P. 26. 158NSF Peer Review, Special Hearings, p. 22. 111 (l) The NSF policy for many years, up to June 1975, was that applicants should receive paraphrased reviewer comments on request but should not know the identity of reviewers.159 (2) On January 1, 1976 the NSF Policy became: applicants should receive verbatim reviewers' comments on request but should not know the identity of reviewers. (3) Applicants should receive signed verbatim peer reviews on request (Rep. Conlan and Heinz Walsdorf believed this would be best). (4) Applicants and the public at large should have access to signed verbatim peer reviews (rejected by all). As indicated above, NSF had itself arrived at a new proposal in June 1975 regarding openness of the peer review system. At the oversight hearings Dr. Donald B. Rice, National Science Board member, presented the argu- ments considered in arriving at that proposal: One: Reviewers will be more candid on all aspects of the proposal, and the qualifications of the proposer to accomplish the work, if their identity and/or verbatim comments are kept con- fidential and not passed on to the proposer. Thus, confidentially, in this view, contributes to the quality of reviews. Two: Providing names of peer reviewers to principal investigators is likely to encourage personality clashes between the proposer and the reviewer, leading to unnecessary ill will and hard feelings among all parties concerned. Three: Program managers might have to take on the role of arbitrators between proposers and 159Ibid. Resolution adopted by the National Science Board at its 174th meeting on June 20, 1975, on Peer Re- view information. 112 reviewers, and would also find themselves spending more and more of their time defending their decisions orally and in writing. This would add measurably to their workload, and make it more difficult to come to the decisions they have to reach. Four: Many potential reviewers would refuse to participate in a system that required them to be identified with their comments to the authors Of the proposals they have reviewed. Five: Unless their identity was protected, re- viewers would be reluctant to comment critically on proposals by others who could adversely affect the career path of the reviewers. Six: The identification of reviewerszwould in- crease the opportunity for political pressures to intrude on the system. The resolution regarding confidentiality and peer review information was actually adopted by the National Science Board at its 174th Meeting on June 20, 1975. It had four resolves: l. The Foundation will publish annually a list of all reviewers used by each Division; 2. Program officers should seek broadly represen- tative participation of qualified individuals as reviewers; 3. Verbatim copies of reviews requested by the Foun- dation after January 1, 1976, not including the identity of the reviewer, will be made available to the principal investigator-project director upon request. The question of including the identity.of the reviewer will be consid- ered further by the National Science Board. 4. The Foundation, upon request, will inform the principal investigator-project director of the reasons 16°Ibid.. pp. 767-768. 113 for its decision on the proposal.161 All reviews requested prior to January 1, 1976 were to continue to be governed by earlier policies since those reviews would have been solicited with a commitment on the part of the National Science Foundation to the con- fidentiality established by that earlier policy. At the hearings Board representatives evinced the belief that the new policy will serve to improve information exchange with the scientific community and allow it to better under- stand the reasons behind National Science Foundation de- cisions.162 But there are dangers, too, when openness becomes a policy, especially when Congress can exercise control over the review procedure. In the hearings several wit- nesses voiced this concern, among them Rep.Tbm.Harkin who pointed out that "the 'Old boys' club is well in place 163 right here in Congress" and further indicated that when he took his bar examination that the names of those law- yers that did the review were kept secret.164 IDr. Norman Hackerman, Chairman, National Science Board, also later made the point that editors of many scientific and tech- nological journals were among the first to use the peer review process to judge the quality of original research 1611bid., p. 60. 162Ibid. 163Ibid., p. 34. 164Ibid, pp. 34,35. 114 articles submitted for publication, and he attributed the high quality of the majority of scientific journals to 165 this process. Rep. Tom Harkin warned of the dangers of political pressures if the names of reviewers were released: "You are going to have the worst kind of pres- sure. You are going to have political pressure, sudden 166 Later he said: political pressure." I feel whenever inquiry and freedom of thought is placed beyond the realm of the politician, then that country acquires a spiritual strength that makes it, in fact, a beacon of liberty in the world. That is what I want to protect. I don't want to become another Soviet Union or China, where people are channelized into their programs. I want freedom of--whatever nature, social sciences, hard sciences--freedom to inquire, and to challenge, 'the freedom to challenge the most deeply rooted be- liefs that people hold. Only by doing that can people really change and grow. I think this prior approval would take away that free- dom of inquiry. And the chairman, Rep. James Symington added: I think whatever kind of a frying pan we may be in now with respect to oversight might turn into a fire of old boyism if Congress became the court of last resort on all of these grants. For a while there would be few intrepid members who would question the grants. Little by little, it seems to me, a form of congressional courtesy would take over which might even turn into an active club on the NSF to go forward with grants which it may have had some suspicion about. I do not, in my own mind, know what grant applications are pending from any state. 165Ibid., p. 76. 1661bid., p. 36. 167Ibid., p. 240. 115 If I were to know that I might be inclined to make some friendly intercession. 53 Perhaps as a conclusion to this section on openness it would be interesting to ponder the results of a survey submitted to the committee by Dr. Vincent S. Haneman, Jr., First Vice President of the American Society for Engineer- ‘ing Education. The survey involved the Engineering Re- search Council whose members were polled on July 29, 1975. Of the seventy-six responses from 172 engineers, Dr. Hane- man reported the following: Question 1 - Concerning whether the faculty of an institution prefer to have their names placed upon the technical portion of the proposal~-33 of 76 said yes, 39 said no, and 4 abstained or did not give an indication which could be placed in either the yes or no category. Question 2 - Would the faculty prefer to have their names identified on the second part or the evaluation of the individual?--11 out of 76 said yes; 64, an overwhelm- ing majority, said no; and one was in question. Question 3 - If it were the will of the Congress that public identity of the reviewers be published along with their reviews, would-the faculty still be available to provide this service?--27 out of 76 said yes; 46 said no; and 3 abstained. Question 4 - Under the conditions of three, that is, the will of the Congress, did the respondents feel that 168Ibid., p. 241. 116 more candid, honest, evaluations would result from those who did volunteer? Only 8 said yes; 64 indicated that evaluations would be less candid, less honest; 4 ab- stained.169 169 Ibid, pp. 630-631.. CHAPTER V CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE RESEARCH It was the purpose of this researcher to present a spectrum of criticism regarding the peer review system employed by the agency-to-individual approach to federal funding 'Of scientific research. That presentation, it was hoped, could then lead to a more knowledgeable and real- istic evaluation of the system's workings. The National Science Foundation became the model in this case study approach because it utilizes peer review and because that utilization was made so visible in a series of Congres- sional oversight hearings in July 1975. But it is impor- tant to note with regard tx> the conclusions to be now Offered that statements made about NSF can be applied in some cases 1x) other federal agencies (such as the National Institutes of Health and the National Institute of Education) which also utilize peer review in some form. Indeed, according to Thane Gustafson, In practice, peer review procedures and patterns of proposal vary widely from one government body to another and even within a single agency. In some agencies peer review is elaborate, proposals are largely unsolicited, and the influence of advisory committees is great; in others peer review is cur- sory, the influence of agency staff outweighs that of outside advisers, and research proposals originate in response to requests from the agency. 117 118 Regardless of its mode of operation, however--a mode whose variances Gustafson outlines, the agency-to-individ- ual approach to federal funding has the same weaknesses and limitations as well as strengths. CONCLUS IONS Peer Review and Scientific Excellence The most important single consideration with re- gards to the peer review system is whether or not it promotes scientific excellence, quality research, effec- tive scientific advancement, innovation and change. The findings on the National Science Foundation by the Con- gressional oversight committee are interesting on this point. Witnesses at the hearings agreed overwhelmingly that some form of peer review should continue to be used in the allocation of federal funds for scientific research. Further, not a single witness suggested that peer review be abandoned. Personal interviews conducted by this author revealed the same attitudes, even from severe-critics. With specific regard to NSF, most hearing witnesses agreed that Foundation staff could not perform as well without the peer review system intact, and several argued that the potential for staff abuse is greater in systems lacking peer review. Bases for this latter argument 17o"The Controversy Over Peer Review," Science (December 12, 1975), p. 1060. 119 include the belief that peer review acts as a restraining influence on the staff in that outsiders (reviewers) act as checks against any agency biases and that the decision-making process is strengthened by the provision of knowledge and expertise from these scientists working in the forefront in the field. Thus, it can be concluded that no method superior to peer review has been found for judging the scientific competence of proposals. Hence, the argument is not whether the system should be replaced, but how it should be modified to enhance its positive aspects. There was not a great deal of testimony at the over- sight hearings directly addressing the issue of the qual- ity of projects supported by NSF. What testimony there was, however, reached consensus on the opinion that an appro- priately functioning peer review system would select for funding those projects at the upper end of the quality scale. Representatives from colleges and universities see the insurance of such quality to be tied to the peer review method of decision-making with regard to federal grants. In fact, they expressed opposition to an alter- native system wherein decisions by full-time federal Officials would be made without advice from "outside" scientists whose expertise guarantees the support of worth- while research. For modification of the system as it exists they did indicate that a sum equal to 15 percent of the total amount of federal research grants and contracts 120 received by each university be made directly available to the university itself rather than individual project researchers so as to avoid the rigidity that occurs when all research funds are ear-marked for narrowly specific purposes. Most testimony on quality research focused on NSF's support of innovation, and there was not much confidence expressed in peer reviewers' willingness to support innova- tive research if it challenged the mainstream of scien- tific thought or if it seemed unlikely to succeed. Per- sonal interviews revealed mixed feelings. Dr. Thomas Clements admitted that he thought that there would be "lots of people who feel they don't get a fair shake in the peer review system to the extent that it does reward orthodoxy."171 Arguments that the system does support innovation followed the reasoning that "nobody wants to turn down the next Nobel prizewinner." In addition, while admitting that many innovative ideas might be rejected, authorities such as Philip Handler, President of the National Academy of Sciences claimed that all types of research were being turned back due to lack of funding. According to Handler, "A growing fraction of all worthwhile research proposals is rejected annually for lack of funds rather than lack 171Personal Interview. 121 of scientific merit."172 Thus, it would appear that the peer review system could do more in the way of seeking out and supporting innovative research. And to insure that the National Science Foundation play a role in this effort, the con- gressional committee recommended that the National Science Board study the support of innovative research and report to Congress. The study is to include: a detailed anal- ysis of the issues covered in the report of the Subcom- mittee on Science, Research, and Technology dealing with NSF Peer Review; a look at major scientific innovations with regard to how they were funded and how they came to be accepted; and an examination of how adequate the decision-making procedure used by NSF programs is for finding and funding innovative research and recommenda- tions. The Congress also made a major decision when it charged the National Science Board, and not Congress, with primary responsibility for the establishment of policies governing peer review at NSF. Peer Review and the Question Of Fairness Whatever the success of the main mission of such a system (promoting scientific excellence), the question remains as to how £31; that system is in the realization of its mission. Is it equitable, impartial in its decision- making? Can it be accused of favoritism? As indicated 172NSF Peer Review, Special Hearings, p. 992. 122 in the previous chapter, the opinions on this matter vary. Several witnesses at the NSF oversight hearings cited the principle drawback to the use of peer review in the agency-to-individual approach as the potential which exists for biased evaluations by reviewers. This potential is based in the tendency for groups of reviewers to be composed of individuals from a common discipline who may praise each other's proposals too highly while giving lower ratings to "outsiders." To peer review's critics such favoritism becomes pOssible when one considers NSF data on reviewer selection. Thus, the New England, Pacific and Mid-Atlantic states together, which employ about 50 percent of the nation's doctoral scientists and engineers, account for nearly two thirds of NSF advisers.173 And for fiscal year 1974 data reveal that California, Massachusetts, Maryland, and Washington, D.C. were repre- sented, in terms of NSF reviewers, more than proportion- ately in relation to their population of scientists.174 Another charge that advisory panels are dominated by mem- bers of prestigious universities is confirmed by NSF statistics that show that within NSF the top eighteen institutions which employed 9 percent of the nation's academic doctorate-holding scientists in 1971 contributed 173Statement by Dr. R.C. Atkinson, Deputy Director, National Science Foundation in NSF Peer Review, Special Hearings. 174 Ibid 0 123 more than one third of the advisers.175 Thus, one could conclude that there is some truth to the charge that a certain amount of geographical/ institutional "favoritism" does exist with regards to participant selection for peer review. Yet whether this disproportionate emphasis on certain regions and schools leads to "cronyisnfl and unfairness in the actual award- ing of grants is another issue. Personal interviews conducted by this researcher revealed that no one doubted that such situations could exist, but none had personally witnessed any. Indeed, all indicated that biased reviews would be rare since the reviewer is putting his profes- sional reputation at stake every time he conducts a re- view. Still, there are data which reveal that patterns of funding within NSF give strong advantage to prestige institutions. Thus, in 1974 research grant awards to the top twenty institutions represented one third of total 176 But there is other NSF obligations for all programs. evidence that concentration does not always work to the advantage of the prestige institutions. A study of the allocation of NSF project grants in metallurgy and mater- ials research shows that they are clustered among the 175Preliminary results of a study of the NSF advis— ory system by N.C. Mullins, Social Sggence Studies (July 1, 1975). 176Gustafson, p. 1062. 124 gigglg ranked institutions rather than the top ones.l77 Therefore, the issue of concentration in funding and reviewer choice is more complicated than imagined by many. The meaning of any information must be measured against the question of whether an award is unfair or uneven. Are we seeking equity over scientific merit? It is ob- vious that those who say the peer review system is unfair are seeking equity, while those who are looking for scien- tific merit are asking whether the distribution of awards is really out of line with the distribution of scientific quality in the country's research institutions. The findings of Congress at the NSF oversight hear- ings reflect the feelings of two camps of opinion with regard to the question of fairness: the camp represented by Rep. John Conlan which believed that a "defensive" system should be devised to cut down on the number of dishonest participants in the peer review system (by de- fensive COnlan meant a system containing a mechanism for continual oversight of the review process, a mechanism which could include Congressional review), and the camp led by Rep. Charles Mosher which assumed the honesty of peer review participants and urged an adoption of trust in the system. Congress found that the NSF staff was dedicated, competent, and honest and established that the operation 177NSF Peer Review, Special Hearings, p. 747. 125 of the system should be based on trust of the staff and reviewers, combined with vigilance to ensure that trust is warranted. The rationale for such an approach is that a system based on trust allows more latitude for staff discretion and results in better performance than those systems based on highly structured procedure. Also a staff member or reviewer who is not trusted will either perform badly or withdraw from the system. Adoption of peer panels, like those used by the National Institutes of Health, was also suggested. These are study sections (convened advisory panels) organized for the review of those research proposals to be supported by grants, an organization which may minimize the possibility of undue influence or bias by program managers or individual re- viewers. In addition, the Subcommittee recommended that the National Science Board conduct a study on how much NSF should rely on peer panel review and report to Con- gress since it felt it did not have sufficient information on peer review systems to come to a definite conclusion at the time. Yet whatever the final judgment, it is doubt- ful if blind reviewing, peer panel review, or Congressional review will ever satisfy the extreme critics or their deep suspicions. In addition to the above, Congress determined that NSF should devise systematic methods which would ensure that competent reviewers be used regardless of institu- tional affiliation or geographic location. Further, the 126 committee recommended that in order to monitor the progress-— or failure--of fairness in the peer review system,NSF con- duct random audits of the decision processes used for the awarding of grants. Peer Review and the Cost Factor The economic costs of peer review in initiating and supporting research efforts in science were not directly addressed at the NSF oversight hearings. Dr. Christian Arnold charged that the system had to be the most costly one ever devised by man to generate research in science and that such a cost factor should not be dismissed with- out solid investigation. Ontflmaother hand, many within the agencies and universities saw no such problem or per- haps would not admit to such. But it is an area of con- flicting opinion. Conversations with graduate students presented in- sights quite contrary to the attitudes of agency Officals and university professors and administrators. For example, two graduate students revealed that they were among ten students within the same university department who received $8,000 each in a research grant for study in a specific scientific area. But only one of the ten was actually engaged and qualified in the field designated; the other nine were doing department related research ppt not_re- search for which the grant was awardgd. Such evidenceaas the above leads one to conclude 127 that the issue of research expenditures by federal agencies might bear further investigation. Though peer review's supporters maintain the high cost of the system is jus- tified by the type of research it encourages, one must also confront the dilemma posed by the divergence which may exist between 'the spurpose of a grant being awarded and the actual research carried out. Peer Review and Confidentiality With regards to openness in the peer review system, one can conclude from the hearings' testimony and personal interviews that people engaged in the review process would be less candid if their identity were made public. Indeed, many individuals would refuse to be reviewers if such publicity became established practice or if signed verbatim reviews were made available to applicants and even the public. It is this researcher's belief that much could be lost by insistence on a totally open system in terms of quality reviews and people willing to participate in the process. And it seems that the National Science Board's change in policy from "applicants should receive paraphrased reviewer comments on request but should not know the identity of reviewers" to "applicants should receive verbatim re- viewers' comments on request but should not know the iden- tity of reviewers" (an NSF policy since January 1, 1976) is a step in the direction of avoiding this loss. 128 There is no doubt that any degree of confidentiality makes a system difficult to defend from charges and crit- icisms such as those which have surfaced in this study. But the detriments that total openness could bring to the peer review system far outweigh the advantages to be gained. The problems of poor quality reviews, personality clashes, refusals by experts to particiapte, threats to future careers, and increased opportunity for the intru- sion of political pressure are powerful reasons for op- posing total system openness. In fact, it would seem that Congress itself is un- willing to move precipitously towards rejection of con- fidentiality. At the NSF hearings it urged caution in dealing with the issue and recommended further study by the National Science Board (via information collection of quantitative data on the review process and opinions from the scientific community) of the effect of confi- dentiality on the Operation of peer review. Specifically with regards to NSF's policy change (referred to above), the subcommittee expressed approval and suggested that "further changes in the level of confidentiality of the Foundation's peer review system should be made slowly if at all.”178 178N§F Peer Review Vol I, p. 59. 129 Summary Evidence of professional and political attacks on peer review without proper data evaluation and discussion was apparent in many cases. Particularly the articles appearing in professional journals and statements by some politicians are "lopsided" and could easily encourage distorted views on the part of other non-involved profes- sionals that the whole peer review system is a disaster contrived and run by persons with less than honorable in- tentions, that it is a system protecting self-seekers without the interests of research as a primary goal. This researcher's investigation did not lead to such conclusions. Yet such attacks have had positive effects in that they have sensitized Congress and the academic community to the issues involved in the workings of the peer review system. The results of such sensitization have been meetings and studies (such as those convened and carried out by the Elliott Committee and the Brookings Institute) and congressional oversight hearings for an agency such as NSF. And it is to the credit of the pro- fessional practitioners in the field, the NSF staff, and especially the members of Congress that a superb sense of balance was maintained in discussions and decisions at the NSF hearings. Confidence in peer review was clearly expressed. And what emerged was not the claim to per- fection of a system, for such a claim should surely mean the end of all growth, but rather a belief that this 130 system has within itself the capabilities for change and improvement making radical departure unnecessary. Ration- ality and balance prevailed as reflected in the basic recommendations of Congress and the opinions of the major- ity of those interviewed. RECOMMENDATIONS FOR FUTURE RESEARCH This investigation led to the acquiring of much information via documents, written opinions, and personal interviews that did not deal directly with the specific topic under investigation and was not used. Such informa- tion, however, did generate contemplation of areas for future study. One of these areas bears indirectly on one of the re- search questions posed for this study, that question which has to do with whether or not the peer review system pro- vides for the advance of science in the most effective way possible. In arriving at a thorough answer to such a query, one must take into consideration the effect of competition on the system. In other words, does the decentralized, com- petitive grant award process, currently promoted by peer review within the agency-to-individual approach to federal funding, encourage solid and innovative research along with an academic and public atmosphere supportive.of scientific excellence? Or does the competition which accompanies such a decentralized funding effort result in negative outcomes such as collusion, mutual isolation, and a one-sided 131 professional emphasis on research at the expense of in- structional performance and goals? In light of these questions, one possible area which merits serious future study is the effect of competition promoted by peer review on academic institutions themselves, their faculties, and students. Let us suppose, then,that a faculty member may be in a university department that gener- ates great pressure upon him to acquire research grants (or his pressure may be self-induced) because attainment of a grant brings the possiblity of publication, extra money, promotion, equipment, prestige or tenure. In other words, what percent of university policies, especially those deal- ing with promotion and tenure, induce such an orientation for faculty? Further, what consequence does such a competi- tive system have for the faculty person and his relationship to his teaching profession and ability? Is it positive or negative? And perhaps most important, what effect does the drive for the research grant have on the students? Are pro- fessors sacrificing their students, graduate and undergradu- ate, to satisfy administrative pressures or personal aims to "publish or perish," to carry out original research? If the answer to the latter question is yes, then per- haps a viable solution would be the hiring of univerSity faculty in a track system. Thus, when being employed, a professor could be given a choice of being hired and eval- uated in Track A or Track B. Track A would be primarily teaching, i.e., eighteen contact hours per week with the 132 expectation of one publication per year. Track B's, pri- mary mission would be research; for example, then, three contact hours of teaching per week might be required but concentration would be on research. The Track A employee would be evaluated on teaching ability and effectiveness, while the Track B person would be evaluated on his ability to conduct and generate re- search. All evaluations of ability, regardless of Track, would determine salary, promotion, and tenure. The advantages in such a Track system might include the following: 1. More professional pride and ability could be generated in the teaching field as a profession. 2. Students, undergraduate or graduate, would be less likely to suffer since professors' prime re- sponsibility would be fixed. 3. Greater faculty-student ratio would occur in- creasing production which would result in obvious economic implications. 4. Individuals would be more content due to their placement in roles fulfilling their needs and capabil- ities; production from both Tracks would thus increase. 5. Greater flexibility would exist within departments of a university; for example, a department could pro— gram the areas it wants to improve or emphasize by calling for a Track A or B person when filling vacan- cies or expanding. Such a system could also permit in— place flexibility by setting up criteria by which 133 professors could shift from Track to Track for per- iods of time. In suggesting the study of such a model for change within the university structure, this researcher is acknow- ledging the tremendous competitive effect the research grant system directed by the federal government has on the academ- ic community. Indeed, an examination of this effect has not yet been forthcoming from either the critics or the support- ers of the peer review system. Yet it is necessary to look at this subject because the viability of the system cannot be maintained if the university community suffers more than it benefits from the system's pressure for scientific excel— lence. And it would certainly seem at this point in history that the system and its pressures are here to stay. Indeed, according to Thane Gustafson: For most types of fundamental research the tradition- al project grant, selected by peer review, with over- all priority among fields and subfields determined at least in part by prOposal pressure, appears to provide the best available guarantee of scientific merit and accurate information. It is important, however, to extend existing safeguards: to choose advisors and agency staff who are representatives of the best sci- ence; to limit their terms of service; to separate as much as possible the evaluation of scientific merit from that of funding so as to reduce the dependence of researchers on the priorities or biases of any one agency or congressional committee; and finally to sub- ject the entire system to periodic review and criti- cism. Given the present--admittedly tentative—-state of our knowledge about the impact of the traditional peer review system, there is not a convincing case that the system's defects warrant the risk of sacri- ficing its virtues.179 179p. 1065. APPENDICES APPENDIX A COMPLETE LISTING OF INTERVIEWEES Dr. Dr. Dr. Dr. Dr. Dr. Dr. Dr. Dr. Dr. APPENDIX A COMPLETE LISTING OF INTERVIEWEES Allen Shinn, Deputy Assistant Director, Science Ed- ucation, National Science Foundation. Interviewed in his office, Washington, D.C., January 29, 1976. Roy Heath, Director of Research Development, Northern Michigan University. Interviewed in his office, Marquette, Michigan, March 1976. Ward Mason, Chief of the RtiD System Support Division, National Institute of Education, Washington, D.C. Interviewed in his office,January 30, 1976. Emerson Elliott, Deputy Director, National Institute of Education. Interviewed in his Office, Washington, D.C., January 30, 1976. Charles Haughey, Chief, Communications and Linkage Branch, National Institute of Education. Interviwed in his office, Washington, D.C., January 1976. Thos. Clements, Chief of the Information Communications Systems Division, National Institute of Education. Interviewed in his office, Washington, D.C., January 30, 1976. Christian Arnold, Associate Director, National Associa- tion of State Universities and Land Grant Colleges. Interviewed in his office, Washington, D.C., January 29, 1976. John Maccini, Program Manager Coordinator for Cost, National Science Foundation. Interviewed in his office, Washington, D.C., January 30, 1976. Richard Worksman, Advisor for Administrative Policy, National Institute of Education. Interviewed in his office, Washington, D.C., January 29, 1976. Joseph Crosswaite, Rotator, National Science Foundation. Interviewed in his Office, Washington, D.C., January 30, 1976. 134 Dr. Dr. Dr. Dr. Dr. 135 Raymond L. Smith, President, Michigan Technological University. Interviewed in his office, Houghton, Michigan, August 1976. Dale Stein, Department Head, Metallurgical Engineering, Michigan Technological University. Interviewed in his office, Houghton, Michigan, August 1976. Alfred Hendrickson, Professor, Metallurgical Engineer- ing, Michigan Technological University. Interviewed in his office, Houghton, Michigan, July 1976. Hunter Mormon, Senior National Institute of Education Associate, Office of Planning, Budget, and Program Analysis. Interviewed in his office, Washington, D.C., January 1976. Lloyd Heldt, Professor, Metallurgical Engineering, Michigan Technological University. Interviewed in his office, Houghton, Michigan, September 1976. Richard Heckel, Professor, Metallurgical Engineering, Michigan Technological University. Interviewed in his office, Houghton, Michigan, September 1976. APPENDIX B INDUSTRIAL RESEARCH OPINION POLL APPENDIX B INDUSTRIAL RESEARCH OPINION POLL Field of Research (Q 9) 141 returns 298 returns 393 returns 261 returns Physics . . . . . . . . . . . Biomedical and/or biolOgical. Chemical. . . . . . . . . . . Other . . . . . . . . . . . . Do you agree with the principle of Peer Review? Phys Biomed Chem Other Total Yes . . . . . . . . 82% 86% 79% 74% 80% NO. . . . . . . . . 18 14 21 25 ' 20 Do you believe that the various Peer Review Proced- ures as they operate in practice are in need of drastic change 34 40 44 46 42 in need of slight change 55 49 47 48 49 fair and reasonable 11 ll 9 6 9 The following comments and criticisms have been made about the operation of the Peer Review System, Please react. It encourages unorthodox ideas. definitely agree .3 5 4 7 5 agree somewhat 14 17 17 18 17 disagree somewhat 35 36 32 31 33 definitely disagree 48 42 _ 47 44 45 It promotes conflicts of interest, "croneyism," and favoritism. definitely agree 37 48 43 45 44 agree somewhat 45 36 44 43 42 disagree somewhat 9 9 10 8 9 definitely disagree 9 7 3 4 S 136 10. 137 During the decision-making process, the applicant has no adequate means to defend his proposal. Phys Biomed Chem Other Total definitely agree 49% 55% 48% 47% 50% agree somewhat 32 30 37 38 35 disagree somewhat 13 10 10 10 10 definitely disagree 6 5 5 5 5 The decision-making process is open and well under- stood. definitely agree 9 8 5 4 6 agree somewhat 21 15 l6 l7 l7 disagree somewhat 36 37 40 39 38 definitely disagree 34 40 39 40 39 Following an adverse decision, the applicant can easily appeal. definitely agree 5 5 2 2 3 agree somewhat 15 ll 9 12 ll disagree somewhat 36 30 42 39 37 disagree definitely 44 54 47 47 49 Funds are equitably distributed between young and estab- lished investigators. . definitely agree 3 6 3 2 4 agree somewhat l4 9 10 10 10 disagree somewhat 33 27 33 35 32 definitely disagree 50 58 54 53 54 How many years of experience in research have you had? less than 5 9 13 15 18 15 5 to 10 23 31 30 22 27 11 to 20 38 33 30 34 33 more than 20 30 23 25 26 25 How many times have you served as a member of a study section, council, or committee directly related to Peer Review decisions? none 57 70 72 57 66 once 7 8 8 10 8 2 to 5 25 14 15 l9 17 more than 5 ll 8 5 l4 9 11. 12. l3. 14. 15. 138 Given the approximate numbers of grant submitted throughout your career: Phys Biomed Chem To NIH, approved and funded None 84% 45% 81% One 5 l3 7 Two 1 9 4 3 to 5 9 18 5 6 to 10 l 10 l 11 to 15 - 2 l 16 to 20 - l - More than 20 - 2 1 To NIH, approved and not funded or not None 81 46 82 One 13 20 11 Two 3 l4 3 3 to S 2 14 3 6 to 10 l 3 1 11 or more - 3 - To NSF, approved and funded None 67 76 75 One 12 9 9 Two 6 9 8 3 to 5 ll 5 5 6 to 10 2 l 2 11 or more 2 - l applicatiOns Other Total 90% 73% 3 7 3 7 3 9 - 3 - 1 l l - 1 approved 93 74 3 12 2 6 l 6 l 2 76 75 11 10 5 7 6 6 2 l - 1 To NSF, approved and not funded or not approved None 59 74 72 One 18 14 12 Two 11 5 9 3 to 5 9 6 5 6 to 10 2 1 2 11 or more .1 - - To AEC, approved and funded None 79 92 88 One 7 3 5 Two 5 l l 3 to 5 6 l 4 6 to 10 l 2 l 11 or more 2 l l 74 11 6 6 2 Iowwoxas I-‘NUJNU'IQ 139 Phys Biomed Chem Other Total To DOD, approved and funded None 55% 90% 86% 68% 79% One 11 l 5 4 4 Two 8 4 l 8 4 3 to 5 15 3 4 10 6 6 to 10 4 2 2 4 3 11 to 15 3 - 1 2 1 16 to 20 2 - - l 1 More than 20 2 l 3 2 To DOD, approved and not funded or not approved None 57 92 84 73 80 One 13 2 7 5 6 Two 13 3 4 5 5 3 to 5 l3 3 4 5 5 6 to 10 7 S l 4 2 11 to 15 l 5 l 2 l 16 to 20 l 5 - - 1 More than 20 2 5 - 3 1 To other federal agency, approved and funded None 67 72 81 69 74 One 12 ll 6 10 9 Two 6 6 6 3 S 3 to 5 6 9 4 9 7 6 to 10 6 2 2 4 3 11 to 15 l - - 2 l 16 to 20 l - - l - More than 20 l - l 2 1 To other federal agency, approved and not funded or not approved None 73 79 81 73 78 One 9 6 8 7 7 Two 8 6 4 4 5 3 to 5 5 5 6 9 6 6 to 10 3 2 l 2 2 11 to 15 - 1 - 2 l 16 to 20 1 - - 1 - More than 20 l l - 2 l 20. 21. 140 Phys Biomed Chem Other Total To private or other foundations, approved and funded None 76% 52% 58% 80% 67% One 8 17 12 9 12 Two 5 ll 6 4 7 3 to 5 8 14 ll 2 9 6 to 10 2 4 2 3 3 11 to 15 l - 1 l l 16 to 20 0 l - 1 5 More than 20 - l - - 5 To private or other foundations, approved and not funded or not approved None 75 6O 73 79 71 One 10 l4 13 10 12 Two 7 10 6 3 7 3 to 5 3 12 6 5 7 6 to 10 2 4 l l 2 11 to 15 - - l - - 16 to 20 l — - - - More than 20 2 - - 2 1 SELECTED B IBLIOGRAPHY SELECTED BIBLIOGRAPHY BOOKS Bailey, S.K. Congress Makes a Law. New York: Columbia University Press, 1950. Bert, John W. Research in Education. New Jersey: Pren- tice Hall, Inc. 1970. Bush, Vannevar. Science - The Endless Frontier. Washing- ton, D.C.: National Science Foundation, 1945. Gottschaulk, L. Understandinngistory. New York: Alfred A. Knopf, 1963. Johnson, S.W. On the Relations that Exist Between Science and Agriculture. New York: State Agricultural Society Trans., 1855. Kahn, R.L. and Connell,C.F. The Dynamics of Interviewing. New York: John Wiley and Sons, 1957. Knoblauch, H.C., Law, E.M. and Meyer, W.P. State Agricul- tural Experiment Stations. Washington, D.C.: U.S. Department of Agriculture, 1962. Orlans, Harold. Science Policy and the University. Washington, D.C.: The Brookings Institute, 1968. Ross, E.D. The U.S. Department of Agpiculture During the Commissionership. Washifigton, D.C.: Agricultural History, 1946. Thomas, Lawrence C. Philosophical Redirection of Education- al Research. Chicago: University of Chicago Press, 1972. Weyl, F. Joachim. Research in the Service of the National Purpose. Washington, D.C.: U.S. Government Printing Off1ce, 1966. 141 142 PUBLICATIONS OF THE GOVERNMENT, THE NATIONAL SCIENCE FOUNDATION AND OTHER ORGANIZATIONS National Academy of Sciences, Committee on Science and Public Policy. "Federal Support of Basic Research in Institutions of Higher Learning." Washington, D.C., 1964. The National Institute of Education: A Brief Outline of *its History, Status, and Tentative Plans. Washing- ton, D.C.: Library of CongresE: Congressional Re- search Service, April 2, 1973. National Institute of Education. The Status of Educa- tional Research and Development in the United States (1975 DatabookTTL Washington, D.C., May 1975. National Patterns of R & D Resources. Washington, D.C.: U.S. Government Printing Office, 1967. National Science Board. "Resolution on Peer Review Infor- mation." Adopted at the 174th Meeting, June 20, 1975. National Science Foundation. 17th Annual Report. Wash- ington, D.C., 1968. . Federal Funds for Research and Develgpment and Other Scientific Activities. Washington, D.C., Fis- cal years 1973, 1974, 1975. . Grants and Contracts Awarded Fiscal Year 1975. Washington, D.C., 1975. . Grants for Scientific Research. Washington, D.C., 1973. . Minority Institutions Science Improvement. Washington, D.C., October 1974. . Mosaic. Washington, D.C., November/December 1975. . Research and Develepment and Economic Growth/ Productivity. Washington, D.C., 1971. . Research Initiation and Support. Washington, D.C., 1976. . Science and Engineering Technician Education Program. Washington, D.C., 1976. 143 Sponsored University Programs for Research and Education. East Lansing: Michigan State Ufiiversity Press, Office of Research and Development, Publication 12, 1971. United States Congress. House Committee on Government Operations. The Administration of Research Grants in Public Health Service. 90th Cong., lst sess., 1968. . Congressional Record. June 24, 1975. . House Subcommittee on Science, Research,and Development, Committee on Science and Astronautics. A Bill to Amend the NSF Act of 1950. 89th Cong., lst sess., 1965. . Federal R & D. Progpams--The Decision-Making Process. 84th Cong., 2nd sess., H.R. 1664, June 27, 1966. . Subcommittee on Research and Technology Pro- grams. Federal R & D Programs--The Democratic Pro- cess. 59th Cong., 2nd sess., January 7-11, 1966. . Government and Science Hearing_. 88th Cong., lst sess., October 15 - November 20, 1963. . Government and Science Hearings. 89th Cong., lst sess., June 23 - August 1965. . Select Committee on Government Research. Hearings on Federal R & D. Pregrams, Pts. 1-3. 88th Cong., lst and 2nd sess., November 18, 1963 - January 22, 1964. . Hearings Before the House Subcommittee on Appropriations. Washington, D.C.: U.S. Government Printing Office, 1963, 1964. . Subcommittee of the House Committee on Appro- priations. Hearings on U.S. Departments of Labor and Health, EducatiOn, and Welfare Appropriations 1968. 90th Cong., lst sess., 1967. . Subcommittee on Executive Reorganization and Government Research. An Inventogy of Congressional Concern with R & D. 9lst Cong., 2nd sess., I97I. . House Committee on Science and Astronautics. Leadership in Science Poliey: The National Scienge Foundation--Its Present and Future. 89tHiCong., lst sess., 1965. 144 . NSF: A Genera1rReview. 89th Cong., lst sess., H.R. 1219, 1965. . National Science Foundation Peer Review. Special Oversight Hearings, Hearings BefOre the House Subcommittee on Science, Research, and Tech- nology, July 1975. . National Science Foundation Peer RevieyLVol. I: A Report of the Subcommittee on Science, Research, and Technology, U.S. House of Representatives. 94EH Cong., 2nd sess. Washington, D.C.: U.S. Government Printing Office, January 1976. United States Department of Navy. 1967 Annual Report - Naval Research Laboratory. Washington, D.C., 1967. . Office of Naval Research - Contract Research Program. Washington, D.C.:' ONR-1, June 1962. United States Office of Naval Research. A Decade ongasic and Applied Science in the Navy. Washington, D.C., March 1957. The White House.. Scientific Prggress: The Universities and the Federal Government. November 15, 1960. PERIODICALS Abelson, Philip H. "Chemistry in the Universities." Science 144, April 17, 1964. . "Congress and Research." Paper delivered at The Second Radioisotope Conference, Gatlinburg, Tennessee, April 20, 1964. . "Departure of the President's Adviser." Science 179, January 1973. .’ "Distribution of Federal Research Funds." Science 144, May 1, 1964. . "Distribution of Research Funds." Science 142, October 1963. . ”New Centers of Excellence." Science 150, October 1965. . "Only One Side of the Question." Science 145, July 1964. 145 . "The President's Science Adviser." Science 142, November 1963. . "Proliferation of Bureaucrach'Science 163, February 1969. . "The Research and Development Pork' Barrel." Science 149, July 1965. . "Science Dropouts." Science 144, June 1964. "Air Force: More Funds to Outside Labs." Industrial Research, 1975. Allott, Gordon. "Massivity in Financing Research." Science, January 1965. "ABC Asks President to Urge Full $460 Million in Grants for Undergraduate Facilities." Science, May 1964. Arnold, Christian K. "Higher Education: Fourth Branch of Government?" Saturday Review, January 1964. "Australia's Famed Hall Institute is Hit Hard by U.S. Budget Cuts." Scientific Research, December 1968. Ben-David, Joseph. "Scientific Productivity and Academic Organization in Nineteenth Century Medicine." Amer- ican Sociological Review 25 ‘December 1960, 6. "Better Administration of R & D Grants Urged." Chemical and Engineering News, August 1964. "Bleak Future Seen for Scientific Research." Machine Design, November 1970, p. 8. Boffey, Philip M. "Mansfield Amendment Curb on Basic Research May Spread." Science 167, March 1970. . "Peer Review Under Attack." The Chronicle of-Higher Education, May 1975. . "Science and the White House." The Chronicle of Higher Education, July 1974. Bross, Irwin D.J. "NIH Career Awards." Physics Today, March 1965. Bush, V. "Vannevar Bush Speaks." Science 142, December 27, 1963. "Career Training: The New Realism." U.S. News and Wopld Report, September 1, 1975, pp. 48-50. 146 Carey, William D. "Peer Review Revisited." Science 189, August 1, 1975. "Circular Letter No. 33, 'National Association of State Universities and Land Grant Colleges,"September 1965. "Congress May Investigate NAS." Science 179, January 1973. "Congress Pressed on Research Cost." The New York Times, May 22, 1964. "Congress vs. NSF." The Chronicle of Higher Education, July 7, 1975. ——' Cowen, Robert C. "Cost of Research Draws U.S. Probe." The Christian Science Monitor, May 4, 1964. Craig, May. "A Woman Writer Takes a Critical Look at America." U.S. News and World Report, February 1964. "Creutz Sees a Gradual Increase in NSF Support of Applied Science." Physics Today, November 1970. Culliton, Barbara J. "NSF Defense of Closed Peer Review System Not Persuasive." Science, August 1975. David, Edward E., Jr. "Directions for R & D." Research Development, September 1975, pp. 18-26. Davis, Harold L. "Peer Review on Trial." Physigs Today, September 1975, p. 96. "Defense Department Will Cut Back Foreign Research 60 Percent by Fiscal 1970." Scientific Research, September 1968, p. 17. ‘ "Dr. Abelson Scores JFK's Science Staff." Houston Press, November 20, 1963. Dupree, A. Hunter. "New Policy for the Government- University Partnership." Science 169, July 1970. "$500 Million Asked for Converting Engineers." Machine Design, August 1971. "Glum Physicists Eye Their Dwindling Funds." gusiness Week, May 1968, pp. 87-89. Goudsmit, S.A. "Editorial: A Drastic Change in Policy." Physical Review Letters 13, No. 3, July 1964. 147 Greenberg, Daniel S. "Congress and Science: NSF Hearings Provide Some Illuminating Insights on the Deterior- ating Relationship." Science 142, October 1963, pp. 368—370. . "Daddario Committee: Hearings to be Held on Overhead Support and Geographical Distribution." Science 144, April 17, 1964. . "Grant System: Elliott Committee Finds Flaws, Diversity in Study of Practices of Federal Agencies." Science 145, August 1964. . "LBJ Directive: He Says Spread the Research Money." Science, September 1965. . "When Pure Science Meets Pure Politics." The Reporter, March 12, 1964, pp. 39-41. Gustafson, Thane. "The Controversy Over Peer Review." Science 190, December 12, 1975, pp. 1060-1065. "Hands Across the Sea." Scientific Research, August 1968. Hotz, Robert. "A Hard-Nosed Budget." Aviation Week and Space Technology, February 1973. . "Toward a Technological Appalachia." Aviation Week and Space Technology, March 1971. "How Things Look in a Big New U.S. Industry." U.S. News and World Report, October 19, 1964. Hudkins, Lonnie. "U.S. Has NO Research Policy." Houston Press, November 19, 1963. "In 1972: 8% More Money for R & D." Machine Desigp, January 1972. "IRRPOS (Interdisciplinary Research Relevant to Problems of our Society) Looks for Relevance to Society's Problems." Physics Today, November 1970. "Italy: Political Turmoil Kills Plan for First Doctoral Program." Science, May 1970. Jones, Robert. "Much of U.S. Research Controlled by Faulty Mechanism." Industrial Research, September 1976, pp. 87-92. “Kennedy Asks More for NSF." Science 168, April 1970. 148 Kerr, Clark. "Society and the Status Quo: The Individual and the Innovative Society." Science 144, April 1964. Lahey, Edwin A. "Research Bias at Midwest." The Kansas City Star, July 15, 1965. Levin, Louis. "Grants and University Authority." Science 146, November 1964. "Massive U.S. Research Cuts Poleax Jobs of Scientists." The Houston Chronicle, November 1970. "MOHOLE: The Project That Went Awry." Research and Devel- opment, January 1964, p. 115. "NIH: End of an Era for Foreign Support." Scientific Research, August 1968. "NSF's Appropriation Has Been Cut by the House Appropria- tions Committee." Chemical and Engineeripg News, May 1964. "NSF Budget: House Group Reacts to Data on Plight of Science." Science 168, April 1970. "NSF Undergraduate Education in Science, Project Director's Meeting." Educational Newsletter, March 1970. Naughton, James. "Nixon's Science Aid Unseen: Is He Heard?" The Milwaukee Journal, March 8, 1972. "The Needs of Chemistry." The New York Times, May 1964. Nelson, Bryce. "A Surplus of Scientists? The Job Market is Tightening." Science 166, October 1969, p. 582. "New Centers of Excellence." Industrial Research, May 1964. "New Role for NASA Research Center." Science 168, April 1970. "Nixon Sets Shift in Science Job." The Milwaukee Journal, January 1973. "Ohio's Old Laissez-Faire Attitude on Linking Education, Research, and Industry Undergoing Change." Science 145, July 1964. Osmundsen, John A. "Hostility Toward Science is Detected." The New York Times, December 29, 1963. 149 Parrish, John B. "Growth and Dispersion of Industrial Research Manpower." Research and Development, May 1964. "Peer Review and Grants." Industrial Research, June 1975. "Pentagon Cheats, Rickover Charges." The Detroit Free Press, February 19, 1970. "Perspectives in Federal Physics Support - an Interview with Chalmers Sherwin." Physics Today, September 1967, pp. 73-74. "Physics Fads and Finance." New Scientist, August 1974, p. 462. "Poverty Pays--Federal Way." The Daily Mininngazette, Houghton, Michigan , February 23, 1970. "Preparing for Change in Defense Spending." Stanford Research Institute, March-April 1964. "Regional Equity, a New Principle in Distribution of R & D Moneys." International Science and Technology, May 1964. "R & D Seeks a New Strategy for the Seventies." Industrial Research, March 1970. "R & D to Top $30 Billion in '73." Machine Design, August 1973. "Research Grants by U.S. Criticized." The Kansas City Star, December 11, 1964. "Research-Poor Areas Must Help Themselves." Chemical and Engineerinngews, May 1964. "Returns are Poor from R & D Spending." Machine Design, June 1970. Roeder, Edward. "The Consulting Con Game." New York Times, November 15, 1974, pp. 26-32. Rowsey, Luther. "Meet the Man Who Splits the Atoms of U.S. Research Policy." Houston Press, Nobember 19, 1963. ' Satchell, Michael. "The Riley Affair." The Washington Star, February 1973. 150 "Science Structure Criticized." Industrial Research, May 1969: Pp. 42-46. "Scientists Gloomy on Science's Future." Scientific Re- search, May 1968, p. 12. Shapely, Deborah. "NSF: A 'Populist' Pattern in Metall- urgy Materials Research." Science, August 1975. . "Surprise Appointments Restore Science to White House Favor." Science, November 28, 1975, p. 859. "Shift of NSF Funds." Science 167, March 1970, p. 1672. "Shrinking Funds Mean Bleak Outlook for Italian Biomedical Research." Scientific Research, March 1968. Spilhaus, Athelstan F. "Man in the Sea." Science 145, September 1964. "Teaching vs. Research on the Campus." The Kansas City Star, June 27, 1965. "U.S. Science Endangered, Board Feels." The Milwaukee Journal, February 29, 1970. Wade, Nicholas. "Agriculture: NAS Panel Charges Inept Management, Poor Research." Science 179, January 1973. Walsh, John. "NSF and Its Critics in Congress. New Pressure on Peer Review." Science 188, June, 1975. "NSF to Make More Peer Review Information Available." Science 189, July 1975, p. 123. . "NSF Peer Review Hearings: House Panel Starts with Critics." Science 189, August 8, 1975, pp. 435-537. "Washington Concentrates: NSF's Appropriation Has Been Cut by the House." The Chemical World This Week, January 1964. "Washington Ramble: News in Brief on Investigations, Accelerators, Anger in NSF, and Other Matters." Science 142, November 1963. Wolfle, Dael. "Geographic Distribution of R & D Funds." Science 147, January 1965. 151 "Woolridge Report: Study of NIH Producing Conflicting Reactions among Congressional Figures." Science 148, Arpil 16, 1965, pp. 351-352. MICHIGAN STATE UNIVERSITY LIBRARIES !|1IIJIHIIHI\llllllHlfllllHllllHllHillllHlHWHHIIIIH 3 1293 03177 3306