. -... 7.“-.. A METHOD FOR MANAGEMENT OF INTERNAL AUDIT RESOURCES IN MULTI-LOCATION MILITARY AUDITS Thesis for the Degree of Ph. D, MIGHIGAN STATE UNIVERSITY IIMIE KUSEL MAJOR, USAF 1972 L!“ 1RY Michigan State University This is to certify that the thesis entitled A METHOD FOR MANAGEMENT OF INTERNAL AUDIT RESOURCES IN MULTI-LOCATION MILITARY AUDITS presented by i Jimie Kusel Major, USAF has been accepted towards fulfillment of the requirements for f\1r 9- degree in A((0Un¥i‘wé\- é)» ' f/ch‘fnc, /L;//;:Ctl/ Major professor Date Dec 7 r I‘YIX. 0-7639 1': amomc av E ' HUAG 8 SUNS, BUUK BINDERY INC; ‘_ .IBRARY amoans -Pmsroltiljlcwsu _ _. “a. _.. . in s . . o ‘. .Hu a» .: \dn V‘ .v I a» ‘, Vs‘ \A .(w A» A; P~ a... 9» ,A u A... L. G N u .— h LE as» O _ ,.. e C vs T. Z i. E? s: .2 .s a. at g .‘ ABSTRACT A METHOD FOR MANAGEMENT OF INTERNAL AUDIT RESOURCES IN MULTI-LOCATION MILITARY AUDITS BY Jimie Kusel Major, USAF JQEAQPBeA Department of Defense internal audit staffs have “Zaireduced relative to the size of the internal audit ;; the primary organization of interest. ,The research concerned an analysis of internal audit iUQQMQleted by the Air Force Audit Agency. The pri— ' ;_. ‘QA'I'A I . 'uy »' :ouou .~'vl' . ‘ Q hop-y -.. On p — Inn-v. 5.5“ -- .- :"'" VAI—fiu' vow-.4... C..- y t A'- ..A'\--. n‘ r ‘0‘“; I5G- e" ‘ u .Ia. V ‘ I - no.1..“ ‘Sn‘o ~-- A 9. CV . . w. Q‘av‘ - 0t R. p. \ ‘§ a: C3“ " ‘ . .I :-».I‘ . ‘6 ‘ ‘ ‘u “a. Q ‘Tisnu ”Ln a V- Jimie Kusel Audit Agency and are therefore of direct applicability to that organization, the methods should be of interest to other multi—location organizations operating in a similar environment. Much of the discussion centers around what are called information derived curves. In simple terms, these curves measure the accumulation of new items of informa- tion (called reportable type items) provided by each addi— tional location included in an audit. It is argued that where statistical techniques cannot be adapted economically to certain internal audit tasks, the information derived curves can be used to provide quantitative data for making auditing decisions. Four basic hypotheses were advanced: 1. The first few locations in a judgmentally selected sample of locations make relatively large contributions to the total amount of information derived from the audit. 2. The information derived curve developed from a judgmental selection of individual audit locations is significantly superior to such curves developed from a random selection of locations. Jimie Kusel 3. The information derived curve can provide an internal audit management tool, the use of which can result in more efficient use of internal audit resources. 4. There is a range of locations for a given audit beyond which the incremental cost of audit tends to exceed the value of the incre— mental information. The research was directed toward confirmation or refutation of each of these hypotheses. Although the obtainable evidence was not of such nature to "prove" that an hypothesis is true or not true, the preponderance of evidence was sufficient support to accept each preposition. The evidence was gathered from a detailed analysis of six— teen subject matter audit reports that summarized informa- tion from more than 1600 individual reports of audit, and by interview with audit managers directly and indirectly associated with these audits. The dissertation is organized into eight chapters. Chapters I and II provide general background to the research area and introduce the reader to the military environment in which the research took place and for which the suggested audit methods were developed. Chapters III ; w . ' ‘ I8 I ‘. Jud oi . ‘ i -:.LY.‘ “‘3‘ ‘."."'.It" . :Fn .'- Tm~Ter \ ‘II “ ‘ ”‘s V ‘2‘. ‘In \"‘ "»u 3 N. ‘ i“ ‘ A,- h ~.(‘e"fi ‘ . ..D O. T 6 Jimie Kusel and IV review the details of the research. Here the eco- nomical range of locations to use in each audit is dis— cussed and the shape of the information derived curve is determined. Chapters V—VII discuss specific methods for improving the management of internal audit resources stemming from the research results provided. Specifically, Chapter V discusses the use of the information derived curve in determining management areas in need of audit attention. Chapter VI discusses the use of the curve in the management of the field test of new audit programs. Chapter VII discusses a new audit technique called the seg- mented audit. The last chapter, Chapter VIII, reviews the extent to which the hypotheses proposed were confirmed. . A. - I.‘ .II Fm I « w . . ‘ \ r« .n1 v be m: AIU A METHOD FOR MANAGEMENT OF INTERNAL AUDIT RESOURCES IN MULTI-LOCATION MILITARY AUDITS BY Jimie Kusel Major, USAF A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Accounting and Financial Administration 1972 . . x m S‘j'lzrliu _; ' . . ‘ w- 140 lad arm 1‘. we _. ‘. ' ‘: '.‘.tu..’r r. ’~ “Wu -' 1‘ ts.- .r,*:: not» t'.‘;l,_ . law «9 ”298(5th ». -I _. _ \. " fifth any. , : iwm been in"eiuair) r-.._ ' , . ,. ’zflo-ix.’ma‘ .bi.\_ ‘ ‘Qfi‘iA-,.‘: 1‘”: - .'-‘." “be" c! m" extant r» ‘3' ‘ ‘rom my father and mother, :T5' is Milan Reamer- ev 3‘ . g . He enry‘ and Juanita Kusel 413'} m Mot-user. 'i‘heu toms-i 1's- ’- L" agar-W. .- 2' t “11‘ 1i)!“ (0 0X91: .s 7 a; u; ; .5. Md" 5.. Teubnur: 'F‘IE -\t::‘;;*..\" ’-u."t-‘."?-‘I '4'). ”Boyd Deputy Altdt'. - General, Fur ~ a - ”Mlish the unearth v.5 1.- 2)... I". r‘c' "1- 7 (2- :‘\“~' 'ffi fli- ond noopew r 1m- Aun 1am to ems-a my appreciation to ‘ywmotwfimm a b ‘ v-u ~ ~ Q... ,‘. M “..n\.‘ u.‘ -A-’!-.n-n ‘- V .- “"’""‘-4‘ ~ 1 I'M.” I L:: T". . ”My . ‘ -‘.. .f‘l, II C I ‘ ‘ T. ed _J 1" c"- ‘5‘. '1» use .I"'L_‘ ‘ ‘ ‘ ""A «to “3‘ -C‘.‘ C ‘. A ‘c ‘ K ‘ s C: -v “Q U‘~~~‘ ‘ i I- '1 ‘F. VAR Huu‘ l‘. C‘t: . h a A. ‘ en. £2.31 L- r .-..»“‘ .I: I! ,‘F “. =Gr., “~~‘ J- I ‘ ‘ «E‘u.:~"' n “_“~: ‘4.- ‘ ‘ -“~.‘ ‘ Nun‘k fang, .2‘. I I ‘ y \§‘erfl "‘ ‘u n‘ 5‘ ‘ ‘ ‘\:~¥l’ ‘ T “I‘s . q‘ ‘F . V..- "\{Eh FP———_ .'.'__- ACKNOWLEDGMENTS I am especially indebted to Dr. Gardner Jones who guided and maintained a very active interest in my work. He was chairman of the Dissertation Committee, advisor, motivator to greater achievement, and friend. His contri- bution has been invaluable and this acknowledgment is only a small token of my appreciation for his efforts. The other members of my committee, Dr. Bruce Coleman and Dr. Maryellen McSweeney also contributed significantly to this dissertation. Their comments, criticisms, suggestions, and interest were greatly appreciated. I would like to express my appreciation to Major General Harold C. Teubner, The Auditor General, and to Mr. Trenton D. Boyd, Deputy Auditor General, for their permission to accomplish the research within the Air Force Audit Agency. A special thanks is also due to my many friends within that agency whose assistance and cooperation greatly aided completion of the research. I would also like to express my appreciation to Jo McKenzie for the outstanding job of typing the ‘ -uv. . €an ‘Ol. . v n 'Q 8.. .. .~nv‘ ". .Q-’ a“ a Co 5.5 --.. . Joann-Hr. I.- j. O odh A. .C e C .. r. e .3 n4 v. u a nu ..o -. .3 .-m t. at .t . . C. .n. v. .C a: i. at V. n» .»u A.- v. .. T. a». I‘fll'n. :- .~‘I .‘ é'fier speed, accuracy, and attention to detail [Fible in permitting the timely submission of the ‘aissertation. Ti3»that owed to my wife Janet. She alone knows what Afififieosts of this dissertation and degree have been. I‘““®~1§lflost all of our married life I have been enrolled at ' I'0 X‘. -. 37,?y fiabsional Air Force schools or involved in working on “ZINE? Wat-u; - »: degree at some college. Yet, through it all she has T fibeside me, encouraging and assisting whenever and =yer possible. To her, my most loving thanks. i Y ¢-..,-_.' 2, l ),\.H'...‘, xr \ I :5 ~ I‘f.‘ 7' X1. . u. -- A": 3322-0323 . .53, I ;.l ' :92» .". r5517. ‘- ._. " ' . _, I- 5 . ' ‘~ U...‘ g‘ . -- ear rsi; . EITC‘AJTLJ - . s - . - ~ » - 77‘" .1' 00.. P v! p. D " .— uo... S. -. w-.- -‘ _ ‘ o -05. .. . - . l'n.. ‘ . Ehapter I. TABLE OF CONTENTS OF TABLES . C I I O I O I O C O O O I C O 0 ”T or F I GIJRE S I l I I l I I O I I O I O I O C INTRODUCT I ON C C I O O O O C I O O O . DOD Internal Audit Agency Responsibility. . . . . . . . . . . The Internal Audit Product . . . . . The Economics of Sampling. . . . . . The -Research Focus . . . . . . . . Research Background and Approach . . Purpose. . . . . . . . . . . . . . . Research Parameters. . . . . . . . . Four Basic Hypotheses. . . . . . . . General Organization . . . . . . . . Footnotes. . . . . . . . . . . . . . THE AIR FORCE INTERNAL AUDIT ENVIRONMENT When Air Force Audit Was Born. . . . Air Force Audit Agency Headquarters - Line and Staff. . . . . . . . . The Regions -- Geographic Slices of ”the WOrld . . . . . . . . . . . . . The Service—wide Systems Division. . Logistics r- The AMA Approach. . . . Acquisition Systems Division . . . . The Audit Report -- A Final Product. ~91 an Evaluation of The Service . . . . 'WK Page viii xi 13 14 20 20 21 23 25 26 27 28 32 33 35 35 36 38 4Q: 41. «'.;—-ev 'w'" . o . l Chapter Page III. RESEARCH - PHASE I AND II. . . . . . . . . . 42 The Sampling Rationale . . . . . . . . . . 43 Mechanics of The Approach. . . . . . . . . 46 Definitions . . . . . . . . . . . . . . 46 Phases of Research. . . . . . . . . . . 60 The Criteria. . . . . . . . . . . . . . 62 Research - Phase I . . . . . . . . . . . . 64 The Objective . . . . . . . . . . . . . 64 The Reportable Type Item. . . . . . . . 64 Selection of Subject Matter Audits. . . 65 ’ Selection of Sample Locations . . . . . 66 I The Research. . . . . . . . . . . . . . 67 | Results of The Research . . . . . . . . 68 L An Evaluation of Results. . . . . . . . 70 ' Management Action . . . . . . . . . . . 73 Additional Research Needed. . . . . . . 74 Research - Phase II. . . . . . . . . . . . 74 The Objectives. . . . . . . . . . . . . 75 The Reportable Type Items . . . . . . . 75 Selection of Subject Matter Audits. . . 77 Selection of Sample Locations . . . . . 79 I The Research. . . . . . . . . . . . . . 87 Results of the Research . . . . . . . . 92 An Evaluation of Results. . . . . . . . 108 Management Action . . . . . . . . . . . 133 Additional Research Needed. . . . . . . 136 Footnotes. . . . . . . . . . . . . . . . . 137 IV. RESEARCH - PHASE III . . . . . . . . . . . . 138 The Objectives . . . . . . . . . . . . . . 138 The Reportable Type Items. . . . . . . . . 141 Selection of Subject Matter Audits . . . . 142 Selection of Sample Locations. . . . . . . 145 The Research . . . . . . . . . . . . . . . 156 Results of the Research. . . . . . . . . . 160 An Evaluation of Results 187 Overview. . . . . . . . . . . . . . . . 187 Objective One . . . . . . . . . . . . . 190 Objective Two . . . . . . . . . . . . . 200 Objective Three . . . . . . . . . . . . 207 Objective Four. . . . . . . . . . . . . 209 Objective Five. . . . . . . . . . . . . 211 Objective Six . . . . . . . . . . . . . 214 vi v- a..- ’(7 Chapter Page Footnotes. . . . . . . . . . . . . . . . . 217 V. USE OF THE INFORMATION DERIVED CURVES IN DETERMINING MANAGEMENT AREAS IN NEED OF AUDIT ATTENTION . . . . . . . . . . 218 The Objective. . . . . . . . . . . . . . . 218 Background . . . . . . . . . . . . . . . . 220 Hypothetical Procedure for Selecting Audit Areas . . . . . . . . . . . . . . . 221 An Evaluation. . . . . . . . . . . . . . . 225 Footnotes. . . . . . . . . . . . . . . . . 235 VI. USE OF THE INFORMATION DERIVED CURVES IN THE MANAGEMENT OF THE FIELD TEST OF NEW AUDIT PROGRAMS . . . . . . . . . . . 236 The Spectrum of Information Provided by a Field Test . . . . . . . . . . . . . 237 The Hypothetical Method. . . . . . . . . . 238 An Application to the Research Data. . . . 241 An Evaluation. . . . . . . . . . . . . . . 243 Footnotes. . . . . . . . . . . . . . . . . 247 VII. THE SEGMENTED AUDIT. . . . . . . . . . . . . 248 Background . . . . . . . . . . . . . . . . 249 The Segmented Audit. . . . . . . . . . . . 251 A Hypothetical Example . . . . . . . . . . 251 An Evaluation. . . . . . . . . . . . . . . 254 Footnotes. . . . . . . . . . . . . . . . . 257 VIII. CONCLUSION . . . . . . . . . . . . . . . . . 258 Hypothesis One . . . . . . . . . . . . . . 260 Hypothesis Two . . . . . . . . . . . . . . 263 Hypothesis Three . . . . . . . . . . . . . 264 Hypothesis Four. . . . . . . . . . . . . . 265 Summary. . . . . . . . . . . . . . . . . . 266 LIST OF REFERENCES. . . . . . . . . . . . . . . . . . 268 Ce A: Z. a: L. . O O C O .. . . . s 1. .7 ..a _ . ?. a uni Rh. I Th e A C A .~ 7,. ‘1‘ .3 s|\ 10. 11. 12. 13. 14. 15. LIST OF TABLES Subject Matter Audit Codes . . . . . . . Subject Matter Audits - Phase I. . . . . Research Results - Phase I . . . . . . . Subject Matter Audits — Phase II . . . . Factor Tests for Materiality . . . . . . Reportable Type Items - Phase II . . . . Cumulative Number of Reportable Type Items Found — Definition 1 . . . . . . . Cumulative Percentage of Reportable Type Items Found — Definition 1 . . . . . . . Cumulative Number of Reportable Type Items Found - Definition 2 . . . . . . . Cumulative Percentage of Reportable Type Items Found - Definition 2 . . . . . . . Cumulative Number of Reportable Type Items Found — Definition 3 . . . . . . . Cumulative Percentage of Reportable Type Items Found — Definition 3 . . . . . . . Reportable Type Items at 4 Percent or Less Locations . . . . . . . . . . . . . Percentage of Reported Information Available from Samples . . . . . . . . . Comparison of Data - Definition 1 vs 2 . viii Page 61 66 68 79 91 94 96 97 102 103 105 106 112 101 114 - I I . I V II I . . . — fl. . ... ._ . . . n . . u u... a; A . aw aw A u — . . _ 9‘ o. v». Q“. P «x» n a u A ' yl .55. n o a o o o o n o ‘o ‘ . L. pr. . .nu .a. p a a a .7 :u .p. ‘4 ~44 AU :3. . . . . . . . .1. . . e A. a L g . L Rd «cu .- L; 1...." Table Page 16. Reportable Type Items at 9 Percent or Less Locations . . . . . . . . . . . . . . . 116 17. Comparison of Data - Definition 2 vs 3 . . . 118 18. Table for Selecting the Number of Locations for Audit when the Objec- tive is to Detect Management Problems that Occur at Ten Percent or More of All Locations — Phase II Research. . . . . . 122 19. Actual Frequencies of Reportable Type Items . . . . . . . . . . . . . . . . . 126 20. Subject Matter Audits — Phase III. . . . . . 144 21. Factor Tests for Materiality — Phase III . . 158 22. Reportable Type Items — Phase III. . . . . . 160 23. Cumulative Number of Reportable Type Items Found - Judgmental Method. . . . . . . 163 24. Cumulative Percentage of Reportable Type Items Found — Random Method. . . . . . . . . 164 25. Data for Composite Information Derived Curves - Judgmental Method . . . . . . . . . 165 26. Cumulative Number of Reportable Type Items Found - Random Method. . . . . . . . . 167 27. Cumulative Percentage of Reportable Type Items Found - Random Method. . . . . . . . . 168 28. Data for Composite Information Derived Curves - Random Method . . . . . . . . . . . 169 29. Cumulative Percentage of Reportable Type Items Detected by Method by Location . . . . 173 30. Percentage of Reported Information Available From Samples . . . . . . . . . . . 178 ix .t .‘Uo h... a 4-. ch at. 0Q. aqd .RJ n(t Table 31. 32. 33' 34. 35. Page Cumulative Percentage of Reportable Type Items Found - Judgmental Method — Phase I and II . . . . . . . . . . . . . . 186 Average Cumulative Percentages of Information Derived Curves - Phase II and III . . . . . . . . . . . . . . 192 Table for Selecting the Number of Locations for Audit When the Objective is to Detect Management Problems that Occur at Ten Percent or More of All Locations - Phase I and II Research - 30 or More Reportable Type Items Per Audit. . . . . . . . . . . . . . . . . . 199 Table for Determining the Percentage of Reportable Type Items not Detected by Samples of Locations From 1 to 15. . . . . . 227 Results of a Seven Location Field Test of the Audit Program . . . . . . . . . . . . 242 . ~ ~..'_. .o‘¢.~ " l o (1') [\P ’14 (.1 CU LIST OF FIGURES Figure Page 1. Information Derived Curve. . . . . . . . . . 10 2. Cost of Sampling Curve . . . . . . . . . . . 11 3. Ratio of Cost to Information Derived . . . . 12 4. Air Force Audit Agency — Organization Chart. . . . . . . . . . . . . . . . . . . . 30 5. Air Force Audit Agency Regions . . . . . . . 32 6. Information Derived Curve. . . . . . . . . . 43 7. Cost of Sampling Curve . . . . . . . . . . . 44 8. Ratio of Cost to Information Derived . . . . 45 9. Construction of the Information Derived Curve. . . . . . . . . . . . . . . . 56 10. Information Derived Curve - Phase I. . . . . 69 11. Information Derived Curves - Definition 1 . . . . . . . . . . . . . . . . 99 12. Information Derived Curves — Definition 2 . . . . . . . . . . . . . . . . 104 13. Information Derived Curves — Definition 3 . . . . . . . . . . . . . . . . 107 14. Composite Information Derived Curves Formed From the Best, Worst, and Average Percentage of Data in Table 25 . . . 166 15. Composite Information Derived Curves Formed From the Best, Worst, and Average Percentage of Data in Table 28 xi 170 I I l I J a.» A,» are A» .C F; I. I. n. .. e 2 . . . . . .u. .p. . pr. nu. . ado . o o u o q. Figure Page 16. Cost of Sampling Curve - Phase II. . . . . . 182 17. Cost of Sampling and Information Curve - II I I I I I I I I I I I I I I I I I 183 18. Cost of Sampling and Information Curve - II I I I I I I I I I I I I I I I I I 185 19. Composite Information Derived Curves Resulting From Both Judgmental and Random Selection Methods . . . . . . . . . . 204 20. Composite Information Derived Curves for the Six Normal Audits. . . . . . . . . . 206 ~-. ... ‘ . -“;';“ :Aa ‘ v ~..'.~‘ dk‘ Hv.‘ c F. i.h‘a.-~, '. I.. . ’I “:C: RI v- u.~.‘ ‘HA U . ~K .v- n~ - P. " H .5. "~“4-7““.‘ ' A C‘A“Vac.~ ‘5‘. \ ‘VVI \. .x ‘ - '- ‘A‘I. ‘ "~ . - ~.‘§\‘~‘ ‘ 1 \ ‘1. ‘y ~ . I“.‘:‘ P I .. p ~‘E-‘hc' s r V‘ ~ .4 F . :fiht‘ 'I_ F‘. . ‘J." 4 E Q' ‘ ‘ ”a" . I l t I t w~ ." ”—4- vww———- CHAPTER I INTRODUCTION Time changes many things. Management methods con— sidered acceptable in former years are, under present con- ditions, often found inadequate. New research and innova— tions have thrown the old methods out of perspective. The same is true for internal auditing. This research proposes new considerations specifically for the management of United States Air Force (USAF) internal audit resources. These proposals, however, should have general applicability to other Department of Defense (DOD) and private multi-location organizations Operating under similar conditions. The purpose of this chapter is to introduce the reader to the research. Accordingly, the first two sec- tions discuss general DOD internal audit responsibilities and the nature of the internal audit product. The third section, called The Economics of Sampling, begins to focus more sharply on the central area of interest which is iden— tified specifically in the following section. The 1 i 1 c 3 2 background and general approach that is taken in the research is followed by a succinct statement of the pur— pose of the research, identification of the research para- meters, and a listing of the four basic hypotheses of the study. The first chapter concludes with a brief overview of the remaining chapters to provide the reader with a roadmap to lead him through the balance of the discussion. DOD Internal Audit Agency Responsibility Although many Americans consider themselves well acquainted with their Department of Defense Armed Services, relatively few, even among those who have served within those Services, are aware of the full range of its activities. The national significance of Armed Services activi~ ties is such that their importance really cannot be mea- sured in common business terms, but they do involve aspects that are susceptible to business measurement. Some of these activities are very large even when compared with the industrial giants of our nation. For example, Air Force assets alone total approximately 85 billion dollars. This is greater than the combined assets of the t0p 50 manufacturing firms of this nation.1 The support necessary for maintaining Department of Defense readiness includes large and varied activities ..C C I In urfl A“ ' .".A\l J“- . :F'V’Zfle ‘.-~. ud'v I r.l 1 u p . I l I I . . . . .2 I n . fly I ‘¢ ‘ I s g$s v- u: a» . v. I; . I .re . I I .. LL .. 5% .‘ ~: 3 :I .f.» r‘ C. . A I .c s. . a 1.. . L. G. M” .nt. .9 n... .u .3 .a. . . «a 5 hr w. . . 2.. . I a. e m... I. . . a: . . u. L" . .. m7 .1 _. an I “In :‘ I ,v DP II!»- II~ .rufl .rI 3 found only in combinations of major industrial and commer— cial enterprises. The range of diversified activities embraces such dissimilar fields as warehousing, transporta— tion, marketing, and communications. This exceptional diversity adds to the problems of management, problems which become the direct concern of the DOD internal audit agencies. These internal audit agencies are charged with the purpose of providing . . . "those responsible for manage— ment . . . with an independent, objective, and constructive evaluation of the effectiveness and efficiency with which managerial responsibilities (including financial, Oper— ational, and support activities) are being carried out."2 To fulfill this obligation, these agencies must continually seek better ways to produce information and advice and to keep pace with management's needs. Over the past several years the need to find more efficient internal audit methods has become increasingly serious. DOD internal audit staffs have been reduced relative to the size of the internal audit workload. This pattern of a diminishing auditor staff concurrent with an expanding internal audit workload has become a matter of serious concern to the Air Force Audit Agency, the primary organization of interest in this dissertation. In the , A i- 'A': [I e UVOUJIOO‘ . . . . ‘nd 9’ V I e. a ...'u ... I» y—IIHU ‘.-_ a p I. ‘ . r ' - .I , ‘ VA.‘ A .-‘ II _‘ I “. I“ .‘ iv '9 ”V . . tw~ .: .., “‘VI-‘ . . . .. F , ~ V ME 5-~ ‘ ""‘.o . . . L 1 y “Ha [‘4 V s3“ “1 t ~E>~ \L‘Q A - ‘N gs \ 4 three years ended JUne 30, 1971, Air Force Audit Agency personnel authorizations declined approximately 22 percent, and there is no reason to expect a reversal in this trend. During this same period Air Force resources grew both in cumulative inventory and annual consumption. There has been no significant reduction in the number of major loca— tions at which the Air Force operates. Recognizing a problem, serious as it may be, is only a first step toward finding and adopting a solution to the problem. Recommended changes in audit methods should meet at least the following very general criteria. They should be: 1. based as completely as possible on objective rather than subjective considerations. 2. designed to assist the audit agency to meet the objectives specified by governing Depart- ment of Defense and Air Force instructions, regulations and policies. 3. effective in providing operating managers the type of information that will help them do a better job of managing. One cannot evaluate a proposal for a new audit method in terms of these criteria without a thorough understanding of t:he purpose and the product of internal audit agencies. [I 'n. . .a ‘ ‘Ifl aI-Q‘R ‘ $1.}: boson. . . ‘ .. 7 ‘FA r‘r .E.... w . 4 64‘ 4..., ~ 'V".~u.. R r IIJII". I - . ‘2'.“ ‘ PO ad.» .l“| 2.4,. Av" ‘OU‘k K“: . I I.' I” : III 5“e “ . ‘RP “a ‘ .. ~C59: :,.v “I ‘ ; ‘ 'Q I‘ ‘I. V “A: .. t) I 'v W 5 As stated above, the purpose of internal audit agencies is to provide an evaluation of the effectiveness and efficiency with which managerial responsibilities are being carried out. Narrowing attention more specifically on Air Force internal audit, how is this purpose achieved? Stated another way, what is the Air Force internal audit product? The Internal Audit Product .__________________________ It is possible to justify the existence of Air Force internal auditing by pointing to the requirements for audit created by law. However, this is totally artificial, for the auditing organization exists because a need exists for auditing services. In terms stripped of formal direc— tive verbosity, Air Force internal auditing satisfies a need for information of two types: 1. Information for management decision-making. 2. Information on management performance. It is this information that is the product of the internal audit effort. The method of deriving this internal audit product is a central focus of this research. Evaluation of method is in turn dependent upon an understanding of the use made of the information. Both the method of deriving information . nap... Cr:r an“). “0-- C n H...‘ pke - that 5‘: \n . ‘F‘ A ‘5 5— II A ‘fiot lb . . Q - air. .. Vat-.d ‘,4_ "v‘v-.... -— . ‘ Ve‘a..-...4-_ . N ““!‘ «C “V .C u... n‘ I, . ’ ,0" “VP..- ‘5‘.“ w J. ‘G'C'Nc, ‘ o "“'-b. >4 1 ‘o' C V "w. "~. L ‘Q :::‘~.. . v‘:. ‘v ‘A _. a“ “a $. I ~fl’ \‘c or A, “Jrh\‘ 1.: . ice-hi“, \- x ‘\ s‘fip J. h, H ——-x ,- ___ v—t ——. ‘ 6 and the use of information are discussed at length in subse— quent chapters. For purposes of acquaintaing the reader with the area of research, however, it is probably advis— able at this point to touch briefly upon the use of in— ternal audit information in the context of the military organizational structure. It makes some sense to View the military base com- mander as in a position analogous to that of the decentral- ized profit manager in the commercial enterprise. The dif- ference is simply that the base commander is a decentralized resource manager concerned with the optimum utilization of resources rather than with profit generation. The base commander needs specific information on resource utilization in order to improve his managerial effectiveness. He needs information to assist him in making decisions. Top management (management above base level) is interested in information that can be used to make deci— sions, but it is also interested in information which will assist in measuring the performance of the base commander. Thus, two types of information are needed from an audit, information £25 management and about management. The experienced auditor realizes that different decisions are made by the base commander and top manage— ment, and different information is needed by each of these ci’c'”: whats-QUAD- l I C p'Ahu I. v 50:! ‘Vll‘ o ‘ a . .. ”e: “N I - U a....~: .. ‘ n*—f-;pn iv. V»Iu.u¢..ut‘ u p 33“. ,. r: .V“"vn.~ 5.; I“ 1 f D! F__——i _ 7 echelons. However, the information needed at base level is frequently useful also to top management. Audit findings dealing with base performance not only assist the base commander in his mission, but also inform top management of system weaknesses and of potential problem areas at other locations. A very important point emerges from this dual infor— mation need. The base commander may wish to have quantita- tive statements about any discrepancies dealing with the resources under his command; top management is much less interested in specific quantitative inferences about base level problems. It is usually sufficient for them to know that a significant problem area exists.3 Hence, with regard to audit programs applied at a given base, the auditor may wish to use statistically determined and applied sampling plans. This is because the base commander desires to have information which not only reveals that a problem exists, but additional data permitting measurements of how material the problem is and how its materiality is changing over time. However, tOp management does not need quantitative measurements of difficulties nearly as frequently as does the base commander. Top management needs to know what the significant problems are, what the likely causes of the problems are, and what . l!‘"“’ a no;- un.=»-o ' ~ film-:5! AP ...u‘arou I A ‘ “yfiv'qq ‘v dav'5U .A n I‘n‘ A . l <~H~fi' ”‘u—uv. -v.. 3r" “:"n- in "no" .4 I a: A ;~;_ u a h : d . . Tilazva' $.‘,._ ‘er1‘ Sue 2?de 1 5a ‘ I. , ~ - ... - "“‘e...£ " a I. M- .M‘ v .. ‘d“ 2 4 ~‘ “ ‘,.:‘. - I ‘0. ‘ 1“.S k; v‘ . i‘ ‘c . ‘ v s.’ “‘1 A ‘ «5 CI“- \ «M‘: n ‘A WI : V‘A 1“ ‘N . “~42,“ -<.,‘z € ‘9‘ “e C.‘ \ ti: H " ..‘ 0.x.1‘ 7" INHI 8 corrective action in the system is needed to remedy the situation.4 Each internal audit accomplished is not required to provide information to both base level and tOp levels of management. Requests for audit service can originate at any management level. It is convenient to think of audits as originating from two basic levels--base level and top management level. Audits requested by and confined to a single loca— tion, such as a base, are primarily designed to provide answers to specific questions of particular concern to management at that location. These audits will be referred to here as local audits. Audits requested by or accom- plished for top level management must be applied at a suf— ficient number of locations within the multi-location organization to provide the information desired. These audits will be referred to as centrally directed audits. It is with the centrally directed audits that this research is primarily concerned. Each of the armed services (Air Force, Army, Navy), for audit purposes, can be viewed as a multi-location organization. Auditable activities within the Air Force, for example, may be operated at approximately 180 locations. Until recently it was the policy to apply centrally V ,.v.~u-n smock .Cu A.” V” owl d a .59: _ fit»..— ‘:_V" Ic.‘ n—u. “._ 1T: . 'CE'V‘H': 0,. a.“ b». ‘1’ «‘\‘ . 9 directed audits concerning an activity at all of the loca- tions at which that activity existed. Efficient management of internal audit organizations requires the application of audit resources only to the ex- tent necessary to derive the type of information required. An all—location application of an audit serves each base manager, but will likely provide more information than is needed to identify a problem area for top management. An audit completed at a sample of locations can provide t0p management adequate information and conserve the amount of audit effort that is centrally directed, but will not pro— vide specific information to base managers not included in the selection. Accepting the fact that centrally directed audits are accomplished by Air Force internal auditors, one is then free to consider methods of managing such audits. One popular method of generating audit information frequently discussed in the audit literature is to employ some type of sampling plan. The Economics of Sampling By applying these audits at a sample rather than a universe of locations the auditor must assume that such procedures have advantages. But what are they? Do .. :aN ~,." 5". v“- at C l'l') . the p... F” C F .ulvd. v.5 "“ht‘ac nun..—-\. . . _ n_A-.'-,. ”h n * lit-o. ‘u‘g. I" ‘ 10 aggregate gains exceed losses? Since information is the product of the audit, a theoretical study of auditing at a sample of locations indicates that a sampling method is at least economically desirable. Assume that a management system is in operation at 180 locations (or bases) worldwide. One can hypothesize that the curve of information derived from the application of an audit to test this system can be drawn as follows: FIGURE 1 INFORMATION DERIVED CURVE f——~—————_fi—_________.,. Cumulative Amount of Information Derived ' 0 Sample size in locations 180 This figure simply indicates that the first few I locations in any sample will make relatively large contri- : butions to the amount of information derived from the audit ’ tests. As the number of locations included in the sample increases, the total amount of information derived may increase, but at a decreasing rate. From data provided by past statistical records, the curve of the cost for the development and applying the . “IF": : 3.....- - ‘- 'r ..., C“ u ’ ‘ A.» .,.‘.C‘F‘ 1‘ h-“»‘ Hat. ‘5'- p. ~V~~ ‘ ‘t‘. J ‘ i ‘ c .01“ . ~.‘¥‘\-‘ . “A'a“" iv“: ‘.‘ W.” .\ “a .~“IF‘-N:‘ F. v- "s 5.“ ,(im ”y" _J ll audits at locations can be drawn somewhat as follows: FIGURE 2 COST OF SAMPLING CURVE Cost of Sampling Sample size in locations 180 The curve does not start at the axis intersection because of fixed costs in the development of the audit program. Cost generated by application at each location is con— sidered relatively stable, increasing total costs at a constant rate as the number of locations in the sample is increased. One can now hypothesize that the two curves com— bined would provide the following figure: I I . . t i v. ‘4. 6» ~ . .0 n p . .. FM n m In n\u . a C .. i r. and Lu . . .n . ~ A u .. .i v . Ca . h . . uu C» vs .3 L .. . a) a... as a u n o .9 .. a a v ‘MFN .\ HE n t n . A d , F... Iu" 1 a e Cb 5C .9. . . ‘ n ANV . Q a a . .2. r; r. a. v § \ .\ L as k is a: I. a a .. 12 FIGURE 3 RATIO OF COST TO INFORMATION DERIVED Cost of Sampling/ Cumulative Amount of Information Derived L..-— 0 "n" 180 Sample size in locations This figure indicates that as the number of loca— tions in the sample is increased, costs increase at a constant rate. Information derived increases, but at a decreasing rate. To maximize the amount of information for the dollars spent, the size of the sample should be where the two curves are farthest apart, or at point "n." The auditor must not, however, believe that sample size "n" is a magic solution. While undoubtedly cost is a vital consideration confronting the auditor who has a test to perform, it should not be an overriding consideration. After all, the lowest cost of auditing is attained by not performing the test at all: An important principle derived from this graph (if indeed the hypothesized information derived curve is lr- I’ -. :mm'v, I‘V‘QH' - . a by ‘V g I: e a fin out . ~.. A. .AR‘ v. Ak»u~- V. V" L... . . H5 a.“ Vs h“ .1 "“L: :4 N an In . c~ - ~ \ I!“ a, F. A J . 5 Ca ‘\ l3 accurately portrayed) is that costs will be avoided as long as the audit program is applied at less than the population of locations. Further, the sample size can often be cut with very little loss of management information. The Research Focus The figures in the preceding section are drawn using theoretical arguments. An important element within the figures is the shape of the "information derived curve." Whereas the curve for actual costs of auditing is rather easily determined, determining the shape of the "information derived curve" is a more difficult task. This involves not only defining items to be measured, the units 015 information, but a determination of how to measure such units as well . Assuming that the "information derived curve" can be eStablished, the important question becomes does knowl— edge of this curve aid in the management of internal audit resOurces? That is, does knowledge of this curve suggest m"lt‘i'lods for more economically abstracting audit information While maintaining acceptable levels of timeliness, quantity 0f information, quality of information, and cost. The primary focus in this research was upon the determination of the shape of the " information derived . v. . v. \ .5 C. i vf. «V AC :. ... . .~« Va A b 7* C. . a a . r. a: .. P: .F» .p.. v a 2‘ w .. >1“ ..~ ,. r: t» :k a...» .. .. l . >. h. vi 1. ‘3 .“ ~74; .... ... o L. C. v“ .. v. .n.. C. V. Akv u" a\. flu. .Au 2. A» nu Va «.5. . a :n . . ha is .u v a . . . .I. 2. C. .1 C. m: a . 14 c\xrwree" and upon the appropriate arguments that support its use- Research Background and Approach My initial interest in the research area was ltinti].ed as early as 1966—67 when I was assigned to the head— quarters of the Air Force Audit Agency, Norton Air Force .Basee, California, as a member of the General Research Divi- sixori- I was one of three individuals detached from routine audij.t: work and given the task of conducting conceptual re- seaarczh on audit methodology. The importance attached to Sucla research is reflected in the words of the then Auditor General of the Air Force, Major General Don Coupland: We constantly seek self-improvement. To aid iJI improving our audit methods, we established Ein Auditor General Research Division. Included VVithin this division are personnel assigned to Sleneral research . . . The number of personnel 5.8 small . . . but the impact of their products Tlas far reaching effects on the organization.5 At that time, audits designed to provide informa— tlorl about an activity to top levels of management were beil'lg'applied at all of the locations at which that activity eXiSted. But as stated earlier, one could reasonably argue that while an all—location application of an audit serves each base commander, it will likely provide more informa- tion than is needed to identify a problem area for tOp . ~ .. .n_fl":a CV ”Linneao-ba . u~~. “a.” v- " a. _ .........,._ _ I‘ v o . -- n'j-v ' "‘s.._?'“ v (I! p- - .rr""~‘— a I-ov“... ' J's . a 1 . 4.. N,- ‘ ‘T‘V-r‘ ‘U p. .4 ‘ .. at). ‘s‘v:\- ‘ k D. LJV" “at-’- I ‘ r . w" ‘ngi " 15 management. This reasoning, together with the reality of a con— sistent reduction in audit manpower brings one to the core of the problem. How can the Air Force Audit Agency main— tain or even expand its service in the face of diminishing numbers of auditors? The seriousness of the problem is reflected time and again in the statements of top level managers (for examples of such statements see pages 38 through 40). My interest was, in turn, motivated by the inqxortance attached to this area by the Air Force Audit Agency. My earliest phase of the research, which is dis- cussed in Chapter III, included the detailed analysis of three separate audits to determine how quickly audit infor- mation is accumulated as additional audit locations are inclnlded in the sample of locations. For each audit, infoxmmtion produced by 10, 20, 30, 60, and the universe 0f l<>cations was evaluated. Although analysis of but threes audits did not provide conclusive evidence as to the “unbfiar of locations that should be included, it was clearly eViGEnt that some number less than the universe would be aPPI'OPriate. Based upon this initial research, the Air Force Audit Agency decided to apply certain audits at 60 locat ions . . - .AF v” "V. on... \ ' "‘ Pav- ‘E::a~~... I I . .‘ 1c '2...~‘. ‘~ ' Q .A"V ~.-. I... ."“ 4.4"; . ‘roav '- ‘cogv. ‘3‘: q . H... r- h. H 1;! ‘ .1 «“ 16 But in my opinion the Air Force Audit Agency did not fully exploit the initial research findings. A basic reason was undoubtedly the lack of more comprehensive research to provide stronger evidence of the probable result of changing audit methods. To provide this evidence my next phase of the research was initiated. In this phase of the research which is elaborated upon in Chapter III, the data from four audits originally applied at from 55 to 152 locations were reviewed. Particular attention was paid to the amount of information that was in evidence at 10 or less locations selected judgmentally from the population of locations at Which the audit was applied. This research indicated that the “information derived curve" (refer to Figure 1, page 10) f°r 'the small number of locations has a distinctive shape that; suggests several audit method improvements, not only indetermining the number of audit sites to include in an andjtt, but for the detection of areas in need of audit and in tlne development of the necessary audit programs. Subse— querlt to this research the Air Force Audit Agency again I'e‘ni-sed their audit methods in line with the research, but it Vvas evident that additional research was needed to shal‘pen the focus of these new methods. After I became a doctoral student in mid-1970 the ”It. -. ‘§ .‘V.‘ ‘u. ‘C ‘ ‘tu..-~~ “'u‘ ‘I F H ‘ in“ ‘3‘ 0‘.‘ ‘ 17 research to date was documented and a proposal prepared to perform additional research into this important management area of the Air Force Audit Agency. Although the research would be specifically within the Air Force Audit Agency records, the results should be of general interest to any multi—location organization at which internal audits are made. With the support of the Air Force Audit Agency and approval from my major professor, the current research was undertaken. As detailed in Chapter IV, the shape of the "infor— mation derived curve" was determined within reasonable liJnits using both judgmental and random selection tech— niques. More specifically, nine subject matter audits were Selected for detailed analysis. Nine is a sufficient number t0 indicate the shape of the "information derived curve" and 'to show that there is a significant statistical differ— enci! between the judgment method and the random selection mathod. Considering first the judgmental method, for each °f tihe nine subject matter audits, Air Force internal audit managers were asked to select 15 locations which they would have: recommended had the audit been restricted. These manag‘ers were interviewed to determine the basis for the Belatction, ie., size of base, quality of audit staff at 18 that location, representativeness of the auditable system at that 10cation, etc. These locations were arrayed by the audit managers in an order of preference from 1-15. The supporting records for the subject matter audit were reviewed to identify the reportable type items. Reportable type items for this research include those items which were found to be present at ten percent or more of the pOpulation of locations at which the complete subject matter audit was applied. Each individual audit input from each location (in order of preference from 1 to 15) was analyzed for each of the nine subject matter audits. Reportable type items were identified and a cumulative "information derived curve” was constructed. The nine cumulative curves were plotted, with analysis made at the 5, 10 and 15 location points. For the random selection method, the above research was dirplicated with one exception. The 15 individual loca- tions for each subject matter audit were selected randomly “Sing' random number tables. The judgmentally and randomly derived curves were the“ <=ompared and appropriate conclusions drawn. Management uses of the information derived curve were 15hen developed. The discussed uses consider such crlteria as risk of nondetection of significant management vIIIIVF"'F'--------'-"'-'---'-""--_—--'-""'_—-‘_—____——____—-—'—f 19 problems, the time required to achieve the subject matter report of audit, the cost in terms of audit hours, and the quantity and quality of the resulting management informa— tion. As elaborated upon in subsequent chapters, risk is a function of the "information derived curves." Timeliness is a function of the "reasonable" time requirements from inception of the audit to delivery of the final audit product. Costs are evaluated in terms of audit hour costs suggested for planning purposes by audit managers. Quantity of information, like risk, is a function of the "information derived curve". Finally, quality of information is a func— tion of the materiality of the reportable type items found at the sample. Materiality, in turn, is judged by a pre- determnined measure. For example, to judge that the sample findjdlg is as material as the finding from the population, the elrror rate must be equal to or greater than the error rate :Erom the population of locations in the complete audit and, 1:0 ensure the information is not clustered at a single l°catidon, at least two of 5, three of 10, or four of 15 l°catiOns must indicate the problem. It is argued that consideration of the "information deriVed curve" suggests a number of possibilities for interrlal audit cost savings. At times these cost savings may be at the expense of one of the other criteria we wish H- v;~rO:‘ .y .Lu.n~|lo , , . vnbp'w'~§- ~ r' athsn-au-g .. Mp g 50. n- I” '1 m f‘v‘e “u-..- r ,. ‘“ It‘th . Tn. .~ 1 (I) 20 to maintain, such as timeliness, quantity, or quality of information. But what is of importance, these likely costs are now quantifiable. Judgments to expand or reduce the number of individual locations included in a subject matter audit can be based on known costs rather than assumed costs. Specific discussion of these points are elaborated upon in subsequent chapters. P_urP°_se Stated succinctly, the purpose of this research was to determine the actual shape of the "information derived curve" and to demonstrate by apprOpriate argument that knowledge of the shape can provide an effective and eco— nomical management tool----a management tool that has not fornually been considered. Criteria used in judging the effectiveness and economy of the proposals included timeli— ness, quantity, quality, and cost of information. Research Parameters This research was restricted to those audits accom- plislled by the Air Force Audit Agency. Although the inves- tigatIive work was limited to Air Force audits, it is antici- Pated that the results will be of interest to other multi- locat ion organizations . A further limitation concerned the age of the 21 information systems reviewed by audit. Since the bulk of the Air Force internal auditors work is concerned with what might be called "established systems," only audits com— pleted on systems that have been in operation for at least a year were selected for detailed analysis. Finally, no attempt was made to determine if the types of information provided to each management level as discussed earlier (pages 5—9) are the types of information that should be provided. The rationale for the types of information that are provided reflects the consensus de— rived from discussions with audit managers of the Air Force Audit Agency. Opinions of operating managers were not solicited. The present research was accomplished with the aSSumLption that information that has been provided to oper- ating‘managers in the past will continue to be provided in the :future. A determination of what information should be PrOVjuded to each level of management is beyond the scope of this research. Four Basic Hypotheses The following four basic hypotheses are advanced: l. The "information derived curve" is uniquely shaped. That is, the first few locations in any sample will make relatively large 22 contributions to the total amount of infor- mation that is derived from the subject matter audit. As the number of locations included in the sample increases, the total amount of information derived may increase, but at a decreasing rate. The "information derived curve" deve10ped from a judgmental selection of individual audit locations is significantly superior (a=.10) to such curves developed from a ran— dom selection of individual audit locations. There is a minimum number or at least a range of locations for a given audit for which audit cost can be considered reasonable. Beyond that range the incremental cost of audit tends to exceed the value of the incremental infor- mation. The "information derived curve" can provide an internal audit management tool, the use of which can result in more efficient use of internal audit resources. 23 General Organization This concluding section provides the reader a road— map to lead him through the balance of the discussion to in: lude insight into both structure and content of each chapter. Chapter II provides a brief discussion of the Air Force Audit Agency organizational structure and of its Product -- the audit report. The purpose is to illustrate the environment in which the research is accomplished and to aid the reader in drawing similarities between this mL‘ll‘ti-location audit agency and any other multi-location organization of interest. Chapter III presents the initial effort to define the individual units of information that compose the "infor— mat ion derived curve" and the results of preliminary re- Sea~J:‘<'.'h to determine the shape of the curve. Changes initiated in audit methods by the Air Force Audit Agency subsequent to this early research are reviewed briefly. Chapter IV provides a comprehensive discussion of the research to determine the actual shape of the "infor— mat ion derived curve." Nine audits, each of which was apPlied at 40 or more locations, are analyzed. Both 311dgmental and random selection techniques are used to GeVelop " information derived curves" for a limited number 24 of locations selected from each audit. The judgmental and random selection techniques are compared and evaluated. Chapters V, VI, and VII explore additional methods by which knowledge of the "information derived curve" can be used for more economically abstracting audit information. More specifically, Chapter V discusses application of the knowledge to determining areas in need of audit attention, Chapter VI to the audit program field test situation, and Chapter VII to the segmented audit. Chapter VIII concludes with a discussion of the initial hypotheses of the research. 25 FOOTNOTES lMajor General Don Coupland, "Current Management —— An JAudit Challenge," The Armed Forces Comptroller, April, 1967, p. 23. ]Department of Defense Directive No. 7600.2, Department <3f Defense Audit Policies. Washington, D.C.: Depart- Inent of Defense, 19 August 1965. 'This was the general consensus of Air Force audit Inanagers charged with the responsibility of briefing top level Air Force managers on the results of internal audits. Based partly upon this belief of what infor— Ination managers need, a decision was made within the .Air Force Audit Agency to apply certain audits at less than the population of locations. No attempt is made in this research to evaluate user needs for this infor- Ination. See note 3. Ibid., p. 24. ‘Ii' ~ 4 . ~ :N“ .R? ‘uu -.J‘ '5‘"?‘ .‘~ v._ e ‘R- '5 5. ‘ F... "a- "“v‘ . . ~t E I 1am.“ ~ ~ C.» ‘ “CV‘I CHAPTER II THE AIR FORCE INTERNAL AUDIT ENVIRONMENTl When supply lines snarl, cash balances collapse, or any phase of management malfunctions, the Air Force auditor is the man on the spot to uncover the trouble and report to the level that can take the necessary corrective action. He is stationed as close by as possible to where he can offer the greatest service. He lives and works where the Air Force lives and works. The purpose of this chapter is to provide the reader a glimpse of the environment in which the research was accomplished. By understanding that environment, it may be easier to ascertain similarities and differences bet“Ween operating methods of the Air Force Audit Agency and other multi-location organizations of interest. The first Se‘i-‘tion traces the history of the organization. This is fc>lJ«awed by a brief discussion of the organizational head— <1uaI‘ters and the separate line and staff elements. The Q1llapter concludes with a capsulized review of the audit app"finch and of the final product, the audit report. 26 rare SUV-A“. . ‘ I ’F‘np U “nau‘ I . ‘ Use: _ 'id. ~ .,._A ‘ u-g v-c '~L,"‘_‘ . ' . AV,‘ ‘ \ u v- . I VI “a. .V.‘ . a A. .“D r J i..~ 0 1" (II In (J I O“ 27 When Air Force Audit Was Born The United States Air Force was established as a separate department by the National Security Act of 1947-- enaing 40 years of Air Force association with the United States Army. The Army Audit Agency continued to perform these 2audit function until, by agreement between the Secre- taries of the Army and the Air Force, the Air Force assumed re ssxpmonsibility for its own audit function. The respon— sifkozihlity was assumed on 1 July 1948 when the United States ALT: Force (USAF) Auditor General was established as a part Of? 1:13e Comptroller. Today, all Air Force auditors, regard- leassgs of where they are located, are part of the Comptroller 0i? tZIne Air Force and are directly responsibile to the USAF ALldnltor General. The responsibility for internal audit was specifi- cail—JLjy set forth in the National Security Act Amendments of Axlgillsst 1949. As defined by the DOD, internal audit is the indieP—Jpendent review and evaluation of the effectiveness and efficiency with which managerial responsibilities are being carried out. It is an independent appraisal of financial, operational, and support activities as a basis for protec- tive and constructive service to management. It is a service to all levels of management, from the smallest based-situated organization up through major r“. . - ”An-2v fl " 5V... I‘nd v I r, 27.9 T. (l ( -a.. -u-3 v-\--. . . \ -v I‘.~‘ ‘ fir Mun- ‘ " ".c-‘V'v - u‘ c ‘.~ ‘K. a e.. 28 command to the tOp levels of the Air Force. To maintain the necessary independence of review, the Auditor General is authorized to use direct lines of communication with the A3 sistant Secretary of the Air Force for Financial Manage- ment. Air Force Audit Agency Headquarters—Line and Staff At the Norton Air Force Base headquarters, The Auditor General of the Air Force and his staff of super— V isory and support personnel furnish guidance to approxi— Ina-1:e1y 1100 professional auditors located at major Air Force installations throughout the world. Having resident auditors and their staffs located at major Air Force installations around the world allows alrnost immediate response to Air Force-wide problems. By directing audit effort from the headquarters at Norton Air Ferce Base, California, the Auditor General can gather in- formation on any type of activity at any or all Air Force has es. Consolidated reports are prepared to communicate audit results to the highest levels of Air Force management. Air Force auditors, increasingly management— Oriented, are not confined to reporting the discrepancies they uncover but dig into problem causes in order to furnish rue-nagement with recommendations leading to greater 0' Ft- Ana': v‘ guy-II». - :: u-y-Au_~ U on ‘Qi/on‘| U .u~.' ‘ \ 9 pp... ‘6 Ivan D I #0 5,. h V- s” v ' Q : "FLA ‘O-i..\,u a .- ‘ s :.A‘. ‘ ea.-. .... .. ‘.'S A I .V I . EVA-‘ F- c ”Oyu‘. . .“ . :‘n.l.. ~§~‘k-b C3~u. .“v ‘- Q 29 Operational efficiencies. Auditors not only determine what is wrong, but also why a method, directive, or procedure has failed, the impact it may have, and what might possibly be done to correct it. Early in July 1968, the functional responsibilities OE the organizational elements at the headquarters were re- aLigned to establish an unequivocal distinction between staff and line functions. The purpose of this realignment was to provide for a more homogeneous grouping, and to create staff elements which have mutually exclusive respon— Sibilities. Figure 4 gives a concise picture of the present oI'gamization. Beginning at the tOp of the organization chart, the office of The Auditor General is staffed by a Major General, his civilian deputy, and his executive officer. The assigned general officer fills a tri-position as Assistant comptroller for Audit (an Air Force Headquarters staff p0Sition) , as The Auditor General, and as Commander-~Air Force Audit Agency. Because of the distance separating the Auditor General headquarters and Washington, D.C., an Associate Auditor General and staff maintain permanent offices in the Pentagon. These personnel can contact the Air Staff and other Pentagon-based officials in direct and tl“\ely communication for The Auditor General. The Associate A m 0 momo¢ m ow < a owes. 855. 85...; moms? 855. q. \E1‘ _ «H mm Hm mN mawumzm mawummm — fl — wamumzm mpfianmofi>uwm cowufimfioco< moflumfiwoq cow ouom tachoumwmm— flamuudomh — :uoummm_ OSHA I mCOHMKWfiQ UCCfime mmmumllu IIIII lll'll1lllll I'lllll... uuoamsm mwow>umm mamam wcowumuwdo mo Hmoowmmmwoum mo mo mumuouowufin mo quHOuoouaa oumuouowufin mumuouowufia .maoaumuoa wafiumuoao ouoa no one oma>uwn=m momo< waome _ F _ mfimhamq< uewz F Hmuwamu Houfip=< oumfioommrus is divided into regions, as shown in Figure 5. Each reggiJDn is established on the basis of geographical bound- aries for the purpose of supervising, assisting, and evalu— atiJngg the work of the Auditor General Representative or . . . . . 2 Res :Ldent Offices Within that region. FIGURE 5 AIR FORCE AUDIT AGENCY REGIONS vb AFAA Hqs Norton AFB I Central Audit Region Eastern Audit Region Region 4 “‘Vnav' ‘Q§“ AJ 1 .w‘ an. ~1 E .34 r t . 33 Supervisory personnel of the Western Region are located at Norton Air Force Base. Supervisory personnel of the Central Region are located at Carswell Air Force Base, Texas; those of the Eastern Region are located at Langley Air Force Base, Virginia; and those of the Euro- pean Region are located at Lindsey Air Station, Germany. A typical Auditor General Representative Office is composed of five to seven auditors, headed by a resident auditor as the office chief. These auditors are always tenants on the base with which identified, independent of all local commanders, and are, regardless of location, directly responsible to The Auditor General through their immediate supervisors. A resident auditor located at a major command head- quarters has a dual responsibility-—he is the Auditor General Representative to the major command as well as the base resident auditor . The Service—Wide Systems Division The ad hoc directing of all or a selected number of regional resident offices into specialized areas of interest to Air Force management is the responsibility of the Ser- vice-Wide Systems Division. The division has the capa‘ bllity to design an audit program that can evaluate a | S Di ”MW”: M- O 'H 9.. ~ ~.— " P u ..a -- r' \A‘ V‘ ~ s ‘u. II «we ~ 4 . RN» -‘ a a . pH.» » ,. .r. . f. Re n1. . . At Q» r t r? a . PM E ,. . 2. ,. . .. . L . at . a s: . t .ws frn a; a: u: A u pt if. . . . . r» as .2 . t he .J nu. u. vs u; ... at ..r ..v n-.. ..~ C. r .1 .3 .rb 2‘ .H‘ C. .: a: 1‘ «\~ U" L . r». x u 34 management system vertically through all levels of manage— ment as well as horizontally across numerous organizations on the same level. Integrated audits performed on selected management systems are designed to focus on not only man— agement at several levels, but also the system's inter- and int ra-relationships . This division provides a tool for making in—depth audit analysis of management areas, analyses which are much broader in sc0pe than would be possible from the effort of a single team of auditors at a given Air Force base. Such audits are called central]; directed audits since they re- sult from programs develOped by this division and sent to selected field offices for accomplishment. The Service-Wide Systems Division is also respon— sible for three unique representative offices. One is located at each of the following Air Force centers: Data SYStems Design Center, Accounting and Finance Center, and Military Personnel Center. The offices are staffed with technicians who are engaged in auditing automatic data proceSsing (ADP) systems under deve10pment and the other management systems under the responsibility of each of these centers . . t..~ h...‘ . '2'- a hunk up. ~‘ '1 in“ O ‘1! (n I!) p "In” 4‘ ‘v, “I r . 'w h 35 Logistics-~The AMA Approach The Logistic Systems Division is located at Wright- pat:t:erson Air Force Base, Ohio. Like the regions, the Log;j_stic Systems Division has full-time auditors assigned. Thj_s; special audit division was established to service a siJiggle major command because of the high value of resources hariéiled by the Air Force Logistics Command (AFLC) and the tuniqfueness of the management systems employed. Auditors witiri specialized training and experience in logistics man the: :resident offices at the Air Materiel Areas (AMAs) to assist management in uncovering and correcting difficulties in ‘tJne world-wide logistics functions. As with other sys— tenl (design centers, a large group of auditors are in resi— denczee with AFLC's Advanced Logistics Systems Center. Acquisition Systems Division Like the Logistic Systems Division, the Acquisition SYSteanns Division is co-located with the headquarters of a majoq: command--in this instance, the Air Force Systems Comn‘and at Andrews Air Force Base. Auditors under the direction of the Acquisition SYStREnus Division are assigned to the major buying centers suck; Eis the Space and Missile Systems Organization (SAMSO) at BIL Segundo and at Norton Air Force Base. California. run 5.1.»; . V 9 If... ~ -‘ b. u.‘. b. .‘A‘ L“H‘ a .505 p "‘V~s~ . I l ira‘ ~ 1" ‘¢;.‘- I\ . '0‘" h. M'L“ 5.4 § ~K'2'v . nu“ hr: (I ‘)l " t" a” . VKJM“ 3“... :3 m (I) (I) 36 The Audit Report--A Final Product Within this organizational framework, audit respon- sibility regardless of the auditor's location is basically approached in two principal manners, one, through locally scheduled audits, and two, through centrally directed audits. In the first instance, the resident auditor on a base, when not occupied in performing directed audits, uses his time to identify audit needs peculiar to his installa- tion and where, in his Opinion, he can best serve manage- ment . He surveys areas of suSpected audit need on his own initiative or in response to requests of local officials, checks out potential problem situations, deve10ps a program, and performs the audit. He issues his audit report to the management level with the authority to assure that appro- Priate action is taken on audit findings and recommendations. To follow the usual procedure, the auditor prepares a draft report of audit and discusses the contents of the rePOJI‘t with base operating officials-—possibly the activity commalnder and key section personnel whose work was reviewed in t}1ee audit--and with the installation commander. Purpose Of the discussion is to insure that all management weak— nesses uncovered during the audit are known and that recC>Iruuended actions contained in the report of audit are understood. “F‘o ‘ loin A... V---». "Yah‘ , L‘ ‘.‘ ‘ Q .‘._h .. «Mc‘ Is it.“ .“ ~° ‘v ‘~ “ s ‘Fw‘fi \‘V.‘: ‘ ‘ K 't \A. A! U . s 37 Following the discussion, the report is finalized and signed by the resident auditor. COpies are provided to managers at base level, with a c0py being sent directly to the major command concerned. The major command, after analyzing the report, may direct that specific actions be taken throughout the command--if there is a possibility that the management weaknesses noted may be in evidence at other command bases. The centrally directed audit approach, on the other hand, is used to evaluate total resource management and to Provide recommendations for systems improvement to tOp leVel Air Force decision—makers. Depending on management's needs, a centrally directed audit may be aimed at a specific functional problem; at an interrelationship of two or more func tional activities such as maintenance and supply: at policies, procedures, and methods at several levels of manalgement; or at an entire system where the total system is treated as one entity, considering all affecting func- tions, management levels, and interrelationships. The results of these audits are summarized and inc luded in Summary Reports of Audit which are issued to the Air Staff. Air Staff comments are incorporated into the reports which are transmitted to the Assistant Secre- t . . at}? of the Air Force for F1nanc1a1 Management and, by ¥ :NV A ‘V‘ . :‘F‘ev he“ ‘- IVA“... ”‘~~v. . I in 38 decree of the Office Of the Secretary of Defense, to the Office of the Assistant Secretary of Defense (Comptroller). An Evaluation of the Service That internal audit service is of considerable \NilJJe to Operating Air Force commanders is evident in a series of letters written by such commanders in early 1971. Ai: ‘that time, motivated by a consistent loss of manpower fcxr' a number of years, the Air Force Audit Agency was con- sidering a prOposal to reconfigure the organization. This PrOposal was apparently viewed as a step toward reduction in audit service. In support of maintaining at least the same degree Of aiudit service, Lieutenant General David C. Jones, Com- maI161er, Second Air Force, on January 28, 1971 wrote the f01-Joowing to General Bruce K. Holloway, Commander in Chief, Stra tegic Air Command: IDuring the past year I have made it a point to stOp and visit with the Resident Auditor sup- ;porting each of my bases. Without exception, I have been impressed with the qualities of these people and the positive results we get from the service they perform. The products and advice of the Resident Auditor are an essential and integral part of the com— mand function throughout Second Air Force. With today's money and manpower reductions and the escalating cost of operations this service is vital if we expect to do more with less. 39 I understand there may be some thought of reducing the number of Resident Auditors throughout the Air Force. If such an action takes place I believe it would be counter-productive. It would take an enormous savings to balance-out the loss of any of this small group of professional critics. On January 29, 1971, Lieutenant General Paul K. Carlton, Commander, Fifteenth Air Force, on the same sub j ect wrote : I recommend that resident auditor forces be retained and that the highest priority on the available ser- vices be given requests for audits of activities that numbered Air Force commanders consider to be of prime importance. In response to these letters, General Holloway, on February 16, 1971, in a letter to General John D. Ryan, Chief of Staff, United States Air Force, stated: During the past year my immediate staff had r1s. New methods are found to improve audit service. And so it is with the present research. Rather t}15111 accepting past methods of extracting information by alléifiLt, these methods were questioned and research support Provided to suggest a better way. 41 FOOTNOTES This chapter is based on an article written by Jimie Kusel, "Man-On-The-Spot", The Armed Forces Corrmtroller, Jilly. 1969' pp. 22-26. For definitions of these offices see page 47. “’1 - I .-_.‘._n. . .Pn h-C fla‘. I15. L) I 1) I r (I) CHAPTER III RESEARCH -- PHASE I AND II The primary purpose of this chapter is to review the first two phases of the research. The chapter is divided into four sections. The first section reviews the rationale for auditing at a sample of locations to reintro— duce the primary subject of inquiry -— the information derived curve. The second section provides key definitions to terms used throughout the research, identifies the Phases of research that were accomplished, and lists the Criteria used to evaluate research results. The final two Sections provide a discussion of the first two phases of the research. Covered by discussion for each phase of the research are: the objective or objectives, a description of what was measured, a description of the selection of both audits and audit locations used in the analysis, a discussion of the research and an evaluation of the resHalts, actions taken by audit managers, and a discussion of the research to follow. 42 43 The Sampling Rationale As pointed out in Chapter I, one can reason that auditing at a sample of locations rather than at the full pepulation of locations is at least economically desirable. Since this rationale is basic to the following research, these arguments are briefly reviewed. It was assumed that a management system is in oper- ation at approximately 180 locations -- the universe of such locations. It was hypothesized that the curve of in- formation derived from the application of an audit to test this system can be basically drawn as follows: FIGURE 6 INFORMATION DERIVED CURVE // Cumulative Percentage of Information Derived 0 0 Locations 18 This figure simply indicates that the first few 10cGUT—ions in any sample will make relatively large contri- b - . . . . utlOns to the amount of information derived from the audit t . . eats} As the number of locations included in the sample t‘ F " 44 increases, the total amount Of information derived may increase, but at a decreasing rate. From data provided by past statistical records, the curve of the cost for the development and applying the audits at locations can be drawn somewhat as follows: FIGURE 7 COST OF SAMPLING CURVE Cost of Sampling / 180 0 Locations It was reasoned that this curve does not start at the axis intersection due to fixed costs in the develOpment of the audit program. Cost generated by application at each loca- tion is considered relatively stable, increasing total costs at a constant rate as the number of locations in the Sal"‘ple is increased. It was then hypothesized that the two curves com- bined would provide the following figure: 45 FIGURE 8 RATIO OF COST TO INFORMATION DERIVED Cost of Sampling/ Cumulative Amount ‘AL‘ 0 f Information Derived ‘ L 0 "n" 180 This figure indicates that as the number of loca- tions in the sample is increased, costs increase at a con— stant rate. Information derived increases, but at a decreasing rate. To maximize the amount of information for true dollars spent, the size of the sample should be where time two curves are farthest apart, or at point "n." The important conclusion reached from this line of reasoning is that costs are avoided as long as the audit program is applied at less than the population of locations. Furrt11er' the sample size can often be cut with very little loss of management information. But an important question left unanswered at the Cone lusion of the discussion was: what is the actual SIIEiIDe of the "information derived curve”? Whereas the c . L13: Ve for actual costs of auditing is rather eaSlly 46 Cieatermined, determining the shape of the information éiearived curve is a more difficult task, one to which an approach must now be develOped. Mechanics of the Approach To this point the discussion has been in general terms. The "information derived curve" has been discussed. It was stated that the curve is composed of ”reportable i type items" but the items were not precisely defined. I Other key terms such as "local audit" and "centrally directed audit" were indirectly defined. The research accomplished was touched upon but, with the objective of only introducing the research tOpic, avoided excessive detail. Now, however, it is necessary to shift the mode 0f presentation -- to become involved with the nuts and bolts of the project. The purpose of this section is to define key terms, Provide a coding system to identify various phases of the research, and to specify the general criteria that are used in evaluating audit methods suggested by the research restilts . \De finitions . First, consider the definitions. A number of i Itlportant terms specifically related to the research are 47 11ssed.throughout the discussion. Some of the terms may be :Ezindliar, others will not be. Some of the terms relate ssgpecifically to the Air Force Audit Agency, others were disaveloped and defined during the course of the research. Some of these terms are again defined and explained ciLiring the discussion of the related phases of the research. {Flierefore to provide this discussion of terms at the outset :irivites a certain amount of redundancy. However, a basic acquaintance with these important terms at this point in 'tlie dissertation provides some insight into the research that follows: The more important and frequently used terms are defined and/or explained below: gggc1it program. An outline of steps to be followed or work to be done in accomplishing an audit. fldit survey. A limited examination to determine the need for audit or the extent to which audit tests are to be applied. E§5§§;§or General Representative. The title of the chief of an Auditor General Representative Office. ggisléigor General Representative Office (AGRQL. An Auditor General office whose audit mission is pri- marily related to an Air Force organization as distinguished from an installation. 48 Egyiitor General Resident Office (AGRQL. An Auditor General office whose audit mission is primarily related to an installation as distinguished from an Air Force organization. Ceqitrally directed audit. An audit scheduled by the Head— quarters, Air Force Audit Agency. The appropriate audit programs are developed in response to requests from the Air Staff, higher levels of management, and internal sources. The programs are usually applied simultaneously at two or more locations throughout the Air Force. Auditors issue reports on their findings at each location in the same manner as for their locally scheduled audits. These reports, however, are usually forwarded to the Headquarters, Air Force Audit Agency, for summarization. The summary report of findings is provided to Major Commands, Air Staff, the Assistant Secretary of the Air Force for Financial Management, and through him to the Office of the Assistant Secretary of Defense, Comptroller. igagiiltrally directed audit program. An audit program 49 prepared by the audit control point to accomplish a centrally directed audit. ,Audit control point. An Auditor General element which is assigned responsibility for managing, pro- gramming, summarizing, analyzing, and reporting a centrally directed audit. Seugmented audit. A subject matter audit that is divided into parts. Each part is accomplished at a different sample of locations. The audit findings from each part are combined and summarized into a single report of audit. Such audits permit the expansion of audit coverage while controlling the total audit hours used. .Lfiagally scheduled audit. Audit work scheduled and per— formed by an AGRO without direction from higher audit management authority. The Resident Auditor issues reports over his signature to the activity manager, the base commander, and the major air commanders. lifiigiident Auditor. The title of the chief of an Auditor General Resident Office. ‘éiliZQject matter audit. An area, system, activity, function etc., on which a centrally directed audit -—-.~l.' "l g, Reportable type 50 has been completed. items. This term assumes different meanings depending upon the phase of research being discussed. In general, however, reportable type items are those items of information that are disclosed by a subject matter audit and that inform tOp levels of management of non compliance with management directives or of poor management practices. For Phase 1 of the research, reportable type items are defined as items of "specific" information that are disclosed by a completed subject matter audit and that inform tOp levels of management of non compliance with management directives or poor management practices. The information is specific if it meets two criteria. First, each item of information must be separate and distinct from every other item of information. For example, for a subject matter audit of a clothing sales store one item may be that physical inventory counts made by auditors do not 51 match balances recorded in perpetual inventory records maintained by managers. Second, the information must relate to inaccurate or undesirable actions that occur with a minimum frequency within the total subject matter audit. Specifically "minimum frequency" here relates to Oper- ating errors or poor management practices that are observed to occur in the field of audit observations at a rate of ten percent or more. Using the example above to illustrate this second criterion, if for the completed subject matter audit ten percent or more of the comparisons between physical inven- tory counts made by auditors and balances recorded in management's records do not agree, then this item of information would be considered sufficiently material to interest tOp level managers and would be judged to be a reportable type item. This definition was develOped through discus— sions with audit managers as one that will encompass most of the items of information 1 “7‘; cacti." .5" ' iii:- 52 reported to higher levels of management. After analyzing the results of the phase 1 research, the definition of report— able type item is revised for phase 2 of the research. The revision pertains to the second of the two criteria discussed above -- that the information must relate to in— .U. -“T—_1 accurate or undesirable actions that occur with a minimum frequency within the total subject matter audit. Specifically, the revision is in the definition of "minimum frequency." For phase 1, the minimum fre— quency is ten percent of that which is ex— amined in the subject matter audit when the subject matter audit is considered as a single unit. Using this definition it is possible that a very serious problem at a single location alone could cause the second criterion to be met. Yet, as previously reasoned, t0p level managers are not so much interested in isolated problems as they are in general problems that are in evidence at a number of locations. After discussion with audit managers, the 53 "minimum frequency" was redefined in terms of the number of locations at which the problem was in evidence rather than as some error rate in the total records examined. The error rate in the records at each loca- tion remains important to judge the mate- riality of the condition at that location, but for this phase of the research, the i 4 materiality judgment is secondary to the F determination of the general sc0pe of the problem across locations. Specifically, for the phase 2 research "minimum frequency" is examined at three different incidence levels. The Operating error or poor management practice: 1. occurs at at least one of the loca- tions in the universe of locations. 2. occurs at at least five percent of the locations in the universe of locations. 3. occurs at at least ten percent of the locations in the universe of locations. Based on interviews with audit programmers 54 and managers, the third alternative is considered the most representative of the type of problems of interest to tOp level managers. Information derived curves based on reportable type items defined by each of ‘._.‘l 4‘. _ the alternatives were develOped, however, to highlight the sensitivity of the curve to changes in definition. Finally, for phase 3 of the research the definition of reportable type items is that used for phase 2 assuming the third alternative for "minimum frequency" -- occurs at at least ten percent of the locations in the universe of locations. llljflormation derived curve. Initially it is best to think of the curve as the accumulation of new items of information (reportable type items) as provided by each additional location included in the audit. For example, assume that an audit in a specific activity (i.e. supply) was accomplished at 150 locations. Further assume that analysis of the total audit information from these locations yielded 100 different independent reportable 55 type items, i.e. the first might be that unauthorized supplies were being ordered, the second that stock record cards were not accurately posted, etc. Now assume that one wishes to determine how much of this reported information could have been detected at a small sample of these 150 locations. The first location selected for the sample (however selected) would yield several of the original 100 kinds of items. Assume for the sake of illustration that at the first location 36 of the 100 kinds of items are detected. Now turn to the hypothetically constructed information derived curve of Figure 9, page 56. Along the horizontal axis of the figure find the first location. Since 36 of a possible 100 kinds of items represents 36 percent of the total, proceed vertically until a position Opposite the 36th cumulative percentage point is located and place a dot. This dot represents the first point on the informa- tion derived curve. Similarly, the second 56 FIGURE 9 CONSTRUCTION OF THE INFORMATION DERIVED CURVE 100- 90‘ 80“ 70“ Cumula t 1V8 60’ Percentage of Reportable Type I tems 50- 40-. 30« 20 10' L---—- L-- H«———-—-.’—-- 4567891011 Locations 12 13 14 15 etc. —-r-1 57 location in the sample would yield perhaps additional items of information not found at the first. Continuing the illustration, assume that the second location yields l8 kinds of items not detected at the first location. In cumulative terms, one will -u IT-j now have detected 54 or 54 percent of the original 100 total items. Again, along the horizontal axis of Figure 9 find the second location. Proceed vertically until a position Opposite the 54th cumulative percentage point is located and place a dot. This dot represents the second point on the information derived curve. As addi— tional locations are included in the sample, additional points on the information derived curve will be located. Finally, connect these points with a smooth solid line as shown in Figure 9 and the informa- tion derived curve is constructed. Now look once again at this important curve, this time in a bit more detail. As shown in Figure 9, it is a solid line drawn on a graph with a vertical axis labeled 58 "cumulative percentage of reportable type items" and a horizontal axis labeled "loca— tions". The curve indicates the cumulative percentage of reportable type items in evi- dence up to any number of audit locations shown on the horizontal scale. Percentage 1 rather than number of reportable type items is used on the vertical axis to permit com— parison of the information derived curves ; for audits that vary in the absolute numbers of reportable type items. It has been stated that the curve is a solid smooth line. The discerning reader may argue, however, that a correct plot of the data results in a series of discrete points, one above each numbered location rather than a continuous line. He might further argue that if the points are to be connected, the connection should be in a stair step fashion as illustrated by the dashed line in Figure 9. Reportable type items are accumulated at a location, not between locations. Mathematically speaking, these arguments are correct. However in 59 this and subsequent chapters a number of information derived curves are compared simultaneously. To more easily visualize these comparisons, they are drawn as smooth continuous lines. While such construction takes some liberties with reality, the logic of the arguments that follow is not affected. 53 curve. Since the primary curve that is referred to in ¥ gort of audit . 33ernal audit. this research is the information derived curve, it may be occasionally called "the curve" for short. A document which identifies the SCOpe of the audit and conveys the auditor's findings and recommendations to the responsible manager or managers for apprOpriate action. The independent, objective, and construc- tive review and appraisal of the effective— ness and efficiency with which managerial responsibilities in all areas (financial, Operational, and support) are carried out at all levels of management. E3 levels of management. TOp levels of management may be defined differently depending upon the 60 perspective of a given job level. To make the intended meaning here perfectly clear, tOp levels of management will refer to management above the base level. A synonym used is higher levels of management. E31§E§es of Research The total research was completed over a three to kaIIr year time span with the bulk of the research completed 54r1 the most recent year. Accordingly the research is Ciii—"vided into three phases called apprOpriately enough: I>li£ase I, phase II, and phase III. Each phase of the r‘esearch brings the actual shape of the information derived cl-‘lrve into sharper focus. An objective of discussing the research in three phases is to provide a complete rationale f(Dcr'the changes in audit methods both adopted following and 5‘53 currently suggested by each phase. Each phase approaches tillsa research in a different manner and uniquely contributes t1C> the total knowledge. Following an initial identification, a special c=x~ focus specifically on the audit codes for phase I and 333C - Here, the numerical digit indicates the phase of the reEssearch —— a "l" for phase I and a "2" for phase II. LC><3king at the audit codes of phase III, notice that there EiITn.to state that: "Efficiency relates to the satisfaction c>f individual motives . . . ."6 "Effectiveness" is used here in Barnard's sense. " Efficiency," however, is used not in the sense of Barnard, taut rather in its more usual engineering sense: the Optimum Iredationship between input and output. The more units of cwutput that are obtained from a given input, the more effi— c:ient is the process or method. Units of output here will Iaeethe reportable type items of information previously ITlentioned. Units of input will be the locations included ill the audit. Additionally, the research results of phase III are Ijtuflged using such additional criteria as cost, timeliness, e confirmed by empirical research. jphe Objective Phase I of the research was accomplished to deter— rnine if the information derived curve is uniquely shaped aas hypothesized. frhe Reportable Type Item As defined previously, reportable type items are in ggeneral those items of information that are disclosed by a ssubject matter audit and inform tOp levels of management of tion compliance with managerial directives or of poor manage- nnent practices. For this phase of the research, an item of informa— tzion must meet two specific criteria to be classified as a Ifeportable type item. First, each item of information must IDe separate and distinct from every other item of informa— ‘t:ion. Repeating an earlier example, one item may be that L>Txysical inventory counts made by auditors do not match balliances recorded in perpetual inventory records maintained 1?3’ Inanagers. Second, the information must relate to inaccu— rate or undesirable actions that occur with a minimum 65 frequency within the total subject matter audit. Specifi- cally "minimum frequency" here relates to operating errors or poor management practices that are observed to occur in tflae field of audit observations at a rate of ten percent or rmore. Continuing the example, if for the completed subject ‘matter audit ten percent or more of the comparisons between gfliysical inventory counts made by auditors and balances reuzorded in management's records do not agree, then this iJmem of information would be considered sufficiently mate- rixal to interest tOp level managers and would be judged by tliis second criterion to be a reportable type item. This definition was developed through discussions “filth audit managers as one that will encompass most of the iiaems of information reported to tOp levels of management. Sealection of Subject Matter Audits For the detailed analysis, audit managers were re- ‘Ituested to provide the supporting data for three completed Stibject matter audits applied at 40 or more locations. Stupporting data for the following completed subject matter audi ts were provided : 66 TABLE 2 SUBJECT MATTER AUDITS - PHASE I (1) (2) (3) (4) Number Number of Audit of Audit Area Code Locations "Items" Axxzcrued Military Pay A1 105 38 Commercial Transportation B1 177 37 Clothing Sales Store Cl 162 41 (Zolumn 3 indicates the total number of locations at which ‘the audit was applied. For each subject matter audit, <:olumn 4 indicates the total number of reportable type items. Three subject matter audits applied at at least 40 Slocations each were selected because of the belief that 1:his would be a sufficient number and size to indicate if 1:he information derived curve rises rapidly enough to give Ipromise of suggesting more economical means of abstracting Eiudit information. Unless this were the case, additional 61nd more detailed research would be useless. Selection of Sample Locations In searching for a bend in the information derived (Zlirve, it was arbitrarily decided to determine the cumula- thJVe percentage of reportable type items that can be 67 detected at 10, 20, 30, and 60 locations. Accordingly, audit managers were requested to select judgmentally the first ten locations that they would have suggested for in- clusion in the audit had the audit been limited to ten locations. Following that selection, audit managers were :requested to select ten additional locations until all s amples were identified . At this point in the research, no attempt was made tzo identify the selection criteria used by audit managers. VVhere appropriate managers were requested to ignore as much 218 possible hindsight knowledge of results in making their sselections. But again, no attempt was made to control specifically for such knowledge. The Research The detailed research consisted of first analyzing true supporting data for each subject matter audit to deter— Inille the total number of reportable type items. Next, data prravided by the selected sample locations were examined to determine the cumulative number of reportable type items detected by each succeeding block of sample locations, i.e. 10. 20, 30 etc. Results of the Research Results of the research are recorded in the follow- 68 ing table: TABLE 3 RESEARCH RESULTS - PHASE 1 Number of Number Of Cumulative Locations Items Percentage Audit Code Used Detected of Items A1 10 30 79 A1 20 31 82 A1 30 38 100 Al 60 38 100 A1 105 38 100 B1 10 37 100 B1 20 37 100 B1 30 37 100 B1 60 37 100 B1 177 37 100 Cl 10 36 88 Cl 20 40 98 Cl 30 41 100 C1 60 41 100 C1 162 41 100 R Plotting these results provides Figure 10, shown on Page 69. “.0. “I I P; 69 oea ofl 8H oHH ooH madamm ca msOHumoog om or on co om 0: on -oa ,oN -om rom rooa H mmmbo nm>Hmma ZOHHaumasaso 70 An Evaluation of Results Evaluating the results of this limited research provides some useful information for management, but sug- gests many additional lines of investigation, the results of which could result in even more effective and efficient Operating methods. On the positive side of the evaluation are listed the following: 1. The information derived curve for each subject matter audit is in general shaped as hypothe— sized in Figure 6, page 43. Full detection of the reportable type items was achieved no later than at the thirtieth location selected from the pOpulation of loca— tions -- at less than thirty percent of the universe of locations. By continuing to audit at the population of locations, more information is being derived than is necessary to brief higher levels of management on general operating conditions. Effectiveness could be maintained and effi- ciency improved if something less than the population of locations is included in each subject matter audit. 71 On the negative side of the evaluation the follow- ing comments are apprOpriate: l. The research was not designed to give a clear indication of the rate at which reportable type items are detected as individual locations are added to the analysis. For example, 79 percent or more of the items for each subject matter audit are detected at 10 or less locations. But is this also true at five or six or seven locations? Such information is useful for: a. Determining the range of locations beyond which the incremental costs of audit tend to exceed the value of the incremental information. b. Assessing at each number of locations the risk of not detecting significant informa— tion of interest to managers. c. Suggesting other audit methods that are dependent upon the representativeness of the measured information at a small number of locations to that found at the universe of locations. These methods involve audit program develOpment, field testing of audit programs, and segmented audits. They are 72 elaborated upon in Chapters V-VII. As a result of discussions with audit managers over the results of the phase I research, changes in the definition of reportable type items were suggested. A definition was needed that will permit not only the detection of management problems within a sample of loca- tions, but also permit some assessment of the materiality of the problem across all loca- tions. Based on contacts with tOp Operating management, audit managers believed materiality must usually meet two criteria. First, the management problem must be revealed material at the specific location at which it is detected. Second, this problem must be in evi- dence at a number of individual locations. If the audit finding fails the first criterion, it is considered a minor problem of local man- agers. If it fails the second, it is a major problem for local managers but not a general problem to tOp level managers. The rationale for using judgmental methods rather than random selection methods to choose the sample locations for the subject matter 73 audit is not provided. Management Action There is an old saying that the proof of the pudding is in the eating. Perhaps the most critical evalu- ation of research results -- the proof of the pudding -- is in the implementation of management methods suggested by the results. One major change in audit method was adOpted by the Air Force Audit Agency following disclosure of the results from phase I. The number of locations to be selected for certain planned subject matter audits was reduced to 60, about one-third of those previously included. Although this is a step toward more economical audit procedure, the data plotted in Figure 10 indicate that even more efficient procedures may be possible. By the tenth location a major portion of the reportable type items are detected. By the thirtieth location all reportable type items are in evidence. While the remaining thirty locations in a sixty location sample may provide a clearer picture of laow extensive the management problem is across locations, cane might ask the question —- is there a point or range at Vihich the incremental value of the information added exceeds -ii:s incremental cost as additional locations are added to 'tlle: sample? Is that point likely to be at less than 60 74 locations? Additional Research Needed To provide an answer to these questions it was obvious that additional research was necessary. For one thing, considerably more must be known about the generation of reportable type items as individual locations are added to the sample locations. Specifically, the shape of the information derived curve through the first ten locations is needed. Establishing a first point on the curve at the end of ten locations conceals this information. The addi— tional research must also be designed in such a way that reportable type items are not only detected, but their materiality evaluated as well. If it can be shown that the management problems detected at small numbers of locations also meet an acceptable criteria for materiality, then the 'basic data is at hand to effectively compare the incremental ‘value of adding an additional location to its incremental cost . Research - Phase II Phase I of the research provided some useful infor- nuation. It indicated that the information derived curve is Shalped basically as hypothesized. Although the curve was 11C>t1 determined with any great degree of precision, it was 75 sufficiently definitive to audit managers to Spark an im— portant economical change in audit method. Considerably more information about the exact shape of the curve is needed, however, before it can be used with precision in determining economical sample sizes or for making other changes in audit methods. Some of this information is pro- vided by phase II of the research. The Objectives The primary objective of phase II of the research is to determine the shape of the information derived curve over the span of the first ten locations of a judgmentally selected sample of locations. Secondary objectives are to analyze changes in the shape of the curve as the definition of reportable type items is varied and to make some assess— Inent as to what percent of the information contained in the report of audit for each subject matter audit could have been provided to Operating managers based on information cierived from the sample of locations. (Phe Reportable Type Items Reportable type items for phase II of the research are defined very similarly to reportable type items for Phase I. Here also an item of information must meet the Same two specific criteria to be classified as a reportable 76 type item. First, each item of information must be separate and distinct from every other item of information. Second, the information must relate to inaccurate or undesirable actions that occur with a minimum frequency within the subject matter audit. But for phase II of the research there is a subtle change in the meaning of minimum frequency. To make sure that the subtle change in the meaning of minimum frequency is made perfectly clear, recall its meaning for the phase I research. There minimum frequency related to Operating errors or poor management practices that were observed to occur in the field of audit observa— tions at a rate of ten percent or more. Using this defini— tion it is possible that a very serious problem at a single location of the many locations included in the subject matter audit could cause the second criterion to be met. It has been reasoned however that tOp level managers are not so much interested in problems that are isolated at one or two locations as they are in problems of general concern to a high percentage of the locations. The subtle <:hange in the meaning of minimum frequency concerns the 19ercentage of locations evidencing a problem. Such items ‘as error rates in the records at a given location are limportant to judge the materiality of a condition at that lxacation, but this materiality becomes secondary to the 77 determination of the general sc0pe of the problem across locations. To provide adequate consideration for the sc0pe of a management problem across locations, "minimum frequency" in the phase II research was considered at three different incidence levels. The Operating error or poor management practice: 1. occurs at at least one of the locations in the universe of locations. 2. occurs at at least five percent of the locations in the universe of locations. 3. occurs at at least ten percent of the locations in the universe of locations. Audit managers considered the third alternative to be most representative of the type of problems of interest to tOp level managers. Information derived curves based on report— able type items defined by each of the alternatives were develOped, however, to highlight the sensitivity of the curve to changes in definition. _§election of Subject Matter Audits An important problem that arises in considering the selection of subject matter audits is how many should be £3elected? For the research in phase I, three were selected 78 based on the rationale that three were sufficient to deter- mine the feasibility of further research. As it turned out, audit managers revised their procedures based on the limited data. Although the adoption of improved audit pro- cedures was not the primary objective of this initial re— search, such action is the motivating force underlying all of the research. For phase II of the research a similar situation is faced. The primary objective of this phase is not to provide specific data that management will act upon, but such action is certainly the ultimate motivating force underlying the research. The problem of determining "how many" was therefore approached by asking audit managers their Opinion. Specifi- cally, they were asked how many replications of the infor- mation derived curve would be required before they would be reasonably certain of its shape over a span of the first ten locations. Without knowing probable uses of the curve, the question was difficult to answer. However, considering the research results of phase I their professional Opinion was that three or four additional curves would be sufficient. Since experience indicated that approximately four work days were required to construct each curve, this phase of the research was limited to the construction of four 79 additional curves. Audit managers were requested to provide the sup- porting data for four completed subject matter audits that were considered to be representative of the type of audits performed and that were applied at at least 40 locations. Supporting data for the following completed subject matter audits were provided: TABLE 4 SUBJECT MATTER AUDITS -- PHASE II (1) (2) (3) Number of Audit Area Code Audit Locations Food Service A2 150 Aviation Fuels Division B2 152 Critical Item Control C2 143 Equipment Management Office D2 55 Selection of Sample pgcations Ten audit locations used in each of the subject matter audits must be selected. But how are they to be selected? In phase I of the research the locations were selected judgmentally by audit managers. Rationale for using a judgmental rather than a probability approach was not provided. Perhaps at this time it is best to digress a bit iind consider in general the various approaches to selecting 80 audit locations. They appear to be two in number, a prob— ability (statistical sampling) approach and a judgmental approach. The probability or statistical sampling approach is considered first. There are two general methods by which a statistical sampling approach might be used to select locations for audit application. The first method would define the total number of locations as the universe, and depending upon the type of information desired from the sampling plan, compute a sample size, and provide for sample selection and summari- zation. This method would imply that the selection of the specific locations for audit application would be on some form of random basis. This first method has several conceptual difficul— ties. First of all, the universe size (approximately 180 bases when dealing with Air Force locations) is too small to result in much savings, for most any sample size would approach universe size if statistically acceptable infer— ences concerning the pOpulation are made from the sample. A second difficulty with this approach is that the loca- tions are not homogeneous in nature (this homogeneity problem is mentioned later), the primary restriction placed upon sample selection under a plan such as method one sug- gests. Such a plan would be difficult if not impossible to 81 design in such a way as to provide economically statisti- cally sound inferences. A second general way to approach the problem is to use some specialized statistical techniques for universe stratification and sample selection. Typical of the tech- niques which could be used are stratification of the uni- verse by some reasonable homogeneous characteristic, such as major command, and/or the use of cluster sampling tech- niques. This approach also presents difficulties. The most important difficulty and the one which precludes this approach arises from the large number of "unique" locations which do not fit in with a larger classification group. For example, within the Air Force, 10 out of the 18 major commands have 10 or fewer bases; 8 of these 10 have 3 or fewer bases. This implies that any statistically designed stratified sample would approach the size of the universe, just as was the case even if all the bases were considered a homogeneous group as in method one. Of course, the ideal statistically designed sampling plan for audit application would define the uni— verse not in terms of such locations as bases, but service (e.g., Air Force-wide), in terms of functions, processes or paper work. This would result in the computation of one sample size which would be spread or allocated to each base 82 on a random basis. In this instance, the audit would be performed at all bases, but the amount of detailed testing performed would be quite minimal at given bases, and at a minimum in total. This alternative does not fit strictly within the objectives of this discussion, for it does not represent the application of audits of selected bases, rather it presents a method for decreasing the amount of detailed testing performed at all bases, where all bases will be audited. Unfortunately, applying a single sample (where the definition of the universe is Air Force-wide and ignores base subdivisions) across all bases is fraught with diffi- culty. There are three basic accumptions which must under- lie such an application, all dealing with the homogeneity of the Air Force-wide information system. The first assump- tion is that the system as designed (including all internal controls and information processing techniques) is substan— tially identical regardless of major command or base. This assumption is reasonable, in many cases. The second assump- tion is that actual internal controls and data handling are substantially identical at all bases; the third assumption is that the administrative and clerical effort and quality are substantially identical. Audits performed at various locations in the past have shown that this third assumption 83 is not reasonable. These three assumptions may occasionally be satis- fied. However, even if this is true, another disadvantage exists. Information drawn under a single sampling plan for all bases would provide little information of use at the base level. The reason for this is that very few actual samples would be selected from any given base. Assume a sample size of 800 (which is certainly sufficient in most applications where a universe of infinite size is used). This would mean that there would be an average of approximately five samples drawn from each base. From a statistical viewpoint, summary information can be made Air Force-wide because of the assumption of homogeneity; un- fortunately, common sense sometimes has to displace the finest of our statistical designs. In this case, it would be difficult to argue with any reasonable man that a sample of five from any base would provide information of value to the base commander. The auditor, even when performing an audit primarily for tOp management, has assumed a reporting responsibility to the local commander, and such an approach is untenable. As far as the selection of certain bases for audit iipplication on a statistical basis is concerned, it does rust appear to be a wise course of action. The primary 84 reasons for this are the nonhomogeneity of the universe, the minimal savings promised because of a high sample size/ universe ratio, and the conceptual argument that except for special studies, statistical inferences dealing with Opera- tions are not needed at higher than base level. Obviously, if statistical methods are not used in selecting locations at which the audit will be applied, then a judgmental method will be used. Use of this method re- quires both a thorough understanding of the method and the difference between it and the statistical method. First, consider the difference in methods. Any sampling plan can be thought of as consisting of three discrete processes: determination of sample size, a selection of the sample, and summarization of sample re— sults. When a statistically designed sample is used, the selection and summary processes are also statistically per- formed. When a sample size is determined judgmentally, summarization may include quantitative statements (e.g., in our sample of 800, we found a 10% error rate), but cannot include probabilistic statements about the results (e.g., the error rate of the total pOpulation is estimated between 8 and 12%, and we can make this statement with 95% confidence). With regard to using a judgmental approach to 85 determining sample size and selecting the sample, there is a central guiding criterion. A sample size should be deter- mined and the specific locations chosen in such a way as to provide information to tOp management which is representa— tive of the total information system under audit. For illustrative purposes, assume that the auditor wishes to select a number of Air Force bases at which to review a particular system in Operation. Then his selection of the particular locations or bases would depend upon answers to such questions as: 1. Who requested that this audit be performed? 2. Why did that request originate? 3. What Specific problem or information need prompted the request? (to what use will the report be put)? 4. Is the information need peculiar to a given portion of the total information system of the Air Force? 5. Are matters dealing with system design, policies, and operations within the system sufficiently uniform so as to be able to make judgments about the total system on a less than total system examination? 6. How many bases are likely candidates for the 86 application of the audit program? 7. What are the characteristics of the bases which bear upon their role in this information system (size, command, etc.)? 8. How many bases would have to be examined in order to provide information as to the general strengths and weaknesses of the system under study? 9. At which bases will the system be the most representative of the Air Force—wide system? 10. In what form should the recommendations be made? Top management must receive results of audit oriented to a total system; it follows that the sample should be selected in such a manner that the bases used for audit application will have Operations which are deemed to be representative of the total system. Answering such ques- tions as listed above is not easy, for specific guidelines are hard to state. Some scheme for classifying bases by the nature of their participation in the information system must be used. Some of the classification parameters could be the size of the system at the base, the volume of paper- work handled, the techniques for information processing, whether the base is domestic or foreign, the major command to which the base is assigned, and experience of the 87 auditor making the audit. All of the parameters are the type of characteristics which lead to formal stratification when statistical sampling is used. In a similar way, they should influence the selection of the bases for audit appli— cation on a judgmental basis. For the samples for the present four subject matter audits, audit managers were requested to select judgmentally for each audit the ten locations that they would have sug- gested for inclusion in the audit had the audit been limited to ten locations. The primary objective for the selection of the locations was to detect by sample the management problems that existed in the universe of locations. As in phase I of the research, no attempt was made at this point in the research to identify the selection criteria used by audit managers. To provide representative rather than biased results, audit managers in making their selections were requested to ignore as much as possible hindsight knowledge of audit findings at specific locations. But again, no attempt was made to control specifically for such knowledge. The Research The four selected subject matter audits had orig- inally been applied at 500 locations. Information from 88 these 500 individual audits was analyzed to determine the number of reportable type items. The shape of the infor- mation derived curve over the span of the first ten loca- tions of a judgmentally selected sample of locations was then determined. That is, points were plotted on a graph in a manner described in considerable detail earlier in the chapter. The points represented the cumulative percentage of reportable type items detected after the first, second, third, and so on up thru the tenth location. The shape of the information derived curve was determined three times for each subject matter audit ~— once for each of the three different definitions given to the reportable type items. One curve was plotted when the definition involved the detection of all audit findings from the universe of audit locations of the completed sub- ject matter audit. A second curve was plotted when the definition involved the detection of those audit findings that were in evidence of five percent or more of the uni- verse of audit locations of the subject matter audit. The third curve was similar to the second except that it in- volved detection of those audit findings that were in evi- dence at ten percent or more of the universe of locations. Since the end product of the audit is information, an analysis was made of the reportable type items composing 89 the third curve to determine the percentage of all informa- tion in the report of audit for the subject matter audit that was also materially present in the sample of locations. Stated another way, an analysis was made to determine what percent of the reported information could be provided to Operating managers based on information derived from the sample of locations. The percentage was determined at the end of the first five locations and at the end of the tenth location for each sample of locations. To make this analysis in a quantifiable manner, some Objective criteria were needed to judge the materi— ality of the reportable type items found in each sample. Note that the materiality of an item that did appear in the report of audit was not questioned. An item considered material by one person may be considered as immaterial by another. Or the same person may be inconsistent in his assessment of materiality depending upon the time and the circumstances. The materiality of an item may depend on its size, its nature, or a combination of both.7 Audit managers generally agreed that materiality of the reportable type items found in the sample of locations could be judged based primarily on two factors: frequency and magnitude. Frequency refers to the number of locations in the sample at which a particular error occurred. 9O Magnitude refers to the extent of the error at any one location. Data characteristics of frequency and magnitude influence the value of derived information, both to system managers and audit managers. To illustrate the importance of these character- istics to system managers, assume that each of ten bases has 1,000 items of equipment in use. An audit discloses that one item at each base is not prOperly authorized. Base frequency for this error is 100 percent (10 of 10 bases indicate the error), but the magnitude of occurrence is less than one percent (10 of 10,000). Considering frequency alone, the condition seems significant. When the charac- teristic of magnitude is added, assessment of the situation changes. To be of value to system managers, to be con- sidered material, a problem area must exhibit a prOper balance of frequency and magnitude. Of course it is not practical to state what that balance should be in all cases. The experienced auditor knows that there are often areas within an Operating activ- ity that are of more concern to tOp level managers than are other areas. For these sensitive areas, the degree of fre- quency and magnitude of management problems required to make them of interest to tOp level managers may be far less than the degree required in other less sensitive areas. 91 Therefore hard and fast rules to test materiality are usually not possible. However, over the years audit managers do develop general guidelines -- rules of thumb to determine the materiality of audit findings. From discussions with audit managers at the time of this research, the following gen- eral guide to judging the materiality of reportable type items found in the sample of locations was suggested: TABLE 5 FACTOR TESTS FOR MATERIALITY Five Location Ten Location Factor Sample Sample Frequency . Two or more locations Three or more locations Magnitude Ten percent, or if less Ten percent, or if less than ten percent, equal than ten percent, equal to the reported per- to the reported per- centage. centage. But to ensure comparability between the information contained in the report of audit and that provided by the sample of locations, two additional actions were taken. First, the frequency of occurrence of all reported infor- mation was reviewed. Those reported management problems that occurred at less than ten percent of the universe of locations were dr0pped from further consideration. Such problems would not qualify as reportable type items for the 92 information derived curve of primary interest. And second, if the data provided by the sample of locations failed the test for materiality specified in Table 5, one additional test was given. The frequency and magnitude of the data in the sample was compared with the frequency and magnitude of the reported information in the universe. If both the fre- quency and magnitude of the data in the sample was equal to or greater than that from the universe, the sample informa- tion was judged to be reportable. This latter modification to the materiality criterion was made so that management problems of a sensitive nature that perhaps need not occur as often as other problems to be of concern to tOp management are given equal consideration both in the universe of locations and in the sample of locations. Results of the Research The primary objective of phase II of the research was to determine the shape of the information derived curve over the span of the first ten locations of a judgmentally selected sample of locations. This objective is met, as elaborated upon earlier, using three separate definitions of "minimum frequency" in determining reportable type items. 93 These are: Definition 1: The Operating error or poor management practice occurs at at least one of the locations in the universe of locations. Definition 2: The Operating error or poor management practice occurs at at least five percent of the locations in the universe of locations. Definition 3: The Operating error or poor management practice occurs at at least ten percent of the locations. The number of reportable type items applicable to each subject matter audit varied with the above definitions as shown in Table 6, page 94. For example, audit A2 had 61 reportable type items when definition 1 was applied. Using definition 2 the number was reduced to 54, and for defini- tion 3, it was further reduced to 40. This reduction in number is as expected since each succeeding definition per- tains to more exclusive information. This general pattern of reduction can be observed for each of the three remain- ing subject matter audits as well. Consider now the reportable type items determined by using definition 1. The cumulative number of these 94 mes oNH NoN No mN om oNe No on oN one Na es en Ho N< m ooeoeoeeoo N ooeoaoeooo H ooNoNoeNoo oooo oeeoa mamuH Oma- mabmuuoemm mo Monasz HH Mmém I mZMBH mmwh. MAQHMOQMM 0 mafia. 95 reportable type items detected by the judgmental sample of ten locations for each of the subject matter audits is} recorded in Table 7, page 96. To illustrate the information contained in the table, the recorded data for audit code A2 is reviewed. At the first location 23 of the possible 61 reportable type items are detected. At the second loca— tion three additional items not detected at the first loca- tion are found for a cumulative total of 26. The other locations contribute information similarly thru the tenth location at which point 49 of the total 61 reportable type items are in evidence. Table 7 provides absolute numbers of reportable type items. Since the total of reportable type items for each subject matter audit differs, here ranging from 61 to 202, it may be difficult for the observer to determine the rate at which new information is being accumulated as loca- tions are added to the sample. It may be easier to visualize this rate if the data for each subject matter audit is con- verted into a common measuring device. This is done in Table 8. In Table 8 the absolute numbers of Table 7 have been converted to common percentages. Again using audit code A2 as an illustration, at the first location 38 per— cent of the total reportable type items are detected. At NNH Noe woe ooN Nod ooH see nee as ea NoN No ooe mm ea om an em NN so on an Gas No NN me No em an we Ne we NN eN omH Na as we Ne oe oe oe me Nm oN NN so Na oa N o N o m e m N H No mesmev oeoo oeeoa mOOfiumooq mamuH mo Honasz Houoa H onHHszmQ II QZDOM mEmBH Nauru. mandamommm mo mmgz manna-go m mania 97 me No No No ow NN eN He me NN No so No as mN oN eN no mm oe no No. me me as on em Nm NN mN HN oN Na Hm mN we oN oN oN oN No Ne mm N< ea e m N o m e m N H oeoo oeeo< mOOHumooq H onHHthma II 9758 magma-H ME MHgH-mommm mo moHHHHHHHm HHH mmHHHHHsmOO OO OO Om HO HO OO HO OO HO mm OH OH HO Om Om ammuawoumm mwmuo>< HO HO OO OO as NO OO OO OH HO HO OH HO OO HH mmmuamuumm “who: OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH HO mwmuamuumm ammm OH OH OH NH HH OH O O a O O O m H H chHumooH moszz Hy50 QMSHMMQ onHgomzH MHHwomZOU Mom Huh—“<93. m N H.348 166 FIGURE 14 COMPOSITE INFORMATION DERIVED CURVES FORMED FROM THE BEST, WORST, AND AVERAGE PERCENTAGE OF DATA IN TABLE 25 Best 100r f I 90 . 1. 46¢» 80 b 70 worst Cumulative 60J Percent of Reportable Type 50 Items 40+ 30~ 20— 1o , 1 2 3 4 5 6 7 8 § 10 11 12 13 1a 15 Locations 167 O O O O O O O O O O O O O O O O OH O O O O O O O O N N N N N H H O OO O O O O N N N O N N N H H O O OH mo OO OO OO OO OO OO OO OO HO OO ON OH NH OH OH HO mm NN ON ON ON ON ON ON ON HN OH OH OH OH O N OO mm OO HO OO OO OO OO OO NO HO HO HO OO OH HH N OO On NO HO OO ON ON ON NN ON OH OH OH OH OH N H OO mo OO OO ON ON NN ON ON ON ON ON NN ON OH O O NO Om OO OO OO OO OO OO ON ON ON ON NH OH NH OH O OO Md OH OH OH NH HH OH O O N O O O O N H ANN mHOmHv mvoo chHumooq mamuH uHO=< Hmuoa OomHmz ZoazHHHOOOO OO OO NO OO NO ON ON OO OO OO OO NO NO ON OH OOOOOOOOOO OOOOO>< OO OO OO OO ON ON OO OO ON ON ON OH OH O O OOOOOOOOOO “who: OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OO ON OOOOOOOOOO OOOO OH OH OH NH HH OH O O N O O O O N H mCOHumooq Q0398; 282.4% I Oggo OWN/HOMO ZOHHOOZMONHZH mHHOOOZOU mom €531 ON mHOOO NO NO NO NO NO OO OO OO OO OO OO OO OO NO HO OOOeOOOOO O OO OO HO HO OO ON ON ON NN NN OO OO HO O O EOOOOO NO NO NO NO NO NO HO HO NN OO OO OO OO ON O OOHO>OO NO NO NO NO NO NO NO OO OO OO ON OO OO OO HO OOOEOOOOO O OO NO NO OO ON ON OO OO OO OO OO ON NN NN OH SOOOOO HO HO HO HO OO OO NO ON ON ON OO OO NO O N OONO>OO HO HO OO OO OO NO OO OO OO OO HN OO NO OO NN OOOaOOOOO O OH OH OH NH HH OH O O N O O O O N H OOOOOz OOOO mcofimwooq uHO=< NOV NNO NHO ZOHHHHmm OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OO unmemOOSO H OOH OOH OOH OOH OOH OO OO OO OO OO OO OO OO NH NH aouamm OOH OOH OOH OOH OOH OOH OOH OOH NO OO OO O O O O wmum>mm OOH OOH OOH OOH OOH OOH OOH OOH OO NO NO NO NO NO NH namsmOOSO O OO OO OO OO ON ON ON OO ON ON ON OH OH O O Eovcmm OOH OOH OO OO ON ON ON ON OO OO OO OH OH OH O wmuw>mm OOH OOH OOH OO OO OO OO OO ON ON ON ON OO OO OO uamEmOODO O OO OO OO OO OO OO OO OO ON ON OO OO HO NO ON Eovcmm OO OO OO OO OO OO OO ON HN OO OO OO OO HO ON wmum>mm OO OO OO OO OO OO OO OO OO OO OO OO ON OO OO ucwamOOSO O OO NO OO OO OO OO NN NN ON OO OO NO OO OO ON Eovcmm OO OO OO OO OO OO OO OO NN NN ON OO OO NN NH mmum>mm OO OO OO OO OO OO OO OO OO OO ON ON OO NO OO acmemOOSO O OH OH OH NH HH OH O O N O O O O N H Oosumz mvoo mGOHumooH qus< NOV NNV NHO OOOONDOOOIION OHOOO 175 method. To illustrate the use of the table, consider the data provided for subject matter audit B. When the judg- ment method for selecting the audit locations is used, 41 percent of the reportable type items contained in the uni- verse of locations is detected at the first location. At the second location an additional 15 percent of the report— able type items from the universe of locations, but not detected at the first location, are in evidence for a cumulative detection of 56 percent of the items over the first two locations. By the eighth location, 94 percent of all reportable type items in the universe of locations are detected. The percentages recorded by location for the reverse and random methods are interpreted in the same way. Obviously the data are now available both to com— pare the effectiveness of judgment and random sample selec— tion methods, and to make some assessment of the effective— ness of the criteria used to select and rank locations when the judgment method is used. However, as was the case with the data relating to the shape of the information derived curves, evaluation of these data is delayed and discussed in the next section of this chapter. To this point the results of research that will allow one to establish beyond a reasonable doubt the shape 176 of the information derived curve, to determine if there is a significant statistical difference between information derived curves developed using audit data generated at locations judgmentally selected as being representative of the pOpulation of locations, and those curves develOped using audit data generated at locations randomly selected from the population of locations, and to assess the effec- tiveness of the criteria used to judgmentally select audit locations from the universe of locations have been pre- sented. Now we will consider the results of research that should allow the extension of knowledge as to the amount of material information that is available from small samples of audit locations, and the establishment of a range of the number of sample locations that should be used for each subject matter audit. In outlining the phase III research, it was stated that an analysis was made to determine what percent of the information contained in the final report of audit for the subject matter audit could have been provided to Operating managers based on information derived from the sample of locations. The percentage was determined at the end of the first five locations, at the end of the tenth location, and at the end of the fifteenth location for each sample of locations. To judge the materiality of the information 177 detected by each sample of locations, tests described on Table 21, page 158 were used. The results of this analysis are provided in Table 30, page 178. NOte that the table contains the combined data of both phase II and III of the research since the criteria for evaluating the information are identical. To illustrate the use of this table, consider the data recorded for subject matter audit AJ. Had the subject matter audit been restricted to five judgmentally selected locations, these five locations would have provided suf— ficient material information to report 56 percent of the total information that was included in the report of audit for the complete subject matter audit. Had the number of locations in the judgmental sample been expanded to ten, the amount of reportable information would have been 71 percent. For fifteen locations, 84 percent of the reported information could have been provided. On an average, about 71 percent of the information contained in the report of audit could be reported using a five location sample. For ten and fifteen location samples, these percentages are 80 and 95 percent respectively. While an audit manager may be comfortable with the averages based on these actual results, the very cautious might reasonably ask what are the lowest percentages of information that the analysis 178 TABLE 30 PERCENTAGE OF REPORTED INFORMATION AVAILABLE FROM SAMPLES Sample Size Audit Code Five Locations Ten Locations Fifteen Locations AJ 56 71 84 BJ 59 69 90 CJ 80 80 100 DJ 50 61 83 EJ 50 75 100 FJ 100 100 100 GJ 100 100 100 HG 50 100 100 IJ 100 100 100 A2 63 73 B2 68 74 C2 73 73 D2 70 70 Average: 71 80 95 AR 7 27 36 BR 28 31 69 CR 0 20 20 DR 28 44 83 179 indicates are present? These percentages are 50 percent at five locations, 61 percent at 10 locations, and 83 per- cent at fifteen locations -- still an extremely high per- centage of reportable information from a very low percentage of the universe of audit locations. Finally, for compari- son purposes, the last four lines of Table 30 provide for four subject matter audits, the percentages of information that could have been reported had random methods been used to select the audit locations. To determine a range of locations for a given sub- ject matter audit for which cost can be considered reason- able, we must have data indicating both the amount of infor- mation that can be derived from each sample of locations of a given size, and the expected cost. The "amount of infor- mation" in the above analysis is defined and measured in two ways. In Table 30 the "amount of information" refers to the percentage of reported information that could also be reported based on information derived from the sample of locations. In Table 24 the "amount of information" refers to the percentage of reportable type items that can be detected using various samples of locations. Each of these interpretations .is used in turn in the following presenta- tions. Cost can be indicated either in terms of dollars 180 and cents, or in terms of manhours of audit time used which can, in turn, be converted to dollars and cents. Since the governmental internal auditor normally schedules audits based on expected audit manhours to be consumed rather than on dollar estimates, costs are expressed in this analysis in terms of audit manhours consumed. But how many manhours are used in a typical audit? The quickest reply one is likely to be given is that this question cannot be answered for there are no typical audits. Subject matter audits vary in the time required to develOp an audit program, in the hours required to apply the pro- gram at each location, and in the number of locations at which the audit is accomplished. For planning purposes, however, audit managers during phase I of the research suggested the following number of hours: Develop Program Locations ggggs .1922; Initial Deve10pment l x 280 Field Test Program 3 x 200 Finalize Program 1 x ‘lgg 1.000 Establish Hours Per Location 1 x 200 Determine Number of Locations 150 x 200 30,000 181 Accepting these suggested hours, the cost of sampling curve originally hypothesized in Figure 2, page 11, can be con— structed as shown in Figure 16, page 182. The curve does not start at the axis intersection but at a point three percentage points up the scale due to fixed costs in the develOpment of the audit program (1000 fixed cost hours divided by 31,000 total hours). Cost generated by application at each location is stable (200 hours per location), resulting in a steady rise in the cumulative cost of the subject matter audit through the final location, at which time 100 percent of costs are expended. Sufficient data is now available to meet the objec- tive of determining a range of locations for inclusion in each sample of audit locations selected from the possible universe of locations -- a range beyond which the incre- mental cost of information tends to exceed the value of the incremental information. To do so we need only to simul— taneously graph both information and cost. Figure 17, page 183, provides one view of this graphed data. The cost curve is drawn as previously described. The information curve is plotted using the average percentage of reported information available from sample sizes of five, ten, and fifteen locations as reported in Table 30. The curve is incomplete due to the 182 msOHumooa OOH cOH OOH ONH OHH OOH om OO ON OO OO OO OO ON OH O P F p b TOO .os umou OOO mo unmoumm m>HumHsaso .OO OOO .OOH HHH mmMDU UzHgmzMDU ZOHHHumHsEso 184 limited sc0pe of the sample sizes used in the research. The vertical dashed lines indicate visually a range of loca- tions in which information appears to be maximized for the costs incurred. Defining information as the detection of reportable type items provides a second view of the integrated data in Figure 18, page 185. Again, the cost curve is drawn as previously described. The information curve is plotted, however, using data shown in Table 31, page 186. Specifi- cally, the information curve is constructed using the aver- age cumulative percentages of Table 31 for a five, ten, and fifteen location sample. Note that Table 31 incorporates the research from both phase II and III. This curve is also incomplete due to the limited sc0pe of the sample sizes used in the research. The vertical dashed lines indicate visually the same range of locations in which information appears to be maximized for the costs incurred. Both figures indicate that the hypothesized "n" of Figure 3 (see page 12) lies in a range between ten and twenty locations. By the fifteenth location, on the aver- age, 95 percent of the information that is recorded in the report of audit for the subject matter audit can be in— cluded in a report of audit based on the judgmentally selected sample. Also by the fifteenth location, 96 185 mGOHumooq OOH Om OO ON OO OO OO OO Olumoo SOHumEoOcHTINIY . HH I m>mso ZOHHHumHsaso 186 Oo Oo Oo Oo Oo Oo so Ho Oo NO OO NN NO OO OO OOOHO>O oo Oo Oo Oo so Oo oO NO OO on NO Oo Oo No Ho OO sO ON OO Ns No NO oO oO so NO HO oN oN sO OO mo NO so so so No No No OO OO OO Hs NO OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OOH OO OH OOH OOH OOH OOH OOH OOH OOH OOH OO NO NO NO NO NO NH om OOH OOH OOH Oo Oo oo Oo OO ON ON ON ON OO OO OO OO Oo oo oo Oo mo Oo oo Oo Oo Oo mm mm ON oO so no mo mo Oo Oo oo no mo mm mm OO ON ON OO Ns OO Om oo Oo Oo Oo Oo Oo ow sO OO ON HN oO OO OO Om OO No No No No Oo so so so so oO OO OO OO NO HO so No No No No No No No so so sO ON oO OO OO Hs OO Ho Ho oO oO oO NO so OO OO OO HN OO Ns Os NN OO OH sH OH NH HH OH o O N O O s O N H OOOO mGOOuMUOH uHO=< HO mqmHH¢HDEDo 187 percent of the reportable type items for the subject matter audit can be detected. In other words, regardless of the way "information" is defined, on the average one can expect 95 percent or more of the information for, assuming 150 locations, just over 10 percent of the cost of a complete subject matter audit. To assure achievement of the remain- ing five percent of the information auditors must incur approximately 90 percent of the total audit cost -- clearly an uneconomical procedure. Again, more will be said about these data in the evaluation of research results presented in the following section. An Evaluation of Results Overview As stated previously, one of the most important stages of a research project is evaluation of results. There the important question of what does it all mean is answered. The phase III research has been extensive as evidenced by the many tables of data and illustrative figures. The following evaluation of these data in brief discloses: 1. that when, as audit managers suggest, report- able type items are defined as Operating errors or poor management practices that 188 occur at ten percent or more of the locations in the universe of locations, and when judg- mentally selected samples of locations are used to detect these items, then the result- ing information derived curves for each sub- ject matter audit are similarly shaped. that information derived curves develOped using audit data generated at locations judgmentally selected as being representative of the pOpu- lation of locations are statistically superior beyond the hypothesized alpha level to those curves developed using audit data generated at locations randomly selected from the population of locations. Specifically, it was hypothesized that if one could expect on the average the in- formation derived curves developed using audit data generated at locations judgmentally selected to lie above those curves developed using audit data generated at locations randomly selected 90 or more times in each 100 compari- sons (a = .10), then the curves develOped from judgmental data would be considered statisti- cally superior. For the nine subject matter audits analyzed in phase III of the research, 189 all nine judgmental curves are above the random curves. The computations on page 202 indicate that this result would occur only 19 times in 10,000 repetitions if either curve were truly likely to lie above the other. In other words, the computed result of .0019 is clearly signifi— cant beyond the .10 alpha level. that general criteria can be develOped to assist in the judgmental selection of locations for inclusion in each audit sample. that approximately 95 percent of the reported management problems can be detected with suf- ficient materiality to warrant inclusion in a report of audit using judgmentally selected audit samples of fifteen locations. that to maximize information for the costs incurred audit samples of from 10 to 20 loca— tions are indicated. For audit samples in excess of this approximate range, incremental cost of audit tends to exceed the value of the incremental information. that a table can be develOped from the empiri- cal research that will permit Objective rather than the customary subjective decisions in the 190 management of internal audit resources. Specifically, consideration of data within the table suggests important new objective methods for determining management areas in need of audit attention, for handling the field test of new audit programs, and accomplishing what are here called "segmented audits". These disclosures are develOped in detail in the remaining portions of this section. Since each of these six disclosures is related to the six objectives specified for this phase of the research, the evaluation of data that follows is organized and presented under the objective to which it relates. Objective One The first objective of phase III of the research is to establish that when reportable type items are defined as Operating errors or poor management practices that occur at ten percent or more of the locations in the universe of locations, and when judgmentally selected samples of loca- tions are used to detect these items, then the resulting information derived curves for each subject matter audit are similarly shaped. Recall that in phase II of the research audit 191 managers were asked how many replications of the informa- tion derived curve constructed from judgmentally selected data would be required before they would be reasonably cer- tain of its shape over a span of the first ten locations. Considering the research results of phase I, their profes- sional Opinion was that three or four additional curves would be sufficient. Accordingly, four additional curves were constructed. The results of that research as recorded in Table 12, page 106 and in Figure 13, page 107 clearly indicate that the information derived curves for the subject matter audits selected for analysis are similarly shaped and that all the curves rise very rapidly over the first few loca— tions included in the sample of locations. It was pointed out that the curves tend to level off between the fourth and fifth locations. Although audit managers at the Air Force Audit Agency were satisfied that the shape of the information derived curve was reasonably determined, the phase III research serves to strengthen these conclusions. Data for the judgmentally selected samples are con- tained in Tables 24 and 25, and in Figure 14 found on pages 164, 165, and 166 respectively. That the information de— rived curves constructed using data from phase II of the 192 research are very similar to the information derived curves constructed using data from the nine subject matter audits of phase III of the research is evident from a comparison of the average percentages for these curves recorded in Table 18 and in Table 25. This comparison is as follows: TABLE 32 AVERAGE CUMULATIVE PERCENTAGES OF INFORMATION DERIVED CURVES - PHASE II AND III - Number Research Phase of (Table 18) (Table 25) (Table 31) Locations 11 III II and III 1 35 36 35 2 55 54 55 3 72 63 67 4 83 72 77 5 87 76 83 6 89 83 87 7 9O 87 - 90 8 92 89 91 9 94 92 94 10 .95 94 95 ll 95 95 12 95 95 13 95 95 14 96 96 15 96 96 193 The information derived curves rise very rapidly over the first few locations of the judgmentally selected sample. Lodking at the line for ten locations, the phase III re- sults indicate that one might reasonably expect to detect about 94 percent of all reportable type items while the phase II results indicate an average detection of about 95 percent of the items. The combined averages for phase II and III (see Table 31, page 186) bear the same character— istics as do the averages for phase II and phase III‘ individually. One may reasonably conclude that if audit managers are willing to accept the average percentages recorded in Table 18, then the percentages recorded in Table 25 that are based on data from more than twice the number of sub- ject matter audits, or the data in Table 31 based on the results of both phases of the research should certainly be accepted. Some question may be raised however, concerning the range between the best and worst percentages of report— able type items detected as the number of locations is ex— panded. Table 18, column 2 indicates that the known range of the percentage of management prOblems that can be detected using the selected number of locations in phase II of the research is very small. By comparing the best and 194 worst percentage data recorded in Table 25, one can observe that the range is larger. It is important to the purpose here, however, to note that the larger range results from an extremely high "best percentage." The "average percent- age" data for both phase II and III of the research are similar, and the "worst percentages" reported are very high reaching, for example, 87 cumulative percent by the tenth location. None the less, two questions should be explored in more detail. Of what importance is this observation to the research? Is there an explanation for these range differences? The importance of the range is a function of the intended use of the information derived curves. This intended use is discussed briefly on pages 129-133 and will be elaborated upon in Chapters V-VII. Suffice it here to say that the curve is intended to provide audit managers with a method of making objective choices between quantity and cost of information. The curve is developed from an empirical examination of the types of audits normally accomplished and is prOposed on the premise that similar management systemm will be subject to future audit -- similar as to the scope of audit and as to the kind of information that will be of interest to tOp management. To illustrate the importance of the range between 195 the best and worst percentage of detection of reportable type items for a given sample of locations, consider the following example. Assume that an audit is to be field tested at seven locations. Table 18, page 122 indicates that, based on a past empirical review of results from similar audits, about 90 percent of the management problems of interest to top level management that are present in the universe of locations will be detected. The known range of such detection is no worse than 82 percent and has been as high as 95 percent. If the field test results do not indi- cate any management problems, one can reasonably conclude, even if only the worst past results are used as a guide, that an application of the subject matter audit to other locations would be fruitless. On the other hand, had the known range at seven locations been much broader, say from 1 percent to 99 percent, the decision to terminate the audit could not be made with as much confidence. Is there an explanation for the larger range dif- ferences as evidenced by the data recorded in Table 25, page 165? More importantly, can this range be reduced? A careful review of the data suggests that one can answer yes to both questions. The subject matter audits analyzed in each phase of the research have one criterion in common. Each audit is 196 applied at a universe of 40 or more locations. This cri— terion, as stated before, is used for two reasons. First, to apply the research in subject matter audits that are most typical of the internal auditor's work. Within the Air Force subject matter audits are usually accomplished in management systems that are in use at 40 or more locations. And second, to demonstrate that the resulting information derived curves rise very rapidly over the first few judg- mentally selected locations no matter how large the number of locations used in the audit. Perhaps, however, an additional criterion should be used. Subject matter audits selected by audit managers for the first two phases of the research as representative of the internal auditors work meet not only the above criter- ion, but also apply to audits that produced 30 or more reportable type items. The number of reportable type items may reasonably be considered a function of the number of management areas to be reviewed, or stated another way, the number of audit steps the internal auditor must accomplish. The experienced audit manager will agree that usually sub— ject matter audits are of sufficient sc0pe to produce 30 or more reportable type items. Occasionally audit programs are develOped to review a very limited number of management practices -- a number that could not result in at least 30 197 reportable type items -- but this is more exceptional than typical. In phase III of the research, data for all nine subject matter audits that met the first criterion -- applied at a universe of 40 or more locations -- were analyzed. A review of data recorded in Table 23, page 163, discloses that six of these nine subject matter audits also met the additional criterion —- research applied in subject matter audits that produced 30 or more reportable type items. Three of the nine subject matter audits, however, did not produce at least 30 reportable type items. Specifi- cally, audits GJ, HJ, and IJ produced 10, 6, and 9 report- able type items respectively. In each case these small numbers of items resulted from very limited audit programs. That audit programs involving a very limited number of audit steps can radically influence the percentage ranges we seek to develop can be illustrated by the follow— ing example. Assume that a subject matter audit has but one reportable type item. If this item is not in evidence at the first location the percentage of detection is zero. If, however, the item is detected at the second location, the cumulative percentage would jump to 100 percent. By contrast, if the subject matter audit has 30 reportable type items, the detection of the first item will increase 198 the cumulative percentage detection of reportable type items by only three percent. By eliminating data for audits GJ, HJ, and IJ from those subject matter audits that are considered more normal in sc0pe, and by combining the research results of both phase I and II, Table 33, page 199 is prepared. Like Table 18, page 122, Table 33 is a table for selecting the number of locations for audit when the objective is to detect management problems that occur at ten percent or more of the universe of locations. Note particularly that even though data for two and a half times the number of subject matter audits are used in the analysis, the known range of percentages reported in column 2 of Table 33 is very similar to the known range of percentages reported in column 2 of Table 18. Further, the averages reported in column 3 of each table are also very similar and exhibit the described characteristics of the information derived curves. One can reasonably conclude that: (1) when interest is in what might be called the normal subject matter audit, one that is applied at 40 or more locations and will likely result in 30 or more reportable type items, and (2) when, as audit managers suggest, reportable type items are defined as Operating errors or poor management practices that occur at ten percent or more of the locations in the 199 TABLE 33 TABLE FOR SELECTING THE NUMBER OF LOCATIONS FOR AUDIT WHEN THE OBJECTIVE IS TO DETECT MANAGEMENT PROBLEMS THAT OCCUR AT TEN PERCENT OR MORE OF ALL LOCATIONS - PHASE I AND II RESEARCH - - 30 OR MORE REPORTABLE TYPE ITEMS PER AUDIT - (l) (2) (3) Known Range of the Per- centage of Management Average Percentage of Problems That Can Be Management Problems Number Detected Using The That Can Be Detected of Selected Number Using the Selected Locations of Locations Number of Locations 1 27 - 51 35 2 40 - 63 54 3 47 - 82 68 4 56 - 89 77 5 70 - 93 83 6 76 - 95 88 7 80 - 98 90 8 80 - 98 91 9 84 - 98 94 10 87 - 99 95 ll 89 - 99 95 12 89 - 99 95 13 89 - 99 95 14 91 - 99 96 15 91 - 99 96 200 universe of locations, and (3) when judgmentally selected samples of locations are used to detect these items, then the resulting information derived curves for each subject matter audit are similarly shaped. These curves rise very rapidly over the first few locations, providing, for example, an average detection of 88 percent of all reportable type items through the fifth location included in the sample. These findings as to the shape of the information derived curve have important implications for the manage- ment of Air Force internal audit resources. But before discussing these implications we must assure ourselves that the information derived curve constructed using judgmental methods for selecting audit locations is best suited to our purposes —- that it is superior to an information derived curve constructed using random methods for selecting audit locations. Objective Two The second objective of phase III of the research is to determine if there is a significant statistical dif- ference between information derived curves develOped using audit data generated at locations judgmentally selected as being representative of the population of locations, and 201 those curves develOped using audit data generated at loca- tions randomly selected from the population of locations. Although in the beginning it was hypothesized that curves constructed from judgmentally selected data would rise more rapidly and lie above curves constructed from randomly selected data, there was no knowledge of which curve, if either, would exhibit these characteristics. If either curve is equally likely to lie above the other, then the probability of one curve being higher than the other for each subject matter audit reviewed is .5. In this described situation one has the ingredients for a binomial experiment. The experiment consists of nine trials or a review of the data for nine subject matter audits. Each trial re- sults in one of two outcomes: the curve using the judgment method to select the data does or does not lie above the curve using random selection techniques. Since there is no prior knowledge of the frequency with which one curve lies above the other, one must assume that either curve is equally likely. The subject matter audits reviewed are independent and interest is in the number of times the curve using judgmental methods lies above the curve that uses the random selection techniques. The formula for the probability distribution for 202 the binomial experiment is: n n- p (y) = Cy pyq y probability of success where: p y = the number of successes observed (one curve lies above the other) n = number of trials (number of subject matter audits reviewed) q = probability of failure The data in Table 29, page 173, indicate that infor— mation derived curves constructed from judgmentally selected data lie above the curves constructed from randomly selected data for all nine subject matter audits. In terms of the formula above, this is equivalent to asking what is the probability that one specific curve will lie above the other in nine consecutive trials. The computations are as follows: p (9) 0 9 9 09 (.5) (.5) 91 53— (5)9 = (.509 = .0019 These computations indicate that if it is truly equally likely that one curve will lie above the other, then the result of the experiment would have occurred no more than 19 times in each 10,000 repetitions. In other words, it is very evident that information derived curves constructed 203 using data judgmentally selected will lie above those curves constructed using randomly selected data. One criticism of the above method of showing statis- tical significance is that it does not make full use of all the data from each exPeriment. Specifically, data are used to indicate direction, that one curve lies above or below another, but data is not used that indicate the magnitude of the direction, how far one curve lies above the other. There are statistical methods for quantifying these differ- ences. However, I believe that for the purposes sought by this research it is best to display these magnitudes visually. Combining the composite information derived curves formed from the best, worst, and average percentage of data for the judgmental method of selecting locations for inclu— sion in each subject matter audit (Figure 14) with the composite information derived curves formed from the best, worst, and average percentage of data for the random method of selecting these locations (Figure 15) provides Figure 19, page 204. Note particularly that until the fifteenth loca— tion the average detection of reportable type items based on random methods of selecting audit locations lies below the worst percentages of detection of reportable type items based on judgmental methods of selecting audit locations. Figure 19 utilizes the information curves of all Cumulative Percent of Reportable Type Items 204 FIGURE 19 COMPOSITE INFORMATION DERIVED CURVES RESULTING FROM BOTH JUDGMENTAL AND RANDOM SELECTION METHODS lOOI , ..... . _____________ 90 80 70 50 4d 20 1 Judgmental Method I -------- Random Method 7 8 9 16 11 12 13 14 15 Locations 205 nine subject matter audits included in phase III of the research. When attention has been reduced to what have been called the six "normal" subject matter audits —- those that are both applied at 40 or more locations and will likely result in 30 or more reportable type items -— the differences between the curves become even sharper as shown in Figure 20, page 206. Note that the best results using random methods are very similar to the worst results using judgmental methods. And again, the average results using random methods are not as good as the worst results using judgmental methods until the fifteenth location is reached. One can conclude that when the objective is to determine from a sample of 15 or less locations the report- able type items in the universe of locations, a greater percentage of these items will be detected when the loca— tions are judgmentally selected. Worded in terms of the first objective, it has been demonstrated that an informa— tion derived curve constructed using judgmental methods for selecting audit locations is superior to an information derived curve constructed using random methods for select- ing audit locations. Accepting the superiority of the judgmental method one must now ask if we can do a good job in making judg- mental selections Of locations that will provide in the Cumulative Percent of Reportable Type 50 Items 206 FIGURE 20 COMPOSITE INFORMATION DERIVED CURVES FOR THE SIX NORMAL AUDITS 100 ,... ,"’ / 9° . "”777; H" "I’ ,H/ ’ $96 /’ H/ ’1’ 80 , I ./’ /"‘” 3" 4‘. / //I / «I ’ r 9' QO/’ 6'" 0 I/ ’ ’/ 70‘ / $9 {9%/"’ / a' 4% ’- I }/ I I ’/ $1 ’ 9/ 60 [I I,’ 90'", I ’ ’ [-- l / / / I I I f / / / I / 4O ,’ I, / I I, / 3o /’ .- 1’ / I I’ I l / I / I 20 I ./ I ———__—__.Judgmental Method l’,’ I, -------- Random Method I 10 I I I / I j 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Locations 207 future the same results observed in the empirical review of past records. Stated another way, can effective criteria be identified to guide these selections? Objective Three The third objective of phase III of the research is to identify criteria to select judgmentally audit locations from the universe of locations. With regard to using a judgmental approach to select the sample, there is a central guiding criterion. The specific locations should be chosen in such a way as to pro- vide information to tOp management which is representative of the total information system under audit. Criteria for selecting locations were identified by Air Force Audit Agency personnel and used by audit managers to select judg- mentally each sample of locations for the nine subject matter audits analyzed. It was hypothesized that if the selection criteria are apprOpriate and if they have been applied properly by audit managers, then information derived curves resulting from analyzing audit data from.the locations in the order ranked will initially rise somewhat faster as additional locations are added to the sample of locations than will such curves resulting from analyzing audit data from these 208 same locations but in the reverse order of their ranking, i.e. the fifteenth location becomes the first, the four— teenth becomes the second, and so on. The data in Table 29, page 173, indicates that this is precisely the case. A comparison of the cumulative per- centages over the first five locations for the "judgment" and "reverse" methods discloses that for all nine subject matter audits the judgment method, using the criteria identified by Air Force Audit Agency personnel, results in higher cumulative percentages. Over the first ten loca— tions the "reverse" percentages rise above the "judgment" percentages for only one audit, audit D. One can reasonably conclude that criteria can be and have been identified that provide the ability to select judgmentally and to rank locations according to their representativeness in the information system under audit. One cannot conclude that the criteria identified are the best or most useful to accomplish the objective, but they are at least minimally sufficient for effective choices to be made. To this point in the analysis we have established the shape of the information derived curve and have dis- cussed its characteristics. Judgmental methods have been demonstrated to be superior in the develOpment of these 209 curves, and criteria for accomplishing the judgmental selections have been identified. The next step is to show how use of this curve can result in more effective and effi- cient management of internal audit resources. But before launching into such a discussion, let us digress and evaluate one other area of interest to the internal audit manager. Specifically, how much of the in— formation that was contained in reports of audit could have been reported based on information derived from a sample of locations? And a closely related question, does the research indicate a sample range beyond which the incremental cost of audit tends to exceed the value of the incremental informa- tion? Turn now to the first of these two questions. Objective Four The fourth objective of phase III of the research is to extend knowledge concerning the percentage of infor- mation contained in the report of audit for the subject matter area that could have been reported based on informa- tion derived from judgmentally selected samples of five, ten, and fifteen locations. The research, summarized in Table 30, page 178, indicates that for five, ten, and fifteen judgmentally selected samples, the average percent of information that 210 one may reasonably be expected to report is 71, 80, and 95 percent respectively. The very cautious observer will note that the lowest percent that is reported from the data analyzed is a reasonably high 50 percent for a five loca- tion sample, 61 percent for a ten location sample, and 83 percent for a 15 location sample. By contrast, if the locations are randomly rather than judgmentally selected, data recorded in the last four lines of Table 30 provide an indication of the probable quality of the result. For example, the percentage of reported information available from a fifteen location sample for audit AR is considerably less than the result that is achieved by a five location sample in audit AJ. The result achieved by a fifteen location sample for audit BR is matched by a ten location sample for audit BJ. The result achieved by a fifteen location sample for audit CR is no better than twenty five percent of the result achieved by a five location sample for audit CJ. Again, the cautious observer will note that the lowest percent that is reported for the four audits analyzed is a very low zero percent for a five location sample, twenty percent for a ten location sample, and twenty percent for a fifteen location sample. These results should not be entirely uneXpected. It has been pointed out that the rate at which reportable 211 type items are detected using judgmental methods of select- ing sample locations is consistently higher than the rate at which items are detected using random methods of select— ing sample locations. Since these reportable type items can be eXpected to form the basis for the report of audit, one should normally expect a greater percentage of this reported information to be available from judgmentally selected samples. The conclusion that can be reasonably drawn from these data is that if judgmental rather than random methods of selecting audit locations are used, a very high percent— age of the reportable information is available from a small percentage of the universe of locations. Assuming a uni- verse of 150 locations, about 95 percent of the reportable information is available from a sample composed of only 10 percent of all locations. This observation leads to the next objective. Objective Five The fifth objective of phase III of the research is to determine a range of sample locations beyond which the incremental cost of audit tends to exceed the value of the incremental information. Figure 17, page 183, and Figure 18, page 185, both 212 indicate that the hypothesized "n" of Figure 3, page 12, lies in a range between ten and twenty locations. As has been already observed, by the fifteenth location, on the average, 95 percent of the information that is recorded in the report of audit for the subject matter audit can be in- cluded in a report of audit based on the judgmentally selected sample. Also by the fifteenth location, 96 per- cent of the reportable type items for the subject matter audit can be detected. In other words, regardless of the way "information" is defined, on the average one can expect 95 percent or more of the information for (assuming a uni- verse of 150 locations) just over 10 percent of the cost of a complete subject matter audit. To insure achievement of the remaining five percent of the information auditors must incur approximately 90 percent of the total audit cost —— an economically questionable procedure. Following the limited research contained in phase I, the Air Force Audit Agency adOpted one major change in audit method. The number of locations to be selected for certain planned subject matter audits was reduced to 60, about one- third of those previously included. It was stated at that time that this is only a step toward greater economical audit procedures. Research in more depth should provide more effective guidance for the selection of sample sizes. 213 The present research provides this guidance. Limiting subject matter audits to from ten to twenty judg- mentally selected locations clearly meets the basic criteria of effectiveness and efficiency. The sample range is effec- tive in that most of the information of interest to tOp management can be found, and it is efficient in that virtu— ally the same report of audit is produced at reduced cost. But the research provides more than this. If one can reasonably assume that the audit environment of the past several years is similar to the audit environment of today, then there is a reasonably objective method of quantifying audit decisions that have to now been made based on subjective judgments. For example, if a subject matter audit is planned in a management system that is in use at more than 40 locations, the tables provided in the research indicate that one can expect from a sample of fifteen judgmentally selected locations to include in the report of audit about 95 percent of the information that would be included if all locations were audited. If random methods are used to select the locations, an approximation of the loss of reportable type information can be made. If for some reason some level of management insists upon audit at the universe of locations and the audit agency is reluc- tant to comply, arguments can be phrased in terms of known 214 incremental cost for the additional information provided. In other words, it is my thesis that simply because auditors are involved in a management area that is not well suited to objective random statistical methods, this does not mean that purely subjective decisions are the only alternative. The decision variables can be quantified. Objective data can and are here provided to make these decisions, not with the exactitude of a precise mathematical model, but with the precision that objectivity can add to reason and experience. Objective Six The sixth and last objective of this phase of the research is to develOp a table, based on an analysis of empirical data, that will provide audit managers a means of objectively determining the number of sample locations needed from a universe of locations to discover a specified portion of all reportable type items. This objective is achieved by the construction of Table 33, page 199. To illustrate the use of the table, assume that audit managers decide to use a sample of seven locations. First locate the number seven under the first column of the table. Reading across, note that judgmentally selected samples of seven locations have in the past 215 detected from 80 to 98 percent of all reportable type items. On the average, 90 percent of the reportable type items may reasonably be expected to be detected. Conversely, assume that one wishes to use a judgmental sample of all locations that can reasonably be expected to result in the detection of about 75 percent of all reportable type items. Reading down the third column of the table locate the percentage closest to this goal, in this case, 77 percent. Reading left across the row it is noted that a sample of four loca— tions has a reasonable chance of providing this objective. Although the table may be used directly or indi- rectly in a number of ways to assist in the management of internal audit resources, three uses will be discussed in the next three chapters. These uses relate to the deter— mination of management areas in need of audit attention, to the management of field tests of new audit programs, and to the accomplishment of segmented audits. These uses of the table are not mutually exclusive. That is, use of the table for one purpose may preclude the necessity of using the table for another purpose. For ex- ample, a method of audit management that uses the table in determining areas in need of audit attention could delete the necessity for a method of audit management that uses the table in managing field tests of new audit programs. 216 Each proposed method for managing internal audit resources will be discussed separately. It is for the interested reader to select the method or methods that will be the most useful. The three prOposed methods for managing internal audit resources are discussed in Chapters V-VII that follow —— discussions to which we now turn. 217 FOOTNOTES This policy is reflected in the following reference: Air Force Audit Agency Regulation 23-6, Air Force Agency Organization and Functions, Norton Air Force Base, California, 1972, paragraph 5, page 40. CHAPTER V USE OF THE INFORMATION DERIVED CURVES - IN DETERMINING MANAGEMENT AREAS IN NEED OF AUDIT ATTENTION The purpose of this chapter is to demonstrate through reasonable argument that data provided by the infor- mation derived curves can be used objectively to determine management areas in need of audit attention. The chapter is organized into four sections. The first section elabor- ates upon the objectives sought. The second section pro— vides background as to how management areas are selected for audit by the Air Force Audit Agency. In section three a hypothetical procedure for selecting audit areas is dis— cussed. The chapter concludes with an evaluation of the prOposal. The Objective As stated, the purpose of this chapter is to prOpose a method of determining management areas in need of audit attention. The primary instrument of this prOposal is the information derived curve previously develOped and 218 219 summarized in Table 33, page 199. Specifically, the objective is to demonstrate by reasonable argument that by using the data provided in Table 33, a method can be develOped to determine objectively the extent of the management problems that will be commented upon in a subsequent subject matter audit REESE to the full I commitment of audit resources to the audit. A full commit- ment of audit resources is interpreted to extend from the audit manager's preparation of audit program through the i completion of the resulting subject matter report of audit. The words "a method" rather than "the method" are purposely used because it is not my intention to suggest one method to the exclusion of others. Rather, the intention is to confine the discussion to an hypothetical example of a method that will suggest how the data produced by research in the previous chapters gag be used, not how it should be used to manage internal audit resources within a multi- location organization such as the Air Force Audit Agency. A specific determination of how it should be used would involve a consideration of Operating variables beyond the present sc0pe of this research. 220 Background Audit programs for subject matter audits within the Air Force Audit Agency are prepared centrally by specifi- cally assigned audit managers.1 The programs are written to review management functions at the various Operating locations around the world. Each subject matter audit, from initiation of the audit program to release of the final report of audit, will consume thousands of audit hours and can require several months of the audit manager's time. To achieve economical utilization of audit resources it is vital that these resources be expended in management areas that will produce results attractive to top level management. Management areas scheduled for coverage by a centrally directed subject matter audit are suggested by a variety of sources such as the Department of Defense, major commands, system managers, and the audit staff. All of these suggestions are well intended. Experience has shown, however, that sometimes subject matter audits completed as a result of these suggestions have not been as successful in detecting significant management weaknesses as was hOped. The fault may not lie with the audit manager nor with the application of the audit program, but with the decision process used to select that area for audit. 221 Decisions as to which management areas should be scheduled for audit have, in the past, been made primarily based upon subjective considerations. More recently, to provide some objective facts upon which to base these critical decisions, two procedures were used. The first procedure was called the Program Audit Development Survey. The second procedure is called the Directed Audit Research Task.2 But as stated in an earlier discussion of these procedures (refer to pages 134-135) a measurement device, such as Table 33, was not used to indicate to audit managers the probable extent of the reportable type items that are detected by the sample of locations used. Consequently, audit managers must rely upon their subjective assessments of the need for audit within an area. Hypothetical Procedure For Selecting Audit Areas Using the results of the research provided in previous chapters as a management tool, the following hypo— thetical example and outlined procedure suggest one method for objectively determining the extent of management prob- lems within an Operating management system. Assume that an audit of the XYZ activity is sug- gested. This activity will be in operation at nearly all of the locations to which Air Force internal auditors are 222 deployed. Assume further, that due to the current internal audit workload, it will be about six months before an audit manager will be available to provide full attention to the project.3 A review of the prOposed objectives for the audit suggest that, if accomplished, one can reasonably expect 30 or more reportable type items to be detected.4 Finally, assume that the audit is to be made in a manage- ment area for which a previous audit program has not been prepared. In other words, there are no "canned audit pro- grams" on the shelf to suggest audit tests that should be made. This latter assumption will be relaxed later in the discussion. Accepting these assumptions, the audit manager on whose schedule the subject matter audit is placed might take the following actions: 1. Identify at one of the Auditor General Resident offices an assigned internal auditor who is experienced in XYZ activity audits and capable of drafting a tentative audit program for this management area. 2. Prepare a letter identifying the audit objec- tives and requesting that the field auditor prepare a tentative program to review the management area. This letter should be drafted 223 several months before concentrated central programming work is scheduled to begin. The degree of detail contained in the letter could vary. For example, the audit manager who will eventually be reaponsible for the subject matter audit could spend from a few hours to several days clarifying the objec- tives, suggesting audit approaches, etc. Other than this, each letter should include information concerning: a. The format of the draft audit program. b. Time alloted the field auditor to develOp the program. c. The estimated number of hours that should be required for application of the finished audit program. d. Processing instructions for the draft audit program. Send the letter to the selected field auditor, requesting acknowledgment of the assignment. Review the draft audit program when received, revise as necessary, and reproduce it in the desired number of COpies. Determine the number of locations at which the audit program will be used to review the XYZ activity. This number will depend upon the 224 percentage of reportable type items that the audit manager wishes to be reasonably certain of detecting. If the manager wishes 80 percent, for example, then Table 33 indi— cates that five representative locations should be selected. In the past, for this type of audit, five representative locations have been sufficient to detect an average of 83 percent of all reportable type items. Identify the required number of locations, determined in step five, that are representa- tive of the information system to be audited. These selections should be made based on criteria such as that outlined in Chapter IV, pages 153-154. Send a c0py of the audit program to the selected locations for application. Receive from the selected locations the applied programs, supporting workpapers, local audit reports, and auditor critiques of the program. All of this packaged infor- mation should arrive prior to the date con- centrated effort on the subject matter audit by the audit manager is scheduled to begin. 225 9. Review the advanced information on this activity being considered for a subject matter audit. By following these steps the audit manager will know before resources are fully committed to the audit just where the actual management problems are and are not evi— denced. He can, therefore, design the audit program for the subject matter audit so that it can provide the most information return for the audit hours expended. He will, in short, concentrate his efforts where audit need exists. Returning to an earlier assumption, if there are usable "canned audit programs" on the shelf, then the steps necessary to make the Objective assessments concerning the extent of management problems in a given area are, of course, reduced. Essentially, the fifth step could become the first step. The requirement to send a cover letter explaining the actions desired by the deployed auditor would replace step seven. But the resultant audit reviews should provide the same valuable information to audit managers. An Evaluation In this and the succeeding two chapters four basic criteria will be used to evaluate the hypothetical methods 226 suggested. These criteria are: (1) provide the Opportunity to assess the risk of not detecting reportable type items, (2) reduce the time required to achieve the subject matter report of audit, (3) reduce the cost in terms of audit hours, and (4) provide the same information to management as is provided by an audit accomplished at the universe of locations. Each of these evaluation criteria is considered in turn. To facilitate discussion of the first criterion, Table 34, page 227 is prepared. The table is prepared using data recorded in Table 33, and therefore assumes the same type of audit. That is, the audit manager is interested in detecting management problems that occur at ten percent or more of the universe of locations and second, the audit is likely to produce 30 or more reportable type items. A rule of thumb way to judge this latter criterion is that the audit will require 100 or more hours to complete at each location. Looking at Table 34, the first column indicates the number of representative locations included in the sample. Columns two, three, and four are labeled "risk preference". Column two is labeled a low risk preference. The percent- ages in this column are obtained by subtracting the lower limit of the range of percentages recorded in column two, 227 TABLE 34 TABLE FOR DETERMINING THE PERCENTAGE OF REPORTABLE TYPE ITEMS NOT DETECTED BY SAMPLES OF LOCATIONS FROM 1 TO 15 (1) (2) (3) (4) Number of Risk Preference Locations Low Average High 1 73 65 49 2 60 46 37 3 53 32 18 4 44 23 ll 5 30 17 7 6 24 ‘ 12 5 7 20 10 2 8 20 9 2 9 16 6 2 10 13 5 1 ll 11 5 1 12 11 5 l 13 11 5 l 14 9 4 l 228 Table 33, from 100 percent. For example, for a sample of one location, Table 33 indicates a low risk preference of 73 percent. This is obtained by subtracting the lower bound of the column two range for a one location sample found in Table 33 from 100 percent (ie. 100 - 27 = 73). The remaining percentages in column two, Table 34 are com- puted in the same manner. The average risk preference per- centages recorded in column three are determined by sub— tracting the percentages recorded in column three, Table 33 from 100 percent. The high risk preference percentages recorded in column four, Table 34, are computed in the same manner as are the percentages recorded in column two except that the upper bound of column two, Table 33 is used rather than the lower bound. Now let us consider how the data in Table 34 can be interpreted and used. The interpretation is straight for- ward. Consider a sample of five locations. The data indi- cate that from the empirical research, the worst result achieved by a sample of five locations from a subject matter area is a failure to detect 30 percent of the report— able type items present in the universe of locations. On the average, there is a failure to detect 17 percent of the reportable type items, while at best only 7 percent of the items are not detected. 229 The table may be used to determine how many repre- sentative locations should be included in a sample given the degree of risk the selector wishes to assume. If the selector has a low risk preference, that is the audit cir- cumstances are such that he must detect at least a certain percentage of reportable type items with little or no margin for error, then the sample will be selected using data recorded in column two as a guide to the number of locations required. If the selector has a high risk pref— erence, that is the audit circumstances are such that he feels very confident that if there are reportable type items at all they will be in evidence in the representative sample, then the data in column four will be used as a guide to the number of locations required. Finally, if the audit is to be accomplished under what may be called average or routine circumstances, the data in column three will be used. It is not possible to specify which of the risk preferences should be used for that is a function of the characteristics of the audit situation and of the individual whose decision it is to determine the sample of locations. But regardless of which risk column is used, the importance of the table is that there is now some objective means of determining how many locations are "enough" given the degree of risk the audit manager is willing to assume. No 230 longer is the audit manager entirely dependent upon his subjective judgments as to the number of locations needed. The table effectively organizes the results of past expe- rience into a useful measurement tool -- a tool that can guide the audit manager in the economical selection of the results desired. Now the reader may point out that all that has been done in Table 34 is to present the data of Table 33 in a different arrangement. In Step five of the hypothetical example rather than use Table 33 as a guide in selecting the sample size, Table 34 could have been used just as easily. This is, of course, true. However, what is dif— ferent is the approach to the interpretation of the data. While the same end result can be achieved regardless of the table used, our total understanding can at times be improved by looking at something from more than one angle. One can conclude, however, that any hypothetical method that can make use of Table 34 to select the sample size can also meet the first criterion of the evaluation. That is, the opportunity to assess the risk of not detect- ing reportable type items is present. Now consider the second of the four evaluation criteria -- reduce the time required to achieve the subject matter report of audit. By "time required" is meant the 231 calendar number of work days extending from the date the audit manager devotes full attention to the project to the date the final subject matter report of audit is released. The amount of time that may or may not be reduced through utilization of the hypothetical method will depend upon the planning and Operating procedures followed by the audit organization. These procedures may vary between organizations and vary within the same organization for different audits. However, in general, one might expect the total time required to complete the subject matter audit will be reduced due to one or more of the following reasons: 1. The time required for the audit manager to prepare the audit program to be used will be reduced. This reduction in time may occur for one or both of two reasons. If a deployed auditor prepares a tentative audit program as suggested in steps one thru four of the hypothetical method, much of the research required to prepare the final program will already be accomplished. And second, even if a "canned audit program" is available for the initial review, the preliminary review of the area will determine those areas within the 232 subject matter audit that are in need of audit attention. Another way of stating this is to say that the preliminary review will determine those areas within the subject matter audit that are 29: in need of audit attention. Those areas can be deleted from the final audit program thereby reducing preparation time. 2. The time required to field test and revise the final audit program may be eliminated. Sufficient information from application of the tentative program may be available to delete the necessity of a field test and subsequent revision of the audit program. 3. The time required at each selected location to complete the audit program steps may be reduced. This follows from the expected reduction in the length of the program brought about by elimination of those por- tions of the proposed subject matter audit for which no reportable type items are detected by the initial review of the area. For these same reasons one might reasonably expect a reduction in the overall cost of audit manhours used to 233 produce the subject matter report of audit. The audit manager's time will be reduced due to the assistance received in the develOpment of the audit program and, since the audit program will be restricted to known prob- lem areas, less audit time will be expended at each par— ticipating location. Finally turn to the fourth and last criterion. Does the hypothetical method provide the same information to management as is provided by an audit accomplished at the universe of locations? Answering this question brings out an interesting feature of the hypothetical model. There are trade—offs in the extent to which the criteria are met. As Table 34 indicates, one can be reasonably certain of detecting most all of the reportable type items only by using relatively large samples of locations and incurring the expense that that requires. On the other hand, one may accept a somewhat larger risk of not detect— ing some reportable type items in favor of a reduction in audit cost required for a larger sample. But once again, and this is important, we are able now to more objectively quantify these decision variables. In summary, one can say that generally the first three criteria of the evaluation can be met. In doing so, however, the fourth criterion may not be met. The fourth 234 criterion may be met, however, if we are willing to sacri- fice at least some cost in doing so. But what is important to note is that the decision can be framed around reason- ably quantifiable variables rather than purely subjective guess work. - "i. r: ‘1 1. 235 FOOTNOTES Air Force Audit Agency Regulation 23-6, Air Force Audit Agency Organization and Functions, Norton Air Force Base, California, 1972, page 31. A description of the program Audit Development Survey was contained in Air Force Audit Regulation 175—101, Internal Audit Procedure, Norton Air Force Base, California, 1969, pp. 4—1 thru 4-2. A description of the Directed Audit Research Task is contained in Air Force Audit Regulation 175-105, Directed Audit Research Task, Norton Air Force Base, California, 1970. This is a reasonable planning time within the Air Force Audit Agency. Another reasonable way of determining this requirement is to state that the audit program will require 100 or more audit hours to complete at each location. The number of reportable type items are a function of the number of audit steps which are, in turn, a function of the number of audit hours allowed for the jOb at each location. CHAPTER VI USE OF THE INFORMATION DERIVED CURVES IN THE MANAGEMENT OF THE FIELD TEST OF NEW AUDIT PROGRAMS A "field test" is a term that is applied to the procedure of trying out a newly develOped audit program at a certain number of field locations. Audit programs cen- trally prepared by the Air Force Audit Agency are not always field tested prior to full scale use of the partic- ipating locations. But when they are, the research data proved in previous chapters can be used to improve manage- ment of these tests. The purpose of this chapter is to describe how the research data can be used to improve management of the field test of centrally prepared audit programs develOped to be used in the review of subject matter audit areas. The chapter is organized into three sections. The first section discusses the types of information the audit manager may seek from a field test of an audit program. There are basically two types: (1) information to assure that the prOposed audit program can be understood and 236 237 applied by the field auditor, and (2) information that will permit him to delete unproductive steps from the program. To illustrate how the research results may be used to obtain this second type of information, a hypothetical method is prOposed in section two. The impact of the prOposed method is indicated in section three where the prOposed field test method is applied to data provided by the subject matter audits analyzed in phase II of the research. The chapter concludes with an evaluation of the proposal. The Spectrum of Information Provided by_a Field Test The number of locations selected for the field test of a prOposed audit program depends upon the information the audit manager for that subject matter area wishes to derive from the test. His needs may vary in a spectrum ranging from securing only enough information to assure that the prOposed audit program as written can be under- stood and applied by the field auditor, to obtaining infor- mation that will allow him to delete those steps in the audit program that, if applied at the universe of locations, will 29: provide information of interest to top management. The audit manager presently concentrates on his need in the lower end of this spectrum, that is, securing enough information to ensure finalization of a workable 238 audit program. As a minimum, feedback concerning the following is sought: 1. The effectiveness of the tentative audit pro- gram in achieving audit objectives. 2. Suggested additions and deletions to the program. 3. Command problems that may be exPerienced in applying the program at certain locations. 4. The audit hours required to complete the pro- gram at each location. Audit managers interviewed stated that a field test at one or two locations is usually sufficient to provide this information. Each audit manager agreed that it would be helpful to move up the spectrum and to identify before hand those steps in the audit program that will and those steps that will not provide information of interest to top management if applied at the universe of locations. But they did not have a method to obtain such data. A Hypothetical Method Fortunately, these research data suggest such a method. Consider the following example that typifies the audit situation faced by the Air Force Audit Agency 239 internal audit manager. Assume that an audit manager is assigned the task of writing an audit program to review the "Z" system of management. The Z system has been in Operation for a number of years at more than 40 locations around the world. The audit manager is not certain of the degree of audit need in the various portions of the system. An audit program can be developed and applied to achieve a compre— hensive coverage of all system areas, but this could result in much of the field auditor's work being spent in areas that produce no fruitful management information. To avoid this situation, the audit manager must first decide what will be considered a fruitful finding to tOp management. In other words, how much audit "clean—up" of the system is desired? The auditor knows from experience that the most serious system weaknesses, for example those that occur at 80 or 90 percent of the locations, usually will evidence themselves at a one or two location sample. Those less serious conditions will require more locations in the field test to ensure their detection. Assume, as has been done in this research, that the audit manager for the subject matter area that includes system Z believes management can best be served by bringing to light those management problems that occur at ten percent 240 or more of the pOpulation of locations. This he believes represents a fairly comprehensive examination into the system and is in accord with management's needs. He may now turn to the data contained in Tables 33 and 34 for assistance. To be reasonably certain that through the field test he will be able to identify 90 per— cent Of the audit program steps that will result in the auditor's involvement with problems of interest to tOp management, he consults Table 33. There he notes that to achieve his objective a sample of seven judgmentally selected locations must be used in the field test. Table 34 indicates that with this sample size he runs a risk of not detecting as many as twenty percent of the audit pro— gram steps that will result in the desired type of infor— mation, but then again the detection rate could be much better than desired. Assessing these possibilities and those provided by other sample sizes, the audit manager can decide just how the field test is to be managed. The final decision as to the number of locations to include in the field test will depend upon a balance between the degree the audit manager wants to be certain of detecting those audit steps that will review actual management problems, and the economics of administering a field test of the necessary 241 size. But again, and this is repeatedly stressed, these decisions are made objectively based on past experience. Following the field test, questions in the audit program are reviewed. Those questions that detect a management problem are identified. Auditors responsible for applying the audit program at other locations will use g only those audit steps identified by the field test. The ‘ audit manager_will use the field test results to delete those steps in the audit program that will not provide information of interest to top management. An Application to the Research Dagg If such a method had been followed for those sub- ject matter audits analyzed in phase II of the research, Table 35 provides an indication of the extent portions of the audit program could have been deleted prior to the application at the universe of locations. 242 TABLE 35 RESULTS OF A SEVEN LOCATION FIELD TEST OF THE AUDIT PROGRAM (1) (2) (3) (4) Percentage of Percentage of All Findings Reportable Type Percentage not Audit in Evidence Items in Evidence in Evidence in Code (Table 8) (Table 12) column (2) A2 76 92 24 32 36 82 64 C2 78 91 22 D2 82 95 18 Er Consider the data recorded for audit B2. Column 2 indicates that 36 percent of all audit findings detected when the audit program was applied at the universe of 152 locations could have been in evidence at a seven location field test of the audit program. Column 3 indicates that included within those findings of column 2 are 82 percent of all information that will likely be of interest to top level management. Audit program steps that ultimately identified 64 percent of all total audit findings would have been completely unproductive in the field test. Had a seven location field test been made of audit B2 and acted upon as suggested above, a considerable por- E tion of the audit program that was applied in its entirety 243 at 152 locations would have been eliminated. Those elimin- ated portions of the audit program would include those audit steps that produced no information for any level of manage- ment plus those steps that produced results of interest only to managers at specific locations (that percentage recorded in Table 35, column 4). If this action had been taken, un- I doubtedly considerable savings in audit costs would have resulted.1 T“ An Evaluation Turn now to an evaluation of the field test method using the four criteria suggested in Chapter V. These criteria are: (1) provide the Opportunity to assess the risk of not detecting reportable type items, (2) reduce the time required to achieve the subject matter report of audit, (3) reduce the cost in terms of audit hours, and (4) provide the same information to management as is pro— vided by an audit accomplished at the universe of locations. The first criterion is clearly met through use of Table 34 as described in the hypothetical method. Follow- ing the assessment of the risk of not detecting reportable type items, a final decision is made as to the number of locations to include in the field test. This number depends upon a balance between the degree the audit manager 244 wants to be certain of detecting those audit steps that will review actual management problems, and the economics of administering a field test of the necessary size. The second criterion -— reduce the time required to achieve the subject matter report of audit -- may or may not be met. This depends upon whether the audit program for the subject matter audit would normally be scheduled for a field test. If so, then a reduction in time can be expected, the extent of the reduction depending upon the 1 . audit time that would have been required to apply those unproductive portions of the audit prOgram deleted follow— ing the field test. On the other hand, if the audit pro- gram is field tested for the specific purpose of refining the program, and if otherwise it would be used directly at the universe of locations, then the time required to achieve the subject matter report of audit obviously would be extended. Audit application at the universe of loca- tions would be delayed until the field test is completed. By using the field test method, cost in terms of audit hours would be reduced. Just how many hours would be saved for a given audit would be a function of the time that would have been required to apply the deleted portions of an audit program times the number of locations at which the refined audit program is applied. 245 Finally, one cannot state that the method will pro— vide the same information to management as is provided by an audit accomplished at the universe of locations. But neither can it be said that it is desirable to do so. Again, as is discussed in Chapter V, there are trade—offs in the extent to which the various criteria are met. As Tables 33 and 34 indicate, one can be reason- ably certain of detecting most all of the reportable type items only by using relatively large samples of locations and incurring the expense that that requires. In terms of the present method, this means that to insure that a step in an audit program that will result in a reportable type item if applied at the universe of locations is not eli— minated, a very large field test of the program must be made incurring the audit cost that that requires. On the other hand, one may accept a somewhat larger risk of eli- minating certain audit program steps that will detect reportable type items in favor of a reduction in the cost required for the larger field test. Just what decision the audit manager will make will depend largely upon the quantifiable decision variables provided by the tables. In summary, we can say that the first and third criteria should be met. Whether the second criterion is met is dependent upon whether the audit program for the 246 subject matter audit is normally field tested prior to application at the universe of locations. If it is, then that criterion is also met. The extent to which the fourth criterion is met is a function of the cost the audit man- ager is willing to incur. But as in Chapter V, by using the research provided this decision can be reached con— sidering reasonably quantifiable variables. 247 FOOTNOTES Based on the research accomplished it is not possible to quantify this result. To do so requires three steps. First, all steps in the audit program that will be eliminated by the field test procedure must be identified. Second, the number of hours that would have been required to accomplish these deleted steps at each location must be determined. And finally, the hours required at each location must be multiplied by the total number of locations at which the refined audit program is applied. These computa- tions must be made for each audit considered. CHAPTER VII THE SEGMENTED AUDIT Methods discussed for determining areas in need of audit attention and for managing the field test of an audit program make specific use of Tables 33 and 34 developed from information derived curves. These curves were devel- oped from judgmentally selected samples of the most repre- sentative locations for each subject matter audit included in the analysis. Each location included in the sample is selected using criteria identified on pages 153—154. From working with audit managers who made the selections it was apparent that selecting and ranking the 15 locations required careful consideration of the merits of each location. For some of the subject matter audits 30 or more locations were con— sidered by audit managers to have representative systems in operation. The problem was in the determination of the mpg; representative. Where these conditions exist one other method for the economical utilization of internal audit resources may 248 249 be considered. The method is based upon the use of several "reasonably" representative samples for a given subject matter audit rather than upon the use of one most representa- tive sample. Consequently, the data provided in Tables 33 and 34 develOped from use of the mpg: representative samples must be used as a guide only. Nevertheless, the method f” in the Opinion of audit managers to whom it was explained, makes good sense and offers sufficient promise of greater economy in the use of audit resources to warrant its con— lj‘ sideration. The purpose of this chapter is to discuss this method called the segmented audit. Accordingly, the chapter is organized into four sections. The first section develOps the background to the need for an audit method that will permit the review of a major portion of a manage- ment system without unduly infringing upon the audit hours controlled by the Resident Auditor. The segmented audit is defined and its use is illustrated by a hypothetical ex- ample. The chapter concludes with an evaluation of such audits. Background Each Auditor General Resident Office has a limited number of audit hours available for use. These hours must 250 be used for both locally scheduled and centrally directed audit efforts. If too large a portion of these limited hours is expended on centrally directed audit efforts, the Resident Auditor is handicapped in providing needed infor- mation to the base commander. Recognizing this fact, centrally directed audits T‘ are generally limited in sc0pe. Most of these audits re— quire from 100 to 200 audit hours at each selected loca- tion. Because of the present size of certain management [I activities such as base supply, base procurement, financial services, installation engineer and the like, this number of hours permits the analysis of only a small portion of each activity at any one time. Further, with the continuing integration of these activities through SOphisticated electronic data processing equipment, the resulting systems are becoming even larger. An auditing method is needed whereby the major por- tion of an activity or integrated system can be audited without unduly infringing upon the audit hours available to and managed by the Resident Auditor. If such a method can be develOped, the audit organization will thereby be provided greater flexibility in its audit capability. One such method is through use of a segmented audit. I 251 The Segmented Audit A "segmented audit" is a term coined in this research that denotes a subject matter audit that is divided into parts. Each part is to be accomplished at a different sample of locations. The audit findings from each part are to be combined and summarized into a single report of audit. Such audits permit the same audit coverage with a smaller expenditure of local audit hours or an eXpanded audit coverage from the same expenditure of local audit hours. To see how this might come about, let us explore a hypothetical example. A Hypothetical Example Accept for the moment the following simplifying assumptions. Assume that to review a major portion of a specified management system at any one location an auditor working alone would require 600 hours. At best, more than three months would be needed to complete the audit. Further assume that this system is in operation at 150 locations at which internal auditors are deployed. About one third of these locations are identified as being "reasonably" representative of the total management system. However, no attempt is made to rank order these 45 locations. There are several audit objectives identified for the audit, but 252 each objective or combination of objectives can be accom— plished independently of the audit data extracted in accom- plishing another audit objective or objectives. The steps in the audit program used to accomplish these objectives can be conveniently divided into three parts, each part requiring 200 hours to complete. Finally assume that although the segmented audit is to be used, we want the l resulting subject matter report of audit to provide tOp I management about 95 percent of the information that would E It: be provided if the complete audit program were applied at the universe of locations. The mechanics of the segmented audit are now rather straight forward. First, a sample size must be selected. Using Table 33 as a guide, one may note that a sample of 10 or more locations will be needed to detect 95 percent of the reportable type items. To assure that these items are sufficiently material to warrant inclusion in a report of audit, Table 30, page 178 indicates that one must use a sample of 15 locations. Accepting a sample size of 15 locations, the next step is to determine the possible number of segments into which the audit can be divided. This number is a function of the number of reasonably representative locations avail- able and the number of independent sections into which the 5 253 audit program may be divided. For the assumed conditions described, the audit may be divided into three segments (45 representative locations divided by the required sample size of 15) each requiring 200 hours (a 600 hour audit program divided into three independent parts of equal size) to accomplish. Use of the segmented audit can result not only in a reduction in the number of locations participating in the subject matter audit (here 45 locations rather than the 150 in the universe), but in a substantial reduction in the number of audit hours consumed at each location as well (200 hours required for each segment while 600 hours are necessary for the complete audit program). One can conclude that the report of audit will con- tain a very high percentage of the information of interest to tOp management at a considerable reduction in cost. However, as stated earlier, one must bear in mind that the amount of information contained in the report of audit will likely approach but not reach those percentages recorded in Table 30 for equal sample sizes since the samples used to abstract the information must be characterized as "reason- ably representative" rather than "most representative". Obviously, for a number of reasons, all subject i matter audits are not apprOpriate for the segmented approach. 254 There may be an insufficient number of locations in the universe. There may not be enough representative locations. The audit program itself may not be suited for division. But when these conditions are met, a segmented audit approach can provide audit economy. An Evaluation Turn now to an evaluation of the segmented audit approach using the four criteria suggested in Chapter V. These criteria are: (1) provide the Opportunity to assess the risk of not detecting reportable type items, (2) reduce the time required to achieve the subject matter report of audit, (3) reduce the cost in terms of audit hours, and (4) provide the same information to management as is pro- vided by an audit accomplished at the universe of locations. The first criterion, to assess the risk of not detecting reportable type items, is not as important to the segmented audit approach as it is to the methods described in the previous two chapters. It has been reasoned that the tables can be used as a general guide for determining sample sizes even though the data in the tables are based upon analysis of results from the most representative sample rather than from several representa— tive samples from each subject matter audit. 7') 255 Where the segmented audit can be used, it should result in a reduction of the time required to achieve the subject matter report of audit. This is obvious from the statistics used in the hypothetical example. It was stated that to review a major portion of a specified management system at any one location an auditor working alone would require 600 hours or more than three months to complete the audit. When the audit is segmented, however, the time needed is reduced to a third of that requirement, permitting the resulting information to be provided to tOp management at a much earlier date. The third criterion, to reduce the cost in terms of total audit hours used for the subject matter audit may or may not be achieved. Again, using the statistics pro- vided in the sample as an illustration, if the segmented approach had not been used and if the audit were applied at the universe of 150 locations, considerable audit hour savings would accrue. Specifically, only one tenth of the number of hours would be used by the segmented approach.1 If the subject matter audit had been limited to 15 of the most representative locations in lieu of the segmented approach, then there would be no change in the total audit hours consumed. The audit burden at each selected loca- tion, however, would be considerably increased.2 256 The last criterion, to provide the same information as is provided by an audit accomplished at the universe of locations, is difficult to assess based on the research accomplished. .Table 30 which has been used to determine the percentage of the reported information available from a sample is constructed using Ehg most representative sample from each subject matter audit. One can only speculate that results achieved by representative rather than Egg repre— sentative sample will approximate these results. Verifica- tion, however, would require research beyond the sc0pe of this investigation. In summary, one can conclude that the method will clearly meet only one of the four criteria -- it will reduce the time required to achieve the subject matter report of audit. One cannot assess the extent to which the third criterion is or is not met without specific data concerning the number of audit locations to be included in the sub- ject matter audit. Assessment of the remaining two criteria cannot be made without additional research that is beyond the sc0pe of this investigation. Nevertheless, the segmented audit makes good sense and offers sufficient promise of greater economy in the use of audit resources to warrant further research, and probably to warrant its consideration even without research. 257 FOOTNOTES The equivalent sample of 15 locations would be used from the universe of 150 locations. Specifically, 45 locations using 200 audit hours at each location is equivalent to 15 locations using 600 audit hours at each location. If the audit program is applied at 15 selected loca- tions, 600 audit hours will be required to complete the audit program at each location. The segmented approach as described will require only 200 audit hours at each location. CHAPTER VIII CONCLUSION This study proposes new considerations specifically for the management of United States Air Force internal audit resources. As previously stated, over the past several years the need to find more efficient internal audit methods has become increasingly serious. DOD internal audit staffs have been reduced relative to the size of the internal audit workload. This pattern of a diminishing auditor staff con- current with an expanding internal audit workload has be- come a matter of serious concern. In the three years ended June 30, 1971, Air Force Audit Agency personnel authoriza- tions declined approximately 22 percent and there is no reason to expect a reversal in this trend. During this same period Air Force resources grew both in cumulative inventory and annual consumption. There has been no sig- nificant reduction in the number of major locations at which the Air Force Operates. The proposals contained in this study are advanced in the belief that they can provide some of the answers to 258 259 the increasingly complex problem of managing internal audit resources. At the heart of these prOposals are four basic hypotheses identified in Chapter I. These hypotheses are in brief: 1. The information derived curve is shaped as TI illustrated in Figure 1, page 10. 2. Information derived curves develOped from judgmental selection of individual audit loca— tions are superior to such curves develOped [I from random selection of individual audit locations. 3. There is a minimum number or at least a range of locations for a given audit for which audit cost can be considered reasonable. 4. The information derived curve can provide an internal audit management tool, the use of which can result in more efficient use of internal audit resources. The purpose of this concluding chapter is to review each of these hypotheses in the light of the research re— sults achieved. 260 Hypothesis One The first hypothesis is that the information derived curve is shaped as illustrated in Figure 1, page 10. That is, the first few locations in a judgmentally selected sample of locations can make relatively large contributions to the total amount of information that is derived from the subject matter audit. As the number of locations included in the sample increases, the total amount of information derived may increase, but at a decreasing rate. All three phases of the research were designed to test this hypothesis. Each phase used a slightly different approach. For the first phase "information" was defined as data about operating errors or poor management practices that were Observed to occur in the figld of audit obser— vations at a rate of ten percent or more. Three representa- tive subject matter audits were selected by audit managers for analysis. For each subject matter audit, audit managers judgmentally selected samples of 10, 20, 30, and 60 loca— tions. The rate at which information was generated by these samples of locations for each subject matter audit was determined. These results are recorded in Table 3, page 68, and illustrated in Figure 10, page 69. It is apparent from these data that most of the information is 261 provided by samples of only 10 locations. However, the phase I research was not sufficiently detailed to determine the specific rate at which this information is accumulated. For phase II of the research the general term "information" was replaced by a more carefully defined "reportable type item". Reportable type items were defined in three separate ways to test the effect of changes in the definition on the resulting shape of the information derived curve. The definition that audit managers believed would encompass those types of problems of interest to tOp level managers is as follows. The Operating error or poor manage— ment practice occurs at ten percent or more of the locations in the universe of locations. Four representative subject matter audits were selected by audit managers for analysis. For each subject matter audit, audit managers judgmentally selected samples of 10 locations. The rate at which report— able type items were generated by these samples of loca- tions for each subject matter audit was determined. The most meaningful of these results are recorded in Tables 11 and 12 and in Figure 13, pages 105-107. It is apparent from these data that the information derived curve rises very rapidly over the first few locations included in each sample. Specifically, the average detection of reportable type items by only the fifth location is more than 80 262 percent. Doubling the sample size to ten locations results in an average detection of reportable type items of 95 per- cent, a small gain for a 100 percent increase in effort. Phase III of the research was designed to remove doubt as to the general shape of the information derived curve from the most skeptical of observers. For this phase of the research reportable type items were defined as Operating errors or poor management practices that occurred at ten percent or more of the locations in the universe of locations. All nine subject matter audits in the files of the Air Force Audit Agency were selected for analysis. For each subject matter audit, audit managers judgmentally selected samples of 15 locations. Again, the rate at which reportable type items were generated by these samples of locations for each subject matter audit was determined. The most meaningful of these results are recorded in Tables 23 and 24, pages 163-164, and in Figure 14, page 166. It is apparent from these data that results are very similar to those achieved in phase II. The results of phase II and III that apply to normal subject matter audits -- those that should involve 30 or more reportable type items -- were combined to pro— vide the data listed in column 3, Table 33, page 199. These combined results indicate that the information 263 derived curve rises very rapidly over the first few loca— tions included in each sample. Specifically, as is the case for the results of phase II and III individually, the average detection of reportable type items by the fifth location is over 80 percent. Doubling the sample size to ten locations again results in an average detection of reportable type items of 95 percent. One can conclude that the research results strongly confirm the first hypothesis. Hypothesis Two The second hypothesis is that the information de- rived curves develOped from a judgmental selection of indi— vidual audit locations are significantly superior to such curves develOped from a random selection of individual audit locations. Phase III of the research was designed to test this hypothesis. The data in Table 29, page 173, indicate that for all nine subject matter audits, information derived curves constructed using judgmentally selected locations lie above those constructed using randomly selected loca- tions. Through use of the statistical formula for the binomial experiment it was indicated that the curves con- structed using judgmentally selected locations are clearly 264 statistically superior. Indeed, as Figure 20, page 206 shows, the best results achieved by randomly selecting the locations were no better than the wOrst results achieved by judgmentally selecting the locations. These differences were particularly evident over the first ten locations in- cluded in each sample. By the fifteenth location the dif— f ferences persisted, but were less severe. F Based upon these research results one can again I conclude that the second hypothesis is confirmed. ! Hypothesis Three The third hypothesis is that there is a minimum number or at least a range of locations for a given audit for which audit cost can be considered reasonable. Beyond that range the incremental cost of audit tends to exceed the value of the incremental information. Phase I and III of the research examined this hypothesis. Phase I of the research provided the results shown in Table 3 and Figure 10, pages 68—69. The research was not sufficiently detailed to pinpoint the desired range, however it was very clear that less than a pOpula- tion of locations should be included in each subject matter audit. Following disclosure of these results, the number of locations selected for certain planned subject matter 265 audits by the Air Force Audit Agency was reduced by a third. In phase III of the research cost of sampling was introduced. Considering both of the variables of informa- tion and cost, the research results indicated that the infor- mation return for the costs incurred was maximized over an approximate range of from 10 to 20 audit locations. This result is indicated in Figure 17, page 183. By the fifteenth location, on the average, 95 percent of the information that was recorded in the report of audit for the subject matter audit could be included in a report of audit based on the judgmentally selected sample. This information could be achieved from a sample that will consume only about 10 per- cent of the audit costs required for a complete subject matter audit. To assure achievement of the remaining five percent of the information auditors must incur approxi— mately 90 percent of the total audit cost —- clearly an uneconomical procedure. Based upon these research results one can conclude that there is a range of locations for a given audit for which audit cost can be considered reasonable. Hypothesis Four The fourth and last hypothesis is that the infor- mation derived curve can provide an internal audit 266 management tool, the use of which can result in more effi- cient use of internal audit resources. The confirmation of this hypothesis was the primary purpose of the presentations provided in Chapters V—VII. Chapter V discussed the use of the information derived curves in determining management areas in need of audit attention. Chapter VI followed with a discussion of the use of the curves in the management of the field test of new audit programs. Finally, Chapter VII concluded with a discussion of a prOposed technique called the segmented audit. m In summary, all four of the hypothesis Specified at the outset of the research are reasonably confirmed by the research results. It follows then that the prOposals for improving the management of internal audit resources within the Air Force Audit Agency or of any other multi- location organization Operating under similar conditions have merit and should be carefully considered. Like a lawyer, however, the researcher can only present his case supported by the evidence of his research and strengthened by the reasonableness of his arguments. But it is management who in the final analysis sits in 267 judgment of the fate of his ideas and suggestions. My evidence is presented. The arguments are made. My case rests. LIST OF REFERENCES Books Anthony, Robert N. Planping and Control Systems: A Frame- work For Analysis. Cambridge, Massachusetts: Harvard University Press, 1965. Argyris, Chris. Integrating_the Individual and the Organ— ization. New York, London, Sydney: JOhn Wiley & Sons, Inc., 1966. Arkin, Herbert. Handbpoky9f_5ampling for Auditing and Accounting. New York, San Francisco, Toronto, London: McGraw-Hill, Inc., 1963. Barnard, Chester I. The Functions of the Executive. Cambridge, Massachusetts: Harvard University Press, 1970. Cadmus, Bradford, Operational Auditing HandboOk. New YOrk, N.Y., The Institute of Internal Auditors, 1964. Conover, W. J. Practical Nonparametric Statistics. New York, London, Sydney, Toronto: JohnWiley & Sons, Inc., 1971. Hendriksen, Eldon S. Accounting Theory. Homewood, Illinois: Richard D. Irwin, Inc., 1970. Koontz, Harold and O'Donnell, Cyril. Principles of Manage- ment. New York, New York: McGraw—Hill BoOk Com- pany, 1968. March, James G. and Simon, Herbert A. Organizations. New York, London: John Wiley & Sons, Inc., 1959. Mautz, R. K. and Sharaf, H. A. The Philosophy of Auditing. Chicago, Illinois: American Accounting Association, 1961. Mendenhall, William. Introduction to Probability and Spatistics. Belmont, California: wadsworth Pub— lishing Company, Inc., 1968. 268 269 Mendenhall, William. The Design and Analysis of Experi- ments. Belmont, California: Wadsworth Publishing Company, Inc., 1968. Willingham, J. J. and Carmichael, D. R. AuditingConcepts and Methods. New York, New York: MOGraw-Hill BoOk Company, 1971. Articles and Periodicals American Accounting Association. "Committee on Accounting Concepts and Standards, 1957 Revision," The Accounting Review, October 1957. Coupland, Don. "Current Management -- An Audit Challenge," The Armed Forces Comptroller, April 1967. Kane, M. "A Clarification of the Broadened Role of Internal Auditing," The Internal Auditor, Spring 1961. Kusel, Jimie. "Man-On-The Spot," The Armed Forces Comptroller, April 1969. Sparrow, H. G. "Mission-Oriented Audits," The Armed Forces Comptroller, Vol. 12, No. 2, April 1967. Sternberg, Alexander J. "What Management Should Expect of the Internal Auditor," The U. S. Army Audit Agency Bulletin, December 1965. Walsh, Francis J., Jr. "Trends in Audit Management," The U. 3. Army Audit Agengy Bulletin, December 1965. Official Documents Air Force Audit Agency Regulation ll-5, Processing Summary Reports of Audit, Norton Air Force Base, California: Department of the Air Force, 1 March 1972. Air Force Audit Agency Regulation 23-3, Areas and Activities of the Air Force Audit Agengx. Norton Air Force Base, California: Department of the Air Force, 31 July 1972. 270 Air Force Audit Agency Regulation 23-6, Air Force Audit Agency Organization ang‘Functions, Norton Air Force Base, California: Department of the Air Force, 31 July 1972. Air Force Audit Agency Regulation 175-101, Internal Audit Procedures, Norton Air Force Base, California: Department of the Air Force, 1 March 1969. Air Force Audit Agency Regulation 175-105, Directed Audit Research Task, Norton Air Force Base, California: Department of the Air Force, 21 September 1970. Air Force Audit Agency Regulation 175-112, Initiatipg_and Controlling Centrally Directed Audits, Norton Air Force Base, California: Department of the Air Force, 1 August 1972. Air Force Audit Agency Regulation 175—116, Statistical Sampling for Auditors, Norton Air Force Base, California: Department of the Air Force, 17 July 1972. Air Force Audit Agency Regulation 175-118, Statistical Sampling, Norton Air Force Base, California: Department of the Air Force, 14 September 1971. Air Force Regulation 175-4, Auditing in the Air Force, Washington, D.C.: Department of the Air Force, 15 June 1972. Department of Defense Instruction No. 7600.1, Summary Report of Audit Operations. Washington, D.C.: Department of Defense, 16 July 1971. Department of Defense Directive No. 7600.2, Department of Defense Audit Policies. Washington, D.C.: Depart- ment of Defense, 19 August 1965. Department of Defense Instruction No. 7600.3, Internal Audit in the Department of Defense, Washington, D.C.: Department of Defense, 25 May 1968.