’1. 33.15%; ’I «:3 \Ik|flv. ' M " -‘. {II 'N‘l' ’7’"; ‘30 t ':..' IIIM’J WWW , , .. .I.' Ilfin'fil 7 ' ‘ . . .‘ -.._ .3425 49.14“?» 7" I???“ _ {‘5' :' ””- ~ ., . NV N -z:. a; m r..- 2-,? . . VI ' ,"I'fisIIII fit}; 1'} L I W1 ‘15 '.' ""1 ,g' ‘ (JR: libr'n‘fl"; It" I u '. “I I 3 ' :‘II I\I I." u"; Is”: i' ' III... w“: ‘. 'I-I' .‘-’| ‘II' '- :1 ‘l n. I I... I III I I ' I I .‘f I . I. II " I I I I: I I I a i l 4h" ‘I. I-3. H . ' v 3 I '1 HI I '{ . l! {I T 7. ‘I:u| » hi. $gfili"? II ‘3'». II IIIIII I a’wmm If)» ‘.I’ 'o ;"‘F\v| 171:7. ' Illil‘I-I . 1"” ['27 ,l'” I it; ”‘7‘. I’II'H‘ .1" I Timi‘imhlt.litfflIlllTllHi l - __ ~ - 31293 006970044 ;. , ..--.-a .’_‘._I_f-i;::] :::‘:: [LL-tr; ; :7 This is to certify that the dissertation entitled USING AUDITOR KNOWLEDGE TO FORMULATE DATA MODEL CONSTRAINTS: AN EXPERT SYSTEM FOR INTERNAL CONTROL EVALUATION presented by Graham Francis Gal has been accepted towards fulfillment of the requirements for Ph.D. degree in Accounting, Business WW (/Vic 02:47 Major professor Date 51? OCT 1785' MS U is an Affirmative Action/Equal Opportunity Institution 0- 12771 LlBRARlES ”- MSU ‘ RETURNING MATERIALS: Place in book drop to remove this checkout from your record. ‘FINES will be charged if book is returned after the date stamped beiow. WM QM USING AUDITOR KNOWLEDGE TO FORMULATE DATA MODEL CONSTRAINTS: AN EXPERT SYSTEM FOR INTERNAL CONTROL EVALUATION BY GRAHAM FRANCIS GAL A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Accounting 1985 ©1986 GRAHAM FRANCIS GAL All Rights Reserved ABSTRACT USING AUDITOR KNOWLEDGE TO FORMULATE DATA MODEL CONSTRAINTS: AN EXPERT SYSTEM FOR INTERNAL CONTROL EVALUATION BY GRAHAM FRANCIS GAL The integration of database technology into the information management activities of business firms has a number of benefits which include the ability to develop overall policies concerning administration and use of the data. However, there are also problems inherent with the formulation of large corporate wide databases, such as monitoring the transactions which add data to the system and providing different subsets of the data to be used to support various decisions. This dissertation addresses these two problems from the standpoint of the independent auditor's evaluation of internal accounting controls. When the independent auditor evaluates accounting controls one decision that must be made concerns the accuracy of the information that is collected by the data processing system. In order to make this evaluation the auditor obtains a subset of data which is considered to provide the necessary information. This study sought to identify the portion of this subset that is obtainable from Graham Francis Gal a corporation's production database and to determine the role it plays in the entire evaluation of the data collection process. In addition this study demonstrated that the idea of database constraints in the form of transaction prototypes could be used to obtain the information from the database that was contained in this data subset. The methodology used in this study to uncover the decision process and database variables used in the decision was to construct an expert computer program. This computer model embodied the auditor's knowledge about the evaluation of accounting controls. The information contained in the system was then analyzed to determine role played by database elements in the decision and the structure of the data model constraints that would provide the information used by the auditor. In addition the study also examined the importance of information external to the database in the evaluation of accounting controls. ACKNOWLEDGEMENTS A doctoral program is a very long and demanding process which is culminated with the dissertation. I would' like to take this opportunity to thank those whose help has been instrumental in the completion of the doctoral program. First I would like to thank my committee chairmen, William E. McCarthy who has been a source of inspiration both personally and professionally throughout my program, and particularly during this dissertation. My readers, D. Dewey Ward and George Stockman, both gave me assistance during my program and this project. Thomas H. Carr provided invaluable observations and criticisms early in this project. Alvin A. Arens offered some particularly important insights during this project. Paul Steinbart engaged in many discussions concerning the development of the system. Bob Beyers made the computer system at Jackson Community College available for the project; this research would not have been possible without his assistance. Finally, my family, Francine, Justin and Lyle, who provided support and the realization that there are things in this world that are far more important than any doctoral program or research effort. But who also gave me the incentive to pursue this career. ii TABLE OF CONTENTS LIST OF FIGURES O O O O O O O O O O O O O O O I O O O Vii INTRODUCTION 0 I O C O O O O O O O O O O O O O O O O 1 DATABASE APPROACH . . . . . . . . . . . . . 1 DATABASE VI Ews 0 O O O O O O O O O O O O O I 2 EVALUATION OF INTERNAL CONTROLS . . . . . . 5 METHOD 0 O O I O O O O O O O I 6 ANALYSIS OF EXPERT SYSTEM . . . . . . . . . 7 SUMMARY 0 O O O O O O I O O O O O O O O O O 9 CHAPTER I. PREVIOUS RESEARCH ON INTERNAL CONTROLS . . . 11 Concept of Control . . . . . . . . . . . 11 Management Control . . . . . . . . . . 12 Accounting Controls . . . . . . . . . . l4 Controls and Data Reliability . . . . . 17 Evaluation of Accounting Controls . . . . 18 Factors Affecting Controls . . . . . . 19 Evaluation Frameworks . . . . . . . . . 23 \ Internal Control Judgments . . . . . 29 Implications of the Control Literature . 33 Structuring of the Evaluation Task . . 33 Levels of Consensus and Expertise . . . 34 II. EXPERT SYSTEMS AND DATA MODELS . . . . . . . 43 Expert Systems . . . . . . . . . . 44 Simulation of Intelligence . . . . . 44 Expert Systems versus Intelligent P r 09 rams O O O O O O O O O O 4 7 Power versus Knowledge Approach to Human Intelligence . . . . . . . . . 50 Production Systems . . . . . . . . . . 53 iii Symbol Manipulation . . . . Production Knowledge in the Search Process . . . . . . . . . Explanation of the Line of Reasoning Expert Systems as Cognitive Theories Level of the Expert System Theory Theories of Expertise . . . . . . Heuristics in Problem-Solving . . summary 0 O O O O O O I O Databases and Data Models . . . . . Data Models as Representations of Declarative Knowledge . . . . . . . . Accounting Data Models . . . . . . . . Accounting Controls as Data Model constraints 0 O O O O 0 C O O I O 0 Accounting Views of a Database . . . Summary . . . . . . . . . . . . . . . . III. RESEARCH METHOD . . . . . . . . . . . . . . Research Tool: EMYCIN . . . . . . . Reasons for Choosing EMYCIN . . . EMYCIN in previous research . . Similarity of medical diagnosis internal control judgments . . EMYCIN features . . . . . . . . . Inference strategy . . . . Structure of the knowledge base Reasoning under uncertainty . . Summary . . . . Construction of the Prototype System Specification of the Problem Area . Identification of the Initial Knowledge Base . . . . . . . . Selection of the firm . . . . Initial knowledge acquisition Refinement of the System . . . . Selection of the Auditor . . . The Refinement Process . . . . . Explanation of the reasoning pro e s summary 0 O O O 0 Verification of INTERNAL—CONTROL-ANALYZER Verification of Goals and Sub-Goals . . summa ry O O I O O O O O O O O O I O O 0 00000.”... :3 000.0000“... 0 O O O O O O O C 8 IV. ANALYSIS OF THE SYSTEM . . . . . . . . . . . The Internal Control Evaluation Model . . The Model's Judgment Process . . . . General overview of the evaluation process . . . . . . . . . . The evaluation model . . . . . . . . iv 102 104 109 109 110 112 113 116 118 119 121 122 123 125 127 129 129 129 130 133 Evaluation of sales transactions . Separation of duties for sales . Population controls for sales . . Completeness controls for sales Authorization control for sales Accuracy controls for sales . . . Comparison controls for sales . Mathematical checks for sales . Evaluation of cash receipt transactions . . . . . . . . . . Completeness controls for cash receipts . . . . . . . . Separation of duties for cash receipts . . . . . . . Accuracy controls for cash receipts . . . . . Comparison of remittance advice Mathematical checks for cash receipts . . . . . . . . Suma ry O O O O O O Refinements to the Prototype System Refinements to the judgment . . . Refinements based on the same information . . . . . Refinements based on different information . . . . . . . . . Refinements in information acquisition . . . . . . . . . . . Accounting Control Evaluation . . . . . The auditor's evaluation process . Possible explanations of the observed decision process . . . . . . . . . summa ry O 0 O O O O I O O O I O O 0 Database Constraints and Transaction Prototypes . . . . . . Transaction Prototypes and Controls Completeness controls . Authorization controls Comparison controls . Mathematical checks . Separation of duties Summary . . . . . . . Database Constraints and Auditing Environmental Factors and Database Design Issues . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . V. Conclusions and Suggested Extensions . . . . General Overview and Research Contribution Possible Future Extensions of the Research 138 138 142 142 147 147 148 150 151 152 153 153 154 155 156 156 157 159 162 165 166 167 171 172 173 174 180 184 185 186 188 190 197 200 203 203 207 Investigation of the Internal Control Decision . . . . . . . . . . 207 Include Environmental Variables . . . 207 Use other auditors to refine the system . . . . . . . . . . 209 Experimental studies of internal control judgments . . . . . . . . . 210 Investigation of Database Issues . . . 210 Complete implementation using prototypes . . . . . . . . . . 211 Use expert systems to identify external data items . . . . . . 211 Integrity measures and internal control judgments . . . . . . . . . 212 Conclusion . . . . . . . . . . . . . . . 213 APPEme A O O O O O O O O O O O O I O O O O O O O C 214 BIBLIOGRAPHY O O O O O O O O O O O O O O O O O O O O 222 vi FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. LIST OF FIGURES DATABASE VIEWS . . . . . . . . . . . . . . . PRODUCTIONS AND BACKWARD-CHAINING . . . . . . Components of INTERNAL-CONTROL-ANALYZER . . . GENERAL OVERVIEW OF THE EVALUATION OF ACCOUNTING CONTROLS . . . . . . . . . . . . THE EVALUATION OF REVENUE CYCLE ACCOUNTING CONTROLS O O O O O O I O O I O I O O O O O O FACTORS AFFECTING CONTROLS OVER TRANSACTIONS FACTORS AFFECTING POPULATION CONTROLS . . . . FACTORS AFFECTING ACCURACY CONTROLS . . . . . TRANSACTION and EVENT PROTOTYPES . . . . . . SALE SPECIALIZATION OF TRANSACTION PROTOTYPE TRANSACTION PROTOTYPES AND THE AUDITING VIEW vii 57 94 124 134 137 141 146 175 181 195 INTRODUCTION DATAEASE APPROACH The 'effective use of corporate data is an important‘ component of a number of decisions, and such importance has influenced many developments whose purposes are to facilitate the collection, processing and dissemination of this data. A particularly significant development, that is both technological as well as conceptual, has been the introduction of database systems to the organization of this data. A database approach can be contrasted with a traditional file orientation by the way in which data is viewed within the firm. Files were initially developed by application area which led to certain problems. Because similar data items were of interest in different applications, this approach generally produced redundancy which further resulted in a lack of overall data consistency and data integrity. 0n the other hand, a database approaches the development of the corporate data pool from an overall perspective. This corporate wide perspective of the data produces a number of desirable features. First, the reduced redundancy increases the overall consistency of the database. Second, certain corporate policies that concerned general properties of the data (such as control over its verification) became feasible. Finally it became possible to develop overall policies that would restrict use of the data. For instance, from this overall or conceptual perspective of the data, various extracted views can be produced and controlled based 1 on the pertinence of the view to a particular decision. DATABASE flfiflfi Figure 1 demonstrates the relationship of particular views to the entire set of data. The source database is controlled by the source database management system. This is the point at which the overall data policies of the firm are administered. From this large collection of data, the data extraction system produces various data sets by using processes such as aggregation and subsetting. Some of these extracted data sets are very limited and therefore useful only in very selected decision settings. For instance, the person responsible for granting credit might only need a view consisting of customer names, numbers, current balances and credit limits. However, the features of the database approach (such as the consideration for overall consistency) also fac1litate decisions that require either views which cross traditional application areas or views which encompass the entire database. It is probably not possible for an individual to view Or SOURCE SOURCE ¢_1 ' DATABASE DATABASE MANAGEMENT SYSTEM ‘F ‘;~_____,. V DATA EXTRACTION SYSTEM financial credit statistical reliability statements authorization abstract abstract DATABASE VIEWS FIGURE 1 an entire source database in making a decision; therefore, certain reductions or aggregations might be required. Financial statements are good examples of such reductions. A firm's financial statements are a view of the entire corporate data pool that has been reduced (with various aggregation rules) to a more manageable subset. While‘ certain features must be lost through the aggregation process, it is assume that most of the pertinent information will be contained in this database abstract. Another use of the view concept has been the development of procedures that produce abstracts which can be used to answer questions concerning properties of the source database. Rowe [1983a a b] discusses a technique that uses various inference rules to develop a ‘top-down' statistical abstract of the source database. Rowe argues that because a database is restricted to certain information about a particular entity, a semantic sample can answer statistical questions that a random sample could not. Auditing also has a similar perspective concerning the degree to which random samples can be used to represent the overall set of information. ‘ Auditors do not simply examine random transactions within a firm. Instead certain types of transactions are considered more important to audit decision making. One particular decision that auditors make concerns the reliability of the-data in the source database. In the process of an audit, certain data (some of which comes from the source database) is examined to reach a conclusion about the possibility of a client's data processing system producing errors. The evaluator of internal accounting controls is specifically interested in obtaining a view of the company which is used in making a judgment about the- overall reliability of the client's data. The purpose of this study was to investigate a possible method for constructing this reliability view and to demonstrate a way in which such a view could be used in the evaluation of internal controls. EYALHAIIQN QE.IHIEBNAL.§QHTBQLS Ideally there would be an objective method to identify the information that should be used to evaluate an existing system of internal accounting controls. Then this information could be used to construct the appropriate database view. However, an examination of professional standards [AICPA, 1979] reveals the use of primarily subjective concepts such as "reasonable assurance,“ “sufficient competent evidential matter,“ and ' auditing judgment.’I Mautz and Winjum [1980] also discuss certain features, such as the control environment within the firm, that become important in evaluating internal controls. However, these two authors also consider the final evaluation to be a matter of judgment. This opinion --that 6 general information about controls can be useful, but that the final evaluation is in the area of judgment-- is found throughout the literature on internal controls. In order to obtain the information concerning an appropriate view and evaluation procedure, it is necessary to study individuals who make internal control judgments. Professional auditors have experience in both obtaining the information and making judgments about such matters; therefore this study concentrated on their decision processes. METHOD The first step in this study was to examine the way in which an experienced professional auditor collected and evaluated pieces of information, as he made a judgment about both the nature of the internal accounting controls and the reliability of data in a client's system. The interest in this examination concerned both the process that .collected information and the process that used this information to arrive at a particular judgment. The importance to this study of obtaining insights into the actual processes used in the judgment required an approach which examined not only outcomes but also the way in which the outcomes were obtained. The method that was used to examine the judgment is derived from the information-processing paradigm proposed by Newell and Simon [1972] to investigate human problem solving. 7 This paradigm was used in a recent study on internal control evaluation [Biggs and Mock, 1983] in which verbal reports or protocols were obtained from auditors as they made certain evaluations. In the conclusion of their work, Biggs and Mock suggest that computer programs could be constructed that would embody the information about the. processes that the auditors used to solve the problem. This approach has been used in other disciplines (such as medicine), and the resultant programs have been termed ‘expert systems'. It is this approach --the construction of a computer program that contained a model of the process an auditor uses to evaluate internal controls-— that was used to collect and verify the information obtained from the auditors. After the program was completed, it was possible to analyze the results to see whether a particular view of a database can be constructed which would allow a reliability evaluation to be made. ANALYSIS 9}; [LEE EXPERT SYSTEM The second portion of this study concerned the analysis of the computer simulation of the decision process. This analysis dealt with two matters: (1) the particular conceptual model that has been proposed as a representation of internal controls, and (2) the way in which a view of the database could be constructed. In a recent work, Gal and McCarthy [1985c] argue that the concept of internal controls is related to the way in which transactions are executed. At a conceptual level, transactions can be thought of as ‘episodes' that a company goes through. These episodes are sometimes grouped by certain similarities into particular cycles. For instance, a sale might be considered to be one episode in the revenue cycle. From all episodes of a similar type (such as sales), a prototype of a particular event could be developed. Gal and McCarthy have called these prototypes of particular events "Event-Scripts“. The authors argue that the Script concept [Schank and Abelson, 1977] can be used as a representation of transactions and as a method for evaluating internal controls. Gal and McCarthy demonstrated that the REA accounting model [McCarthy, 1982] can be combined with the concept of Event-Scripts. These transaction prototypes can also be considered constraints on occurences of'a database and would contain information such as the agents within the firm that can fill certain roles in a transaction or the sequence of events that must occur for a transaction to be completed "correctly". Given that the database contains a conceptual representation of the way in which transactions should occur, it would be possible to identify exceptions to the prototype for the particular event. It is these exceptions that would become part of the evaluation of the overall internal controls in the client's system. SUMMARY Gal and McCarthy have argued that the exceptions to prototypical Event-Scripts can serve as a view of the database that contains information about internal controls,. and therefore about data reliability. The information contained in the expert system will be the process that an experienced auditor goes through to obtain information and to make an evaluation based on this information. This information will concern either data that is available from the source database (in the form of a view) or data that is outside the database. The analysis of the results of the construction of the expert system will consist of a comparison of the proposed Event-Script method of building this view and the data actually used by auditors. In addition, the analysis will look at the way in which the auditor uses the information from the database to assess the controls which ensure overall data reliability. The remainder of this dissertation is organized as follows. Chapter I reviews both the issue of internal control and previous research which has examined the evaluation of controls. Chapter II discusses the theoretical basis for using the expert systems methodology to study decision making behavior and examines work in the area of data models. Chapter III discusses the actual 10 method that was used to construct the expert system. Chapter IV analyzes the resulting system in terms of the internal control decision, the use of prototypes to provide an appropriate view (or data subset) for this decision, and the implications for database design. Chapter V summarizes the findings and recommends some areas for future research. CHAPTER I PREVIOUS RESEARCH ON INTERNAL CONTROLS This chapter examines the concept of control and the various approaches that have been used to study its components and possible methods of evaluation. The first section discusses the different views that exist about controls and their purpose. The second section then examines a particular type of controls, accounting controls, in terms of the factors that influence them and the literature that has studied their evaluation. The final part of this chapter discusses the implications of the previous sections for this study. Concept of Control At a very general level there would probably be little disagreement among accountants and managers about the necessity of good controls to a well-run business. However, a recent study [Mautz et al., 1980] examined the current status of controls and found a number of different responsibilities and risks that were particular to each profession. Mautz and Winjum [1980] identify and discuss two of these definitions that cover a good deal of this 11 12 diversity: management control and accounting control. The following sections discuss the nature of these two concepts of control, and the particular definition of control which is important for this study. Management Control In' their study, Mautz and Winjum argue that the A development of the concept of management control does not have a well-defined body of literature from which a general consensus has emerged; instead, it is a view advocated by various groups (such as controllers, internal auditors, chief financial officers, and operating executives) who are responsible for different functions within a firm. Mautz and Winjum discuss a number of features that seem to capture the essence of the various positions expressed by these groups, which identify management control as: 1) an integral part of management responsibilities, 2) a broad concept which includes positive goal directed and error reduction activities, and, 3) a personnel-oriented concept [p. 14] The first feature identifies management control to be one of the many activities that are the responsibility of corporate management. The study defines the nature of management responsibility as the task ”of putting company assets at risk for a profit [p. 14]." That is, the responsibility of management is to identify methods which 13 will attain corporate profit goals and to deal concurrently with the risks that are inherent in the plan to achieve these goals. The exposure that management faces in using corporate assets to achieve corporate goals are subsumed under the general heading of business risk (i.e. factors such as. competition and economic conditions that might keep a company from achieving its goals). One component of this risk “internal control risk" is the result of relying on the actions of people to achieve corporate goals. Thus management control is exercised in an environment that uses fallible humans in the formulation and implementation of corporate plans. The second feature mentioned by Mautz and Winjum are those actions that fall under the heading of management control. Included are both positive and precautionary measures that are related to realizing corporate goals. The measures that reward certain activities and those that attempt to reduce errors and irregularities all fall under the heading of management control. The final feature concerns the personnel orientation of management control. This area is related to the features discussed previously: however, the authors make a distinction between failure due to personnel and failure due to deficiencies in management control. Management controls are used (1) to motivate personnel 14 toward positive actions, (2) to deter personnel from negative actions, and (3) to allow timely discovery of errors or irregularities. Because these controls are personnel-oriented, it is possible to have errors and irregularities even if the management controls are not defective. No system can prevent personnel failure and therefore the presence of errors and irregularities does not imply a defective system [Mautz, Reilly, and Maher, 1979]. In conclusion, management control is a component of the overall responsibility of management. It is designed to deal with risks that arise from using fallible humans to carry out corporate plans, and it is possible to have a good management control system and! still have errors and irregularities. Accgggting Contrgls The deve10pment of the notion of accounting controls can be traced by examining a specific body of. official pronouncements and codifications. The definition within this professional literature has gone through a number of revisions designed to limit the scope of activities that would be included under the heading of accounting controls. In addition, the definition of accounting controls contained in the Foreign Corrupt Practices Act (FCPA) is essentially identical to the one presented in section 320.28 of the professional standards [AICPA, 1979]. This history tends to give specific authoritative acceptance to the following 15 definition of accounting controls: Accounting control comprises the plan of organization and the procedures and records that are concerned with safeguarding of assets and the reliability of financial records and consequently are designed to provide reasonable assurance that: a. Transactions are executed in accordance with management's general or specific authorization. Transactions are recorded as necessary (1) to permit preparation of financial statements in conformity with generally accepted accounting principles or any other criteria applicable to such statements and (2) to maintain accountability for assets. Access to assets is permitted only in accordance with management's authorization. The recorded accountability for assets is compared with the existing assets at reasonable intervals and appropriate action is taken with respect to any differences. The purpose for giving a very specific ‘and very limiting definition to the term "accounting controls'I can be understood by examining the independent auditors second standard of field work which states: This There is to be a proper study and evaluation of the existing internal control as a basis for reliance thereon and for the determination of the resultant extent of the tests to which auditing procedures are to be limited [AICPA, 1979: 320.01]. requirement that independent auditors examine internal accounting controls necessitated a definition that 16 would fit into the specific objectives of the audit function. Therefore, the present definition of accounting controls in the auditing standards is designed to deal with both the risks associated with the audit process and the nature of the information examined. An audit is conducted to provide reasonable assurance . that the financial statements fairly present the current state of the firm within the framework of generally accepted accounting principles. The risk faced by auditors results from this audit process. . Arens and Loebbecke [1980] identify two components of this risk: (1) that the firm's accounting system will make material errors that are undetected, and (2) that the audit tests will fail to uncover these errors. Another way of looking at audit risk is presented by Cushing and Loebbecke [1982] where a comprehensive risk framework, to be incorporated in audit planning, is constructed which includes a number of components. One of these components is internal control risk which is the risk that the client's system will produce undetected errors (the first component of audit risk identified by Arens and Loebbecke). Vasarhelyi [1980, p. 43] defines these errors ”as a discrepancy between the empirical relational system (ERS) (containing all transactions, economic entities, and levels within the system) and its numerical relatinal system (NRS) (representing the measurements of these entities made within 17 a framework of measurement rules).' That is, accounting controls relate specifically to differences between the ‘real world' value and the value as measured and recorded by the information system. Controls and Data Re abi In summary, there are two concepts of control which have emerged. Management control is a very broad concept with an emphasis on the formulation and implementation of corporate plans designed to attain certain goals. It is aimed at dealing with fallible humans as they fit into the corporate realization of these plans. On the other hand, accounting control is a very narrow, well-defined concept whose purpose is to describe audit responsibilities. Accounting controls specifically relate to the data within the client's accounting system. The purpose of this study, as previously mentioned, is to investigate the possibility of constructing a database view that contains the information necessary to evaluate the reliability of the data. ' Therefore, it is accounting controls and their evaluation that are important to this study. Further, it is the independent auditors that work with this concept of controls, and it is an assumption of this study that they possess the appropriate expertise necessary to identify specific controls which are necessary 18 to ensure accurate data. The next section examines the literature that pertains to the evaluation of accounting controls. Eyaluation 9; Accounting Controls Earlier it was argued that the accounting controls influence audit risk, because they affect the existence of errors in the client's accounting system. The result of the evaluation of these accounting controls will therefore influence the amount of audit tests that need to be conducted. The szcond standard of field work contemplates the relatioship between a firm's accounting controls and the tests that auditors perform to achieve a given level of assurance. Within the complete standard, there is some general guidance on both the way in which the evaluation might be conducted and the impact that this evaluation would have on audit tests. Mautz and Mini [1966] attempt to provide a framework to aid independent auditors in making minimum audit program adjustments based on the results of the study of the accounting controls. This study also offers some guidance in the evaluation process, but both the standards and the Mautz and Mini work place the evaluation in the area of judgment. Mautz and Mini argue that a logical analysis should permit agreement as to the presence of an internal control weakness, but an assessment of the seriousness of a weakness would require a judgment on the 19 part of the auditor [p. 291]. The literature that deals with internal control evaluation is essentially of three types: (1) work that examines factors that can have an influence on controls, (2) work that attempts to assist the judgment process by providing a framework for the judgment, and (3) work that. examines empirically the components of the judgment. Each of these types is discussed in a section that follows. Factors A££§25129.QQEEL21§ The literature included in this section deals with various factors that affect the nature of controls within an organization. These factors will also have an impact on the ways in which controls need to be evaluated. Therefore the factors discussed in this section serve as a basis for a number of studies in the next section. A particularly important environmental factor which dramatically affects internal control concerns the technological complexity of a firm's information system. More specifically, it concerns the extent to which a firm uses computers in its data prrocessing operations. The impact of this development on the control of data and its processing was described by Vasarhelyi as follows: Manual systems had allowed for informal controls of a pattern recognition nature by human information processors. Special emphasis was given to supervision. Automated systems partially changed the nature of control systems. The 20 emphasis now is on system design and integrity as consistency is substantially assured [1980, p. 41]. In response to this technological development a number of studies have examined specific factors that relate to controls in a computer environment. The Stanford Research Institute (SRI) study [1977] was conducted for the Institute of Internal Auditors under a grant from IBM. The stated puspose of the SRI study was to aid persons responsible for controlling and auditing systems by reporting on various methods that have demonstrated value in the audit and control of computer based data processing systems. As suggested by Vasarhelyi, the study places an emphasis on the importance of the design and implementation of computer systems. The position that auditors, both internal and external, should understand and participate in the systems development process is a reflection of,a number of opinions concerning computer system controls. First is the view that to use most of the sophisticated audit techniques (required by computer systems) a necessary condition is an understanding of the functioning of the system (a point made in the SRI study). Second is the belief that it is important to begin instituting control procedures and audit capabilites early in the system development stages [see Grabski, 1983 for more discussion of the literature in this area]. In addition, this view (that 21 it is important to exercise control. during the early development of systems) is a reflection of a point made by Mautz and Winjum concerning the importance of a general control environment. There are a number of studies that are similar to the SRI work in that they attempt to bring together information. from a number of areas that have an impact on the internal control environment. The National Bureau of Standards (NBS) Reports [1977 and 1980] were undertaken with the support of the General Accounting Office. These studies were designed to obtain opinions from leading experts on the topic of computer security. The government as a large user of computers to process information, was interested in the current state of controls that provide data security. Many of the topics that were addressed concerned issues (such as qualifications and training of personnel) that affect controls. The reports also dealt with various factors that are management-oriented (such as the organizational structure and the presence of control standards and plans to institute these controls). One particular area that is discussed in the NBS study is the affect of database systems on the security and control of data. The overall perspective of a database system requires certain types of controls that other types of applications do not. The tendency to share data across traditional application areas makes control over access and audit trails 22 more complex and difficult to monitor. A second general factor that has affected internal control is a result of the passage of the Foreign Corrupt Practices Act (FCPA). The FCPA uses the same definition of accounting controls contained in the professional standards with only minor changes. The FCPA was concerned with . certain types of payments by corporations, and therefore the investigation implied was not only to serve as a basis for the modification of the audit program but also to serve as a method for dealing with these payments. In response to the passage of the FCPA, Mautz et a1. [1980] conducted a study for the Financial Executives Research Foundation. The authors of the study were interested in the currently held view of control and the possible impact that the passage of the FCPA might have. Mautz et al. found a number of different definitions of the general concept of control and number of different procedures which were used to control the activities of the firms. While Mautz et a1. did not find any specific changes that corporations or audit firms were making in response to the FCPA, the passage of the act did provide a renewed interest in the control of data processing systems. This section has discussed a number of factors that have an influence on the control over data. These factors are technological (such as computers and database systems), environmental (auditor involvement in system design and 23 implementation), orgnizational, and legal. They not only have affected controls but also have produced a number of frameworks to evaluate controls. The next section deals with some of the studies which have incorporated these factors in various evaluation frameworks. Eyaluation Frameworks The SRI study (and to some extent the other works in the previous section) attempted to provide assistance to the evaluation of controls by examining a number of techniques or identifying issues that affect controls. In contrast, the studies examined in this section attempt to assist the evaluation process either by placing controls in a particular framework or by providing guidance in the collection of evidence. One category of this work attempts to place internal controls in a mathematical framework. The basis for this work is the view that controls deal with the reliability of data which in turn is related to the probability of errors. Both the terms reliability and probability can be dealt with in traditional mathematical models. Yu and Neter [1973] deal with the relationship between system inputs and outputs as a stochastic process that can be altered through the use of various reliability increasing procedures. Cushing [1974] developed a model of system reliability that includes an expected cost measure, which could be used to evaluate alternative controls systems. Hamlen [1980] expands on 24 these approaches by developing an integer programming model of internal accounting that would allow judgment of the effectiveness of a system based on its ability to meet specific objectives at a minimum cost. One problem associated with the use of these models is that probabilities (some of which will necessarily be very ‘ subjective) must be assigned at some point to various events. Another line of research that deals with internal controls in terms of mathematical representation is the TICOM project [Cash, Bailey, and Whinston, 1977: Bailey et al., 1983a & b, 1985] This ongoing project has attempted to represent the internal control system in terms of a mathematical structure. This representation then permits an analysis of the system based on its formulation. Another view of controls places transactions that occur in the course of operations in the context of episodes that a firm goes through. By representing the changes that a company goes through in terms of a prototype, a Script [Schank and Abelson, 1977] of these transactions can be developed. The use of prototypes for different activites in the development of information systems has been used in a number of projects [Borgida, Mylopoulos, and Wong, 1984]. In this view accounting controls attempt to ensure that the episodes that a company goes through conform to the prototype or script for the particular transaction. 25 Deviations from the script are considered control violations. The work of Gal and McCarthy [1985c] develops the idea of Event-Scripts that conform to the REA data model [McCarthy, 1982]. In this form, resources and agents within the company play roles in these Event—Scripts. This research project is based on this view of controls and] attempts to identify the components of these event prototypes which can be used to evaluate the necessary controls. In contrast to studies that propose a representation controls that may assist their evaluation, practitioner guides deal specifically with methods of evaluation. The public accounting firms are particularly interested in assisting their auditors in making these types of judgments. In addition, it is the type of judgment that is made in the normal course of their business: therefore may firms have developed guides that will assist the evaluation process. The guides that were available [Arthur Andersen & Co., 1978; Ernst & Whinney, 1979; and Peat, Marwick, Mitchell & Co., 1980] reflect a number of similarities (such the use of objectives in terms of the nevaluation process): however, they also reflect the individual nature of the different aproaches used by each firm. A number of additional works also deal with methods that can be used in the evaluation process, although they are not as comprehensive as the work of the public 26 accounting firms. Mautz and Winjum [1980] identify four components that are important to the evaluation process: (1) the control environment, (2) the analysis of risks, (3) the matching of procedures with risks, and (4) the monitoring of the control system. Miotto [1980] deals with these components and considers the corporate plan for controls to ‘ be one of the most important factors. Felix and Goodfellow [1979] deal with the way in which audit tests can be used for internal control reliance, and discuss the use of sampling procedures to gather evidence about the nature of the controls. Rittenberg and Miner [1981] include the use of cost/benefit trade-offs in the analysis of internal control procedures. Loebbecke and Zuber [1980] deal with the impact of the FCPA on the evaluation process and attempt to demonstrate the use of Operational objectives in examining internal accounting controls. Loebbecke, Mullarkey, and Zuber [1983] extend such analysis to those situations in which the client uses computers and recommends an approach that follows flows of transactions through the system. The effect of database systems on controls was mentioned earlier. A number off studies have examined aproaches to the evaluation of controls when a database approach is used. Roberts [1980] developed a questionnaire that would assist auditors in gathering evidence which might be useful in evaluating controls in a database environment. 27 Techavichit [1979] disccussed the evaluation of internal controls in an environment that used a database system to store data. He divided controls into technical and non- technical catagories, and he pointed out issues that must be dealt with at various levels within these groups. This section's review of literature covers many ‘ different authors with a wide variety of interests. This broad spectrum of work can be characterized as representing two distinct approaches to the issue of information systems that produce reliable information. The first approach, as represented by Yu and Neter, Cushing and Hamlen works, views controls and the processing of data in an information system in a mathematical framework. These works appear in what might be identified as academic publications and have not received much acceptance by those individuals that actually evaluate systems as part of their professional responsibilities. This lack of acceptance is probably due to a number of factors which are implicit in the use of these mathematical frameworks. The first problem, which was alluded to earlier, is that these approaches require the assignment of probabilities and/or reliability measures to certain processing activities within the system. The assignment of these numerical values is very subjective and Einhorn and Hogarth [1981] have argued that normative decision models are generally not followed for precisely this reason. Another problem with the approaches that place 28 controls in a mathematical framework is that they ignore certain inputs that are difficult to quantify, such as the user of the information system and the quality of the control environment. The second approach, as represented by other works reviewed, reflects the view expressed by Mautz and Winjum ‘ [1980] that there are many different procedures that can be used to produce the desired result (an evaluation of the controls that are functioning within the system) and that there is probably no unique solution to the problem of reliable information system design. These works appear in publications that are more practitioner oriented, or have subsequently been adapted (as was the case of the Roberts [1980] work) to be useful to professionals in this area. Therefore, these works do not attempt to remove the judgment from the evaluation, but instead have as their purpose to assist the auditor making an assessment of the controls that are functioning within an accounting information system. These works provide the auditor with a framework to gather evidence that the authors feel should have an impact on the decision but generally do not attempt to develop a normative decision model based on this evidence. The evidence from this section indicates that in order to understand how certain factors (some of which were discussed in the previous section) affect the control of an information system it is necessary to study individuals that 29 evaluate these systems. Therefore it is necessary to examine professional auditors that actually use these frameworks and see the affect that certain factors have on the judgments of system reliability. The next section reviews the literature that has taken this approach in an attempt to understand auditors' judgments and those factors which affect it. WWW Many of the studies in this section have been classified under the general heading of behavioral accounting research. Ashton [1982] and Libby [1981] examine behavioral studies in accounting and argue that the purpose of this research is to improve the way in which the decisions are made. Both of the authors point out that, in order to improve a decision, it is first necessary to understand the methods currently being used by. decision makers. The behavioral studies that examine internal controls judgments study the evaluation of the quality of internal controls and the relationship of such evaluations to audit program planning (the influence on the amount of audit tests). Due to the lack of objective criteria for either internal control quality or sufficiency of audit tests, these studies examine the impact of certain variables and the level of consensus among subjects making the judgments. 30 One of the initial studies that examined internal control judgments was done by Ashton [1974]. This study used independent auditors and presented them with six internal control factors that the author considered to be important based on his examination of literature in the field. The auditors were asked to make evaluations of the _ internal controls based on these factors. Ashton and Kramer [1980] replicated this study but used students so that a comparison could be made between experienced auditors and subjects that did not possess any expertise in the area. Ashton and Brown [1980] used the same instrument as the initial Ashton study but added two additional cues. Reckers and Taylor [1979] examined the same judgment, internal controls in the payroll cycle, but used a longer questionnaire that was obtained from an accounting firm. Trotman, Yetton, and Zimmer [1983] used a similar approach as the previous studies except that evaluations were made both individually and in groups. These studies had a number of results concerning the levels of consensus obtained which are important to this work. The first point concerns the degree of inter-subject consensus that was found in these studies. Individual consensus levels among auditors ranged from .66 in the Ashton and Brown study to .70 in the original study conducted by Ashton. The students that were used in the Ashton and Kramer and in the Trotman et a1. studies had a 31 level of consensus of .66 and .56 respectively. Although none of the other studies achieveed individual consensus levels equal to the .70 in the original Ashton study, their results can be considered similar. The Trotman et a1. study also had similar levels of consensus among groups that made internal. control judgments: .61 for two member groups and . .68 for groups with three members. On the other hand, the work of Reckers and Taylor had a level of consensus that was considerably lower (.15) than the other studies. This difference is probably related to the difference in the task structure, a point which is extremely important when people are faced with the problems of making decisions [Einhorn and Hogarth, 1981] or exercising judgments [Newell, 1968]. Ashton [1979] noted that the level of consensus between individuals in his study was below the .70 average in many cases. In addition, Ashton observed that there were great individual differences in the importance attributed to each factor. When these works are considered together it appears that although there is some consensus in the evaluation of internal controls even with a small number of factors (six in the Ashton study and its replications) the agreement varied, and as the number of factors increase, the consensus dropped dramatically (.15 for the Reckers and Taylor study). A second point that can be made on the original Ashton study and its subsequent replications has to do with the consensus subjects had concerning important control factors. 32 The studies used a number of different factors such as whether or not the documents were prenumbered and whether personnel were rotated between particular functions. The subjects considered the question that dealt with the separation of duties in the hypothetical payroll cycle to be most important. In all of the studies, approximately 50% of the variance in judgments could be explained by this one environmental feature. Therefore, there would appear to be some consensus about certain factors that are important. A final point that was examined in a number of these studies was the difference between judgments of groups with various degrees of experience. In these studies, it did not appear that experience had any effect on the judgment measures that were examined. The studies cited previously in this section simply examined internal control evaluation. Earlier, it was noted that these evaluations would have an impact on the audit tests that should be performed, i.e., the judgments made about internal controls should have an impact on judgments concerning subsequent portions of the audit. The study done by Joyce [1976] examined the impact of information about the accounts receivable subsystem on the number of man-hours that would be allocated in an audit program. The level of consensus in this study was low (.37) in terms of the amount of audit work, but the separation of duties factor accounted for most of the variance. 33 The next section examines some of the implications for this study of these preceeding sections. Implications 9f Che lnrernal Control Literature From the literature that dealt with the nature of controls and the possible approaches to control evaluation, a number of points can be summarized which have implications for this study. The following sections will examine the task structure that was used and the effect that the levels of consensus have for this study. Strucrurlng 9: Che Evaluation Task In the previous sections which reviewed research that examined internal control judgments, the task was structured around a particular accounting cycle (payroll in all of the studies with the exception of accounts receivable in the work of Joyce). In addition, for most of the work that provided guidance to the evaluation process, the authors suggested that it is necessary to approach the evaluation by breaking down the entire system into smaller more manageable components. The suggestion that the appropriate method for doing this is by transaction cycle was found in all of the studies examined. The manuals from the accounting firms specifically use this approach (in particular, see Arthur Andersen & Co., [1979]). Therefore the structuring of the evaluation task by accounting cycles would certainly seem to be very close to the way auditors approach this analysis. 34 This point is important in terms of limiting the scope of judgments to be examined while maintaining a realistic task structure. Levels 9f Consensus and Experrisg A second issue that is important for this study concerns the consensus measures that were obtained. One' observation that could be made concerning the studies examined earlier (all of which examined either control factors or evaluation frameworks) is that no overall best method emerged. Except for some very general features, such as defining control objectives, many different methods to either control or evaluate systems were examined. Therefore, it would seem unlikely that absolute consensus could exist. Furthermore, in the decisions studied, consensus must begin at a very elementary level and continue throughout the entire process. In Ashton's initial study, for instance, the consensus levels that were obtained were contingent upon agreement over (1) the weakness (or strength) represented by each of the factors, (2) the importance of the factors, and (3) the appropriateness of the quality assignment. In the Reckers and Taylor study, the number of factors considered was increased, thus augmenting the number of items that must be agreed upon in order to achieve consensus. In the Joyce study, the level of consensus must continue to yet another decision beyond the evaluation of the controls, that is to the point of deciding on the appropriate amount of audit 35 work. The distinction between the different decisions that must be made is similar to an argument made by Carr [1979] concerning goal-setting processes. He identified two stages to this process: (1) identifying an appropiate cue, and (2) deciding on the response to the cue. Thus consensus in these studies could be affected by the number of decisions' that must be made in arriving at a final answer. Closely related to the effect on consensus levels of the complexity of the decision is the task structure used in these studies. By restricting the auditors to a particular set of information, they were forced to make a judgment in a context that might not correspond to the way in which they would evaluate an actual system.. The importance of task structure [Einhorn and Hogarth, 1981] is considered critical in studying decisions, and therefore, the restricted setting will not necessarily improve consensus. In addition to the general issue of consensus, the difference (of lack thereof) between the consensus measures for groups with various levels of experience is also important. In these studies, there wasn't any significant difference between auditors with different years of experience. Joyce and Libby [1982] report on a study by Hamilton and Wright where the purpose was to examine differences due to experience. In that study there was a greater degree of consensus among auditors with more than three years of experience as compared with less experienced 36 subjects. The combined results from all of the studies examined does not give conclusive evidence of any systematic effects of experience. This raises the question of the appropriate definition of auditing expertise and selection of a subject for this study. However, the problem of generally non—significant differences between groups of different levels of experience could be a problem of task structure. The fact that the single cue related to separation of duties accounted for most of the variance is an indication that the studies presented such simplified task structures that judgment differences which would be made by more experienced auditors were not required in the particular experimental setting. These two issues, general consensus and effects on consensus due to experience, are important to the particular method that was used in this research to investigate the auditor's judgment. Joyce [1976] argues that in situations without an objective answer, a necessary (but not sufficient) condition for expertise is high level of consensus among the experts. His argument is based on views expressed by Einhorn [1974] who also makes the point that different schools of thought on a particular judgment will influence the level of consensus. In addition to this view expressed originally by Einhorn, Joyce also examines Mautz and Sharaf's [1961] argument concerning a prudent man (an expert auditor) exercising as sound a judgment as someone 37 else when presented with the same information. There is a fundamental difference between agreement, or consensus, over a final evaluation of internal controls and exercising sound judgment. Mautz and Mini [1966, p. 291] emphasize this point: Personal standards could undoubtedly affect an auditor's assessment of the seriousness of a given weakness (probable irregularity), but not his conslusion as to the presence of that weakness. Thus, experts should agree over the existence of internal control weaknesses, but not necessarily over the quality (seriousness of the weakness) of the internal control system. Therefore, when an expert decision maker is evaluated, the soundness of the judgment (as evidenced by the reasoning process) would seem to be more important, than agreement over the final decision. The method that was used in this research to investigate the auditor's decision attempted to reproduce the judgment process, and therefore represented a different orientation than previous studies. These studies were based on what has been called a stimulus-response paradigm [Lachman, Lachman and Butterfield, 1979] which seeks to find the relationship between variables and decisions using linear statistical models [Libby, 1981]. In studies that use these types of models, it is necessary to have cues that have low correlations, if an interpretation of their 38 importance is the aim of the study. If cues or independent variables are highly correlated then the interpretation of their importance is very difficult [Kerlinger and Pedhazur, 1973]. More realistically, however, information is related, and cues are therefore correlated. The redundant nature of the environment is a characteristic of most decision) settings. In the evaluation of internal controls, the literature suggests that this is the case; there are many different controls and many different combinations that could result in a well controlled system. In such situations, the reduction of the redundant environment to a few cues can make the task artificial. Ashton [1979] commented that even within the restricted set of cues (in his initial study) individuals varied greatly in the importance they placed on various factors, an observation that is not apparent from an examination of the data. It is these points (reproduction of the judgment . process, construction of a realistic decision setting, and the examination of the importance that individuals place on different information) that distinguish the method to be used in this study from previous studies that examined internal control judgments. Recently, a different approach to the examination of decisions has been used in a number of studies. This alternative approach is based on an information-processing paradigm [Newell and Simon, 1972]. In the studies mentioned 39 previously the process that came between the cues and the decisions was not examined. By not examining the process that is used to reach a decision (an area of interest in this study), components that are abstracted away by linear models are not available to the researcher. The information that comes from the studies concerning - evaluation can not be used in a setting that does not have cues of the same type presented to the subjects. In this study, the interest is in the types of cues that the auditors consider important in a realistic setting, and in the process that is used to evaluate this information. The process tracing methodology [Einhorn, Kleinmuntz, and Kleinmuntz, 1979] is used to collect data on different dimensions of the process that an individual uses to attack a particular problem. In a recent study by Biggs and Mock [1983], a particular form of this methodology was used to collect data from auditors with different experience levels as they evaluated internal controls and made audit decisions. The method used was the collection of verbal reports or protocols from the individuals as they solved the problem. The use of these reports as data allows the researcher to examine the way in which the decision is made [Ericcson and Simon, 1980]. This enabled the researchers to observe (l) the effect that certain pieces of information had on the importance of other pieces of information, (2) the way in 40 which cues were combined to reach certain conclusions, and (3) the amount of time spent in making the decision. From the data presented, it did not appear that experience caused any systematic differences. That is, the experienced auditors did not always look at more (or less) information or always take less time to reach a decision. Einhorn et~ al. [1979] point out that certain features of process tracing and linear models might be similar in certain situations, but there are also pieces of information that will not be available from linear models. The differences that can be observed in the Biggs and Mock study reflect what Newell [1968] identified as situation-specific algorithms, or processes that are constructed by the decision maker to meet the characteristics of the particular task. The literature examined earlier also suggests that there will be many situational factors that will affect the evaluation process. For instance, the type of system that is used to process data can differ along a number of dimensions (including the degree to which database systems are used), and each different type of system could affect the evaluation process. In linear models, the differences across situations not specifically considered would be cancelled as all the decisions are aggregated. In addition, it is important that the number of factors not be too large due to the limitations of linear models in the area of interpre- 41 tation. In process models, these situational differences become part of the data, and they therefore can be analyzed directly. There are other methodologies that have been used to examine the process that is used in problem-solving tasks. Retrospective verbal reports [Ericcson and Simon, 1978] collect the verbal data after the task. This technique was also used in an accounting context [Larker and Lessig, 1983]. The technique that was used in this study consisted of building a computer program that captured and represented the knowledge from the decision maker in a form that would allow the computer to simulate the decision maker. These simulations are also related to a specific individual (in this case an auditor), and they attempt to encode the subject's knowledge in a particular area as opposed to simply capturing a particular decision that was made in a certain setting. Computer programs, such as the ones described above, have been termed ‘expert' or knowledge based systems. They are attempts to simulate the knowledge used by a particular individual to solve a particular task. This desire to build a model that could act like an individual decision maker was contemplated by Libby [1975] in his development of a linear model. The difference between these two models, however, is that linear models must act the same across many different situations within the same task; while expert systems do not 42 contain a single general algorithm that would apply in all situations. Instead, expert systems (as simulations) attempt to model the situation-specific algorithms that Newell [1968] mentioned and that seem to be functioning in the Biggs and Mock study. In fact, Biggs and Mock suggest that computer programs of these decision makers could be a next A step in this line of research. The next section of this dissertation examines two areas; (1) the concept of expert or knowledge based computer programs and the ability of such systems to simulate human problem solving behavior, and (2) the literature on database models and the representation of constraints or restrictions on the possible states in these models. CHAPTER II EXPERT SYSTEMS AND DATA MODELS This chapter discusses two concepts. The first is the idea that computer programs can be constructed which simulate certain features of human problem solving ability. These programs have been referred to as expert or knowledge based systems and represent the knowledge that a particular person uses to solve problems in a domain. If the person is an expert in the area then a model of this person can be considered an expert system. These systems are models of how certain pieces of data can be used to solve problems in certain areas. In the second section of this chapter the concept of data models is discussed. These are models of the data which pertain to a particular portion of the world. In this sense data models are also models of knowledge; models of knowledge about the data which describes that part of the world. This chapter discusses these two types of knowledge: models of procedures that use data to solve 1>roblems in a particular domain and models which represent ltnowledge about the data that can describe a portion of the world . 43 44 EXPERT SYSTEMS This section examines some of the literature which discusses the use of computers to build models of problem solving behavior. The first section examines the basis for using computers to simulate human intelligence. The next section . discusses the difference between intelligent_ programs and expert or knowledge based systems. The third section discusses a particular type of computer program, a production system, that has acheived expert problem solving behavior in certain domains. Finally, the fourth section examines the role of these Systems in building models of human intelligence. Simiatiensflflellissass The idea of using computer simulations as a tool to study a particular phenomena is certainly not new. This general technique has been very useful in a number of disciplines as a method of testing the adequacy of certain ideas about the phenomena under investigation. One of the first questions that a researcher using a simulation must answer concerns expectations about the result. For instance, when an econometric model is constructed there might be some preliminary expectation that the model be able to predict certain features of an economy. However, when a researcher attempts to build an expert system -— a computer model of a human expert solving a jproblem —— there is a fundamental difference between the 4S expectations of the results of this type of model as opposed to other computer simulations. This point was expressed by Miller [1981, p. 220]: If a computer were used to model weather, no one would fear that a cyclone might destroy the computer center. [However] A computer that models an intelligent brain is expected to be a brain, to display actual intelligence. This feeling that somehow the capabilities of a computer are more closely related to the production of intelligence, i.e. the capabilities of the human mind, than to the production of a cyclone or an economic system is a very powerful comparison that has brought together researchers in the fields of artificial intelligence (computer science) and cognitive psychology. Newell and Simon [1972] see this similarity as being attributable to the view that both computers and human minds are processors of information. Newell [1980] suggests that this similarity goes beyond the activity of information processing to include the type of information used by both humans and computers; the ability to use and manipulate information in the form of symbols. In a previous work, Newell and Simon [1976] argue that this concept of a physical symbol system, and its accompanying hypothesis that humans and computers use similar symbols, is the most fundamental contribution to the joint work of artificial intelligence and psychology. 46 The development of this notion of a physical symbol system becomes a very powerful argument to explain the expectation expressed by Miller. However, its importance goes beyond simply providing a theoretical basis for this expectation. Newell [1980] argues that not only are human minds and computers examples of physical symbol systems but. also physical symbol systems are instances of universal machines. In this context, machine ”can be understood as an abstract mathematical system, or an abstract process, whose states and changes of state can be described by four or five elementary operations [Lachman et al., 1979; p. 95]. By demonstrating that a physical symbol system is an instance of a universal machine, two important results follow: (1) a physical symbol system "has as much flexibility as it is possible to obtain [Newell, 1980: p. 151]“, and (2) "any effective procedure (one that can be specified) that can be accomplished and characterized by human problem solvers can be acheived by a (universal) ‘machine' [Lachman et a1. discussing Turing [1936]; p. 95].“ Therefore the development of the concept of a physical :symbol system and its relation to universal machines, "provide[s] a basis for characterizing abstract mental operations [Lachman et al.: p. 96].“ The characterization of a mental process in terms of a computer program becomes what Newell and Simon [1972] call 47 “sufficient theories" of the process under study. That is, the development of any program to represent a theory of a particular cognitive process, such as problem solving, can be evaluated on a sufficiency criteria based on its ability to produce the behavior under study [Payne, Braunstein, and Carroll; 1978]. However, it is important to stress the type - of sufficiency that is required by a particular study. That is, in a particular study is a sufficient theory of the task or of the person performing the task required. In this study the examination of certain aspects of the process was important and therefore a simulation of the problem solver was required. The next section discusses the distinction between programs that solve intelligent problems and those that simulate a human problem solver. This is an important feature that differentiates expert systems from other types of computer programs. ELPELL EYELEES TELSHS Intelligent RLQQLAmS The notion of universality (flexibility) expressed by Newell implies that computers, as instances of physical symbol systems, can be instructed to perform an unlimited number of tasks. Even within the same task, there are likely to be a large number of strategies that will achieve the desired outcome. Within this broad range of possible approaches to the performance of a particular task, researchers interested in building expert systems seek to 48 simulate the subset of these approaches that model expert behavior [Young, 1979]. In defining the properties of an expert system, Brachman et a1. [1983] includes these features: (1) reasoning with symbols, (2) using expert rules, especially to avoid blind search, and (3) explaining the lines of‘ reasoning used to solve the problem. These characteristics of expert systems distinguish them from computer programs that obtain the appearance of intelligence through the use of powerful computer techniques. This distinction can be demonstrated by examining computer programs that play chess. For a person to be good at chess requires some intelligence. Therefore, if a computer could play chess very well, it might be considered intelligent. That is, the quality of the output from the program (whether the program wins most of its games) can be used to evaluate any computer program which plays chess. Many of the programs that are good at chess determine the next appropriate move by examining a very large number of alternative results to each possible move, i.e. they play out a number of move sequences to see what the results would be. These programs use a somewhat blind approach to chess playing and rely on the ability of the computer to examine many board positions in a short period of time. A different approach to the problem of computer chess playing attempts to limit the number of board positions examined by including information from human 49 chess players in the chess playing program. A program by Campbell and Berlinger [1983] includes ‘chunks' of knowledge from human chess experts about the strengths of various board positions in the program and uses this knowledge to pursue only those move sequences which result in strong positions. This difference in approach corresponds to the dichotomy between programs that are intelligent artifacts and those that psychological theories [Feigenbaum, 1979]. Lachman et a1. [1979: p. 105] describe two types of systems that can be simulated: (1) those that are well developed, and where laws and principles are complex but known, and (2) those that are underdeveloped. Chess would seem to be an example of the first type, and therefore strategies to perform the task could simply use the laws and principles of the game as opposed to requiring an examination of human experts. It is possible to determine if the program has an adequate appreciation of the laws and principles of chess and included them appropriately in its playing strategies by examining its ability to win a sufficient number of matches. This outcome can be used to evaluate the program whether it uses powerful computer techniques or human knowledge of chess to achieve the result. On the other hand, the evaluation of accounting controls is quite different. Earlier it was argued that there~isn't any well accepted objective criteria that can be 50 used to determine if someone has come to a correct conclusion about controls. There are effectively an infinite number of data items that can be looked at and it is not clear what each piece of data means individually or collectively. Because the laws and principles of this problem domain are not well known and the outcome of the‘ decision is hard to measure objectively it would not be possible to use any of the ‘blind' strategies that have been successful solving chess problems to evaluate accounting controls. That is, it would-not be desirable (or possible) to take advantage of any powerful computer techniques, such as the computer's ability to examine many different positions in a short period of time, in solving the control evaluation problem. Therefore in order to simulate intelligence in this particular area, it is necessary to study the methods of human experts as they solve this problem. This distinction beween power and knowledge has also been made within those studies that seek to simulate human strategies. MELLLGSSWWQWW Goldstein and Papert [1977] identify what they call a ;paradigm shift in the understanding of intelligence: The fundamental problem of understanding intelligence is not the identification of a few powerful techniques but rather the question of 51 how to represent large amounts of knowledge in a fashion that permits their efffective use and interaction [p. 85: emphasis added]. Pylyshyn [1973] discusses this difference not so much as a paradigm shift but more as a distinction between competence and performance theories of cognitive behavior. He suggests that this is a difference in emphasis between what McCarthy] and Hayes [1969] identify as, “epistemological and heuristic problems in the design of intelligent automata [Pylyshyn, 1973: p. 22]." Pylyshyn argues that the difficulty in understanding the epistemological aspects of a cognitive ability is due to the lack of well developed methods to examine the mental structures which generate the “moment-to- moment“ (or heuristic) problem solving procedures that are observable. It is these heuristic problem solving methods that were studied in this dissertation using computer simulations. This shift in emphasis, or distinction between different approaches to models of cognitive abilities, serves to differentiate the work done by Newell and Simon [1969] on GPS and by Fikes and Nilsson [1971] on STRIPS from subsequent efforts. Both ‘GPS and STRIPS attempted to simulate human problem solving on a computer using general strategies that the researchers though might be applicable ‘to diverse situations, i.e. strategies that were required to be competent at solving a wide range of problems. The sgeneral strategy that was used in GPS involved assessing the 52 difference between the current state and the goal state, and then applying operators that would most effectively reduce this difference. The STRIPS program uses the general problem solving (GPS) approach and applies it to the task of theorem proving. The shift away from attempts to simulate intelligence through the use of general problem solving skills to one that seeks to uncover and build into a simulation program domain specific knowledge to achieve expertise was due both to the lack of adequate methods to examine the mental representations discussed by Pylyshyn as well as to the inability of the general problem solving techniques to deal with certain types of situations.‘ Winston [1977] points out that certain situations required 'a model for going deeper into the process by which salient facts are recognized and resulting facts are deduced" [p. 144: emphasis added]. Therefore, as opposed to those general skills, current researchers seek to uncover and transfer expertise or rules of ”good judgment" used by the expert practitioner [Feigenbaum, 1979: p. 7] to the programs that they are building. The next section examines a particular type of system, production systems, that have been used in a number of astudies that have attempted to develop computer programs t:hat contain these rules of good judgment. 53 W mat Production systems are programs that have been used to model expert problem solvers in a number of domains. Examples of these types of programs include: DENDRAL for deducing molecular structure [Feigenbaum et al., 1971], PROSPECTOR for mineral exploration [Duda and Reboh, 1983],~ MYCIN in the area of medical diagnosis [Shortliffe, 1976], AUDITOR to evaluate the allowance for bad debts [Dungan, 1983: Dungan and Chandler, 1983], and in the ACT and ACT* systems [Anderson, 1982 a 1983] to simulate general cognitive activity. The diverse nature of the domains in which these programs operate is an indication of the flexibility that can be achieved using production system architecture. In these systems, the problem solving knowledge from experts is encoded in the form of productions [Newell and Simon, 1972]. Productions are rules of the following form: SITUATION -> ACTION. When a production is selected, the program searches its collection of data to see if the particular situation exists. If it does, then the action portion of the rule is executed. The following is an example of a production that might exist in an expert system that examines accounting controls: IF (1) the client has a good control environment, (2) the controls adequately cover the risks, and 54 (3) the client has good monitoring of controls: THEN conclude that the client has effective accounting controls. This example serves to illustrate a number of characteristics of expert systems that were mentioned by Brachman et a1. [1983]. These include the ability to use. and manipulate symbols, the ability to use knowledge (especially to avoid blind search), and the ability to explain the line of reasoning that was used to reach a particular solution. The next three sections deal with these features in production systems. MW Symbols are collections of Characters that are used by physical symbol systems to represent particular concepts internally. In the production example cited above, symbols representing client, risks, and control environments are used in the situation portion of the production. These symbols have two features that make their use important to the construction of these systems. The first feature concerns a hypothesis expressed earlier: that computers and human minds not only use symbols but that the symbols are the same. The symbols used in this particular prodution deal with real-world concepts that a person familiar with the domain of accounting controls might use. Therefore the symbols used and manipulated have semantic properties that give the system the ability to deal 55 with the problem in the same context as the expert. This allows the expert to understand the reasoning process that is being used. The second feature of symbol manipulation is that expert systems use concepts in their processing. This can be contrasted with traditional program that use syntactic- processes to perform tasks. This difference between expert systems and other computer programs was noted by Gorry and Krumland [1983] in their examination of artificial intelligence and decision support. They argued that one difference between expert systems and other decision support programs concerns the use of implicit and explicit information about the particular problem. In traditional programs, the knowledge is implicit in the program: the reasons for particular actions are not part of the program. In contrast, the ability of expert systems to use and manipulate the symbols related to the problem makes the knowledge explicit. Further, the goals are also represented in terms of symbols as opposed to being implicitly encoded in the branching decisions of the program. Newell [1982] describes knowledge in terms of the principle of rationality. This description ascribes knowledge to a particular agent, if its behavior can be computed in terms of rational selection of actions to meet goals. Furthermore, the selection of these goal directed actions are conditioned on states of the world. Newell 56 [1980] argues that the use of symbols is both a necessary and sufficient condition for this rational and intelligent behavior. The explicit use of symbols to represent goals rather than implicitly embedding them into the system allows expert systems to adapt to conditions, and to respond to these conditions rationally (that is, to meet the goals‘ that are explicitly represented in terms of symbols). RLQQHQLIQD. Kngfllsdss is. the Essrgh. RLQQQSE There are basicaly two types of search strategies that have been used in production Systems: data driven and goal driven. The selection of strategy depends on the problem domain. Backward chaining (goal driven search) is effective in situations that have few goals in relation to the possible pieces of evidence [Nilsson, 1979]. In contrast, forward chaining (data driven) is effective when there is a small amount of data compared to the possible goals. DENDRAL [Feigenbaum et al., 1971] uses mass spectral and nuclear response data to identify candidate molecular structures. In this system, the search for candidate structures is data driven. The productions limit the search by pruning implausible substructures from further consideration [Brachman et al., 1983]. The search strategy that was used in this study was backward chaining or goal-driven. In this strategy, the productions serve to acquire (search for) information that 57 PRODUCTION 1 IF (1) the client has a good control environment, (2) the controls adequately cover the risks, and (3) the client has good monitoring controls; THEN conclude that the client has effective accounting controls. PRODUCTION 2 IF (1) the firm has a hierarchical organizational structure, (2) the client has personnel policies which ensure competent employees, and (3) the client has a competent internal audit staff: THEN conclude that the client has a good control environment. PRODUCTION 3 IF (1) the client has well defined personnel requirements, (2) the client has adequate training programs, and (3) the client has adequate supervision of employees; THEN conclude the client has personnel policies which ensure competent employees. PRODUCTION 4 IF (1) the clients internal audit staff is headed by a CPA, (2) the internal audit staff reports to an appropriate level within the firm, and (3) the internal audit department is not restricted in its investigations: THEN the client has a competent internal audit staff. PRODUCTIONS AND BACKWARD-CHAINING FIGURE 2 58 is relevant to the particular problem. In Figure 2 there are some examples of productions that might represent some of the knowledge necessary to evaluate a firm's accounting controls. Production 1 might represent the initial production selected by the system because it makes a conclusion about accounting controls which is the major goal] of the system. One of the conditions in the production relates to the control environment of the client. This is a piece of knowledge that must be obtained to make a conclusion about the controls. In order to obtain this knowledge, the system must look for a production that would allow this piece of knowledge to be deduced. In this example production 2 makes a conclusion about the control environment, therefore a system that is goal-driven would examine this production next. In order for production 2 to make a conclusion about the control environment the three conditions must be met. In order to satisfy the first premise the system would search for a production that concludes about the organizational structure of the particular firm. In this example there aren't any productions which make a conclusion about corporate structure, so the system would request that this data be supplied. The second condition of production 2 requires information about the personnel policies of the firm. A search of the productions in the example reveals that production 3 can make a conclusion about the personnel 59 policies which in turn is used to satisfy the major goal of the system. This production would be selected and its premises would used by the search procedure to see if there were productions which make conclusions about this data. This search strategy is called backward or goal—driven because the search for information proceeds backward from. the goals to the individual pieces of information. Therefore, the search is based on meeting particular knowledge requirements, or as Schank and Abelson [1977] term them, instrumental (I-goals) goals, such as the quality of the control environment or the organizational structure. In this context, the complex decision or main goal is seen (and therefore modeled) as a series of simpler decision rules applied sequentially in time [Montgomery and Svenson, 1976]. The procedure which searches for information represents a plan that will yield knowledge about a particular situation. Hayes-Roth and Hayes—Roth [1979, p. 276] define a plan as, "a course of action aimed at achieving some goal.” In the case of internal controls, the plan involves acquiring the knowledge necessary to make some conclusion. The flexibility of the production system architecture allows the firing of different productions based on information that becomes available to the system in the process of reaching a goal. Thus the plan used to attack the problem can be adapted to the particular situation. 60 In production systems, therefore, search is not blind. It is based on the ability to identify the necessity of certain information or on the impossibility of using a particular path, to achieve some goal. The ability to use symbols to represent goals and the ability to take actions (search for information or molecular structures) to meet those goals provides the system with a rational (as opposed to a blind) approach to the particular problem. Explanarlon 9f,the.LiB§ 2f.Beasenins The requirement that expert systems be able to explain and therefore justify their line of reasoning is an essential feature of these types of programs. This ability can serve as a further point of distinction between mathematical or statistical models and expert systems. The outputs of mathematical models, in whatever form, do not allow for reflection on the process that gave the particular result [Minsky, 1975]. Therefore, in situations that do not have objective solutions, mathematical models can obscure the process that arrived at the number; a process which in many cases is as important as the solution. In evaluating decisions that are made in areas that do not have well- structured solutions, it will be this reasoning process that is examined. In the area of internal control evaluation, the ability to justify or provide a line of reasoning is crucial to understand the final judgment. For this reason, in a simulation of an internal control decision an 61 explanation facility would seem to be extremely important. There are, however, some arguments which suggest that auditors (or any problem solvers) will not be able to provide accurate insights into their decision process and therefore this explanation or line of reasoning facility will not necessarily be beneficial in constructing a theory. of the audit judgment. Nisbett and Wilson [1977] have questioned the ability of problem solvers to access the mental processes actually being used and then to accurately construct verbalizations about this process. They argue that in situations which are characterized by a high degree of automaticity it is even more doubtful that there will be access to the mental processes. In the context of auditing decisions there are a number of factors that might contribute to an auditors ability to construct explanations of their mental process. Auditing and in particular the internal control evaluation judgment is what Einhorn [1980] calls an "outcome irrelevant learning structure". These are learning situations in which feedback will not be useful in correcting judgment errors. This is due to the fact that it is hard to identify situations in which professional auditors are obviously incorrect. In recognition of the difficulty of simply reviewing final decisions, the evaluation of auditors is done through a review process which in part examines the formulation of the judgment. As 62 a result of this review process, the ability (or requirement) to justify or provide the reasoning for a judgment is crucial as it serves as a basis for subsequent evaluation. Part of the training of auditors is in the process of reaching judgments and in the documentation of these judgments. This type of training speaks to arguments] made by Ericsson and Simon [1979 and 1980] concerning the ability to access and verbalize about the process being used to solve a particular problem. First, because auditors have training in developing documentation of their decision processes arguments made by Ericcson and Simon indicate that the information required to construct a line of reasoning will be in verbal form and will be readily accessible. Second, to the degree that the repetition of the auditing task will result in a degree of automatically this will result in 'metastatements' [1980, p. 227] about the process being used. If this problem does exist it Should be mitigated by the documentation requirements. Because the reasoning is so important to any auditing judgment it is also an important component of any cognitive theory of an auditor's judgment. Production systems facilitate this explanation capability by keeping track of the different pieces of information used and the reasons for their use (the production whose premise was being satisfied by the information). This type of architecture allows lines of 63 reasoning to be constructed. The process that is used to search for information —- backward-chaining or goal-driven-- is therefore an integral part of the ability of these systems to provide explanation facilities. The next section examines the use of production systems as theories of cognitive ability in humans. Expert Systems ee Cogpltlye Theories Previously it was argued that the notion of a physical symbol system provided a basis for the expectation that the computer would be able to simulate human intelligence. In addition, a number of studies were identified that used a production system architecture to achieve a level of performance that was comparable to human experts in complex problem environments. Despite these observations and despite the fact that these systems required the representation of a great deal of human—like knowledge to achieve their performance levels, this is not apriori evidence that humans must also work with large amounts of knowledge and production rules to be experts. This section will examine some evidence that might suggest that these are appropriate methods to represent a theory of human expertise. mgmmmm If an expert system is to be viewed as a theory of human problem-solving, it is important to determine the level at which the theory operates. Longuet-Higgins [1981, 64 p. 198] discusses three levels at which a theory of cognitive behavior can be expressed: (l) by examining and building a theory of the task itself, (2) by specifying effective procedures "for proceeding from initial state to final state", and (3) by proposing steps that will be followed at the neural level. It is not the purpose of this. study to make any claims that the resulting simulation is a theory of neural level process that are required by a human to perform the particular problem-solving task. In order to simulate natural systems with a computer, it is necessary to move from the physical to the logical level which is the only level at which a computer provides an analogy of the mind [Lachman et al., 1979]. At this logical level, Young [1979] identifies a number of different views of the production system architecture: (1) the language level (as a convenient way of representing the language of the problem), (2) the rule level (as a ”commitment to the psycholoqical reality to rules"), and (3) at the immediate processor level (“as a theory of the structure of human cognitive process [where, for instance] actions on the righthand side of the rules... are taken to be the ‘elementary information processes'). This study will not attempt to develop a theory of the physical or neural activity of the human mind, but instead will build a model of the knowledge necessary (in the form of rules) that is necessary to solve internal control 65 evaluation problems. If the distinction is made between .the brain structure, the mental operations that such a structure can perform, and the knowledge that is used as data for these operations then it would appear that the study of intelligence in auditing is more appropriate at the- knowledge level. This assumption is based on the physical symbol hypothesis expressed earlier in this section [Newell and Simon, 1976, Newell 1980]. This hypothesis argues that a physical symbol system is both a necessary and sufficient condition for intelligence. Newell [1980] enumerates the mental operations, or what Pylyshyn [1981] identifies as type I processes that these systems must be able to perform. If all intelligent agents must be able to perform these operations then intelligence in different areas can not be the result of differences at this level, it must be due to differences in the knowledge that the agent can bring to the problem. Goldstein and Papert [1977] have made a similar argument; that human intelligence is not due to the development of different brain structures or mental operations, but it is due instead to the acquisition of knowledge in the area. In additionto these arguments, there is also empirical evidence [Card, Moran, and Newell, 1983: Chi, Feltovich, and Glaser, 1981: Klix, 1979] which suggest that intelligence in a particular area is related to experience, and therefore knowledge, about the domain. The 66 computer analogy is very strong on this point: the hardware of a computer only facilitates certain activities, a program is still required to produce the activities. Newell [1982] argues for this knowledge level as consisting of activities at the symbol level. The "representation exists at the symbol level...(data structures and processes) that realize a body of knowledge at the knowledge level [p. 100]." By working at the knowledge level, it is possible to predict and understand behavior without having an operational model of the processing being carried out to exhibit that behavior. In this study, a theory of the knowledge that an auditor uses to evaluate internal controls is being developed. It is a logical theory at the rule level that is not attempting to describe the physical process that are underlying the problem-solving behavior. By developing a theory at this level, it is possible to understand how knowldege is used. That is, our theory passes the requirements of being useful [Schank and Abelson, 1977]. In addition, a theory at this level will allow an evaluation based on its ability to generate the step-by-step process without dealing with the physical reality that is supporting the outward behavior [Payne et al., 1978]. This theory of intelligence has two implications. First, it is human orientated and must be studied in humans. Second, learning is an important factor, and therefore 67 situations in which the knowledge was acquired become important in understanding the knowledge. The next section examines the idea that in certain areas (or situations) particular people will have acquired a higher level of knowledge than others and will be identified as experts. Theories pf Expertise The difference between experts in a particular domain and novices in that same domain has been studied by a number of researchers and it is important to an understanding of expertise. Card, Moran, and Newell [1983] connected experience with the control knowledge of the person. The authors noted that with a low level of experience, the person ‘wandered' in search of a particular goal. As experience increased, the person became more efficient at controlling the search for the goal. The ability of experience to make problem-solving more efficient. was alo identified by Klix [1979] in a study that contrasted mathematically-gifted with normal adults. When presented with difficult math problems, the gifted group, "revealed in the first trial a higher efficiency in their strategy“ [p. 3]. That is, they were able to gain information on the problem structure in a more intensive fashion than the control group. Similar observations were made by Chi, Feltovich, and Glaser [1981] in their study of the way in which experts and novices solved physics problems. They 68' noted that the experts thought in terms of general physical principles (such as conservation of energy) while the novices dealt with the particular items specifically represented in the problem. The results of these studies indicate a number of important aspects of expertise. The first point is that expertise is related to the\ amount of experience that the person has with the particular task. The more experienced a person is with a particular problem environment the greater will be the skill (in the case of the work by Card et al. the greater the efficiency) of the person. A second point is that expertise can be related to the level at which the problem is represented. This point, concerning the importance of representation to the problem-solving process, has been made by a number of authors. Newell [1968] and Einhorn and Hogarth [1981] have argued that understanding a particular judgment or decision process requires an understanding of the represenatation of the environment that is being used. Einhorn [1980], in discussing the results from studies using "problem isomorphs', indicated that making heuristic models more predictive would require models that deal with the task as represented. A final point is that expertise is procedural in nature. The fact that experts had higher level representations of the problem indicates the presence not only of knowledge of how to solve a problem once it is placed in a particular representation but also of knowledge 69 of when to use the particular model. In this case, the cognitive theory deals with the mechanism that determines how knowledge is put to use [Scandura, 1979]. Therefore, expertise consists of knowledge about actions as well as knowledge about situations in which these actions are appropriate. This concept of expert knowledge is the basis. of its representation in terms of the productions discussed earlier. There is also some additional evidence that production rules can express the knowledge of individuals, expert or otherwise. Young and O'Shea [1981] developed a production system to model the way in which children solve subtraction problems. The system that they developed produced errors that were similar to those made by the children, and in addition, they were able to correct these errors by adding certain productions to the system. They argued that the errors were not due to an inappropriate application of a set of rules, but to an absence of rules that dealt with actions required in certain situations. Hunter [1962] studied a mathematician who was able to perform impressive calculations in his head and concluded that this person had a collection of mathematical facts and a knowledge of the situations in which these facts should be used. This collection of evidence seems to indicate that expert knowledge is acquired through situations that require the use of that knowledge. Furthermore, productions which 70 identify both situations and appropriate actions can model this type of knowledge. In addition, it becomes important to understand the situations in which the knowledge was obtained or learned to understand the use of the knowledge. The next section deals with the nature of heuristics and the relationship of heuristics to the recognition of situations. that cause certain actions. WEW In many task environments, researchers have found that decision makers to not use normative models. Einhorn and Hogarth [1981] argue that this is because the assumptions of normative models are generaly not met, or because subjective evaluations (especially of probabilities) are often required. Einhorn [1980] identifies a change in current psycological research on decision making away from the develOpment of normative models and toward an emphasis on the study of heuristic models of problem-solving behavior. Problem-solving behavior has been characterized as a search on a number of different levels: a search for information, a search for appropriate goals, and a search for a particular representation that will allow the problem to be solved. Simon [1980] has argued that the principle mechanism of intelligence is dealing with problem environments through the use of heuristic search. Simon discusses learning and heuristics in terms of adaption. Learning, he argues, is a semi-permanent adaptation to the 71 environment in which certain types of knowledge about situations was obtained. Heuristics then are the short-term adaptations to particular situations which are encountered. Thus, to understand the heuristics that a person uses to solve a particular problem it is necessary to investigate the situation that caused the process to be used. Further‘ when a heuristic model of the problem-solving behavior is constructed these situations must be included as part of the model. Some researchers have argued that the use of certain heuristics in decision making will produce biases which will result in an inappropriate solution. Tversky and Kahneman [1974] discuss a number of different heuristics and attempt to show that these approaches to a particular situation will result in an inappropriate response. There have been a number of studies of decision making that used auditors as subjects to investigate their reliance on inappropriate heuristics (Kinney and Uecker, 1982: Joyce and Biddle, 1981a and 1981b). These studies have concluded that in certain situations auditors use heuristics, anchoring and adjustment in these experiments, ' and therefore responded inappropriately to the situations in the studies. Einhorn [1980] has argued that the heuristic problem-solving observed in a particular situation is the result of an application of meta-heuristics (such as anchoring and adjustment) to generate rules to meet the observed structure 72 of the problem. If the use of heuristics is a response to a particular situation, as Simon [1980] has also argued, then the problem of studying the heuristics in artificial task settings would seem to confound the results of these studies, ie., once again the importance of realistic task structures is critical in understanding the problem-solving behavior. In situations where a person does not have the necessary experience, it is possible (and the evidence from Tversky and Kahneman, Kinney and Uecker, etc. suggests probable) that an inappropriate structure will be identified and the heuristics used to solve the problem will result in an inefficient or inappropriate result. On the other hand, the evidence from Klix [1979], Card et a1. [1983] (and to some degree suggested by the Biggs and Mock [1983] work) seems to indicate that through experience (or expertise) in a particular area, a person will be able to 'induce a structure that will be appropriate and efficient for the problem. Based on the structure that is observed people form an internal representation, or problem space (Newell and Simon, 1972), for the particular task they are confronted with. The internal representation will influence the efficiency of the solution method and will be affected by factors such as the experience of the problem solver. It can also be argued that the internal representation will determine the type of 73 information that is used to solve the particular problem. Lachman et a1. [1979] argue that cues will be selected for attention based on their importance to the current coqnitive activity. The internal representation that is being used will determine what portion of the available information is considered to be pertinent to the particular] problem-solving method. Viewed in this manner, the search for particular information to solve a particular problem is similar to an idea expressed by Norman [1968] concerning attention and consciousness. -His view was that some form of processing was done to all information in the environment (that is at some level humans are sensitive to all cues). However, cues were accepted for further processing (attended to) based on their semantic importance. Therefore the internal representation will have an impact not only on the efficiency of the solution, but will also have an influence on the information that will be considered important to the problem. In this sense, the heuristics used in response to a certain problem situation determine the internal representation which in turn controls the information that will be used to solve the problem. In production systems rules are formulated so as to capture not only the particular actions that a person uses in solving a problem, but also the situations that resulted in the selected action. In addition the structure which is used to connect the productions and select rules for 74 execution restricts the information that will be used by the system to those cues that are important to the particular problem-solving situation that the system is confronted with. Therefore production systems are able to model two important aspects of heuristic problem-solving: (1) the situations which cause certain heuristic problem—solving' methods to be selected, and (2) the search for information that the problem solver considers important to the task. SHEEQLY This section has examined the basis for using a computer to simulate certain mental processes such as problem solving. Evidence was presented that indicates that it is necessary to obtain and encode domain specific knowledge into programs that attempt to model expert problem solvers. Further, productions that identify situations in which particular actions should be used, serve as appropriate mechanisms to encode the heuristic nature of the knowledge of human experts. The next section deals specifically with the concept of data models and its relationship to expert systems. 75 DATABASES AND DATA MODELS This section reviews the literature that has examined different approaches to the modelling of data. In many instances data models were developed as part of database. management system, although this section will also discuss some approaches to the representation of data in connection with theories of cognitive processes. The first part of this section will discuss data models and their use in the representation of declarative knowledge. After this will be an examination of data models in an accounting context. Rate 11.94.06 s as. Mental ons 52f W fineness: Production systems are models of procedural knowledge that can be used to solve problems in a particular domain. However, to actually use these procedures, data or declarative knowledge is required. For instance, in the process of reaching a conclusion about accounting controls, declarative knowledge (such as the client's type of business and the frequency of certain errors in the client's, system) might be used by various productions. Therefore, declarative knowledge is an important element of the problem-solving process. Data models deal specifically with the representation of declarative information. As such, they are concerned with the organization of data in a form that facilitates its 76 use. Some data models have been developed in an attempt to represent declarative knowledge as part of a theory of human cognitive processes, while others are more concerned with data organization as part of a database system. Quillian [1968] developed a data model to deal with the organization of symbols in the human mind. His model, a» semantic network, attempted to explain how concepts could be arranged to accomplish certain types of natural language activities. The model used a hierarchical structure to represent the organization of semantic concepts in human memory. A typical hierarchy might represent the fact that a canary "is a“ bird which in turn "is a' animal. Quillian demonstrated that his model was useful as a theory of declarative knowledge by coupling it with a procedural component [Quillian, 1969] and using it in a program which understood English text. Pople [1977] has also used the semantic network data model as part of an expert system (INTERNIST) in the field of medical diagnosis. Database systems atttempt to organize data in certain computer application areas. Tsichritzis and Lochovsky [1982], in discussing data models, point out that an important feature of data is its connection to a particular meaning. For instance, the number “4275" or the word “hammer" have ambiguous meanings, but when they are attached to a particular concept (such as an invoice number or a customer's name), these pieces of data can be interpreted. 77 The introduction of computers into the data processing function initially resulted in a separation of data from its meaning [Tsichritzis and Lochovsky, 1982: p.41. Database management systems, through the use of data models, attempt to store both the data and its meaning together in the computer. The conceptual model of a database is a representation of the meaning of the data. Date [1982] argues that in order for a conceptual (or data) model to capture this meaning, it must be able to show two components of this data: (1) how the data is organized into relevant objects, and (2) the relationships between these objects. There are potentially many different conceptual models that could be used to represent these properties of data. There are, however, three data models that have achieved wide acceptance and which serve as a basis of comparison of other data models [Date, 1982: Tsichritzis and Lochovsky, 1982]. These three data models, --the hierarchical, the network, and the relational—- have been referred to as first generation data models [Lum et al., 1979]. These three data models have particular characteristics that have made them useful in the development of particular database management systems. Network and hierarchical data models evolved from early file processing or report generation systems [Tsichritzis and Lochovsky, 1982: p. 91]. In comparison, the relational 78 model was based on a mathematical theory of relations [Codd, 1970]. Due to the way in which these models were developed, certain inflexibilities resulted. To overcome such problems, Tsichritzis and Lochovsky [1982, chaps. 8-11] discuss a number of “higher-level" data models that provide more flexible structuring capabilties, and more explicit. constraint specification capabilities. One particular high-level (or second generation) data model that has received attention is Chen's [1976] Entity- Relationship (E-R) model. Sakai [1981] uses this particular data model to demonstrate a method for representing certain information structures in a conceptual schema (or model). There have also been many other research efforts that have used this particular conceptual model [Chen, 1980 and 1981]. An important feature of the E-R model is its ability to specify an overall or "enterprise schema" [Chen, 1977]. This facility has made this data model particularly appropriate as a method of representing corporate, and therefore accounting data. The next section discusses the literature that has examined data models (and also database systems) as representations of accounting data. We n PataHedels Due to the nature of the accounting profession, there is an inextricable connection between accountants and 79 business data processing. Therefore, there is a great deal of accounting research that is concerned with ways in which data can be used or presented effectively. When database management systems became integrated into the data processing activities of firms, it was quite natural for accountants to investigate the effect that such systems. would have on the accounting and auditing functions. This interest led a number of researchers to investigate the way in which accounting could be done in a database environment [Colantoni, Manes, and Whinston, 1976: Everest and Weber, 1977]. These researchers also attempted to demonstrate that the traditional accounting framework, which developed from bookkeeping requirements, might be inappropriate in a computer environment. This group of research has adapted the ‘Events' approach [Sorter, 1969: Johnson, 1970] to particular database systems (for a more complete discussion of the work combining 'Events' theory and . database approaches, see McCarthy [1981]). The data models that were used in these studies were first generation models. The work that followed from these studies developed accounting data models which included semantic properties of the data and therefore used "higher-level" data models. McCarthy [1979] extended the concepts developed in previous accounting data work to Chen's E-R model. The purpose of McCarthy's study was to demonstrate the development of an accounting model that represented 80 corporate information that was of interest, not only to accountants, but also to other users within the firm. McCarthy used the concept of "artifact—free" to describe the lack of any particular accounting conventions in the data model. Furthermore, this model was formulated using semantic properties of the data of interest, an approach_ which makes it independent of restrictions connected with particular application areas within a firm. In an advancement of this model, McCarthy [1982] used the concepts of generalization and aggregation [Smith and Smith, 1977a and b] to develop a general accounting data model. McCarthy argued that his general framework emerges through an examination of the important entities and relationships in his original model. He demonstrated that the accounting data in a business environment could be modeled in terms of the Resources, Events and Agents in a particular firm and the relationships among those entities. The REA model is therefore a general semantic representation of the entities and the relationships that characterize business environments. Tsichritzis and Lochovsky [1982] have suggested that higher-level data models (such as the REA model) can be translated into the three first generation data models discussed earlier. The appropriateness of the REA data model has been demonstrated by studies that have used its framework in the development of accounting systems in a 81 network database [Gal and McCarthy, 1983] and a relational database [Gal and McCarthy, 1985b]. The REA model attempts to capture certain semantic (or real world) features of data in a business environment. Tsichritzis and Lochovsky [1982] identify two components of a data model that are necessary to represent the real world._ The first is the static structures that are allowed to exist. One particular static structure that McCarthy deals with in the REA model is generalization hierarchies [Smith and Smith, 1977b]. An example of this type of structure would be the generalization of entities such as warehouse workers, secretaries, and accounting clerks into an entity called employee. In addition to these static structures, it is also necessary to show how Ythe real world changes. Therefore, a second component of a data model concerns the operations that transform particular occurences (or database states) into subsequent occurences. As part of these operations, certain restrictions or constraints are specified which identify either disallowed operations (such as the retrieval of privileged salary information) or disallowed states (such as a sale made to a nonexistent customer). The REA model deals specifically with the static structures of a business environment and as such, it does not represent explicitly the operations or constraints that would be necessary to capture certain semantic properties of corporate data. 82 Earlier in Chapter I of this work, the importance of controls in an accounting context was discussed. The nature of accounting controls relates directly to the concept of a constraint, i.e. the restriction of certain operations or states from occuring. In the terms that Vasarhelyi [1980] developed to define errors, a constraint would prevent the_ existence of certain internal or measured values that do not correspond to the actual conditions. Therefore an important feature of an accounting data model is the constraints that restrict certain states from occurring. The next section develops the concept of constraints and then relates them specifically to the REA model. Amour: 92m saéflafafledslmtrsiata Tsichritzis and Lochovsky identify a number of ways to view data model constraints. One such classification has to do with implicit and explicit constraints. An implicit constraint in the REA model is represented in terms of the structures that are allowed, i.e. all data items must be classified as belonging either to resources, events and agents or to the relationships between these entities. Explicit constraints are part of the data model and are represented separately. They insure that the information system represented by the data models, I'accurately reflect the real-world situation" [Tsichritzis and Lochovsky, p. 39]. Therefore explicit constraints ensure that the semantic and the integrity properties of the data model are 83 enforced. Theerachetmongkol and Montgomery [1980] use the concept of semantic integrity constraints to explain how the enforcement of certain types of restrictions result in a database that can identify certain configurations of the data that reflect real-world configurations that should not‘ exist. For instance, to maintain certain management requirements, it might not be allowed for employees be able to obtain salary information of other employees. The data model should represent this restriction. This is an example of an explicit constraint that is static in nature. There are also constraints which relate to operations and which therefore are dynamic in nature. An example of a dynamic constraint might be one that states an employee cannot be assigned to two departments. This constraint is dynamic, because it is related to an operation, the assigning of an employee to a department. The concept developed earlier in this proposal is essentially operation oriented, and it deals with constraints on data that reflect transaction occurrences. In an accounting context, the constraints or controls ensure that the particular occurrence of the database after the execution of a transaction matches the way in which the financial position of the business would be affected by the actual transcation. Accounting transactions (such as a sale) are translated into many database operations (such as 84 altering the balance of a customer, and reflecting the change of each item of inventory appearing on the sale). Therefore, to maintain integrity, a data model must be able to represent the execution of an accounting transaction with an overall or conceptual perspective of the necessary operations. Furthermore, an individual accounting~ transaction is part of an entire cycle which contains a number of different transactions. For instance, the revenue cycle includes both sales and cash receipt transactions. This indicates that to maintain integrity the data model requires an overall view not only of the operations of the individual transactions, but also of the operations for transaction cycles. Gal and McCarthy [1985c] have argued that, by using the concepts of generalization and aggregation [Smith and Smith, 1977a & b] prototypes of transactions can be represented as part of the REA.data model in ’terms of episodes that a business goes through. They also demonstrated that these episodes could be modeled with the Scripts concept developed by Schank and Abelson [1977], and then connected into a prototypical representation of traditional accounting cycles. The methodology that is used to represent these transaction prototypes comes from the TAXIS project [Borgida, Mylopoulos, and Wong, 1984]. Therefore, this conceptual representation of the execution of transactions represents the constraints that will 85 determine which occurrences of the data are possible. Fernandez, Summers, and Wood [1981] indicate that security over database objects should exist at all levels (conceptual, logical and physical) but that: fundamental access rules should apply at the conceptual level. The reason is that the conceptual level provides a global view of the data, where their semantics are explicit [p. 87]. These Event-Scripts provide for this security by dealing with constraints at the conceptual level. In general, all transactions will not be executed as prescribed. The constraints may be violated due to the, necessities of the business environment. As an example, it might be required for integrity purposes that the extension (price times quantity) of each inventory item on a sale sum to the total amount of the sale. However, if there is an error in an inventory item, a violation of integrity might exist in the system for some period of time. Svanks [1982] demonstrates a technique to analyze the degree to which integrity violations exist in a particular database, and identifies this as "auditing with the database". The purpose of this proposed research is to understand the way in which certain pieces of information (some of-which might come from an integrity analysis based on violations of certain predetermined constraints) are used in the evaluation of the integrity of the database. The next section deals with the concept of views (or database 86 subsets) and in particular those views that are of interest to auditors as they analyze data integrity. Accounting yiepe pf rpe,Darebese Martin [1975] identifies two components of a functioning database: (1) the production database and (2) the information system. The production database contains the overall stored data and is updated with transactions to reflect the current environment. The users of the system generally do not require all of the stored at any one time: instead, they require subsets or views of this data. It is the information system that provides these data subsets. In a typical business environment, many of these views would be constructed based on the policy of least privilege [Fernandez et al., 1981] and separation of duties [Gal and McCarthy, 1985a]. Some of these views can also be used to make inferences about the information that is not (examined. Rowe [1983] demonstrates a procedure, which uses inference rules to develop a database abstract or ‘semantic' sample that could be used for statistical analysis of the database. Auditors also are not interested in examining all of the data in a system. Through rules of good auditing judgment, they determine what items to examine in reaching a conclusion about the integrity of the data, i.e. they construct an auditing view of the system and then evaluate the contents of this subset. To construct these views in a 87 functioning database certain software could be included in the system to obtain necessary information. Embedded audit routines can collect data from various points in a functioning data processing system. The SRI study [1977] noted that this technique gives the auditor a "window"_ with which to view the operation of the system. Auditors can select any view or window that would meet their specific requirements for information, but the data that is to be reviewed must be part of the system. The idea of embedding certain audit assistance features into application syStems is beginning to receive some attention in both the practitioner and academic communities. In a recent book, Weber [1982] discusses what he calls concurrent auditing techniques. These concurrent auditing techniques are designed to deal with the evidence collection needs of complex systems whose requirements cannot be satisfied with ex post collection and evaluation [Weber, 1982, p. 474]. Weber mentions a number of reasons for the inadequacy of these "after the fact" techniques in continuously monitoring advance systems, and discusses some techniques for obtaining the information from the system. One particular technique that Weber examines is the System control Audit Review File (SCARF). This technnique involves collection of evidence using various software modules which are embedded in the application system at the appropriate points. There are a number of different types 88 of information that can be collected by these SCARF routines. Weber [1982, p. 483] enumerates the following categories: 1. Application System Errors SCARF routines that provide an independent check on the existence of design or programming errors that either were part of the initial application system or that were introduced through modification or maintenance. 2.Policy and Procedurel yerianeee SCARF routines that collect data on the variations from the administrative and technical policies and standards. 3. System Exeeprieps SCARF routines can be used to monitor different types of application system exceptions that may be allowed if they are within specified tolerance. 4. Statistical Semples The use of SCARF routines to obtain statistical information from various points in the application system. 5. Spepehepe egg Extended Reeerds These types of records can be written to the SCARF file. 6. Performance Measgrement Here The use of embedded SCARF routines to collect information that may be useful for measuring or improving the performance of the system. The nature of the information that can be collected by the embedded SCARF routines would seem to be valuable to auditors, as they evaluate the degree to which certain internal controls are working within the system. For instance, evidence collected concerning possible policy or procedural exceptions can give information on the degree to 89 which "transactionns are executed in accordance with management's general or specific authorization [AICPA, 1979, AU 320.28(a)]." In addition, by examining certain types of transactions (such as error correction transactions) additional insight can be gained concerning the ability of the data processing system to detect and correct errors in a~ timely manner. Furthermore, by embedding internal control evaluation evidence-gathering procedures into the SCARF routines, the examination of transactions exectued throughout the period under audit (as suggested by the ACIPA [1979, AU 320.61 a 63]) can be accomplished. Weber argues that there are essentially two design issues that must be considered when using these embedded routines. The first concerns the exact subset of information that will be collected. By examining the declarative knowledge that is used by the auditor in reaching a conclusion about internal controls, the portion of the database that is required could be identified. Further, the "Event-Scripts" could also be used to identify the subset of the database that does not conform to predetermined constraints. A second design issue that Weber identifies concerns the reporting method and frequency that would be used. The SRI [1977] study notes that: The comprehensiveness of transaction review, which is the chief advantage of this technique, can also cause the production of voluminous data [p. 135]. 90 By embedding the evaluation knowledge as part of the audit view, the problem identified by this study could be reduced. This idea of coupling expert knowledge with a production database has been successful in a number of projects [Jarke and Vassiliou, 1983: Walker, 1983]. One problem that Jarke and Vassiliou identified was the. inefficiency of dealing with sequential queries on the database by the expert system. The approach that they were using to reduce the time required for the expert system to get the necessary information was to group these requests and therefore get subsets of the database. This method is similar to the approach used in the embedded audit tecchniques, except that it takes chunks of data, some of which may not be neccessary, and makes it available to the expert system. The information used by the expert system in this study could be used in the development of the Event- Scripts and therefore construct views of the database that contain little extraneous information. This is similar to the approach used by Rowe [1983] in obtaining an abstract of the database from which statistical inferences could be made without examining the entire database. umm This section has examined models of declarative knowledge. The information that is part of a data model attaches certain meanings to the data it represents. These data models have been examined in the context of studies 91 that have dealt with the way in which accounting information can be produced in a database environment. It has been argued that in order to represent acccounting information, it is necessary to include certain constraint specifications as part of the data model. In the REA model, constraints that deal with occurences of the database after the. execution of transaction cycles can be represented in the form of prototype episodes or SCRIPTS. Violations of these SCRIPTS in a database system can be used to construct a view of the database that could be examined as part of an evaluation of internal controls. Therefore, the transaction SCRIPTS could answer certain design issues that have been raised concerning the use of embedded audit routines. The next section will deal with the method that was used to construct the expert system and evaluate the results. CHAPTER III RESEARCH METHOD This chapter describes how INTERNAL-CONTROL-ANALYZER, A an expert system for evaluating the internal accounting controls in the revenue cycle, was developed. This chapter is divided into four parts. The first section discusses the tool, EMYCIN, that was used to help build INTERNAL-CONTROL- ANALYZER and explains the reasons for its selection. The second section describes the construction of the initial prototype version of the INTERNAL-CONTROL-ANALYZER. Section three. discusses the refinement of this prototype system. Finally, section four discusses the verification of INTERNAL-CONTROL-ANALYZER. Research Teel: EMYClN An expert system can be viewed as a complex interaction of a number of different components. These systems are generally considered to consist of a knowledge base (which contains the facts and rules that pertain to a particular problem domain), a working memory (which keeps track of information about the status of the particular problem the system is currently examining), and an inference engine (which controls the actual behavior of the system by 92 93 determining the order in which the rules will be used). This modular type of system architecture has allowede tools to be developed which assist in the construction of expert systems. Expert systems have been developed which make distinctions between the knowledge base, the working memory, and the inference engine components, and therefore make it possible to remove the knowledge base from a system and use the same procedures for the working memory and inference engine on a different knowledge base. These general systems provided the software needed to maintain and manipulate any knowledge base which was added to the system. This meant that by using one of these tools to build the expert system it was possible to develop a theory of an auditor's internal control evaluation process at what Newell [1982] identifies as the knowledge level. That is, the theory being developed in this study does not attempt to argue for a particular physical organization of the expert's mind being modeled, only that the rules and facts that will be part of the system constitute the knowledge that is needed to evaluate controls. Thus, by using these tools to develop this expert (or knowledge based) system it is not necessary to deal with the computer science issues of how to manage and use the knowledge base only with the structure of the expertise necessary to solve internal control evaluation problems. This study makes use of a software tool, EMYCIN [Van 94 Melle et al., 1981] in the construction of the system. Figure 3 illustrates the relationship between EMYCIN and the other components of INTERNAL-CONTROL-ANALYZER. EMYCIN performs three functions on the knowledge base: (1) monitoring its structure, particularly during its refinement, (2) manipulating it to solve problems during a. consultation, and (3) serving as an interface with the users of the system. The knowledge base consists of the produc- tion rules which represent the expertise needed to evaluate internal accounting controls. The purpose of this research was to build this knowledge base and then to analyze its structure. INTERNAL-CONTROL-ANALYZER EMYCIN working memory - bookkeeping functions USER: s: inference engine - rule application interface - dialogue with user KNOWLEDGE BASE L____auditcr1s_exnsrtise Components of INTERNAL-CONTROL-ANALYZER FIGURE 3 95 The next section discusses the reasons that EMYCIN was selected as the software tool to assist in the construction of INTERNAL-CONTROL-ANALYZER. Following this discussion is a more detailed examination of some of the important features of EMYCIN. Reasons for Choosing EMYCIN There were two reasons that EMYCIN was considered an appropriate tool to be used in the construction of the system. The first reason concerns the fact that EMYCIN has been used successfully as tool in the development of expert systems in a number of domains (including accounting). The second reason for its selection concerns the similarity of the task for which EMYCIN was originally created (medical diagnosis) and internal control judgments. EMYCIN ip preyious reeeereh. EMYCIN originally was developed as part of a project to construct an expert system, MYCIN [Shortliffe, 1976], to perform medical diagnosis. EMYCIN is the working memory and inference engine portion of MYCIN with the medical knowledge base removed. This same tool was then used in the construction of expert systems in a variety of domains, including: PUFF - to analyze pulmonary function data, SACON - to analyze structural data, and DART - to advise on computer tele- communication problems. EMYCIN has also been used in a previous study in accounting to develop an expert system 96 that gives advice in the area of estate tax planning [Michaelsen, 1982]. This evidence suggests that EMYCIN is a versatile software tool that can be applied to a wide range of problems areas. In addition the evidence also suggests that rule-based systems can capture expertise in a number of different areas. Finally, and perhaps more pragmatically, I this wide range of use provides assurance that many of the problems inherent in the use of any piece of complex software have already been dealt with. Similarity pf medical diagpesie egg iprernal eentrel jpdgments. A second reason for selecting EMYCIN as the tool to develOp the expert system in this study has to do with its ability to provide a number of features which are important components of internal control judgments. The fact that the system has these features is due to similarities between the task for which EMYCIN was originally designed and the judgment examined in this study. EMYCIN stands for Essential MYCIN and, as previously mentioned, is the core of a diagnostic system with the medical knowledge base removed. A very general examination of the diagnosis process reveals a number of similarities between medical diagnosis and auditing judgments involved in the evaluation of internal controls. Stefik et a1. [1983] describes diagnosis 68: 97 the process of fault-finding in a system (or determination of a disease state in a living system), based on interpretation of potentially noisy data. Requirements include those of interpretation. A diagnostician must understand the system organization (its anatomy) and the relations and interactions between subsystems. Key problem areas follow: (1) Faults can sometimes be masked by the symptoms of other faults; ... (2) Faults can also be intermittent; ... (3) Diagnostic equipment can itself fail; a diagnostician has to do the best possible with faulty sensors. (4) Some data about the system can be inaccessible, ... (5) Because the anatomy of natural systems such as the human body is not fully understood, a diagnostician may need to combine several partial methods [p. 83]. The similarities between this view of diagnosis and internal control evaluation exist at many levels. The point that diagnosis is a process of faultfinding corresponds almost directly to the control evaluation process. Internal control risk concerns the risk that the firm's information system which captures and records data will produce errors, ie., the risk that there will be faults in the system that processes the firm's data. In order to make an evaluation of the possibility that the system will produce errors, an understanding of the way that the system functions (including the interaction of various processing subsystems) is required. This need to understand the workings and relationships within a firm's data processing system is the basis for rather a thorough documentation of the client's 98 system as part of the audit process (see for example Peat, Marwick, Mitchell & Co., 1980). In addition to these major areas of similarity some of the problems faced by auditors in the faultfinding process correspond to points raised by Stefik et al. For instance, auditors are faced with the problem of determining whether a_ particular control problem will produce a material error, ie., the error might not occur with enough frequency or to enough large transactions to warrant a complete investigation of an area. In addition, the work examined in Chapter I suggests that the data which is available to evaluate accounting controls is generally not conclusive: the auditors must use data which is potentially noisy to make their judgments. In addition to this similarity between the task characteristics there is also a similarity between the state of the knowledge in the domains. Shortliffe [1976, p.32] observes that medical diagnosis must deal with knowledge about diseases that "are not sufficiently well understood to be characterized by well-defined mathematical formulae." The literature in Chapter I revealed this same view of knowledge about controls. Although there were some attempts to develop mathematical representations of internal controls, they were not successful because they did not represent the control systems with sufficient detail and explanatory power. 99 The evidence from the review of the internal control literature in Chapter I suggests that auditors must be able to use various types of data, quantitative as well as qualitative. In addition auditors must combine data which is incomplete or inconclusive in the formulation of an internal control judgment. Finally, because of the nature. of the data and the task domain, auditors must make assumptions and must be able to provide explanations for the assumptions and other judgments that they make. The connection between internal control evaluation and other diagnostic-type tasks indicated that certain features which were considered important to the successful development of the MYCIN system and therefore were part EMYCIN would also be important for internal control judgments. The next section examines some aspects EMYCIN that supported these diagnostic-type activities and were important in the construction of INTERNAL-CONTROL-ANALYZER. EMYCIN Features This section examines certain features of EMYCIN which were considered important in the development of the expert system which is part of this research. The first aspect of EMYCIN which will be examined is the inferencing procedure. This is the part of the system that determines the order in which the rules will be applied to the particular problem. Following this will be a discussion of the various methods that are used to represent the knowledge within the system. 100 Finally, there will be an examination of the way in which the system deals with uncertainty in the knowledge. inference strategy. EMYCIN uses a goal—driven inference procedure. This means that the system begins its problem solving with a major goal, such as reaching a. conclusion about the level of internal controls. The inference engine (the part of the EMYCIN software that controls the inferencing procedure) then examines the knowledge base for productions that make a conclusion about this major goal. It then looks at the premise or situation portion of these productions to see what information is needed to make conclusions about this major goal. The acquisition of this information now becomes a sub-goal which must be met in order to acheive the major goal. In order to satisfy this sub-goal, the inference engine then examines the knowledge base for productions that would allow the system to make conclusions about this set of information. The acquisition of the information in the situation portion of these productions becomes an additional sub-goal for the system. The system continues to follow this chain of productions until either it can no longer find a production that would allow it to make a conclusion about a sub-goal, or it already knows the value for the premises of the production. If the system has the piece of information that is required, then it satisfies the particular sub-goal and 101 proceeds to other sub-goals that must be met to reach the final goal. If the piece of information is not available at this point, the dialogue component of the system asks the person running the session to supply that piece of information. In the INTERNAL-CONTROL-ANALYZER system the data that is requested from the user corresponds to the» information that is collected on the worksheets. (This point will be examined more fully in the discussion of the development of the prototype.) Because the chain of productions is followed backward from the major goal to the data that is used to make the conclusion, this goal-driven inferencing procedure is also called backward chaining. This backward chaining (or goal-driven) approach ensures that only data that is needed for the particular conclusions (or sub-goals) of the system is obtained. This differs from the forward-chaining approach used by other systems. Nilsson [1979], in contrasting these two approaches, identifies backward reasoning as appropriate in situations where there are a few major goals and many pieces of data which could be examined. In the evaluation of accounting controls there are potentially many pieces of data that could be used in the decision. The evaluation frameworks examined in Chapter I attempt to assist the auditor in restricting the examination of data to that portion that will have an impact on judgments in a few major areas. The goal of internal control evaluation is to 102 reach conclusions about these areas. Nilsson also calls the backward approach a form of planning in that a chain of actions is constructed that will restrict the pieces of data examined to those that will affect the particular goal. In the development of an audit program auditors plan specific actions that they feel will most efficiently meet the objectives of the audit. Therefore, the goal-driven or backward chaining approach (by restricting the examination of data to items which are important to the major goals) corresponds rather well with the method actually used by auditors. The use of a goal-driven strategy to solve the problem also allows the system to construct a line of reasoning. By keeping track of the various sub-goals that must be satisfied in order reach a conclusion about the major goal, the system has a record of the process it is using to solve the problem. Aiello [1983] compared a number of edifferent search or inferencing strategies and noted that the backward chaining strategy did provide an explanation or line of reasoning that was easier to follow. Struetpre er the tpoyledge peee. EMYCIN maintains and uses two types of knowledge: declarative and procedural. Declarative knowledge pertains to certain parameters that are used within the system. Some of these parameters may have actual values attached to them during a particular session. Examples of this type of knowledge might be the 103 name of the company or the nature of the data processing facility. The definition of a parameter consists of describing certain attributes. In general the attributes that must be specified for each parameter are the entity or context that it relates to, the values it may have, the way in which it will be interpreted, and the way in which' questions about it should be phrased. For instance, the parameter ‘NAME' might have the following attributes: CONTEXT: CLIENT VALUE: ANY TRANSLATION: THE NAME OF THE CLIENT PROMPT: WHAT IS THE NAME OF THE CLIENT? Declarative knowledge is represented in the form of context, parameter, and value triples. As an example, the following representation might be present once the name of the client has been obtained: "CLIENT:NAME:XYZ CORP.". Procedural knowledge is the knowledge obtained from the expert concerning the way in which the problem should be solved. This knowledge is represented as productions or rules of the form: SITUATION => ACTION. The situation portion identifies a particular occurrence or set of occurrences which will be tested if the action portion is required for a current sub-goal. If the production is selected and the parameters in the situation portion have the appropriate values then the action portion is executed and the conclusion is added to the knowledge base. For instance, the following production might be used in a system 104 to evaluate controls: IF 1) the separation of duties is adequate, THEN conclude that there are not any problems with incompatible functions. In this case the parameter ‘separation of duties' for the context ‘client' will be examined for the value ‘adequate'.‘ If it has this value then the system will conclude that the parameter ‘incompatible functions' has the value ‘no'. Contexts are used to break the knowledge base into logical associations, thus connecting rules and parameters into categories. In the above example the production and the parameters all are related to the context ‘client'. This context would be connected with all the rules and parameters that make inferences about certain features of the client. In this way the contexts serve as higher level knowledge structures. Reasening ppger pncertainty. In the discussion of the general features of diagnostic type tasks it was indicated that some of the information that is used in the judgment process is inherently noisy or uncertain. EMYCIN allows for this uncertainty by attaching to each piece of knowledge (either rule or parameter) a degree of certainty. This certainty factor (CF) represents how well the particular knowledge has been established. These CFs are used by EMYCIN when it accesses a piece of information during its reasoning process. The CF for a 105 rule or piece of information can be interpreted as the subjective degree of belief in that assertion. Shortliffe and Buchanan [1975] define CFs as the difference between the degree of belief in some rule or data, given the current evidence, and the disbelief of the information given the same evidence. Thus CFs are a combination of both the supporting as well as disconfirming evidence. CFs can range in value from -l.0, which represents complete certainty that the assertion is not true, to +1.0, which represents complete certainty that an assertion is true. These CFs will be attached to both the parameter values as well as the action portion of the productions in the system. For a parameter like ‘NAME' that has been determined to be ‘XYZ CORP.‘ the system will attach a CF of +1.0 which indicates that this fact is known with certainty. A CF of .8, for instance, which is attached to the action portion of a rule indicates that whenever the premise of the rule is determined to be true, the inference or conclusion can be made with a high (but not complete) degree of certainty. As mentioned previously, rules are fired by EMYCIN in order to make conclusions about certain goals (or subgoals). The selection of rules is governed by their ability to make inferences about the particular case. CFs enter into the inferencing process by determining whether the premises can be considered to be met. In order for a rule to be fired 106 and the inference made, the information in the premise must be established at a minimum level of certainty. This minimum level used by EMYCIN is .2. The calculation of this level of certainty for the premises is determined by the relationship between the clauses that make up the situation portion of the rule. For example, INTERNAL-CONTROL-ANALYZER~ uses the following rules: RulelOl IF: 1) There is a population control to count or identify customer checks, 2) There is a procedure to review the completeness of customer checks, and 3) The person that performs the procedure that counts or identifies customer checks is not the person that review the completeness of customer checks THEN: It is definite (1.0) that the boundary controls over the completeness of customer checks is effective Rule73 IF: 1) The functions involved in the sales transactions at the boundary are: adequately separated, or 2) The functions involved in the sales transactions at the boundary are adequately supervised THEN: It is definite (1.0) that the responsibilities for the boundary functions involved with sales transactions are adequately separated or supervised ‘ RulelOl is a conjunctive rule. This means that all three of the clauses in the premise must be satisfied for the conclusion to be made. Therefore the strength of the 107 inference made in this rule will be a function of the degree of belief in the weakest data that is to be used. In conjunctive rules the CF attached to the action or conclusion is equal to the minimum CF of the clauses in the premise times the CF for the conclusion. For instance, if the CFs for the clauses of RulelOl were .3, .4, and .5' respectively, then the strength of the inference would be the minimum (.3) times the CF for the rule (1.0) or .3. In conjunctive rules the certainty of the weakest piece of evidence determines the strength of the inference that is possible. The second rule is disjunctive. In disjunctive rules only one premise needs to be established in order to make the specified conclusion. In these cases the certainity of inference will be a function of the premise that has been established with the highest degree of certainty. Therefore, the strength of the conclusion for Rule73 will be equal to the maximum CF for the clauses times the CF for the rule. For instance, if the clauses had been established with .5 and .6 degree of certainty, the conclusion would be established with .6 times 1.0, or .6 degree of certainty. In disjunctive rules the piece of information that has been established with the greatest certainty determines the strength of the inference that can be made. The inferencing procedure of EMYCIN attempts to gather evidence for each goal or subgoal until it has been 108 established with absolute certainty (a CF of +1.0 or -1.0). This means that the system will apply all of the rules that establish a certain parameter, unless this certainty value is reached. In order to ensure that additional information will not result in an assertion that exceeds absolute certainty, the following method is used to update the CF. If a piece of information has been established with a .8 degree of certainty, then additional information can only make assertions on the amount that is uncertain. Thus, if a second rule results in an- assertion on the piece of information with a degree of certainty of .6, this is interpreted as additional certainty on only the uncertain portion. Mathematically this new degree of certainty is obtained using the following formula: .8 (previous degree of certainty) plus .6 (additional certainty) * .2 (the remaining uncertainty) = .92. The use of CFs and the way in which new information is combined with previous information allows knowledge based systems which use EMYCIN to handle missing or incomplete information in making inferences. In addition these systems will look at all possible ways of establishing conclusions about the various goals that must be met in order to make an assertion about the major goals of the system. If information which would allow the system to make conclusions using one approach is unavailable, these systems can use alternative methods which will accommodate the data that is 109 available. Summary. The process of studying a decision that is made by a person requires the use of different tools depending on the nature of the investigation. These tools allow the researcher to focus on the analysis of the~ decision as opposed to the development of tools. In the same way that computerized statistical packages have enhanced the ability of researchers to use certain methodologies to gain an understanding of decisions, tools like EMYCIN have enhanced the ability to use the expert system methodology. The selection of this particular tool to build the system was based both on EMYCIN's success in constructing expert systems in other areas and on the similarity of the task characteristics of medical diagnosis and internal control evaluation. The next section in this chapter describes the development of the initial prototype version of INTERNAL-CONTROL-ANALYZER using this tool. Censtruction er the Prototype System In order to construct a system such as the INTERNAL- CONTROL-ANALYZER a number of decisions must be made. These decisions concern the specific domain which will be examined by the system and the source of the knowledge base. These decisions are related in that a specification of the problem domain will determine the source of the knowledge that is 110 used to solve these problems. The acquisition of knowledge for a particular domain is generally an interative process. This "knowledge acquisition" [Buchanan et al., 1983] becomes a process of both addition and refinement of the system's knowledge base. The knowledge acquisition portion of EMYCIN, .Teiresias [Davis and Lenat, 1980], permits most of this process to be done interactively with an expert. The expert uses the current version of the system to perform a selected task, identifies the areas in which the performance of the system is inadequate, and then suggests additions or corrections to the knowledge base that will alleviate the problem. In order for this interactive process to be carried out there must be an initial knowledge base to refine. This section discusses the development of this initial knowledge base by examining the two areas mentioned previously: specification of the problem area, and identification of the pertinent knowledge for the prototype. Speeifiication pf the Problem Aree The specification of an exact problem that the final system will deal with is critical to obtaining a suitable level of performance. If the problem is too broad the construction of the system would not be a manageable project; the MYCIN system represents the work of approximately 10 different people over a period of fifteen years. On the other hand a problem definition that is too narrow might result in an task which has characteristics 111 that are too artificial and the knowledge obtained from the expert would not really be from an area in which they had expertise. In Chapter I, previous work was examined which had studied internal controls and this same problem of task definition was approached by both accounting firms that‘ needed to offer guidance to their auditors and by researchers that were investigating internal control judgments. In each case the overall task of control evaluation was divided into more manageable components by examining a firm's controls within particular cycles. In this study the same approach was used: examination of controls in a particular cycle, the revenue cycle. There was, however, one additional restriction that was placed on the decision that was studied. The purpose behind examining internal control evaluations in this study concerns the ability to use data that is part of a firm's physical database to assess its reliability. If a database is structured using the REA model [McCarthy, 1982] then it will contain a record of events that have changed the financial position of the firm. These economic events should be carried out in accordance with management's authorization. The effectiveness of controls will then determine whether or not transactions are executed in accordance with this authorization. Therefore, examination of the actual physical database should be able 112 to provide a view of the data that will at least be useful in determining the reliability of the data. By restricting the data that was used by the INTERNAL—CONTROL-ANALYZER it those items that could be part of the physical database it would be possible to assess the role that transactions data plays in the analysis of controls. In addition it would' also be possible to determine the value of data outside the database to the internal control evaluation. In summary, this study examined the evaluation of controls in the revenue cycle and restricted the data in the INTERNAL-CONTROL—ANALYZER to that portion that would be found in the firm's physical database. The next section will examine the source of the information that was used to construct the prototype of the INTERNAL-CONTROL-ANALYZER. ldentificetion pf the In't a Kno ed Base The identification of the knowledge to be used in the construction of the prototype for the INTERNAL-CONTROL- ANALYZER consisted of two steps. First, it was necessary to identify a public accounting firm that would be suitable for this study both in terms of the structure of their internal control decisions and their ability to provide subject(s) for the study. The second step consisted of analyzing the firm's information to identify that portion which would be appropriate to include in the prototype version of the system. 113 Selection ef_tpe firm. Eight public accounting firms with offices in Detroit were contacted initially to request information concerning their approach to the evaluation of internal controls. All of the firms provided information and expressed some interest in the nature of the project. The information that was provided was quite diverse ranginge from a few pages out of an audit manual to a few books on the subject of control evaluation. This information confirmed the evidence from the academic literature which indicated that there were considerable differences in both the approach and terminology used in the evaluation of accounting controls. At this point it was necessary to decide whether to build a generic system that would include the most appropriate information from each firm, or to build a firm specific model. It was decided for a number of reasons that it would be more beneficial for this study to examine one firm. One reason for selecting a single firm concerned a point made by Einhorn [1974] that differences in background and training, in particular when they represent alternative ‘schools of thought', will probably lead to lack of consensus among experts from these different schools. The information from the firms indicated that they were definitely different schools of thought on this subject and therefore a system which tried to combine all of these 114 points of view would probably not be possible. A second reason, which was related to the different approaches, concerned the very different terminology that was used by the firms to express similar ideas. This is a problem because the purpose of the study was to examine the way in which experienced auditor(s) make internal control. judgments, not to train them in a particular approach. Therefore, the prototype must approach the problem in the same manner as the auditor that was being studied; which includes the use of similar terms. Buchanan et a1. [1983] argue that one of the most difficult tasks in knowledge acquisition is to "identify and formalize domain concepts [p.129]", and it was felt that the use of a single firm would alleviate this problem by using the concepts that were already familar to the auditor. Thus the construction of INTERNAL-CONTROL-ANALYZER would only require the auditor to add knowledge to the system about the way in which they would gather and evaluate evidence differently from the prototype as opposed to defining terms that the system should use. These two reasons determined that it would be more appropriate for this study to examine a single firm. The next step was to decide on the firm whose method of evaluating controls would best fit into the study. This decision was based on two criteria: (1) structure of the internal control evaluation and (2) distinction between 115 database and non-database information in the evaluation. The degree of structure in the decisions of different firms had been noted previously by Cushing and Loebbecke [1983] in their review of the audit manuals of twelve accounting firms, and this was also evident in the material provided on the internal control decisions. For this study it was rimportant not to select a firm which had too much structure because the construction of the system could turn into an exercise in computerizing their written material, with little input coming from the expert. On the other hand the selection of a firm that considered the evaluation of controls to be very unstructured could cause problems in the construction of the prototype and its subsequent refinement. The firm that was selected had some structure in terms of the data that was initially considered and offered some general guidance for the evaluation judgment in their training manuals but still considered experience important to the refinement of the control judgment. I A second criterion that contributed to the selection of the firm used in the study concerned the data that was considered important in the evaluation. Because this study concentrated on the evaluation that would result from an examination of data that would be part of the physical database, it was necessary to select a firm that made some distinction between this and other types of data. The firm that was selected did not specifically identify these 116 different types of data, however they did use different worksheets to collect and evaluate ‘environmental' factors that might have an effect on controls and data which was specific to particular cycles. In this firm it was possible to restrict the system's evaluation to the data that would be part of cycle worksheet, therefore the expert could be. asked to form an evaluation based on a logical subset of the data. Initial 3511212121332 agguisitien. Once the firm had been selected the next step was to decide on the information that would be used to construct the prototype version of the system. Buchanan et a1. [1983] discuss a number of approaches that can be used to obtain this knowledge. One such approach is to incorporate the knowledge that is contained in textbooks from the area. This approach was used successfully in the construction of systems which performed medical diagnosis [Van Melle et al., 1981] and planned for individual estate taxes [Michaelsen, 1982]. This latter system contained a great deal of rule-based knowledge which was taken directly from the tax code. The firm that was selected provided a large amount of information that could be added to the initial prototype version. Most of this information was extremely general and discussed terminology rather than any specific guidance for the evaluation process. However, there was one source of information that did offer guidance at a very elementary 117 level. This information was contained in the material that was provided to every entering auditor as part of a course on the documentation and evaluation of controls. The course material gave examples of the way that the various worksheets should be completed and then possible evaluations of this material. It was decided that only this course material should be included in the prototype. The selection of this material as the sole source of information for the prototype allowed for a different type of evaluation of the final system. Generally the initial knowledge acquisition is done solely to provide the expert with a system to interact with to bring the system to a desired level of performance. However, in certain studies [Young and O'Shea, 1981] there is an examination of the incremental knowledge that must be added to correct specific deficiencies in the system. That is, not only is the final level of performance important but also the types of incremental knowledge that must be added to transform the system from one which exhibits only a novice level of performance to one that might be considered an expert in an area. By providing the initial prototype with only the knowledge contained in the course material from the firm it was possible to analyze the incremental knowledge in terms of the effect 'of experience on the evaluation process. Because each new member of the firm went through this course very early in training, the 118 differences between the prototype and the refined version could be analyzed in terms of the learning that takes place from participation in actual audits. The system that resulted from the structuring of this information in the form of production rules was able to reach the same conclusions about accounting controls as the‘ course material cases. At this point it was possible to begin the iterative process of refining the system using actual audit workpapers. The next section describes the refinement process which consisted of interactions between the expert and the researcher to fill in rules which reflected the professional judgment that had been acquired through experience in making control evaluations. Refinement er the System The use of the expert system methodology to identify and construct a knowledge base in a particular domain is a long process which is never really finished [Davis and Lenat, 1980]. Davis and Lenat suggest that this is due to the approached that is used to construct these systems, "... the approach to competence [is] incremental (and occasionally asymptotic)," and to the domains being studied, "... those which are still under active development [p. 471]." Because this refinement process is never really done it means that for a particular project there must be some milestones that can be identified. Gaschnig et a1. [1983, 119 p. 258] have identified nine stages in the life cycle of knowledge based systems to assist in their development and evaluation: 1. The initial definition of the goals of the project. 2. Construction of the prototype system. 3. Refinement of the prototype by means of interaction with prospective users. 4. Structured evaluation of system performance. 5. Structured evaluation of user acceptance. 6. Extended service and maintenance. 7. Follow—up studies of usefulness. 8. Modifications and revisions to system. 9. General release and marketing. The purpose of the INTERNAL-CONTROL-ANALYZER was to build a descriptive model of an auditor's internal control evaluation process and to examine the importance that physical database items have in this process. Therefore the project was only designed to progress to the third stage of the lifecycle. The remainder of this section will discuss the selection of the auditor that refined the system and the refinement process. Selection ef the Auditor Once the firm had been selected the next step was to obtain individuals within the firm that were capable of participating in this study. The refinement process for other expert systems has sometimes required years. Although it was anticipated that for the INTERNAL-CONTROL-ANALYZER this refinement would take approximately six months, this is still a major commitment for a practicing auditor. 120 Therefore, there was an initial meeting with a partner of the firm to discuss the project and the amount of time that would probably be needed. Based on this meeting three senior managers within the firm were identified by the partner as available to participate in the project. Initially it was hoped that a partner would participate in] the project, but based on the interviews it was decided that it would be more appropriate to use managers as subjects because they make control evaluations as part of their responsibilities, whereas partners do not. There was a preliminary meeting with the three managers in which the project was discussed in some detail. During this meeting there was also some indepth discussion of their individual approaches to the evaluation of controls. Based on this interview and some subsequent realignments within the firm it was decided to use only one of the managers in the project. In addition. to some practical issues such as identifying mutually agreeable meeting times there were some other issues which determined that it would be more appropriate to use a single manager to refine the system. EMYCIN is not able to handle contradictory rules in its knowledge base. Therefore, if more than one manager was used to refine the system and they were not able to agree on the action to be taken in a particular situation, it would be necessary for the researcher to select among the possible 121 rules. This would mean that the performance of the system would be determined in part by the reseracher selecting which knowledge was more appropriate to be included. A second consideration involved the analysis of the final system. If the final knowledge base was not attributable to a single manager then it would not be possible to analyze. the system in terms of what an certain individual auditor has learned through their experience in the firm. For these reasons it was decided that only one manager would be used for this particular study. Once an individual manager had been identified the refinement consisted of using selected audits in which this person had made the evaluation of the controls and allowing the system to use this data in its evaluation. This next section describes how EMYCIN was used to refine the system. The Refinement Precese The refinement process required approximately seven months and consisted of eight meetings with the manager to examine the results of different changes that were made to the system. The first few sessions consisted of refinements to the information acquisition portion of the system. Once this aspect of the system had been refined the next step was to allow the auditor to examine the system as it attempted to make actual evaluations. The last few sessions involved the system evaluating an actual set of workpapers for the revenue cycle. In the next chapter the specifics of the 122 actual changes that were made to the system to improve its information acquisition and evaluation abilities will be examined. The next section will discuss the question- answering and explanation capabilities of EMYCIN which allowed the auditor and the reseracher to examine the way in which the system was making its evaluations for a particular] case. Explanation Qfithé reaeening progess. EMYCIN has the ability to respond to questions both during and after the evaluation of a particular case. During the examination of the information that EMYCIN already has obtained there will periodically be a requirement for additional information about the case; at this point the system asks for this data. When the system is requesting information, it is permissable to ask the system, "WHY". This is interpreted as a request for the reason that the information is needed. The system responds with an explanation of exactly how this information is to fit in with its current reasoning process. If the person using the system wants to continue the explanation of the reasoning process it is possible to type in "WHY" again and pursue the reasoning further. An example of a session which includes these questions can be found in Appendix I. In addition to the "WHY" questions the system is also able to answer "HOW". This type of question is generally used after a "WHY" question and must refer to a particular conclusion that the system has made. The system interprets 123 these "HOW" questions as a request for the particular rule that allowed it to make a conclusion. It is possible to ask another "HOW" question to get the system to respond with the English translation of the rule. Shmmary This refinement process continued until the system was able to handle the selected cases appropriately. The refinement of INTERNAL-CONTROL-ANALYZER was a long process which required a great deal of effort to track down and correct pieces of knowledge which were affecting the overall performance of the system. The selection EMYCIN's as a tool to build the system greatly enhanced the ability to develop the large knowledge base that was required to make evaluations in this domain. The next section discusses the verification of the system at the end of the refinement process. yerificetien pf INTERNAL-CONTRQL-ANALYZER The refinement of INTERNAL-CONTROL-ANALYZER was considered done when the system was handling the selected cases appropriately. For this particular study ‘appropriately' consisted of acquiring only that information which was considered important for the particular case and reaching conclusions about the important goals (and sub- goals) with the information that was available that were within a range of what the auditor considered important. 124 ACCOUNTING CONTROLS REVENUE CYCLE r A ENVIRONMENTAL FACTORS corporate personnel safeguards ‘L (THESE CONTROLS PERTAIN TO SALES AND CASH RECEIPTS) POPULATION SEPARATION ACCURACY CONTROLS OF DUTIES CONTROLS COMPLETENESS AUTHORIZATION COMPARISON MATHEMATICAL CONTROLS CONTROLS - CONTROLS CONTROLS (THESE CONTROLS PERTAIN TO EACH DOCUMENT IN THE REVENUE CYCLE) GENERAL OVERVIEW OF THE EVALUATION OF ACCOUNTING CONTROLS FIGURE 4 125 The next section examines the verification of the system's ability to acquire information and use it to meet the goals of internal control evaluation. yerification Q: anl§_agg §ub-§oals Figure 4 shows the relationship between the various. sub-goals and the major goal of the system which was to make a conclusion about the accounting controls in the revenue cycle. The lowest sub-goals, the evaluation of complete- ness, authorization, comparison, and mathematical accuracy controls, affect the sub-goals at the next higher level. For instance, the decisions about the 'completeness and authorization sub-goals affect the sub-goal of determining the population controls which in turn is a factor in evaluating the overall accounting controls. The environmental factors affect decisions at each level of the hierarchy, but they have the most effect on the higher level decisions. For this reason the auditor had a higher level of agreement with the lower level decisions made by the system. Gaschnig et al. [1983] have argued for an evaluation of knowledged based systems depending on the particular stage in the life cycle. If the system has been developed for commerical release the evaluation should be based on its ability to provide accurate answers to the problems ‘it is presented with. However, in the earlier stages of 126 development the emphasis should be placed on the line of reasoning, '... getting the right answers for the right reasons [p. 252].” In this study the verification was based on the system's ability to reach appropriate conclusions on all of the sub-goals and the major goal. In the final system the verification specifically consisted of an examination of three aspects of the decision: (1) the data that was used, (2) the way the decision was reached (line of reasoning), and (3) the level of certainty (CF) attached to the decision. For the lower level decisions (completeness, authorization, etc.) the auditor generally agreed with the system's evaluation in each of these areas. This same agreement continued for the evaluation of the intermediate level decision on the adequacy of the separation of duties for the cases presented to the system. For the other intermediate level decisions concerning population and accuracy controls the auditor agreed with the system's use of data and with its line of reasoning but felt that in some cases the certainty factor was not correct. The auditor also felt that the system's overall evaluation of accounting controls was adequate in terms of the worksheet data that was used and the reasoning that resulted from the use of this data but the certainty factor determined by the system was generally different from one that they would assign. For most of the cases presented to the system this same evaluation was given by the auditor; 127 the use of worksheet data and reasoning were correct but the certainty factor for the overall evaluation was different than the one that they would assign. There are a number of reasons that became apparent for this lack of agreement with the system's degree of certainty. The first reason concerns the environmental_ factors that were explicitly excluded from the system's decision. Because actual cases were used it was hard for the auditor to exclude data that had been considered in the evaluation for the audit. A second reason is that auditors do not use certainty factors in their evaluation of controls. The worksheet simply requires a yes or no response to the question, "Are the controls adequate?'. There was never a disagreement on whether or not the controls were adequate only how certain the system should be with the information that it used to make its evaluation. Summary The verification of the final system indicated that two important components of the evaluation of controls were met: data acquisition and line of reasoning. For expert or knowledge based systems that are not constructed to progress beyond the third stage of the life cycle proposed by Gaschnig et al. [1983] these two factors are of primary importance. In fact the system also had agreement with the auditor on the final evaluation of controls; it was simply the degree of certainty that did not match the auditor's 128 evaluation. The next chapter examines the knowledge base that made these evaluations and discusses the database constraints that could result from information contained in the system. CHAPTER IV ANALYSIS OF THE SYSTEM This chapter examines the information contained in the INTERNAL-CONTROL-ANALYZER system. The first section of this chapter will discuss the model of the internal control decision that is represented in the system. The second section will relate the information in the system to the database issues of formulating data model constraints using SCRIPTS (or transaction prototypes) that could contain these restrictions and the design of databases to support decisions such as the evaluation of controls. Ih£.InL£In£l.QQDILQI.EEQIQBLIQB.HQQ£1 This section will review the internal control evaluation model which is contained in INTERNAL-CONTROL- ANALYZER. This discussion will concentrate on three areas: (1) the way in which the decision is made by the system, (2) the changes that were made to the prototype system which reflect the effect of experience on the decision, and (3) an examination of the nature of the evaluation decision. mmmmm This section will discuss the way in which INTERNAL- CONTROL-ANALYZER actually makes its evaluation of a 129 130 particular firm's accounting controls in the revenue cycle. A transcript of an actual session is contained in Appendix A. The discussion of the model will not necessarily correspond to the order of this transcript, but it will examine all of the possible paths that the model could take. At various points, particular rules that are used to make the evaluation will be presented in the English translation provided by the system. The first section will give a general overview of the process of evaluating accounting controls, and the second section will examine the portion of this process that is represented in the INTERNAL-CONTROL- ANALYZER SYSTEM. Genmlmrxiswefthsmlmmnw. Figure4 shows a decision tree overview of the internal control evaluation decision. The overall evaluation of the controls in the revenue cycle is determined by the presence of environmental factors and by the presence (or absence) of specific controls over the capture of transaction information. The information on the environmental factors is collected on a separate worksheet but incorporated in the evaluation of specific controls (population, accuracy, and separation of duties) particularly in those situations in which the specific controls may not initially appear to be 131 ACCOUNTING CONTROLS REVENUE CYCLE f A ENVIRONMENTAL FACTORS corporate ) personnel safeguards L L (THESE CONTROLS PERTAIN TO SALES AND CASH RECEIPTS) POPULATION SEPARATION ACCURACY CONTROLS OF DUTIES CONTROLS COMPLETENESS AUTHORIZATION COMPARISON MATHEMATICAL CONTROLS CONTROLS CONTROLS CONTROLS (THESE CONTROLS PERTAIN TO EACH DOCUMENT IN THE REVENUE CYCLE) GENERAL OVERVIEW OF THE EVALUATION OF CONTROLS FIGURE 4 132 adequate. The environmental factors fall into a three categories: corporate factors, personnel factors, and particular safeguards. Within the corporate factors are things like the degree of centralization, the presence of an internal audit staff, the level within the corporate structure that the internal audit staff reports to, and the degree to which management participates in information system design. Personnel factors include the corporate policies on hiring, training, and supervision of employees. Safeguard controls include protection of source documents and financial records. . The specific controls are evaluated for each of the transactions in each of the cycles. For the revenue cycle this includes an evaluation of both the sale and cash receipt transactions. The specific controls fall into three categories: population controls, accuracy controls, and separation of duties. The evaluation of population controls is based on an examination of completeness and authorization controls and ensures that only (and all) valid transactions are processed. The evaluation of accuracy controls is determined by the evaluation of comparison and mathematical controls and ensures that the processed transactions are correct. The third category of specific controls are those which ensure that there is proper separation of duties. The controls at the bottom of the decision tree of Figure 4 (completeness, authorization, comparisons, and mathematical 133 controls) are evaluated for each document that is used in processing of the transactions. For instance, for sales transactions the completeness and authorization controls for sales orders, shipping reports and sales invoices are evaluated to come to a conclusion about population controls for the sales portion of the revenue cycle. This overview provides a very general description of the evaluation process used in the examination of accounting controls for the specific firm used in the study. The next section discusses the way in which the judgment model approaches the evaluation. Thile the environmental factors are generally not included in the model the discussion will indicate some instances in the model in which certain of these factors were mentioned by the auditor. This is done to provide some indication of the possible role that these factors might play in the evaluation process. The eyeleetien mggel. This section describes the model of internal control evaluation which is represented in the production rules of the system. Figures 5 - 8 illustrate the judgment model. These figures are presented in a decision tree format. In each of these figures individual data items that are either collected or inferred by the system have a question mark after them and items which are decisions based on collected data items are presented in capital letters. Thus, "type of sales method?“ is a question posed by the system to obtain a data item 134 REVENUE CYCLE ACCOUNTING CONTROLS IL CONTROLS OVER CONTROLS OVER type of sales method? SALES CASH RECEIPT TRANSACTIONS TRANSACTIONS THE EVALUATION OF REVENUE CYCLE ACCOUNTING CONTROLS FIGURE 5 135 about how the firm handles most of its sales transactions, whereas "CONTROLS OVER SALES TRANSACTIONS“ represents a decision that will be made by the system. Each decision made by the system is broken down further in a subsequent figure to its individual data elements. Figure 5 indicates that the major decision (or goal). of the system is to reach a conclusion on the adequacy of the accounting controls in the revenue cycle. The figure indicates that this decision is based on the type of sales method used and an evaluation of the controls over sales and cash receipt transactions. There are four productions that make a conclusion about the accounting controls in the system, and each production has premises which require the system to evaluate controls over cash receipts and sale transactions. These productions ensure that all of the information about the transactions will be collected before the system attempts to evaluate the accounting ‘controls. RULEOG7 is one of these productions and has the following translation: RULE067 [This rule applies to firms, and is tried in order to find out about the adequacy of accounting controls in the revenue cycle if properly performed] IF: 1) The method that accounts for most of the firml's] sales transactions is on-account 2) The controls over the collection of data from sales transactions is effective, and 3) The controls over the collection of data from cash receipts in payment of sales on 136 account is effective THEN: It is definite that the accounting controls within the firm's revenue, if properly performed are effective The other three productions that make a conclusion about the major goal of the system are similar except they deal with the situations in which the controls over sales~ and/or cash receipts are not completely effective. In order to satisfy the requirements of these rules, the system must first determine if the company has most of its sales on account. This piece of information is obtained directly from the person running the consultation. The next two clauses provide the system with new sub-goals that must be met and therefore determine the pieces of information that should be collected next. The evaluation of controls over sales and cash receipts do not have equal weight in the decision about the accounting controls. The controls over cash receipts are given approximately twice the importance in the final evaluation of revenue cycle controls. That is, in the evaluation of accounting controls if the controls over the collection of cash are inadequate this is considered more serious than inadequate controls over sales transactions. Of course if controls in both areas are inadequate then the revenue cycle controls are considered inadequate. The estimation of the degree of importance that should be attached to the controls in each of these areas was a difficult portion of the refinement process and 137 mmumzu Q¢0H80 mAOMBZOU wzHBUmmmd mmOBU¢m Amumuooou smsov mommoameo umumsua mo wsoflmw>uomsm unsavont m0 «cowussuomcfi uousm «soaumuocfiwcoo umooo¢\uo>«amn «cofiuommcmuu mazmzouon 0» Suwu uflaaoo mo zomHm0 mAOMBZOU mqomazou mqomazou ZOHfimNHmOmBD< mmmzmamqmzoo / maomazou ZOHB 0) duties separated? (recording unit is not equal to custody unit and recording unit is not equal to operation unit and custody unit is not equal to operation unit) valid agent? (economic agent is valid) authorized? (operating unit is valid) event completeness? (EVENT-NUMBER in order or EVENT-TIME in sequence) ACTIONS: add economic-resource to stock-flow of economic-event add economic-event to participate with economic-agent add economic—event to control of economic—units-recording add economic-event to authorized of economic-units-operation ECONOMIC-EVENT with ATTRIBUTES: date: EVENT-DATE time: EVENT-TIME (EVENT-DATE:TIME$) document number: EVENT-NUMBER amount: EVENT-AMOUNT stock-flow: set of economic-resource participate: set of economic-agent control: set of economic-units-recording authorize: set of economic-units-operation TRANSACTION and EVENT PROTOTYPE FIGURE 9 (From Gal and McCarthy, 1985c) 176 captured by the system. To determine whether or not there was a completeness control INTERNAL-CONTROL-ANALYZER requires that the client have a procedure to count or identify exchange documents. This is different from an evaluation of the control, which would be based on the results of compliance testing. In a manual system which uses paper documents the obvious choice is to prenumber them and then later verify their order to see if any are missing. This type of completeness control is addressed in the first part of the completeness question in the transaction prototype. However, the use of prenumbered documents as the completeness control would not be useful (or appropriate) in two instances. The first case concerns those situations in which the document numbers assigned to the transactions are either determined by the outside party or do not enter the system in document number order. Cash receipts are examples of this type of transaction. The check number is assigned outside of the firm and the remittance advices will not necessarily be returned in any particular order. The second case corresponds to data processing systems that use the computer to capture the information about the event and to create the paper documents. The computer will only assign document numbers to the transaction it identifies and not to the ones that may be missing, therefore these numbers could not be used to determine whether transactions were 177 missing. In addition controls such as batch totals also can't be used to monitor completeness because they can only determine whether all the captured information is later processed: they can not determine that all events are captured. However, the time construct can be used to monitor the completeness of the events in the two situations previously identified. In McCarthy's original description of a semantic model of accounting data, "TIME" is used as an attribute to identify the date and time each economic event occurred. This relationship between an event and its time of occurrence is important if the data is to be used for certain accounting activities (such as the preparation of financial statements). The identification of a time for each event in the system is equivalent to creating a log of economic events that were recognized by the system and thus provides a basis for determining the completeness of the information. The problems associated with representing and using time in databases have been examined by a number of researchers (see Bolour et al. 1982 for a discussion). Many of the issues which have been investigated are more concerned with the semantics of time information in a database rather than the issue of connecting particular events to the time that they occurred (Lamport [1978] presents a very good discussion of using time as a key for events). The important point made by Lamport concerning the 178 use of time as an attribute is that transactions can be uniquely identified using the time stamps. The information contained in INTERNAL-CONTROL-ANALYZER indicates that a completeness control is one that identifies individual transactions. For cash receipt transactions the use of logs is considered an appropriate completeness control, and _ because time can uniquely identify transactions it can also be used as a completeness control for events which are captured initially by the computer. The prototype of the economic event in the bottom of Figure 9 represents a description of an event which conforms to McCarthy's REA model and includes the time attribute. The use of time for a completeness control is represented in the second part of the completeness question in the prerequisite portion of the transaction prototype. The evaluation of a completeness control based the relationship of a transaction to its time of occurrence is a different issue and might require a different approach. In a system which uses prenumbered documents it is possible to see if there is a break in the sequence of used documents and then interpret this as missing events. When the computer is responsible for capturing information about the event and time is used as a key for the events, this same type of evaluation is not possible. If there is a break in time between two sales for instance, it is not readily apparent whether or not there should be additional 179 sales between them. For this reason it might appear that evaluation of a completeness control is more difficult than one based on prenumbered documents, but some of the same types of issues must be considered in both cases. For instance in a system which uses prenumbered documents if there is a break in document numbers, the auditor must determine whether the missing transactions are material. Even if all of the documents are accounted for, the auditor must still determine whether some transactions weren't placed on documents. In a computer system the problem of determining whether transactions were missed in a long time span between transactions may require an evaluation of general controls of the computer environment, such as the use of system logs and backup procedures. In a system which uses prenumbered documents the auditor may need to evaluate the reliability of the personnel that initially place the information on the document. The evaluation of these general controls can give some evidence of the possiblity that there were uncaptured transactions during a certain time period, and may be required regardless of the type of completeness control used. The use of time as a key for each economic event can be used as a control to monitor the completeness of the transactions actually captured by the system. The INTERNAL- CONTROL-ANALYZER requires that a completeness identify individual transactions; because the time that an event 180 occurs can be unique, this component of the prototype can be used to monitor this control. The use of logs or time stamps may require an auditor to approach the examination of the control differently than a control based on prenumbered documents, but there are also similarities in the information needed to determine the effectiveness of the] control. Authprization cpntrpls. A second category of controls evaluated by the INTERNAL-CONTROL-ANALYZER system were those that ensured the authorization of all transactions. The prototype in Figure 9 indicates one of the roles that must be filled is the operating unit for the transaction. These roles used are outlined by Cushing [1982, p. 89]. The system also requests information about the person (or department) that commits the entity (firm) to the exchange as this was also a role identified in the manual of the firm. used in the study. For the authorization controls, only the way in which the operating unit role is filled is important. The prerequisite portion of the transaction indicates that authorized means that the operating unit is valid. For the INTERNAL-CONTROL-ANALYZER, the adequacy of this control was determined by whether or not it was performed; the person that filled the role of authorizer was not important. In the transaction prototype the performance of this control would be indicated by someone actually filling this role. The validity of the authorization can be verified by 181 SALE: SPECIALIZATION of ECONOMIC-TRANSACTION with ROLES: economic-event:SALE (ORDER NUMBER) economic-resource:INVENTORY (INVOICE LINE-ITEMS) economic-agent:CUSTOMER economic—unit-recording:SALESPERSON economic-unit-custody:SHIPPING—DEPARTMENT economic-unit-operation:CREDIT—MANAGER ACTIONS: create invoice based on ORDER LINE-ITEMS send message (shipping document) to custody unit to ship INVOICE LINE-ITEMS to CUSTOMER-ADDRESS send message (sales invoice) to economic agent at BILLING-ADDRESS indicating completion of sale LARGE-SALE: SPECIALIZATION of SALE with ROLES: economic-event:SALE (AMOUNT > $10,000) SALE SPECIALIZATION OF TRANSACTION PROTOTYPE Figure 10 (From Gal and McCarthy 1985c) 182 examining the job function of the person that eventually fills the role. In a large integrated database system, this can be accomplished by accessing the information about the employee and verifying that their job function is compatible with authorizing the particular transaction (see Gal and McCarthy, 1985a for a discussion of using job functions to~ restrict access to particular database activities). Figure 10 shows a specialization of the general prototype for a sales transaction with the particular roles and actions which are unique to a sale and indicates that CREDIT-MANAGER is the appropriate job function to authorize this transaction. This becomes a specific prototype for a sale and all those attributes not mentioned are inherited from the general prototype. Using the prototype, the control over the authorization of transactions can be monitored and those transactions that do not have the prOper authorization can either be blocked or simply saved for further evaluation. The selection of the action to be taken would depend on whether or not the particular authorization violation is crucial enough to void the transaction (possibly due to the size of the transaction; > $10,000) or whether the evidence should be accumulated for subsequent compliance evaluation. The system also evaluated authorization controls as being adequate if the firm used an approved customer list. The prototype of a sale transaction in Figure 10 also allows 183 this type of authorization control to be represented. The prototype identifies a customer as the appropriate outside agent for the transaction. Within any particular organization, this approval of a customer (as a compensation for lack of specific authorization) can take different forms such aS< assigning a credit limit. The prototype can‘ accommodate this disjunctive form of authorization. In systems that support the use of prototypes and exceptions (like TAXIS [Mylopoulos, Bernstein, and Wong, 1980]) the exception handling mechanism could be used to identify those transactions that do not have an appropriate operating unit but whose agent (customer) is valid and therefore determine that the transaction is appropriately authorized. This same exception handling mechanism could also accommodate transactions (like cash receipts) that would not have the authorization role filled by specifying that this violation of the prototype is not a control problem ‘and the transaction is acceptable. The exception handling capabilities of systems which support prototypes imply that a separate prototype is not required for groups of transactions that do not have all of the components identified in the general prototype; the designer (auditor) only needs to supply an appropriate set of procedures to handle the deviations from it. The controls which ensure that the transactions are authorized can be represented directly in a prototype such 184 as that shown in Figures 9 and 10. The identification of exceptions is very straightforward, particularly when compared to the identification of exceptions to completeness, and simply involves validation of the way in which certain roles in the transaction are filled. The prototype can accommodate both the specific authorization from a credit manager (for instance) and the indirect or compensating authorization in the form of approved customers. Qomparison controls. The third category of controls examined by the INTERNAL-CONTROL-ANALYZER system were the comparison controls. These controls attempt to ensure that information is consistent across all the documents that are used to complete a transaction. For instance, the comparison controls for sales attempt to ensure that the items that are ordered are also those that are shipped and subsequently billed for. For cash receipts, the comparison between the remittance advice and the check is supposed to ensure that the amount credited to the customer (from the remittance advice) agrees with the amount deposited in the bank. For a system which would create the documents after receipt of the order (as is the case in the prototype in Figure 10) the consistency of information on subsequent documents is insured and the auditor would be more interested in the design of the system. A similar point was 185 raised previously by Vasarhelyi [1980] when he noted that, for computer systems, consistency type errors would be eliminated and therefore the concern for the auditor would be the correctness of the initial system design. For cash receipts, the problem is slightly different in that) the system does not create the remittance advice from the check (or vice versa). The customer sends in both of these documents and therefore the comparison control is a verification of the consistency of the customer's system. In an integrated system, it would probably not make sense to use both the remittance advice and the check as only the check from the customer contains information which must be entered. Mathematical checks. The fourth control identified comprised the mathematical checks. These controls are designed to ensure that the mathematical accuracy of the transaction information is maintained. A prototype for a transaction would specify certain mathematical relationships, or constraints on the data, that must be true. These constraints are included under the general heading of semantic integrity constraints which include many restrictions on data that must be true (some of which can be specified mathematically). The mathematical checks that the auditor was looking for were things like extensions and totals on invoices. The representation of constraints on these mathematical relationships as part of a prototype is 186 straightforward but the question of how to enforce or evaluate them is very situation specific (see Gal and McCarthy, 1985a for a more indepth discussion). The information contained in the INTERNAL-CONTROL-ANALYZER suggests that even in an environment in which the computer does not specifically capture the information, the» importance of the mathematical checks is still reduced because of the use of other devices which improve the mathematical accuracy. Seperation pt_ pptiee. The final type of control examined by INTERNAL-CONTROL-ANALYZER was the separation of duties. In the development of the general prototype, the specification of the various roles that must be filled is used in the prerequisite portion to define the incompatible functions that must be separated. In a specific prototype (such as the one in Figure 10 for a sale) there are two possible approaches to the identification of the people that can fill these- roles. The first would specify particular people that are performing the different job functions at a certain point in time. This type of constraint on the system's ability to accept certain people performing various functions is a static constraint in that it is necessary to change the constraint as people change jobs. Gal and McCarthy [1985a] show how the constraints on the performance of different roles can be specified by job 187 function rather than for individuals. In this way the constraints in the system are dynamic [Zloof, 1978] and adapt to changes in job assignments. The system also allowed for situations with inadequate separation of duties to be evaluated as adequate if complete separation would not be cost effective because of the size of the accounting department, and there was adequate supervision. The use of supervision (an environmental factor) compensates for an inadequate control. Although the use of an approved customer list was identified as a compensating control for the lack of specific authorization, there are some differences. The customer list is another way to authorize transactions: the authorization is customer specific rather than transaction specific. On the other hand supervision is not another form of separation of duties. The environmental factor "supervision" truly compensates for lack of separation of duties. This environmental factor was included in the system because the representation of supervision can be accomplished directly_ within the database. The relationship between employees and their supervisior can be represented (for example) simply by including an attribute for each employee called “MANAGED" which would have the employee number of the person that supervised the particular person. The question of the definition of terms like "adequate" and how to include them in a database is still unanswered and will be addressed 188 later in this chapter. The representation of separation of duties is accomplished directly in the definition of roles and comparison of the individuals that actually fill them during the execution of a transaction. In the studies from Chapter I that examined the effect of different variables on. internal control evaluation, separation of duties accounted for most of the variation in the judgments and therefore might be considered a very important feature of representation of a transaction. Summary. This section has examined the different controls that were reviewed by the INTERNAL-CONTROL-ANALYZER system. This review discussed the types of errors that the controls were designed to prevent and the way in which the control could be represented in a transaction prototype developed from an REA perspective. The way in which the prototype could be used to identify deviations .was also discussed. This section raised the question of the evaluation of these deviations. However, it did not present a model for compliance testing these controls. The INTERNAL-CONTROL-ANALYZER only examined a system to see if there were controls in five major areas: completeness, authorization, comparison, mathematical checks, and separation of duties. In traditional systems, the auditor would take a sample of transactions, identify the deviations and then 189 make inferences about the entire population. In advanced systems in which the computer is a participant in collecting and monitoring transactions, some of these traditional approaches may not be effective. vasarhelyi [1980] did raise the issue of the change from error detection based on pattern 'recognition to an approach which emphasizes the analysis of the system design process. There are, however, some indications that the use of prototypes does provide a basis for examining data from the system itself to evaluate controls. A major difference might be in the statistics used to evaluate deviations from a transaction prototype. If prototype or transaction SCRIPT were used to monitor the actual transactions, it could identify ell transactions that do not follow the predetermined structure of the prototype. This would mean that the auditor would not have to make inferences from the errors in a sample to errors in the population. A second difference was noted earlier in the examination of completeness controls and that is when different indicators of the control are used different inferences are appropriate. If time is used to verify completeness instead of prenumbered documents, then a different approach is called for to determine if transactions are missing. Svanks [1981] raises some of these issues and argues for EDP auditing with the data as Opposed to auditing the process which produces the data in the database. The next section examines some of the 190 implications that the current study has for the use of database constraints in the continuous monitoring of a database and in the construction of an audit abstract. Database gpnstreihte epp Auditing In. the previous section, various constructs of the. transaction prototype developed in Gal and McCarthy [1985c] were integrated with controls that were examined by the INTERNAL-CONTROL-ANALYZER. In this section the idea of database constraints and restrictions on states of the database will be related to the controls which the auditor considered important and the data that was used to conclude that a particular control was present. Typically constraints on the data in a database system are formulated to ensure that certain states or configurations of the data will not occur. Zloof [1978] and Theerachetmongkol and Montgomery [1980] demonstrate the construction of semantic integrity constraints in a QBE system and use a number of examples of states that should not be allowed. For instance, if a company has a policy that an employee must not make more than their supervisor, then an attempt to change a salary that violates this should be blocked. This idea of constraint can be extended to operations performed on the data. For instance, the authority to change a salary might be granted only to the appropriate person in the personnel department and the 191 constraints in the system should not allow changes to the salary field except by that person. Constraints such as these are similar to preventitive controls and allow for a very straightforward response by a computer system if there is an attempt to violate them; the transaction is blocked and the state of the database is not changed. With the_ economic transactions represented in the prototype, a different approach to constraints is needed. The transactions that would be specific instances of the prototype of Figure 9 change both the database state and the economic position of the firm. If a violation of the constraint concerning the relationship of employee and supervisor is allowed, a change has occurred to the database state but until the payroll programs are run nothing has occurred to change the economic position of the firm. This distinction between the general use of constraints and the use of transaction prototypes as constraints means that violations of constraints need to be interpreted differently. The prototype for a transaction must be considered a description of the way in which a sale should look. The question about controls that must be answered is whether deviations from this structure lead an auditor to believe that the information which has been captured is not correct, ie., does not match the way in which the economic position of the firm has changed. There are some restrictions in the REA model and the 192 transaction prototype that can't be violated. For instance, the REA data model is strictly typed [Tsichritzis and Lochovsky, 1982] which implies that all data must pertain to prespecified types (either Resources, Events, Agents, or to some relationship between them). Information that does not fit into_one of these types will not be captured. Authority_ controls might also fit into the category of constraints that can't be violated and will therefore cause an economic transaction to be disallowed. However, controls such as completeness and separation of duties are not control problems if they only occur for a single transaction. It would not be possible to look at a single transaction and conclude that completeness controls had been violated. Similarly inadequate separation of duties in a single transaction would not be evidence that the information contained in all of the transactions was incorrect. Further there are other types of separation of duties which are discussed in Gal and McCarthy [1985c] which can only be evaluated in the relationship of transactions in a cycle. For instance, if the same person authorized a particular purchase order and then subsequently was assigned to a position which authorized the payment of the invoice, the problem with separation of duties is at the cycle rather than at the transaction level. It would not be possible to disallow the original purchase because it already occurred and the economic position of the firm was affected. 193 Therefore the violations of the controls represented in the prototypes should not be considered as evidence of errors in the system; the data contained in the database may actually represent the economic position of the firm. The violations of the constraints in the prototype should be considered as evidence whose cumulative weight must be evaluated to determine whether the information contained in the system does not correspond to the economic position of the firm. It is for this reason that the use of prototypes can be connected with the use of System Control Audit Review Files (SCARF) discussed in Weber [1982]. In his discussion of concurrent auditing techniques, Weber points out some of the problems with the use of embedded audit routines. He mentions the problems of determining the information to collect and the way in which the information will be collected [p. 483]. His emphasis in the discussion of the SCARF routines is as a tool to monitor certain transactions as they proceed through the processing activities. Much of this processing is designed to use the information from transactions to update various data files. In integrated database systems such as those contemplated by McCarthy [1982] in his development of the REA model much of this type of processing is eliminated as information is kept in a single location and format for use by multiple users. In addition these systems will use the computer as a participant in the data collection process itself. The 194 information contained in the INTERNAL-CONTROL-ANALYZER identifies the types of controls that the auditor considered important to ensure that the collection process captures data that will correspond adequately to the economic exchanges it is supposed to represent. The previous section related . those controls to various portions of the transaction prototype. Therefore the model of the transaction presented extends the idea of concurrent auditing not only to the processing of the data but also to the collection of it. If the auditor is concerned that the transactions actually captured by the system do not represent all changes to the economic position of the firm (completeness controls are in question), information about the time between transactions should be examined. The prototype begins to answer the question about what information to capture during the continuous monitoring of the data collection process. The prototype (does not determine how the information should be used. In this way the prototype has identified the information from the database that should be used if an evaluation of certain controls is the decision that is to be made. The difference between declarative (data) and procedural (how to use it) knowledge about accounting controls is represented in Figure 11. The specification of a transaction in terms of the prototype is declarative in that it only identifies an appropriate configuration for the ECONOMIC 195 TRANSACTION AND CYCLE PROTOTYPES Q 4 SOURCE TRANSACTIONS EVALUATION‘é PROCEDURES DATABASE 1 gr” AUDITING VI (SCARF) i TRANSACTION PROTOTYPES AND THE AUDITING VIEW FIGURE 11 196 data which pertains to economic transactions and cycles. This appropriate configuration can then be used to construct a view of the database which would correspond to the SCARF mentioned by Weber. The information in the INTERNAL- CONTROL-ANALYZER is procedural in that it specifies how to use certain pieces of data to reach a decision. The. productions (procedures) use data about the identification of individual documents (transaction) to determine whether a particular control is present. This procedural-declarative distinction can also be demonstrated by reviewing the information that would lead the auditor to determine that there was a compensating control for inadequate separation of duties. During the refinement of the initial system through interaction with the auditor there were cases that didn't have complete separation of duties but were judged to be sufficiently controlled because there was . adequate supervision. It was decided that the system would look for these situations so that the cases would be handled correctly. The decision to include this situations in the system was also made because the declarative aspects of supervision can be represented in a database which has been designed using the REA model. It was not possible to determine the procedural knowledge which concludes that a particular supervision instance represented "adequacy". The INTERNAL-CONTROL-ANALYZER asked for the output of the 197 auditor's decision procedure because some of the declarative knowledge about supervision could be represented as part of the REA model. There were other environmental factors besides supervision that entered into the evaluation. Many of the differences between the conclusions made in the actual workpapers and those made by the system were due to the role played by these factors in the evaluation. The next section examines some of these factors and discusses some database design issues raised by the fact that they were external to the database. Wmmwmm Date [1982, p.3] describes a database system as a computer-based recordkeeping system which records and maintains information that is significant to an organization and that may be important to decision-making. The definition provided by Date emphasizes the role of the database in providing data that might be used by various decisions. This study has examined a decision which concerns the evaluation of the quality of the data in the database. An important point identified in this research was the role that external data plays in this decision. If this external data is truly valid and useful in this particular decision, which has great importance to other decisions that use the data, then it makes some sense to determine whether it can be included in the formal information system. This 198 is even more critical if a form of compliance testing is to be built into the system (evaluation of not only the presence of and deviations from certain controls but also of the importance of the deviations) as an ongoing process. This section will examine the possibility of identifying and including information about environmental factors in a- production database. A review of the information requested on the worksheet which is used to evaluate the environmental factors leads to the conclusion that what is being collected is the output of procedures. The results of these decision processes can be viewed as data at some level but it is certainly not the type which is traditionally part of a database. For instance, one environmental factor that the auditor felt was particularly important in the evaluation of controls was the control consciousness of the CEO (chief executive officer). The auditor considered this to be an important factor, because it affects decisions about the possibility of management override and general adherence to controls. This one piece of data demonstrates two critical issues which concern the use and collection of environmental factors in the evaluation of accounting controls. The first problem is that it is unclear exactly how environmental factors should be used in the decision. In the discussion of factors (such as the control consciousness of the CEO) with the auditor there wasn't any particular 199 control that would be improved or any specific errors that would be reduced. This may not be a problem due to the lack of a specific normative definition of reliable data. Further, the nature of expertise may be the ability to identify those situations that call for the use of factors such as these. The inability of the auditor to specify the‘ precise effect that the external data has on the evaluation is not important in terms of the representation of the data in the database. The second concern is that the value for the control consciousness is most certainly the output of a procedure as opposed to a data element such as the CEO's salary, bonus, or education level. The auditor examined certain attributes of the CEO and then made the evaluation that this person had an appropriate level of consciousness about the controls in the organization. To make “control consciousness“ an intensional feature (attribute) of the CEO (or any manager) hides both the declarative elements which determine its value as well as the procedural components which would produce its value for a particular manager. In addition because it is the output of a procedure (an auditor's decision process) the question of whose procedure to use must also be answered. McCarthy [ 1982] makes the the distinction between the declarative aspects of the REA model and the procedural features that use these facts. McCarthy [1982: p. 556] regards database design as, 200 "... a process during which an attempt is made to mirror aspects of an identified reality (called an object system) in an abstract model...'. In the case of the environmental factors some of their features can be represented as declarative facts as part of the REA model. Facts such as an employee's supervisor, the workloads of employees, and the education level of the CEO are declarative elements and therefore can be represented as data elements in a database. The information contained in the INTERNAL-CONTROL-ANALYZER can be used to identify information which is used in the control decisions and external to a database system. The analysis of this external set of information can then lead to an understanding of the deficiencies in a current database in terms of its ability to provide information that is used in corporate decisions. If the reporting level of the internal audit staff is important, then it makes sense to design database structures which mirror this aspect of a corporation or object system. In this way the construction of a knowledge based or expert system can provide a decision support orientation towards the design of database systems. Summery This section has examined the information contained in the INTERNAL-CONTROL-ANALYZER in terms of the ability of various portions of the transaction prototype to represent controls reviewed by the system. In the first section the 201 relationship between the types of controls, the auditor required in the system, and components of the transaction prototype was discussed. It was argued that all of the controls (completeness, authorization, comparison, mathe- matical verification, and separation of duties) can be moni- tored within the general specification of an economic. transaction. The second section discussed the relationship of these controls to database constraints. The argument was made that these controls do not represent typical constraints on states of the database because the transactions represent changes in economic states which are occurring outside the database. Therefore these constraints represent a desired form of a transaction and the deviations must be analyzed in a cumulative manner to evaluate the realtionship between the stored data and the economic position of the firm. This view of the prototype and the analysis of deviations from it answers some of the questions raised by Weber in his discussion of the use of SCARF routines as a method for concurrent auditing of an information systems. In particular, it addresses those situations in which the computer participates in the collection of information during the execution of the economic transactions. In the third part of this section the use of information in the internal control decision which is external to the database was discussed. The discussion 202 of these environmental factors concluded that the problem with making them part of a production database was that they were procedural as opposed to declarative in nature and therefore hard to represent as intensional features of the database. The section concluded by arguing that the use of expert systems can therefore answer certain database design_ questions because the decision model embodied in the productions can give certain insights into the role played by external data and identify those items which can and should be included as attributes in the database. This concludes the analysis of the knowledge base constructed as part of this dissertation. The next chapter summarizes the findings, discusses the implications of the research for future work in modeling decision processes and designing database systems, and then suggests some possible extensions. CHAPTER V CONCLUSIONS AND SUGGESTED EXTENSIONS The previous four chapters of this dissertation have presented an analysis of internal accounting controls, provided a theoretical basis for modeling an auditor's control evaluation and representing the data to be used in the evaluation, discussed the particular tool and method used in this study, and finally analyzed the knowledge base that was developed. This chapter will begin by summarizing the results of this research project and relating it to other work. Following this will be some suggestions of possible extensions to this dissertation. men Wandws c W As companies begin to integrate more of their information resources into large corporate databases, the problems of ensuring that this data is accurate will become more acute. In these large systems, it will be crucial for the computer to participate to a greater extent in the control process particularly as data, in the form of transactions, is added to the data pool. The main purpose of this research was to investigate the role which data from 203 . . .- .—". ,_... _ . .- 204 a database plays in an auditor's evaluation of these controls, and the ability of prototypes of transactions to identify this data. In addition this research project has examined the use of the expert systems methodology in the modeling of decision processes. The INTERNAL-CONTROL-ANALYZER system contains the_ knowledge base necessary to make internal control evaluations. The system identifies various pieces of information which influence the decision. The information pertains to controls in three general areas. The first of these control areas attempts to ensure that the valid transactions will be captured by the information system and the invalid ones will not. The second type of control concerns the accuracy of the infOrmation, in terms of its consistency and its mathematical integrity. The final type of control evaluated by the system concerns the presence of adequate separation of duties. This final control is less related to specific errors. In addition, the system also contains information to identify compensating controls in these areas, such as the use of an approved customer list to validate transactions when specific authorization is lacking. In previous research on the evaluation of internal controls, the researcher identified different factors which were considered important. The researcher then constructed different cases based on combinations of these factors. 205 This approach made it difficult to examine the decision process used to evaluate the situations represented in the cases. In these studies the analysis was necessarily confined to inferences about the portion of the overall decision variance explained by certain factors. Ashton [1979] commented that in his initial study [1974] specific auditors' varied greatly in their use of different factors, but the combined data does not make this apparent. One of the contributions of this study was its use of the expert system methodology to study the auditor's evaluation strategy. A primary benefit of this approach was in the identification of the factors that the decision maker consider important. The use of this methodology to examine the decision helped to identify certain conditions that intuitively would appear to be important, but through the refinement process it was determined that they did not have any effect on the auditor's judgment. Further, the use of the expert system methodology provided (for the identification of factors which led the auditor to conclude that certain situations had adequate controls. Although the knowledge base was developed using a particular auditor from a single firm, there are still benefits to be derived from such a project. Simon [1980] argues that an important step in understanding problem solving behavior is in developing a taxonomy of alternative strategies used in a particular task. This research _ _____.__.:91- .n-‘h ...-.3.}qu ”b 206 represents one step in the development of such a taxonomy. In addition Dukes [1965] noted that research projects which used single subjects have played a major role in psychological studies particularly as they may help in, ”... clarifying questions, defining variables, and indicating approaches [p. 78]." The information used by the system was restricted to that portion which could be incorporated in a functioning database system. This research project discussed the concept of database constraints and related them to SCRIPTS [Borgida et al., 1934; Schahk and Abelson, 1977]. These transaction prototypes could be used to construct a view of the entire database that would support an evaluation of the various controls. The use of these constructs was examined within information systems design languages such as TAXIS. This represents a different approach to the integration of decision models with data models. Previous work has either structured the data specifically for the particular decision (as is the case with expert systems which use EMYCIN) or required procedures to optimize queries on the database (as is the case with the work of Jarke and Vassiliou [1983]. In addition the use of only database items in the evaluation provided some insight into the role played by external information in this particular decision. The identification of this information which is useful to the auditor as a decision maker can also answer certain database design 207 questions concerning the types of data which should be included in a production database to increase its ability to support decisions. The next section will present some possible extensions to this work. Reeeihle Estexe.fixteneiene.e£ the Beeeereh The future extensions of this research project are of two types: 1) further investigation of the internal control decision, and 2) further investigation of database use in decision support. In the following sections, different possibilities in each of these areas will be discussed. In1e§tiset12n.2£ the Intexnal Centfel Desieien Based on the results of this work, there are a number of possibilities for further investigation of the internal control evaluation decision. The first is to expand the present system to include the environmental variables which were excluded. A second possibility would be to use the same prototype system and investigate changes made to the system by other auditors from the same firm. A third direction that might warrant future investigation would be to use information contained in the system to build cases to investigate the importance other auditors place on the factors used by the subject in this study. Each of these possibilities is explained below. Inelude W1 varieties. There were certain variables that were specifically excluded from the system 208 constructed as part of this project. Based on a comparison of the conclusions reached by the system and the auditor, it was apparent that these factors were important to the decision. Therefore, in order to improve the system's ability to simulate the decision strategy of an auditor, it would be necessary to include these variables in the system. Once the system uses these items in decisions, it would be possible to determine whether the items play a systematic role in the decision or whether their use is very case specific. Such a determination will not be easy due to the nature of the information that must be included. This observation is based on the results from a previous study by Stansfield [1980] which attempted to develop a commodity expert. One of the problems Stansfield encountered concerned attempts to represent and to include certain information in the evaluation process used by the commodity expert. The information was very similar to the environmental information used by the auditor in the present study. The difficulty Stansfield experienced was in the interpretation of the information and in the identification of its causal relationship in the decision process. For instance, his initial attempt was to use ‘live' data from news articles and one of the examples he provides of difficult data to interpret was “rainfall in the Mid-west 209 improved crop prospects.‘I He concluded that the domain was far too complex for the knowledge representation tools presently available. While the domain of auditing might be more restricted in the types of information it uses to make its evaluations (particularly in the area of controls) it still may prove to be an extremely difficult task to include factors such as the control consciousness of the CEO. flee pther apditors tp refine the System. In the analysis of the refinements to the prototype system, it was argued that the changes represented the effects of experience. This suggests two possible directions for investigation of changes other auditors might make. The first extension in this area would require obtaining another auditor from the firm that had a similar background in terms of clients audited. The refinements made by this person would be expected to be similar. A second possibility is based on the observation that the auditor that participated in the study had not seen a company that had poor internal controls. Due to this experience, the evaluation might have been based on an expectation that the controls were adequate, and the audit papers support this. The extension of the research in this area would require an auditor from the firm that had experience auditing firms that did not always have adequate controls or ones with a high risk of failing. The refinements made by this person would be expected to quite different. In both of these extensions 210 the refinements would be of interest to determine the effects of different experiences on the evaluation of accounting controls. Experimental studies pt internal control judgmentS. The analysis of INTERNAL-CONTROL-ANALYZER identified various factors that influenced the decision of the auditor. These‘ variables suggest a number of different controlled experimental studies which would investigate various influences on decisions of auditors. One particular set of factors which had some importance in the evaluation of controls were the compensating factors identified by the auditor. The results of experiments which varied the compensating controls could determine whether other auditors actually consider them to compensate for the controls generally used. Another possible set of factors that might be varied in an experimental setting are the environmental factors that were excluded from the system in this study. In the next section the possible extensions of this work in the database area will be examined. Investigation pt Database Issues This section will present some possible extensions of the dissertation to investigate some of the database design issues. There are three areas that are suggested by the work represented herein. The first is to do a complete implementation of an REA accounting system similar to the 211 Gal and McCarthy [1983, 1985a & b] using TAXIS or similar language. The second is use the expert systems methodology to fully investigate the data elements that are used in various decisions which are external to database systems. The third possibility would be to use currently developed integrity measures to investigate violations in a production database. and to compare this with the evaluations made by practicing auditors. Each of these possibilities will be examined in more detail below. Wm e imelemeetatien E1119 Wee. The previous work of Gal and 'McCarthy [1983, 1985a & b] demonstrated that the REA model was a realistic approach to the modeling of accounting data because it was useful as an implementation guideline for database management systems which use first generation data models. As an extension of this implementation research, the REA model, the transaction prototypes presented in Gal and McCarthy [1985c], and the information about controls identified in this study, could be used to implement an accounting system using the TAXIS [Wong, 1981] system or one that is similar (e.g. Simula, [Dahl and Hoare, 1972] or Smalltalk, [Ingalls 1978]). Use expert systems tp identify exterhal date items. In this dissertation, data items identified as external to a traditional production database were not included in the expert system. One important use of the methodology would 212 be first to identify those environmental factors which are important in the evaluation decision. After this was done it would be possible to begin to identify the data elements that ,are used in the decision about the environmental factors. For instance, if the ”control consciousness of the CEO" was identified as important, then either expert systems~ or other experimental methods could be used to identify specific declarative elements used in the procedures which determine a value for this environmental variable. One of the goals of database design is to include items which allow the system to support corporate decisions. These same methods could be applied to other decisions which are made at all levels within organizations and must use data which is external to the formal information system. Integrity measures ehp internal control jpdgmente. One problem with the current methods for evaluation of accounting controls is their lack of objective measures of errors. In accounting firms, there are some attempts to make the evaluation less subjective and yet there is little work on relating objective measures of data integrity with auditor's judgments. The work of Svanks [1980] represents an approach to the objective measurement of errors in a database system. An extremely interesting area for future investigation would be to use the integrity measures suggested by Svanks and to have auditors use their judgmental techniques to asses the possibility or errors in 213 the same set of data. A comparison of the two evaluations would yield insights which would be important for the refinement of auditor's methods and might also allow for the refinements of the integrity analysis approach. nc us'on This dissertation has demonstrated the application of the expert systems methodology to the study of judgments of internal controls. In addition, it has investigated the issues of the use of database elements in the decision. This research has also presented an approach to the identification of the information from the database that would be useful in this decision. It is hoped that the line of research which is represented by the dissertation will ensure that future accounting and management information systems will be able to integrate the full set of conceptual and technological advantages represented by database and artificial intelligence developments. APPEND IX APPENDIX A Sample Session with INTERNAL-CONTROL-ANALYZER lO-Jan-BS 10:42:17 ---------- Firm-ll4--------- 1) This system is designed to collect and evaluate information about the controls over the collection of data for revenue cycle transactions (sales and cash receipts). The system recognizes the following personnel that participate in these transactions for a large number of firms: Acct-clerk (to differentiate a l or 2 may be added to the end, Acct-clerkl), credit-mgr, treas-sec, asst—treas, president, edp, shipping-clerk, shipping-supervisor, sales- clerk, production, and controller. If the function is not listed use OTHER as the response to questions about people that perform certain activities. 2) What is the name of the client? ** Firm 114 Inc. 3) What is the major line of business for Firm 114? ** manufacturing 4) Is there a procedure to count or identify sales orders? ** yes 5) What is the procedure to count or identify sales orders? *1: 7? Expected responses are: N/A, LOGS, PRENUMBERED, COUNTERS or DAILY-LISTING ** prenumbered 6) Who performs the procedure? ** sales-clerk 7) Is the completeness of sales orders reviewed? ** Why [i.e. Why is it important to determine if the completeness of sales orders is reviewed?] 214 215 [1.0] This will aid in determining whether the completeness controls for sales orders is effecive if properly performed If [1.1] information has been collected about the completeness controls for sales orders, and [1.2] there is a procedure to count or identify sales orders, and [1.3] the completeness of sales orders is reviewed, and [1.4] the person that performs the completeness procedure is not the same person that reviews the procedure then it is definite (1.0) that the completeness controls for sales orders is effective if properly performed, and it is definite (1. 0) that one of the observations based on the information provided is that: THERE IS A PROCEDURE TO ENSURE COMPLETENESS OF SALES ORDERS AND IT IS REVIEWED BY A DIFFERENT PERSON; THIS CONTROL IS CERTAINLY EFFECTIVE ** no 8) Is there a procedure to count or identify shipping reports? ** no 9) Is there a procedure to count or identify sales invoices? ** yes 10) What is the procedure? ** prenumbered 11) Who performs the procedure? ** edp 12) Is the procedure reviewed? ** yes 216 13) How often is the completeness of sales invoices reviewed? ** before-shipment 14) Who reviews the completeness of sales invoices? ** acct-clerk 15) How is the review documented? ** ?? Expected responses are: INITIALS, SIGNATURE, ATTACH-COPY, N/A, or NONE ** initials 16) Are sales orders specifically authorized? ** yes 17) When is the sales order authorized? ** before-shipment 18) Who authorizes sales transactions? ** sales-clerk 19) How is the authorization documented? ** initials 20) Is the authorization reviewed? ** yes 21) When is the authorization reviewed? ** before—shipment 22) Who reviews the authorization? ** credit-mgr 23) How is the review documented? ** initials 24) Is the sales order compared to verify its accuracy? ** yes 217 25) What is the data source that is compared to the sales order? ** sales—invoice 26) When is the SALES—INVOICE compared to the sales order? ** before-shipment 27) Who compares the SALES-INVOICE to the sales order? ** sales—clerk 28) How is the comparison documented? ** initials 29) Is there a comparison to ensure the accuracy of shipping reports? ** no 30) Is there a comparison to ensure the accuracy of sales invoices? ** yes 31) What document is compared to the invoice to verify prices? ** price-list 32) When is the PRICE-LIST compared to the invoice? ** before-shipment 33) Who compares the PRICE-LIST to the invoice? ** sales-clerk 34) How is the comparison of the PRICE-LIST with the invoice documented? ** none 35) What data source is compared to the invoice to ensure the correctness of the items listed? ** sales-order 0 I Ann... “I W C‘E.-'|- I- ~v.. 218 36) When is the SALES-ORDER compared with the invoice? ** before-shipment 37) Who compares the SALES-ORDER with the invoice? ** sales-clerk 38) How is the comparison documented? ** initials 39) Is there a mathematical check of the accuracy of sales orders? ** no 40) Is there a mathematical check of the accuracy of shipping reports? ** no 41) Is there a mathematical check of the accuracy of sales . invoices? ** yes 42) When is the mathematical check of invoices performed? ** before-shipment 43) Who performs the mathematical check of the invoices? ** credit-mgr 44) How is the mathematical check of invoices documented? ** none 45) Who commits the firm to the sales transaction? ** sales-clerk 46) Who delivers the physical consideration to the customer? ** shipping-clerk 47) Who is responsible for entering the data from sales transactions into the processing system? ** acct-clerk 219 48) Is there a procedure to count or identify remittance advices? ** yes 49) What procedure is used? ** logs 50) Who performs the procedure? ** acct-clerkz 51) Is the procedure to ensure completeness of remittance advices reviewed? ** yes 52) How often is the procedure reviewed? ** daily 53) Who reviews the the procedure? ** asst-treas 54) How is the review documented? ** none 55) Is there a procedure to count or identify customer checks? ** yes 56) What procedure is used? ** logs 57) Who performs the procedure? ** acct—clerkz 58) Is the procedure to ensure completeness of customer checks reviewed? ** yes 59) How often is the procedure reviewed? ** daily 220 60) Who reviews the procedure? ** asst-treas 61) How is the review documented? ** none 62) Is there a procedure to compare remittance advices to another data source to ensure accuracy? ** yes 63) What is the data source? ** check 64) When is the remittance advice compared to the CHECK? ** ‘3? Expected responses are: N/A, DAILY, MONTH-END, AT-TINE-OF- CASH-POSTING, UPON-RECEIPT, AT-TIME-OF-DEPOSIT, MONTHLY, PERIODICALLY or TEST-BASIS ** at-time-of-cash-posting 65) Who performs the comparison? ** acct-clerk 66) How is the comparison documented? ** none 67) Is there a procedure to check the mathematical accuracy of remittance advices? ** no 68) Who accepts the check from the customer? ** acct-clerk 69) Who is responsible for entering the cash receipt data into the processing system? ** asst-treas -b Kiri?“ I)“ :7" The the The 221 following are the observations and conclusions based on responses provided: There is a procedure to ensure completeness of sales orders but there is not a review, therefore the control is less effective ‘ Because there isn't a procedure to ensure completeness of shipping reports there is a missing conrol Because there isn't a comparison of shipping reports there is a missing control The most important mathematical check for sales documents is for the invoice, therefore even though the checks for the other documents are missing the control is still fairly effective Without some form of documentation for a review it is probably necessary to observe the procedure for some of the remittance advices Because there isn't a mathematical check for remittance advices there is a missing control accounting controls for the revenue cycle of Firm 114 if properly performed is as follows: EFFECTIVE (.4) BIBLIOGRAPHY BIBLIOGRAPHY Aiello, N., ”A Comparative Study of Control Strategies for Expert Systems: AGE Implementation of Three Variations of PUFF,“ Proceedings pt the National gonference pp Artificial lntelligenpe (August 1983): PP. 1-4. American Institute of Certified Public Accountants, Alggh grofessiOpel Stapderds 101. l (Commerce Clearing House, 1979). Anderson, J.R., “Acquisition of Cognitive Skill,“ Reyghp; lpgical Reyiep (1982, 89:4): PP. 369-406. ________. The Arehiteeture of Qesnitien (Harvard University Press, 1983). Arens, A.A. and J.K. Loebbecke, Legitipge.Ap,lptegreted Approach (Prentice-Hall, 1980). Arthur Andersen & Co., A Spipe fer Studying epd Eyelpetlpg lpternal Accounting Qpntrols, (Arthur Andersen, 1978). Ashton, R.H., 'An Experimental Study of Internal Control Judgments," Journal.ef Aeeepntiug Beeeareh(8prin9 1974), pp. 143-157. , "Comment: Some Observations on Auditors' Evaluations of Internal Controls," Spurnal pt pppppptipgy'huditipg e Einance (Fall 1979), pp. 56-66. , uman Information Processipg ip Accounting (American Accounting Association, 1982). , and P.R. Brown., "Descriptive Modeling of Auditors' Internal Control Judgments: Replication and Extension," Jeernel.ef,Aeeounting Researeh (Spring 1980), pp- 267- 277. Ashton, R.H. and 8.8. Kramer, "Students as Surrogates in Behavioral Research: Some Evidence,“ qurnal pt Aeeeentiae Beeeareh (Spring 1980), pp- 1-15- Bailey, A.D., J.H. Gerlach, R.P. McAfee, and A.B. Whinston, 'An OIS Model for Internal Accounting Control Evaluation," ACE Traneaetien§.en.9ffiee Infermatien Systems (January 1983a), pp. 25-44. 222 223 Bailey, A.D., G.L. Duke, J. Gerlach, C. Ko, R.D. Meservy, and A.B. Whinston, "A Mathematical Contraction and Automated Analysis of Internal Controls," (University of Minnesota, 1983b). , "TICOM and the Analysis of Internal Controls," The Accounting Review (April 1985): PP. 186-201. Biggs, S.F. and T.J. Mock, "An Investigation of Auditor Decision Processes in the Evaluation of Internal Controls and Audit Scope Decisions," Qournel pt Accounting Research (Spring 1983), pp. 234-255. Bolour, A., T. Anderson, L. Dekeyser, H. Wong, "The Role of Time in Information Processing: A Survey," SIQMQD preppp (April 1982), pp. 27-50. Borgida, A., J. Mylopoulos, and H.K.T. Wong, "Generaliza- tion/Specialization as a Basis for Software Specification," in M. L. Brodie, J. Mylopoulos, and J. W. Schmidt (eds) Qa.§eneeptual He_ell_ngi.ferspeetixee from Artifieial Intelligeneel.na_aha§eel and grpgreppipg Languages (Springer-Verlag, 1984), pp. Brachman, R.J., S. Amarel, C. Engelman, R.S. Engelmore, E.A. Feigenbaum, and D.E. Wilkins, "What are Expert Systems?" in F. Hayes-Roth, D.A. Waterman, and D.B. Lenat (eds), Building Expert Systeme (Addison-Wesley, 1983): PP. 31-57. Buchanan, B.G., D. Barstow, R. Bechtel, J. Bennet, W. Clancy, C. Kulikowski, T. Mitchell, and D.A. Waterman, "Constructing an Expert System," in F. Hayes-Roth, D.A. Waterman, and D.B. Lenat (eds), Building Expert Systems (Addison-Wesley, 1983): PP. 127-167. Campbell, M. and H. Berliner, "A Chess Program that Chunks," groceedings pt The patipnal gpnfereppe pp Artifiplel Iptellience (August, 1983): PP. 49-53. Card, S.K., T.P. Moran, and A. Newell, The Esychology pt Human-Computer interaction (Erlbaum Press, 1983). Carr,T.H., "Consciousness in Models of Human Information Processing: Primary Memory Executive Control and Input Regulation," in G. Underwood and R. Stevens (eds), Aspepts pf Qonsciousness yell l; Esychologipel Issues (Academic Press, 1979), pp. 123-153. 224 Cash, J.I., A.D. Bailey, and A.B. Whinston, "The TICOM Model---A Network Data Base Approach to Review and Evaluation of Internal Control Systems," proceedings AFIPS Netipnal computer gonference, 1977, pp. 843-854. Chen, P.P., "The Entity-Relationship Model—-—Toward a Unified View of Data," ACM Transactions pp Database Systems (March 1976), pp. 9-36. ,"The Entity—Relationship Model: A Basis for the Enterprise View of Data," Prpceedings thPS National Qomputer conference, 1977, pp. 77-84. ____, (ed.) Entity-Relationship Approach tp Systems Anelysis and Design (North Holland, 1980). ...... (ed.) W WA :2 I_f_n o_rm__t_i__a on hodeling and Apelysis (North Holland, 1983). Chi, M.T.H., P.J. Feltovich, and R. Glaser, "Categorization and Representation of Physics Problems by Experts and Novices," gpgnitive Science (April-June 1981), pp. 121- 152. Codd, E.F., "A Relational Model of Data for Large Data Banks," communications pt the ppm (June 1980): PP. 377-387. Colantoni, C.S., R.P. Manes, and A.B. Whinston, "A Unified Approach to the Theory of Accounting and Information Systems," The Acepuntipg‘Reyiey (January 1971): PP. Cushing, B.E., "A Mathematical Approach to the Analysis and Design of Internal Control Systems," The pppppptipg Reyiem (January 1974), pp. 24-41. , Accounting information Systems and Business Organirations (Addison—Wesley, 1982). , and J.K. Loebbecke, "Analytical Approaches to Audit Risk: A Survey and Analysis," University of Utah Department of Accounting Working Paper, 1982. Dahl, O.-J. and C.A.R. Hoare, "Hierarchical Program Struc— tures," in O.-J. Dahl, E.W. Dijkstra, and C.A.R. Hoare (eds), Structured Prpgrammipg (Academic Press, 1972), Date, C.J., hp Introductipp tp_Qeteheee Systems (Addison- Wesley, 1982). 225 Davis, R. and D. Lenat, Knowledge-Based Systems ip Artificial Intelligence (McGraw-Hill, 1980). Duda, R.O. and R. Reboh, "AI and Decision Making: The Prospector Experience," Symposium pp Artificiel Intelligence Applications fipr Susiness (New York University, 1983). Dukes, W.F., "N = 1," sychological Bullentin, (1965, 64:1), pp..74-79. Dungan, C.W. "A Model of an Audit Judgment in the Form of an Expert System" unpublished Ph.D. dissertation, Department of Accountancy, University of Illinois at Urbana-Champaign, 1983. , and J.S. Chandler, "Analysis of audit Judgement Through an Expert System," University of Illinois at Urbana working paper #982, October 1983. Einhorn, H. J., "Expert Judgement: Some Necessary Conditions and an Example," Journal pf Applied P_yehplpgy (August 1974), pp. 562- 571. , "Learning from Experience and Suboptimal Rules in Decision Making," in T. S. Wallsten (ed. ), gpgpitiye Erpcesses ip ghoipe anp pecision Aehayipr (Erlbaum , and R.M. Hogarth, "Behavioral Decision Theory: Processes of Judgement and Choice," Apppel Eeyiey pt Pexehelegx (1981). pp- 53-88- Einhorn, H.J., D.N. Kleinmuntz, and B. Kleinmuntz, "Linear Regression and at Process-Tracing Models of Judgement," Psychological Reyiey (September, 1979), pp. 465-485. Ericcson, H.A. and H.A. Simon, "Retrospective Verbal Reports as Data," C.I.P. working paper no. 388 (Carnegie- Mellon University, 1978). , "Verbal Reports as Data," Psychological Reyiey (May 1980), pp. 215-251. Ernst & Whinney, Eyalpetipg Internal gontrpl, 1979. Everest, G.C. and R. Weber, "A Relational Approach to Accounting Models," The Apcountipg Beyiey (April 1977). pp. 340-359. 226 Feigenbaum, E.A., "Themes and Case Studies of Knowledge Engineering," in D. Michie (ed), Expert Systems ip the Micro Electronic Age (Edinburgh University Press, 1979): PP. 3-25. , G. Buchanan, and J. Lederberg, "Generality and Problem Solving: A Case Study Using the DENDRAL," in D. Meltzer and D. Michie (eds), Machine Intelligence S (American Elsevier, 1971): PP. 165—190. Felix, W.L. and J.L. Goodfellow, "Audit Tests for Internal Control Reliance," Symposium pp Auditipg Aesereph ill (University of Illinois at Urbana—Champaign, 1979), PP. 85-105. Fernandez, E.B., R.C. Summers, and C. Wood, hatehase Seeuritx.aud.lntegritx (Addison-Wesley. 1981). Fikes, R.H. and N.J. Nilsson, "STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving," Artificiel intelligence (1971),3/4, pp. 189—208. Gal, G. F. and W. E. McCarthy, "Declarative and Procedural Features of a CODASYL Accounting System," in P. P. Chen (ed): En.it1:3elatienehip Appreaeh (to Information..2deling.a_d.Analx_ie (North-Holland: 1983): pp. 195- 211. , "Specification of Internal Accounting Controls in a Database Environment," gpmpptere end Secprity (March 19853), pp. 23-32. , "Operation of a Relational Accounting System," forthcoming in Apyeppee_ip Acpounting (1985b). . , "Conceptual Representation of Internal Accounting Controls: The Concept of "Event- Scripts," Michigan State University Department of Accounting working paper (1985c). Gaschnig, J., P. Klahr, H. Pople, E. Shortliffe, and A. Terry, "Evaluation of Expert Systems: Issues and Case Studies," in F. Hayes-Roth, D.A. Waterman, and D.B. Lenat (eds), Building Expert Syeteme (Addison-Wesley, 1983): PP. 241-280. Gibbins, M., "PrOpositions about the Psychology of Profes— sional Judgment in Public Accounting," Jpprpel pf Aeeeunting Reeeareh. (Spring 1984): pp. 103-125. 227 Goldstein, I. and S. Papert, "Artificial Intelligence, Language, and the Study of Knowledge," Qognitiye Soieooe (1977: 1): pp. 84- -123- Gorry, G.A. and R.H. Krumland "Artificial Intelligence Reseach and Decision Support Systems," in J.L. Bennett (ed), Building Decision Support Syetems (Addison-Wesley, 1983), pp. 205-219. Grabski, 8., "A Comparison of Perceptual and Technical Skills among Systems Analysts, Internal Auditors, and EDP Auditors," unpublished Ph.D dissertation, Arizona State University, 1983. Hamlen, 5.8., "A Chance-Constrained Mixed Integer Programming Model for Internal Control Design," The Apcounting Beyiey (October 1980), pp. 578-593. Haseman, W.D. and A.B. Whinston, "Design of a Multi- dimensional Accounting System," The Accppntipg.Reyiey (January 1976): PP. 65—79. Hayes-Roth, B. and F. Hayes—Roth, "A Cognitive Model of Planning," gpgnitiye Science (October-December 1979), pp. 275-310. Hunter, I.M.L., "An Exceptional Talent for Calculative Thinking," Sritish Spurnal pt Esypology (August 1962): PP. 243-258. Ingalls, D.B., "The Smalltalk-76 Programming System: Design and Implementation." confereooe.neoore.of the.fif_h Apnual Agh Symppsium pp grpgrammipg Languages (ACM, January 1978). Jarke, M. and Y. Vassiliou, "Coupling Expert Systems with Database Management Systems," ymposium pp Artificial Intelligence applications for Business (New York rd an ‘Events' Theory of Accounting," The Accounting Review (October 1970), pp. 641-653. Joyce, H.J., "Expert Judgement in Audit Program Planning," Journal of Aooouotino.Reeearoh.(Supplement 1976): PP. 29-600 and G.C. Biddle, "Anchoring and Adjustment in Proba- bilistic Inference in Auditing," Qpprpel pt Accpuntipg Aesearch (Spring 1981a), pp.120-l45. Il' 228 , "Are Auditor's Judgments Sufficiently Regressive?" Journal pt Accounting Aesearch (Autumn 1981b): PP.323-349. Joyce, E.J. and R. Libby, "Behavioral Studies of Audit Decision Making," Journel pt Accountipg Literetpre (Spring 1982): PP. 103-123. Kerlinger, F.N. and E.J. Pedhazur, Multiple Regressipn ip Sehavipral Research (Holt, Rinehart, and Whinston, 1973). Kinney,W. and W. Uecker, "Mitigating the Consequences of Anchoring in Auditor Judgments," The Appppptipg Reyiey (January 1982), pp. 55-69. Klix, F., "On the Interrelationships between Natural and Artificial Intelligence Research," in F. Klix (ed): Romeo.and.Artifioiel.lhtellioehoe (North- HOlland' 1979), PP. 1-90. Lachman, R., J.L. Lachman, and E.C. Butterfield, ngpitiye £§12h21292.§HQ Information Prooeeeingi.hh.Introduotion' (Erlbaum, 1979). Lamport, L., "Time, Clocks, and the Ordering of Events in a Distributed System," gommunipetipne pf the AQM (July 1978): PP. 558-565. Larker, D.E. and V.P. Lessig, "An Examination of the Linear and Retrospective Process Tracing Approaches to Judgment Modeling," The Apppupting Reyiey (January 1983): PP. 58-77. Libby, R., "The Use of Simulated Decision Makers in Information Evaluation," he Accounting Aeyiew (July 1975): PP. 475-489. , Accountipg and Human Information Processinge_Theory and Applications (Prentice-Hall, 1981). Loebbecke, J.K., J.F. Mullarkey, and G.R. Zuber, "Auditing in a Computer Environment," The Jpprpel_pt,Acppuptency (January 1983), pp. 68-78. - Loebbecke, J.K. and G.R. Zuber, "Evaluating Internal Control:" The Journal.of Aoooohtahox (February 1980):49—56. Longuet-Higgings, R.C., "Artificial Intelligence - A New Theoretical Psychology?" nghltipp (August-December 1981): PP. 197-200. 229 Lum, V., S. Ghosh, M. Schkolnick, D. Jefferson, S. Su, J. Fry, T. Teorey, and B. Yao, "1978 New Orleans Data Base Design Workshop Report," Research Report RJ2554 (IBM Research Laboratories, San Jose, CA, July 1979). (Abridged version also published in Proceedings pt the Eifth Internetiopal Sonferenpe pp yery Large Deta Seses, 1979, pp. 328-339. Martin, J., Managing the Data-Sass Enyironment (Prentice— Hall, 1983). Mautz, R.R., W.G. Kell, M.W. Maher, A.G. Merten, R.R. Reilly, D.G. Severance, and E.J. White, lpterpel gontrol ip_pTSe Qprporations; The State pt the Art (Financial Executives Reseach Foundation, 1980). Mautz, R.R. and D.L. Mini, "Internal Control Evaluation and Audit Program Modification," The_Apppphtihg.Reyiex (April 1966), pp. 283-291. Mautz, R.R., R.R. Reilly, and M.W. Maher, "Personnel Failure, the Weak Link in Internal Control," Elpenpiel- Erepptiye (December 1979): PP. 22. Mautz, R.R. and H.A. Sharaf, The_2hilpepphy pt Agfiiting (American Accounting Association, 1961). Mautz, R.R. and J. Winjum, griterie for hanegement gpptrpl Syeteme (Financial Executives Research Foundation, 1980). McCarthy, J. and P. Hayes, "Some Philosophical Problems from the Standpoint of Artificial Intelligence," hephipe iptelligeppe 4 (Edinburgh University Press, 1969) McCarthy, W.E. "An Entity-Relationship View of Accounting Models," The Accounting Review (October 1979), pp. 667-686. , "Multidimensional and Disaggregate Accounting Sys- tems: A Review of the ‘Events' Accounting Literature," __§MA Qommonioation (July 1981): pp. 7-13- , "The REA Accounting Model: A Generalized Framework for Accounting Systems in a Shared Data Environment," The Accppptipg Beyiep (July 1982), pp. 554-578. Michaelsen, R.H. "A Knowledge-Based System for Individual Income and Transfer Tax Planning," unpublished Ph.D. dissertation, University of Illinois at Urbana-Champaign, 1982. 230 Miller, G.A., "Trends and Debates in Cognitive Psychology," cognition (August-December 1981), pp. 215-225. Minsky, M., "A Framework for Representing Knowledge," in P.H. Winston (ed), Esycholpgy pt 9pmppter yisipn (McGraw-Hill, 1975): PP. 211-277. Miotto, N.J., "Evaluating Internal Accounting Controls," Management Accounting (July 1980), pp. 15-18. Montgomery, H. and O. Svenson, "On Decision Rules and Information Processing Strategies for choices among Multiattribute Alternatives," Steppipeyiep,Jpprpel pt ESYQholoQY (1976):3: pp. 283-291. National Bureau of Standards, meppter Spiepce §.I§£hn212921 Auditahdhxalhotiohofgohphterw (U-S- Department of Commerce, 1977). , Qomputer Science 5 Technology; A.__udit ..oan ___1___1__Eva uat'on of oompoter Seooritx 111 fixatem yplperabilities epp,gpptrple (U.S. Department of Commerce , 1980). Newell, A., "Judgment and its Representation: An Introduc- tion:" in B. Kleinmuntz (ed): Ethel Representation of human Judgement (John Wiley, 1968), pp. 1-16. ______J "Physical Symbol Systems," gpgpitiyp Scjgngg (April- June 1980), pp. 135-183. _____: "The Knowledge Level:" Artifieial Intellioehoe (January 1982), pp. 87-127. , and H.A. Simon, "GPS, a Program that Simulates Human Thought," in E.A. Feigenbaum and J.A. Feldman (eds), Computers and Thought (McGraw-Hill, 1963). , Human Problem Solving (Prentice- Hall, 1972). , "Computer Science as Empirical Inquiry: Symbols and Search," gpmmunicetions pt_the AQM (March 1976), pp. 113-126. Nisbett, R. and T. Wilson, "Telling More Than We Can Know: Verbal Reports on Mental Processes," Psychological Reyiem (May 1977): PP. 231-259. 231 Nilsson, N.J., "Some Examples of Artificial Intelligence Mechanisms for Coal Seeking, Planning and Reasoning," in F. Klix (ed), Humap end Artificial iptelligepoe (North-Holland, 1979), pp. 11-36. Norman, D.A., "Toward a Theory of Memory and Attention," Psychological Aeyiey (November 1968), pp. 522-536. , "Analysis and Design of Intelligent Systems, in F. Klix (ed): Human and Artificial intelligence (North- Payne J.W., M.L. Braunstein, and J.S. Carroll, "Exploring Predecisional Behavior: An Alternative Approach Alternative Approach to Design Research," Qrgeniration Behauior and Human Performanoe (August 1978) : pP- 17- 44. Peat, Marwick, Mitchell 8 Co., SEAQQQ, 1980. Pople, H.E., "The Formation of Composite Hypothesis in Diagnostic Problem Solving: An Exercise in Synthetic Reasoning:" We of the Fifth International Joint §__er___eonf enc on Artirieial Intellioenoe:1977: pp. 1030- 1037. Pylyshyn, z. W., "The Role of Competence Theories in Cognitive Psychology." Journalot Baxehounouietie Aesearoh (1973),1, pp. 21- 50. , "Psychological Explanations and Knowledge- Dependent Processes," gogpitipp (August-December 1981), pp. 267-274. Quillian, M.R., "Semantic Memory," in M. Minsky (ed), Semantic information processing (MIT Press, 1968), pp. 216-270. , "The Teachable Language Comprehender: A Stimulation Program and Theory of Language," gommunications pt the ACM (August 1969), pp. 459-476. Reckers, P. M. J. and M. E. Taylor, "Consistency in Auditors' Evaluations of Internal Accounting Controls," of Ao_____'_arcount1n Auditing a__nd Einanoe (Fall 1979): pp- 42- 55. Rittenberg, L. E. and D. L. Miner, "Performing Cost/Benefit Analyses of Internal Controls," The lpterpel Auditor (February 1981), pp. 56- 64. 232 Roberts, M.B., "An Investigation of Guidelines for Review and Evaluation of Accounting Controls in Data Base Management Systems," unpublished Ph.D. Dissertation, Georgia State University, 1980. Rowe, N.C., "Top-Down Statistical Estimation on a Database," SiSMQD BTQQAp_(May 1983a) pp. 135-145. , "Rule-based Statistical Calculations on a Database Abstract," unpublished Ph.D dissertation, Department of Computer Science, Stanford University, 1983b. Sakai, H., "A Method for Defining Information Structures and Transactions in Conceptual Schema Design," gropeeoipge oi the Seventh internetional gontererpoe op_yery,herge Data S S: 1981: Pp. 225-234. Scandura, J.M., "Human Problem Solving: A Synthesis of Content, Cognition, and Individual Differences," in F. Klix (ed): mu an ma Artifioial Intelligence (North- Holland, 1979), pp. 59-88. Schank: R. and R. Abelson: Seriotal.21ane1.§oala and Underetanoinu (Erlbaum: 1977): shortliffe: E-H. Computerzuaeed.hedioa1 Qoneultation1.MXQIN (American Elsevier, 1976). and 3.6. Buchanan, "A Model of Inexact Reasoning in Medicine," Methematicel Sioeoiepoee (1975, 23), Pp. 351-379. Simon, H.A., "Cognitive Science: The Newest Science of the Artificial," gpgpitiye Science (January-March 1980), PP. 33-460 Smith, J.M. and D.C. Smith, "Database Abstractions: Aggregation," Communications pt the ACM (June 1977a), pp. 405-413. , "Database Abstractions: Aggrega- tion and Generalization," ATM Transeptiops pp petehase Syeteme (June 1977b), pp. 105-133. Sorter, G.H., "An ‘Events' Approach to Accounting Theory," The Apcouhtipg Reyiey (January 1969), pp. 12-19. Stanford Research Institute, Systems Auditability p Soptrol Study; Data Processing Audit Preptices Report (Institute of Internal Auditors, 1977). 233 Stansfield, J.L. "Conclusions from the Commodity Expert Project," MIT AI memo no. 601, November 1980. Stefik, M., J. Aikins, R. Balzar, J. Benoit, L. Birnbaum, F. Hayes-Roth, and E. Sacerdoti, "Basic Concepts for Building Expert Systems," in F. Hayes-Roth, D.A. Waterman, and D.B. Lenat (eds) Buildipg Expert Systems (Addison-Wesley, 1983), pp. 59-86. Svanks, M.I., "Integrity Analysis: A Methodology for EDP Audit and Data Quality Control," unpublished Ph.D. dissertation, University of Toronto, 1981. Techavichit, J.V., "Evaluation of Internal Control in a Database System Environment," unpublished Ph.D. dissertation, Univeristy of Missouri-Columbia, 1979. Theerachetmongkol, A. and A.Y. Montgomery, "Semantic Integrity Constraints In the Query Example Data Base Management Language." _h_e Australian Lnutero u Journal (February 1980), pp. 28-42. Trotman, K.T., P.W. Yetton, and I.R. Zimmer, "Individual and Group Judgments of Internal Control Systems," Jourpel pt Acpoupting Aeseerch (Spring 1983): pp. 286-292. Tsichritzis, D.C. and F.H. Lochovsky, Data models (Prentice- Hall, 1982). Turing, A.M., "On Computable Numbers, with an Application to the Entscheidungsproblem," roceedings ot_the hopdpp nethematics Society; Series 2, 1936, pp. 230-265. Tversky, A. and D. Kahneman, "Judgment Under Uncertainty: Heuristics and Biases," Science (September 27, 1974), pp. 1124-1131. Van Melle, W., A.C. Scott, J.S. Bennett, and M. Peairs, The EMYCIN Manual (Department of Computer Science, Stanford University, 1981). Vasarhelyi, M.A., "A Taxonomization of Internal Controls and Errors for Audit Research," in D.R. Nichols and H.F. Stettler (eds) Auditing Symposium ye groceedipgS ot the lSSQ Touche Ross University ot_§anses Symposium on Auditing grohlems (University of Kansas, 1980), pp. 41- 58. Walker, A., "Data Bases, Expert Systems, and Prolog," Surnpoeium on Ar_1f.i__.lt' cia _n__111oen_e1 to c Applioatione f_ro Susipess (New York University, 1983). 234 Weber, R., 322 Auditing; Qopceptuel Soundetiops opp greetioe (McGraw-Hill, 1982) Winston, P.H., Artificial Intelligence (Addison—Wesley, 1977). Wong, H.K.T., "Design and Verification of Interactive Information Systems using TAXIS," Technical Report CSRG-129, University of Toronto, April 1981. Young, R.M., "Production Systems for Modelling Human Cognition," in D. Michie (ed) Expert Systems ip the Microelectronic Age (Edinburgh University Press, 1979), pp. 35-45. , and T. O'Shea, "Errors in Children's Subtraction," gognitiye Science (April-June 1981): PP. 153-177. Yu, S. and J. Neter, "A Stochastic Model of Internal Control Systems," Journal or Accopnting Aeeeeroh (Autumn 1973): pp. 273-295. Zloof, M.M., "Security and Integrity within the Query-by- Example Data Base Management Language," IBM Research work paper RC 6982, February 1978. "Al