2-H 1‘ ‘7 545m; { .‘ ‘41. P It... 3. p : ., hen! . 4.33:]...1; . ‘I. u. thbJ...l.‘..:VI\ . .2 - “zap A. 5 . .135. V_l .5: live: xLIEs Iranian-1‘ i. .) n.us.«.iu.n ] :2 974... (L . V S » .‘I...\ {:7 .r.lu I .s ‘ .2 ‘ , .111...) 2.4“. 11., p u a}. J :1)». [J (3:!- #a: b 5:? z -fimm .91 .P.» . .v I .. r . I ,. hunt: . :{II :14 ..:(v...ln .u. $5.! ‘ I . .v . .7: Ltd... .s in) . .l .!\J\. ....\.‘ L i. 31:27. ... 1‘ 231 I11 .\ C. .. .J. JV :3: 1.0;- o. ,I. BIT, II— V E. n ......er2 akidyr? dink“: a . 4 :3; {a {gift}: .A?. V ‘1. nipDD‘rILIi-n .i. ..- £515!er (munnfioh kink nix.» 15 HAHICD llllllllmill1l...\\\ll\l\l ““8 LIBRARY Michigan State University L Al This is to certify that the dissertation entitled THE EFFECTS OF FEEDBACK, FORECAST, AND STANDARD ERROR INFORMATION ON POLICY DECISIONS IN AN UNCERTAIN ENVIRONMENT presented by THOMAS J. CALLAHAN has been accepted towards fulfillment of the requirements for PH. D . BUSINESS ADMINISTRATION degree in Major professor MAY 11, 1990 Date MS U i: an Affirmative Action/Equal Opportunity Institution 0-12771 ___—___ _.__ _ _ ______ PLACE IN REFURN BOX to remove this checkout from your record. TO AVOID FINES return on or before date due. DATE DUE DATE DUE DATE DUE _J I MSU Is An Affirmdtve Action/Equal Opportunity lnetltulon cmmm THE EFFECTS OF FEEDBACK, FORECAST, AND STANDARD ERROR INFORMATION ON POLICY DECISIONS IN AN UNCERTAIN ENVIRONMENT BY THOMAS J. CALLAHAN A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department Of Management 1990 ABSTRACT THE EFFECTS OF FEEDBACK, FORECAST, AND STANDARD ERROR INFORMATION ON POLICY DECISIONS IN AN UNCERTAIN ENVIRONMENT By THOMAS J . CALLAHAN Contrary to popular belief, learning from trial and error experience in decision-making tasks is a slow and difficult process (Brehmer, 1980). Decision makers' ability to predict future events based on their experience with past and similar decisions has been seriously limited by the feedback they receive from the environment regarding the success of their prediction (Brehmer, 1980). This phenomenon is not due to any problems with the feedback itself, but rather is due to the expectations of human decision makers that they live in a deterministic or perfectly predictable world (Brehmer, 1980; Einhorn, 1980). A number of researchers in various areas of decision- making have attempted to overcome decision makers' expectations that there must be a one to one correspondence between events, with very limited success having been reported (Hagafors & Brehmer, 1983; Steinman, 1974). This research represents an attempt to improve subjects' decision-making performance by providing them with decision aids representing predictions from linear models. It was predicted in this study that subjects who were given access to a forecast and standard error before each decision trial and feedback after each decision trial would outperform subjects who had access to less than all three of these factors. Results indicate that providing subjects with either of the decision aids resulted in superior performance in comparison to subjects who received neither of the decision aids on most measures of decision-making performance. The expectation that the standard error factor and the forecast factor together would overcome the previously noted deleterious effects of outcome feedback was not confirmed. It appears that in most cases the effects of both of these factors were redundant. Managerial implications from this study and suggestions for future research conclude this work. Copyright by THOMAS J. CALLAHAN 1990 ACKNOWLEDGEMENTS I would like to express my deep appreciation to Michael Moch who was the guiding force behind this project. I would also like to express my appreciation to Michael Lindell for his expert knowlege of the lens model literature and methodology. John Hollenbeck has been an invaluable member of my committee and I thank him for his pragmatic views on both the methodology and logic of this research. Dan Ilgen's thorough knowledge of the area of feedback has greatly advanced this dissertation process. TABLE OF CONTENTS Page LIST OF TABLES ......................................... ix LIST OF FIGURES ........................................ xii Chapter I: INTRODUCTION Background and Purposes of Research ....... 1 Chapter II: Chapter III: Chapter IV: LITERATURE REVIEW Learning Task Structures Through Trial and Error ..................................... 5 Cue Probability Learning ................ 8 Management Coefficient Theory ............ 19 Subjects' Learning of Decision Task Structures Under a Variety of Feedback and Environmental Conditions: A Summary... 22 RESEARCH OBJECTIVES AND HYPOTHESES Primary Research Objective ............... 26 Hypotheses ........................... 27 Effects Of Shifts in Task Predictability on Decision-making Performance ............................ 35 METHODOLOGY AND DATA ANALYSIS Method.... ............................... 42 Data Analysis ........................ 51 vi Chapter V: Chapter VI: Chapter VII: Chapter VIII: Chapter IX: References Appendices RESULTS MANOVA Analysis .......................... 53 MANOVA Results ....................... 6O ANOVA Results ............................ 69 METHODOLOGY AND SECOND DATA COLLECTION Problems Associated with Data Collection ............................ 103 Method ............................. 105 RESULTS MANOVA Analysis ......................... 109 ANOVA Analysis ...................... 110 SUMMARY AND DISCUSSION OF RESULTS Effects of Decision Aids on Decision-making Performance.. ......................... 139 Limitations of This Research .......... 149 MANAGERIAL IMPLICATIONS AND SUGGESTIONS FOR FUTURE RESEARCH Managerial Implications ................. 152 Suggestions for Future Research ....... 156 ............. ........................... 161 Appendix A ............................. 173 Appendix B ............................. 176 Appendix C ............ _ ................. 180 vii Appendix D ............................. 181 Appendix E ............................. 197 Appendix F ............................. 205 Appendix G ............................. 208 Appendix H ............................. 209 Appendix I ............................. 213 Appendix J ............................. 214 Appendix K ............................. 221 viii Table Table Table Table Table Table Table Table Table Table Table Table Table 10 ll 12 13 LIST OE TABLES Hypothesized Effects of Feedback, Forecast and Standard Error Factors .......... . ....................... 39 Summary Statement of Hypotheses .......... 40 Example of 5x5 Matrix for Developing Combinations of Cue and Criterion for the Post Experimental‘Task ........................ 46 Comparison of Significant Results for the Pillai-Bartlett Trace ............ 57 Effects of the Feedback Factor on Linearly Combined Dependent Variables ................................ 62 Effects of Standard Error Factor on Linearly Combined Variables ........... 63 Effects of the Forecast Factor on the Linearly Combined Variables .......... 64 Effects of the Forecast and Standard Error Interaction on the Linearly Combined Variables .............. 66 Effects of the Feedback and Standard Error Factors on the Linearly Combined Variables .............. 67 Effects of the feedback and Forecast Factors on the Linearly Combined Variables ....................... 68 Effects of feedback and shift on Linearly Combined Variables ............. 7O Testable Hypotheses By Univariate ANOVAs Based on Preliminary MANOVA Analysis ......................... 71 Achievement-ANOVA Results ................ 78 ix Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table Table 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 Summary of Hypothesized Results of the Achievement Measure ............... 79 Consistency-ANOVA Results ................ 84 Summary of Hypothesized Results for the Consistency Measure .............. 85 Matching-ANOVA Results ................... 91 Summary of Hypothesized Results for the Matching Measure ................. 92 Accuracy-ANOVA Results ................... 97 Summary of Hypothesized Results for the Accuracy Measure ................. 98 Effect of Group Membership on Card Bin ................................ 101 Effects of the Forecast Factor on Linearly Combined Variables ............. 111 Effects of the Forecast By Standard Error Interaction on Linearly Combined Variables ............. 112 Testable Hypotheses Using Univariate ANOVA Based on Preliminary MANOVA Analysis ............. 113 Achievement-ANOVA Results ............... 116 Summary of Hypothesized Results for the Achievement Measure ............ 120 Consistency-ANOVA Results ............... 122 Summary of Hypothesized Results for the Consistency Measure ............. 124 Matching-ANOVA Results .................. 129 Summary of Hypothesized Results for the Matching Measure ................ 131 Accuracy-ANOVA Results ................. 134 Summary of Hypothesized Results for the Accuracy Measure ................ 136 Means and Standard Deviations for Group Knowledge Task...,. ............... 138 X Table Table Table Table Table Table Table Table Table Table 34 35 36 37 38 39 4O 41 42 43 Achievement-Means and Standard Deviations After Z-Score Transformation ................. Consistency-Means and Standard Deviations After Z-Score Transformation ................. Matching-Means and Standard Deviations After Z-Score Transformation ................. Accuracy-Means and Standard Deviations After Z-Score Transformation ......... ........ Correlation Matrix For Dependent Variables For Data Collection I Achievement-Means and Standard Deviations After Z-Score Transformation ......... .. ...... Consistency-Means and Standard Deviations After Z-Score Transformation ....... .......... Matching—Means and Standard Deviations After Z-Score Transformation ........ . ........ Accuracy-Means and Standard Deviations After Z-Score Transformation ............ ..... Correlation Matrix For Dependent variables For Data Collection II . ................. xi ........ 176 ........ 209 ........ 210 ........ 211 ........ 212 ........ 213 FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE 10 ll 12 13 LIST OF FIGURES Achievement - Forecast By Standard Error Interaction ..................................... 75 Achievement - Feedback By Shift Interaction ..... 76 Consistency - Forecast By Standard Error Interaction........ ..... . ....................... 81 Consistency - Feedback By Shift Interaction ..... 82 Matching - Forecast By Standard Error Interaction ..................................... 88 Matching - Feedback By Shift Interaction ........ 89 Accuracy - Group Performance For All Combinations Of Feedback, Forecast and Standard Error......... ...... ..... .............. 94 Accuracy - Feedback By Shift Interaction ........ 96 Achievement - Forecast By Standard Error Interaction.......... ........ . .................. 118 Consistency - Forecast By Standard Error Interaction ..................................... 119 Matching - Forecast By Standard Error Interaction.......... ........................... 127 Accuracy - Forecast By Standard Error Interaction ..................................... 128 Achievement - Block 1 ...... . .................... 181 xii FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 3O Achievement - Block 2 .......................... 182 Achievement - Block 3 .......................... 183 Achievement - Block 4 .......................... 184 Consistency - Block 1 .......................... 185 Consistency - Block 2 .......................... 186 Consistency - Block 3 .......................... 187 Consistency — Block 4 .......................... 188 Matching - Block 1 ............................. 189 Matching - Block 2 ............................. 190 Matching - Block 3 ....................... . ...... 191 Matching - Block 4 ............................. 192 Accuracy - Block 1 ............................. 193 Accuracy - Block 2 ............................. 194 Accuracy - Block 3 ............................. 195 Accuracy - Block 4 ............................. 196 Achievement - Forecast By Standard Error Interaction ..................................... 197 Achievement - Feedback By Shift Interaction ..... 198 xiii FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE 31 32 33 34 35 36 37 38 39 4O 41 42 43 44 45 46 47 Consistency - Forecast By Standard Error Interaction ..................................... 199 Consistency - Feedback By Shift Interaction ..... 200 Matching - Forecast By Standard Error Interaction ..................................... 201 Matching - Feedback By Shift Interaction ........ 202 Accuracy - Group Performance For All Combinations Of Feedback, Forecast and Standard Error .................................. 203 Accuracy - Feedback By Shift Interaction ........ 204 Achievement - Block 1 .......................... 214 Achievement - Block 2 .......................... 215 Achievement - Block 3 .......................... 216 Achievement - Block 4 .......................... 217 Consistency - Block 1 .......................... 218 Consistency - Block 2 .......................... 219 Consistency - Block 3 .......................... 220 Consistency - Block 4 .......................... 221 Matching - Block 1 ............................. 222 Matching - Block 2 ............................. 223 Matching - Block 3 ............................. 224 xiv FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE FIGURE 48 49 50 51 52 53 54 232 FIGURE FIGURE 55 56 Matching - Block 4 ............................. 225 Accuracy - Block 1 ............................. 226 Accuracy - Block 2 ............................. 227 Accuracy - Block 3 ............................. 228 Accuracy - Block 4 ............................. 229 Achievement - Forecast By Standard Error Interaction ..................................... 230 Consistency - Forecast By Standard Error Interaction ..................................... 231 Matching - Forecast By Standard Error Interaction ..................................... 233 Accuracy - Forecast By Standard Error Interaction ..................................... CHAPTER I INTRODUCTION Wank; Learning_Ernm_Exnerience At the core of most theories of learning is the assumption that human beings learn from experience (Bandura, 1977; Freud, 1959; Skinner, 1977; Thorndike, 1971). Somewhat surprisingly, the proposition that human beings effectively learn from experience in decision-making situations has been questioned by a large body of empirical evidence (e.g. Abelson & Levi, 1985; Brehmer, 1973, 1978, 1980; Einhorn, 1980; Slovic, Fischoff, & Lichtenstein, 1977). Although it appears that during very early periods in the human being's life, learning in decision-making situations is primarily a result of trial and error experience (Piaget & Inhelder, 1974), there is little evidence that this type of inductive learning guides individual's choice behaviors through much of their lifetimes (Einhorn, 1980). Rather, it appears that early inductive learning about action-outcome sequences (i.e., a specific action leads to a specific consequence) creates a series of templates or schemata (Einhorn, 1980; Tversky & Kahneman, 1974) which serve as frameworks for interpreting all new trial and error experiences. The advantages from the utilization of these schemata are directly related to a reduction of information processing requirements for the decision maker (Einhorn, 1980). Additionally, it appears that schemata based on early action-outcome sequences guide most interactions with the environment (Brehmer, 1980). E | . . l' M E 1 1.]. I' E . . E . | On the other hand, there is much evidence that causal schemata may be based primarily on deterministic trial and error experience, in which a specific action leads directly to a specific outcome (Brehmer, 1980; Einhorn, 1980). In probabilistic environments in which a specific action figmetimes leads to a specific outcome, decision-making based on deterministic schemata often results in decision error (Einhorn, 1980). This decision-making error has great costs for individuals in their everyday lives (Einhorn, 1980; Shugan, 1980), and researchers have attempted to determine in which situations and under what conditions these errors occur (e.g., Bowman, 1963; Brehmer, 1971, 1973, 1978; Wason, 1960, 1968). It appears that decision-makers make the greatest number of errors and most casnlx errors in environments in which there is a substantial amount of uncertainty, that is, a non-deterministic decision environment where the probability that a specific action leads to a specific outcome is much less than one (Brehmer, 1980). E J E E I] l . E . . _ 1' Optimal decision-making performance in non-deterministic settings requires some understanding of Bayesian probability, but this understanding is not innate in human judges and learning probability from experience with the environment may be severely constrained by cognitive limitations (Einhorn, 1980). Due to these latter facts, decision makers Often ignore feedback which contradicts or disconfirms their predetermined notion that the underlying rule for solving a decision task is a deterministic one. Instead they rely almost exclusively on evidence that confirms these deterministic decision rules (Tversky & Kahneman, 1974). Since attending to feedback which disconfirms a deterministic rule is assumed to be the primary method through which decision makers can inductively learn a probabilistic decision task, judges' behavioral responses to both confirming and disconfirming feedback have been an important topic for researchers attempting to improve decision-making performance (eg. Bowman, 1963; Brehmer, 1971, 1973, 1978; Wason, 1960, 1968). This empirical research has been directed along a number of lines. First, researchers (Mynatt, Doherty, & Tweney 1977, 1978; Wason, 1960, 1968; Wason & Johnson-Laird, 1972) have directly attempted to induce subjects to utilize disconfirmatory feedback rather than confirmatory feedback in order to learn the underlying structure of a decision task. A second line of research (Hagafors & Brehmer, 1983; Hammond, Summers, & Dean, 1973; Steinman, 1974) has utilized Brunswik's (1952, 1955, 1956) lens model to measure subjects' understanding of and performance in probabilistic decision tasks. A third line of research has focused on assessing the effects of feedback on bias and consistency in human judges' decision-making performance (Carter, Remus, and Jenicke, 1978; Ebert, 1972; Remus, 1978; Remus, Carter, and Jenicke, 1984; Remus, Carter, and Jenicke, 1979). The primary goal of this discussion is to propose a study of decision—making performance in which a variety of techniques directed toward guiding decision makers in the learning of appropriate decision strategies in probabilistic decision tasks can be tested and compared. Before accomplishing this purpose, it is necessary to discuss and integrate the specific findings from the above three lines of research. CHAPTER II LITERATURE REVIEW W The first line of research mentioned above follows a method developed by Wason and his colleagues (Wason, 1960, 1968; Wason & Johnson-Laird, 1972) and has been directed toward establishing how subjects inductively discover problem task structures. In this series of studies, the task for subjects was to determine a general rule for a short series of numbers by using a trial and error process of inquiry. A simple set of numbers, such as 2-4-6, was presented to the subjects who were then asked to discover the rule through the process of trial and error. Subjects were allowed to test as many triplets of numbers as they wanted and were given feedback regarding whether those triplets conformed to the general rule. Because the general rule for the 2-4—6 series of studies was "Any three numbers in ascending order," subjects who gave responses such as "Even numbers separated by two," would be given feedback about the incorrectness of their conjecture about the general rule. The sessions ended when the subjects discovered and announced the correct' general rule. This series of studies indicates that subjects began with what they considered to be an "obviously true" hypothesis, relied on only confirmatory evidence, and 5 expressed surprise at their error in attempting to determine the correct rule. After failure, the subjects generated new hypotheses, with 70% of Wason's (1960) subjects eventually discovering the correct rule on later trials. The most interesting finding from the Wason (1960) study was that those subjects who correctly discovered the rule on their first guess were more likely to have used disconfirmatory strategies than subjects who failed to solve the problem or solved it after one or more incorrect guesses. Regardless of this specific finding, in both the Wason (1960) and Wason (1968) studies, the majority of the subjects did not attempt to eliminate hypotheses nor seek disconfirmatory evidence when attempting to find the problem solution. Studies using more complex tasks than the 2-4-6 problem have generally found that strategies aimed at inducing subjects to use disconfirmatory evidence search have not resulted in more successful completion of a problem task. For example, Mynatt, Doherty, and Tweney (1977; 1978) used brief instructions regarding the nature of the task in an attempt to induce a confirmatory strategy in some subjects and a disconfirmatory strategy in others, but this manipulation produced little effect as evidenced by the fact 'that subjects failed to seek disconfirmation of hypotheses or test alternative hypotheses in either of the experimental conditions. The strongest evidence for inductive learning in this line of research has been provided by Tweney, Doherty, Worner, Pliske, Mynatt, Gross, & Arkkelin, (1980). In a series of studies, these researchers attempted to induce subjects to disconfirm their working hypotheses for the 2-4-6 game. In one study, subjects received information that was designed to overcome confirmatory biases. In a second study, subjects were allowed to develop strongly held hypotheses and then were introduced to instructions informing them of disconfirming strategies. In both of these studies, the provision of these instructions led to no increase in the efficiency of rule discovery. In a third experiment, subjects were taught to use multiple hypotheses at each step- a strong inference approach (Platt, 1964), but performance after this treatment actually worsened. Finally, in the fourth experiment, Tweney et al. (1980) semantically redefined the feedback given to subjects regarding the correctness of their responses. Subjects were told that the 2-4-6 game had two general rules for generating the triplets of numbers. One of the two rules was termed DAX and was the correct ascending order rule. The other rule was termed MED, and included all incorrect rules, although the subjects were led to believe that it was an alternate and correct rule. This problem structure, directed toward the discovery of two general rules, resulted in dramatic improvement in subjects' performanCe in comparison to subjects' performance in the preceding conditions. Sixty percent of the subjects solved the task on their first announcement of a rule. Tweney et al. (1980) referred to Chamberlin's (1897) model of a logical tree in order to explain the mechanisms of subjects' inference processes. They hypothesized that subjects, given the information that two hypotheses existed, were able to interrelate them in such a way that they could recognize the scope of each. In this situation, negative feedback could be classified as meaningful information, rather than something to be ignored. Background A second line of research directed at assessing the ability of subjects to learn in problem solving situations, is termed cue probability learning (CPL). Slovic and Lichtenstein (1971) describe cue probability learning research as an ecologically valid methodology in which the subject must integrate information from several sources. The lens model (Brunswik, 1952) has provided conceptual and analytical guidance for most of these studies. Typically in this type of research study, the subject has been presented with a set of cues, asked to make a quantitative judgment on the basis of these cues about some criterion, and then has been informed of the actual criterion value. For example, in this type of study a subject might be asked to predict a number of students college grade point averages from a number of predictors (e.g. high school G.P.A., S.A.T. scores, A.C.T. scores). After each prediction, subjects are shown the actual college grade point average for the previous trial. Cue probability learning research has primarily focused on how subjects utilize information from trial and error experience in a probabilistic decision environment. A number of manipulations have been hypothesized to improve decision makers' performance in these types of environments, and the more relevant ones are reviewed below. WW As discussed previously, in order for subjects to exhibit inductive learning in a task situation, learning must have been based on feedback from their previous performance. In the cue probability learning literature, this type of feedback is termed outcome feedback. The fact that the subject must learn information about the structure of the task from outcome feedback has necessarily required that the subject have no prior knowledge about the task or its structure (Sniezek, 1986). The operational assurance of this condition has been to identify the cues and task itself with abstract or meaningless labels (Sniezek, 1986). The efficacy of outcome feedback at improving decision-making achievement in most of these studies has been, at best, small (Hammond, 1971; Hammond & Summers, 1972; Todd and Hammond, 1965; Lindell, 1976). At worst, some studies have shown outcome feedback to be detrimental to learning in cue probability learning tasks (Hagafors & Brehmer, 1983; Hammond, Summers, & 10 Dean, 1973; Steinman, 1974). Attempts at manipulating cues to aid learning have also not been very successful. For example, in one study, Hoffman, Earle, & Slovic, (1981) arranged the cue-criterion pairs in logical sequences, so that relational patterns of feedback would be evident to subjects, but this treatment resulted in lower achievement on the part of subjects than cue-criterion pairs arranged in random sequence. Schmitt, Coyle, & King, (1976) have explained the detrimental effects of outcome feedback on subjects' performance as being the result of decreases in subjects' linear consistency. Linear consistency in this case refers to how well a subject consistently assigns the same weights to each of the cues on successive trials. It appears that feedback from previous trials confuse the subjects and cause them to change their strategies after each trial. In a probabilistic task, such as those found in cue probability learning studies, the actual value given as feedback contains error that has been a function of the overall uncertainty (random error) in the decision task. Failure on the part of the subject to recognize the less than perfect predictability of the system indicated by feedback values, has reduced the subject's consistency and overall decision performance (Hagafors & Brehmer, 1983). The one study in which outcome feedback appears to have been effective for learning with abstract cues in a probabilistic task environment has been reported by Peterson 11 & Pitz, (1985). These authors found an increase in subjects' knowledge about the proportionality of one weights after requiring them to keep the same one weights for periods of five consecutive trials. Peterson & Pitz (1985) attributed this improvement to subjects' utilization of outcome feedback alone. Given the fairly negative results regarding the efficacy of outcome feedback in lens model research, attempts have been made to indicate the probabilistic structure of the cue- criterion relationship by providing subjects with what has been termed feedfgruazd (Bjorkman, 1972) or more commonly, task information (Schmitt et al., 1976). Task information has been provided to the subject in the form of a simple zero-order correlation between each cue and the criterion, but beta weights or beta weights times correlations have been used in a number of studies (Adelman, Stewart, & Hammond, 1975; Balke, Hammond, & Meyer, 1973; Schmitt et al., 1976). Schmitt et al. (1976) and Lindell (1976) have reported improvements in subjects' knowledge of the relative proportionality of cue weights in the presence of task information. Lindell (1976) has indicated that these improvements have been most evident in complex decisions tasks. On the other hand, Schmitt, Coyle, & Saari (1977) have found that subjects receiving no forms of feedback performed better than those receiving correlational or beta 12 weight information about the task at least on some measures of decision-making performance. In what may be the most relevant study of the influence of task information on cue probability learning, Sniezek and Reeves (1986) have tested the effects of what they term a feature cue on subject learning in a high certainty environment (multiple R = .89). Before each trial, the subject was presented with the feature cue, which represented both the least-squares and maximum likelihood prediction of the criterion for that cue. Sniezek & Reeves (1986) reported that subjects partially used the feature cue as a decision aid, and this utilization resulted in both improved subject decision consistency and task knowledge. Sniezek & Reeves (1986) were not able to explain the reasons that none of the subjects simply gave the feature one as their response, since this would have been the easiest response and would have optimized their performance on all lens model measures. They did however, hypothesize that the error inherent in the least squares optimal criterion led subjects away from its use as a response. Hoffman, Earle, & Slovic (1981) have applied computer graphics (Hammond, 1971) to the study of the influence of task information and outcome feedback on learning in cue probability tasks. In their study, subjects received visual representations of both task information and outcome feedback after each block of twenty-five trials. This combination produced what the authors describe as superior learning in a 13 complex (nonlinear) task. Hoffman et al. (1981) explained the superior performance of subjects with access to a computer graph of relationships in the decision task and outcome feedback in comparison to subjects given numerical task information and numerical outcome feedback as being attributable to the visual presentation of this information. According to these authors, the important element in this study was that task information and outcome feedback were designed as perceptual rather than cognitive stimuli. HOffman et al. (1981) hypothesized that in their study and previous studies using computer graphics (Hammond, 1971; 1981) "many tasks that appear to require symbolic cognitive activity and a rational approach can be transformed to perceptual-motor analogues, whereby learning will prove to be more efficient" (p. 78). Hammond (1971), the first study in which computer graphics were used in lens model research, has explained the superior performance of subjects provided graphs as decision aids in a similar but simpler manner, i.e., subjects receiving visual information about their own judgment rules and the rules implied by the relationships in the task have learned more easily. Wes Sl'I'I . Olgilvie and Schmitt (1979) have suggested that in order to improve learning in cue probability decision tasks, the cues and criterion should be labeled in a manner that is meaningful to the subject. In other words, at least in most 14 adult learning situations, some elements of the structure of the decision-making task must have already been known to the subject from previous experience with relationships between cues and criterion in similar situations in order for measurable learning to occur. For example, in the Koele (1980) study, the cue probability learning task for subjects was to predict a test achievement score from two cues, an intelligence index and a motivation index. The implicit assumption in this study was that subjects already had some knowledge of the general association between these, or similar cues and the criterion. Experiments such as this have relied on subjects' previous knowledge about similar and previously experienced decision tasks to direct and encourage induetixe learning. Much evidence in the cue probability literature that has shown meaningful variable labels which agree with the statistical nature of the task have improved subject performance, in particular subject consistency (Adelman, 1981; Koele, 1980; Miller, 1971; Muchinsky & Dudycha, 1975; Sniezek, 1986). Somewhat surprisingly, there has also been some evidence that meaningfully labeled variables lead to superior performance even if the statistical structure (sign or nonlinearity) of the task has been contradicted by the labels (Brehmer and Kuylenstierna, 1980; Muchinsky and Dudycha, 1975; Sniezek, 1986). Explanations for the latter finding, that incongruent labels can lead to improved subjects' performance, again have 15 assumed that variable labels direct subject attention to the statistical structure of the task, but in cases in which labels have been incongruent, the subject has been able to recognize the unexpected and opposite relationship and change strategies accordingly (Sniezek, 1986). Subjects' consistency has been enhanced by both congruent and incongruent labels, since the subject, having some prior knowledge about the structure of the task, has changed strategies less frequently (Adelman, 1981; Sniezek, 1986). Indications that subjects' teak kndwiedge or understanding of the proportionality of the cue weights, improved in the presence of meaningful labels were reported in the Sniezek (1986) study, although not in the Adelman (1981) study. Positive as opposed to negative and linear as opposed to nonlinear cue-criterion relationships have appeared to be more easily learned by subjects based on the data from a number of studies (Adelman, 1981; Brehmer, 1973, 1976; Brehmer & Kuylenstierna, 1978; Brehmer and Lindberg, 1970; Gray, 1979; Naylor & Clark, 1968; Slovic, 1974; Sniezek, 1986; Sniezek & Naylor, 1978). These findings have generally been explained in terms of Naylor and Clark's (1968) observation that subjects approach most tasks with an expectation that the cue-criterion relationship will be both positive and linear. A number of studies, (Brehmer, 1971; Gray, 1979; Naylor & Clark, 1968) have reported that negative 16 or curvilinear cue criterion relationships result in a slower rate of learning, but given a sufficient number of trials, subjects have reached equivalent levels of performance on positive and negative tasks and on linear and simple curvilinear tasks. EffectsJLInstructinn Closely related to studies which have attempted to improve subject performance through the provision of task information or variable labels, have been those studies which have attempted to improve lens model performance measures by increasing subject awareness of the statistical properties of the task through verbal instructions. Brehmer and his colleagues (Brehmer & Kuylenstierna, 1978, 1980; Johnasson & Brehmer, 1979; Kuylenstierna & Brehmer, 1980) have attempted to induce consistency in subjects by informing them of the general probabilistic nature of the task, the type of strategy that should be used, and the level of error that might be expected in the task. The results from this series of experiments have indicated that these instructions did net lead to a change toward a more consistent strategy on the part of the subjects, although overall achievement did improve due to subjects' ability to better estimate the correct proportionality of the weights of each of the variables. Brehmer and Kuylenstierna (1980) have explained these less than optimistic findings as being the result of cognitive processing limitations that prevent subjects from 17 using a statistically based approach. The nature of these limitations has not yet been empirically demonstrated, with Brehmer and Kuylenstierna (1980) hypothesizing that either subjects' inability to apply a known strategy or failure to find an appropriate strategy has caused this less than optimal decision-making performance. Using instructions that were specific to the sign and function form of the task, Sniezek, Dudycha, & Schmitt (1978) reported that instructions and high levels of mathematical sophistication led to faster learning rates and higher performance when the task required learning negative cue- criterion relationships. Statistical sophistication by subjects has been shown to be positively related to one probability learning performance by Miller (1971) and Brehmer & Lindberg (1970), while Hammond and Summers (1965) and Rothstein (1986) have shown that specific instructions facilitate subject learning in curvilinear cue-criterion relationships. BEE I E E . | J E l' I l'l'l S 1. | , I . The importance of cue-criterion validities, or system predictability (the multiple R) on subject learning has been shown in a number of cue probability learning studies (Brehmer, 1976, 1978; Gray, 1979; Hagafors & Brehmer, 1983; Brehmer and Kuylenstierna, 1978; Lindberg & Brehmer, 1976; Naylor & Clark, 1968; Sniezek, 1986; Schmitt et al., 1976). As might be expected, the evidence has indicated that 18 subjects were best able to learn the cue-criterion relationships if the system predictability was high. Possibly the most important issue in this area has been related to the tendency for subjects to change their linear consistency as a monotonic function of environmental predictability (Naylor & Schenk, 1968). It has appeared that subjects have displayed higher levels of consistency in very predictable environments and lower levels of consistency in environments where predictability has been low (Brehmer, 1976; Naylor and Schenk, 1968). As Brehmer (1976) has pointed out, in order to optimize in lens model tasks, linear consistency should be unity, regardless of the uncertainty perceived in the task. Empirical evidence (Brehmer, 1973, 1976, 1980; Brehmer & Kuylenstierna, 1980; Gray, 1979; Johnasson & Brehmer, 1979) has indicated that subjects tend to match their consistency to the uncertainty in the environment, overmatching (the subject's linear consistency is greater than the multiple R of the decision task) when environmental predictability has been high and undermatching (the subject's linear consistency is less than the multiple R of the decision task) when environmental predictability has been low. In environments of low certainty, feedback values contain more error. These "noisy" feedback values lead subjects to change strategies frequently, which has resulted in low consistency values. In environments in which environmental certainty has been high, 19 feedback values contain less error and subjects change strategies less frequently (Brehmer, 1978). Background The importance of linear consistency in decision-making tasks has been a central area in the above lines of decision- making research. Therefore, it is not surprising that the influence of linear consistency on decision-making performance has been the central concept in what has come to be known as Bowman's (1963) management coefficient theory. Bowman (1963) has stated that although managers are biased in their judgments, these biases (systematic errors) are less harmful than inconsistencies (variance) in determining judgmental performance. Bowman (1963) further stated that an algorithm computed from an average of all the manager's past decisions on a task should perform better than the manager on the same task, almost entirely due to the greater consistency in the application of the manager's eyetege decision rule. All of these predictions have been made with the assumption that the manager had some previous knowledge of relationships among the decision-making variables (Bowman, 1963; Kunreuther, 1969). 20 WW 3 EE" E 'I' A number of researchers have looked at the efficacy of linear models of human judgment which outperform the human judges from whose responses they were developed. Goldberg (1970), Dawes (1971), Dawes & Corrigan (1974) have indicated that these so-called bootstrapping—models often outperform a human judge if the judge's responses have any validity, since filtering out the error by applying the stable model weights will increase accuracy. In other words, linear models derived through bootstrapping distill underlying policy from error prone and variable human judgments (Abelson & Levi, 1985). A substantial amount of research on the management coefficient theory has expanded and refined this concept. For example, Kunreuther (1969) found support for the most of Bowman's (1963) propositions in his study of a manufacturing organization, although he noted that if forecasts were available to managers and were accurate representations of future events, then the inconsistency associated with the manager's decision should be beneficial to task performance as long as those inconsistencies are directed toward congruence with the decision rule implied by the forecast. A later series of studies (Carter, Remus, and Jenicke, 1978; Ebert, 1972; Remus, Carter, and Jenicke, 1984; Remus, Carter, and Jenicke, 1979) used computer generated production and marketing games to assess learning in managerial decision- 21 making within Bowman's (1963) research framework. These studies supported the propositions that decision makers do learn inductively. Furthermore, linear consistency in decision-making has been an important determinant of both learning and performance. Similarly, MOskowitz and Miller (1975) studied the effects of differing levels of error in forecasts and differing lengths of forecast periods using graduate student subjects in a cost minimization decision task. Results indicated that the dependent variable, average costs, decreased with lengthened forecast horizon (up to three months) and increased as the forecast became less accurate. Variance in managerial decision-making behaviors tended to increase costs as the forecasts became less accurate and forecast horizons increased (beyond three months), as would be predicted from Bowman's (1963) propositions. Finally, Hogarth and Makridakis (1981) assessed the performance of competing teams in a marketing simulation game, in which performance results were based on competing teams inputs as well as a number of complex algorithms. In this study, teams of subjects competed against each other and against two "teams" that were artificially constructed by the experimenters. These artificially constructed "teams" were really decision rules, the first being a consistently applied decision rule and the second being a randomly applied decision rule. The influence of consistency on decision- 22 inaking was again demonstrated, since the consistent artificial "team" outperformed 41% of the human teams. The above studies on the management coefficient theory imply that subjects were inductively learning decision rules over time, although like most of the studies in the previously discussed cue probability learning area, the variables were meaningfully labeled and the subjects, business students, were presumed to have had some knowledge of the task structure beforehand. S 1 . I ' I . E E . . I 1 SI I It is useful at this time to identify and summarize studies from both the psychological and management decision- making literature that reported subjects' learning of the structure of a decision-making task through ante induction, that is, solely on the basis of outcome feedback from their own performance. From the psychological concept formation literature, the Tweney et al. (1980) study reported inductive learning in the 2:4:fi game after subjects were led to believe that there were two correct, but unknown rules for the task. Tweney et al. (1980) explained this learning as being attributable to the ability of subjects to see and interrelate two clearly testable hypotheses. They further believe that the existence of these two hypotheses made 23 disconfirmatory feedback meaningful, rather than something to be ignored. The other study that reported significant inductive learning in the presence of outcome feedback alone was that of Peterson and Pitz (1985), who required subjects to remain consistent long enough to test their hypotheses. The Sniezek (1986) study, where nenttal labels identified the variables, seems close to, but not exactly within, this narrow definition of inductive learning. A larger number of studies in the cue probability learning area discussed above (eg. Adelman, 1981; Koele, 1980; Miller, 1971; Muchinsky & Dudycha, 1975; Sniezek, 1986) imply that inductive learning was occurring within a previously learned framework; (i.e., one that was implied by labels, training, etc.) The most optimistic studies, (those that assumed learning from feedback) have been those associated with the study of the management coefficient theory (Carter et al., 1978; Ebert, 1972; Hogarth & Makridakis, 1981; Remus et al., 1984; Remus et al., 1979). A common element to this group of studies was that forecasts were provided to the subjects as decision aids. Although with the exception of the Hogarth and Makridakis (1981) study, the effects of these forecasts were not the primary focus of these studies, it is conjectured here that the provision of forecasts to subjects in these studies may have increased inductive learning in at least two ways. First, the forecast provided information to 24 subjects about the relationships in the task, because it was a statistical estimate calculated from the relationships in that decision task. In line with this reasoning, Sniezek and Reeves' (1986) feature cue was actually a statistical forecast for the upcoming cue probability trial. Although subjects did not use the numerical value of the "feature cue" as their response (which would have represented the optimal statistical strategy), Sniezek and Reeves (1986) reported that subjects appeared to have adjusted their responses in the direction of the feature cue's numerical value when making their decisions. This behavior resulted in both increased task knowledge and consistency. Second, it is conjectured that the provision of forecasts to subjects may have had an effect similar to the Tweney et al. (1980) strategy of leading subjects to believe that another testable hypothesis existed, the manipulation in the most successful of the '2-4—6' studies. In this instance, subjects who were given forecasts as decision aids, may have been made aware of two competing hypotheses, their own hypothesis about task variables and the more statistically accurate one implied by the forecast. In this case, feedback values that disconfirm the subject's own hypothesis may be seen as evidence that confirms the hypothesis represented by the forecast. Regardless of the possible explanations as to why forecasts may improve learning in decision tasks, there seems to be little doubt that they do facilitate learning of task structures (Carter 25 et al., 1978; Ebert, 1972; Moskowitz and Miller 1975; Remus et al., 1984; Remus et al., 1979). CHAPTER III RESEARCH OBJECTIVES AND HYPOTHESES E . E 1 :1. |° The primary research objective in this study is to assess the effects on subjects' performance of providing them with decision aids directly derived from the statistical relationships between the predicted cues and outcome criteria of a decision-making task. More specifically, before each trial, some subjects will be provided with the forecast and/or the standard error derived from the least squares regression equation which statistically describes the relationships between the predictive cues and actual outcome value on that specific simulation trial. The most general proposition of this study states that under conditions in which subjects are informed that the forecast and/or standard error are based on predictions from the least-squares regression equation describing the relationships between the cues and actual outcomes, subjects provided with these decision aids will outperform subjects who are not provided with these decision aids. Specific hypotheses and the rationales underlying each of the hypotheses are discussed more fully below. 26 27 Hypotheses makinLEerfnrmance Much evidence indicates that outcome feedback alone is not useful and may even be deleterious to learning and performance in decision-making tasks (Hagafors & Brehmer, 1983; Schmitt et al., 1976; Steinman, 1974; Tweney et al., 1980; Wason, 1960, 1968; Wason & Johnson-Laird, 1972). As discussed previously, Schmitt et al. (1976) have attributed subjects' learning problems in the presence of outcome feedback to the inconsistencies in subjects' application of decision strategies which may be caused by the confusing aspects of feedback values containing random error. Furthermore, Brehmer (1980) has hypothesized the existence of a hierarchy of decision strategies, in which a positive linear decision strategy is the most commonly elicited from human decision makers when faced with most types of decision tasks. Because it has been shown that most decision tasks can be at least adequately approximated by a positive linear decision strategy (Dawes & Corrigan, 1974), this type of decision strategy may represent a "best-bet" response for human decision makers in any type of decision task (Brehmer, 1980; Einhorn, 1980). In light of these statements, subjects given outcome feedback which contains random error will be confused by that outcome feedback and change strategies frequently. On the other hand, subjects who are not given outcome feedback will be more likely to remain with their 28 'first-choice' strategy (postive and linear) throughout a series of decision trials. Therefore, Hypothesis_1 states that decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. EEE EEIJ'IE I 1:" Reefermanee A second issue in the decision-making literature has been related to the effects on performance of providing subjects with statistically derived predictions (forecasts) before each decision trial. A large number of researchers have indicated that decision makers exhibit a tendency to incorporate the forecast into their decision strategies, as evidenced by higher levels of performance on the part of subjects who have received accurate forecasts (Carter et al., 1978; Ebert, 1972; Moskowitz and Miller 1975; Remus et al., 1984; Remus et al., 1979; Sniezek & Reeves, 1986). As previously mentioned, Sniezek and Reeves (1986) have shown that even when subjects are presented with a statistically valid forecast before each decision trial, they do not give the forecast as their response on the majority of decision trials. These findings indicate that providing subjects with a forecast will improve decision-making performance above levels achieved if no forecast were provided. It is also understood that this performance level will not be as high as that which could be attained if subjects did give the 29 forecasted value as their response on each trial. With the above discussion in mind, Hypothesis_11 states that decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in which subjects have not been provided with valid forecasts. This main effect will be primarily attributable to subjects' tendency to adjust their decision responses in the direction of the forecast. Wu Qeeisiemmmenee Another central question in the research on learning in decision-making situations has been directed toward establishing under what conditions subjects have been induced to exhibit more linear consistency in their decision-making responses, thereby improving subjects' learning and performance. Researchers have reported some success in accomplishing this task (Brehmer & Lindberg, 1970; Brehmer & Kuylenstierna, 1978; Hammond & Summers, 1965; Johansson & Brehmer, 1979; Kuylenstierna & Brehmer, 1980; Rothstein, 1986) primarily through manipulations directed at loWering subjects' expectations that accurate predications of the criterion are possible on every trial. It is proposed in this study that presenting subjects with information about expected variability in the criterion values will inform subjects that a range of values is acceptable on each trial rather than any one specific value. If providing the standard error to subjects increases subjects' tolerance of 3O variability, subjects' consistency should be improved, along with. subjects' performance and learning. Therefore, Hypothesis_111 states that decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. WWW We Closely related to the above discussion are questions concerning the interaction effects on decision-making performance of providing subjects with combinations of a forecast and feedback. In order to hypothesize the interaction effects of the first two combinations, it has once again been necessary to reference the Sniezek and Reeves (1986) study on subjects' reactions to optimal criterion information. Post-hoc conjecture by these researchers indicated that subjects did not fully utilize this information due to the noise (random error) in feedback values. Regrettably no studies have compared the performance of a forecast and no feedback group with the performance of a group receiving a forecast and feedback. Schmitt et al., (1976) and Hagafors and Brehmer, (1983) did find "no feedback" groups outperformed groups receiving task information and feedback, although the statistical task information provided was a numerical correlational index of the relationships between the cues and actual criterion 31 values. This body of research implies that providing subjects with statistically valid decision aids (correlational indices) did not overcome the negative effects of feedback. It is assumed that if the decision aid had not been presented as an index representing a bivariate density function (correlation index), but as a prediction estimate with the same numerical scale as the actual feedback value, the positive effects on subjects' performance of such a manipulation will be stronger. There is also no reason to believe that the effects of providing a forecast to subjects will overcome the confusing aspects of feedback, because the forecast value for any given trial will almost never be the same as the actual (feedback) value for that specific trial. Although over a number of trials a prediction from a least squares regression equation minimizes random error, a prediction value of this type does necessarily coincide with the actual (outcome feedback) value on any given trial. Therefore, it is assumed that the feedback and forecast treatments will not significantly interact, but will be additive functions of their respective main effects. WW SLandardJmmncLEeedheeLEaetermn Reefermenee Similar predictions are made for the interaction effects of the feedback and standard error factors as were made above. In this case, the standard error is similar to the forecast in that the standard error is a decision aid that indicates ranges of likely predicted criterion values. 32 Expectations would be that its positive effects as a decision aid would be more than those found after providing a bivariate density function index, such as has been used in previously discussed studies (Schmitt et al., 1976; Hagafors and Brehmer, 1983), but there is no reason to believe that the standard error factor will interact with the feedback factor in any way other than an additive manner. MW We Likewise, there is no reason in the literature or in logic to believe that the combination of the forecast and standard error decision aids will result in significant interactions. Although both decision aids represent components of the same least-squares predition equation, they convey different information about the task structure. For example, the forecast represents a specific numerical estimate for the actual outcome value with error minimized for that trial, whereas the standard error identifies a range of values which are most likely to contain the actual outcome value. The forecast is a specific estimate, but contains no information about the variability of the environment on a specific trial, whereas the standard error indicates a range of values based on the variability of that environment. Because different types of information are provided by each of these decision aids, it is assumed that their joint effects will be additive functions of their respective main effects rather than statistical interactions. 33 WW: W The above hypotheses have focused on the negative effects of feedback on subjects' decision-making performance reported by previous researchers (e.g. Hagafors and Brehmer, 1983; Schmitt et al., 1976). At this point, one performance prediction for this study diverges from those previous findings. It is now predicted that the combination of a forecast and standard error information will help subjects to successfully utilize feedback values so that their performance with feedback will be greater than subjects given all other combinations of a forecast, standard error information, and outcome feedback. As discussed previously, Schmitt et al. (1976) attributed the negative influence of -feedback on performance in less than perfectly predictable decision environments to the confusing aspects of actual criterion values containing random error. In lens model experiments, subjects who have developed "adequate" knowledge of the cue-criterion relationships have rejected that knowledge after encountering outlying feedback values that appeared to contradict their previous learning about cue- criterion relationships. It is proposed that the interaction of a forecast, standard error, and feedback will make feedback values more interpretable to subjects. As Sniezek and Reeves (1986) have pointed out, and as has been previously hypothesized, it is expected that subjects will modify their predictions in the direction of the numerical 34 forecast values. Regardless of this fact, providing subjects with a forecast alone does not resolve the problem associated with the fact that the actual criterion values are rarely equal to the forecast values. It is here that a combination of a forecast and information regarding the expected variability of feedback values (the standard error) is predicted to overcome the confusing aspects of those feedback values. The combination of the standard error and the forecast will provide subjects with an estimate of the range in which approximately two-thirds of the actual criterion values should fall as long as the forecast is accurate. Given this information, it is predicted that decision-makers will learn to focus on the relatively stable feedback regarding cue-forecast relationships, which contains less error, rather than on the variability of feedback regarding cue-criterion relationships, which contains large amounts of error. More simply put, the combination of a forecast, standard error information and feedback (1) guides subjects to a numerical response close to the forecast, (2) shows them that not all feedback values are relevant for learning cue- criterion relationships, and (3) allows them to simplify their decision process to one in which the cues are reliable indicators of the optimal predicted criterion value, as represented by the value of the forecast. Furthermore, since previous research has shown that subjects approach most decision tasks as essentially linear problems (Brehmer, 1980), subjects who receive the 35 combination of forecast, standard error and feedback in a linear decision task will be most likely to consistently apply that strategy over a number of trials, since they have some understanding of the relationship between the variability of the feedback values and their relationship with the criterion values. Therefore, HprLheSi§_l¥ states that decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of fOrecasts, standard errors, and feedback. The concept of environmental uncertainty has often been discussed as a factor in determining decision-making performance. A number of studies (Brehmer, 1976, 1978; Gray, 1979; Hagafors & Brehmer, 1983; Brehmer & Kuylenstierna, 1978; Lindberg & Brehmer, 1976; Naylor & Clark, 1968; Sniezek, 1986; Schmitt et al., 1976) have indicated that subjects' decision-making performance increases as the predictability of feedback values increased. As discussed previously, the reason for this phenomenon has been the tendency for human judges to monotonically match their linear consistency to the level of uncertainty in the environment as a response to the variability of feedback values (Brehmer, 36 1976, 1978). Therefore, as the system becomes more uncertain, feedback values become more variable and subjects become less consistent in their decision-making performance. Because, after this type of shift, groups receiving feedback will be confused by the greater variability in criterion values and groups receiving no feedback cannot be confused by the increased variability in the feedback values, Hypothesis 1 states that following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. I | I' EEE l E E | 1 $ 'fl W It has already been hypothesized in this study that the effect of providing subjects with a forecast will be to induce subjects to adjust their responses in the direction of the optimal predicted criterion values. Because the forecast will remain the optimum predicted criterion value even after a shift in environmental uncertainty, HynenheSiS_yl states that following a shift in the decision environment to more uncertainty, subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. 37 It is further believed that providing subjects with standard error information will visually draw attention to the shift in system variability after its occurrence as long as the standard error information is based on relatively recent data points (previous ten trials). In addition to drawing attention to the shift, it is believed that the standard error will also provide subjects with guidance for their decisions after a shift. In this instance, subjects will be informed by the standard error that their predictions may contain more variance and still be within acceptable limits. Therefore, Hypothesis_yll states that following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. WW Beefermanee Finally, following the same reasoning discussed in support of Hypothesis IV, the combination of a forecast, standard error information and feedback will effectively aid subjects following a shift toward more environmental uncertainty. Subjects will be informed of the shift and the range in which feedback values are most relevant by the standard error information. Also, as already mentioned, the 38 forecast remains the optimal statistical estimate of the criterion regardless of the variability of the system, and the relationship between the cues and the forecast remains error free. Therefore, Hypetheeie.¥lll states that following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. Table 1 - Hypothesized Effects Of Feedback, Forecast and Standard Error Factors, is provided in order to give the reader a clearer idea of the formal hypotheses and their relationships to the MANOVA and ANOVA models in which they are to be tested. In addition Table 2 - Summary Statement of Hypotheses provides the reader with restatements of the above hypotheses in tabular format. 39 TABLE 1 HYPOTHESIZED EFFECTS OF FEEDBACK, FORECAST AND STANDARD ERROR PREDICTIONS FOR DEPENDENT VARIABLES No Feedback Feedback No Forecast Forecast No Forecast Forecast No SE SE No SE SE No SE SE No SE SE GRP l 3 4 5 6 7 8 HI GROUP (l,2,3,4) > GROUP (5,6,7,8) HII GROUP (3,4,7,8) > GROUP (l,2,5,6) HIII GROUP (2,4,6,8) > GROUP (1,3,5,7) HIV GROUP (8) > GROUP (l,2,3,4,5,6,7) Predictions for Dependent Variables Following Shifts to Lower Levels of Environmental Certainty No Feedback Feedback No Forecast Forecast No Forecast Forecast No SE SB No SE SB No SE SE No SE SE GRP 1 3 4 5 6 7 8 Pre-shift Post-shift HV GROUP (l,2,3,4) GROUP (l,2,3,4) GROUP (5,6,7,8) GROUP (5,6,7,8) HVI GROUP (3,4,7,8) GROUP (3,4,7,8) GROUP (l,2,5,6) GROUP (l,2,5,6) HVII GROUP (2,4,6,8) GROUP (2,4,6,8) GROUP (1,3,5,7) GROUP (1,3,5,7) HVIII GROUP (8) GROUP (8) GROUP (1,2,3,4,5,6,7) GROUP (l,2,3,4,5,6,7) 40 Table 2 Summary Statement of Hypotheses Hypothesis I: Decision-making performance in conditions in which subjects have not been provided with outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. Hypothesis II: Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. Hypothesis III: Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors. have been provided to subjects. Hypothesis IV: Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. Hypothesis V: Following a shift toward more environmental uncertainty, the decision- making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. Hypothesis VI: Following a shift in the decision environment to more uncertainty, subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. 41 Table 2 (Continued) Summary Statement of Hypotheses Hypothesis VII: Hypothesis VIII: Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will experience either decrements or no change in their performance. CHAPTER IV METHODOLOGY AND DATA ANALYSIS Subjects One hundred and seventy six students enrolled in the business policy course at a midwestern university served as subjects. They were rewarded with class credit. The data? for five subjects were dropped from the analysis due to discrepancies in the post experimental verification of their group membership. More specifically, the experimenter recorded five subjects' experimental conditions differently than the subjects' recorded their own experimental condition. In order to make sure that no subjects' data were analyzed in an incorrect cell, these five subjects' data were excluded from all analyses. DeeimmLBreeednre Subjects were randomly assigned to one of eight conditions: two feedback conditions, two forecast conditions, and two standard error conditions with repeated measures on each of the dependent variables. This experiment was designed to be a 2 (feedback/no feedback) X 2 (forecast/no forecast) X 2 (standard error/no standard error) X 4 (blocks of 20 consecutive trials representing four different levels of environmental predictability) factorial design. The feedback, forecast, and standard error treatments were analyzed as 42 43 between subject factors and the four blocks of twenty trials under different levels of environmental predictability were analyzed as the within-subject factor. In all conditions subjects were required to make 80 predictions of a hypothetical airline's total weekly passenger miles based on two meaningfully labeled cues numerically presented to them on a computer screen (See Appendix A). All subjects completed 20 iterations (one block) of the simulation in four levels of environmental uncertainty (multiple R = .93, multiple R = .38, multiple R = .64 and multiple R = .38)1 . All groups were asked to make predictions about the upcoming criterion value based on the values of two numerical cues. The intercorrelation between the two cues was .11, .21, .10, and .29 for the four consecutive blocks of twenty trials. The correlations between the first cue and the actual passenger miles were .54, .26, .36, and .24 respectively. The correlations between the second one and the actual passenger miles were -.60, -.18, -.43, and -.22, respectively. In Group 1, subjects made predictions without feedback, forecast or standard error. In Group 2, subjects were shown a graph indicating the forecast's standard error before each trial, but did not receive the forecast or feedback. In Group 3, subjects were presented with a forecast before each trial, 1 Originally the study was planned with two environments of multiple R = .80 and multiple R = .53. Because of experimenter error during the design of the simulation, the above multiple R's represented the environment. 44 but received no feedback or standard error. In Group 4, subjects saw a forecast before each trial with that forecast's standard error indicated but were not provided with feedback. In Group 5, subjects received the actual criterion value as feedback after each trial, but were not given access to the forecast or standard error. In Group 6, subjects received the actual criterion value after each trial and were shown the standard error of the forecast before all trials, but were not shown the forecast itself. In Group 7, subjects received a forecast before each trial and feedback after every trial, but were not shown the standard error. In Group 8, subjects received feedback, a forecast, and the forecast's standard error for each trial. After each block of twenty trials all subjects experienced a shift in the level of certainty in the environment. E: I- . I 1 I I E S l . I In order to ensure that subjects' performance in each QQndition was based on learning the cue-criterion relationships, rather than simply relying on one or more of the decision aids, immediately following the completion of the 80 trials subjects were asked to place twenty-five cards Qontaining examples of cue-criterion pairings in bins rlumbered one to five. Subjects were told, "Randomly spread out before you are twenty-five cards containing possible values of one of the cues and the actual airlines passenger miles. You also have 45 :five bins placed before you which are numbered 1 to 5. Place eeach of the twenty-five cards in the bin that represents your kJelief about how likely it is that the pairing of the cue and 'tfihe actual passenger miles on the card would have occurred during the simulation, with bin 1 representing 'very lizmlikely' and bin 5 representing 'very likely'." An attempt vases made to provide subjects with an anchored scale telling t:11enu "You must begin this exercise by placing the one card j§zcyu believe is least likely to have occurred into bin 1 and t:ln£e card you believe is mast likely to have occurred into bin 55 - After that you may put any card into any bin that you be lieve is appropriate . " The twenty-five cards were produced by rank ordering the ‘\r21lues of each cue and the values of the criteria, selecting tllle values of each that occured at the 10th, 30th, 50th, .7()th, and 90th percentiles. These values were arranged on tllme respective axes of a five by five matrix, so that the J-eft to right diagonal contained the actual pairings ‘C>ccurring at the respective percentiles, with combinations Ifrom the off-diagonals becoming less likely to occur as a IEunction of their distance from the diagonal. In this €3xperiment, bin 5 represented the pairings on the diagonal Eind bins 4 through 1 represented the next four off-diagonal Ipairings as shown in Table 3 - Example Of 5x5 Mbtrix For Developing Combinations Of Cue And Criterion For The Post-Experimental Task . 46 Table 3 Example Of 5x5 Matrix For Developing Combinations Of Cue And Criterion For the Post-Experimental Task (Ordered Seasonality Values = 64,76,80,82,96; Ordered Criterion values = 46,51,54,58,77) 46 51 54 58 77 64 64,46 64,51 64,54 64,58 64,77 76 76,46 76,51 76,54 76,58 76,77 80 80,46 80,51 80,54 80,58 80,77 82 82,46 82,51 82,54 82,58 82,77 96 96,46 96,51 96,54 96,58 96,77 (LEFT TO RIGHT DIAGONAL REPRESENTS MOST LIKELY COMBINATIONS BASED ON ACTUAL CUE-CRITERION PAIRS IN THE SIMULATION) Measures Both the forecast and standard error information were <2<1mputed on the basis of the least squares regression €3llowing manner: ra = rmReRs + \l c (1-R2e) (1-R25) f . Acenraex The measure of accuracy in the study was defined as the Eaverage variance of the difference score between the actual ‘Czriterion value and the subject's responses across trials. 'IPhis definition of accuracy is the same as the definition Ilsed by Sniezek and Reeves (1986): The extent to which a prediction is an inaccurate estimate of Ye is given by I, the difference score Y3 - Ye. The distribution of I will have parameters ui and 012. Mean instantagy over trial n trials, u: = ZI/n, can 49 be thought of as judgment bias. The variance of I scores, oi2, reveals the extent to which judgments deviate from uI, the point of average inaccuracy. As Naylor (1967) points out, the variance of the difference scores is limited in that it is based on deviations from the point of average, not zero, error. Therefore, the performance measure inaccuracy is defined with respect to deviations from the point of zero error: 0102 = 2(I - O)2/n = 212/n. Since the value of an average squared score equals the variance plus the squared mean, it follows that inattttsty = 0102 = 012 = u12. Thus prediction becomes more inaccurate as the variance of I = Y5 - Ye increases or as judgment bias increases (p. 301). 52:] 'E' I. E I ll 1] I! . 1] Before moving on with this discussion, it is important ‘t:<3 clarify the meanings of these lens model variables. In 1:Lihis case, it must be kept in mind that the lens model Qquation relies on regression and correlational analysis to ItIeasure the congruence between the judge's decision-making I;>olicies and that policy which best describes the statistical E>roperties of the judgmental task. Therefore, the J.imitations of correlational measures such as these should be Iioted. For example, ra is an overall achievement score, and in the literature has occasionally been discussed as a ‘measure of the subject's accuracy. As Naylor (1967) and later 50 Sniezek and Reeves (1986) have pointed out, accuracy and achievement are not synonymous in lens model research. Since achievement (ra) is a correlational index, it is possible for a judge to reach high levels of achievement, while being grossly inaccurate. In this case, the correlation between the judge's predictions and the criterion values are very high, but the judge's predictions differ greatly in absolute terms from the criterion values. as Brehmer (1973) has pointed out, rm Furthermore, (matching) measures the degree to which subjects develop least squares prediction equations which are amnertienai in terms of regression weights to the optimal least squares regression equation for that particular task. Due to these facts, it is not necessary for the subject to give the same magnitude to cue weights as are found in the optimal It regression equation in order to attain perfect matching. is only necessary that the subject maintain the same proportional relationship between/among the cue weights that exist in the optimal environmental equation. Some clarification of the concept of consistency (Rs) Eilso is warranted, since in the lens model, Rs refers to the degree to which a subject consistently applies a linear I>olicy. Absolute consistency (choosing the same response Jregardless of cue values) or the consistent application of nonlinear policies on the part of the subject can result in a very low Rs index in lens models tasks (Koele, 1980) . 51 We Amheiefleesnresmismaking Eerfermanee Multivariate analysis of variance (MANOVA) was used to aissess the simultaneous effects of the factors on the four Ibearformance measures, followed by separate univariate earialysis of variance on each of the four dependent variables aaczhievement, consistency, matching, and accuracy. W W The analysis of the post-experimental card sorting task Vwais intended to determine the answers to two questions: (1) 13:1d groups significantly differ in their placement of cards lili the five bins? and (2) Given that groups significantly ciiffered in their placement of cards in the five bins, did saroups predicted to have more knowledge of the statistical J:elationships in the simulation more closely approximate the iaetnai frequency value for each of the five bins? For this analysis, subjects placed the cards containing Igarticular cue-criterion relationships into the each of the five bins. After each subject had finished putting all the cards into the bins, the experimenter recorded the aetnai simulation frequency of each of the cards in each bin. It was possible to record the aetnai frequency of the cards containing the cue-criterion relationships for each subject 52 AfOr each bin since they had been color-dot coded on the back ()f the card before the experiment. As an example of this czalculation, the subject put four cards in bin 3. One cue- (:riterion's relationship was the second most likely to occur, aenother was the fourth most likely to occur, another was the :nnost likely to occur and.the last was the third most likely The subject's score for bin 3 (implying a midrange t 0 occur. calculated as the mean of 4 + ;1;ikelihood of occurrence) was :2 + 5 + 3 = 3.5. The mean of the cards in each of the five Zt>jrns was calculated for each subject and analyzed using an 8 (gyroup) by 5 (bins) repeated measures MANOVA design, with ezlroups representing the between-subjects' factor and the five bin scores for each subject representing the within-subjects' ifaictor. CHAPTER V RESULTS WM As indicated by Bray & Maxwell (1985), four assumptions are required for the application of multivariate analysis of variance techniques: (1) units (subjects) have been randomly sampled from the population of interest, (2) observations are statistically independent of one another, (3) the dependent variables have a multivariate normal distribution within each group, and (4) the k groups have a common within-group population covariance matrix (p.32-33). The design of the experiment ensured that the first requirement had been met, in that subjects were arbitrarily assigned to conditions based on their self-selected appointment time for the experiment. The second requirement also was fulfilled, since although subjects had multiple observations at different times, the observations between-subjects were statistically independent. Preliminary linear analyses indicated that the last two assumptions (multivariate normality and equality of ~covariance matrices) were not met in this data set. Specifically, the data appeared to deviate strongly from the assumption of multivariate normality within each group. Stem and leaf plots of the data indicated that the distributions 53 54 of the dependent variables were in three cases negatively skewed (achievement, consistency, and matching), while in one case was positively skewed (accuracy). Since univariate normality has been shown to be necessary but not sufficient condition for multivariate normality (Carroll, 1961), it was apparent that the multivariate linear combinations of each of the variables also deviated from normality. Although a number of Monte Carlo studies (Ito, 1969; Mardia, 1971; Olson, 1974) have shown that deviations from normality have only slight effects on Type 1 errors (rejecting the null hypothesis when it is in reality true) in multivariate analysis of variance, the multivariate linearly combined variables were normalized using an orthonormal transformation before begining the analysis (Hand and Taylor, 1987, p. 57). The means and standard deviation for each cell, correlation matrix for dependent variables and stem and leaf plots for the dependent variables are presented in Appendix B, Appendix C, and Appendix D, respectively. The MANOVA requirement that the covariance matrices of these data be equivalent presented a more complicated problem. The hypotheses in this study were based on previous theoretical findings indicating that one of the primary differences among treatment groups in decision-making experiments has been in their variances, in particular among groups of subjects given feedback and subjects not given feedback. The expectation that the experimental groups would share a common within-group population covariance matrix was 55 contradictory to the predicted results of this study. Even though these data had been tranformed into Fischer 2 scores in order to standardize the means and standard deviations of the dependent variables with different scales before beginning the analysis, it was not surprising that the Bartlett-Box F test for univariate homogeneity of variance for each dependent variable and the multivariate Box M test for homogeneity of dispersion matrices were highly significant (p < .001, in both cases). Since both of these tests were negatively influenced by the departures from normality in the data, Levene's test (Conover, Johnson, & Johnson, 1981, p. 19) was performed on the data. This test, which entails doing one-way analysis of variance on the absolute value of the difference betweeen each observation and the mean for that observation, has been found to be the most robust test for homogeneity of dispersion matrices in situations in which the data is skewed (Conover, Johnson, and Johnson, 1981, p. 19). Application of this test to each dependent variable at each time period indicated that the assumption of homogeneous group variances was rejected in all cases for each of the dependent variables. Regarding this problem, Olson (1974, 1976) has found that the Pillai-Bartlett trace has been robust in situations where covariance matrices are unequal aineng as cell sample sizes are equal. Therefore, the data were reanalyzed with cell sizes arbitrarily made equal (N=20) by random deletion of records. Comparison of the Pillai-Bartlett trace 56 indicated that results which met the criterion of .01 significance level in the equal cell analyses also met this criterion in the analyses where cells were not equal, as indicated in Table 4 - Comparison of Significant Results using the Pillai-Bartlett Trace. The requirement that the experimental groups share a common within-group covariance matrix was complicated by the inclusion of a repeated measures component in the design. In this situation, not only must different groups share a common within-group covariance matrix, but the same groups must share a common within-group covariance matrix over time. Huyn and Feldt (1970) have proposed a downward adjustment of both the treatment degrees of freedom and the error degrees of freedom in order to provide an appropriately conservative F test. The Huynh and Feldt adjustment produces degrees of freedom that are appropriate for a reduced design consisting of a t-test using the subject as his or her own control. According to Barker and Barker (1986), this reduction requires no assumption of homogeneity of covariance matrices nitnin groups over time (p. 94). Based on the same statistical principles as the Huyn and Feldt (1970) adjustment factor, the Greenhouse-Geiser epsilon factor has the advantage of being calculated as a specific integer, rather than calculated as a range of upper and lower bounds as has been the Huyn and Feldt (1970) adjustment factors. In this study, the Greenhouse-Geisser epsilon adjustment factor was calculated for each repeated measures 57 ooo. oo.o ooo. Ho.o uuaem an xomnommm omo. mo.H oma. mo.H swarm an ummoouom ooo. oo.H moo. oo.m upwem an uouum osmocmum mam. om.H omm. vH.H unacm an ummoouom an xomhoooh VNH. mm.H Hma. mv.H umwcm sh uouum oumocmum an xownooom moo. mo. was. me. umonm an uouum oumocmum an ummoouom mam. mm. mam. mo.H storm as souum osmocmum an ammoouom an xomhooom ooo. mm.sm ooo. mo.mm somnuomm ooo. oo.HH ooo. oo.ss ummomuom ooo. om.o ooo. om.os house nuancmum ooo. oo.m ooo. mm.m ammomuom an somnoowm ooo. mm.v ooo. mo.m nonum oumocmum >9 xomhomom ooo. mo.m ooo. mo.o uouum oumocmum an unmoouom OQO . MH . N HQO . 0m . N .HOHHW UHflUCflUW “Q “mmumHOh %Q 30mgmmm NNwZWoN omuz mocmofimocmflm m oocmoauacmwm m uouomm moose uuoauummuamaafim who mcflma muasmom Demoamacmom mo comaumeEoo o momma 58 relationship tested in the data. The multiplication of both the numerator and denominator of the F-ratio by the Greenhouse-Geisser epsilon factor has been described as a conservative method of minimizing Type 1 errors when analyzing repeated measures' data with unequal covariance matrices (SPSSX User's Guide, 1988, p. 610). E I I. E . I W In order to interpret the relatively large number of univariate results which were the primary focus of the hypotheses of this study, two procedures were utilized in order to minimize Type I errors, that is, rejecting the null hypothesis when it is true. First, the protected F or Least Significant Difference test (Bock, 1975; Cooley & Lohnes, 1972; Wilkinson, 1975) was used to ensure that the overall multivariate tests provided protection from an inflated alpha on the univariate tests of the-four dependent variables. By interpreting only univariate tests for those relationships which were significant on the multivariate results, the overall alpha level for the four univariate tests should have been held near the experiment-wise specified nominal alpha (.05). Since it has been shown that the protected F test does not adequately control for Type I error within each of the separate univariate tests of the dependent variables (Miller, 1966, p.93), the Bonferroni procedure (Harris, 1975) was used in conjunction with the protected F tests to minimize the . probability of rejecting the null hypotheses when they are 59 true. This procedure involved dividing the nominal alpha (.05) by the number of main effects in the study and using that modified alpha as the minimum level of significance for interpretation purposes. In the present analysis, the Bonferroni adjusted alpha was .05/4 = .012, rounded off to .01 for the purposes of this study. It must be noted here that this interpretation of the Bonferroni procedure has utilized the concept of comparisons as corresponding to the main effects, rather than to the summation of main effects and their possible interactions. This interpretation of the Bonferroni approach has been suggested by both Barker and Barker (1984, p. 36) and Bray and Maxwell (1985, p. 51-52) for studies in which planned rather than unplanned comparisons are to be made. Although, to the writer's knowledge, combining these two procedures has not been recommended in the literature, it was known in the planning stages of the study that the dependent variables would be highly intercorrelated and Type I errors would therefore be a concern on the univariate ANOVAs. With this in mind, a conservative approach was deemed warranted for interpreting the relatively more powerful univariate results and tests of specific hypotheses which follow the multivariate tests. The reader is also asked to keep in mind that the multivariate results presented below have not been intended to provide direct determinations of the veracity of specific hypotheses, but rather to provide protection against Type I error or, the incorrect decision 60 that the research treatments under study produced the differential outcome observed in the dependent variables, when actually the differences were produced by chance or random events. “1|. . I BEE I E I] E i] 1 EEQLQI The results indicated that effects of feedback (p < .001) were significant for all three multivariate measures. (Pillias-Bartlett trace, Hotellings T, Wilks Lambda). Levels of significance for these three measures have been determined according to three parameters S, M, and N. Barker and Barker (1986) have presented the mathematical formulas for these parameters as follows: 5 = min (dfh,p) M = ( dfh - p - 1)/2 N = (dfe - p - l)/2 where dfh corresponds to the degree of freedom associated with the treatment effect, dfe corresponds to the degrees of freedom associated with the error term, and p corresponds to the number of dependent variables (p. 23). Examination of the correlations among dependent and canonical variables, an indicator of the strength of the relationship between the pairs of canonical variates (Hair, 61 Anderson, Tatham, & Grablowsky, 1979, p. 183) revealed that the consistency and accuracy measures accounted for the most variance in the overall canonical function. Table 5 - Effects of the Feedback Factor on Linearly Combined Dependent Variables also shows the raw discriminant function coefficients and the standardized discriminant function coefficents. These measures provide information concerning the degree of necessity of retaining each variable in the complete set of discriminators (Bray & Maxwell, 1984 p. 45). H J!’ . I BEE I E I] SI I l EIIQI_EBCLQL Significant effects were also found for the standard error treatment on all three multivariate tests of significance (p < .001). Examining the correlations between dependent and canonical variables indicated substantial contribution by all four dependent variables in explaining differences among groups as presented in Table 6 - Effects of Standard Error Factor on Linearly Combined Variables. H II. . I BEE I E I] E Easter Likewise, Table 7 - Effects of the Forecast Factor on the Linearly Combined variables indicates that main effects for the forecast treatment were significant on the three multivariate test statistics (p < .001). Correlations between dependent and canonical variables again indicated 62 Table 5 Effects of the Feedback Factor on Linearly Combined Dependent Variables Multivariate Tests of Significance (S = 1, M = 1, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .37 23.75 4 160 .000 Hotellings .59 23.75 4 160 .000 Wilks Lambda .62 23.75 4 160 .000 Roy's Root .37 Raw discriminant function coefficients Function No. Variable 1 Achievement .523 Consistency -.873 Matching -.122 Accuracy -.527 Standardized discriminant function coefficients Function No. Variable 1 Achievement .667 Consistency -1.189 Matching -.174 Accuracy —.731 Correlations between dependent and canonical variables Canonical Variable Variable 1 Achievement .094 Consistency -.485 Matching .000 Accuracy -.493 Estimate of effect .753 63 Table 6 Effects of Standard Error Factor on Linearly Combined Variables Multivariate Tests of Significance (S = 1, M = 1, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .20 10.38 4 160 .000 Hotellings .25 10.38 4 160 .000 Wilks Lambda .79 10.38 4 160 .000 Roy's Root .20 Raw discriminant function coefficients Variable Achievement Consistency Matching Accuracy Standardized discriminant function coefficients Variable Achievement Consistency Matching Accuracy Function No. 1 .483 -.006 -.110 -.460 Function No. 1 .617 -.009 -.157 -.638 Correlations between dependent and canonical variables Variable Achievement Consistency Matching Accuracy Canonical Variable 1 .849 -.627 .647 -.914 Estimate of effect .498 64 Table 7 Effects of the Forecast Factor on the Linearly Combined Variables Multivariate Tests of Significance (S = 1, M = 1, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .23 11.97 4 160 .000 Hotellings .29 11.97 4 160 .000 Wilks Lambda .76 11.97 4 160 .000 Roy's Root .23 Raw discriminant function coefficients Variable Achievement Consistency Matching Accuracy Standardized discriminant function coefficients Variable Achievement Consistency Matching Accuracy Function No. 1 .176 -.421 -.339 -.270 Function No. 1 .225 -.573 -.486 -.374 Correlations between dependent and canonical variables Variable Achievement Consistency Matching Accuracy Canonical Variable 1 -.799 -.876 -.828 .736 Estimate of effect -.534 65 nearly equal contribution of the four dependent variables in differentiating among treatment groups. H Jl' . I EEE I E I] E I E StandarcLErreLIhteraetien The forecast by standard error interaction was also significant for the linear combinations of dependent variables (p < .001). Correlations among dependent and canonical variables and the estimate of effects for canonical variables are presented in Table 8 - Effects of the Forecast and Standard Error Interaction on the Linearly Combined Variables. H JI° . I BEE I E I] E I] l E Winn All three multivariate tests also indicated significant results for the feedback by standard error interaction (p < .001). As shown in Table 9 - Effects of the Feedback and Standard Error Factors on the Linearly Combined variables, the linearly combined accuracy measure was the most important in discriminating among treatment groups. H JI' . I EEE I E T] E II 1 E ForecastJnteraetien The feedback by forecast interaction was significant (p < .01) as can be seen in Table 10 - Effects of the Feedback and Forecast Factors on the Linearly Combined variables. The estimate of effects for canonical variables 66 Table 8 Effects of the Forecast and Standard Error Interaction on the Linearly Combined Variables Multivariate Tests of Significance (S = 1, M = l, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .16 7.62 4 160 .000 Hotellings .19 7.62 4 160 .000 Wilks Lambda .84 7.62 4 160 .000 Roy's Root .37 Raw discriminant function coefficients Function No. Variable l Achievement .047 Consistency -.364 Matching -.213 Accuracy .343 Standardized discriminant function coefficients Function No. Variable 1 Achievement .060 Consistency -.496 Matching -.306 Accuracy -.475 Correlations between dependent and canonical variables Canonical Variable Variable 1 Achievement -.820 Consistency -.854 Matching -.784 Accuracy .812 Estimate of effect .426 67 Table 9 Effects of the Feedback and Standard Error Factors on the Linearly Combined Variables Multivariate Tests of Significance (S = 1, M = 1, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .12 5.62 4 160 .000 Hotellings .14 5.62 4 160 .000 Wilks Lambda .88 5.62 4 160 .000 Roy's Root .12 Raw discriminant function coefficients Function No. Variable 1 Achievement .345 Consistency -.079 Matching -.044 Accuracy .820 Standardized discriminant function coefficients Function No. Variable 1 Achievement .441 Consistency -.108 Matching -.063 Accuracy 1.137 Correlations between dependent and canonical variables Canonical Variable Variable 1 Achievement -.343 Consistency -.346 Matching -.311 Accuracy .962 Estimate of effect .366 68 Table 10 Effects of the Feedback and Forecast Factors on the Linearly Combined Variables Multivariate Tests of Significance (S = 1, M = 1, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .08 3.55 4 160 .008 Hotellings .09 3.55 4 160 .008 Wilks Lambda .92 3.55 4 160 .008 Roy's Root .08 3.55 Raw discriminant function coefficients Function No. Variable ‘ 1 Achievement .118 Consistency .395 Matching .291 Accuracy .779 Standardized discriminant function coefficients Function No. Variable 1 Achievement .150 Consistency .538 Matching .418 Accuracy 1.081 Correlations between dependent and canonical variables Canonical Variable Variable 1 Achievement .255 Consistency .409 Matching .351 Accuracy .550 Estimate of effect .291 69 indicated that the variables as a group contributed a relatively small amount of variance toward differentiating groups. Table 11 - Effects of Feedback and Shift on Linearly Combined variables indicates that the only significant result involving the within-subject factor of shift was associated with the interaction of feedback by shift (p < .001). These effects remained at the same level of significance following the multiplication of the numerator and denominator degrees of freedom by the Greenhouse-Geisser Epsilon factor (.65). Table 12 - Testable Hypotheses by Univariate ANOVAs Based On Preliminary MANOVA Analyses is provided as a summary of the hypothesized relationships that are testable using univariate tests given the results of the multivariate analysis. E 1' | H . . I E II The two-way interaction effect for groups receiving combinations of forecast and standard error treatments was significant, F(1,163) = 20.85 p < .001. Scheffe's test indicated that the group that received no decision aids was significantly different from the groups receiving either or both of these treatments, as presented in Figure 1 - 70 Table 11 Effects of Feedback and Shift on Linearly Combined Variables Multivariate Tests of Significance (S = 1, M = 1, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .19 8.31 4 1464 .000 Hotellings .22 9.00 4 1454 .000 Wilks Lambda .81 8.69 4 1286 .000 Roy's Root .37 Raw discriminant function coefficients Function No. Variable 1 2 3 Achievement 1.404 -.288 -.053 Consistency -.815 -1.527 —.651 Matching -.787 .809 -.814 Accuracy -.142 -.506 1.300 Standardized discriminant function coefficients Function No. Variable 1 2 3 Achievement .967 --199 -.036 Consistency -.460 -.862 -.367 Matching -.481 .494 -.497 Accuracy -.066 -.234 .601 Correlations between Dependent and canonical variables Canonical Variable Variable 1 2 3 Achievement .747 -.223 -.527 Consistency -.246 -.834 -.485 Matching -.294 .379 -.700 Accuracy -.349 -.210 .757 Estimate of effect .727 Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: 71 Table 12 Testable Hypotheses by Univariate ANOVA Based On Preliminary MANOVA Analyses H Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. TEST WARRANTED? YES SIGNIFICANT TWO-WAY INTERACTION OF FEEDBACK AND STANDARD ERROR FACTORS F(4,160)=5.61 p < .001 AND FEEDBACK AND FORECAST FACTORS F(4,160)=3.54 p < .01 max INCLUDE CONFIRMATION OF MAIN EFFECTS FOR THE FORECAST FACTOR. Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. TEST WARRANTED? YES III: SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS F(4,l60)=7.62 p < .001 AND FEEDBACK AND FORECAST FACTORS F(4,160)=3.54 p < .01 MAY INCLUDE CONFIRMATION OF MAIN EFFECTS FOR THE FORECAST FACTOR. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. TEST WARRANTED? YES SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS F(4,160)=7.62 p < .001 AND FEEDBACK AND STANDARD ERROR FACTORS F(4,160)=5.61 p < .001 MAX INCLUDE CONFIRMATION OF MAIN EFFECTS FOR THE STANDARD ERROR FACTOR. 72 Table 12 (Continued) Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: IV: Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. TEST WARRANTED? NO THREE WAY INTERACTION OF FEEDBACK, FORECAST, AND STANDARD ERROR FACTORS WAS NOT SIGNIFICANT ON MULTIVARIATE TESTS F (4, 160) = 2.29 p > .06. ‘V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. TEST WARRANTED? YES TWO WAY INTERACTION OF FEEDBACK AND SHIFTS WERE SIGNIFICANT ON MULTIVARIATE TESTS F(12, 152) = 7.01 p < .001 ‘VI: Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will 'experience lowered performance. TEST WARRANTED? NO NO INTERACTIONS INVOLVING THE FORECAST AND SHIFT FACTORS WERE SIGNIFICANT ON MULTIVARIATE TESTS. VII:Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. TEST WARRANTED? NO NO INTERACTIONS INVOLVING THE STANDARD ERROR AND SHIFT FACTORS WERE SIGNIFICANT 0N MULTIVARIATE TESTS. 73 Table 12 (Continued) Hypothesis VIII:Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. UNIVARIATE TEST WARRANTED? NO COMMENTS: THE INTERACTION OF THE FEEDBACK, FORECAST, STANDARD ERROR.AND SHIFT FACTORS WAS NOT SIGNIFICANT ON MULTIVARIATE TESTS F(12, 152) 1.06 p > .39. 74 Achievement - Forecast by Standard Error Interaction.2 These results indicate that the subjects who were given both the forecast and standard error either did not recognize the differential information provided by these decision aids or if they did, they did not apply that information. Because groups receiving either or both decision aids reached about equal levels of achievement, it appears that the effects of the forecast and standard error on decision performance are best described as interchangeable substitutes for one another. The interaction effect for feedback by shift was significant for the achievement measure F(3,489) = 17.91: p < .001. Multiplying the numerator and denominator degrees of freedom by the Greenhouse-Geisser Epsilon factor (.92) did not affect the level of significance (p < .001) of this interaction. Paired t-tests indicated that all comparisons of within subject scores across the shifts were significant (p < .01) with the exception of the no feedback subjects' performance across blocks three and four (p > .2). This interaction is visually presented in Figure 2 - Achievement - Feedback by Shift Interaction. Subjects receiving feedback did decrease their performance following shifts to greater environmental uncertainty, as did the subjects receiving no feedback following the first shift. 2 All graphs of interactions found in the data collection are presented as Z-score transformed means. Appendix E presents graphs for interactions in actual lens model measures. 75 16:8 n >ozo02o<¢3o3 1 mooncmox 3 m2: 5889.0: mooncmox Zo moon cmox 77 Table 13 - Achievement - ANOVA Results presents the ANOVA univariate results for all main and interaction effects associated with the achievement dependent variable. Table 14 - Summary of Hypothesized Results for the Achievement Measure presents the way in which each hypothesis was comfirmed or discomfirmed by these results. Many—Results Univariate results were significant for the feedback treatment F(1,163) = 22.77, p < .001. Subjects not receiving feedback outperformed subjects receiving feedback on the consistency measure. The interaction of forecast and standard error treatments was significant F(1,163) = 22.64, p < .001. All treatment groups outperformed the controls as shown in Figure 3 - Consistency - Forecast by Standard Error Interaction. Again this interaction indicated that the effects of the forecast and standard error on subjects’ performance were so similar as to be redundant. The feedback treatment interacted with shift on the consistency measure, F(3, 489) = 4.94, p < .01 and this interaction remained significant after adjustment by the Greenhouse-Geiser Epsilon factor (.97). Paired t-tests indicated that subjects receiving feedback differed Significantly among all blocks (p < .01) with the exception of subjects' mean consistency scores of block 2 and block 4 (p > .7). Again refering to Figure 4 - Consistency- Feedback by Shift Interaction, it can be seen that the 78 TABLE 13 mmm. vH. mo. m mH. DMHnm xb uouum Cumocmom an ummomuom an Moancmmm vvm. mm. mm. m an. DHHCm an uounm Unmocmum >b ummomuom Hmm. vv.H mm. m mo.m DCHBm an nouns pumpcmpm an xombpmmm mmH. mm.H hp. m Hm.m uuHCm an ummomuom an gumbommm ooo. me.~ mH.H m am.m DHHBm an uouum pumpcmum Nmo. mv.~ bH.H m Hm.m DHHnm >9 ummomuom ooo. Hm.pH om.m m Hm.mm DoHsm an Momnpmmm mam. Ho. Ho. m we. DHHnm be. mmv mH.mmm mHHOo :HCDHS m mo .mHm m m2 mo mm coHumHnm> mo mouoom muommmm uommbom cHsqu mo mummy mmo. mo.m mm.¢ H mm.e uouum pumpcmum an ummumuom an xomhpmmm ooo. om.om mm.mm H mm.mm .uouum nnmpcmnm Sn ummomuom «OH. os.~ oq.v H ov.v uouum pumpcmum an somnpmwm mmm. «m. «m.H H Hm.H ummomuom Ha xomnpmmm ooo. mm.om v>.mv H v>.mw uouum oumocmum ooo. mH.Hm m>.om H m>.om uwmomuom mmm. mm. ov.H H ov.H Momncmom Hum. mo. vo. H «0. ucmumcoo mm.H mmH Hm.mmm mHHmu cHnqu m mo .mHm m m2 mm mm coHumHum> mo mousom muoommm MHOOmbsmucoosuom mo mummy manmmm ¢>oza . Damsm>mHBo¢ MH magma 79 Table 14 Summary of Hypothesized Results for the Achievement Measure Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: I: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? NO MAIN EFFECT FOR FEEDBACK FACTOR WAS NOT SIGNIFICANT F(1,163)=.86 p >.36. II: Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. CONFIRMED? NO SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS INCLUDED NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. III: Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS INCLUDED NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. IV: Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. CONFIRMED? NO THREE WAY INTERACTION OF FEEDBACK, FORECAST, AND STANDARD ERROR FACTORS WAS NOT SIGNIFICANT F(1,163)=3.05 p > .08. 80 Table 14 (Continued) Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: ‘V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. CONFIRMED? NO BOTH FEEDBACK AND NO FEEDBACK GROUPS EXPERIENCED SIGNIFICANT DECREMENTS IN PERFORMANCE FOLLOWING A SHIFT TO LOWER LEVELS OF ENVIRONMENTAL CERTAINTY. 'VI: Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. CONFIRMED? NO THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = 2.46 p > .06. VII:Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. CONFIRMED? NO THE STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = 2.49 p > .06. VIII: Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. CONFIRMED? NO THE FEEDBACK BY FORECAST BY STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = .06 p > .93 81 .otm panama Btu Embed-m oz an. I 6320“. o 3. mm. 8820“. en. 22829:. totem otmpcmbm >m $390.... I 5:063:00 m 059”. 82 mm... x0380“. 5... MNMMI\\I\\\II\I Z I mp. VN. mm. mm. x883”. oz 5582:. rim 5 x0330“. I 5:083:00 v 2:9". 83 performance of subjects who received feedback did decrease at block 2 following the first shift to a less environmental certainty, but increased during block 4 after a shift to less environmental certainty. On the other hand, the consistency scores of subjects not receiving feedback did not significantly differ across any blocks (Figure 4). Table 15 - Consistency-ANOVA Results includes all the MANOVA univariate results for the consistency measure and Table 16 - Summary of Hypothesized Results for the Consistency Measure presents the confirmation or discomfirmation of each hypothesis for the consistency measure . W As was true with the previously discussed dependent variables, the forecast and standard error interaction was significant at the p < .001 level: F(1, 163) = 19.10. Again this interaction effect was attributable to lower performance on the part of the control subjects and further indicated that the group receiving both the forecast and standard error did not significantly outperform groups receiving either one of these two decision aids as can be seen in Figure 5 - Matching-Forecast by Standard Error Interaction. The feedback by shift interaction for subject matching was small in terms of the f statistic: F(3,489) = 3.82, but effects of the Greenhouse-Geisser factor adjustment (.95) led to no change in the significance level of this effect, p < .01) . Figure 6 - Matching-Feedback by Shift 84 mmm. «H. co. m mH. Dqum an Houum pumpcmum >b ummomuom an xomgoomm vvm. mm. mm. m mp. HHHCm an uounm pnmocmum an ammoouom Hmm. q¢.H mm. m mo.~ DHHBm an noun: pumpcmum sh xomnpmmm one. vm. om. m om. DHHCm xb ummomuom an Romnooom Hoo. nm.m m>.H m mm.m DHHCm xb gonna oumocmum mmm. mm.H mv. m om.H DHHnm >2 ammoouom moo. vm.v rm.H m N>.v HMHnm >n xombpmom ooo.H oo. co. m oo. uanm mm. mmv m>.mmH mHHOO :Hnqu m mo .mHm m m2 mo mm :OHumHum> mo mouoom muommum somflnsmIcHnqu mo momma «mo. GH.m mm.m H mm.m noun: pumpcmum Sn ammomuom an xomhpmmm ooo. vo.mm oo.mv H oo.m¢ uouum oumocmum >n ummomuom OOH. vh.m mo.m H mo.m uouum pumpcmum an xomnpmmm NNH. mv.~ me.q H mv.v ummomuom an xumnpmmm ooo. HG.GH hm.om H pm.om “chum pumpcmum ooo. vv.>m mv.mm H o>.mm ummoouom ooo. hh.mm vm.mv H vm.mv Momnommm Hmm. oo. oo. H oo. ucmumcoo mm.H mmH Nv.mom mHHmO cHnqu m mo .mHm m m2 mo mm :oHumHnm> mo mouoom muommmm muomflbsmucmozuom mo mumoe muHommm m>oz< I HOGOUMHmcou mH OHQMH 85 Table 16 Summary of Hypothesized Results for the Consistency Measure Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: I: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? YES II: MAIN EFFECT FOR FEEDBACK FACTOR WAS SIGNIFICANT F(1,163)= 22.77 p < .001 Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. CONFIRMED? NO III: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO IV: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. CONFIRMED? NO THREE WAY INTERACTION OF FEEDBACK, FORECAST, AND STANDARD ERROR FACTORS WAS NOT SIGNIFICANT F(1,163)=5.16 p > .02. 86 Table 16 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. HYPOTHESIS CONFIRMED? NO COMMENTS: Hypothesis VI: ALTHOUGH THE NO-FEEDBACK GROUP DID NOT EXPERIENCE SIGNIFICANT DECREMENTS IN PERFORMANCE FOLLOWING A SHIFT TO LOWER LEVELS OF ENVIRONMENTAL CERTAINTY, BETWEEN BLOCKS 3 AND 4 THE FEEDBACK GROUP EXPERIENCED AN INCREASE IN PERFORMANCE FOLLOWING A SHIFT TO A LOWER LEVEL OF ENVIRONMENTAL CERTAINTY. Following a Shift in the decision environment to more uncertainty, subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = 2.46 p > .06. 87 Table 16 (Continued) Hypothesis VII: Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly Change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: ALTHOUGH THE STANDARD ERROR BY SHIFT INTERACTION WAS SIGNIFICANT ACCORDING TO THE UNIVARIATE TESTS F(3,489) = 5.57 p = .001, THE MANOVA ANALYSIS PRECLUDED INTERPRETATION OF THIS UNIVARIATE RESULT. Hypothesis VIII:Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FEEDBACK BY FORECAST BY STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = .14 p > .93 88 Cotm Esp—Em cotm 23:8 02 m—. 5395”. 0 an. I .8020“. :26895 Cotm 23:me >m H2820". I 0:30.22 m 059.... No. 89 «HI . xombpomu. o—I No. I No. I > F No. No. 2. Z. 3330.... oz 8:092... £5 E x0330". I 6:2on m 059". 90 Interaction indicates that the significant interaction was due primarily to differences in the performance of subjects receiving feedback across blocks. Paired t-tests indicated that four of the six comparisons were significant (p < .01) for this group, while only one of the comparisons was significant for the control subjects. Table 17 - Matching-ANOVA Results presents all univariate results for the matching measure and Table 18 - Summary of Hypothesized Results for the Matching Measure provides a summary of the confirmation or disconfirmation of hypotheses, for this measure. W The three-way interaction of feedback, forecast, and standard error was significant F(1,163) = 7.29, p > .01. As can be seen in Figure 7 - Accuracy-Group Performance for all Combinations of Feedback, Forecast and Standard Error, this interaction is attributable to more accurate performance by subjects who received any of the decisons aids either alone or in their combinations as verified by Scheffe tests at the (.01) probability level. Observation of subjects who had no access to the feedback or the decision aids by the experimenter during data collection indicated that this group of subjects often responded with numerically low predictions of passenger miles, keeping Close to the bottom of the computer screen. Because the feedback about actual passenger miles, the forecasts, and the standards errors were near the center of the computer screen, these subjects probably scored 91 mmv. Sm. mm. m mo.H uanm an uouum pumpcmum pp ummoouom pp Hombommm mmm. mH. mo. m pH. umHCm pp uounm Cumocmum pp ummoouom «pm. pm. mm. m mp. DMHCm pp uouum oumccmum an Mombommm mHm. mH.H vv. m mm.H HHHnm an ummoouom pp xombomom pHo. om. mm. m pm. HHHCm an uouum Cumpcmum mmm. mp. pm. m Hm. HHHnm >Q ummomuom oHo. mm.m Nv.H m pm.v HHHCm an Hombomom ooo.H oo. co. m oo. uanm pm. mmv mm.mmH mHHOO :HCDHS m mo .mHm m m2 mo mm :oHumHnm> mo mousom muommmm Domnnsm :Hnqu mo mumoa Hem. mm.H mm.m H mm.~ pouum pumpcmum an ammomuom an Homnpmmm ooo. OH.mH pm.mm H p~.mm uouum ppmpcmum Ha ammomuom mmH. HN.N Hm.q H Hm.e uouum pumpcmum an xomhpmmm mmH. mp.H pe.m H pm.m ummomuom Hn xomhpmmm ooo. Np.pH mv.mm H mv.mm uouum Cumocmum ooo. Nv.mm mm.mm H mm.mm ummomuom mam. oo. oo. H oo. xomnommm mHm. Ho. No. H No. ucmumcoo mo.~ mmH mo.mmm MHHOU :Hnqu m mo .mHm m w: mo mm :oHumHum> Ho wousom muoommm mDOOpbsmncmozuom mo mummp. mDHsmmm <>oz¢ I mcHnoumz pH OHQMB Summary Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: 92 Table 18 of Hypothesized Results for the Matching Measure I: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? NO II: MAIN EFFECT FOR FEEDBACK FACTOR WAS NOT SIGNIFICANT F(1,163)=.01 p > .915 Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in which subjects have not been provided with valid forecasts. CONFIRMED? NO III: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. IV: Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. CONFIRMED? NO THREE WAY INTERACTION OF FEEDBACK, FORECAST, AND STANDARD ERROR FACTORS WAS NOT SIGNIFICANT F(1,163)=1.39 p > .241.. 93 Table 18 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. HYPOTHESIS CONFIRMED? NO COMMENTS: Hypothesis VI: THE FEEDBACK GROUP EXPERIENCED A DECREMENT IN PERFORMANCE FOLLOWING THE FIRST SHIFT TO LESS ENVIRONMENTAL CERTAINTY, BUT EXPERIENCED AN INCREASE IN PERFORMANCE FOLLOWING THE SECOND SHIFT TO LESS ENVIRONMENTAL CERTAINTY. FURTHERMORE, THE NO FEEDBACK GROUP EXPERIENCED A DECREMENT IN PERFORMANCE FOLLOWING THE SECOND SHIFT F(3,489) = 3.82 p = .01. Following a shift in the decision environment to more uncertainty, subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: Hypothesis VII: THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3, 489) = .73 p > .53. Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: Hypothes is VI I I THE STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = .60 p > .61 :Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, while subjects in all other conditions will either experience decrements or no change in their performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FEEDBACK BY FORECAST BY STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3, 489) = .10 p > .95 94 2330.2 58:08:. 55.052 0568.). onm 33.4 Home: o. EEO :_ .0208 .0835. coon 05... «03.9 208 N ”902 O x O x O x O x cotm Data—3m O O x x O O x x .3020“. O O O O x x x x x0330". - - L - n - n - p... I moo. | /\\// l O _.m. cotm .830ng HE: 3320". Joan—00050 20.353600 __< 3.... oo:aE:OtOn. H.390 I 38:84. p 059”. 95 poorly on the accuracy measure because their response estimates of actual passenger miles were more distant from the actual passenger miles than the response estimates of other subjects. Although the direct interpretation of three- way and other higher order interactions in general has not been recommended (Cohen & Cohen, 1983, p. 344-348), in this particular case it does appear that for the accuracy measure, feedback and the two decision aids supply similar information to subjects. The feedback by shift interaction was significant F (3, 489) = 4.61, p <.05. Paired t-tests indicated that during all time periods the accuracy measure differed within each of the two groups of subjects. The results presented in Figure 8 - Accuracy-Feedback by Shift Interaction show that both groups significantly changed over time. Because of the calculation of the accuracy measure, as the variance of feedback values increased (decreased), the variance of the differences between the subjects' response and the actual passenger miles also increased (decreased). This may represent a statistically based reason for these results. Furthermore, as mentioned earlier, some of the subjects in the groups who received no feedback did have access to information that was related to feedback values. Subjects who received forecasts and/or standard errors without feedback could have used, and probably did use, those sources of information as substitutes for feedback. Table 19 - Accuracy-ANOVA Results and Table 20 - Summary of oasooos. 32:03:. :9... .658 23322 mootsoo< Eomoa o. cooco :_ poaooo Cooao>oa :oon o>oc 32.9 29va No.02 .V m N ... 96 n — - —_ _ _.II 5.63666”. 62 8. .. .VN. I mm. l .. I. II m... I ’JI’IIIIII![\I\ . {_H mm mm. Reappooi Hm. 5.8825 firm .6 xoanooou. I 58:00.4 w 230.”. 97 oom. oH.H mm. m op. nnHom an nonnm onoooonm HQ umooonom an xoobpoom «Ho. oo. om. m om. nnHom so nonnm onoooonm so noooonon oNH. Ho.H oo. m NN.H nnnom no nonnm onoooonm an noonooon omH. op.H om. m oH.H nnHom so noooonon an goooooon mom. oo.H op.H m oo. nonom an nonnm onoooonm omo. mm. mo. m mm. noHnm an nmooonon moo. Ho.o pm.H m mo.m . nnHom so Hoonooon ooo. Ho. oo. m Ho. nnnom Hm. moo om.ooH oHHoo oHnnHz n no .mHm n m: no mo ooHnoHno> no oonoom. muoomwm noopnsm :Hnqu mo mummy ooo. om.p No.oH H ~o.oH nonnm onoooonm an umooonom HQ Hoobooom ooo. oo.om om.om H om.om nonnm onoooonm so noooonon ooo. om.HN No.oo H No.oo nonnm onooconm no noonooon omo. om.o oo.o H oo.o noooonon on noonooon ooo. om.mm po.oo H po.oo nonnm onoooonm ooo. mo.om Ho.om H Ho.om noooonon ooo. mm.mm om.oo H om.mo Hoonooon «op. HH. Hm. H Hm. noonoooo mo.H moH oo.mHm oHHoo oHonHz n no .on n m: no mo connoHno> no oonoom ouoowmm muoopbsmIcooznom mo muooa manoom ¢>oz< I moonsoofi oH oHnoo Summary Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: 98 Table 20 of Hypothesized Results for the Accuracy Measure I: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? NO SCHEFFE ANALYSIS OF SIGNIFICANT THREE-WAY INTERACTION OF FEEDBACK, FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES AMONG GROUPS RECEIVING ANY OF THESE FACTORS. Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. CONFIRMED? NO III: SCHEFFE ANALYSIS OF SIGNIFICANT THREE-WAY INTERACTION OF FEEDBACK, FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES AMONG GROUPS RECEIVING ANY OF THESE FACTORS. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO IV: SCHEFFE ANALYSIS OF SIGNIFICANT THREE-WAY INTERACTION OF FEEDBACK, FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES AMONG GROUPS RECEIVING ANY OF THESE FACTORS. Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. CONFIRMED? NO SCHEFFE ANALYSIS OF SIGNIFICANT THREE-WAY INTERACTION OF FEEDBACK, FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES AMONG GROUPS RECEIVING ANY OF THESE FACTORS. 99 Table 20 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. HYPOTHESIS CONFIRMED? NO COMMENTS: Hypothesis VI: BOTH FEEDBACK AND NO FEEDBACK GROUPS EXPERIENCED DECREMENTS IN PERFORMANCE FOLLOWING SHIFT TO INCREASED ENVIRONMENTAL UNCERTAINTY. Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: Hypothesis VII: THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = .50 p > .65. Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: Hypothesis VI II: THE STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = 1.06 p > .35 Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FEEDBACK BY FORECAST BY STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = 1.10 p > .34 100 Hypothesized Results for the Accuracy Measure are presented in order to report the results of the univariate tests of the accuracy measure and as a summary of hypotheses tests, respectively. Post-experimental TESL Qfi SNDjECIS' Learning As discussed earlier, evidence for subject learning of the underlying cue-criterion relationships in the simulation was to be substantiated by answering two questions:(1) Did groups significantly differ in their placement of cards in the five bins? and (2) Given that groups significantly differed in their placement of cards in the five bins, did groups predicted to have more knowledge of the statistical relationships in the simulation more closely approximate the actual frequency value for each of the five bins? Since forty-five of the entire sample of 171 subjects were pretest subjects who were later included in the lens model analysis but were not asked to do the card sorting task, data from 126 subjects were included in the MANOVA.ana1ysis of this data. As can be seen in Table 21 - Effect of Group Membership on Card Bin Mhans, the eight groups of subjects did not significantly differ in their placement of the cards in any of the five bins when attempting to described the seasonality indices' relationship with the actual passenger mile values or the price/promo indices' relationship with actual the actual passenger mile values. Because this requirement was 101 TAENHB 21 Effect of Group Membership on Card Bin Means Tests of Subjects' Knowledge of Relationship Between the Seasonality Cue and Actual Passenger Miles Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .29740 1.10219 35 610.00 .318 Hotellings T .33842 1.12548 35 582.00 .288 Wilks Lambda .72845 1.11489 35 498.81 .302 Roy's Root .16423 Tests of Subjects' Knowledge of Relationships Between the Price/Promo Cue and Actual Passenger Miles Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .24354 .89236 35 610.00 .649 Hotellings T .26732 .88904 35 582.00 .654 Wilks Lambda .77491 .89070 35 498.81 .651 Roy's Root .1I384 102 not met, analysis of the proximity of each group's mean responses to the actual bin value was not warranted. CHAPTER VI METHODOLOGY AND SECOND DATA COLLECTION E l] E . I i H'I] E . E I C J] I. A number of issues arose during the data collection and subsequent analysis of the first data collection. First, the number of trials (20) following each shift in environmental certainty may have included too few iterations for subjects to stabilize their performance following that shift. Previous research on the effects of shifts in environmental certainty on subjects' lens model performance has used substantially more trials before the shift (Dudycha, Dumoff, Dudycha, 1973; Lindberg & Brehmer, 1976; and Sneizek and Reeves, 1986). AS an example, Sniezek & Reeves' pre-shift and post-shift blocks consisted of seventy-five trials each. Although the results in the first data collection indicated Shift effects only in conjunction with the feedback factor, the argument could be made that given a more stable environment, shift main effects along with shift by forecast and shift by standard error interactions may have been found. Secondly, analysis of subjects' matching measure from the first data collection indicated that groups with access to the forecast did not fully utilize this decision aid. A plausible explanation for this behavior was that subjects did not believe that they had the option of responding with the forecast value as a prediction even though the instructions 103 104 were worded deliberately to not preclude this behavior. If subjects resisted inputting the forecast due to perceived demand characteristics of the design, a number of problems could have arisen concerning the interpretation of results from the first data collection. On the other hand, if subjects resisted inputting the forecast value as their decision for other reasons, such as the forecasts' deviations from feedback values, discussion of the hypotheses from a theoretical perspective would be more appropriate. Additionally, the post-experimental test of subject knowledge met with negative reactions from subjects. Most subjects, after they understood what they were required to do with the twenty-five cards and five bins, stated that they could not perform this task. With experimenter encouragement, all subjects did complete the task, but many subjects stated after its completion that they had guessed which card should be put in each bin. Because the previously discussed analyses indicated that there were no differences in performance on this task among groups, subjects' doubts about their ability to perform this task could not be ruled out as a potential cause of these subjects' performance at this task. The second data collection was designed to be a partial replication of the first data collection and an attempt to rule out the potential problems discussed immediately above. 105 Subjects Eighty-eight students enrolled in the business policy course at a midwestern university served as subjects for class credit. Care was taken to equalize the number of subjects in each of the four groups for this data collection. This was accomplished through the assignment of students who had missed their originally scheduled appointments to groups with the objective of equalizing the number of subjects in each cell. This attempt at equalization required that the last three students who rescheduled their appointment were given alternate assignments and were not used as subjects in this data collection. W Subjects were randomly assigned to one of four sets of conditions: two forecast conditions (forecast/no forecast) and two standard error conditions (standard error/no standard error) with repeated measures on each of the dependent variables. In this experiment, all subjects received the actual criterion value (feedback) after each trial. The decision to give feedback to all subjects was made for a number of reasons. First, the differences between feedback and no feedback groups in Data Collection I were quite robust and mirrored previous lens model findings for these groups. Second, the underlying purpose of the first data collection was to study the effects of decision aids on subjects 106 responses to feedback which contained random error, although the MANOVA model in which they were tested required a full factorial design. The four "no feedback groups" therefore served primarily as controls in the first data collection. Given these considerations, the decision was made to test subjects in a two (forecast/no forecast) by two (standard error/no standard error) by two (environmental certainty) by four (blocks) factorial design. Environmental certainty and the four blocks represented within subjects factors. In this analysis the block factor was nested within the environmental certainty factor. Subjects were required to predict the total weekly passenger miles for a hypothetical airlines based on two meaningfully labled cues. In this data collection the multiple R of the two blocks within the pre shift trials was .80 and the multiple R of the two blocks within the post shift trials was .51. The equivalence of the two blocks within the shift factor was assured by repeating the same cue-criterion combinations in the respective adjacent blocks. No subjects reported noticing this repetition during or following collection of data. Intercorrelations between the cues were .22 for both of the first set of two blocks and .30 for the second set of two blocks. The correlation between the first cue and the actual passenger miles for the pre shift and post-shift periods were .51 and .27 respectively. The correlations between the second cue and the actual passenger miles were -.46 and -.30, respectively. 107 In order to ameliorate the perception that the decision aids were not valid predictors of the actual passenger miles, all subjects were required to compute a least squares regression equation for the simulation based on the first twenty cue-criterion pairs in the simulation, although they were told that these cue-criterion pairs were from the preceding twenty weeks (i.e., before they took over simulation). Using a micro-computer statistical package, subjects were shown a computer screen with the cues and actual passenger miles data on the screen. They were then informed of the statistical package commands needed to complete the regression analysis. After completing this analysis, subjects were shown the multiple R (.80) from the regression equation on the computer screen, but not the regression weights. The experimenter told the subjects that for the real world, a multiple R of .80 was good. Subjects who were to receive the forecast/standard error information were further told, "The forecast/standard error that you will be using throughout the simulation is based on the regression equation you just developed. I, (the experimenter), guarantee you that the forecast/standard error you will see during the simulation is valid. It will help you predict the actual passenger miles." Subjects receiving only feedback with no decision aids were told, "A multiple R of .80 is good, I quarantee you that the cues will help you predict the actual passenger miles in the simulation." Appendix F 108 contains the instructions for the statistical package and decision-making simulation. A post-experimental test (Appendix G) was given to each subject after the completion of the simulation in order to measure the extent to which subjects learned the directionality (positive/negative) of the cue-criterion relationships. This post-experimental test was deliberately designed to be less demanding on the subjects' cognitive processes than the post-experimental test following the first data collection. W As in the first data collection, multivariate analysis of variance (MANOVA) was used to assess the simultaneous effects of the factors on the four performance measures, followed by separate univariate analysis of variance on each of those dependent variables (achievement, consistency, matching, accuracy). As in the first data collection, the MANOVA analysis was not designed to be a test of individual hypotheses, but rather a necessary and preliminary analysis of linearly combined dependent variables directed at providing protection against Type I error. The simplified post-experimental test of subject knowledge of cue—criterion relationships was analyzed using one-way analysis of variance on the total number of correct responses for each subject. CHAPTER VII RESULTS MANQMA_BesnlLS As was true in the first data collection, stem and leaf plots indicated that the data for each variable were skewed. In addition, the Bartlett-Box F test for univariate homogenity of variance for each dependent variable and the multivariate Box-M test for homogeniety of dispersion matrices were highly significant (p > .001). Since cell sample sizes were equalized during data collection, the Pillia-Bartlett trace was used as the primary test for multivariate significance of data with unequal covariance matrices (Olson, 1974, 1976). Orthonormal transformations before data analysis ameliorated the problem associated with skewed data. Appendix H has been provided to show the means and standard deviations of the data, Appendix I presents the correlation matrix for the dependent variables and Appendix J has been provided to show the stem and leaf plots for the dependent variables. Significant effects were found for the forecast factor on the four linearly combined multivariate dependent variables (p < .001). Correlations between dependent and canonical variables indicated that the achievement measure accounted for the most variance shared with the underlying 109 110 composite (-.98), although as can be seen in Table 22 - Effects of the Forecast Factor On Linearly Combined variables, the other dependent measures also shared a substantial amount of variance with that composite. In addition, the forecast by standard error treatment factors were significant (p < .01) according to all three multivariate test statistics. Table 23 - Effects of the Forecast by Standard Error Interaction On Linearly Combined variables indicates both these results and the discriminant function coefficients and canonical correlations for this multivariate interaction. No significant effects were associated with the standard error, block, or shift factors either alone or in combination with each other. Table 24 - Testable Hypotheses Using Univariate ANOVA Based On Preliminary MANOVA Analysis presents the hypotheses which may be tested using univariate methods based on the above multivariate results. E 1. | H . . | E 1| The interaction of the forecast and standard error treatments was also significant for the achievement measure F(3,84) = 13.24 p < .001) . Scheffe's tests indicated that the control group's mean was significantly lower than the groups receiving either or both of these treatments. Similar to the 111 Table 22 Effects of the Forecast Factor on Linearly Combined Variables Multivariate Tests of Significance (S = l, M = 1, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .28 7.94 4 81 .000 Hotellings .39 7.94 4 81 .000 Wilks Lambda .72 7.94 4 81 .000 Roy’s Root .28 Raw discriminant function coefficients Function No. Variable 1 Achievement -.597 Consistency .021 Matching -.103 Accuracy .134 Standardized discriminant function coefficients Function No. Variable 1 Achievement -.753 Consistency .030 Matching -.147 Accuracy .178 Correlations between dependent and canonical variables Canonical Variable Variable l Achievement -.988 Consistency -.786 Matching -.809 Accuracy .891 Estimate of effect -.611 112 Table 23 Effects of the Forecast by Standard Error Interaction On Linearly Combined Variables Multivariate Tests of Significance (S = l, M = l, N = 79) Test Name Value F Hypothesis Error Sig. Degrees of Degrees of Freedom Freedom Pillais-Bartlett Trace .19 4.61 4 81 .002 Hotellings .23 4.61 4 81 .002 Wilks Lambda .81 4.61 4 81 .002 Roy's Root .19 Raw discriminant function coefficients Function No. Variable l Achievement .185 Consistency -.243 Matching -.021 Accuracy .684 Standardized discriminant function coefficients Function No. Variable l Achievement .234 Consistency -.347 Matching -.029 Accuracy -.909 Correlations between dependent and canonical variables Canonical Variable Variable 1 Achievement -.832 Consistency -.821 Matching -.663 Accuracy .979 Estimate of effect .466 113 Table 24 Hypotheses That May Be Tested By Univariate ANOVA's Based On Preliminary MANOVA Analysis Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: Hypothesis UNIVARIATE COMMENTS: I: TEST WARRANTED? II: TEST WARRANTED? III: TEST WARRANTED? IV: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. NO FEEDBACK WAS NOT A.FACTOR IN THIS DATA COLLECTION. Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. YES SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS F(4,81)=4.60 p < .01 MAX INCLUDE CONFIRMATION OF MAIN EFFECTS FOR THE FORECAST FACTOR. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. YES SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS F(4,81)=4.60 p < .01 MAX INCLUDE CONFIRMATION OF MAIN EFFECTS 'FOR THE STANDARD ERROR FACTOR. Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with forecasts, standard errors and feedback. TEST WARRANTED? NO FEEDBACK WAS NOT A FACTOR IN THIS DATA COLLECTION. 114 Table 24 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. UNIVARIATE TEST WARRANTED? NO COMMENTS: FEEDBACK WAS NOT A FACTOR IN THIS DATA COLLECTION. Hypothesis VI: Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. UNIVARIATE TEST WARRANTED? NO ' COMMENTS: NO INTERACTIONS INVOLVING THE FORECAST AND SHIFT FACTORS WERE SIGNIFICANT ON MULTIVARIATE TESTS. Hypothesis‘VIIzFollowing a shift in environmental uncertainty, subjects who have not been provided with standard errors will not significantly change their performance, whereas subjects who have been provided with standard errors will experience lowered performance. UNIVARIATE TEST WARRANTED? NO COMMENTS: NO INTERACTIONS INVOLVING THE STANDARD ERROR AND SHIFT FACTORS WERE SIGNIFICANT ON MULTIVARIATE TESTS. Hypothesis‘VIII:Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. UNIVARIATE TEST WARRANTED? NO COMMENTS: FEEDBACK WAS NOT A FACTOR IN THIS DATA COLLECTION. 115 results found in the first data collection, these results indicated the redundancy of these decision aids' effects on the achievement measure. This interaction is presented in Figure 9 - Achievement-Forecast by Standard Error Interaction. and Appendix K presents all univariate relationships in lens model measures. Table 25 - Achievement-ANOVA Results presents all univariate results for the achievement measure. In addition Table 26 - Summary of Hypothesized Results for the Achievement Measure reports confirmation or disconfirmation of each hypothesis. As with the achievement measure, an interaction of the forecast and standard error treatments was significant F(3,84) = 12.89; p < .01). Scheffe's test indicated that groups receiving the forecast and the forecast with the standard error outperformed the control group on the consistency measure at the .01 level of significance, as shown in Figure 10 - Consistency-Forecast by Standard Error Interaction. The subjects receiving only the standard error did not significantly differ from any other group. At least on the consistency measure, it appears that the presence of a forecast was the important factor in determining high levels of performance. Table 27 - Consistency-ANOVA Results shows the univariate results for the consistency measure, and Table 28 116 voo. m>.o mv.m H mv.m Hoenm oumocmum ma uneconom «mo. mm.m mm.m H mm.m nanm an “chum nnmucmnm omo. o>.m vm.m H vm.m DMHnm >b ammomuom oo.H oo. oo. H oo. uanm mo. vm mm.mm mHHmO cHnqu m mo .mHm m m2 mo mm COHumHnm> mo mousom How. mH. no. H no. xoon an uouum oumocmum an uneconom «no. om. oo. H oo. xoon an uouum oumocmum va. mm.~ mm. H mm. HUOHm sh ammomuom. ooo.H oo. oo. H oo. mEHe om. em oH.om mHHmO cHnqu m mo .mHm m m: we mm COHumHum> mo mousom muoommm muommbsmucHnqu mo momma ooo. m~.mH mo.HN H mo.HN “chum oumnamum an ummomuom moo. mm.m vm.m H em.m Houum oumocmum ooo. oH.Nm om.Hm H vN.Hm ummoouom ooo.H oo. oo. H oo. ucmumcoo mm.H em m>.mmH mHHoO cHsqu m mo .mHm b m2 mo mm COHumHum> mo mousom muoomwm muownbsmucmmzuom mo mumme muHSmmm ¢>oz< I ucoEo>mHno< mm wanes 117 uuHrm an Hoon owe. me. Ho. H Ho. an uouum nuancmum an nmmomuom Hmo. mo. No. H No. DMHQm an HOOHm mm uouum onmocmum me. mo. mo. H mo. DMHBm >2 xoon an ammomuom oo.H oo. oo. H oo. DMHbm >b Hoon Hm. vm mm.mv mHHmO cquHz m mo .mHm m m2 mo mm :oHanum> mo mousom HomsaHucooo mm mHnme 118 totem oaocsm Loam unsung oz No. I mh. I 689.0". 02 Hmaoeaom 53082:. harm otaocme >m $320“. I 2.0805294 m 239". .otm oatcflm _ totm 98286 02 119 50. I .8080“. 02 POI 9. H8095”. c2882... norm Bap—5m >m .8020... I 3:033:00 H: 2:9”. 120 Table 26 Summary of Hypothesized Results for the Achievement Measure Hypothesis HYPOTHESIS Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS I: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? NOT TESTED II: Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. CONFIRMED? NO III: SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS INCLUDED NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS INCLUDED NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. CONFIRMED? NOT TESTED 121 Table 26 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. HYPOTHESIS CONFIRMED? NOT TESTED Hypothesis VI: Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = 3.76 p > .05. Hypothesis VIIuiFollowing a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = 3.83 p > .05. Hypothesis‘VIII:Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, while subjects in all other conditions will either experience decrements or no change in their performance. HYPOTHESIS CONFIRMED? NOT TESTED 122 mmH. vm.m Ho.N H Ho.m Houum oumocmum an unmomuom hoe. HH.m eo.m H eo.m nuHrm an uouum eumncmnm emu. me. em. H em. nuHhm sh nmmomnom oo.H oo. oo. H oo. nuHrm om. Hm mm.om mHHmo cHnnHz m H0 .mHm m w: mo mm coHannm> mo monsom mmH. H>.H mm. H mm. Hoon an nonnm nuancmnm sh nmmomnoe boo. em. mo. H mo. Hoon an uonnm uumncmnm. emu. ho. Ho. H Ho. xoon >b ummomuom ooo.H oo. oo. H oo. meHa Hm. em be.eH mHHmo :HrnHz m we .mHm m m2 mo mm coHannm> no monsom muommmm muomnbsmnchqu mo mumps Hoo. mm.NH mH.mN H mH.mN uounm unmncmnm an nmmomuom mew. mm. mH.H H mH.H Houum oumccmum ooo. vm.o~ mm.Hv H mm.Hq nmmumuom ooo.H oo. oo. H oo. ncmnmcoo mo.m Hm Ne.oeH mHHmo :HnnHz e no .aHm m m2 mo mm coHannm> Ho monsom muoommm movemQSmlcoozumm mo mummy manmom <>oz¢ I >ocmumHmcoo hm wanme 123 uuHam an Hoon emm. we. Hm. H Hm. an uounm nuancmnm an nmmomuom Hmm. Ho. Ho. H Ho. nuHrm an Hoon an Honum nuancmnm amp. HH. mo. H mo. uanm an Hoon an nmmomuom oo.H oo. oo. H oo. nHHrm an HUOHm He. em Hm.vm mHHmo cHrnHz e no .mHm m m2 mo mm :oHannm> mo monsom Humscanoo. em mHhmH 124 Table 28 Summary of Hypothesized Results for the Consistency Measure Hypothesis HYPOTHESIS Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS I: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? NOT TESTED II: Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. CONFIRMED? NO III: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO IV: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision—making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors, and feedback. CONFIRMED? NOT TESTED 125 Table 28 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. HYPOTHESIS CONFIRMED? NOT TESTED Hypothesis VI: Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = .45 p > .50. Hypothesis VII} Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = 3.44 p > .02 Hypothesis‘VIII:Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. HYPOTHESIS CONFIRMED? NOT TESTED 126 - Summary of Hypothesized Results for the Consistency Measure reports confirmation or disconfirmation of each hypothesis in light of the above results. The interaction effects of the forecast and standard error treatments were also significant for subjects' matching F(3,84) = 8.42; p < .01. Scheffe's test indicated that subjects receiving the forecast and the forecast with the standard error significantly outperformed subjects receiving only the standard error and subjects receiving neither of the two treatments (Figure 11 - Matching-Forecast by Standard Error Interaction). These represent results similar to those found for the consistency measure above. Again it appears that the presence of the forecast is the important factor in determining relatively higher performance on the matching measure. TabLe 29 - Matching-ANOVA Results indicates the univariate Manova results for the matching index and Table 30 - Summary of Hypothesized Results for the Matching Measure reports on the confirmation or disconfirmation of hypotheses in light of these results. 2 fl . . | E 1| The forecast by standard error interaction was significant F(1,84) = 18.31; p < .001 for the accuracy measure. As presented in Figure 12 - Accuracy-Forecast by Standard Error Interaction, all groups outperformed the 127 norm Empcflw Loam Enocflm oz KI 5320.... 02 Ho. RN. 6320". 22882:. totem Eaucflm >m $320“. I 0:20.22 H.059". 128 norm panama"... term 235%. oz mm. I 532.0". 02 OF. E. .3020”. E. 52099:. .otm Enocgm >m $390”... I 38:86. N. 2:9”. 129 mom. ow. HH. H HH. Houum oumocmum an ummomuom ooH. oo.H oo.H H oo.H nuHom on uouum nuancmnm mom. oo.H oo. H oo. HHHnm on nmmomuom oo.H oo. oo. H oo. noHnm oo. oo oo.mm mHHmo cHnnHz m mo .mHm m m: we mm coHanHm> mo monsom mom. Ho. «a. H vs. Hoon on Houum nnmncmnm on ammomuom omH. om.~ mH.H H mH.H xuon on uouum nuancmnm. mHo. oH.o oo.~ H oo.~ HUOHm on nmmomnom ooo.H oo. oo. H oo. msHe oo. oo om.ov mHHmo cHrnHz m H0 .mHm m m: on mm coHanum> Ho monsom mnomuum whomflnsmucHanz Ho momma moo. mo.o oo.oH H oo.oH nonnm nuancmnm on ammomuom moo. mm.m mH.o H mH.o uouum nuancmnm ooo. mm.Hm om.mo H om.mv nmmomnom ooo.H oo. oo. H oo. namumcoo Ho.~ Ho oo.ooH mHHmo cHnnHz m we .mHm m m2 mo mm coHHmHnm> Ho monsom muoommm muooflbomucomzumm mo mummy manmmm ¢>oz< I mcHronmz mm manna 130 nHHsm on Hoon ooo. oH. oo. H oo. an nouum nuancmnm on nmmomnom ooo. HH. so. H Ho. nanm on HUOHm on gonna nuancmnm How. mm. oo. H oo. HHHnm on Hoon on nmmomuom oo.H oo. oo. H oo. HMHnm an EOOHm om. vm om.om mHHmU GHSDHB o co .mHm m m: we mm coHanum> Ho monsom HomocHucoov mm mHQmB Summary Hypothesis HYPOTHESIS Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS 131 Table 30 of Hypothesized Results for the Matching Measure I: Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? NOT TESTED II: Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. CONFIRMED? NO III: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO IV: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF FORECAST AND STANDARD ERROR FACTORS FOUND NO SIGNIFICANT DIFFERENCES BETWEEN THE GROUP RECEIVING THE FORECAST WITH THE STANDARD ERROR AND THE GROUP RECEIVING NO FORECAST WITH THE STANDARD ERROR. Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. ‘ CONFIRMED? NOT TESTED 132 Table 30 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. HYPOTHESIS CONFIRMED? NOT TESTED Hypothesis'VI: Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = 1.22 p > .27. Hypothesis VII} Following a shift in environmental uncertainty, subjects who have not been provided with standard errors will not significantly change their performance, whereas subjects who have been provided with standard errors will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = 2.63 p > .10 Hypothesis‘VIII:Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. HYPOTHESIS CONFIRMED? NOT TESTED 133 controls on the accuracy measure. Table 31 - Accuracy- ANOVA Results shows the univariate results of this analysis of the accuracy measure and Table 32 - Summary of Hypothesized Results for the Accuracy Measure indicates the confirmation or disconfirmation of each hypotheses. Enst_Experimental_Test_of_Subiectsl Knmledge Means for the test of subjects' knowledge of the sign of the relationship between each of the cues and the criteria were analyzed using one way analysis of variance. This analysis indicated that there were no significant differences among the groups F(3,81) = .98; p > .40. Given that a score of 4 would represent a chance level of response, u was calculated for each mean. This analysis indicated that all means were significantly higher than chance. Table 33 - Means and Standard Deviations For Group Knowledge Task shows the means and standard deviation for each group. 134 ooo. om.m oo.H H oo. nonnm onmocmnm on nwmomuom mmo. NH. mo. H oo. uuHcm an Houum oumocmum ovm. om. mH. H mH. HMHnm an unmoouom oo.H oo. oo. H oo. DHHbm mm. vm m>.mv mHHmU cHnuHZ m mo .mHm m m: we mm coHumHHm> mo mousom vmo. Ho. oo. H oo. Hoon Ha nouum unmncmnm an ammomnom. Hom. vm. «H. H vH. xoon we uonum cumocmum moo. om.v oo.H H oo.H HOOHm an unmomuom ooo.H oo. oo. H oo. mEHe Hv. om o>.vm mHHoo chqu m mo .mHm m m: we mm coHumHHm> mo monsom muommmm muooflbsmIchqu mo momma- ooo. Hm.oH mm.mm H mm.~m Honnm unmncmnm Hn nmmomnom omo. HH.m Ho.m H Ho.m uouum oumocmum ooo. mH.oN mH.ov H mH.ov unmomuom ooo.H oo. oo. H oo. ucmumcoo oo.H vm mN.ovH mHHmU Combos m mo .mHm m m: mo mm COHumHHm> mo mousom muoowmm muooflbsmncmozumm mo momma muHsmom m>oz< I >0musoo¢ Hm OHQMB 135 uqum an Hoon mvm. om. mm. H mm. mm Houum cumocmum an ummoonom who. mH. oo. H oo. DMHnm an HoOHm an Houum oumocmum «mo. NH. vo. H vo. HHHHm an Hoon we unmoouom oo.H oo. oo. H oo. Hmonm an xoon mm. em nw.mm mHHmO chuHB m mo .mHm m m2 mm mm coHumHHm> mo mousom Humscanoov Hm mHnme Summary Hypothesis HYPOTHESIS Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS COMMENTS: Hypothesis HYPOTHESIS 136 Table 32 of Hypothesized Results for the Accuracy Measure Decision-making performance in conditions in which subjects have not been provided outcome feedback will be superior to decision-making performance in conditions in which subjects have been provided outcome feedback. CONFIRMED? NOT TESTED II: Decision-making performance in conditions in which subjects have been provided with valid forecasts will be superior to performance in conditions in subjects have not been provided with valid forecasts. CONFIRMED?-NO III: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF THE FORECAST AND STANDARD ERROR FACTORS FOUND THAT SUBJECTS RECEIVING ANY OF THESE FACTORS OUTPERFORMED THE CONTROLS. Decision-making performance in conditions in which standard errors of valid forecasts have been provided to subjects will be superior to decision-making performance in which no standard errors have been provided to subjects. CONFIRMED? NO IV: SCHEFFE ANALYSIS OF SIGNIFICANT TWO-WAY INTERACTION OF THE FORECAST AND STANDARD ERROR FACTORS FOUND THAT SUBJECTS RECEIVING ANY OF THESE FACTORS OUTPERFORMED THE CONTROLS. Decision-making performance for subjects who have been provided with forecasts, standard errors, and feedback will be higher than for subjects who have not been provided with the combination of forecasts, standard errors and feedback. CONFIRMED? NOT TESTED 137 Table 32 (Continued) Hypothesis V: Following a shift toward more environmental uncertainty, the decision-making performance of subjects who have not been provided with feedback will not significantly change, whereas the performance of subjects who have been provided with outcome feedback will decrease in response to the increased variability of the decision environment. HYPOTHESIS CONFIRMED? NOT TESTED Hypothesis VI: Following a shift in the decision environment to more uncertainty subjects who have been provided forecasts will not significantly change their performance, whereas subjects who have not been provided with forecasts will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FORECAST BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = .37 p > .37. Hypothesis VIIR Following a shift in environmental uncertainty, subjects who have been provided with standard errors will not significantly change their performance, whereas subjects who have not been provided with standard errors will experience lowered performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(1,84) = .12 p > .73 Hypothesis‘VIII:Following a shift toward more environmental uncertainty, subjects who have been provided with a combination of forecasts, standard errors and feedback will improve their performance, whereas subjects in all other conditions will either experience decrements or no change in their performance. HYPOTHESIS CONFIRMED? NO COMMENTS: THE FEEDBACK BY FORECAST BY STANDARD ERROR BY SHIFT INTERACTION WAS NOT SIGNIFICANT F(3,489) = 1.10 p > .34 138 Table 33 Means and Standard Deviations For Group Knowledge Task Group Mean Standard N Deviation Forecast/Standard Error 5.86 1.88 22 Forecast/No Standard Error 6.77 1.66 22 No Forecast/Standard Error 6.43 1.71 22 No Forecast/No Standard Error 6.18 1.96 22 CHAPTER VIII SUMMARY AND DISCUSSION OF RESULTS EEE | E E . . E'l : E . . _ I. E E The results from the first data collection discussed above have indicated that providing subjects with either of the two decision aids before each decision trial significantly improved their decision-making performance, on the achievement, consistency, matching, and accuracy dependent measures used in these analyses. Based on the results from the first data collection it appeared that the effects of the forecast and standard error on subjects' decision-making performance were redundant, in that subjects with either or both of these two decision aids were not significantly different in their performance on any of the dependent variables measured in the first data collection. Still referring to the first data collection, results for the accuracy measure indicated that the effects of both the forecast and standard error were also redundant with the effects of outcome feedback. These results were interpreted as indicating that the optimal decision aids provided much the same information as did feedback about the values of the criterion, although the decision aids were provided before each decision trial and outcome feedback was provided after the decision trial. At the same time, subjects may have been 139 140 utilizing the decision aids or forecast in different ways, but this study did not identify those differences. These general findings were disappointing at least from perspective of the stated hypotheses of this study. A three- way interaction was predicted in which the standard error would provide subjects with information that would desensitize them to the variability of feedback values and the forecast would provide subjects with information from a least squares regression model about the task structure of the relationship between the cues and criterion values. This interaction did not occur in this study. Furthermore, the significant two way interactions of the forecast and standard error treatment factors indicated that either of these were substitutes for each other in the first data collection, although in the second data collection subjects who received only the standard error did not significantly outperform the controls on the consistency and matching measures. This was interpreted as indicating that the standard error was more useful to subjects when environmental change was large and relatively frequent, as it was in the first data collection. Significant effects for the within-subjects shift factor on decision-making performance were found in the first data collection and were associated with the presence or absence of feedback for all four measures of decision-making performance. Shift effects on decision-making performance 141 were not found in the second data collection when feedback was provided to all subjects. . . iIbgecfS_InCazpQz?IianEQ£EIhe_EQrecaSL_ln Stong evidence from both data collections indicates that subjects utilized the forecast in their responses. This finding is neither new nor surprising from a logical perspective or previous empirical research (Sneizek & Reeves, 1986). That subjects did not incorporate the forecast more in their responses represented an interesting phenomenon. In particular, the procedures used in the second data collection were designed to minimize subjects' resistance to using the forecast as an input response. Subjects calculated the regression equation for the forecast or standard error and were shown the multiple R of .80. It was explained to subjects that a multiple R this high would provide good estimates of the actual criterion value. Regardless of these manipulations, subjects only partially utilized the forecast value. As an example, in the second data collection, the group receiving only the forecast and feedback values attained an mean matching score of approximately .91 across the four blocks. This represented the highest mean for any group in either of the two data collections. Since the matching measure represented the correlation between the least squares predictions for each trial from the linear equation describing the subjects' judgments and the forecast for each trial, with perfect matching being represented by a correlation of 1.00, these scores indicated that subjects did 142 rely heavily, but not totally on the forecast. It must also be remembered that there was small to moderate intercorrelation between the cues (between .20 and .30) and this has been the attributed cause of artificially high matching scores in past research (Brehmer, 1976). In addition, if subjects used a strategy in which they avoided inputting the forecast, but instead added a constant to the forecast and input that as their estimate, the matching measure would also have been inflated, because of the correlational nature of this index. Although in this study one cue was positive and one cue was negative and this complicated the task. One potential reason that subjects may have not utilized the forecast, despite the manipulations in the second data collection directed at encouraging them to use it, was related to the demand characteristics of the simulation itself. In simulation data collections such as those discussed above, it may not have been possible to convince subjects that inputting the forecast value as their response was acceptable. Subjects were told that they were the load planners for the hypothetical airline and it was their job to make predictions about future passenger miles. Given this fact, it may have been difficult for subjects to have believed that the forecasts' predictions rather than their own were expected. Removing this implied need for a response other than the forecast would seem to present a difficult task in laboratory studies such as these. 143 On the other hand, it was also difficult to believe that demand characteristics explained all of subjects' resistance to total utilization of the forecast value. Subjects receiving the forecast and feedback rarely encountered trials in which the outcome feedback values and forecast for that trial were equal. An optimal decision aid with outcome feedback apparently also confused subjects and led them to change strategies. For subjects in the first data collection who received only a forecast with neither outcome feedback or the standard error, the statement that "the whole trick is to decide which variables to look at and then know how to add." (Dawes and Corrigan, 1974, p. 105) did not appear to have been a strategy that subjects chose to follow. Because the forecast represented an additive function of the equally weighted positive and negative cues along with a constant, a matching score mean of .75 across the four blocks does not seem impressive, given that the two cues and the forecast were the only information available to these subjects during the simulation. .A motivational factor may have caused decision makers' resistance to incorporating information from predictive models in their decision-making behaviors in this study. Dawes (1976) has identified a factor which he termed cognitive conceit. It has appeared that decision makers often assume that their own decision-making abilities are superior to those of predictive statistical models. Einhorn 144 and Hogarth (1987) have recently discussed a situation in which subjects prefer their own predictive guesses over more predictive strategies based on a very simple probability rule that could be discovered by counting frequencies in a dichotomous decision task. In the present research, the experimenter asked a number of the subjects in both data collections why they had not used the forecast as their input response. In the first data collection, the most common response from subjects was that they believed that they should not have used the forecast on every trial due to demand characteristics. In the second data collection, the most common response was that the forecast did not seem to be correct and they believed that they could make better predictions than the forecast. It did appear that cognitive conceit on the part of subjects was a factor in at least the second and possibly the first data collection, although how much this factor affected subjects performance was impossible to measure in these experiments. 5 1' I I l' :E I] SI I I E I I] . E . . -M 1' Berformance Recently, Doherty and Balzer (1988) have discussed what they consider to have been a surprising gap in lens model research. This gap was described as the failure of investigators to develop means of representing uncertainty for subjects in lens model tasks, especially in light of Brunswick's emphasis on the probabilistic nature of the environment. They noted that 145 ..... None has presented, for example, error bands around estimates of parameters. Point estimates of overall error have been represented (e.g., Re and RS or trans-formations thereof) as well as the scatter about least square function forms. But no investigation has represented the error associated with estimates of ecological validities, utilization coefficients, Rs, ra, etc. It seems to us that veridical means of representing uncertainty might have a significant impact on people, especially given the widespread assumption that probabilistic environments are deterministic. (p. 185). This study has now filled this gap in lens model research, with standard error information presented to subjects in both data collections of this study. Subjects in the first study did appear to incorporate the guidelines provided by the standard error in their decision-making performance as evidenced by the significant two-way interactions of the forecast and standard error. In these interactions, Scheffe's tests indicated that subjects who had access to the standard error exhibited higher performance than subjects who received no decision aids on the achievement, matching, and consistency measures. In the second data collection, the group receiving only the standard error did not significantly differ from any of the other three groups on the matching and achievement 146 measures. These results may lead to the conclusion that the standard error does improve performance on the consistency and accuracy measures, although not greater than the forecast alone or the combination of the forecast and standard error together. It was hypothesized that the standard error would give subjects information about the variability of the environment. Given the results of the standard error on the consistency performance measure, the possibility that this occurred cannot be ruled out, although a range estimate, such as the standard error, was not superior to a point estimate, such as the forecast. Subjects in this study were relatively statistically sophisticated, over 70% of the subjects were either accounting or finance majors. This fact, along with the experimenter's verbal instructions over and above the written instructions provided to subjects about the statistical information represented by the standard error did not logically lead to the possibility that subjects did not have an understanding of the standard error's purpose as a decision aid. The lack of even additive effects of the standard error and forecast decision aids, much less the multiplicative interactions hypothesized, can be seen as evidence that subjects did not utilize this decision aid as completely as expected. 147 . I . ihbgecfiS_TIncofanfatiog_0€_fieedback_ln Outcome feedback was a factor only in the first data collection. The effects of outcome feedback on consistency was the sole main effect found in this study and was attributable to superiority in performance of the subjects who did not receive outcome feedback in comparison to the performance of subjects who did receive outcome feedback. A number of previous studies have found similar results. Feedback did appear to confuse subjects and cause them to change strategies. This main effect for the consistency measure and the lack of interaction effects between feedback and the forecast or feedback and the standard error indicated that the combination of these decision aids with outcome feedback are not sufficient to overcome the negative effects of this factor on decision makers' consistency. The outcome feedback factor did interact with the shift factor in the first study for all four dependent variables measures. For the achievement measure, subjects provided with outcome feedback actually improved their performance following the first shift to less environmental certainty. This may have been because improved matching by these subjects which overcame this group's less consistent decision behaviors. Accuracy also improved after the first shift to less environmental certainty. These improvements in the achievement, matching, and accuracy measures following the shift to less environmental certainty are not explainable 148 from a theoretical perspective, but may be attributable to the the relatively high environmental predictability in the first block (multiple R = .93) and the relatively frequent and severe changes in shift. WW Knowledge MANOVA results for the post-experimental test of subjects' knowledge following the first data collection indicated that groups did not differ in their placement of cue-criterion realtionships into bins representing the frequency of their occurrence. Based on this test, no statements about actual learning of the relationship between the cues and criterion were made. The simplified test of subjects' knowledge used in the second data collection was designed primarily to see if subjects were able to express the signs (positive/negative) of the respective cues as they related to criterion values. Again there were no significant differences between groups on this test, but all groups exhibited knowledge above a chance level. Evidence that certain groups of subjects learned the relationships between the cues and criterion was necessary to show that these subjects were actually learning rather than simply following the forecast value or remaining within the standard error. Given the results of the test of subjects knowledge from the first data collection, a statement cannot be made about learning on the part of subjects. That all 149 groups significantly performed above a chance level in the second data collection may at least have indicated that they accomplished some level of learning about the relationships between the cues and criteria in the simulation. Brehmer and Brehmer (1988) have identified a number of categories of issues that have been found to be important in assessing the validity and generalizability of policy capturing studies. The issue of judges' previous experience with similar judgment tasks has represented the first of these categories. As Brehmer and Brehmer (1988) have pointed out, in most lens model studies, judges are often college students, and these subjects often have had no real experience at the experimental judgment task. These subjects must approach the decision task without having developed any a priori judgment policy. In these types of experiments, the subjects must engage in a process that Brehmer (1988) has termed policy construction. Essentially naive subjects cannot have been expected to have previously learned elements of the task structure such as the distribution of actual cue values or intercorrelations between cues. Therefore, the representativeness of the sample for the validity and 150 generalizability of the results represents an important issue in lens model research. In the two data collections discussed in this research, the subjects were graduating seniors in a school of business, and as far as the experimenter assumed, had no experience as a load planner for an airline, if such a job even exists. Because the cues and criteria were calculated from a statistical package, it was highly unlikely that any of the subjects had seen these cues or criteria before the simulation. Because of this, it was impossible to identify when subjects were nnnstrnnting policies or applying policies. Therefore, the generalizability of these results to managerial decision-making situations in which experienced judges make decisions was limited. The second category of issues discussed by Brehmer and Brehmer (1988) again is associated with the issue of representativeness, but in this case the issue is representativeness of the task. Phelps and Shanteau (1978) have found that experienced judges were able to incorporate information from between nine and eleven dichotomous cues in their judgments. In the above discussed experiments, two linear metric cues were presented to subjects for each of their decision trials. It is admitted that this was a very limited number of cues and relatively simple task structure to have been a representation of an actual decision task. 151 Furthermore, the simulation was just that: a simulation. There were no claims made in this study that the decision trials were identical to those made by managers in business. In particular, the design of the simulation and the experiment did not allow for a realistic time frame between decisions. Although the decision trials represented quarters in the simulation, subjects input decisions within seconds of completing the previous trial. If decisions had been made with longer intervals of time between periodicy, results may have differed. Finally, Brehmer and Brehmer (1988) have discussed the validity of the linear model as a general theory of human judgment. They concluded that there does not appear to have been sufficient evidence that linear models accurately describe the processes of human decision-making. Some studies have established that subjects exhibit nonlinear decision—making strategies (Hoffman, Slovic and Rorer, 1968; Slovic, 1969; Wiggins and Hoffman, 1968); therefore, the linear analyses used in the present studies may not have accurately described the decision strategies of all subjects. CHAPTER IX MANAGERIAL IMPLICATIONS AND SUGGESTIONS FOR FUTURE RESEARCH 11 . J I 1' I. Annlicatinns_for_Decision_finnnert_S¥5tems The importance of this research for the applied practice of decision-making in managerial settings is obvious in a number of areas. First, to this researcher's knowledge, this endeaver represents the first attempt at directly assessing subjects' incorporation of a forecast and/or standard error information in their decision-making. As mentioned previously, a number of studies have provided subjects with forecasts (Carter et al., 1978; Ebert, 1972; Hogarth & Makridakis, 1981; Remus et al., 1984; Remus et al., 1979), but these studies assumed that subjects utilized forecasts in their decision-making and no comparisons were made between groups receiving different types of decision aids and control subjects. The study of optimal decision aids and decision makers' incorporation of them in decision-making is extremely important in light of the proliferation of decision support systems in business contexts (Mann and Watson, 1984). In the area of expert systems, decision aids that emulate the process that a human expert would follow in solving problems (Basden, 1983; O'Keefe, 1986; Talluru and Akgiray, 1988), little empirical research assessing managers' acceptance of 152 153 decisions from these models has been done. Specific findings from this study would indicate that decision makers will not fully incorporate predictions from expert systems. This partial utilization is attributable to at least two problems. First, decision makers would be expected to rely less on predictions from expert systems due to the system's inability to perfectly predict actual events. As was true in this research, outcome feedback differs from even a statistically optimal predictions. Applications of decision support systems must recognize that human decision makers appear to expect valid predictions to be accurate on every trial, rather than to be statistically optimal over a number of trials. Secondly, above and beyond decision makers' misinterpretation of outcome feedback values leading to a lack of trust in the validity of decision aids, the results of this study indicated that motivational factors interfere with decision makers' incorporation of decision aids. In this study, many subjects expressed the expeCtation that they could outperform the statistically optimal forecast. These expectations continued even after subjects had calculated the regression equation from which the forecasts were predicted. Overcoming these expectations in managers may result in significant improvement in decision- making performance. One study (Hill, 1988), found that user involvement and training had a positive relationship on managers' satisfaction with expert systems and managers' 154 perceived improvement in decision-making. It should be understood that when developing decision support systems it is not a given that managers will utilize them completely, even if managers have been involved in their development. 1malications_for_¥alue_ludements Since the results of this study and more generally, the results of most of the research reviewed above, have shown that decisions from statistical models almost always outperform decisions from human decision makers, it might be logically deduced that human decision makers should be removed from the decision process. Einhorn and Hogarth (1987) cite a study (Showers and Chakrin, 1981) reporting the effects of a statistically based decision rule, which resulted in an estimated annual reduction of $137 million in bad debts for one company. Their point was that in many managerial situations, statistical decision aids can and should be used without what essentially has been the biased judgment of managers. Despite examples such as this one, the fact is that not all decisions can be modeled using the technology and techniques of statistical modeling. In support of this contention, Einhorn and Hogarth (1987) gave the example of an artificial intelligence program designed to simulate the comprehension of newspaper headlines. The program interpreted the headline: "World shaken. Pope shot." as "Earthquake in Italy. One dead." (p. 70). The technology of artificial intelligence and even more basic statistical 155 modeling is still not at a point where the human decision maker can be replaced entirely. Beyond the limitations of even the most state-of—the-art statistical modeling techniques, the decisions which human judges must make often include value judgments. Brehmer (1988) states that differences in value judgments have appeared to be more important than differences in judgment policies with respect to factual matters in applied settings, in particular in settings where confict is an issue. Social Judgment Theory (Hammond, Rohrbaugh, Mumpower, and Adelman, 1977; Hammond, Todd, Wilkins and Mitchell, 1964; Hammond, 1965) has been developed from the lens model framework as an interpersonal learning and conflict reduction technique. In the typical study based on Social Judgment Theory, statistical comparisons are made between the decisions of two judges' rather than actual outcomes or statistical predictions. These types of studies have provided a number of judges insight into their own decisions and insight into the decisions of others. Rohrbaugh (1988) has reviewed a number of applications of the conflict reduction techniques developed within the Social Judgment Theory framework and concluded that these techniques often led to group consensus on decision tasks where conflict had occurred. Applications of Social Judgment Theory have been reviewed in the disciplines of accounting (Waller, 1988), medicine (Wigton, 1988; Smith and Wigton, 1988), social work (Dalgleish, 1988) and risk judgment (Earle and Cvetkovich, 1988). The overall 156 conclusion of these reviews is that policy-capturing methods have resulted in conflict reduction in applied decision- making settings. W In the opinion of this author, the Sniezek and Reeves (1986) study represented a breakthrough in decision-making research. As far as is known, it is the first study that did not assume that subjects would recognize, understand, and appropriately interpret an optimal decision aid (feature cue) which shared the same measurement units as the criterion. The present research further developed this research line by informing subjects that the optimal decision aid was a forecast for the upcoming decision trial. The results in both sets of studies were essentially the same, in that subjects exhibited partial utilization of the regression based estimate of the criterion. Sniezek and Reeves (1986) hypothesized that the reason subjects did not fully utilize the feature cue in their decisions was that feedback values confused them and led them away from this strategy. The present research included a group which received a forecast without outcome feedback, in which the outcome feedback factor could not have been the cause of subjects' partial utilization of the forecast value in their decisions. Motivational factors, such subjects' belief that they could do better than the forecast were offered as another 157 potiential cause of their resistance to fully utilizing the forecast. Future research should be directed toward identifying factors which influence subjects' utilization of optimal decision aids. Studies assessing means of lessening the negative effects of feedback on subjects' utilization of a forecast might include a manipulation in which the optimal estimate is provided as outcome feedback rather than as feedforward. It is possible that in applied decision settings, linearly smoothing outcome feedback values may lead subjects to better learning and performance on decision tasks, due to the smaller amount of variance in these smoothed feedback values. Doherty and Balzer (1988) expressed the opinion that providing subjects with direct information about the probabilistic nature of the environment might improve subjects' learning and performance in decision tasks. The present research provided subjects with standard error information toward that purpose. Subjects who were questioned by the experimenter were able to state the type of information this decision aid would provide to them in the simulation. Still the standard error factor's effects were not consistently significant on all measures of performance across the two data collections. It appears that a range estimate, such as the standard error, does not provide any more information than a point estimate, such as the forecast, and in some situations may provide less information. 158 Therefore, future research might still be directed toward (1) identifying types of information that will aid decision makers in interpreting the outcome feedback values which contain error and (2) motivating them to apply those interpretations in their decisions. In light of the rather blatant attempts at accomplishing these two goals in the present research, no specific suggestions for future research will be offered in this area. Regarding the limitations of this research described in the previous chapter, studies specifically focusing on managers' behavioral reactions to optimal decision aids should and can be assessed in actual, rather than simulated decision-making contexts. Research studies associated with the Social Judgment Theory paradigm have used policy capturing techniques to analyze group decisions in a large number of applied settings. Providing subjects with statistical decision aids during the decision-making process should be relatively easy to accomplish and provide measurable results. Furthermore, a research study within an actual decision context might naturally include all of the factors managers are considering when making decision, including ethics and values. As Doherty and Balzer (1988) have pointed out, decision makers probably know their own policy weights reasonably well, but have difficulty expressing them. The two attempts to use this fact in both data collections to assess subjects' knowledge of the relationship between the cues and the 159 criterion were not very successful. In the first data collection, it appears that subjects were overwhelmed by the task, whereas in the second data collection the task was rather simply directed at determining whether subjects knew the signs of the cues. Reilly (1987) has recently reported a study where subjects were asked to identify their own policies using an array of cue weights. Of eleven subjects, seven were able to identify their own policies. Rather than showing the potential cue-criterion combinations to subjects as was done in the first study, a sample of cue weights, including the actual weights in the simulation, might have been shown to the subjects and this might have resulted in more meanful results. Conclusion The purpose of this research was to assess the effects of decision aids on human decision-making performance. In most cases, either one or both of these decision aids did improve subjects performance in comparison to subjects who received neither of these decision aids. Providing these decisions aids to subjects did not overcome the confusing aspects of feedback. Some of the subjects in this study were given access before each trial to a statistically optimal response, the forecast. It is likely that those subjects did not and still do not understand the concepts of probability and linearity 160 in decision-making. Consistency, a major contributor to optimality in decision-making is a difficult concept to teach decision makers. Experience does not seem to be the best teacher in decision-making. Before recommending that decision-making be taken away from human decision makers and turned over to perfectly consistent statistical algorithms, it must be noted that this study and its results follow a long line of pessimistic studies of the performance of human decision makers. Christensen-Szalanski and Beach (1984) provide evidence that studies showing the follies of human judges have a higher incidence of publication and citation. They point out that human judges have made an infinite number of decisions that have allowed the species to survive to the present time. Admittedly, the results of this project have been interpreted from the perspective of viewing a glass as half empty rather than half full. The performance of the subjects in this study did not confirm the authors expectations, but subjects were not trying to fulfill those expectations. In fact, most of the subjects in this study made adequate decisions, that is, their decision were part of some strategy that was better than a random strategy. The recommendations for future research in this study did not identify major directions in which decision makers might be led toward optimality. That does not mean that they do not exist. LI ST OF REFERENCES REFERENCES Abelson, R. R. & Levi, A. Decision-making and decision theory. In G. Lendz & Aaronson, E., (eds.) Handbnnk_nfi Snoial_£aynhnlngy. Reading, Ma: Addison Wesley. 1985. Adelman, L. The Influence of formal, substantive, and contextual task properties on the relative effectiveness of different forms of feedback in Multiple Cue Probability learning tasks Qroanizational_Behaxior_ano_Hnman Performance. 1981, 21, 423-442. Balke, W M Hammond, K R and Meyer, G D An alternate approach to labor~management relations. Administratjye Soienoe_Qnarterl¥- 1973 13 311-327 Bandura, A. Sogial_Laatning_Ihanty. Englewood Cliffs, N. J.: Prentice Hall. 1977. Barker, H. R. & Barker. B. M. Multixariate_Anal¥sie_of Marianoe_lMANQyAL. University of Alabama: The University of Alabama Press, 1984. Basden, A. On the application of expert systems. International_Journa1_of_Man:Maohine_Studies,1983,.19. 461- 477. Bjorkman, M. Feedforward and feedback as determinate of knowledge and policy: A note on a neglected issue. Soanoina1ian_lonrnal_of_ES¥oholoox- 1972. 13. 152-158. Bock. R. D. Multixariate_Statietiel_methoos_in_hehaxiorel taaaatnn. New York: McGraw-Hill, 1975. Bowman, E. H. Consistency and optimality in managerial decision—making. IManagamant_Snianca. 1963, 9, 310- 321. Bray. J- H- and Maxwell, S. E. Mhltixariate_Anal¥Sis_of Matianna. Beverly Hills, Sage Publications, 1985. Brehmer, A. and Brehmer, B. What have we learned about human Judgment from thirty years of policy capturing? In B. Brehmer and C.R.B. Joyce (Eds.) Human_Jndgmant: The .Sl1l1l_yiau. North Holland: Elsevier Publishers, 1988. 161 162 Brehmer, B. & Kuylenstierna, J. Content and consistency in probabilistic inference tasks. nganjzatjona] BEL . ano_Human_Eerformanoe. 1980, 25, 54-54, Brehmer, B. & Kuylenstierna, J. Task information and performance in probabilistic inference tasks. Qroenizationa1_Beha1ior_ano_Human_Eerformanoe. 1978. 22, 445-464. Brehmer, B., Kuylenstierna, J. & Lilgergen, J. Effects of function form and cue validity on subjects' hypotheses in probabilistic inference task. Qrganizatignal_aanayinr_ang Human_2erformanoe. 1974 11. 338-354. Brehmer, B. & Lindberg, L. The relation between cue dependency and cue validity in single cue-probability learning with scaled, cue criterion variables. Qroanizationa1_Performanoe_and_Human_Eerformanoe. 1970. 5, 542-554. Brehmer, B. & Qvarstrom, G. Information integration and subjective weights in multiple cue judgment. ' ' ' . 1976, ll, 118-126. Brehmer, B. Cognitive dependence on additive and configural cue criterion relationships. Amerinan_fionrnal_of Psychology. 1969, 33, 490- 503- Brehmer, B. Cue utilization and cue consistency in multiple- Cue probability learning. Qroanizational_Behayior_ano Human_2erformanoe. 1972, 8. 286-296- Brehmer, B. Effects of task predictability and cue validity on interpersonal learning of inference tasks involving both linear and nonlinear relations. Organizational Behaxior_and_Human_Eerformanoe. 1973, 10. 24-46. Brehmer, B. In one word: Not from experience. Agta Psycholooia. 1980. 45. 223-241. Brehmer, B. Inference behavior in a Situation where the cues are not reliably perceived Qrganizational_Behaxior_and Human_2erformanoe. 1970, .5. 330*347- Brehmer, B. Note on clinical judgment and the formal characteristics of clinical tasks. Psychological Bulletin. 1976, 83, 778- 782. Brehmer, B. Response consistency in probabilistic inference tasks. Qroanizationa1_Behaxior_ano_fluman_Eerformanoe. 1978, 22, 103-115. 163 Brehmer, B. Subjects' ability to use functional rules. Psychonomio_Soienoe. 1971, 24. 259- 260 Brehmer, B. The development of Social Judgment Theory. In B. Brehmer and C.R.I. Joyce (Eds.) Human_lndgment: The s2Q212_yien. North Holland: Elsevier Publishers, 1988. Brunswik, E. Eeroeot1on_and_the_Representatixe_oesisn_of Psyoholosioa1_Exneriments. Berkeley: University of California Press, 1956. Brunswik, E. Representative design and probabilistic theory in a functional psychology. Psychological_3eyieu. 1955, 52, 193- 217. Brunswik, E. The_Conoentual_Eramework_of_2syoholooy- Chicago: University of Chicago Press. 1952. Campbell, D. T. Blind variation and selective retention in creative thoughts as in other knowledge processes. Psycholooioa1_8eyiew. 1960 51. 380- -400 Carroll, J. B. The nature of the data, or how to choose a correlation coefficient. Psycncmetrika. 1961, 25, 347- 372. Carter, P. L., Remus, W. E., & Jenicke, L. 0. Production scheduling decision rules in changing and non- linear environments. International.1ourna1_of_2roouotion Research. 1978, lfi, 493-496. Chamberlin, T. C. The method of multiple working hypotheses. MW- 1897! 5.5.! 837-48- Christensen-Szalanski, J. J. & Beach, L. R. The Citation bias: Fad and fashion in the judgment and decision literature. American_Esyoholooist. 1984, 39. 75- 78 Cohen. J. & P. Cohen. Aeolieo_Mn1tiole B . CC 1 Ii E J . E I] E 1 . J Sciences. Hillsdale, New Jersey: Lawrence Erlbam Associate, 1983. Cohen. J. Statistioal_ooner_analysis_for_the_hehayioral sciences. New York: Academic Press, 1977. Cohen, L. F. Can human irrationality be experimentally demonstrated? Ihe_Beha1ioral_and_Brain_Soienoes. 1981 A, 317- 331. 164 Conover, W. J., Johnson, M. E., and Johnson, M.M. A comparative study of tests for homogeneity of variances, with application to the Outer Continental Shelf Bidding Data. Teohnometrios 23, 351-361, 1981. Cooksey, R.W. and Freebody, P. Generalized multivariate lens model analysis for complex human inference tasks. 1 985 I 35’ 4 6-72 e Cooley. W. W. and P. R. Lohnes. Mmlfixeriate_oata_anelysis1 New York: Wiley, 1971. Dalgleish, L.I. Decision-making in child abuse cases: Applications of Social Judgment Theory. In B. Brehmer and C-R-I- JOYCE (Eds. ) Human_1udgment:The_SIJII1_Yiew North Holland: Elsevier Publishers, 1988. Dawes, R. M., & Corrigan, B. Linear models in decision- making. Psycholooioa1_Bnlletin. 1974, 81. 95-106. Dawes, R. M. Shallow psychology. In J. 8. Carroll and J. Payne (Eds. ) Coon1tion_and_Sooial_BehaIior.Hillsdale, N. J.: Erlbaum, 1976. Dean, D. H., Hammond, K. R., & Summers, D. A. Acquisition of knowledge in complex inference tasks. Jonrnal_cf Exoerimenta1_Esyoholooy- 1972 92, 20- 26. Doherty, M.E. and Balzer, W.K. Cognitive feedback. In B. Brehmer and C.R.I. Joyce (Eds.) Human_1ncgment: The s2s212_yieu. North Holland: Elsevier Publishers, 1988. Dudycha, A. L. Dumoff, M. G. and Dudycha, L. W. Choice behavior in dynamic environments. Qroanizational_Behaxior_and Human.£erformanoe. 1973, 9, 323-338. Dudycha, L. W., & Naylor, J. C. Characteristics of the human inference process in complex choice behavior situations. Organizational Behaxior and Human Egnfcrmance. 1966,11 110-128. Earle, T.C. and Cvetkovich, G. Risk judgment, risk communication, and conflict management. In B. Brehmer and C-R.I- JoyCe (Eds. ) Human_lndoment: .Ihe_SIJITI_Yiew. North Holland: Elsevier Publishers, 1988. Ebert, R. J. Environmental Structure and Programmed Decision Effectiveness. Management_§cience. 1972, 15, 435-445. 165 Einhorn, H. J. Learning from experience and suboptimality rules in decision-making. In T. S. Wallstein, (Ed.), chnitiye Erccesses in Chcice and Decisicn Behayicr. New York: Earlbaum, 1980. Einhorn, H. J. The use of nonlinear, noncompensatory models in decision-making. .Esychologioal_BnlleLin. 1970, 13, 221-230. Einhorn, H. J. Use of nonlinear, noncompensatory models as a function of task and amount of information. Qroanizational_BehaIior_ano_Human_Eerformanoe. 1971, 13. 221-230. Einhorn, H. J., Kleinmutz, D. N. & Kleinmutz, B. Linear regression and process tracing models of judgment. Psycholooioa1_3eyiew. 1979. 86. 465- 485 Einhorn, H.J. and Hogarth, R.M. Decision-making: going forward in reverse. Harxaro_Business_Bexiew. 1987 65. 66- 70. Freud, s. Be2ono_the_21easnre_2rinoiole. New York: Bantom Books, 1959. Goldberg, L. R. Man versus model of man: a rationale, plus some evidence, for a method of improving on clinical inferences. ESECthQQiCal_BnlleLin. 1970,13, 422- 432. Gray, C. W. Ingredients of intuitive regression. Qroanizational_Behayior_&_Human_Eerformanoe. 1979, 23, 30-48. Hagafors, R., & Brehmer, B. Does having to justify one' s judgments change the nature of the judgment process. . 1983, 3T, 223-232. Hair, J.F., Anderson, R.E., Tatham, R.L., Grablowsky, BlJ. multixariate_Data_Analysis. Tulsa. Okla: Petroleum Publishing Company, 1979. Hammond, K. R. Computer graphics as an aid to learning. Science. 1971, 112, 903-908. Hammond, K. R. New Directions in research on conflict. Journal_of15ooial_lssues. 1965, 21, 44- 46. Hammond, K. R., & Summers, D. A. Cognitive control. Psychological semen. 1972, 12, 58-67 166 Hammond, K. R., Rohrbaugh, J., Mumpower, J. L. & Adelman, L. Social judgment theory: Applications in policy formation. In M. F. Kaplan & S. Schwartz (Eds.), Human_jucgment_anc oeois1on_orooesses_in_anolied_settinos. New York: Academic Press, 1977. Hammond, K. R., Todd, F., Wilkins, M. M., & Mitchell, T. O. Cognitive conflict between persons: Application of the lens model paradigm. ' Psychology. 1965. 2. 343-360- Hammond, K. R., Summers, D. A. and Deane, D. H. Negative effect of outcome- -feedback in multiple- cue probability learning. Qroanizational_Behaxior1_ano_Human_Eerformanoe. .9. 30- 34. 1973. Hand! D-Jor and Taylor, C.C. WWW .Yarianoe_and_Beoeated_Measures. New York: Chapman and Hall, 1987. Harrie. R. J. .A_Primer_of.multixariate_Statistios. New York: Academic Press, 1975. Hill, M. M. The relationship between user involvement and decision system success. Proceedings of the annual meeting of the Decision Sciences Institute. 1988, Nov., 210-212. Hoffman, P. J., Earle, T. C., & Slovic, P. Multidimensional functional learning and some new conceptions of feedback. Qroanizationa1_Behaxior_and_Hnman_Eerformanoe. 1981.21. 75-102. Hoffman, P. J., Slovic, P. and Rorer, L. G. An analysis of variance model for the assessment of configural cue utilization in clinical judgment. Esycholcgical_Bulletin. 1968, 52, 338- 349. Hogarth, R. M. & Makridakis, S. The value of decision-making in a complex environment: An experimental approach. .Manaoement_Soienoe. 1981. 23. 93-107. Hursch, C., Hammond, K. R., & Hursch, J. Some methodological considerations in multiple-cue probability studies. Psycholooioa1_8exiew. 1964. ll. 42-60- Huyn, H., & L. S. Feldt. Conditions under which mean square ratios in repeated measurements designs have exact F. distributions. Journal_of_the_American_Statistioal .Associaticn. 1970, 55, 1582—1589. 167 Ito, P. K. On the effect of heteroscedasticity and nonnormality upon some multivariate test procedures; in P. R Krishnaiah (ed) multifariate_Anal¥sis_ll New York: Academic Press, 1969. Johnasson, R., & Brehmer, B. Inferences from incomplete information: A note. Qrgan1zatinnal_Behaxinr_and_Hnman Eeffnrmance. 1979, 25, 22-31. Koele, P. The influence of labeled stimuli on non-linear multiple cue probability learning. ngan1zat1gnal Behax1or_and_fluman_zerformance. 1980. 2f. 22-31. Kunruether, H. Extension of Bowman's theory on managerial decision-making. .Management_ficisnce. 1969. 15. 415-439. Kuylenstierna, J., & Brehmer, B. Memory aids in the learning of probabilistic inference tasks. Qrgan1zat1onal_fiehay1or and_Human_Eerformance. 1981, .28. 415-424. Lindberg, L., & Brehmer, B. Transfer in single—cue probability learning. Qrganizaf12na1_Beha11Qr_and_Human Performance. 1976. 15. 177-192. Lindell, M. Cognitive and outcome feedback in multiple cue probability learning tasks. Journal_of_fixperimental Esx9hnlogx1__Human_Laarn1ng_and_Memnr¥. 1976. 2. 739-745. Mann, R.I. and Watson, H.J. A contingency model for user involvement in DSS Development. MlS_Qna£Lezl¥. 1984, 8, 27-37. Mardia, K. V. The effects of nonnormality on some multivariate tests and robustness to nonnormality in the linear model. Binmetrika. 1971, 58, 105-121. Miller, P. Do labels mislead? A multiple cue study within the framework of Brunswik's probabilistic functionalism. Qrgan1zat1ona1_Eehax1or_and_fluman_2erfbrmanse. 1971 f. 480-500. Miller. R. G. Jr. S1mn1taneous_atat1sfisal_1nference. New York: McGraw-Hill, 1966. Moskowitz, H. & Miller, J. Information and decision systems in production planning. Managementjnience. 1975. 22.. 359-370. Muchinsky, P. M. & Dudycha, A. L. Human inference behavior in abstract and meaningful environments. Qrganizat1onal Baha11nr_and_fiuman_Eerformance. 1975,13. 377-391- 168 Mynatt, C. R., Doherty, M. E. & Tweney, R. D. Confirmation bias in a simulated research environment: An experimental study of scientific inference. Quarterl¥_lournal_of Experimenta1_Esxchnlog¥. 1977.29. 85-95. Mynatt, C. R., Doherty, M. E., and Tweney, R. D. Consequences of confirmation and disconfirmation in a simulated research environment. Experimenta1_Ee¥eheleg¥. 1978. 99. 395-406. Naylor, J. C. Some comments on the accuracy and the validity of a one variable. Jeurna1_ef_Mathemariea1_Ee¥eheleg¥. 1967, 4, 154-161. Naylor, J. C., & Clark R. D. Intuitive inference strategies in interval learning tasks as a function of validity, magnitude, and sign. Performance. 1968, ‘3, 378-399. Naylor, J. C., & Schenck, E. A. The influence of cue redundancy upon the human inference process for tasks of varying degrees of predictability. Organ1zat1onal Behax1er_end_fluman_2erfermanee. 1968. 91 47- 61. Newton, J. P. Judgment and feedback in a quasi-clinical situation. Qeurna1_efLEereena11rz_and_Seeia1_Ee¥eheleg¥. 1965. 1. 336-342. Nystedt, L., & Magnusson, D. Cue relevant and feedback in a Clinical Prediction task. Qrganizatienal_Behaxier_and Human_2erfermanee. 1973. 9. 100-109. O'Keefe, R. Simulation and Expert systems - A Taxonomy and some examples. fi1mnla§1gn 1986, 45, 10-16. Olgilvie, J. R., & Schmitt, N. Situational influences on linear and non linear use of information. Beha11er_and_flnman_2erfermanee. 1979. 29. 292-306. Olson, C. L. Comparative robustness of six tests in multivariate analysis of variance. Journal_of_Amer1can Statistiea1_Aesne1ar1en. 1974, 59. 894-908. Olson, C. L. On choosing a test statistic in multivariate analysis of variance. Eexehelegiea1_an11etin. 1976. 99. 579- 586. Peterson, D. K., & Pitz, G. F. Explicit cue weighting in a prediction task. ' ' ' Dee1e1en_Ereeeeeee. 1985. 95. 289-304. 169 Phelps, R. M. and Shanteau, J. Livestock judges: How much information can an expert use? ' and.fluman.£erfermanee. 1978. 21. 209-219. Piaget. J. and Inhelder. B. W W. London: Routledge and Kegan-Paul, 1974. Platt, J. R. Strong Inference. 591enge, 1964, 146, 347-353. Reilly, B.A. The importance of job attributes in the recruitment of accountants: a policy capturing approach. Unpublished masters thesis, 1987. Bowling Green State University, Bowling Green, OH. Remus, W., Carter, P. & Jenicke, L. Regression models of decision rules in unstable environments. Journa1_of Busineee_Beeeareh. 1979,11, 187-196. Remus, W., Carter, P., & Jenicke, L. Improving decision- making by using performance feedback: An empirical study. QneratmLBesearchJetters. 1984.9. 105-109. Rohrbaugh, J. Cognitive tasks and small group processes. In B. Brehmer and C.R.I. Joyce (Eds.) Human_lndgment: The §1J1I1_y1ew. North Holland: Elsevier Publishers, 1988. Rothstein, H. G. The effects of time pressure on judgment in multiple cue probability learning. We. 1986. 91. 83-92. Schmitt, N., Coyle, B. W., & King, L. Feedback and task predictability as determinants of performance in multiple cue probability learning tasks. Organ1zationa1_Behav1or anLHummBerfermanee. 1976,16. 388-402. Schmitt, N., Coyle, B. W. and Saari, B. B. Types of task information feedback in multiple— cue probability learing. QrganizmenaLBehaxierJMumanmfermanee. 1977.19. 316-328. Showers, J. L., and Chakrin, L. M. Reducing uncollectible revenue from residential telephone customers. Innerfaces, December 1981, 21. Skinner. B. F. W Analysis. New York: D. Appleton-Century. 1977. Slovic, P. Analyzing the expert judge: A descriptive study of stockbroker's decision processes. Journal_of_Appl1ed 9331191932. 1969. 5.9. 255-263. 170 Slovic, P. and Lichtenstein, S. Comparison of Bayesian and regression approaches to the study of information processing in judgement. Qrganizatienal_BehaM1er_and Human_2erfermanee.1971..f. 648-745. Slovic, P. Hypothesis testing in the learning of positive and negative linear functions. Organ1zat1onal_Behayior and_Human_Eerfermanee. 1974.11. 368-376. Slovic, P., Fischoff, B. and Lichtenstein, S. Behavioral Decision theory. Annual_Be11ew_ef_Rsyeheleey. 1977. 29. 1-39. Smith, D. G. and Wigton, R. 8. Research in medical ethics: The role of Social Judgment Theory. In B. Brehmer and C. R. I. Joyce (Eds. ) Human_Judgment: The_919111_91ew. North Holland: Elsevier Publishers, 1988. Sniezek, J. A. The role of variable labels in cue probability learning tasks. Qrganizatiena1_Behayier_and Human_Dee1sien_Breeesses. 1986. 99. 151-161. Sniezek, J. A., & Naylor, J. C. Cue measurement scale and functional hypothesis testing in cue probability learning. 1978, 22, 366-374. Sniezek, J. A., and Reeves A. P. Feature cues in probability learning: Data base information and judgment. Qrgan1zarienal_Behexier_and_Hnman_Eerfermanee. 1986..9l. 297-315. Sniezek, J. A., Dudycha, A. L., & Schmitt, N. W. Learning of negative cue validity as a function of statistical sophistication and instruction. Esycholog1ca1_Beports. 1978, A}, 1093- 1101. SESS_H§§£1§_Gn1dg. 3 Edition, Chicago, IL; SPSS Inc., 1988. Steinman, D. O. Transfer of lens model training. Qrgan1zariena1_Behayier_and_Hnman_Eerfermenee. 1974. 12. 1-16. Stevens, J. P. Comment on Olson: Choosing a test statistic in multivariate analysis of variance. Esycholgg1cal Bullet1n. 1979, 86, 355-360. Stevens, J. P. Four methods of analyzing between variation for the k-grOUP MANOVA problem. Multiyar1are_Beha11era1 Research. 1972, 1, 498- 522. 171 Stevens, J. P. Power of the Multivariate Analysis of Variance Tests Psychologiea1_Bu11etin. 1980. 99. 728- 737. Summers, D. A., & Hammond, K. R. Influence behavior in multiple cue tasks involving both linear and non linear relations. Jeurna1_nf_Exnerimenta1_Eeyehelee¥. 1966. 11. 751-757. Talluru, L. R., and Akgiray, V. Man-Machine Interface issues in the acquisition of knowledge from end users. Proceedings of the Annual Meeting of the Decision Sciences Institute. 1988, Nov., 274-276. Thorndike. E. L. Ihe_Eundamentals.ef_Learn1ng. 1971. New York: AMS Press. Todd, F. J. & Hammond, J. R. Differential feedback in two cue-multiple cue probability learning tasks. Behax1ora1 Sg1enge. 1965, 10, 420- 435. Tucker, L. R. A suggested alternative formulation in the development by Hursch, Hammond, & Hursch, and by Hammond, Hursch, & Todd. Psychologiea1_8eyiew. 1964, 11, 528-530. Tuersky, A. and Kahneman, D. Judgment under uncertainty: heuristics and biases. Science. 1974, 185, 1124-1131. Tweney, R. D., Doherty, M. E., Worner, W. J., Pliske, D. B., Mynatt, C. R., Gross, K. A., & Arkkelin, D. L. Strategies of rule discovery in an inference task. Quarter1y_fiourna1 ef_Experimenta1_Esxehelegy. 1980.92. 109-123. Waller, W. S. Brunswikian research in accounting and auditing. In B. Brehmer and C. R. I. Joyce (Eds.) Human Judgment: The_fi1J1I1_y1eu. North Holland: Elsevier Publishers, 1988. Wason, P. C. Reasoning about a rule. Quarter1y_fiourna1_of Experimental_Esxehnleg¥. 1968. 29. 273-281. Wason, P. C. On the failure to eliminate hypothesis in a conceptual task. Quarter1y_Jeurnal_ef_Exper1mental Esxehelegy. 1960. 12. 129-145. Wason. P. G. & Johnson-Laird. P. N. Psychology_ef_3easen1ngi Structure_and_gontent. Cambridge, MA: Harvard University Press, 1972. 172 Wherry, R. J., Sr., Naylor, J. C., Wherry, R. J., Jr., & Fallis, R. F. Generating multiple samples of multivariate data with arbitrary population parameters. Esychgmetr1ka. 1965, 30, 303-313. Wiggins, N. and Hoffman, P. J. Three models of clinical judgment. 9eurna1_ef_Abnerma1_Esyehelegy. 1968. 19. 70- 77. Wigton, R. S. Applications of judgment analysis and cognitive feedback to medicine. In B. Brehmer and C. R. 1- Joyce (Eds.) Human_9udgment: The_919111_91ew. North Holland: Elsevier Publishers, 1988. Wilkinson, L. Response variable hypothesis in the multivariate analysis of variance. Psycholog1gal Bullet1n. 1975, 32. 408- 412. APPENDICES APPENDIX A INSTRUCTIONS FOR DATA COLLECTION I INSTRUCTIONS FOR DATA COLLECTION I Your assignment (should you choose to accept it) is to be a load planner for a major air carrier during the period immediately preceding and following deregulation (October, 1977 through August, 1979). The main job of a load planner for an airline is to predict the weekly total number of passenger miles that are flown on the airline nationwide. Your job is extremely important to the airline because you supply the one unknown estimated variable necessary for the utilization of a cost—minimization scheduling algorithm. Given your input (the estimate of the total number of weekly passenger miles) the algorithm schedules the optimal size of plane, crew, and support staff for each of its routes. As you may have already surmised, the effectiveness of this algorithm depends almost entirely on the accuracy of your estimate, since the algorithm is "fixed" except for your estimates. You have two decision aids at your disposal designed to help you make your weekly scheduling decisions. The first is a numerical index that takes into account the effects of the upcoming week's adjusted seasonality on total passenger miles flown. For example, the fourth week in December and the second week in April may be represented by the same index, since both weeks include a holiday, but the fourth week in July may also have the exact same index because it is near the close of the business fiscal year. Due to the statistical 173 nature of this seasonality index, it does not matter which week of the year you are working in, since the purpose of this index is to give you information about weekly fluctuations in a standardized form. The second decision aid will be an index derived from other competing airlines' recent promotions and prices. From these decision aids you must accurately predict total weekly passenger miles for the next 80 weeks. A quick example may help you to understand how these cues can be successfully utilized. When you were admitted to college, the school asked for a few predictors of your future college performance, including ACT and SAT scores. Although you may not have known it at the time, your numerical scores on these "cues" allowed the college officials to predict your approximate performance level (ie. GPA) during your college career. For any individual these cues may not have been extremely accurate, but across thousands of applicants it is known that these cues do have some measure of predictive accuracy. Although ACT and SAT scores are not on the same scale as GPA scores, they can still be used to predict GPA. Similarly, the cues you will be given are not in miles, but they can be used to predict total passenger miles. You will also have a forecast of the upcoming week's total passenger miles (based on the regression of past cue values on total passenger miles) that may also help in making your decisions. 174 Finally, you will have the forecast's standard error shown with each forecast. As you may already know, given an accurate forecast, about two-thirds of the actual total passenger mile values should fall within the standard error. On the computer screen your estimate will be graphed like this: ------------------------------ (Almost solid line) The actual passenger miles flown for the previous week will look like this: .............................. (Dotted line) The forecast of total passenger miles for the upcoming week will look like this: ---------------- (Dashed line) The standard error information will look like this: Now boot up your computer, change the directory to a:, insert the simulation disk, type "LOADSIM" and you're flying. NOTE: You will have ten training trials on this task. If you are confused by what is going on after these ten trials, tell your boss. You will be fired if you use paper and pencil to predict passenger miles (we could do that without you) or if you randomly plunk in numbers without any effort (ditto). You may want to keep this instruction sheet handy so that you can refer to it if you forget which graph lines are associated with ygnr_estimate, etc. 175 APPENDIX B MEANS AND STANDARD DEVIATIONS FOR Z-SCORE TRANSFORMED DATA om NN om Hm mm mm mm mm Nv vv Nv mv ov mv vv vv Nv av vv vv vm mm mm mm mm mm 2 wNN.H vow. NOH.H mmv. Nmm. omv. mam. haw. omH.H mhm. mam. ave. amm.H mmm. NHv. mHv. vmm.H Nmm. mew. mmm. mHH.H mam. Nmo.H 5mm. HmH.H mHv. cowumfi>mo nuancmum mow.H| mam. mmo.l wwm. omo.u mvm. va. chm. o~m.l mwm. NmH. Nmm. mmw.l mmm. mmH. Ham. vam.n vHH. who. Nov. mom.| amN. omN.I mmN. mmN.I ovm. Cmmz uouum oumocmum oz .ummomuom oz .xomnommm oz uouum Unmocmum .ummomuom oz .xomnomom oz uouum oumocmum oz .ummomuom .xomnommm oz uouum osmocmum .ummomuom .xomnommm oz uouum oumocmum oz .ummomuom oz .xomnommm uouum oumocmum .ummomuom oz .xomnommm uouum oumocmum oz .ummomuom .xomnoomm uouum oumocmum .ummowuom txomnommm uouum oumocmum oz .ummowuom oz uouum Unmocmum .umwomuom oz uouum oumocmum oz ~ummomuom uouum oumocmum .ummomuom. uouum Unmocmum oz .xomnowmm oz uouum oumocmum .xomnomwm oz uouum oumocmum oz .xomnomwm Houum Unmocmum .xomnommm ummomuom oz .xombooom oz ammomuom .xomoommm oz ummomuom oz .xomnommm ammomuom .xomnomwm uouum oumocmum oz uouum oumocmum ammomuom oz uncommon xomnommm oz xomnommm Houomm coaumEuommcmua muoomIN mound mcofiumfl>mo numocmum new wood: n momusoom hm OHQMB 179 APPENDIX C CORRELATION MATRIX FOR DEPENDENT VARIABLES FOR DATA COLLECTION I doc; «and wnwd wnwd coo; wnwd snhd Odo; oRd ado; §< n00< N00< FOO< wwdd. anmd- owéd- mwvd- ooo. _. v 10h; 10.53 10.22 10.53 wand. nwmd. mvnor 3w: ophd 80... n novd- vad. awnd- ahvd- wwwd awwd ooo. F N wand. 5%.? av Pd. wwwd. wowd pawd npwd cow; 9 wand- wand. vnmd. 53d- anwd w Fwd _.wmd ahvd Odo. .. v200n200~200 vnnd- awnd. wnNd- wand- amvd _.owd _.wwd ahvd wnhd ooo. _. wand- awnd- chad- wand. mnnd w _.vd nmwd swvd vvwd wand ooo. F H cofiuomaaoo mumo Mom . moaQMfium> ucmocwmwo uom xfiuumz coflumamuuoo an magma. vwvd- Qwvd- wwvd. nnnd. wwnd. awnd- pwwd. N :d. vad whcd awwd vwwd a _.wd nhvd vmwd nacd onwd awnd wwwd wawd NNwd wvwd ooo. .. wvwd Odo. .. F200 vIO< 3?? hand- wand- N 3d. whvd vmwd n Fwd nwvd wwwd wnwd nwvd nhvd vnvd Odo; wand- Pwvd- w Fwd- n Fwd- havd wand nhvd awvd R Pd an _.d an Pd awnd «and awnd ooo. — m .. wd. chad. ade- wwwd. nowd wawd thd bond _. ..wd pvwd «and 3nd wawd wowd aawd ooo. p nIO< N$0< 50¢ noo< noo<. 3.5.52 9.0.22 «IQ—.53 £0.22 =..o< 180 APPENDIX D STEM AND LEAF PLOTS FOR Z-SCORE TRANSFORMED DATA FROM DATA COLLECTION I mZO_._.<>mmmmO mmoomuN F x0015 .. ._.Zm=>_m_>w_IO< n? MEDGE 181 nZO_._.<>mman mmOOmIN ow N XOOAm .. ._.Zm=>_m>m=IO< E MEDGE 182 m20_._.<>mmmm0 mmoomuN ON n x0015 .. ._.Zm=>_m>m_IO< 9. MEDGE 183 m20_._.<>mmmmo mmoomuN v x0015 .. ._.ZmS_w>m=IO< wF mEDOE 184 mzo_._.<>mmmm0 mmOOnIN on _. XOOAm .. >Ozm._.w_wZOO t HEDGE 185 mZO_._.<>m_mmmO mmoonuN wN N XOOJm .. >OZm_._..w_.wZOO Q. meOE 186 n20_._.<>mmmm0 mmOOmIN o 0—. mp ON oooooooooooooooo ................................................................................................................................................................................................................................................... In“ .on n XOOAm .. >OZMHw_wZOO a_. MEDGE 187 n20:.<>mmmmo mmOOnIN wN v x0015 I >OZm_._.w_wZOO ON MEDGE 188 m20_._.<>mmmm0 mmoomuN ....................................................................................................................................................................................................... F x0015 .. 02.10.25. FN MEDGE ON 0* OO OO OOF ON_. 189 n20.._.<>mmmm0 mmoomuN N XOOAm .. 02.10.25. NN MED—0.“— O_. ON on O? on 190 n20_._.<>mmmm0 mmOOOIN m x0015 I GZ_IO._.<_>_ nN MEDGE ON O? ow on O9. 191 OZO_._.<>mmmmO mmOOnIN vnu v XOOAm .. OZ_IO._.<_>_ VN MEDGE 0—. ON on OV ow ow ON 192 m20_._.<>mmmm0 mmoomIN vn ............................................................................................................................ .................................................................................................................................................................. .................................................................................................................................. .................................................................................................................................................................................................................. 0| _. XOOAm I >Ommmm0 mmOOOIN N 200.5 I >0mwnm0 mmoomIN n v.00I_m I >0mmmm0 mmoomIN OO v 50.5 - >om 5320". mm 2:9“. I 29:92:04 3. Ne. O 197 63 cool 02 c2899.... 5.5 3 x0330“. I EoEo>oEo< on 2:9... 198 .otm Emu—Em oz cotm 2355 0 0m. 9 $390“. 02 w 05. Oh. up. $390“. 00.? c3089:— totm p.355 >m 5320”. I. 3:06:38 5 2:9“. x0380“. nw. Nw. Oh. \‘Ifl/ wh. x063 000"— 02 53022:. :25 >m x0330“. I 5:066:00 mm 2:9". wk on. 200 Eun‘.‘ 1. iy‘g :otm papcflw totm 2.656 02 $320.... 02 A N. K. ah pa. .9820“. 5:88: :otm o 329w >m 5320.... I 92222 8 2:9”. 201 00... c.0380: 3. c.0380”. oz 8:83.... arm 5 c.0380". I 83%: A can: 202 0530.2 20238:. :a£:o£m: Samoa—2 >0...aI:.:IoIoI< E085 o. .090 :_ cocoon .5305: :00: 03: «mag o:oowN ”202 o x o x o x o x teem caucsm o o x x o o x x .883". o o o o x x x x x0380". — - — P n — - - o «3 3 O 2 and. amn— Nmé \ ,./ 3.: 8.3 \ . RS 38 cm :otm Paocsm ocm 5890.... 203ng 30:23:60 :4. :0". 355.0%: 96:0 I >um500< mm 2:9“. 23322 52:08:. 55 .052 05802 523094 E32: 2 :65 :_ cocoon U335: coon 9a: 82; ”902 n e a F — _ _ — 2. I o t. . 5d oaIWlllIII‘lI/ xomnooom oz . 2 8.8 . / on : é om 8 romance: 8 8. z. :o=oo._2:_ cEm >m xombooom I >oflsoo< 8 23m: 204 APPENDIX F INSTRUCTIONS FOR DATA COLLECTION II CON .1) 10 ll 12. INSTRUCTIONS FOR DATA COLLECTION COMEHIEB_INSIBUCIIQNS TURN ON COMPUTER, BOTH SCREEN AND BASE. AFTER THE MENU COMES UP, PRESS THE ESCAPE KEY. HIT RETURN AT "F>" PROMPT TYPE a: HIT RETURN KEY TYPE regress HIT RETURN KEY AT THIS POINT YOU WILL SEE DATA REPRESENTING THE VALUES OF THE SEASONALITY AND PRICE/PROMO INDICES ALONG WITH THE ACTUAL PASSENGER MILES FOR THE PREVIOUS TWENTY WEEKS (BEFORE YOU GOT THE JOB). TYPE retrieve 'a:simdata' TYPE regress c1 on 2 in c2 c3 TYPE stop YOU HAVE NOW REGRESSED THE TWO CUES ON THE ACTUAL PASSENGER MILES. THE MULTIPLE R OF THIS REGRESSION IS APPROXIMATELY .80, WHICH INDICATES THAT OVER TIME THE CUES ARE RELATIVELY STRONG PREDICTORS OF THE ACTUAL PASSENGER MILES. THE REGRESSION THAT YOU JUST DID WILL NOW BE USED TO CALCULATE THE FORECAST/STANDARD ERROR INFORMATION FOR YOUR SIMULATION EXPERIENCE. THERE IS NO TRICK IN THIS, 205 THE SEASONALITY AND PRICE/PROMO INDICES WILL BE VALID PREDICTORS OF THE ACTUAL PASSENGER MILES AND THE FORECAST/STANDARD ERROR INFORMATION WILL BE BASED ON THEIR RELATIONSHIPS. 13. TYPE loadsim.bat AND FOLLOW INSTRUCTIONS FROM THE SIMULATION. SIMULAIIQN_INSIBUCIIQNS Your assignment (should you choose to accept it) is to be a load planner for a major air carrier during the period from October, 1977 through August, 1979. The main job of a load planner for an airline is to predict the weekly total number of passenger miles that are flown on the airline nationwide. Your job is extremely important to the airline because you supply the one unknown estimated variable necessary for the utilization of a cost-minimization scheduling algorithm. Given your input (the estimate of the ltotal number of weekly passenger miles) the algorithm schedules the optimal size of plane, crew, and support staff for each of its routes. As you may have already surmised, the effectiveness of the algorithm depends almost entirely on the accuracy of your estimate, since the algorithm is fixed except for your estimates. You have two decision aids at your disposal designed to help you make your weekly scheduling decisions. The first is a numerical index that takes into account the effects of the 206 upcoming week's adjusted seasonality on total passenger miles flown. For example, the fourth week in December and the second week in April may be represented by the same index, since both weeks include a holiday, but the fourth week in July may also have the exact same index because it is near the close of the business fiscal year. Due to the statistical nature of this seasonality index, you do not even have to know which week of the year you are working in since the purpose of this index is to give you information about weekly fluctuations in a standardized form. The second decision aid will be an index derived from other airlines' recent promotions and prices. From these decision aids you must accurately predict total weekly passenger miles for the next 80 weeks. You will have also have information about the standard error range that can be expected around your forecast value. As you may already know, for an accurate forecast about two- thirds of the actual passenger miles flown will fall within this standard error range. On the computer screen yuur estimate of total passenger miles will be graphed like this: The.acnnal passenger miles flown for that week will look like this: 00.0.0.0....OOOOOOOOOOOOOOOOCO The standard error of the forecast will look like this: 207 APPENDIX G POST-EXPERIMENTAL TEST OF SUBJECT'S KNOWLEDGE FOR DATA COLLECTION II PUT YOUR STUDENT # HERE Please fill this out after you are finished with the simulation. REMINDERzThe indices looked like this on your screen: Seasonality __ Price/Promo __ Mark each of the following items either true (T) or false (F): As the seasonality index increased, the actual passenger miles increased. As the seasonality index increased, the actual passenger miles decreased. As the price/promo index increased, the actual passenger miles increased. As the price/promo index increased, the actual passenger miles decreased. As the seasonality index decreased, the actual passenger miles decreased. As the seasonality index decreased, the actual passenger miles increased. As the price/promo index decreased, the actual passenger miles decreased. As the price/promo index decreased, the actual passenger miles increased. Thanx for flying with us!!!!!!!!!! 208 APPENDIX H MEANS AND STANDARD DEVIATIONS FOR Z-SCORE TRANSFORMED DATA FOR DATA COLLECTION II NN NN NN NN vv vv vv vv 2 f... 1.1.x. xSPEh mam. mow. mmv. mvb. mam. omm. vvh. va. cowumfl>wo numocmum vwb.l mHO.I vow. amN. NNH.I NNH. Nam.I Nam. com: uouum pumocmum oz .ummomuom oz uouum oumocmum .umwomuom oz uouum Unmocmum oz .ummomuom uouum oumocmum .ummomuom uouum Unmocmum oz nouum oumocmum unmomuom oz ummomuom HOUOMM COHUQEHOMwCMHB GHOUmIN HOUMG MCOflHMfl>wQ UHMUCMUW UCM mfimmz l HGGE®>mfl£0¢ am magma 209 NN NN NN NN vv vv vv vv 2 wmr. mww. wvm. mww. man. wwh. Ohm. hvh. coflumfl>mo Uumvamum vow.I NHo.I 5mm. wNH. amo.l wwo. mvm.I mvm. com: uouum oumocmum oz .ummomuom oz uouum onmocmum .ummomuom oz uouum oumocmum oz .ummownom uouum oumocmum .ummomuom uouum oumocmum oz nouum Unmocmum ummomnom oz uncommon HOUUMh :ofiumEuommcmua muoomIN umumd mCOADMH>mQ Unmocmum ocm mcmmz I zocmumflmcoo ov manme 210 NN NN NN NN vv vv vv vv 2 mna. va. waN. mob. wan. omo. oaa. mvm. coflumfi>oo numocmum mHF.I HHO. wNv. vhN. mvH.I mvH. Hmm.l Hmm. :mmz momma oumocmum oz .ummomuom oz uouum oumoomum .ummomuom oz uouum oumocmum oz .ummomuom nouum oumocmum .ummomuom uouum oumocmum oz nounm oumocmum ummomuom oz unmomuom HOHUMM :oflumEuommcmuB ouoomIN umumd mcofiumfl>mo oumocmum ocm mono: I acflnoumz Hv OHQMB 211 Z va. Nmm. wvm. vba. hnh. mar. bvb. bah. COfiDMfl>mO oumocmum me.I HoH. mom. NHN. Nwm.I owH. Nwm.I Nwm. :mwz uouum oncocmum oz .ummomuom oz uouum osmocmum .ummomuom oz uouum osmocmum oz .ummomuom uouum oumocmum .ummowuom uouum osmocmum oz uouum oumocmum unmowuom oz unmoouom Houomm :oflumEuommcmua muoomIN umumm mooHDMA>mo osmocmum ocm memo: I zomnsoom Nv OHQmB 212 APPENDIX I CORRELATION MATRIX FOR DEPENDENT VARIABLES FOR DATA COLLECTION II 213 OOOé nvwo. awwd avnd vawd. —Nwd. avvd. Ovnd. _.de. dand. vad. 55nd. onad. aoad. svnd. avnd. v00< OOOé nond :wd wavd. nvwd. anwd- wavd. howd. h—wd- wwwd. wwwd. de. FNad. anwd. avwd- n00< ooo., Nawd- vnwd- nwvd. VOhd- and. hand. anwd- awnd. awnd- pNnd. Sad. avad. howd. N00< . OOOé amvd. anwd. wwvd- Nwwd. vad- novd- wNwd. awnd- 3nd- vad. avwd. Nwad. _.00< OOOA Fwwd nvwd wawd a-wd anvd nNnd .vavd wOwd nowd aavd wNwd 3.5.—RS OOO... ahvd nawd pOwd oavd wand pnwd avwd anwd _.hvd wwwd £0.22 ooo.. vwvd waNd anwd nowd vavd ade Sad de oavd £0.25 Odo; End and ONvd nwwd awnd wand avvd _.kd 50.25 OOOé ovwd aand anwd Ohmd Rad sOvd aavd 3,50 OOOJ Opwd anwd whnd thd Nwwd wvvd n28 ooo.? Nwhd vFNd wwad aaad Owwd N200 OOOJ nNnd tad vwhd wwad P200 OQRFAEYOMENO 23d 33? OOOé Nwwd vad n10< .59? 23d $32 25; FIQ< vQSmeQ< ”X2_—OQ< v n N p «200 MEOO ~200 p200 vIS‘m¥Q< Qkfilwlgv 10.25 10.22 10.22 I0h<2 HH cofiuomaaoo mum: mom mmHQMflum> ucmocmdmo Mom xfluumz :oflumHouuoo mv manna. APPENDIX J STEM AND LEAF PLOTS FOR Z-SCORE TRANSFORMED DATA FROM DATA COLLECTION II mzo_h<>mmmm0 mmOOaIN NI ...................................................................................................................................................................................................... ION wN _. XOOAm I HZm_>_m_>m_=I_0< hm. MEDOE 214 a20_._.<>mmmm0 mmOOmIN ON wN an N 200.5 I H2m_>_m>m=10< an MEDOE 215 a20_._.<>mmmm0 mmoomIN wN an a v.00I_m I ._.Zm=>_m>w_I0< an MEDOE 216 a20_._.<>mmmm0 mmooaIN NI O up ON v XOOILm I ._.Zm=>_m>m=IO< 0v MEDGE 217 aZO_._.<>mmaaO mmOOOIN NI _. XOOJm I >02m_._.w_w200 In MEDGE O or up ON MN on 218 aZO_._.<>mwamO mmOOOIN NI O O_. 0—. ON wN on N xoSm -. >ozm5_wzoo we #59: an 219 m20:.<>mmmm0 mmOOmIN _. 0| a v.00I_m I >02m._.w_wZOO av MEDGE O 2. mp ON 220 m20:.<>mwam0 mmooaIN . O or up ON v XOOAm I >02m_._.w_wZOO . vv MED-0.“. 221 n20...<>mmaa0 mmooaIN ................................................................................................ ..................... F v.00..m I 02.10.25. mv MEDGE O Or ON on ov ow 222 ......................................... ............................................................................................................................................................................................................... ............................................................................................................................. az°....<>mmam0 mmoonIN oooooooooooooooooooooooooooooooooooooooooooooooooooooo O or ON on ov N 200.5 I 02.10._.<_>. wv mmD0_n. 223 az°....<>mmam0 mmOOOIN n v.00I.m I 02.10...<.>_ hv mmD0.n. O_. ON On Ov on 224 a20....<>mmmm0 mmoowIN v 200.5 I 02.10._.<_>_ av “1130.“. O O_. ON on ov on 225 az°....<>mmmm0 mmoonIN 0| .............................................................................................................................................................................................. OF ON .................................. oa ov F 200.5 I >0mmmm0 mmoonIN _.I N 200.5 I >0<1300< Om me0_n. 9. ON on ov on 227 a20...<>mmmm0 mmooaIN a 200.5 I >0mmmm0 mmOOaIN on ....... ...... 0V ow v 200.5 I >0oEo< 8 can: Era 2356 02 :otm 2355 II . 0 mm. $320". 02 m 2 aw. _.h. 5320”. IFIa. _ co... :m Emocflm .6 .8020”. I 5:05.800 em 2%: :owoemzc. :o :05. 2356 :otm 23:45 02 $320". 02 hv. 232 an. .3008”. 3.89.95 :otm Ragga >a 589.0“. I 9203.2 mm 9:9: .Otm DEEP—flaw .05.—m twang—9m OZ O 5:020“. 02 m 05.0 2 «BWFIIIIIIIIIII\III\\\I\I\III\I\I\III\II 5.2 . 6320.... ha F F m F :28225 :05. 2%:me .5 63.9.0”. I 32:84. 8 2:9”. "IIIIIIIIIIIIIIIIIIIIIIIIIS -‘_-_-_-----fll