"34 “a; ’3’. l4. .44mfil: ‘ . .1 i I‘ a.“ 3,2 . 4 “'?"3:3 w flag." 'Ql'é'“ ”)3. .L:. 4R9). :4" 2; - 4 ‘44 . 4'4‘ “K II "' A» 1-“! 4:( “f ‘ ‘fi'fl. ’ . ., riff 33-'2-4’5'4 344534143331... J ,:f. ‘ ‘ '- - 335.3. 5 ' 5'""'54i3"4;5;-‘<’;R\§.44a“' 131‘ ~ 5&‘4155: ""wé 4. ’ILI,‘ 4' J'ffl‘ q! MU I" I'll l' 4‘ > '4‘ . fifl-fiumr :r'fci fi‘g" . .4 \_ f,"'14{.}5 1." ‘1 :g‘figha 15_: 4‘— N ‘L‘ :3". ‘ 1. L 0 - 4. 4, 4 4* -.- Qi‘i-é‘ . 49$" " , :4 _- . WM. " - f, U - F. \ 'h ... v’ —- "4":‘51: 5 4, ‘ ,.":- $.13" 4. 34- 4 4 I, , ' I, D l} . 4‘, ‘ .m ‘& ,4.t 4‘ ‘ (5:. 35‘ 5. fi‘fgf?! s 4 > 42;; . ‘q .- - _._._.. , '-M..o_' \ ,. §>-’ VP Affifi < I Y ' 1 t . 4" - - h": ‘4 .49 49513344422:- ’ . 4 .5 .f ' :1]! 3444!! 39.3% ‘ 4Y4) v ‘V‘I’Vdciflz 2‘ ' ‘ 4 3' 4“ ’J I 5U M I ”‘I‘Jfi' 94'? ‘J‘ v 094., \L- .— ,__ ._ ‘. 1'4?“ . )‘3’ ‘ 3‘ ::1l— _ "{‘Jn $44,323" ”94' R ‘9. . 4, 4;“ 41;. :43‘II‘EI4‘I’H I" ‘ ' t¥.£|1.;"l...l;(l.(“;; 4:4: TERI-I“ ’4 I H ' "."a‘v 4. ;' fl 4 44 4 jlifilf.;. '4'" 5| 344‘} it ‘4. 4'3- 4", 413311133. 33-. nw-LH’ 4 I '4 Iwhlf Q91}; ,‘I‘. ”3' 'l'l‘w: [Fiji I'M" «4| . 4 4 .' ”I; ['4‘ ~ ‘4 :1 :50}. ' I. E] 10“}.(lq‘fi ‘Af'i‘S'l‘J‘J/i” ‘ [’31’jlynqn' 4'43: :3 F‘ a.“ 5444 4.445, . ; ; m}? .crfg. ':4,x~;£34'4>3.~: 3 4f:- {NV '5; ‘ .‘Hi, 34...? I" 'l’ ”Ili'l'lw l‘l i'fllt'm “£4. ' N", 3" “hurt? 4 H 3 A.’144.1t.,. 9 ' yd 'r“ 4 W3“) g‘fl'. 1 3“”! ”if I Mia ' ”KW“ '1 74:“: ""4 (1‘ 4k: ,‘4';.'¢,t-\; ".4 . 5‘5 464'” M 4 . Mun“: .141“; ' .’I “ I l H“ 5‘49 I '3, “ ‘mv '4‘ N '3. Jug. 4:.44‘4 44'er 6:" 1%! 43:14:; simwl l M00339}? W "JAM...“ v_ 4 1424936 x’rw'é' rr‘ £33144 W 19”." 6 ... .‘u‘ U INIWWHWU”NH/iiiilUiiiimi/NI ”WWW 310671 8731 LEZIJZELY (I. . i .:§§t=:ijfji.::;.. ff fate ‘YAN‘9-&J’CE deak"-j I L U". 51" WW 5" 'T"‘ .VY I This is to certify that the dissertation entitled The Interactive Effects of Format and Training on Performance Appraisal Accuracy presented by Elaine Diane Pulakos has been accepted towards fulfillment of the requirements for Ph.D. Psychology degree in Major professor Date fi/y/J/y / 7/ MSU is an Affirmative Action 'Equul Opportunity Insulation 012771 MSU LIBRARIES .—:— RETURNING MATERIALS: Place in book drop to remove this checkout from your record. FINES will be charged if book is returned after the date stamped below. MW 0 «1/ M m THE INTERACTIVE EFFECTS OF FORMAT AND TRAINING ON PERFORMANCE APPRAISAL ACCURACY By Elaine Diane Pulakos A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Psychology 1984 ABSTRACT THE INTERACTIVE EFFECTS OF FORMAT AND TRAINING ON PERFORMANCE APPRAISAL ACCURACY By Elaine Diane Pulakos The purpose of this research was to investigate the interactive effects of format and training on performance appraisal accuracy. Two rating formats, behaviorally anchored rating scales (BARS) and behavioral observation scales (308), were considered, and accuracy training was developed in accordance with the cognitive demands placed on raters by each type of scale. In order to investigate training x format interactions, the effects of each training procedure were evaluated for both BARS and 808. Differences in program effectiveness for various job performance dimensions were also assessed. Finally, it was hypothesized that congruent training (BARS training with BARS and DOS training with 308) would increase interrater agreement. One hundred and forty-four subjects were randomly assigned to 1 of 6 cells defined by the combination of training and format (BARS training with BARS, BOS training with BARS, no training with BARS, BARS training with 808, BOS training with BOS, and no training with B08), and raters evaluated videotaped ratees. Six 3 x 5 (training x dimension) ANOVAs with repeated measures on the dimension factor were conducted to compare differences in accuracy (measured in three ways) among the treatment conditions within each format, and a 6 x 5 (training x dimension) ANOVA with repeated measures on the last factor was performed to compare training effects across the formats. The results showed that for those rating with BARS, accuracy was increased the most when BARS training was used. When BOS was used to evaluate performance, only BOS training had the effect of increasing rater accuracy. Comparisons across the formats indicated that no one format, training procedure, or combination thereof was consistently best, although congruent training was generally better than incongruent or no training. There was also a tendency for BARS ratings to be more accurate than BOS ratings, but caution must be exercised in interpreting these results. Dimension x training interactions suggested that training effectiveness can not be considered independent of the rating content. Finally, interrater agreement was higher in congruent training conditions. Implications, limitations, and directions for future research are discussed. ACKNOWLEDGEMENTS First, thanks to my parents for patiently supporting a child who insisted on twenty-two years of schooling (I think I'm finished). Then, there is my "fine" brother, who listened to dissertation woes, typed large amounts of data into the computer, and was, in general, terrific through the whole process. Many thanks are also extended to my committee, especially for their willingness to facilitate an "express," five month dissertation. I greatly appreciate the work of my Chairpersons, Neal Schmitt and Dan Ilgen. They advised me in designing the study, provided fast feedback on drafts, and were very enjoyable to work with. In addition, thanks to Ken wexley and John wagner for serving on the committee. They asked good questions and provided useful comments. ii TABLE OF CONTENTS Page LIST OF TABLES.................................................... v LIST OF FIGURES.................................................. vi INTRODUCTION...................................................... 1 The Rating Process.............................................. 4 The Roles of Attention, Categorization, and Recall in Performance Appraisal................................. 7 Attention..................................................... 7 Categorization............................................... 10 Recall....................................................... 12 Rating Process Implications for Scale Development.............. 16 Rating Scale Implications for Accuracy Training................ 21 Objectives of the Present Research............................. 25 WODOOCOOOOOOOOOOO0..OO...00......OOOOOOOOOOOOOOOOOOOOOOOO0.... 27 Subjects....................................................... 27 Experimental Design............................................ 27 Procedure...................................................... 28 BARS and Videotape Development................................. 29 Behaviorally Anchored Rating Scales.......................... 3O Generating Intended "True Scores" for Performers............. 30 Developing and Videotaping Performance....................... 32 BOS Development................................................ 33 Behavioral Observation Scales................................ 33 Obtaining Final True Scores.................................... 34 Behaviorally Anchored Rating Scales.......................... 34 Behavioral Observation Scales................................ 35 Format Independent True Scores............................... 35 Rater Accuracy Training (RAT) Programs......................... 38 Program I: BARS ............................................ 38 Program II: BOS ............................................. 41 Summary of the Training Programs............................. 45 iii Pretesting of Training Programs.............................. 45 Data Analysis Procedures....................................... 46 Differential Accuracy........................................ 46 Distance from True Scores.................................... 47 Hit RateOOOOOOOOOCOOOOOOOOOC...0.0.000COOOOOOOOOOOOOOOOOOCOI. 48 Format Free Correlations..................................... 49 Analyses Of varianceOOOOIOOOOOOIOCOOCOOOOOOOOOO0.00.00.00.00. SO Interrater AgreementOOIOO0000......OOOOOOOOOOOIOOOOO0.0.0.... 50 RBUIJTSOOOOOOOC0.000.000...O...00......OOOOOOOOOOOOOOOO00.0.00... 51 Behaviorally Anchored Rating Scale Results..................... 51 Behavioral Observation Scale Results........................... 61 Training and Dimension Effects Across Formats.................. 7O DISCUSSION..OCOOCCCOUCCCCCOOOOOCOCOOCOOOOCOCCOOOOCCOOCO0.0.00.000 76 Practical Implications......................................... 82 Limitations Of the ResearChoo00000000000000000000000000.0000... 87 Summary and Future Research Directions......................... 89 MWBWHESH.H.u.u.n.n.u.u.u.u.u.n.u.n.u.n.u.u..9O A. BEHAVIORALLY ANCHORED RATING SCALES........................ 90 B. BEHAVIORAL OBSERVATION SCALES.............................. 96 C. SCRIPTS FOR VIDEOTAPED PERFORMANCES........................1OO D. BARS CONGRUENT TRAINING....................................133 E. BOS CONGRUENT TRAINING.....................................139 F. RAW DATA AND CODING INFORMATION............................146 REFERENC$OOOOOOOOOOOOOOOOOOOOOOO...O...OOOOOOOOOOOOOOOOOOCO0.00.173 iv LIST OF TABLES True Score Ratings of Performance for BARS................. True Scores Frequencies of Behaviors for BOS............... Format Independent True Score Ratings...................... Means, SDs and Intercorrelations of BARS Variables......... Means, and SDs of BARS Dependent Variables by Treatment.... Results of the Analyses of Variance for BARS............... Intraclass Correlations for BARS by Treatment.............. Means, SDs, and Intercorrelations of BOS Variables......... Means and SDs of BOS Dependent Variables by Treatment...... Results of the Analyses of Variance for BOS................ Intraclass Correlations for BOS by Treatment............... Means and SDs of Format Free Correlations.................. Results of the FF Analysis of Variance..................... Mean Comparisons (FF) - Training x DIM Interaction......... Page ‘36 37 39 52 53 54 6O 62 63 64 69 71 72 74 Figure LIST OF FIGURES Feldman's Rating Process M6del............................. Mean Mean Mean Mean Mean Mean Mean Data (DA) for BARS - Training x DIM Interaction....... Data (DIST) for BARS - Training x DIM Interaction..... Data (HITS) for BARS - Training x DIM Interaction..... Data (DA) for BOS - Training x DIM Interaction........ Data (DIST) for BOS - Training x DIM Interaction...... Data (HITS) for BOS — Training x DIM Interaction...... Data Format Free - Training x DIM Interaction......... vi Page 5 56 57 58 66 67 68 73 INTRODUCTION The most widely used instrument for obtaining performance measures is the rating scale (Borman, 1979). Ratings, often referred to as performance appraisals, are frequently the only criteria available for validating selection procedures, promoting employees, and selecting individuals for training programs. The crucial criterion for judging the quality of performance appraisals is accuracy, which has been defined as the degree to which observed ratings are relevant to true performance scores (Dunnette & Barman, 1979). However, difficulties inherent in obtaining true scores, especially in field settings, have led researchers to concentrate on rating errors (e. 3., halo, leniency, and central tendency) in lieu of directly assessing accuracy. This focus was justified under the assumption that error and accuracy covary negatively. Hence, error reduction seemed to be a reasonable approach for increasing performance rating accuracy. Towards this end, researchers have generally adopted one of two strategies for reducing errors: (1) modifying the rating formats themselves or (2) training raters to eliminate rating errors. Unfortunately, the results of many format comparison studies suggest that scale modification has not been particularly useful for reducing rating errors. For example, although the Behaviorally Anchored Rating Scales (BARS) format is an elegant methodology for developing performance rating scales, little if any psychometric superiority has been evidenced by this approach as compared to others (Bernardin, 1978; Dunnette & Barman, 1979; Schwab, Heneman, & DeCotiis, 1977). In fact, other formats have outperformed BARS at times (Bernardin, Alvares, & Granny, 1976; DeCotiis, 1977), although this may have been due to variations in scale development and scoring procedures not entirely consistent with the original BARS methodology (Bernardin & Smith, 1981). Psychometric comparisons between another sophisticated rating format, Behavioral Observation Scales (BOS; Latham & wexley, 1981), and other types of scales have also produced inconclusive results regarding BOS superiority (or lack thereof) (Bernardin & Kane, 1981). In sum, no one performance rating format has been found to consistently yield appreciably higher quality ratings than the others. Rater training programs have, however, demonstrated some success in reducing common rating errors (Bernardin, 1978; Bernardin & Halter, 1977, Bernardin & Pence, 1979; Barman, 1975, 1979; Latham, wexley, & Purcell, 1975). Although this would seem to imply that error training is useful for improving the quality of performance ratings, recent research, which has focused on accuracy rather than rating errors, has raised serious questions regarding this conclusion. In fact, data from five studies (Berman & Kenny, 1977; Barman, 1975, 1979; Pulakos, 1983; warmke, 1980) have shown error training to have virtually no effect on rating accuracy. Further, the correlations between error and accuracy were extremely low (£_= .00 to .26) bringing into question the once accepted error/accuracy negative covariation assumption. These findings have led researchers (e. g., Cooper, 1981; Ilgen & Feldman, 1983) to call for a moratorium on rating error research and in its place, to recommend that rating accuracy be studied directly. In sum, although a great deal of research attention has focused on improving rating quality through developing sophisticated formats and training programs, these efforts have shown little utility for improving accuracy. Further, researchers have been criticized for taking too narrow a view by concentrating on githg£_rating format g£_ rater training (Zedeck & Cascio, 1982). As Landy and Farr (1980) have noted, further progress towards increasing the accuracy of performance ratings is unlikely to occur until consideration is given to the interactive nature of variables such as format, training, purpose of the appraisal, etc. The purpose of the present research was to investigate the interaction of rating formats and rater training methods on accuracy of performance evaluations. Murphy, Martin and Garcia (1982) have recently provided evidence which suggests that various rating formats place different emphases on the cognitive tasks required of the rater. Implicit in this suggestion is the notion that rater training programs should be developed so as to be congruent with the demands of the particular rating format being used. By drawing on literature from cognitive and social/cognitive psychology (reviewed below), several implications for what types of training might be most useful for increasing accuracy with different formats are suggested. The next section presents a model which is used as a framework for the present research. The Rating Process It has recently been suggested that without attention to the cognitive processes involved in the rating task and the variables influencing these, further gains in accuracy may be difficult to achieve (Atkin & Conlon, 1978; Barman, 1979, 1982; Cooper, 1981; Feldman, 1981; Ilgen & Feldman, 1983; Landy & Farr, 1980; wexley & Klimoski, 1984). While there are variations in cognitive process models of ratings, Feldman's (1981) model is both general and specific enough to be used as a basis for the present research. This model proposes that the cognitive processes involved in the rating task are a special case of a more generalized information processing model. Specifically, Feldman conceptualized the performance appraisal process as a combination of four interacting cognitive tasks (see Figure 1 for an illustration of this model). First, the rater must recognize and observe relevant information about those who are being rated. Second, the information must be stored for later access. New information must also be integrated with previously gathered data. The third step involves recalling relevant information when judgments about performance are required. Finally, Attention Categorization Information Integration Figure 1. Feldman's (1981) Rating Process Mbdel the rater must be able to integrate information into some kind of summary evaluation for most appraisal tasks. While it may appear that these processes occur in a precise temporal order, Feldman cautions that they are interacting, dynamic, and cyclical. Ilgen and Feldman (1983) have further suggested that the central and most important component of the model is categorization, as it serves to link together the other three processes. That is, the categories in use by individuals guide their attention to observing particular stimuli while largely ignoring others, and, in addition, form the basis for subsequent categorizations and recall. What follows is a discussion of each of the components of Feldman's (1981) model as they relate to the performance appraisal process. For the sake of greater clarity, each of these is discussed under a separate subsection. However, it should be remembered that the processes are intimately related, resulting in some degree of overlap among them throughout the presentation. Following this review, two rating formats (BARS and BOS) are considered in terms of this information processing model, and implications for what types of training seem most appropriate for each format are suggested. The Roles of Attention, Categorization, and Recall in Performance Appraisal Attention Individuals have a limited capacity to process the vast amount of information available at any given moment and must therefore be selective with respect to what is actually attended to on a conscious level (Glass, Holyoak, & Santa, 1979). Further, there is a great deal of research which indicates that the majority of everyday stimuli are automatically processed and thus, they are not consciously attended to (Abelson, 1976; Langer, 1978; Nisbett & Wilson, 1977; Schneider & Shiffrin, 1977; Shank & Abelson, 1977; Shiffrin & Schneider, 1977). Race, sex, cues of dress, speech, height, attractiveness, etc. are just a few of the stimuli that can be and often are automatically recorded (Feldman, 1981). For example, upon observing a woman, one does not typically ask, "Is that a female and what difference does it make if it is?" One more generally recognizes sex automatically and therefore reacts, in part, in terms of that classification. Additional research indicates, however,that when sued, subjects can accurately recall those stimuli which they have been told to observe. For example, Averbach and Coriell (1961) conducted an experiment in which two rows of eight letters each were flashed in front of subjects for a tenth of a second. When subjects were subsequently asked to recall as many letters as possible, very few accurate recollections resulted. Subjects were then told to focus their attention to specific positions on the screen (e. 3., they were told to focus on the third letter in the second row). The vast majority were then able to accurately recall those stimuli to which their attention had been directed. The results of similar research by Ericsen & Collins (1969) have also shown the positive effects of directed attention in increasing recall accuracy. In related efforts, Treisman and her colleagues (Treisman & Geffen, 1967; Treisman & Riley, 1969) investigated the effects of directed attention to specific cues. In a typical experiment, these researchers simultaneously presented subjects with a list of digits to each ear, only one of which they were told they would later be asked to recall. Occasionally, a letter was presented with the digits and students were instructed that when they heard a letter in either ear, they should tap their desks with a ruler. If the students had been equally aware of both the "attended" and "unattended" ear, they should have detected the letter equally often in both ears. The results showed that subjects accurately detected about 80 percent of the letters presented to the attended ear but only about 23 percent of the letters presented to the unattended ear. Finally, Lawrence (1971) used a tachistosc0pe to flash a list of words at a person one at a time (at a rate of 20 words per second). When a series of words was presented in this way, he found that subjects could recall very few, if any, of them. However, he additionally discovered that subjects could be cued in advance to read a particular word. In Lawrence's experiment, subjects were told that one word in the series would be in all capital letters, and they were to focus on that word. The results showed that subjects were better able to identify the "target" word than when no cuing occurred. One conclusion drawn was that individuals' attention could be focused to a particular stimulus object rather than the entire modality (i. e., everything they saw). Further, because the target word was defined by a discriminating feature (capital letters), it enabled the participants to more effectively attend to it which, in turn, increased their ability to recall the word. Taken as a whole, this research indicates that individuals can be cued to become consciously aware of particular stimulus objects in their sensory fields, and that such directed attention facilitates recall of pertinent information. A potential caution in interpreting these results in light of their relevance to the present research is that these experiments involved very simple attention and recall tasks (1. e., attending to letters/words presented to subjects for a short period of time). The research proposed here attempts to build upon the theoretical conception discussed above by applying the notion of directed attention to more complex performance evaluation criteria. A more detailed explanation of this is provided subsequent to the discussion of categorization and recall processes. 10 Categorization Within a cognitive psychological framework, no discussion of attention, per se, is complete without a discussion of categorization. This is because the two concepts are intimately dependent upon each other. Bruner (1957, 1958) discusses this interdependence in his contention that attention, hence perception, .i§_the categorization of stimuli whereby individuals assign identity and meaning to an object. That is, individuals attend to and interpret their environment in terms of the cognitive categories most available to them. As such, hypotheses about category membership follow from whatever categories the individual most typically uses to organize and make sense out of the environment. A category has been defined as a cognitive structure that partially consists of the representation of some defined stimulus domain. Categories can further be thought of as pyramid—like structures, organized with general information at the highest level of abstraction and more complete information nested within these general groupings. The lowest level in the hierarchy consists of specific examples of category relevant objects, traits, events, etc. These organizational properties represent an individuals knowledge of the way in which the world is structured. When a stimulus configuration is encountered in the environment, it is matched to some category, and the ordering of the relations among elements in the category are imposed on the elements of the stimulus 11 configuration (Marcus, 1977; Minsky, 1975; Teaser, 1978). This process of ordering and structuring the elements is important because it influences subsequent recall of information and provides the basis for inferences and predictions (Taylor & Cracker, 1981). While it is beyond the scope of the present proposal to review all of the relevant research involving categorization systems, it is worthwhile to note the central role that categories play in phenomena such as implicit personality theory (i. e., categorizations based on trait labels; Hastorf, Schneider, & Polefka, 1970; Lord, Binning, Rush, & Thomas, 1978) and stereotyping (i. e., categorizations based on cues such as race and sex; McArthur & Post, 1977; Taylor & Fiske, 1978). Further, Kelly's (1955) personal construct theory has delineated the sometimes profound individual difference that exist in individual category systems. For example, it has been shown that cultural factors (Triandis, 1964) and individual difference variables such as prejudice and cognitive complexity (Feldman & Hilterman, 1975) make different categories salient for different people. Additionally, situational factors (such as how often a category is used or how recently a category has been used; Wyer & Srull, 1980) affect which aspects of a given stimulus person or object will be used in categorization. Evidence supporting the notion that recall is dependent on the category system employed by the perceiver is presented in the next section. 12 Recs; When confronted with a stimulus configuration (e. g., person, object, or situation), one could conceivably recall any of a variety of stimulus attributes. Information is easier to recall, however, if it is structured in some meaningful way. Further, there is evidence that people structure their observations to facilitate recall (Bousfield, 1953). Because categorization provides a means for structuring and organizing what is observed, it has been suggested that either imposing a category system on stimulus configurations or encountering a stimulus configuration that is a good match to already established categories increases the recall of category relevant information. This contention has been given empirical support by a number of research efforts. For example, Taylor, Livingston, and Crocker (1982) presented graduate students in different departments with an academic folder of a hypothetical student. Subjects were later tested on recall: English students recalled more English relevant material (e. 3., English courses, languages, and writing skills) while psychology graduate students recalled more psychology relevant information (e. g., research experience, psychology courses, math background), even though the experimental task did not require selective use of this material. Thus, the category systems already in use by the students seemed to facilitate attention to and recall of information consistent with these categories. 13 In a study on occupational stereotypes (Cohen, 1977), subjects observed a videotape of a woman performing some daily (non-work) activities, having been told either that she was a waitress or a librarian. In a free recall task, subjects remembered stereotype consistent information more accurately than irrelevant or inconsistent information. Other research has similarly demonstrated the effects of imposed category systems for improving recall of category relevant information (Picek, Sherman, & Shiffrin, 1975; Potts, 1972; Sulin & Dooling, 1974; Well & Yopp, 1978). Recall of events and episodes has also been shown to be selectively improved by the imposition of category systems from external sources. For example, Zandy and Gerard (1974) had subjects observe a videotape of two people poking around an apartment. Some subjects were told that the people were anticipating a drug bust and were looking for their dope so they could remove it. Others were told that the two were planning to rob the apartment, while a third group was told that the two were waiting for a friend and had become stir crazy. The results showed that subjects remembered more features appropriate to the particular scenario they had been given. Other studies have shown that the presence of a theme predicts what specific items, in a set of information items, will later be accurately recalled (Bower, Black, & Turner, 1979; Fredericksen, 1975; Rummelhart, 1975; Thorndyke, 1977). It has also been shown that the information processing task itself influences category selection and hence what is recalled. For 14 instance, Cohen and Ebbesen (1979) and Hoffman and Mischel (1980) have found that information is categorized differently depending on whether one's purpose is to memorize behavior or "form an impression" of a person. Specifically, recall was found to be more accurate when the intent was to memorize and was the most schematically biased when impression formation was the objective. It has been suggested that these results were due to the use of different categories for each task. For example, Hoffman and Mischel (1980) found that "actor's goal" categories were used when the purpose was to memorize, whereas "trait" categories were used when the processing objective was impression formation or behavior prediction. However, Hamilton, Katz, and Leirer (1980) found that impression instructions as compared to memory task instructions facilitated recall, and they concluded that the nature of the material was also an important consideration. In sum, the research reviewed here provides strong evidence that either imposing a category structure on stimulus configurations or encountering stimulus configurations that are already good matches to existing categories increases the recall of category relevant information. It is also apparent that the set created by the observer's purpose influences the way individuals categorize information (Cohen & Ebbesen, 1980) and hence, what they are able to recall (Hamilton, Katz, & Leirer, 1980; Lingle, Geva, Ostrom, Leippe, & Baumgardner, 1979). It is important to note, however, that there may be an upper bound on the degree to which individuals are able to 15 recall very specific information (e.g., what precisely was said, particular behaviors, all relevant attributes) about a stimulus person or object once categorization has taken place. This is because categorization is often conceptualized as a process whereby a stimulus is matched to some category prototype. Further, unique stimulus characteristics become more difficult to recall over time because they are colored in such a manner to be consistent with characteristics of the prototype to which they were matched (Wyer & Srull, 1980). This process is one of cognitive economy and results from the fact that humans have limited information processing capabilities. Consequently, when a rater, for example, is asked to recall information for performance evaluations, some of the information will accurately describe the person in question while other information may not (Cantor & Mischel, 1977, 1979; Sentis & Burnstein, 1979; Spiro, 1977; Tsujimoto, 1978; Tsujimoto, Wilde, & Robertson, 1978; wyer & Srull, 1980). Multiple categorizations are possible (Ilgen & Feldman, 1983), however, and seem dependent on one's expertise and the degree of differentiation in the observer's category system (Rosch, 1978; Rosch, Mervis, Gray, Johnson, & Boyes-Braem, 1976). The above reviewed research indicates that specialized category systems can be imposed on individuals and that these can, in turn, result in increased recall of category relevant information. However, questions remain concerning the degree to which very specific items can be remembered. The following section summarizes the key ideas 16 presented regarding attention, categorization, and recall, and, in addition, discusses rating scale formats in terms of this information processing model. Rating Process Implications for Scale Development Feldman's (1981) rating process model provides a useful theoretical basis which suggests several implications for what types of rating scales might facilitate accurate evaluations of ratees. This model states that the interdependent processes of attention, categorization, and recall play a vital role in the performance appraisal process. Further, although much information processing is carried out automatically, 1. e., without consciousness (Abelson, 1976; Nisbett & Wilson, 1977; Shiffrin & Schneider, 1977; Schneider & Shiffrin, 1977), and there are often profound individual differences in the stimuli attended to, the way they are categorized, and what is recalled, the previously reviewed research on attention, categorization, and recall indicates that these processes can not only be brought into consciousness, but they can also be externally influenced. For example, it has been shown that people can be cued to become consciously aware of certain stimuli in their sensory field, and that this attention increases individuals' ability to recall what is observed (Averbach & Coriell, 1961; Ericsen & Collins, 1969; Lawrence, 1961; Treisman & Geffen, 1967; Treisman & Riley, 1969). It has also been shown that the category systems in use by individuals direct their attention to particular stimuli and provide 17 the basis for interpreting it (Cohen & Ebbesen, 1980; Hamilton, Katz, & Leirer, 1980; Marcus, 1977; Minsky, 1975; Tesser, 1978). Finally, it has been shown that meaningful category systems can be imposed on people from external sources, and that these facilitate recall of category relevant information (e. g., Bower, Black, & Turner, 1979; Zandy & Gerard, 1974). Performance appraisal formats which capitalize upon the major elements of Feldman's model are well represented in the fields of Industrial Psychology and Organizational Behavior. It must be noted, however, that the originators of these approaches have not acknowledged their theoretical consistency with the cognitive psychology area in general. Nevertheless, Behavioral Observation Scales (BOS; Latham & Wexley, 1977, 1980) and Behaviorally Anchored Rating Scales (BARS; Smith & Kendall, 1963) seem to be constructed and used in ways that compliment human information processing capabilities. First, the specificity of the behavioral examples that comprise these instruments would be expected to cue raters' attention to relevant performance requirements of a job. Second, the dimensionality of these types of instruments seem analogous to the structure of cognitive categories. Specifically, BARS and BOS are characterized by several job performance dimensions, each of which is further defined by examples of specific employee behaviors. Hence, on a lower level of abstraction, the organization inherent in personal category systems is replicated in these instruments because the general performance dimensions seem similar to broad cognitive 18 category domains, and the employee behaviors (which may serve to facilitate the development of prototypes of effective and ineffective employees) represent more specific information comprising these "categories." It seems that these instruments would act to impose a common schema or categorization system on raters whereby relevant employee behaviors would be similarly defined, organized, interpreted, and recalled. Further, Ilgen and Feldman (1983) have contended that job-relevant category systems are the source of valid variance in performance appraisals. It would thus seem reasonable to hypothesize that accuracy should be higher when performance ratings are made using BARS and/or BOS. As mentioned previously, however, no clear-cut advantages have been found for these particular types of rating scales as compared to others. One potential reason why such behavioral formats have not generally been shown to be superior is that merely instructing people to use a certain format may not be sufficient to really impose the category structure inherent in that format on their thinking. After all, the typical practice in an organization that is developing these types of rating scales is to include a small subsample of individuals familiar with the job who then aid in developing the performance appraisal dimensions and behaviors. The participation of these individuals could be expected to facilitate their acceptance and use of the category system they mutually conceive of as correct. However, the majority of people who would then be asked to use the new format but who did not participate 19 in its development might be less accepting of the new category system. This relative lack of acceptance may be due to simple unfamiliarity with the category system (general dimensions of job performance) and/or a lack of awareness of relevant behaviors that attend upon it. Previous research has shown that people do tend to use category systems that are familiar to them (wyer & Srull, 1980). Thus, although BARS and BOS formats may provide raters with the ability to rate accurately, persistence in the use of previously learned categories may represent a lack of awareness and/or requisite motivation to really use the category systems provided by these formats. Indeed, the majority of raters are most likely unaware (on a conscious level) of the categories they use to evaluate others. Further, even if this awareness does exist but raters are not convinced that their personal, familiar categorization processes are inadequate, there is little reason to expect that they will embrace a newly imposed system. Several authors have suggested, however, that acceptance and use of an appropriate category system for evaluating ratees might be accomplished through training (Bernardin & Buckley, 1981; Barman, 1979, 1982; Landy & Farr, 1980). In fact, preliminary support for this notion has been provided by Pulakos (1983), who attempted to ‘impgggngpgg raters categories that corresponded to the dimensional system they were using to rate performances of videotaped managers interviewing a problem subordinate. Specifically, a Rater Accuracy 20 Training (RAT) program was developed whose purpose was to familiarize 27 student raters with the meanings of five Behaviorally Anchored Rating Scale dimensions and to provide trainees with examples of the types of behaviors representative of each dimension and their corresponding effectiveness levels. By using the BARS as a training tool, Pulakos hypothesized that the development of a common frame-of-reference for evaluating ratee behavior would be promoted. Further, by focusing trainee attention to the particular effective, average, and ineffective behaviors that defined each rating dimension, trainees were provided with easily detectable standards of what constituted good and poor ratee performance. The ratings of those students who received RAT were shown to be significantly more accurate than the ratings of students who received only error training, both error and accuracy training, or no training. Given that the training strategy used by Pulakos is consistent with and seems to facilitate human information processing capabilities in general, it is intuitively very appealing. However, it is not entirely clear whether or not this type of training would be equally effective with all types of rating formats. After all, such a global strategy does not consider the particular and potentially different cognitive demands that are required of raters when using different types of scales. For example, although BARS and BOS are similar in their basic structure (1. e., they each contain several performance dimensions which are further defined by examples of specific effective and ineffective employee behaviors), in 21 actuality, these formats are quite different in terms of the cognitive demands they emphasize. Thus, while "frame—of-reference" or category system training may be perfectly appropriate for raters using a BARS format, it is not certain that this type of training would facilitate performance rating accuracy on, for example, a DOS. A more complete rationale for this notion is developed in the following section. Rating_Scale Implications for Accuracy Training_ Although BOS and BARS are structurally similar, the components of the rating process that are emphasized with each format are not entirely identical. With BARS, for example, raters must attend to or consciously look for relevant ratee behaviors (i. e., those representing the performance dimensions on which evaluations are made). Once observed, these behaviors must be stored in a manner that will facilitate their recall. As outlined in the discussion of the rating process, this is best accomplished through the use of an appropriate category system. Raters must also make evaluative judgments about what they have observed and translate these into numerical ratings for each performance dimension. In terms of Feldman's model, it can be seen that all four cognitive tasks (attention, categorization, recall, and evaluation) are required in using the BARS format. With a BOS format, raters must also attend to relevant ratee behaviors. Because performance appraisals are often carried out on 22 an annual or semi-annual basis, raters must also categorize their observations in such a manner so that they can be easily recalled at some later point in time. However, a rater's ultimate task with BOS is to indicate, using a Likert-type rating scale, the freguency with which s/he has observed each specific, critical behavior. Thus, BOS involve observation, categorization, and recall, but they do not require raters to make complex evaluative judgments, as is the case with BARS. One implication, as stated by its originators (Latham & Wexley, 1981), is that BOS pose a simpler task for the rater. Although a number of authors have recognized the usefulness of treating behavioral observation and judgments about performance as separate components of the overall rating process (Barman, 1978; Landy & Farr, 1980; Murphy, Garcia, Kerkar, Martin, & Balzer, 1982), questions have been raised regarding raters' ability to recall accurately the large amount of behavioral information required of the BOS format. For instance, there is some evidence that raters may not simply recall events but may first make evaluative judgments about these events. Such judgments may be serving as simplification strategies as subjects map single events onto an evaluative dimension and perhaps then recall events that are consistent with these evaluative judgments. Murphy, Martin, & Garcia (1982) reported that recall for behaviors was structured according to traits when using the BOS format. Again, traits may be serving as a simplification system for recall. It seems that raters do not simply observe and store behavioral information about ratees but rather organize that 23 information around general impressions (Hamilton, Katz, & Leirer, 1980). Further, research on frequency judgments suggests that such judgments represent complex inferences rather than simple counts from a mental checklist (Estes, 1976; Marques & Howell, Note 1). One example of such an inferential process might proceed as follows: Joe is a good leader; therefore, Joe must have frequently exhibited behaviors A, B, and C. Given the need to recall large numbers of specific events with BOS, it is likely that the "frame—of-reference" or category system training suggested for BARS may not be entirely sufficient for training raters using a BOS format. That is, although raters might be trained to understand the dimensional system inherent in a BOS format and might be primed to attend to the types of behaviors that correspond to each dimension and their effectiveness levels, such a strategy may do little, if anything, to assist raters in actually recalling the freguency with which they have observed particular acts. As mentioned previously, this is a result of the fact that once a person is categorized vis—a-vis particular behaviors or characteristics, the features of the category prototype to which they were matched come to characterize the individual, making accurate recall of specific information more difficult to achieve. Therefore, although appropriate category systems do promote recall of category relevant information, they are probably not sufficient to help raters remember the amount of specific information required by the BOS format. The implication is that in order to train individuals to 24 evaluate performance accurately using a BOS, it would seem necessary to sharpen raters' observational skills (so that critical incidents are attended to) and to provide them with aids to facilitate recall of these specific events. Dimensional training, on the other hand, seems much more appropriate for a BARS format. First, since raters must make evaluative judgments of performance on each dimension, one prerequisite to accuracy would certainly be understanding the dimensional system itself. Second, by focusing trainee attention to the particular ineffective, average, and effective types of behaviors associated with each dimension, they are provided with fairly explicit standards for judging performance. Thus, upon observing a particular set(s) of behaviors, it would seem that raters could immediately evaluate their effectiveness and, in essence, would need only to store one "scale value" for each dimension rather than trying to keep a mental count of how often a relatively large number of specific behaviors have occurred. As more performance information is presented, raters using BARS might also be encouraged to recallibrate where a particular ratee "stands" on a given dimension; but again, they would need only remember one scale value per dimension. As such, the demands on raters' recall are significantly less with a BARS than with a BOS format. 25 Objectives of the Present Research The purpose of the present research was to investigate potential interactive effects of rating format and rater training on accuracy of performance evaluations. It has been suggested that different rating formats may differentially emphasize certain cognitive tasks required of the rater. In order to increase accuracy, training programs must be developed with these particular requirements in mind. Two rating formats, BARS and BOS, were considered in this study. Although the two formats are structurally similar, different cognitive demands are emphasized with each scale. The BOS format requires that raters Egggll_the frequency with which a number of specific ratee behaviors have been exhibited, whereas the BARS format requires that raters make evaluative judgments of ratee performance on several rating dimensions. Based upon these differences, two training programs were developed (one for each type of format) with the intent of increasing rater accuracy. Specifically, the BOS training procedure focused on teaching raters observational skills and providing them with a strategy to facilitate remembering how often a large number of critical ratee behavior occurred. With respect to the BARS format, training was focused on providing raters with an appropriate set of standards for evaluating the effectiveness of ratee performance. In order to investigate training x format interactions, the present research evaluated the effects of each general training procedure for both the BOS and BARS formats. Also 26 of interest was the assessment of any potential differences in the formats used and/or in training program effectiveness for different job performance dimensions. Finally, to the degree that training is effective for increasing accuracy, it seems reasonable to expect that interrater agreement would also be increased. Thus, higher rater agreement should be evidenced in groups which receive congruent training than in groups which receive incongruent training or no training. METHOD Subjects Participants in the study were 144 undergraduate students enrolled in introductory psychology courses at a large midwestern university. The total sample consisted of 80 females and 64 males. Their mean age was 19.42, and 83 reported having previous experience with performance appraisals (either having had their performance rated or rating the performance of others). Students were randomly assigned to one of six experimental groups described under the Experimental Design section below (N = 24 per group). Experimental Design The present research employed a 6 x 5 (training x dimension) experimental design with repeated measures on the last factor. The first factor, training, consisted of the following conditions: (1) BARS training followed by rating performance using BARS; (2) BOS training followed by rating performance using BARS; (3) No training followed by rating performance using BARS; (4) BARS training followed by rating performance using B08; (5) BOS training followed by rating performance using BOS; (6) No training followed by rating performance using BOS. This analytical strategy was chosen to compare the effects of training across the formats because there were 27 28 slight differences in the BARS and BOS training programs dependent upon the rating scale being used. That is, the BARS training used with BARS was slightly different from the BARS training used with BOS. Similarly, the BOS training used with 308 was not exactly the same as that which was used with BARS. Since the two BARS programs and the two BOS programs were not identical, it was necessary to consider them as unique treatments. The precise nature of the program differences are described in the section that presents the training treatments. Due to the different scales used, comparisons across the BARS and 808 formats were precluded with some accuracy measures (see the Data Analyses section below). Thus, one-way analyses, comparing the three treatments within each format, were performed for these measures. Procedure The research was explained to potential subjects by informing them that the study involved performance appraisals and they would be asked to rate videotaped performances of a manager talking with a problem subordinate. Participation in the research was voluntary. However, extra credit points were given to those individuals who agreed to be involved in the study. Subjects placed in training treatments attended their respective programs within the next three weeks. In order to keep group sizes manageable, 12-15 students participated in each session. Immediately following the training program, subjects observed and rated videotaped managers. 29 subjects who did not receive training were asked only to observe and rate each manager's performance. However, in order to keep the laboratory time constant for all experimental groups, subjects in the no training groups participated in an one and one-half hour role play feedback exercise (Meier, 1967) prior to viewing the tapes. Specifically, control subjects were divided into pairs and each person was given a written description of either a manager's or a subordinate's role. After reading their respective parts, the "manager" conducted a performance feedback interview with the "subordinate." Subjects exchanged roles and conducted another feedback session. Explicit instructions were given not to discuss how well each person conducted the feedback interview. Subsequent to the role play exercise, the group was told that they would be viewing videotapes of eight managers interviewing a problem subordinate and that their task was to rate how well the manager conducted the feedback session. No mention was made of the rating scales, the performance dimensions, or the types of behaviors indicative of effective and ineffective performance. BARS and Videotape Development This section presents the procedures as described by Borman (1977) for developing the videotapes and BARS used in the present research. 30 Behaviorally Anchored Rating_Scales Performance rating scales for a manager talking with a problem subordinate were developed using behavior scaling methodology (Dunnette, 1966; Smith & Kendall, 1963). Seven-point rating scales were used to represent the following five dimensions of the manager's job: 1. Structure and Control of the Interview. 2. Resolving Conflict. 3. Developing the Subordinate. 4. Establishing and Maintaining Rapport. 5. Motivating the Subordinate. Each dimension was defined by both an overall defining statement as well as scaled behavioral anchors describing the seven different effectiveness levels (see Appendix A for these scales). Generating Intended "True Scores" for Performers To make the videotaped performances as realistic as possible, "intended true scores" were generated in the following manner: First, two realistic covariance matrices were formed by asking experts to estimate the "true" means and standard deviations of performance on each dimension and the "true" intercorrelations among dimensions. Profiles reflecting the "correct” covariance structure were then generated for eight ratee performances. Mere specifically, five expert judges knowledgeable about the 31 job and the concept of correlation were asked to independently estimate the level of correlation expected between each pair of dimensions when the job is actually being performed. To accomplish this, they used a 1 to 7 scale, where 7 indicated.£.- 1.00; 6, 5_- .67; 5, .E.‘ .33; 4, 3;- .OO; 3, g;- -.33; 2, E" -.67; 1, £2 -1.00. A descriptive estimate of the reliability associated with these judgments was obtained by using an ANOVA procedure to compare the variability in different judges' ratings of the same dimension pairs with total variance in the judgments. The resulting intraclass correlation for these judgments was .81 (p < .01), suggesting acceptable reliability for the judgment task. Mean ratings (on the 1 to 7 scale) were computed for each dimension pair, and these means were transformed directly to correlation coefficients (e. g., 4.5 was transformed to +.17). Following a procedure outlined by Naylor and Wherry (1965), the resulting correlations with dimension means of 4.0 and standard deviations of 1.5 were then used to generate an intended true score matrix for ratees. As an example, presented below are intended performance profiles for two managers: 32 Performance Dimension Profile 1 Profile 2 Structure and Control of the Interview. 6.0 2.0 Establishing and Maintaining Rapport. 4.5 6.0 Resolving Conflict. 6.0 5.0 Motivating the Subordinate 5.0 2.5 Developing the Subordinate 3.5 3.5 The procedures above thus enabled the development of realistic multidimensional performance profiles for eight individuals on the managers' job. Developing and VideotapinglPerformance . Eight scripts were written depicting 5- to 9-minute performances of a manager talking with a problem subordinate (see Appendix C for these scripts). The scripts reflected the performance levels defined by the intended true scores as closely as possible. Eight different actors played the eight different manager roles while the same actor played the subordinate in all eight performances. Each actor was given explicit instruction and was given ample preparation time to insure close conformance to the scripts during videotaping. 33 BOS Development Behavioral Observation Scales The BOS format consists of a series of behavioral statements related to effective and ineffective job performance on each dimension. Raters are asked to indicate, on a Likert-type scale, the frequency with which they have observed ratees engage in each behavior. A total score is then determined by summing the observer's responses to the behavioral items for each performance dimension. Critical incidents originally collected in developing the BARS scales were used to develop the B08. Specifically, BOS items were based upon specific manager behaviors that had been reliably retranslated into a particular dimension and effectiveness level. In order to ensure high correspondence between observed and scaled behaviors, two graduate students carefully reviewed the videotapes and rewrote items as necessary to match more directly the behaviors actually exhibited by the managers. These procedures resulted in several items describing critical incidents that could be observed for each of the five performance dimensions. Each item had a scale ranging from O to 3+, on which raters were asked to indicate the number of times each manager exhibited the behavior (see Appendix B for these scales). It should be noted that these scales differed from typical BOS in the sense that they required raters to report the number of times each behavior occured rather than the relative frequency of its occurance (e. g., 0 to 25 percent of the time). Obtaining Final True Scores Behaviorally Anchored Rating,Scales Fourteen expert raters were selected to evaluate the effectiveness of each performer. Seven of the raters were graduate students in psychology and seven were practicing industrial psychologists working for a psychological consulting firm or in the personnel research department of a large manufacturing company. All of the raters were very familiar with the performance demands of the job. The scripts were revised as necessary to reflect the verbal behavior actually depicted in the performance, and raters were asked to study these scripts and the rating scales prior to coming to the rating sessions. Experts' ratings were analyzed using an indirect validation approach. Interrater agreement among the 14 experts was computed for each dimension using intraclass correlations. The resulting eight intraclasses ranged from .91 to .98 with a median of .97. Further, correlations between mean expert ratings and intended true scores were all above .70, with a median £_= .93. These results indicated considerable agreement between the expert judges and intended true scores. The high interrater agreement obtained for each dimension suggested that the few times the mean expert ratings did differ from the intended true scores, the discrepancies were most likely due to the scripts reflecting unintended levels of performance and/or the actors failing to project the intended effectiveness levels. The 35 mean expert ratings (see Table 1) were therefore adopted as the "true scores" for subsequent uses of the tapes. Behavioral Observation Scales Ten expert raters who were familiar with the performance demands of the job were selected to evaluate the effectiveness of the managers using the B08. They were given the scripts and the rating scales and were asked to study these prior to making their ratings of the manager performances. True scores were determined for each item if at least eight of the ten raters agreed upon the number of times it occurred. Interrater agreement was quite high for the BOS. Out of the 192 judgments made (24 items for each of eight ratees), there was 1002 agreement on 146 of the items, 902 agreement on 35 of the items, and 80% agreement on the remaining 11 items. The BOS item and dimension true scores appear in Table 2. Format Independent True Scores In order to compare the effects of training across the two rating formats, it was necessary to generate true scores of performance that were independent of the two rating scales themselves. In a similar format comparison study, Borman (1979) developed an expert rating task to obtain true scores that utilized a format which was quite different from the formats to be used in this study. Specifically, eight experts were required to place the managers on a 24-point scale for each dimension. One of the ratees 36 Table 1. True Score Ratings of Performance for BARS Manager Dimension 1 2 3 4 5 6 7 8 Structure and Control of the Interview 3.31 2.79 2.79 6.92 2.07 4.54 4.38 3.08 Establishing and Maintaining Rapport 3.69 1.50 5.93 3.26 5.00 5.23 3.08 1.38 Resolving Conflict 5.69 2.07 4.07 5.62 4.00 4.31 2.85 2.08 Mbtivating the Subordinate 5.77 2.29 4.86 4.62 3.71 6.15 2.77 2.08 Developing the Subordinate 6.08 2.71 3.07 3.38 2.93 6.62 4.54 1.38 37 True Score Frequencies of Behaviors for B08 Table 2. Manager Dimension Structure and Control of the Interview -2 0 Total 011000 010000 000000 000000 030000 000000 t r 0 a» an m... in hi 8D .11 18 .3... m1 5.. 0 Total 02100 01000 00100 01100 10020 00200 02000 20100 Resolving Conflict 3 Total 00001 22010 10000 Metivating the Subordinate 4 Total 001000 100200 200202 000000 000210 010010 010010 110202 Developing the Subordinate 4 Total 38 was assigned a "24" (most effective), another ratee was assigned a "1" (least effective), and the other six ratees were interval scaled into the remaining 22 positions. The mean expert ratings were used as the final true scores (see Table 3). Interrater agreement for these expert ratings as indexed by the intraclass correlations ranged from .91 to .98. Agreement with the expert ratings obtained on the BARS and BOS scales was considerable: correlations (by dimension) between the format free and BARS ratings ranged from .83 to .94, with a mean of .90; correlations between the format free and BOS ratings ranged from .84 to .96, also with a mean of .90. Finally, the correlations between the BARS and B08 expert ratings was also high (range from .91 to .97; mean £_= .94). Rater Accuracy Trainingfi(RAI) Programs Program I: BARS Pulakos (1983) developed a training program to increase rating accuracy. The training focused on providing raters with a common categorization system and set of standards for evaluating ratee performance. This was accomplished by using a behavioral rating instrument as a training tool along with focusing rater attention to the particular job performance dimensions and examples of what types of behaviors constituted various effectiveness levels within each. The general strategy used by Pulakos was employed in the present study to train raters using both the BARS and BOS formats. 39 Table 3. Format Independent True Score Ratings Manager Dimension 1 2 3 4 5 6 7 8 Structure and Control of the Interview 13.9 8.3 5.6 24.0 1.9 19.6 14.8 6.4 Establishing and Maintaining Rapport 10.5 2.4 19.4 14.9 13.0 20.8 11.3 2.3 Resolving Conflict 17.6 2.1 7.0 22.9 8.0 13.0 8.5 2.6 Developing the Subordinate 18.0 3.8 12.4 17.5 9.0 23.4 9.3 3.4 Mbtivating the Subordinate 14.7 1.9 11.1 19.5 8.1 19.4 11.6 3.1 40 Those who received this training were first lectured on the multidimensionality of most types of jobs and the need to pay close attention to ratee behavior in terms of those dimensions (see Appendix D for a script of the training program). Participants were then given the actual rating scales they would use to evaluate each manager (1. e., one group was given BARS and the other was given BOS) as well as a separate list of the dimensions and their definitions. After discussing the meaning of each dimension, the trainer presented the group with examples of specific behavioral incidents that corresponded to it. For those who would be using BARS, the trainer discussed these behaviors in terms of what would be expected of a manager who should be rated a "7" as compared to a "5" versus a "4" versus a "2" etc. For those who would be using BOS, the trainer discussed each of the behavioral incidents in terms of whether it represented effective or ineffective behavior on the dimension. Because the BOS group knew they would ultimately be asked to rate the frequency with which particular behaviors occurred, it was necessary to provide them with an explanation that would legitimize their training program. Thus, prior to discussing the dimensions and critical incidents, the trainer stressed that an important prerequisite to making accurate ratings was understanding the dimensional system inherent in the scales, and that the best way to gain such an understanding was to discuss the effectiveness levels of various behaviors representative of each performance category. 41 Interviews with subjects who participated in the training program pilot tests revealed that this explanation seemed quite reasonable to them and did not foster any questions regarding the appropriateness of the training content for their particular rating task. Further, no questions were raised during the actual experimental sessions which indicated that subjects might be skeptical about why their particular training was undertaken. In both the BARS and BOS groups, subjects practiced using their respective rating scales by rating two of the eight videotaped managers. After viewing each tape, the group(s) discussed their ratings and received feedback on their accuracy. The BARS group received evaluative feedback (i.e., what scale value was the correct rating for each dimension), while the BOS group received frequency feedback (i.e., how often each critical behavior occurred). Because two of the tapes were used for training, the experimental results that follow are based only on the remaining six manager performances. ‘Egggram II: BOS In order to rate accurately using a BOS format, two components of the rating process seemed most critical. The first was that raters attend to and recognize critical behaviors that are exhibited by ratees. Second, because raters were required to recall the frequency with which they had observed a large amount of specific behavioral information, it seemed important to provide them with an aid to facilitate the recall process. As discussed in the 42 introduction, this was necessary because of limited information processing capabilities and the inability to accurately remember large quantities of specific information. Further, congruent with the BOS philosophy (Latham & wexley, 1981), raters should also be encouraged only to observe, rather than to evaluate the effectiveness of, relevant ratee behaviors. Those who received Program II first received a lecture on the importance of attending to relevant ratee behaviors (as opposed to traits) and on the difference between observation and evaluation. The importance of not making judgments regarding the quality of performance was stressed (the training script appears in Appendix E). Participants were then given lists of specific critical behaviors that corresponded to each of the five rating dimensions. In order to minimize the possibility that raters would form general impressions of effective versus ineffective performers (hence providing them with evaluative prototypes to which ratees could be matched), no mention was made of the degree to which any of the behaviors were characteristic of good or poor performance. Further, the behaviors were randomly ordered within each dimension as opposed to being listed, for example, from most favorable to least favorable. Trainees were told that their next task was to memorize the behaviors that appeared within each dimension. Specifically, they were told to read over the behaviors and to mentally rehearse the ones that corresponded to each dimension. Without referring back to the scales, trainees were asked to write down the dimension titles 43 and the items that fell within them. This procedure was repeated twice and subsequent to each trial, subjects corrected their responses by consulting the list of behaviors. The purpose of this component of the training program was to sharpen raters' observational skills by teaching them which particular behaviors they should recognize as important when observing the managers' performance. Further, given that the behaviors were presented in meaningful dimensional groupings, it was hypothesized that this memorization task would facilitate the retention of more information than what might otherwise be possible. This seemed reasonable, given that retention has been shown to be a function of activities such as rehearsing information and identifying inputs as meaningful chunks (in this case, behaviors as representative of particular dimensions) (Bousfield, 1979). Trainees were then told that they should use a mental checklist when observing the tapes. Specifically, when observing the managers, it was suggested that subjects ask themselves the following questions: (1) Is the behavior I am observing one of the critical ones that I should be attending to? If no, ignore. If yes, have I seen this behavior before? If no, store a 1 for the behavior. If yes, how many times; add N + 1. Craik and Lockhart (1972) argue that information can be processed and rehearsed at various levels and that deeper levels of analysis result in superior long term retention. Thinking about first, whether or not a behavior is relevant and second, going through the counting process described above was 44 hypothesized to promote a deeper level of analysis, which was expected, in turn, to facilitate more accurate recall of how often each of the important ratee behaviors occurred. As was necessary when BARS training was used with the BOS format, an explanation was provided to the BOS training/BARS group to legitimize the training procedure. Specifically, those using BARS were told that people often make immediate evaluative judgments of ratees that are based on far too little information and are thus often incorrect. They were further told that although they would ultimately have to evaluate each ratee using a seven point scale, it was very important to postpone making any judgments until they had adequately sampled ratee behavior. It was explained that focusing their attention.ggly_on observing relevant ratee behaviors should help them'gg£.to judge prematurely which, in turn, would facilitate more accurate evaluations. Finally, they were told to translate all of their observations into an appropriate scale value upon completion of each tape. However, no instructions were given regarding specifically how this translation process should be accomplished. Once again, interviews with pilot test subjects indicated that this rationale for training seemed quite plausible, and actual experimental subjects raised no questions that suggested otherwise. Subjects viewed the same two videotapes used in Program I and practiced observing and rating the managers' performances. The group discussed what ratings they had given to each ratee, and the trainer provided feedback on these responses. Frequency feedback was given 45 to those using the B08 and evaluative feedback (i.e., the correct scale values for each dimension) was given to those using the BARS. Summary of the Training_Prggrams Based on the previous discussion, it can be seen that both training programs were developed to elicit active trainee participation and to provide raters with practice and feedback on their judgments. Also, the same two videotapes and relevant performance rating criteria were used in both programs to train participants. Further, in order to control for variance due to differences in the amount of actual training time, all experimental sessions were limited to approximately one and one-half hours in length. In summary, the programs were identical with respect to their training components (i.e., practice and feedback), their training tools, and duration. However, the orientation of each program was different. Program I (BARS) focused on providing raters with an appropriate organizing schema and a set of standards for evaluating ratee performance on each dimension, whereas Program II (BOS) focused on observation as opposed to evaluation and provided aids to facilitate recall of the frequency information required of the BOS format. Pretesting:of Training_Programs Prior to the experimental treatments, Program I using the BOS format and Program II using both the BARS and BOS formats were each 46 pilot tested with two groups of 10 to 15 students. Pretests of Program I using the BARS scales were not conducted because this program had been tested extensively in previous research (Pulakos, 1983). These pilot tests were performed to provide the trainer with practice conducting the training sessions and also to discover any potential problems with the programs so that modifications could be made prior to the actual research. Data Analyses Procedures Items on the BOS that depicted ineffective manager performance were rescored prior to computing the accuracy scores so that large values on each dimension reflected better performance. Specifically, a 1 was scored as -1, a 2 as -2, etc. Thus, for all dimensions on both the BARS and BOS, higher values indicated better performance. A total of four accuracy measures (Distance from true scores, Differential Accuracy, Hit Rate, and Format Free correlations) were calculated and are described in detail in the following subsections. Differential Accuracy Accuracy was assessed using Cronbach's (1955) differential accuracy (DA) measure, which is the correlation between true and observed scores. For the BARS, DA was calculated for each of the five performance dimensions by correlating the observed scores with the true scores across the six ratees. For the BOS, a correlation was computed for each dimension between the true scores and the 47 observed scores across all items and ratees. That is, for example, on the first dimension, the N size for the correlation was 12 (two ratings for each of six ratees were correlated with the true scores for these items); on the second dimension, the N size for the correlation was 36 (six ratings for each of six ratees were correlated with the corresponding true scores); etc. Although some observations were nonindependent (i. e., each ratee's scores across items), it was not feasible to assess DA at the item level in any» other way. Prior to analyzing these data, all correlations were transformed to z scores using Fisher's r-to-z transformation. Distance from True Scores A Distance measure was also computed for each of the dimensions, which assessed how close the observed scores were to the true scores. For each subject, this resulted in five mean deviation scores across ratees, with lower deviations indicating higher accuracy. The formulas used to calculate Distance (DIST) for the BARS and BOS are presented in formulas l and 2 below, respectively. Slightly different calculations were necessary because DIST was assessed at the dimension level for the BARS (i. e., one rating for each dimension) and at the item level for BOS (i. e., multiple items for each dimension). It is worthwhile to note that DIST could not be as great with BOS (four scale points) as it could be with BARS (seven scale points). R DIST -( 2 D)/R (1) r-l where: DIST - accuracy scores for each dimension across ratees. R 2 number of ratees (6). D - absolute difference of the observed score from the true score. R I DIST -[ I ( 2 D)/I]/R (2) r-1 1.] where: DIST . accuracy scores for each dimension across ratees. R a number of ratees (6). I - number of items per dimension. D - absolute difference of the observed score from the true score. Hit Rate Hit Rate (HITS) was defined as the percentage of correct ratings made by observers on each performance dimension. The formulas used to calculate HITS appear below for the BARS and BOS, respectively. Again, slightly different calculations were necessary because Hit Rate accuracy was assessed at the dimension level for BARS and at the item level for BOS. Also, because the BARS true scores were the mean values of the experts' ratings, it was necessary to round each true score to the nearest whole number prior to calculating the HITS measure . 49 R HITS -( 2 H)/R (3) r-l where: HITS - percentage of correct ratings for each dimension across ratees. R a number of ratees (6). D - number of correct ratings per dimension. R I HITS -[ 2 ( 2 H)/I]/R (4) r-l i-l where: HITS c percentage of correct ratings for each dimension across ratees. R - number of ratees (6). I . number of items per dimension. H a number of correct ratings per dimension. Format Free Correlations In order to compare training effects across the different rating formats, Format Free (FF) accuracy was computed by correlating the format free true scores with the observed BARS and BOS dimension scores. This analysis was performed across ratees for each performance dimension, resulting in five Format Free correlations for each subject. BOS dimension scores were calculated by summing, for each ratee, the individual items within each dimension. Prior to subjecting these data to an analysis of variance, Fisher's r—to-z 50 transformation was applied to all FF correlations. Analyses of variance A total of seven analyses of variance were conducted. First, for only those groups using the BARS format, three 3 x 5 fixed-factor ANOVAs with repeated measures on the dimension factor were performed to assess training and dimension effects on accuracy measured in terms of DIST, DA, and HITS. These analyses were then repeated comparing only the B08 format groups. Finally, the Format Free correlations were subjected to a 6 x 5 fixed-factor ANOVA with repeated measures on the last (dimension) factor. This analysis was conducted in order to evaluate training and dimension effects across the two types of rating scales. Interrater Agreement In order to evaluate the effects of training on interrater agreement, intraclass correlations were computed for each dimension on the ratings made by subjects within each of the experimental groups. Differences between these intraclass correlations were compared separately for the three conditions in which BARS were used and for the the three conditions in which BOS were used. RESULTS This chapter is divided into three major sections: results of the analyses for those rating with Behaviorally Anchored Rating Scales, results of the analyses for those rating with Behavioral Observation Scales, and results of the analyses that compared training and dimension effects across the two formats. Behaviorally Anchored Rating_§cale Results The means, standard deviations, and intercorrelations for subjects' accuracy scores (DA, DIST, HITS, and FF), sex, age, and previous experience with performance appraisal are presented in Table 4. The four accuracy measures were quite highly intercorrelated (ifs ranged from .47 to .82,‘p.< .05). DIST, of course, was inversely related to the other three accuracy measures, as smaller absolute differences from the true scores indicated higher accuracy. The means and standard deviations for the three within format accuracy measures (DA, DIST, and HITS) by training condition appear in Table 5. Three (one for each dependent accuracy measure) 3 x 5 (training x dimension) fixed-factor ANOVAs with repeated measures on the second factor were used to assess training and dimension effects. Results of these analyses showed significant main effects for training on each DA, DIST, and HITS accuracy (see Table 6). 51 52 Table 4. Means, SDs, and Intercorrelations of BARS Variables Variable Mean SD‘ (1) (2) (3) (4) (5) (6) (7) 1. DA 1.01 .28 2. DIST 1.14 .28 . HITS .31 .09 -.77 3 4. FF 1.08 .28 -.69 .47 So SEX 1.61 .50 -006 .00 003 6. AGE 19.82 2.02 .05 -.09 .01 -.11 .10 7. EXPER 1.49 .49 —.08 .06 -.12 .21 .05 .00 Note. Correlations greater than .18 are significant, p < .05. DA = Correlation (transformed to z scores) between true and observed scores; DIST - Average difference between observed and true scores; HITS a Percentage of correct ratings; FF 3 Correlation (transformed to z scores) between observed scores and format free true scores. 53 Table 5. Means and SDs of BARS Dependent Variables by Treatment Dependent Differential Variable Accuracy (DA) Distance from True Scores (DIST) Percentage of Correct Ratings (HITS) BARS BOS N0 TRNG TRNG TRNG BARS BOS TRNG TRNG N0 TRNG BARS BOS N0 TRNG TRNG TRNG DIM 1 DIM 2 DIM 3 DIM 4 DIM 5 1.73 1.43 .92 (.49) (.51) (.59) .72 .83 .80 (.37) (.43) (.55) 1.08 .83 .63 (.43) (.43) (.43) 1.23 .86 .94 (.63) (.46) (.47) 1.33 1.03 .87 (.34) (.27) (.33) .58 .95 (.20) (.30) 1.23 1.23 (.30) (.35) .87 1.22 (.29) (.34) .83 1.24 (.41) (.34) .91 1.25 (.25) (.32) 1.57 (.46) .75 (.45) 1.20 (.37) 1.26 (.59) 1.22 (.40) .58 .33 .33' (.17) (.19) (.17) .22 .24 .15 (.14) (.18) (.12) .39 .31 .22 (.24) (.16) (.16) (.18) (.20) (.18) Tbtals 1.21 1.00 .83 (.21) (.22) (.28) .89 1.17 (.16) (.16) 1.36 (.28) .39 .28 .24 (.08) (.06) (.07) Note. The numbers in parentheses are standard deviations. Dimensions are: 1 - Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 - Resolving Conflict; 4 . Motivating the Subordinate; 5 - Developing the Subordinate. Table 6. Results of the Analyses of Variance for BARS Dependent Variable DA DIST HITS Effect df F “,2 F .,,2 F “,2 Training 2 15.86* .30 33.02* .47 28.78* .44 Subjects within 69 (.28) (.21) (.02) groups Dimension 4 19.51* .19 13.74* .15 10.92* .11 Dimension x Training 8 4.05* .06 2.23* .03 2.85* .04 Dimension x subjects within groups 276 (.19) (.11) (.04) Note. Numbers in parentheses are the mean square error associated with the F tests directly above them in the table. The omega squares are computed separately for the between and within subject effects and are thus not directly comparable. 55 Mean comparisons using Scheffe tests revealed similar results for the DA and DIST measures. Those who received BARS training had significantly more accurate ratings than those who received BOS training or no training. Interestingly, ratings from the BOS training group were significantly more accurate than ratings from the no training group. With respect to the HITS measure, BARS training again produced the most accuracy, but there was no difference between the BOS training and no training conditions. Thus, students who used BARS and received training that was congruent with that particular type of rating scale produced more accurate ratings than students who received incongruent (BOS) training or no training. For each accuracy measure, a significant main effect for dimension and a significant training x dimension interaction (see Table 6) also resulted. The interactions for DA, DIST, and HITS are presented in Figures 2, 3, and 4, respectively. Again, similar results were observed for DA and DIST. Specifically, analyses of the simple main effect (Winer, 1971) for training showed that on Structuring and Controlling the Interview and Resolving Conflict, BARS training yielded higher accuracy than BOS training, which, in turn, yielded higher accuracy than no training. For Motivating the Subordinate and Developing the Subordinate, there was no difference between BOS training and no training, but these treatments produced less accuracy than did BARS training. Finally, there were no significant differences in accuracy between the groups for 56 1,301.. . 1.73 1.70 -- 1.60.} 1.50 dr- 1.40-- 1.43 , 0 1.33 1.30 .. 1.23 1.201b 1.10 J. 1.07 O 1.00 .. .93 1.02 .92 .90" .83 .83 k . .87 .80 4+ .80 v 0 .71 8'1 .70 .. I I 1 l ’T 1 2 3 4 5 1 a Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 = Resolving Conflict; 4 - Metivating the Subordinate; 5 - Developing the Subordinate 0—0 BARS TRNG *-—* N0 TRAINING IF--IIBOS TRNG Figure 2. Mean Data (DA) for BARS - Training x DIM Interaction 57 I l l l T 1 2 3 4 5 1 a Structuring and Controlling the Interview; 2 a Establishing and Maintaining Rapport; 3 a Resolving Conflict; 4 - Metivating the Subordinate; 5 2 Developing the Subordinate 0—0 BARS TRNG *—*N0 TRAINING IF--IIBOS TRNG Figure 3. Mean Data (DIST) for BARS - Training x DIM Interaction 58 .30 «- .25-- .20«- .15 ah .10 db l I l l l 2 3 4 5 1 a Structuring and Controlling the Interview; 2 a Establishing and Maintaining Rapport; 3 x Resolving Conflict; 4 = Motivating the Subordinate; 5 a Developing the Subordinate 0——0 BARS TRNG *—* N0 TRAINING IF--IIBOS TRNG Figure 4. Mean Data (HITS) for BARS - Training x DIM Interaction 59 Establishing and Maintaining Rapport. For the HITS measure, an analysis of the simple effect due to training revealed that on Resolving Conflict, BARS training was superior to BOS training, which was superior to no training. Those who received BARS training had higher accuracy on Structuring and Controlling the Interview and Motivating the Subordinate than did those who received BOS training or no training (there was no difference between the latter two groups). For Establishing and Maintaining Rapport, BARS training and BOS training produced equal levels of accuracy, while the no training group was significantly less accurate. No differences were observed with respect to Developing the Subordinate. To evaluate training effects on interrater agreement, five intraclass correlations (one per dimension) were computed on the ratings made within each of the experimental groups, and these are shown in Table 7. Although comparisons between the groups resulted in only one pair of significantly different correlations, a consistent pattern resulted which suggested training generally had the effect of increasing rater agreement. Specifically, those who received congruent (BARS) training had higher agreement (mean £_ across dimensions a .70) than those who received incongruent (BOS) training (mean £_- .62); but, both of these groups had higher agreement than that which resulted in the no training condition (mean £_=.50). Table 7. Intraclass Correlations for BARS by Treatment BARS BOS N0 Treatment TRNG TRNG TRNG DIM 1 .82 .70 .47 DIM 2 .69 .62 .52 DIM 3 .61 .57 .40 DIM 4 .65 .51 .55 DIM 5 .73 .68 .61 Mean 2'. .70 .62 .50 Note. Dimensions are: 1 = Structuring and Controlling the Interview; 2 = Establishing and Maintaining Rapport; 3 s Resolving Conflict; 4 a Motivating the Subordinate; 5 a Developing the Subordinate. A significant difference resulted for only one pair of correlations: for DIM 1, BARS TRNG (.82) > N0 TRNG (.47). 61 Behavioral Observation Scale Results For those groups rating with BOS, the means, standard deviations, and intercorrelations of the accuracy measures (DA, DIST, HITS, and FF), sex, age, and previous experience with performance appraisal can be found in Table 8. Although all four accuracy measures were significantly correlated, the correlations between the Format Free scores and the other three measures were somewhat lower than those reported above for BARS. This is most likely due to the fact that BOS dimension scores were used in the Format Free analyses, whereas individual item scores were used to calculate DA, DIST, and HITS. Alternatively, all BARS accuracy measures were computed on the dimension scores. Also of interest was the fact that the DIST and HITS measures were correlated nearly 1.00 for BOS, while their correlation was somewhat lower (£.‘ -.77) when BARS were used. Otherwise, the accuracy measures were correlated at approximately the same level for both formats. The means and standard deviations by treatment for DA, DIST, and HITS are presented in Table 9. The analyses aimed at evaluating training and dimension effects employed each of the three accuracy measures in a 3 x 5 ANOVA, with training and dimension (repeated measures) as fixed factors. The results of these ANOVAs were similar and revealed significant main effects for training (see Table 10) on each dependent measure. Scheffe tests showed that for DA, DIST, and HITS, training congruent with the BOS rating format yielded 62 Table 8. Means, SDs, and Intercorrelations of BOS Variables Variable Mean SD (1) (2) (3) (4) (5) (6) (7) 1. DA .74 .17 N . DIST o 34 o 14 “'0 77 o FF 1005 .20 058 -048 049 . SEX 1.50 .50 .07 -.11 .09 .12 . AGE 19.01 1.27 .08 .05 -.03 -.14 -.14 \JO‘U‘Ibw o EXPER 1.45 .45 .04 om -002 .08 -007 .04 Note. Correlations greater than .18 are significant, p < .05. DA 2 Correlation (transformed to z scores) between true and observed scores; DIST - Average difference between observed and true scores; HITS - Percentage of correct ratings; FF - Correlation (transformed to z scores) between observed scores and format free true scores. 63 Table 9. Means and SDs of BOS Dependent Variables by Treatment Dependent Variable Differential Accuracy (DA) Distance from True Scores (DIST) Percentage of Correct Ratings (HITS) BARS BOS NO TRNG TRNG TRNG BARS BOS N0 TRNG TRNG TRNG BARS BOS N0 TRNG TRNG TRNG DIM 1 DIM 2 DIM 3 DIM 4 DIM 5 1.28 1.95 1.29 (.43) (.68) (.50) (.09) (.07) (.09) .45 .56 .40 (.09) (.06) (.09) .75 .94 .74 (.16) (.26) (.30) .36 .18 .36 (.13) (.12) (.21) .45 .19 .42 (.19) (.08) (.12) (.17) (.08) (.16) .33 .14 .32 (.17) (.06) (.13) .70 .83 .68 (.10) (.10) (.13) .64 .83 .65 (.11) (.06) (.09) .71 .78 .70 (.10) (.05) (.11) .58 .75 .58 (.11) (.07) (.10) .71 .87 .73 (.12) (.05) (.09) Totals .68 .89 .66 (.12) (.15) (.14) .40 .20 .41 (.12) (.04) (.11) .67 .81 .67 (.08) (.03) (.07) Note. The numbers in parentheses are standard deviations. Dimensions are: 1 - Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 c Resolving Conflict; 4 - Metivating the Subordinate; 5 - Developing the Subordinate. Table 10. Results of the Analyses of Variance for BOS Dependent Variable DA DIST HITS Effect df F 012 F “,2 F ",2 Training 2 21.64* .37 34.70* .49 41.41* .53 Subjects within 69 (.09) (.05) (.02) groups Dimension 4 200.11* .18 33.03* .29 27.43* .26 Dimension x Training 8 7.34* .12 3.39* .04 2.69* .03 Dimension x subjects within groups 276 (.07) (.01) (.01) Note. Numbers in parentheses are the mean square error associated with the F tests directly above them in the table. The omega squares are computed separately for the between and within subject effects and are thus not directly comparable. 65 significantly higher accuracy than did BARS training or no training. Further, not only were there no differences between BARS training and no training, but the means for these two conditions were virtually identical for each accuracy measure. Significant main effects for dimension and significant dimension x training interactions also resulted from these ANOVAs. The dimension x training interactions are shown in Figures 5, 6, and 7 for DA, DIST, and HITS, respectively. With respect to DA, analysis of the simple main effect for training revealed that BOS training was superior to BARS training and no training on three (1. e., Structuring and Controlling the Interview, Motivating the Subordinate, and Developing the Subordinate) of the five rating dimensions. There were no differences between the conditions on Establishing and Maintaining Rapport and Resolving Conflict. For the DIST and HITS measures, BOS training produced significantly more accurate ratings than BARS training or no training on all_performance dimensions. Again, no differences between the BARS training and no training groups were evidenced. Finally, rater agreement for those using BOS was assessed by calculating intraclass correlations on ratings from each treatment group. These data appear in Table 11. As might be expected based upon the ANOVA results, interrater agreement was higher within BOS training group (mean £_across dimensions - .76) than within both the BARS training group (mean E.‘ .64) and the no training group 2.00.. 10% up 66 1.301. 1.29 1.20 -- 1010 '1' 1.00 a. .80 .- .70.. .604, .50-1 .40.. .30.- ] . Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 = Resolving Conflict; 4 - Metivating the Subordinate; 5 = Developing the Subordinate 0—0 BARS TRNG HNO TRAINING E}--CJBOS TRNG Figure 5. Mean Data (DA) for BOS - Training x DIM Interaction 67 034 ‘h .30 4- .26 ,. .22 .- 018 “r 019 014 0 .14 l I 1 2 3 4 5 — - Cd 1 = Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 = Resolving Conflict; 4 - Metivating the Subordinate; 5 3 Developing the Subordinate HBARS TRNG HNO TRAINING E}--EJBOS TRNG Figure 6. Mean Data (DIST) for BOS — Training x DIM Interaction 68 l l 1 2 wad .k U! 1 - Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 - Resolving Conflict; 4 - Metivating the Subordinate; 5 2 Developing the Subordinate 0—0 BARS TRNG HNO TRAINING E}--CIBOS TRNG Figure 7. Mean Data (HITS) for BOS - Training x DIM Interaction 69 Table 11. Intraclass Correlations for BOS by Treatment BARS BOS N0 Treatment TRNG TRNG TRNG DIM 1 .75 .85 .69 DIM.2 .58 .69 .50 DIM 3 .57 .68 .54 DIM 4 .63 .72 .60 DIM 5 .71 .87 .75 Mean 5 .64 .76 .62 Note. Dimensions are: 1 . Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 - Resolving Conflict; 4 - Metivating the Subordinate; 5 . Developing the Subordinate. 70 (mean E.‘ .62). Although there were no significant differences between these correlations, consistent patterns again resulted for all dimensions. It is worthwhile to note that given the small sample sizes (N . 24 per condition), relatively large differences between the correlations would have been necessary in order to find significant differences. Training and Dimension Effects Across Fbrmats The means and standard deviations of the Format Free accuracy scores are shown in Table 12. A 6 x 5 (training x dimension) ANOVA with repeated measures on the second factor was used to evaluate training and dimension effects on accuracy across the two rating formats. Results of that analysis revealed a significant main effect —_.__. for training, a significant main effect for dimension, and a significant training x dimension interaction (see Table 13). The interaction is diagrammed in Figure 8. Overall, it did not appear as if one rating scale or one format and training combination was consistently superior across all dimensions. Summarized in Table 14 are results of the mean comparisons between training treatments within each dimension. On Structuring and Controlling the Interview, use of the BARS format with either BARS or BOS training produced more accuracy than any other format and training combination. The BOS format with BOS congruent training yielded the next highest degree of accuracy and was also more 71 Table 12. Means and SDs of Format Free Correlations BARS BOS BARS BOS N0 BARS BOS NO TRNG TRNG TRNG TRNG TRNG TRNG 1.57 1.56 .93 1.00 1.17 1.03 DIM 1 (.55) (.57) (.62) (.29) (.32) (.37) 1.14 1.14 1.04 1.09 .96 1.03 DIM 2 (.49) (.35) (.54) (.34) (.48) (.44) .92 .59 .50 1.15 1.18 1.14 DIM 3 (.40) (.26) (.40) (.59) (.40) (.62) 1.23 .98 1.13 .83 1.17 .85 DIM 4 (.47) (.43) (.49) (.35) (.55) (.33) 1.16 1.14 1.14 1.00 1.20 .96 DIM 5 (.40) (.36) (.50) (.37) (.30) (.26) Totals 1.20 1.08 .94 1.01 1.13 1.00 (.25) (.22) (.30) (.23) (.19) (.16) Note. The correlations were tranformed to z scores. The numbers in parentheses are standard deviations. Dimensions are: 1 - Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 - Resolving Conflict; 4 - Motivating the Subordinate; 5 - Developing the Subordinate. 72 Table 13. Results of the FF Analysis of Variance Effect df F “,2 Training 5 4.37* .14 Subjects within 138 (.26) groups Dimension 4 9.95* .05 Dimension x Training 20 5.95* .14 Dimension x subjects within groups 552 (.18) Note. Numbers in parentheses are the mean square error associated with the F tests directly above them in the table. The omega squares are computed separately for the between and within subjects effects and are thus not directly comparable. 73 1.60-- 1.50-- 1.40 +- 1.30 11- .a . 07".- 1.10 . O) 1.004 , . 0% '1’ 0 0804’ / .70 .. .60 ‘1' .50 "" 040 "’ l l 1 l I 1 2 3 4 5 1 - Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 . Resolving Conflict; 4 - Motivating the Subordinate; 5 - Developing the Subordinate ¢I--4I BARS/BARS TRNG C>-—C>BOS/BARS TRNG H BARS/BOS TRNG H BOS/BOS TRNG «x—-—* BARS/NO TRNG H BOS/N0 TRNG Figure 8. Mean Data Format Free - Training x DIM Interaction 74 Table 14. Mean Comparisons (FF) - Training x DIM Interaction BARS BARS BOS BOS BOS BARS Format/ Fermat/ Format/ Fbrmat/ Format/ Format/ BARS BOS BOS N0 BARS N0 DIM 1 TRNG TRNG TRNG TRNG TRNG TRNG 1.56 - 1.57 > 1.17 > 1.04 3 10m ‘ 093 BARS BARS BOS Format/ Format/ Format/ BOS BARS BOS DIM 2 TRNG TRNG TRNG 1.14 - 1.13 > .96 BOS BOS BOS BARS BARS BARS Format/ Format/ Format/ Format/ Format/ Format/ BOS BARS N0 BARS BOS NO DIM 3 TRNG TRNG TRNG TRNG TRNG TRNG 1.18 . 1.15 g 1.14 > 092 > 059 ‘ .50 BARS BOS BARS BARS BOS BOS Format/ Format/ Format/ Format/ Format/ Format/ BARS BOS N0 BOS N0 BARS DIM 4 TRNG TRNG TRNG TRNG TRNG TRNG 1023 - 1.17 - 1.13 > 098 > .85 3 .83 BOS BARS BARS BARS BOS BOS Format/ Format/ Format/ Format/ Format/ Format/ BOS BARS N0 BOS BARS N0 DIM 5 TRNG TRNG TRNG TRNG TRNG TRNG 1.20 3 1.17 - 1.14 g 1014 > low > 095 Note. Mean Comparisons were made using Tukey Tests (Winer, 1971). Dimensions are: 1 - Structuring and Controlling the Interview; 2 - Establishing and Maintaining Rapport; 3 - Resolving Conflict; 4 - Metivating the Subordinate; 5 - Developing the Subordinate. 75 accurate than the remaining conditions. 0n Resolving Conflict, those who rated with a BOS format had more accurate ratings than those who rated with a BARS format, irrespective of training. In terms of Motivating the Subordinate, congruent training treatments (1. e., BARS with BARS training or BOS with BOS training) yielded more accuracy than the other treatments (an exception was BARS with no training, which was not significantly different from the congruent training treatments). For Developing the Subordinate, there were no differences in accuracy between groups which used BARS (regardless of training), and these groups had as much accuracy as the congruent BOS training condition. The incongruent BOS group (BARS training) and the BOS/no training group had significantly less accurate ratings than did the aforementioned groups. Finally, on Establishing and Maintaining Rapport, there were very few conditions that differed significantly. The BARS format with either BARS or BOS training produced more accuracy than the BOS format with BOS training; all other mean comparisons were nonsignificant. DISCUSSION The results of the present study indicate that rating accuracy can be increased by training individuals in a manner that is consistent with the cognitive demands required by the particular rating format in use. Specifically, when a BARS format was used to evaluate performance, accuracy was increased the most when a congruent (i. e., BARS) training program was used. For those using BOS, only congruent (i. e., BOS) training was effective for increasing rater accuracy. Use of an incongruent (i. e., BARS) training strategy with BOS had no effect whatsoever on accuracy. These findings, then, reinforce the contentions of several authors (Landy & Farr, 1980; Zedeck & Cascio, 1982) who have maintained that progress towards increasing rating accuracy is unlikely to occur until consideration is given to the interactive effects of relevant variables. Interestingly, BOS training was shown to produce higher accuracy than no training when a BARS format was used to evaluate performance. Although this result was not hypothesized apriori, it does not seem unreasonable. Given the focus of BOS training, trainees were most likely able to recall which behaviors they had seen each manager exhibit. Further, since the critical incidents in BOS training did 76 77 correspond to dimension anchors that appeared on the BARS, it seems reasonable that raters may have been able to match their observations to a generally appropriate effectiveness level, resulting in fairly accurate ratings. Unlike BARS training, however, BOS training probably did not enable raters to make fine discriminations within general effectiveness levels. The converse (i. e., that BARS training would facilitate accuracy on a BOS) does not seem as likely. Recall that the goal of BARS training was to impose upon raters a common set of standards for evaluating ratee performance. This was accomplished by providing trainees with prototypical examples of what constituted effective and ineffective manager performance for each rating dimension. Because of this training, raters probably did observe the videotapes with an evaluative orientation. When trainees were then required to report the frequency with which specific behaviors occurred, their recall of these behaviors was most likely structured according to their general evaluative impressions (Estes, 1976; Hamilton, Katz, & Leirer, 1980; Murphy, Martin, & Garcia, 1982). There is little, if any, reason to believe that these judgments could have been translated to accurate frequency counts. This is a result of the fact that once a ratee is categorized as, for example, an effective performer, the features of the prototype to which s/he was matched come to characterize the individual (Cantor & Mischel, 1977,1979; Sentis & Burnstein, 1979; Tsujimoto, 1978; Wyer & Srull, 1980), making accurate recall of specific information difficult to achieve. 78 Training congruent with a particular rating format may facilitate accuracy through multiple mechanisms. First, by focusing on components of the rating process that are most important in using a particular scale, relevant rater skills are developed and/or enhanced. Second, research has shown that the goals of the rating task influence how information is processed in terms of what data are sought, how they are stored (Cohen & Ebbesen, 1979), and what can be recalled (Hamilton, Katz, & Leirer, 1980; Lingle, Geva, Ostrom, Leippe, & Baumgardner, 1979). Congruent training makes clear the rating "goals" (frequency counts versus evaluation, in this case) and seems to have the effect of motivating raters to use the rating instrument in an appropriate manner. Finally, several researchers (Bernardin & Buckley, 1981; Borman, 1983; Ilgen & Feldman, 1983) have suggested that rating accuracy can be enhanced by training individuals to use a common, job-relevant frame-of-reference when considering ratee performance. Although frame-of-reference training has been discussed previously as similar to the present BARS program (i. e., providing an understanding of the dimensional system itself and what behaviors constitute different effectiveness levels), the BOS training procedure can also be thought of as one which provides raters with a common (albeit different from BARS) frame-of-reference for approaching performance appraisals. The finding that interrater agreement was consistently higher for congruent training conditions seems to support the notion that raters were similarly oriented to their rating task. 79 The present results also revealed that irrespective of how rating accuracy was measured (with a correlational index, in terms of distance from true scores, or as the number of correct ratings), similar training and dimension effects were observed. Although use of a difference score has been criticized in the past (Borman, 1979; Cronbach, 1955), it is the most conceptually appropriate measure of accuracy for the present study. The hypotheses under investigation here concerned whether or not individuals could be trained to accurately report a ratee's level of performance (for BARS) and a ratee's frequency of behavior (for BOS). By measuring accuracy as a correlation between the observed ratings and the true scores (1. e., differential accuracy), information regarding the degree to which raters were reporting the correct level of performance or the number of times a behavior occurred was lost. Indeed, a rater's evaluations could have conceivably covaried perfectly with the true scores but been quite discrepant from the true score values. In that the DIST and DA measures of accuracy were highly correlated in this and in previous research (Pulakos, 1983), it may not be of great practical importance which measure is used. However, dependent upon the research question, one or another measure of accuracy may be more conceptually meaningful. Another finding of interest was that the training programs employed here were not always uniformly effective in increasing accuracy across the performance dimensions. For example, although BARS training and BOS training generally produced higher accuracy for 80 the BARS format, neither of these training programs had an effect on Establishing and Maintaining Rapport. While only post hoc explanations of this result are possible, this dimension seemed to be less concrete than the others in terms of its general definition. Further, the specificity of the behavioral anchors corresponding to various effectiveness levels within it were somewhat more abstract than those associated with the remaining four dimensions. It is interesting to note that similar results have been found for this dimension in previous research (Pulakos, 1983). When accuracy was measured in terms of HITS, BARS training also had no effect on Developing the Subordinate. For the DA (correlation between true and observed scores) measure of accuracy, BOS training coupled with a BOS format increased accuracy on only three of the five performance dimensions. These were: Structuring and Controlling the Interview, Motivating the Subordinate, and Developing the Subordinate. Again, because of abstractness and lack of specificity, raters may have found Establishing and Maintaining Rapport a difficult dimension to evaluate. It is less clear why BOS training failed to work on Resolving Conflict. However, when accuracy was conceptualized in terms of DIST or HITS, BOS training with a BOS format was uniformly effective in increasing accuracy across all five dimensions. Even beyond looking for general format x training interactions, the present results indicate that useful information might be available through further analysis of training effectiveness within particular formats. Just as certain concepts and ideas are easier to 81 communicate than others, it may be that certain dimensions or traits are easier to rate than others. Concreteness, for example, may play an important role in a rater's ability to understand and hence rate a given dimension. Or, it may be that more easily rated dimensions match more closely with raters personal constructs relevant to job performance. Certainly, future assessments of the particular dimension characteristics that inhibit or facilitate accuracy should prove valuable for rating scale development as well as rater training efforts. In terms of the analysis aimed at comparing training and dimension effects across the two rating formats, it appears as if the highest degrees of accuracy resulted either from training that was congruent with the format (BARS with BARS or BOS with BOS) or from use of the BARS format. Ratings were generally less accurate when a BOS format was used with either incongruent (BARS) training or no training. The notable exception to these trends occurred for the Resolving Conflict dimension, on which the BOS format was superior, in all cases, to the BARS format. A word of caution seems warranted, however, in drawing definitive conclusions based on the Format Free data. Attempting to compare a BOS (frequency) rating format with a BARS (evaluative) format is, at best, a rough approximation and deals only with the rank order of individuals on dimensions. It is also likely that valuable accuracy information may have been lost in this analysis, especially with respect to the BOS. This is because dimension scores 82 were necessary in computing the Format Free correlations, which precluded consideration of accuracy at the item level. Further, it was quite possible for raters to achieve the correct BOS dimension rating but be quite inaccurate at the item level. This was due to the fact that the BOS used here was scored from -3 to +3. So, for example, the correct item scores on a dimension could have been 0, -l, O, 1, 2, yielding a total dimension true score of 2. However, there were numerous ways a rater might have achieved a total score of 2 (e. g., -1, -2, 1, 2, 3 also would have yielded a dimension total of 2). Thus, the degree to which those using BOS were rating accurately in terms of the dimension totals may not have been indicative of their accuracy at the item level. On a more optimistic note, if the ultimate goal is to accurately rank order ratees on dimensions (which is probably somewhat more realistic than expecting completely accurate ratings at the item level), this may not pose a particularly serious problem for interpreting the format free correlations. After all, advocates of the BOS format would maintain that one of its main advantages is increased reliability due to multiple items within each dimension. Thus, as psychometric theory would dictate, random errors at the item level on a BOS should cancel each other out, leading to a more reliable dimension rating. Practical Implications On a practical level, the results of this study indicate that there is no "one best way" of training raters to make accurate 83 performance ratings. Rather, the most appr0priate training strategy must take into account and focus on the cognitive demands placed on raters by the particular format in use. Similar sentiments have been echoed by a number of researchers (Borman, 1979, 1983; Ilgen & Feldman, 1983) in their contention that further advancement in the area of rater training is unlikely without appropriate attention to a process-oriented view of performance appraisal that considers the cognitive processing functions of information gathering, storage, recall, and integration. With respect to the present training strategies, however, several questions remain concerning the degree to which such training will facilitate accuracy in ratings that are, for example, based on six month's or a year's performance. After all, literature from the areas of social and industrial and organizational psychology clearly demonstrates that individuals are £2£_able to recall accurately large amounts of specific information. This is a result of the fact that humans have limited information processing capabilities and consequently, must employ strategies to simplify and condense stimulus information. Thus, regardless of whether raters may be using a BARS or a BOS format, it is probably unreasonable to expect that they will be able to recall specific behaviors over time (Heneman & Wexley, 1983; Murphy et a1, 1982). It may be essential, then, that raters record relevant behaviors as they occur. Otherwise, recall will certainly be limited and may be more a function of general impressions than actual, specific events. 34 In addition to potential limitations with respect to recall, there are other factors that might inhibit generalizability of the present results to typical performance appraisal situations. First, it may be unrealistic to assume that raters will immediately be able to embrace a newly imposed category system for considering ratee job performance. Although training may convince raters that performance is best considered in terms of job-relevant categories, they may naturally revert back to using more familiar categories once they return to their jobs. Further, this may not necessarily be a conscious choice but rather a function of the fact that raters are more comfortable using familiar categories. However, if it were possible to reinforce training at frequent intervals back on the job, acceptance and use of a new category system might be promoted. Another factor that might inhibit generalizability of these results to the field is rater motivation to accurately assess performance. After all, in typical appraisal situations there are many factors (e. g., purpose of the appraisal, interpersonal relationship between a rater and ratee, climate of the organization, whether or not the rater is responsible for developing ratees, etc.) that may account for the majority of the rating variance, and also have the effect of decreasing a rater's motivation to provide accurate ratings. In spite of such potential limitations, however, the training strategies employed here seem potentially useful. Training with the dimensions of the rating scales should promote a more complete understanding of the performance domain and should also propel raters 85 to consider ratee performance in job-relevant ways. In that job-relevant category systems are the source of valid variance in performance appraisals (Ilgen & Feldman, 1983), the types of training suggested here are, at a minimum, a first step towards accuracy. Aside from performance appraisal, however, there are other rating situations for which the present accuracy training effects may be more generalizable. Consider, for example, an assessment center in which ratings of assessee performance are typically made subsequent to each exercise, and thus, demands on a rater's ability to recall are minimized. Further, many of the extraneous factors that may decrease motivation to rate accurately in the field are virtually nonexistent in an assessment center situation. Thus, the training effects obtained in the present study might be equivalent to what would be expected if similar strategies were used to train assessors. Further, recent work of Sackett and Dreher (1982) has brought into question the stability of dimension ratings across exercises, suggesting that assessment centers may not be measuring the intended constructs. Specifically, "exercise factors," as opposed to "trait factors," have resulted from several centers. Given the complexity of typical assessment dimensions such as leadership and analytical skills, the level of inference in moving from observing behavior in a set of exercises to making overall ratings on each dimension is great and may be responsible for the lack of dimension stability. Sackett & Dreher (1984) have also made the point that although assessment center exercise construction is quite rigorous, more attention must 86 be paid to the scoring systems used with them. One possibility may be to develop more behaviorally-oriented rating scales for certain exercises, which tap similar (or possibly different) dimensions to those already used in assessment centers but which lessen the inferential leaps that must be made by assessors. Such rating instruments coupled with the types of training employed here might yield higher consistency of dimension ratings across different exercises. Another rating situation in which training similar to that employed here might prove especially useful is the employment interview. Again, this rating task does not require recall of information over time, and it is also void of many extraneous factors that may account for significant portions of the rating variance in performance appraisals. Previous research has shown that interviewers who rated applicants on specific job dimensions that were relevant to critical behaviors involved in the job were better able to accurately discriminate between more and less qualified applicants than were interviewers who rated applicants using general dimensions (Osburn, Timmreck, & Bigby, 1981). Similarly, other research (Langdale & Weitz, 1973; Schmitt, 1977; Weiner & Schneiderman, 1974) has shown that when job information was available to both experienced and inexperienced interviewers, it was used more readily in their decisions and it decreased the effects of irrelevant information. These findings suggest the importance of providing interviewers with job-relevant rating instruments for evaluating 87 applicants. As the present study would suggest, however, merely providing raters with an appropriate rating format may not be enough to convince them to use the dimensional system within the scales or to focus on the behaviors that attend upon it. Thus, to insure proper use of a rating instrument, training congruent with the scales in use may be necessary. Limitations of the Research Although the hypotheses set forth in this study were confirmed, there are limitations to the research that should be considered in drawing conclusions based on these data. First, undergraduate students and not managers were used as subjects, and consequently, the results can only tentatively be generalized to a true manager/supervisor population. However, it has been shown that employment decisions made by students in laboratory settings are similar to those made by professional interviewers (Bernstein et al, 1975; Schmitt, 1976). Further, the issues addressed in the present study concerned how humans process and evaluate stimuli in their environments. There is no indication from the cognitive psychology literature that this process is appreciable different for students versus "real world" appraisers of performance. What might be appreciably different, though, are the personal constructs of job performance that managers/supervisors have developed versus those of the students. Given that category development is a function of education and experience (Rosch et a1, 1976), it seems logical that 88 the constructs already in use by more experienced managers might be more well-defined than those of a relatively inexperienced student group. Hence, convincing experienced individuals to part with their "folk theories of work behavior" (Borman, 1983) and accept a new strategy for assessing ratee performance might require additional training procedures to those employed here. For example, part of the program may have to be geared toward assessing the personal constructs already in use by trainees and convincing the "owners" of inappropriate ones (i. e., ones that are not based on relevant job performance dimensions) that their present means for evaluating their subordinates is somehow inadequate. Perhaps only then will raters be receptive to training and motivated to use the instruments in an appropriate fashion. It is worthwhile to note, however, that approximately half of the present subjects reported having had previous experience with performance appraisal. Thus, the degree to which experience may or may not necessitate changes in the present training procedures can only be evaluated by future research. Another potential limitation concerns the fact that observations were made from videotaped rather than live performances. It is doubtful that this limitation is severe, as research reviewed by Lifson (1953) suggests that filmed performances are rated the same as live performances. On the other hand, the use of videotaped ratees precludes consideration of many factors that most likely influence "real world" performance appraisals. For example, a ratee's past performance and expected future performance and/or his or her 89 relationship with the evaluator may affect the judgment process in ways that cannot be evaluated using videotapes of unknown individuals. Of course, in an assessment center or interviewing situation, factors such as these would be of much less importance. Also, in light of the inherent difficulties associated with obtaining true scores of performance in field settings, criticisms associated with the use of videotapes do not seem particularly salient, especially given the present research questions. Summary and Future Research Directions This study could be replicated and extended by using more experienced raters and possibly variants of the present training procedures. Of perhaps more importance is the need to investigate the stability of the training effects over time. Further, it is important to note that the training programs developed here were each relevant to a specific, behaviorally-based rating format. Whether or not training could be developed that would increase rating accuracy with other types of formats (e. g., trait) is an interesting question for future research. However, given that accuracy is the crucial criterion for judging the quality of performance ratings, the results of this study should be viewed with optimism. When consideration is given to relevant requirements and demands of the rating task and training is developed congruent with these, it seems that it is possible to increase rater accuracy. APPENDIX A BEHAVIORALLY ANCHORED RATING SCALES 90 91 .§IBUCTURING AND CONTROLLING THE INTERVIEW Clearly stating the purpose of the interview; maintaining control over the interview; displaying an organized and prepared approach to the interview versus not discussing the purpose of the interview and displaying a confused approach; allowing the subor- dinate to control the interview when inappropriate. High Level Performance 7 o Outlines clearly the areas to be discussed and skillfully guides the discussion into those areas. 0 Displays good preparation for the interview and 6 effectively uses information about the subor- dinate to conduct a well-planned interview. Average Level Performance 5 0 States the purpose of the interview but fails to cover some areas he intended to discuss. _______’4 0 Appears prepared for the interview but at times is unable to control the interview or to guide it into areas planned for discussion. 3 Low Level Performance 2 o Fails to indicate the purpose of the inter- view and appears to be unfamiliar with the file information. 0 Appears unprepared for the interview and is 1 unable to control the subordinate during the interview. 92 ESTABLISHING AND MAINTAINING RAPPORT Setting the appropriate climate for the interview in a warm non- threatening manner; being sensitive to the subordinate versus setting a hostile or belligerent climate; being overly friendly or familiar during the interview; displaying insensitivity toward the subordinate. High Level Performance 7 o Draws the subordinate out by projecting sincereity and warmth during the interview. 0 Discusses the subordinate's problems in a can- 6 did but nonthreatening and supportive way. Average_Level Performance 0 Displays some sincereity and warmth toward the subordinate and indicates by his response to the subordinate and his problems that he is resonably sensitive to the subordinate's work— 4 related problems. 0 Uses mechanical means to set the subordinate at ease, i. e., orders coffee Low Level Performance 2 0 Projects little feeling or sensitivity towards the subordinate; makes no friendly gestures. o Is confrontive and inappropriately blunt 1 during the interview. 93 RESOLVING CONFLICT Moving effectively to reduce the conflict between Valve and the subordinate; making appropriate commitments and setting realistic goals to insure conflict resolution; providing good advice to the subordinate about his relationship with Valva, subordinates, etc. versus discussing problems too bluntly or lecturing the subordinate ineffectively regarding the resolution of conflict; failing to set goals or make commitments appropriate to effective conflict resolu- tion; providing poor advice to the subordinate about his relationship with Valva, subordinates, etc. High Level Performance 7 o Effectively reduces conflict between the subordinate and others by making appropriate and realistic comments to help the subordinate get along better in the department. 6 0 Provides good advice about solving problems and about improving the subordinate's.poor rel- ationships with his subordinates, Valva, etc. Average Level Performance 0 Puts forth some effort to reduce conflict between the subordinate and others but usually does not commit himself to helping with this conflict resolution. 0 Tends to smooth over problems and provide reasonably good advice to the subordinate about conflict situations. Low Level Performance 2 o Lectures ineffectively or delivers inappro- priate ultimatums to the subordinate about im- proving his relationships with others or chang- ing his "attitude" towards people or problems. 1 o Fails to make commitments to help the sub- ordinate resolve problems or provides poor advice to the subordinate about his relation- ships with Valva, subordinates, etc. 94 MOTIVATING THE SUBORDINATE Providing incentives for the subordinate to stay at GCI and to perform effectively; making commitments to motivate the subordinate to perform his job well, to remain with GCI, and to help GCI ac- complish its objectives; supporting the subordinate's excellent past performance versus providing little or no incentive for the subordinate to stay at GCI and perform effectively; failing to make commitments encouraging the subordinate's top continued performance; neglecting to express support of the subordinate's excellent perfor- mance record. High Level Performance 7 0 Provides encouragement and appropriate incentives to persuade the subordinate to stay with GCI and perform his job effectively. 0 Uses compliments of the subordinate's technical 6 expertise and excellent past performance to motivate the subordinate to meet the objectives of the department. Average Level Performance 0 Compliments the subordinate appropriately at times but is only moderately effective in using these compliments to encourage high 4 performance, loyalty to GCI, etc. 0 Provides some incentives for the subordinate to perform effectively at GCI, but generally makes few if any personal commitments to 3 support the subordinate in his job. Low Level Performance 2 o Fails to express support for the subordin- ate's past performance. 0 Provides little or no incentive for the 1 subordinate to remain at GCI. 95 DEVELOPING THE SUBORDINATE Offering to help the subordinate develop professionally; displaying interest in the subordinate's professional goals; specifying devel- opmental needs and recommending sound developmental actions versus not offering to aid in the subordinate's professional development; displaying little or no interest in the subordinate's professional growth; failing to make developmental suggestions or providing poor advice regarding the subordinate's professional development. ‘fligh Level Performance 7 0 Displays considerable interest in the subordinate's professional development and provides appropriate high quality developmental suggestions. 6 0 Makes commitments to help professionally in the subordinate's development. Average Level Performance 0 Provides general developmental suggestions but usually fails to make a personal commitment to aid in the subordinate's professional devel- opment. 0 Shows moderate interest in the subordinate's development; may direct the subordinate to seek developmental suggestions elsewhere. Low Level Performance 2 o Expresses little or no interest in the sub- ordinate's professional development. 0 Fails to offer developmental suggestions or 1 provides poor advice regarding the subordin- ate's professional growth and development. APPENDIX B BEHAVIORAL OBSERVATION SCALES 96 97 A. STRUCTURING AND CONTROLLING THE INTERVIEW 1. Outlines specific topics that he wishes to discuss during the interview (personnel problems, a communications failure, Whipker's relationship with his subordinates, Whipker's career goals). 2. Brings up irrevalent topics of conversation such as Whipker's private life, wife, family, courses he took in college. B. ESTABLISHING AND MAINTAINING RAPPORT l. Begins the interview bluntly by saying something like, "What's going on in that department of yours?" or "I've been getting a lot of complaints about you lately." 2. Encourages Whipker to talk about problems (e. g., asks if there is anything special Whipker would like to talk about, tells Whipker to feel free to discuss any problems or asks if there are any issues Whipker wants to get off his chest or problems that he is having due to the transition period). 3. Based on the contents of Whipker's personnel file, brings up negative information about him (a past run-in with Ecklund; Thompson's previous performance appraisal of Whipker). 4. Tells Whipker that there must be a good reason he has so many problems. 5. Tells Whipker that feeling sorry for himself is not going to get him what he wants from Valve. 6. Tells Whipker that he hopes they will have a good working relationship. 98 C. RESOLVING CONFLICT 1. 2. Baxter says he'll approach/talk to valva. Reprimands Whipker by saying that he he has to learn to work and get along with Valva and/or people in general. Reassures Whipker that he is sure things can be worked out with Valva and that the problems with Valva can't be insurmountable. Suggests that Whipker outline in writing specifically what he needs from Valva. States that Valva's department is running so Whipker must be the problem. D. MOTIVATING THE SUBORDINATE 1. Compliments Whipker (e. g., on his high levels of technical expertise, on the excellent job he did on the United Fund campaign, etc.). Tells Whipker that he is really valuable to the company. Tells Whipker that he had better start straightening up his act, be more patient and considerate, or show more respect. Makes commitments to Whipker to check into higher level job openings and to help him get promoted. Tells Whigker he could be doing a much better jo at GCI. 99 E. DEVELOPING THE SUBORDINATE 1. Offers specific developmental suggestions to Whipker like taking the Dale Carnegie course, a seminar dealing with people, courses in the Industrial Relations school. Offers general (as opposed to specific) developmental suggestions like reading books on management and going to personnel to check on possible courses. Tells Whipker that it is up to him to change and is unable to come up with any developmental suggestions, even if Whipker asks for them. Says he wants to meet with Whipker within a certain time period (one week, two weeks) to further discuss his development and/or what he's learning and how it applies to GCI. Tells Whipker that his door is always open and to feel free to come in to talk about developing his management skills, how he could get along better better with others, or any problems he might be having. Gets personally involved in Whipker's development (e. 3., offers to sign Whipker up for a course that will help him in in his development; offers to actually take a course with Whipker). APPENDIX C SCRIPTS OF VIDEOTAPED PERFORMANCES 100 101 MANAGER 1 Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Come in. You're Whipker, aren't you? Yeah. I've been looking over some of these personnel folders, and I think there are some things we should be talking. about. What's this going to be -—- some sort of inquisition or something? No-No-—Nol Why would you think that? I've just got a new job here and I've been meeting with all my people to try to get to know them a little better. I just want to get going on the right foot. I've been trying to learn about what some of your problems are and the way you're approaching some things in the department, and how they work out and I need to talk about some of my problems and see how we can work every thing through as a team. Yeah—-so what do you want from me? I mean you already know what my biggest problem is. Well . . . yeah, you're having troubles with Valva. But that can't be insurmountable. We ought to be able to work something out on that score . . . Look, I'd like to get to know you a little better, Marshall. Tell me about yourself. What the hell do you want to know? I mean, I think every- body knows I pull my weight around here. I like my job but there's just no way of getting ahead—~and then there's this Valva who's always dragging his butt, I can't get work out the way I should. You better do something about that bastard. You sure are hot under the collar about that, aren't you? I can understand the situation and how you feel. You're a damn good technical man, and you set pretty high standards. And I guess part of the problem with Valva is because I didn't get the word on that Tech Services cutback to you. I'm sorry about that, but let's not get hung up on that thing forever. Look, I want to get this straightened out so that your hands aren't tied with other departments. 102 Whipker: Yeah-that sounds just fine, but it's just not that easy. Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Still needs to be solved. Look, I'll talk to valva and explain to him that it was all a part of this misunder— standing. I'll tell him that we're sorry about what happened-—especially that I didn't get the word to you about the Tech Services cutback. Then I can tell him that we ought to try to work together a little better. And I think that I can convince him that he should cooperate with us. I guess it's worth a try, but I wouldn't give great odds on it-—but I'll go along with it if you say so . . . But that doesn't solve anything about getting ahead around here. What are you going to do about that? I mean how come you got that job when I was obviously in line for it? That's what I don't understand. . . .Well—-look, I want you to know, Marshall, that I'm entirely behind you. You definitely deserve to move up in this company. Quite frankly, I don't know all the ins and outs of why I'm in this job and you're not, but since I am in this job I'm going to do what I can to see to it that the next promotion comes your way. Now I can't do that all alone. You've got to give me a hand in this. What do you mean? Well-you know you're a top notch technical man, and got a lot of technical stuff that goes into an R & D record such as you've got. Now, being a manager involves technical knowledge too. It isn't all just straightforward. I know personally that I got some of the technical things about management when I took some night school courses at the local community college in Fairbanks. Now I guess that's about what I really mean when I saying you've gotta give me a hand in this. Maybe take a few courses. Read a few books on manage- ment, and get some of these technical details so that you're ready when the right spot comes along. That's all I'm asking. Well-I guess that sounds okay. . . Well, I got an idea what a good start might be——there's a Dale Carnegie course coming up, and I'd like you to give it a try. Now, they don't have all the answers, but they do a fairly good job of getting into some of the Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 103 simpler things. Now I took it myself about 10 years ago and I thought it was okay. If you're willing to give it a try and attend their sessions, we could get together every week or so and chat about what they're doing in the sessions and what you think of it, and how they might be tying in with the job here at GCI. And that way I could give you a few pointers on what I think about managing, too. What do you think about that? Do you want to give it a shot? I suppose so. I don't have much to lose. Well, that'l be just great. I'll send your name in and then they'll be getting in touch with you. Well why don't we get together in another couple of weeks. By that time I'll have talked to Valva and kind of touched base on how things are going in that situation. Then you'll have been to the first Dale Carnegie session, and we can sort of talk about how that thing looks. How's that sound to you? Well-it's okay to give it a try, but they sure as hell better not try to make me into any four-flusher! Well, you know that can't happen to you, Marshall. You're too solid a citizen for that. Look, I'll be seeing you in a couple weeks and in the meantime I'll sign you up for the Dale Carnegie course. And I'll look into the promotion situation for you. Nowa—I'm not promising anything on that promotion, but if we can find a job that seems right then at least we'll have something which you can be aiming for . . . Well, we haven't yet talked about several things that I wanted to bring up, but I guess this is okay for a start. Okay. Good-Bye. I'll be seeing you. 104 MANAGER 2 Baxter: You must be Marshall Whipker.? Whipker: Yes. Baxter: I'm Dick Baxter. Grab a chair. Whipker: Thank you. Baxter: I called you in to find out what's going on in that department of yours. Whipker: What do you mean? Baxter: Well, I've been reviewing production records and they indicate your department is . . . is doing pretty well but that you've been having some problems with Valva. Whipker: What do you mean problems? Baxter: Well, I understand you swore at him the other day. Whipker: Valva is a dunder head. He's incompetent. He doesn't know what the hell he's doing. Baxter: Well, I've been reviewing your . . . your personnel file, Whipker. I noticed that . . . that you've worked for several companies in the past. Why did you leave Lyon's Podium? Whipker: The supervisors there were incompetent. Baxter: You left IBM because of too much busy work and Seal Electronic because they have advancement potential. Whipker: Right. Baxter: do you see a pattern there? Whipker: What do you mean? Baxter: Well . . . I've also been looking at Thompson's appraisal of you. Whipker: What appraisal? Baxter: Well, the one Thompson . . . you can't have that- Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: 105 that's confidential company records-—the one that he- he did on you the 3rd of May. Let me . . . let me read some of his comments on you. Under "motivating others," "Marshall does not have good rapport with the engineers that report to him, nor is he friendly with persons at his own level of authority." He further says that you're defensive when criticized and intolerent of others who appear less motivated and less committed to their job. Thompson does give you high marks on your technical expertise. He says, for example, that you're highly motivated, always current and up to date, that you always do your job well, and that your technical judgments are excellent. How do you feel about that, Marshall? Well, I agree with some parts and disagree with others.. What parts do you disagree with? I agree with my technical competence. I work hard at that. Well, how about the rest? Thompson feels that . . .that you're weak in human relations. What do you mean, human relations? Look at my produc- tion records. If I had human relations problems, do you think my department would produce the way it does? I've got the best department in the whole plant and my records show that. You come in here and start throwing your weight around-—I should have had that job of yours and I should be sitting in that chair of yours right now. Whipker, I don't like your attitude. That's probably why you didn't get the job, running around blowing up at people—Valve, Ecklund, and now me. What about Ecklund? Well, there's a story here that says you blew up at him. When was that? 1967. Oh my God, two years ago, and . . . and you're . . . you're talking to me about that now? Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 106 It's the same problem, Whipker. You're going to have to straighten up and fly right around here. Ecklund is a good man, but needs to be developed. I'm trying to make him more resourceful by encouraging . . . encouraging to do things by himself and . . . and not depending on others. Are you having trouble at home, Marshall? That's none of your business. it is if it affects your work. Thompson felt that your family situation might be unstable. Is that true? I said that's none of your business. What I do on my own time is of no concern to this company. Look, I didn't have anything to do with getting this job. Management selected me to do this job, and I'm going to do it. As far as I can see, you're a good man at getting the work out, but . . . but you need some development. Why don't you go down to Personnel and find out what classes are available. Try to find something in the people relationship area. What kind of people relationships? Well, something on . . . on how to work better with people. Do you think I would have been able to meet the United Fund goal if I didn't get along with people? I saw that, Whipker. How did you do that? I divided up the goal by departments and told every- one what their fair share was. I see. Now getting back to this Valva business. I don't want that situation to get any worse. Stop bad- mouthing and . . and try to work with him. Valva has his problems, too, and we should try to be more tolerent of each other. You don't know Valva. He's just incompetent. I'll have to find that out for myself, Marshall. In the meantime, you are gonna have to work more effectively with him. I also saw, Whipker, where you 107 were thinking about another job. Whipker: What do you mean? Baxter: I heard a rumor that . . . that you might leave. Whipker: That's none of your business . . . What did you call me in for anyway? I don't think we've solved any of our problems as far as I can see. Baxter: Well, I wanted to get to know you a little better. I hope now that we understand each other, Whipker. I want you to know that . . . that I have an open-door policy and if you have any problems you can always feel free to come in and talk . . . talk to me about what's on your mind, okay? Whipker: Well, I don't see what that's going to accomplish. Baxter: Well, let's try it and see. Thanks for coming in, Whipker. Let's not make it the last time. 108 MANAGER 3 Baxter: 0h hello. Come on in. You must be Marshall Whipker. Whipker: Right. Baxter: I'm Dick Baxter. Very nice——very nice to meet you . . . Go ahead and have a seat. Whipker: Thank you. Baxter: Well, you know, being new to this job, I haven't really gotton much of a chance to know my people yet. That's why I'm really glad that we have this opportunity to visit together today. And I'd like to ask you how are things going on the job? Anything special you'd like to talk about today? Whipker: I don't know. Well, anyway, you're the one that called me in here. I assumed ou were planning on doing most of the talking. t's the way these things usually go. Baxter: Oh, not necessarily. Surely you must have a few things to get off your chest. Whipker: Well, I guess maybe I do. Baxter: What kinds of things would those be? Whipker: Well, for one-—Valva, that jackass down in Tech Services. Baxter: Oh, yeah, that's right. I heard you have been having some problems with him. Why don't you tell me about it? Whipker: Well, he's got those . . . those, uh, guys of his down there screwing around on some mickey mouse project printed circuit boards. And in the meantime, I've got a lot of important work to do and, ah, I can't get it out of 'em because of him and those other jerks. Baxter: Have you tried talking things out with him? It could be a misunderstanding. Whipker: What's the use? I mean I can't get any help from him when when I need it. Like this morning, okay, I go down there and-get a load of this!-he's sitting around playing pgker with those guys he's got working for Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: 109 him . . . well, if you can call it working. Ah, and he gives me some crap about Him not being able to get out, ah, this work for me because his boys are . . . are too "busy" on some other project. Well, you know that's just a bunch of crap. well, I don't know Valva personally, but from what I've heard he seems to be an alright guy to work with. At least I haven't heard any complaints from anyone. Look, I'm sure it's just a misunderstanding between the two of you and it will work out in time . . . Well, by the way, I've . . . I've been hearing, uh, from various people about the excellent job ygg_did last year on the United Fund campaign. Oh yea, really? Right, right. Mm-hm . . . in fact, if you'd like to, I'd really appreciate your taking on the job again this year. Uh-I'm sure you can handle it based on last year's fine performance. Well, okay, why not? I'll do it. Good, I'll talk to Peen, then, and make the arrange- ments. Let's see, uh, where were we? Oh, right-—your problem with Valva. Well, besides that, has anything else been bugging you lately about the job? Ah, yea . . . you! Me?! Well, you certainly are not wasting any time getting to the point! What is it about me that's bugging you? Well, I think I should have had this job instead of you. Everyone knows I'm a hell of a lot more qualified for it. You probably know that. Uh . . . well how so? Well, I've been around GCI for a long time nowe— a lot longer thaniygg, I might add. And I've been doing a damn good job. I know how to handle people and they know that I expect results from them. They know what they have to do-and they‘gg.it. And I don't put up with any of this screwing around that goes on in other departments around here. If you've looked at my . . . my personnel file, you can see that I've never had a bad performance review. I've always met my project deadlines 110 on time and I've made money for the company. And nobody has ever given‘gg any shit about gy_work, see. Baxter: Well, yes, I can see that . . . It looks like your work has been first-rate all the way. Whipker: Okay, then, what's the problem? I mean—-why in the hell do they pass over me in favor of some hot shot like you when it comes to handing out promotions. I don't see it. Baxter: Well, I guess they had their reasons. With Thompson gone, I guess they needed to find a replacement for him as soon as possible and uh-—GCI has been relying very heavily on your technical expertise—-as they have always, of course-—and you've been involved recently in several important projects. Plus the fact that I have had a lot of managerial experience already on my other job. So I guess they opted for someone who could step into Thompson's job right away and more or less pick it up where he left off. Whipker: Okay, okay, but I would like to get into management. I mean I'm not getting anywhere in engineering. Baxter: Oh? Whipker: And uh, besides, that uh, management's where the money is. I mean, it's sure as hell not in engineering. Baxter: Okay, well, Marshall, I suggest that you hang in there for awhile longer on your job before making any decision one way or another. Ah, you've been doing such fine work for us at GCI and we rely so much on your tech- nical competence. Still, if you decide at some point that you want to get into management, feel free to come in and talk with me about it. You know my door's always open. In the meantime, you might want to do some reading on your own, to get a better feel for what a manager's job is all about. In fact, uh, I may be able to give you some suggestions along that line. Does that sound okay? Whipker: Yeah, that sounds good to me but that doesn't help me right now. I mean I still don't know what to do . . . to get into management . . . and I still don't know what to do about that jackass Valva. Baxter: Well, I suggest you take your time. I know you'll make the right decision about management and how to improve 111 yourself. As for Valva—-please don't worry about it. Things like that have a way of working out. Listen, I'm really glad-very glad-we had a chance to get acquainted today. Ah, I do have a meeting to go to in five minutes. And I've enjoyed chatting very much with you though, today, and I hope we can make plans to get together again soon. Whipker: Yeah, I . . . I hope so. I have some things I'd like to talk over with you too. Baxter: Well, we'll definitely have to do that. Whipker: Yeah. Baxter: Good-bye for now. Very nice talking with you. Whipker: Good-Bye. Baxter: Good luck with the United Fund! Whipker: Yeah, thanks very much. 112 MANAGER 4 Baxter: Come in, Marshall, and have a seat. Whipker: Ah, thank you. Baxter: I've been looking forward to talking with you and I'm sorry we haven't had a chance to talk earlier. Whipker: Uh-hum. Baxter: I'd like to cover three major areas with you today, Marshall. The first area pertains to the Technical Services situation and your relationship with Valva. Whipker: Uh-hum. Baxter: The second area pertains to your relationship with your subordinates, and the third, I'd like to talk with you about your career goals and what you want to do at GCI. First, let's talk about the Technical Services situation and the Valva situation. As you see things right now, Marshall, what kind of cooperation are you getting with Technical Services? Whipker: Well, as far as I'm concerned, those guys over there are a bunch of idiots. Baxter: What makes you feel that way, Marshall? Whipker: I feel that way because Valve is a complete dunderhead. He doesn't know what the hell he's doing. Baxter: Exactly what is it about Valva that causes you to feel that he doesn't know what he's doing? Whipker: He isn't able to give me any help on my contracts. Baxter: Have you outlined specifically what it is you need from valva? Whipker: I've tried that with him, but he just doesn't know what I'm talking about. Baxter: Well, my suggestion to you, Marshall, is that you put down on paper exactly what it is you need from Valva and then you and I can go over it. If it looks reason- able, I want you to approach Valva and see if he will Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 113 be able to meet your deadlines. Does that sound reason- able? I think the whole thing is hopeless, but if you want me to, I guess I'll try it one more time. Before we move away from the Technical Services area, I'd like to ask you what happened when you and Valva had that rather heated exchange recently. Nothing really happened, except I called him a jackass and stormed out of the area. Well, I think you can see, Marshall, that that kind of situation can create a lot of friction in the organiz- ation and it might get in the way of our getting our job done. In the future, if you have any criticisms to make of Valva, I'd like you to do it in the privacy of his office. Do you think you can handle the situa— tion in that manner? Screw it. Valva isn't worth my time. Well, when you put your objectives down in writing and tell Valva exactly what you need, then you can start worrying about whether it's worth your time or not. In the meantime, I don't want any more situations like we've had in the past and I would like for you to bring any potential problems with Valva to my attention before they get out of hand. Is that clear? Yeah. The next area I'd like to cover is your relationships with your subordinates. How are you getting along with them right now? Okay. What does okay mean? They do their jobs, they stay out of trouble, and they get any flak from me. Is that what supervising is all about? What else is there? I make a profit, don't I? Let's take some specifics. How are you and Ted Ecklund getting along right now? Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 114 Not bad. Do you think Ecklund feels free to bring his problems to you? Why wouldn't he? He's a pretty young guy and he may need direction, and I don't know if you're giving him the kind of direction and the kind of help that he may need. As far as I know he hasn't had any complaints. I'm not too sure about that, but I'd like for you to give this some thought. Specifically, I would like for you to sit down with Ecklund and have a man-to-man talk with him. I want you to talk to Ecklund about whether he's receiving the necessary supervision and help from you and I'd like for you to report back to me on the results of that conversation in one week. Before we move on to any more of your ridiculous agenda items, I'd like to know why you think you're such a hot-shot? I've been here for all these years and instead of getting the job they give it to some "outsider" like you. Why do you feel you should have got the promotion instead of me? Because I'm a better engineer than you. Regardless of how you feel, Marshall, I got the job and you didn't. Now I understand how you feel about it and I understand why you're upset about it, but it wasn't my decision and there's really nothing we can do about it now except look to the future and try to determine what would be best for the two of us and our relationship. Oh, what a crock! Let's talk about your goals, Marshall. What would you like to be doing that you're not doing right now? I'd like to have a big pay increase and a nice job like yours. Does that mean that you want to move into higher mana— Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: 115 gerial ranks? What does it sound like? Well, how high do you want to go? As high as I can get. Can you be more specific? I want to be vice president of engineering. What qualifications do you feel you have that would enable you to perform well in that job? I'm a damn good engineer. Is that all it takes? What else is there? What about managing people and working with people so that you are more of a manager than just a techni- cal contributor? What about it? I'd like to know how you really feel about managing others, Marshall. It is possible you would like to remain an individual contributor rather than moving into management and administrative role? You can't make any money that way. Well, maybe you can't make as much money as you would if you were vice president, but I think you're a very valuable man-I think you're the kind of person that GCI should keep. I'd like to have you give some thought to what you want to do with your future and I'd like to ask your cooperation. You have done an outstanding job on all of your projects, and you meet your deadlines, and you always make a sizable profit. You're too valuable of a person for GCI to lose and I'd be glad to help you in any way I can. If you have any questions about developing your man- agement skills further, my door is always open. How are you feeling about things right now? I don't know. Baxter: Whipker: Baxter: Whipker: 116 Well, maybe we both need some time to think about these things. I'd like to close the interview by suggesting that we meet one week from today and find out how you made out with Ecklund. I'd also like to talk to you about what your plans are for using the Technical Services people. After we go over the plan, we can think things through about the valva situation and make sure there's a smooth relationship. I'd like for both of us to approach Valva within the next two weeks and just make sure that he understands what our needs are. Does that sound reasonable to you? Well, I guess that sounds pretty good. Okay, Marshall. Well, thanks for stopping by today, and I'll be talking to you next week about some of these things. Right-bye. 117 MANAGER 5 Baxter: Come in. Hello Marshall. I'm Dick Baxter. Whipker: Hi. Baxter: Have a seat and make yourself comfortable. Whipker: Thank you. Baxter: Well, now tell me, how is it going for you, Marsh? Whipker: Well, so—so. Baxter: Is everything alright with Anne and the kids? Whipker: Yes, things are basically okay. Baxter: Let's see-I see here you took preliminary math, science, and engineering courses in school and that you were able to get grades in the A range. Gee, I have always found those technical and math courses difficult myself and I have to admire somebody who's diligent enough to stick it out and do well in courses like that. In fact, I started out in college with a major in chemical engineering, but towards the middle of the sophomore year, I had to switch over to business administration, partly because I was tired of spending all those hours with a slide rule, and wanted to work more with people. And also, Marsh, between you and me, I was having a hard time with the math and all the technical stuff that's involved in engineering. So I guess I'm really glad to have somebody as technically competent as you working for me. And I'll be relying on you a lot to hold up the technical end of the shop. Ah, looks like you enjoyed your math and science courses a lot more than the humanities. Whipker: Well, the engineering courses made a lot of sense to me, Baxter: but those damn English and history courses . . . Jeez, you know, I could just care less about that . . . about Faulkner's death imagery or those battles in the War of 1812. You know-that stuff is a lot of crap as far as I'm concerned. Yea, I can see where a lot of the so-called liberal arts courses they're pushing these days probably is more geared toward giving the profs something to do in the lecture Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: 118 halls than to getting people a set of skills they can put to use in the real world. I think you're right. Ah, I see it says here that you've been with GCI for 9 years now, is that right? Yeah. Do you like working here? Yeah, it's all right-I guess. I've heard rumors, Marsh, that you're thinking about a job with CDC. Who did you hear that from? Oh, just through the grapevine, you know . . . That's what's wrong with this place-—it seems like whenever you tell somebody something in confidence, the word spreads like wild—fire. But, for really important information, you're lucky to find out what's happened two years or two months or whatever after it happens! What do you mean? You know very well what I mean. I mean, for instance, the memo about reduction in Tech Services hours. Why wasn't I informed about it? How come I had to get the word from that fool Valva instead of from you? Didn't you get a copy of the memo from Peen on this? No, I didn't get the memo from Peen on this. Well, I guess that's my fault. Looks like you're pretty upset about this, and I can't say I blame you. I'll try to tighten up communications around here. Okay. And another thing-when is somebody going to do something about that son-of-a-bitch Valva? Oh, that's right, you had a run-in with valva. What was it about? If you read your in-basket memos, you'll know that L Baxter: Whipker: Baxter: Whipker: 119 Company sprung some last minute design changes on the F;104O Ferits we're making for them. So this puts us a little behind schedule on that contract, so I thought I'd get some overtime out of Valva's Tech Services people so, you know, to get us back on target. So, anyway, when I asked Valva about it, he just shook his head and mumbled something about his being restricted to printed circuit boards and not being able to give me the develop- ment help I need. According to Valva, you were pretty abusive to him. He says that you hollered and stomped around and insulted him and his subordinates. So what if I did. That dunderhead deserved it. Damnit, I am sick and tired of putting up with the incompetence and hand-holding that goes on around here. Okay, take Valva. Nobody does any work in his shop. All they do is drink coffee, and read magazines, and play with that secretary, Miss Buxomberry, or whatever her name is. Whoever promoted Valva into that job must have been blind. And the promotion policy around here! Um. . . I don't know how it works, but it sure isn't giving the guys the promotion who are doing the work to get ahead. Damnit, you know, I've been busting my buns for nine years, and I've got the best profit margins in the company-so what happens when my boss Thompson dies? They fly some Flash Gordon up from California in here who's not even an engineer. Um-—looks like you've been fuming about this for some time now. Gee, I'm not really sure why you haven't been promoted, but I suspect it might have something to do with your temper. You know, according to Valva, you really blew up when you saw him, and from your reactions right here, I'd say you probably could learn to control your temper more and maybe learn better how to get along with people. You know as well as I do that we just can't run an organization if people are running around blowing their stacks all the time. Maybe you and I could sit down sometime and try to work out ways to help you get along with others better. This thing with Valva is really too bad. I'm sure you can find a way to-oh-approach him and sit down and work something out that would help the two of you get along and work smoothly together without coming to blows each time you see each other. Okay, maybe I can work on that. Is there anything else? 120 Baxter: Ah, no, I guess not. Let me just add, Marsh, that all in all you've been doing fairly well here at GCI and it would be a shame to lose you to CDC. Let's sit down and talk about how we can work together, you and I, to get you to the point where you'l be even more effective as a supervisor. Okay? Whipker: Sure. Baxter: Well, thanks for coming in, Marsh. I'll be talking to you later. Whipker: Okay. 121 MANAGER 6 Baxter: Whipker: Baxter:‘ Whipker: Baxter: Whipker: Baxter: Whipker: Good afternoon. I'm Lee Baxter. Marshall Whipker, I believe. Did I have that right? Please have a chair. I'm glad we finally managed to get together, Marshall, to talk about any difficulties you might be experiencing. Specifically, Marshall, I'd like to discuss any personal problems that you may be having, but feel free to talk about any other problems you might be having as well. Well you could start by getting rid of that jackass Valva. Well, I don't that firing Valve is the best solution to the problem and frankly, I was hoping for some good insights by you into the problem-—some more helpful suggestions. What's that supposed to mean? I mean, I'm not the source of the problem; he is. I'm not the one that's stalling around on the new PC boards or behind schedule on the control package wiring; he is. If you could just get some work out of him, there wouldn't be any problems. I'm trying to improve the work flow, Marshall, but it's important to understand that barging into a guy's depart- ment and telling him off publicly is not the way to solve the problem. It's just not the way to do it. Now, in reviewing your records at GCI, it's clear that you have great technical knowledge and expertise and you've consistently met or exceeded production goals. I just thought you might have some good ideas on how to work this thing out. Well, for one thing, he never has materials on hand and he's constantly held up by shortages. I might get something out of his shop if he could get his orders out earlier. Umm—hmm . . . that's the kind of solid, substantial suggestion we need, Marshall. Any others? Well, I guess he could sure improve his machinery layout so his men wouldn't have to run all over the damn shop to put together one part. But before I solve all your problems, I just want to say that a lot of good my technical expertise and past record have done me. I mean at any time an opportunity for promotion comes up, they bring in some tin-horn dude like dude like you with no Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 122 experience. And then you have the nerve to come in and ask me to solve all your problems. Well . . . I . . .I'm really not sure how to reply to that, Marshall, except that perhaps to say that a managerial position such as the one I'm in requires more than just technical knowledge and expertise. Perhaps even more important is the ability to relate to and deal with people. Maybe I am a little raw in technical areas, but a lot of my training was in dealing with people. Don't you see dealing with people and people problems as being important to a manager? Of course they're important, but I judge a man's abilities by his results and I get results from my people. I know that, Marshall, and that's why you're so valu- able to GCI, but sometimes you have personnel problems like everyone does, for example, with Valva. You can't fire every problem. What other ideas can you suggest other than firing? Well, I would call him in and lay it to him on the line. Either he shapes up on his output or out he goes. Well surely you can't face a man with being fired every time you want him to improve in some area. What other kinds of tactics could you bring in to smooth over a problem? Umm . . . I could try a little . . . What I've been trying to show you, Marshall, is that while your technical knowledge and experience have proven to be very valuable and very complete, perhaps additional development in the area of managerial skills in dealing with people would round out your abilities as a manager. You'll probably work something out with valva, but more important, let's set up a comprehensive development plan for yourself, Marshall, a development plan for you that includes a seminar in dealing with people that you might have read about in the company newsletter. All right? I've been thinking of attending that seminar myself. Tell you what-—why don't we attend it together and after each of the sessions, let's get together and talk about the principles and how they might apply to our situation here. I'm very enthusian— tic about trying to solve some of the personnel problems we have in the department, applying those principles to our own situation. And, I think that that could be Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 123 very, very helpful. I've heard that the seminar leader is a good, no-nonsense kind of leader and maybe we might think it wasn't worth worth out time afterwards, but if he is really a good, no-nonsense kind of guy, we might well get something out of it. Back to your development, if you can get something out of this semi- nar personally, I'll be the first to push for your promotion once those promotional opportunities come along. How does that sound? Well, I guess it sounds pretty interesting, but I don't know that . . . Well, the important aspect of this initial development plan, Marsh, is that we get together after the seminar to integrate the material and talk about it afterwards and I'll be happy to help you integrate to the extent that I can since I've attended a number of these kinds of seminar. So why don't we plan to get together afterwards and I'll give you whatever help I can. I'm sure I'm going to get some insights from you, too. So with your okay, I'll sign us both up-the company picks up the tab, okay? I also know of a couple night courses in the Industrial Relations Center that I think will prove to be useful for you in your development, but we can talk about that later after this initial seminar. Right now, I'm mostly interested in how this initial step-especially the company seminar-—sounds to you. Well, I suppose I've nothing to lose in trying, and I sure don't want to be stuck forever where I am. well, I agree. Just to make sure that doesn't happen, let's agree to meet on a bi-monthly basis to assess our progress and to talk about any other problems that might arise. Okay by me. Marshall, your experience and knowledge are highly valued by GCI and by me personally and with a little further development, I have every confidence that you will make the step up to the next organizational level. That seminar starts in three weeks. I'll have a brochure sent out to you right away, okay? Well, I guess that's all for now? Yeah, but don't forget our meeting two weeks from today. Same time, okay? 124 Whipker: Yeah, I think so, otherwise I'll get back. Baxter: Okay, Marshall, great talking with you. 125 MANAGER 7 Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Come on in. Why don't you sit here in this chair? Well, Marshall, since I've only been here a short time, I thought it would be a good idea to call people in and get to know them a little better. I'm calling everyone in to find out haw their work is going and if there are any problems caused in the transition period while I'm learning the new job. How are things going for you? Just like they always do. Well I've had some communications from other people in the company that indicate to me that there are some problems. Well, it's nice you're getting communications, because I'm certainly not getting any. How do you expect me to run my department if i"m not even notified about cutbacks in overtime? It's very unfortunate that my office slipped up on notifying you of that, Marshall, and I hope we can work together to improve our communication system, but it still seems to me that you've overreacted to the problem. He says . . . somewhere, let's see . . . that you "stomped around in his department and literally hollered and swore at him in front of his people." valve is a complete idiot, and that's the only way I can get any action out of him. You can see how he handles things-he complains to the "big boss" rather than working things out himself. I think I'll start writing some memos on him-—maybe that's the only way I'll get anything done around here. Well, I don't think that that would be a very good idea. Why don't you see if you can get along with him a little better? How do you get along with an idiot? Would you be able to run your department if Valva decided not to work with you? No, I guess not. Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: 126 Well, then, it seems to me you can hardly expect Valva to run his business if you decide not to work with him. Marshall, we're running a business here and your personal feelings about someone have no place in this business. I've read your personnel file in prepar- ation for this interview, just like I'm reading every- one's before they came in, and it sort of implies to me that you frequently let personal feelings about someone get in the way of your work. Do you think that's true? Well, that's a nice thing Thompson said about me-— where does it say that? Let me see. I'm sorry, Marshall, but personnel files are confi- dential. It wouldn't be fair to Thompson to let you read something he thinks he wrote confidentially. However, I am sure you are aware of the contents of your last performance review, since he stated that he talked it over with you. Well, he told me he thought I did a good job working on the United Fund. Well, that's true-he also said you did an adequate job elsewhere, but you have a problem working with people. well, I don't know what you would call problems with people . . . My department is consistently on schedule and we have the most profitable division in GCI-if that's having problems with people where you come well, I guess I do then. You are doing a pretty good job, Marshall, but you can probably do a little better. There's always room for improvement. Well, maybe you could help me by giving me a raise. At least that would help make up for the fact that I'm not going anyplace around here. Well, I'm not going to promise you a raise at this time, since we review everyone's salaries at the same time. I wanted to be promoted earlier, but uh . . . you got the job. You know, it's really a laugh that Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 127 you're sitting there asking me if I want a promotion. In the last several years, it's been, ”Marshall, you're the best engineer we've got," and "Marshall, it's really wonderful you're working so hard for the com- pany," but somehow old Marshall gets forgotton when it comes time to hand out the promotions. Well, maybe there was a reason I got the job and you didn't. Maybe it had something to do with your dealing with people? Well, I don't knows-but it certainly doesn't seem worth staying around here to have it happen again, though. Maybe I can help you do something about that. I'd like you to take the Dale Carnegie course we're offering next month, and see if that isn't inter— esting to you. That's what I really need at this point-—how to win friends and influence people. I'm an engineer, not a backslapper. Nevertheless, though, it does appear as if you're being held back in this area. I agree with you that you're an engineer, but an engineer has to work with people too. I wanted to run my department smoothly, and as far as I can tell, you seem to be good in many areas and could use some development though in handling people. Well, what's in it for me? I mean I've taken these courses before, and all I get are promises. Maybe I have to threaten to leave to get any action. You don't expect me to stay around here with no thanks, do you? well, there's more to working than just money, Marshall, and I seem to remember from your personnel file that you moved from jobs quite often. Well, it's just that it's the same old story each time- promises, but no payoffs. As far as I can see, there's no place for me at GCI right now. Oh, I disagree, Whipker. GCI considers you to be a good engineer, and I'm sure we'd like to keep you in the company. Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 128 wa11, if you need me so much, why don't you give me a raise? No, as I said, you'll be reviewed along with everyone else. However, some effort on your part now will pay off later. I don't feel we've covered all the ground that I wanted to today. I'd like to see you again next week-how about Thursday about 2 o'clock. Yeah. At that time, we can set up the Dale Carnegie course, and we can chat about some of the things we didn't cover in this interview. How does that sound? Okay, I guess. Okay, good-I'll see you next Thursday at two. 129 MANAGER 8 Baxter: Come in. Marshall Whipker, right? Whipker: Right. Baxter: I'm Baxter. Have a seat. Whipker: Thank you Baxter: Well, is everything going okay? Whipker: Yeah, I guess so. Baxter: Good. Ah, well, Marshall, do you have any idea why I've asked you to come in here today? Whipker: No, I don't . . . why don't you tell me] Baxter: A lot of people have been coming to me with complaints about your behavior on the job. Whipker: My "behavior" on the job is just fine . . . Ah, maybe you're referring to that little run-in with Valva? That idiot. Baxter: He's not an idiot and what do you have to say about that confrontation that you had? Whipker: What about it? Baxter: That's what I want your to tell‘mg, Whipker: All that happened is I had a little argument with Valva about his work on my projects. Is that why you called me in here? Baxter: Well, Valva didn't fill me in on all the details, but he did say that you started ranting and raving at him in front of his men when he told you about the cut in Tech Services hours. He said that apparently you hadn't gotton the word from Peen about it and you really lost control. That knid of outburst is something we can do just fine without here at GCI. Whipker: No, I didn't get the memo. Why do you think I was so pissed off when Valva had to tell me about it. Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: 130 Well, I don't know, but valva was pretty pissed off, too, about the way you blew up at him, and he wanted me to talk to you about it. You know, that memo came right from Peen's office. Now, if you didn't get it, that's too bad, but that's sure as hell no excuse for the way you treated Valva. It's not his fault that you didn't get the memo. Okay, okay, so maybe it wasn't his fault, but can‘ygg. tell me, Baxter, how in the hell am I supposed to meet my deadlines around here now that they've cut the hours on me? Especially when I've got to depend on Valva and those other jackasses down in Tach Services? Well, I think that's something you'd better talk to Valva about. Peen had his reasons for sending out the memo, and there's nothing we can do about it now. I think you'd better talk it over with Valva and try to come to some kind of agreement about priorities. If 122 have deadlines to meet, I'm sure he'll under- stand and try to find a way to fit it into his schedule. Fat chance! Listen, even before the cut in over- time hours, I couldn't get any work out of him and the rest of those SOBs down there are just as bad. When I go down there and I ask them for a little help, half of the time they're standing around twiddling their thumbs and telling nursery rhymes to one another. And then they say they're too busy when I ask them for a favor. The way things are now, well I'll be lucky to make‘ggy of my deadlines. Okay, listen can't you talk to Peen and ask him to change his mind about those overtime hours? I already told you, Whipker, I can't do anything about that now. Now, instead of feeling sorry for yourself, you're just going to have to find some way of getting what you want from Valva. Okay, and how in the hell am I supposed to do that? well, from what I've heard, this isn't the first time you've blown your stack around here, now. You're just going to have to cool it a bit when you're dealing with valve and the other people. You may think you've got problems-—okay, maybe you‘ggr-but things aren't always easy for them either. 131 Whipker: That's for damn sure I've got my problems. Getting any cooperation from anyone around here is worst than pulling teeth. Baxter: Well, there must be a good reason why you're having so much trouble. You know, Whipker, I just can't emphasize enough to you the importance of treating everyone with respect. You can't just expect cooperation, you've got to earn it. Now we're all in this together. It's really not important what your personal feelings are for Valva, or anyone else for that matter. The important thing is to at least treat them as human beings and try to work together as a team. Now, don't you agree? Whipker: I suppose so . . . Baxter: Fine, I know you two will work things out somehow. Other than that, how are things going? You haven't had any other problems? Whipker: Things are okay, I guess, Baxter . . . well, there is one thing that's been eating at me. Baxter: Yea, what's that? Whipker: Okay, I've been here at GCI for a long time and doing one hell of a good job as an engineer and alwys bringing in profits for the company on all my projects. Okay, so what do they do? Instead of putting someone on the job who really knows what he's doing, they bring in someone who doesn't know a fetit from a flatiron like you! Baxter: What? You think you should have gotton this job instead of me? Whipker: You're damn right I'm saying it. Baxter: What in the hell makes you think your're better qualified for it than I am? Whipker: Well, for one thing, I'm a damn good engineer. And, people know who's boss with me. I tell them what I expect from them-I expect results-and I get it. Just look in your files-it's all in there. Baxter: This is all well and good, Whipker, but the decision Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: Baxter: Whipker: 132 was made and the fact is you didn't get the job. Now, I guess there's not mush that can be done about it now, is there? Well, anyway, it sounds like you're doing okay on your job now. All of the reports I've gotton on your work seem satisfactory. Besides, what makes you think you'd want to get into management if you had the chance? Why not? I'm not getting anyplace where I am right now. Besides there's good money in management. And from what I've seen you don't have to work very hard to get it, either. Oh, is that what you think? Well, I don't know what you thing being a good manager is all about, but let me tell you, it's not all fun and games. If you expect to make it as a manager, you're going to have to find some better ways of dealing with people. Such as? We've covered all that already. Like you know, you've got to be more patient and considerate with Valve and the other people around here. I'll leave it up to you how you want to handle it. If you really want to change badly enough you can do it if you put your mind to it. Can you be a little more specific? I mean all that human relations stuff is a little abstract, don'y you think? I mean can't you come up with any real, concrete ideas about how I can get a better job? Well, I'll geiv it some thought. Right. Well, I'm kind of busy right now, Whipker, but it's been nice talking with you. I always like to keep on top of what my people are doing. Now keep in mind some of the things we've talked about this afternoon. Hang in there and let me know what you decide to do. Sure thing. Good-bye, Marshall. Good-bye. APPENDIX D BARS CONGRUENT TRAINING 133 134 BARS Congruent Training What follows is a step by step procedure for the trainer to follow when conducting BARS training. The double spaced text is a detailed script of what the trainer should say during the program. Other directions for the trainer are single spaced. There are slight variations in training dependent on which format (BARS or BOS) trainees are using to appraise the managers, and these appear in special subsections labelled "BOS format group" and "BARS format group. All trainees. Today, you will be participating in a training program that will help you learn how to appraise the job performance of others accurately. Once we have finished the training program, I will be showing you videotapes of six managers interviewing a problem subordinate. After viewing each of these videotapes, you will rate each manager on how well s/he conducted the interview. In order to rate the behavior of others correctly, there are a few things that you should know about how performance appraisal systems are set-up. First of all, most jobs can be thought of as consisting of various categories or dimensions of performance. In fact, you can think of any job as a pie that can be cut or divided into various pieces. Whenever we evaluate an employee's job performance, it is very important that we rate this person in terms of these important categories of performance. The reason for this is because these are the crucial elements of the job. As I mentioned before, today we will be rating six managers 135 conducting an interview with a problem subordinate. The manager's name is BAXTER (in each tape, Baxter is played by a different person). The subordinate's name is WHIPKER (the same individual plays Whipker in all tapes). Baxter is a new manager at GCI (company name) and is meeting with Whipker for the first time. As you'll see, Whipker has several problems, one of which is a peer named VALVA. Our goal will be to rate how well each Baxter performs the interview with Whipker. In order to do this, it is first necessary to identify what aspects of the interviewing job are most important. There are five dimensions of performance that we will want to focus on. What I am passing out to you now are the actual rating scales we will use to evaluate the managers. I am also distributing a list of the five dimensions and their definitions. For those rating with BARS, pass out the BARS scales. For those rating with BOS, pass out the BOS scales. The dimension definition hand-out is identical for the format groups. BARS format group. We will be rating the managers by making evaluative judgments of how effectively s/he performs on each dimension. As you can see, there is a seven-point rating scale for each dimension. Further, the scale points are anchored by examples of behaviors that correspond to different effectiveness levels within each dimension. 136 BOS formatpgroup. We will be rating the managers on how frequently s/he performs each one of the effective and ineffective critical incidents that appear within each dimension. Even though we will ultimately make frequency ratings, an important prerequisite to making accurate ratings is understanding the meaning of the dimensional system itself. The best way to gain this type of understanding is to discuss the degree to which various behaviors within each dimension are effective, average, or ineffective. All trainees. The first category we will use to rate the manager's performance is how well s/he STRUCTURES AND CONTROLS THE INTERVIEW with the subordinate. A manager who does a good job with respect to this dimension will do such things as clearly state the purpose of the interview, maintain control over the interview, and be organized and prepared for the interview. A manager who does not perform well with respect to this category will pp£_discuss the purpose of the interview, will display a confused approach, and will allow the subordinate to control the interview at inappropriate times. Similarly go over all performance dimensions by giving a global definition (from hand—out) of what constitutes generally effective and ineffective performance on it. All trainees. Now that we have our five performance dimensions 137 and global definitions of each, the next thing I would like to do is give you more specific examples of what types of behavior are considered high level performance, average performance, and low level performance for each rating category. For the BARS format trainees, go through specific examples of behaviors that correspond to each scale value. Present the behaviors by comparing and contrasting them. For instance, describe a behavior and what effectiveness level it represents on the dimension (e. g., "6") as compared to another behavior and what effectiveness it represents (e. g., "5") versus another behavior and its effectiveness level (e. g., "3"). Provide trainees with several examples of specific behavioral items for each scale value within each dimension. Those rating with BOS are to be presented with the same specific behaviors that are presented to those rating with BARS. However, rather than discuss each behavior in terms of whether it is a "4" or a"2" or a "1, " present the behaviors as being either highly effective, average, or highly ineffective. As I mentioned, what I would like to do now is give you some practice using the rating scales. I am going to show you a five minute videotape and when the tape is finished, you will rate the manager on the five performance dimensions we have been discussing. Do not take notes while the videotape is playing, because you might miss things the manager does. As you are watching the tape, remember to look for examples of behaviors that the manager exhibits that correspond to the five performance categories. Try to match effectiveness levels to behaviors as you view the tapes. This should help you to remember how well the manager performed on each dimension and thus, rate his/her performance more accurately. 138 Show videotape 1. When the tape is finished, ask trainees to put their first names on each scale and give them approximately three minutes to make their ratings. Put trainee names on a flipchart while they are rating the manager. When they are finished, ask them to hand-in their ratings for the first dimension. Record each rater's rating on the flipchart next to his/her name. Generate a group discussion that focuses on any discrepancies among the ratings. Use the scale anchor descriptions to evaluate the effectiveness of each behavior discussed. Give feedback to trainees. In BARS format group, tell the group the correct scale value for each dimension. For the BOS format group, tell trainees how many times each critical behavior occurred within the dimension. Repeat this process for the other four dimensions/rating scales. Tell trainees that they will now rate another videotaped manager. Show tape 2. Follow the exact same procedures and instructions that were followed for tape 1. After the second rating exercise, summarize and end the training program as follows: All of you should now understand how we are to evaluate the managers' performances. You will be rating six more videotapes of different Baxters interviewing Whipker. We will not be discussing your ratings of the tapes, but I will collect them and evaluate how well you did. At the end of this session, I will give you a written summary of this study and my phone number. If you would like the results of this study, you may call me next term and I will provide them to you. Once again, as you are viewing the videotapes, keep in mind the five categories you will be rating the managers on. As we did in training, look for specific behaviors that will help you identify which level of performance effectiveness the manager is exhibiting. Finally, please do not take notes while you are viewing the tapes. APPENDIX E BOS CON GRUENT TRAINING 139 140 BOS Congruent Training What follows is a step by step procedure for the trainer to follow when conducting BOS training. The double spaced text is a detailed script of what the trainer should say during the program. Other directions for the trainer are sin 1e spaced. There are slight variations in training dependent on whi format (BARS or BOS) trainees are using to appraise the managers, and these appear in special subsections labelled "BOS format group" and "BARS format group. All trainees. Today, you will be participating in a training program that will help you learn how to appraise the job performance of others accurately. Once we have finished the training program, I will be showing you videotapes of six managers interviewing a problem subordinate. After viewing each of these videotapes, you will rate each manager on how well s/he conducted the interview. In order to rate the behavior of others correctly, there are a few things that you should know about appraising performance. First of all, when people are asked what they think of another person, they will usually describe them in terms of traits, such as creative, out-going, shy, etc. Even though we naturally store information in this way (traits), it is not an appropriate way to think about and rate people in a job situation. This is because the traits we use are often ppp_job relevant and thus, they should not be used in performance appraisals. In work situations, it is crucial that we rate individuals in terms of important job behaviors. These are 141 observable acts that people engage in which contribute to ineffective or effective performance on the job. As I mentioned before, today we will be rating six managers conducting an interview with a problem subordinate. And, the rating task will specifically involve looking for specific, critical behaviors that are exhibited by the managers. Another thing that we typically do is make evaluative judgments about what they observe. Again, although this is our natural tendency, it is not always the most appropriate thing to do. Our rating task today will focus on observation as opposed to evaluation. As we view the managers' performances, it is extremely important that you not think in terms of how effectively or ineffectively the manager is performing. Rather, our goal will be only to look for specific behaviors and try to count or keep track of how frequently each specific behavior occurs. Before we continue with the training program, I'll tell you a little about the "cast of characters" that appear in the videotapes. The manager's name is BAXTER (in each tape, Baxter is played by a different person). The subordinate's name is WHIPKER (the same individual plays Whipker in all tapes). Baxter is a new manager at GCI (company name) and is meeting with Whipker for the first time. As you'll see, Whipker has several problems, one of which is a peer named VALVA. What I would like to do next is pass out a list of the behaviors that we will be looking for on the tapes. As you can see, the 142 behaviors are divided into five dimensions or categories of performance. These five categories are the most important aspects of the interviewing job. I am also distributing rating scales to you. These are the scales that you will be using to make performance ratings of the managers. For those rating with BARS, pass out the BARS scales. For those rating with BOS, pass out the BOS scales. BARS format_group. we will be rating the managers by making evaluative judgments of how effectively the s/he performs on each dimension. As you can see, there is a seven-point rating scale for each dimension. Further, the scale points are anchored by behaviors that correspond to different effectiveness levels within each dimension. Some of you may be wondering why it is necessary to focus on keeping track of specific behaviors when you will eventually have to make evaluative judgments of performance. As I mentioned earlier, people naturally do make evaluative judgments. However, because these judgments are made almost immediately, they are often based on far too little information and may be incorrect. Therefore, it is better to try and postpone making any judgments until you have observed enough to make an accurate judgment. Focusing your attention pply on observation should help you not judge too quickly. This, in turn, should help you to rate performance more accurately. 143 BOS formatigroup. We will be rating the managers on how frequently s/he performs each one of the effective and ineffective critical incidents that appear within each dimension. Remember that it is important not to think of whether the manager is performing well or poorly. Only look for the specific behaviors that appear on the list. All trainees. In order to ensure that you recognize important behaviors when they occur, the first thing we are going to do is memorize the critical incidents. What I would like you to do is read over the behaviors and mentally rehearse the ones that correspond to each dimension. Rehearsing these behaviors in dimensional groupings should help you to learn them faster. Give trainees a few minutes to go over the behaviors. Now what I'd like you to do is turn over the list of behaviors so that it is face down in front of you. We are going to try and write down the behaviors that correspond to each dimension. First, write down the dimension titles and then try to remember as many behaviors as possible. Don't worry if you can't remember all of them or if you only remember a few. When you have written down as many behaviors as you can remember, correct your answers by referring back to the list of behaviors. 144 Repeat this exercise a second time. Remember that our rating task requires that we keep track of which behaviors occurred. To help you with this, I suggest that you view the tapes with the following questions in mind: Is the behavior I am observing one of the critical ones I memorized? If no, ignore it. If yes, has the manager exhibited the behavior before? If no, store a 1. If yes, store N + 1. What I would like to do now is give you some practice using the rating scales. I am going to show you a five minute videotape and when the tape is finished, you will rate the manager on the rating scales I passed out earlier. Do not take notes while the videotape is playing, because you might miss things the manager does. As you are watching the tape, remember to look for examples of behaviors that the manager exhibits that correspond to the five performance categories. Show videotape 1. When the tape is finished, ask trainees to put their first names on each scale and give them approximately three minutes to make their ratings. Put trainee names on a flipchart while they are rating the manager. When they are finished, ask them to hand-in their ratings for the first dimension. Record each rater's rating on the flipchart next to his/her name. Generate a group discussion that focuses on any discrepancies among the ratings. Give feedback to trainees. In BARS format group, tell the group the correct scale value for each dimension. For the BOS format group, tell trainees how many times each critical behavior occurred within the dimension. Repeat this process for the other four dimensions. 145 Tell trainees that they will now rate another videotaped manager. Show tape 2. Follow the exact same procedures and instructions that were followed for tape 1. After the second rating exercise, summarize and end the training program as follows: All of you should now understand how we are to observe and rate the managers' performances. You will be rating six more videotapes of different Baxters interviewing Whipker. We will not be discussing your ratings of the tapes, but I will collect them and evaluate how well you did. At the end of this session, I will give you a written summary of this study and my phone number. If you would like the results of this study, you may call me next term and I will provide them to you. Once again, as you are viewing the videotapes, focus on only observing and keeping track of the specific behaviors we have been discussing. And, try not to make any evaluations about the effectiveness of each manager's performance. Finally, please do not take notes while you are viewing the tapes. APPENDIX F RAW DATA AND CODING INFORMATION 146 92111211. $2.11 1 1 2 1 3-6 1 7 - 8 1 9 1 1o 1 11 - 15 1 1-15 2 1-15 3 1-15 4 1 - 15 5 1-15 6 1-15 7 1 - 15 8 147 Codingilnformation for BARS Raw Data Variable Format (1 a BARS, 2 - BOS) Training (1 - BARS, 2 - BOS, 3 = no training) Subject Code Number Age Sex (1 - Male, 2 - Female) Previous Experience with Performance Appraisal (1 a Yes, 2 - No) Ratings of Manager 1 (training tape) Same as Same Same Same Same Same as 88 card card card card card card card 1. but for Manager (training tape) but for Manager but for Manager but for Manager but for Manager but for Manager but for Manager 2 NO‘U'IkLO 110012022 110012022 110012022 110012022 110012022 110012022 110012022 110012022 110021921 110021921 110021921 110021921 110021921 110021921 110021921 110021921 110031821 110031821 110031821 110031821 110031821 110031821 110031821 110031821 110041811 110041811 110041811 110041811 110041811 110041811 110041811 110041811 110052822 110052822 110052822 110052822 110052822 110052822 110052822 110052822 110061921 110061921 110061921 110061921 110061921 110061921 110061921 110061921 110072021 110072021 110072021 33344 42143 35343 75666 45564 55467 55645 33222 54443 46433 33241 76661 23271 54437 45666 32221 54655 32443 35444 76652 24232 65467 32331 43221 54463 11123 37463 76675 53264 46566 46136 31211 25466 41413 25‘13‘1 . 76565 13233 44556 45456 32222 34453 66213 36464 77626 35454 46467 “5525 22112 14464 51313 45454 H18 110072021 110072021 110072021 110072021 110072021 110082021 110032021 110082021 110082021 110082021 110082021 110082021 110082021 110092111 110092111 110092111 110092111 110092111 110092111 110092111 110092111 110102012 110102012 110102012 110102012 110102012 110102012 110102012 110102012 110112121 110112121 110112121 110112121 110112121 110112121 110112121 110112121 110121911 110121911 110121911 110121911 110121911 110121911 110121911 110121911 110132121 110132121 110132121 110132121 110132121 110132121 76555 24232 55466 45446 42332 45564 31122 36353 75663 24242 34346 42333 51111 23666 31233 46354 76676 35445 67467 54445 32333 35666 32432 34433 76765 13142 55367 46234 21223 34543 42333 25343 75665 23332 65666 33435, 32221 44463 43423 26243 76553 24332 46554 43334 31121 34554 22313 45454 7675“ 38331 36666 149 110132121 110132121 110142022 110142022 110142022 110142022 110142022 110142022 110142022 110142022 110152712 110152712 110152712 110152712 110152712 110152712 110152712 110152712 110162021 110162021 110162021 110162021 110162021 110162021 110162021 110162021 110171922 110171922 110171922 110171922 110171922 110171922 110171922 110171922 110181922 110181922 110181922 110181922 110181922 110181922 110181922 110181922 110192111 110192111 110192111 110192111 110192111 110192111 110192111 110192111 110202111 43236 31121 34653 32153 34444 65643 34444 55456 43345 32221 33455 42433 37344 76567 13342 66667 64434 32211 34666 52333 34343 63645 32432 56467 44445 31211 34564 31212 24334 75445 13233 65567 42334 22111 34466 21333 25333 77776 23234 54566 44355 23231 12343 31343 33232 76664 13232 44367 33235 32121 35556 150 110202111 110202111 110202111 110202111 110202111 110202111 110202111 110211822 110211822 110211822 110211822 110211822 110211822 110211822 110211822 110222011 110222011 110222011 110222011 110222011 110222011 110222011 110222011 110231822 110231822 110231822 110231822 110231822 110231822 110231822 110231822 110242211 110242211 110242211 110242211 110242211 110242211 110242211 110242211 126011911 126011911 126011911 126011911 126011911 126011911 126011911 126011911 126022021 126022021 126022021 126022021 32233 35543 75666 34655 66677 34333 11111 46655 42222 34456 75553 23544 46666 52435 31322 22544 31343 34352 75555 23321 35246 33145 34332 46655 42222 34456 75553 23544 46666 52435 31322 33334 32433 43433 66554 23433 45456 33444 22232 35466 61243 16242 76671 15232 66667 45544 31331 35424 63233 46353 66756 151 126022021 126022021 126022021 126022021 126031821 126031821 126031821 126031821 126031821 126031821 126031821 126031821 126041822 126041822 126041822 126041822 126041822 126041822 126041822 126041822 126051822 126051822 126051822 126051822 126051822 126051822 126051822 126051822 126061822 126061822 126061822 126061822 126061822 126061822 126061822 126061822 126071811 126071811 126071811 126071811 126071811 126071811 126071811 126071811 126081911 126081911 126081911 126081911 126081911 126081911 126081911 43433 67777 55334 43221 56657 56657 44253 77676 25356 66676 55466 33233 36646 55545 66456 76666 15351 65666 55666 42341 36656 55443 46566 76666 23242 55656 55676 44231 76767 32232 56545 76677 14432 25266 66666 22121 45656 21223 46566 76766 25556 67657 33335 22321 31213 66566 24343 66744 23231 66677 44324 152 126081911 126091922 126091922 126091922 126091922 126091922 126091922 126091922 126091922 126102722 126102722 126102722 126102722 126102722 126102722 126102722 126102722 126112022 126112022 126112022 126112022 126112022 126112022 126112022 126112022 126121922 126121922 126121922 126121922 126121922 126121922 126121922 126121922 126131822 126131822 126131822 126131822 126131822 126131822 126131822 126131822 126142112 126142112 126142112 126142112 126142112 126142112 126142112 126142112 126151812 126151812 42131 65566 12212 34233 76676 24431 66557 54343 22111 35636 35636 65655 65666 25223 65656 35655 11111 46566 22235 56665 76556 55565 55656 65655 23122 56466 11234 56565 75667 24433 67677 55456 22324 55356 76446 56271 76646 15233 54336 66565 22232 45656 11143 17164 72576 15351 66477 55435 32211 35456 51243 153 126151812 126151812 126151812 126151812 126151812 126151812 126161921 126161921 126161921 126161921 126161921 126161921 126161921 126161921 126172322 126172322 126172322 126172322 126172322 126172322 126172322 126172322 126181911 126181911 126181911 126181911 126181911 126181911 126181911 126181911 126192112 126192112 126192112 126192112 126192112 126192112 .126192112 126192112 126201821 126201821 126201821 126201821 126201821 126201821 126201821 126201821 126211911 126211911 126211911 126211911 126211911 46476 75633 37642 66477 54342 42111 55555 42233 27576 76666 27566 67677 66776 21232 35626 61556 12112 76562 12442 67767 45143 42222 35666 61233 14335 76666 13461 65767 32266 21211 45355 22456 15354 76677 15234 45677 45354 22132 55667 31123 27476 75221 13131 65657 32555 32221 46646 22445 65353 76456 14554 154 126211911 126211911 126211911 126222021 126222021 126222021 126222021 126222021 126222021 .126222021 126222021 126232021 126232021 126232021 126232021 126232021 126232021 126232021 126232021 126241921 126241921 126241921 126241921 126241921 126241921 126241921 126241921 132012111 132012111 132012111 132012111 132012111 132012111 132012111 132012111 132022022 132022022 132022022 132022022 132022022 132022022 132022022 132022022 132031822 132031822 132031822 132031822 132031822 132031822 132031822 132031822 56456 24332 32232 35656 42232 23123 66765 24241 46557 44344 31121 55665 31232 67676 75555 13232 66677 55465 42221 65256 34324 37163 75767 23252 46757 43446 12142 55545 66456 76666 77667 22233 45567 35365 34342 45666 52454 36435 66564 26235 47667 65354 32134 35666 56555 32355 76777 14445 55565 54565 44433 155 132041822 132041822 132041822 132041822 132041822 132041822 132041822 132041822 132051922 132051922 132051922 132051922 132051922 132051922 132051922 132051922 132062211 132062211 132062211 132062211 132062211 132062211 132062211 132062211 132071911 132071911 132071911 132071911 132071911 132071911 132071911 132071911 132082212 132082212 132082212 132082212 132082212 132082212 132082212 132082212 132092011 132092011 132092011 132092011 132092011 132092011 132092011 132092011 132102011 132102011 132102011 35566 56645 27235 76667 13222 56667 65555 45232 32556 32214 46134 76767 33333 45366 65426 33211 66223 11211 77345 77467 16634 56377 36254 31411 25656 75665 76667 75677 23613 66767 56345 22222 25156 61233 35173 76675 67553 65477 52163 11211 32533 66636 44345 66655 12221 65663 64323 32424 36465 51547 27257 156 132102011 132102011 132102011 132102011 132102011 132111812 132111812 132111812 132111812 132111812 132111812 132111812 132111812 132121822 132121822 132121822 132121822 132121822 132121822 132121822 132121822 132131722- 132131722 132131722 132131722 132131722 132131722 132131722 132131722 132142022 132142022 132142022 132142022 132142022 132142022 132142022 132142022 132151821 132151821 132151821 132151821 132151821 132151821 132151821 132151821 132162111 132162111 132162111 132162111 132162111 132162111 76677 36531 66677 56257 42411 67666 42333 36664 66465 22122 67677 45444 12111 35776 31233 27354 76665 15131 46666 34345 32121 35533 61112 35535 66766 26131 67666 55555 62221 56553 76665 56555 77777 57555 66677 55555 31121 46676 52235 67677 75666 27365 67767 66655 45555 65556 31232 67145 55132 22123 67777 157 132162111 132162111 132172021 132172021 132172021 132172021 132172021 132172021 132172021 132172021 132182122 132182122 132182122 132182122 132182122 132182122 132182122 132182122 132192022 132192022 132192022 - 132192022 132192022 132192022 132192022 132192022 132201912 132201912 132201912 132201912 132201912 132201912 132201912 132201912 132212021 132212021 132212021 132212021 132212021 132212021 132212021 132212021 132221812 132221812 132221812 132221812 132221812 132221812 132221812 132221812 132232121 32113 21111 65377 61243 35213 75677 23231 66677 56545 22112 35566 61233 67464 75767 56675 36567 65555 32233 65676 71212 57555 35756 24456 47777 14433 12221 53233 67756 71122 33233 75445 24334 77777 22111 55666 51232 67145 55133 22123 67777 32113 21441 25366 32213 25142 76555 45555 35756 55545 32232 56565 158 132232121 132232121 132232121 132232121 132232121 132232121 132232121 132242012 132242012 132242012 132242012 132242012 132242012 132242012 132242012 21233 67466 62623 15233 67666 45543 42232 54667 33243 67666 65656 12133 45367 67675 62332 159 160 Codinngnformation for BOS Raw Data Column Card Variable l 1 Format (1 s BARS, 2 - BOS) 2 1 Training (1 a BARS, 2 . BOS, 3 a no training) 3 - 6 1 Subject Code Number 7 - 8 1 Age 9 1 Sex (1 - Male, 2 . Female) 10 1 Previous Experience with Performance Appraisal (1 - Yes, 2 a No) 11 - 12 1 Ratings of Manager 1, Dimension 1 (training tape) 14 — 19 1 Ratings of Manager 1, Dimension 2 21 - 25 1 Ratings of Manager 1, Dimension 3 27 - 31 1 Ratings of Manager 1, Dimension 4 33 - 38 1 Ratings of Manager 1, Dimension 5 l - 38 2 Same as card 1, but for Manager 2 (training tape) 1 - 38 3 Same as card 1, but for Manager 3 1 - 38 4 Same as card 1, but for Manager 4 1 - 38 5 Same as card 1, but for Manager 5 1 - 38 6 Same as card 1, but for Manager 6 l - 38 7 Same as card 1, but for Manager 7 1 - 38 8 Same as card 1, but for Manager 8 213011822 213011822 213011822 213011822 213011822 213011822 213011822 213011822 213031911 213031911 213031911 213031911 213031911 213031911 213031911 213031911 213042111 213042111 213042111 213042111 213042111 213042111 213042111 213042111 213051811 213051811 213051811 213051811 213051811 213051811 213051811 213051811 213061822 213061822 213061822 213061822 213061822 213061822 213061822 213061822 213071812 213071812 213071812 213071812 213071812 213071812 213071812 213071812 213081811 213081811 213081811 11 12 00 3O 12 20 10 10 1O 02 00 10 02 10 10 00 01 02 01 10 03 00 10 00 11 00 00 10 03 10 20 00 01 11 00 1O 02 10 1O 00 00 02 00 10 03 00 00 00 01 00 00 100001 113101 020001 010001 111000 020001 001001 010001 101001 103211 020002 022002 022101 021001 012001 011110 100101 203101 020001 021001 001002 030000 002000 200012 000001 203100 020000 000001 000000 000000 012001 200010 000011 102210 020001 001001 011000 010001 111100 110100 010001 102111 010001 010111 000001 010101 011110 110110 100000 103010 010001 20100 12000 10100 10020 00000 01000 00100 11100 10100 11000 00200 00010 11100 01000 01000 02200 21100 02000 00100 00120 01100 01000 03100 02100 20100 02000 00200 10120 01000 00000 01000 03100 20100 02000 00100 00110 01100 00100 01100 02100 12200 12000 00100 02010 01100 01000 02000 02000 21100 02000 00200 22010 00200 11010 11100 11000 21110 11111 11101 21010 01201 31000 11000 11000 22020 11000 00201 21020 11101 32000 10000 11000 21020 31001 00100 21202 00301 31000 01000 01000 21020 11001 00200 03030 11100 12000 01010 22000 03010 12101 02200 21030 10101 23000 12001 22000 12011 11101 10101 02030 10201 21000 161 122201 101110 010110 100110 000000 200201 100101 011100 110101 011020 010020 000210 001020 110211 200200 001000 230202 101010 010110 000110 000200 310312 200200 001000 230202 010010 010110 000110 001000 300203 200100 001000 210202 011010 011010 010200 011010 000201 200100 001000 120201 021010 010000 010110 000000 120101 100100 000000 210200 101000 010010 213081811 213081811 213081811 213081811 213081811 213092111 213092111 213092111 213092111 213092111 213092111 213092111 213092111 213102411 213102411 213102411 213102411 213102411 213102411 213102411 213102411 213111911 213111911 213111911 213111911 213111911 213111911 213111911 213111911 213121821 213121821 213121821 213121821 213121821 213121821 213121821 213121821 213132012 213132012 213132012 213132012 213132012 213132012 213132012 213132012 213141821 213141821 213141821 213141821 213141821 213141821 30 03 20 00 00 11 02 00 20 00 00 10 21 01 O1 00 10 03 00 1O 00 10 22 01 10 02 1O 10 10 10 32 10 10 22 20 00 00 00 01 00 10 02 00 00 00 00 3O 00 20 02 00 010011 010001 020001 112000 111110 111011 103200 000000 000001 102101 020211 110110 100312 010001 213000 030000 030001 001000 001001 010000 011010 020101 102000 020001 020000 010101 010011 010001 111020 100101 213101 110100 000101 110001 110102 011100 012100 000102 102111 020002 020001 010001 020001 011101 101120 000010 323000 020000 000000 000000 010000 11130 10200 11100 02100 12201 21111 02100 00100 00030 11021 21000 01102 21123 30200 01000 00200 11020 01100 01000 10100 01100 20200 12100 10100 11120 00100 00100 01000 12000 31100 01101 00100 12110 11100 01200 01100 13100 20100 01000 00200 10120 00100 11000 01100 01000 01000 03000 00000 00010 00000 00000 03000 01010 03020 10001 00100 32030 30300 30020 00001 12311 11010 21110 00213 32021 20100 21000 22100 11000 11001 12021 00101 32020 00200 32000 12000 12000 12020 12001 10200 22020 20201 33100 33101 22100 33021 22110 11300 20021 00101 33000 21010 22101 22011 12001 00201 00100 00300 20000 21000 21000 11010 162 001110 001010 110212 101100 001000 310213 002000 030030 001100 110011 303020 011112 011110 120101 011010 010020 000210 000000 100101 210201 001000 200111 011110 010110 010110 000100 210202 200101 002000 220201 012110 010000 000110 000110 230212 200100 002000 211201 002010 000020 010110 010110 110101 110101 001000 001000 022000 000000 000100 000100 200101 213141821 213141821 213151812 213151812 213151812 213151812 213151812 213151812 213151812 213151812 213161822 213161822 213161822 213161822 213161822 213161822 213161822 213161822 213171922 213171922 213171922 213171922 213171922 213171922 213171922 213171922 213181921 213181921 213181921 213181921 213181921 213181921 213181921 213181921 213191911 213191911 213191911 213191911 213191911 213191911 213191911 213191911 213202021 213202021 213202021 213202021 213202021 213202021 213202021 213202021 213211912 001000 000010 100102 223002 030100 000001 002101 030002 001001 000010 131220 213101 130101 021102 010100 121000 110012 100120 101103 233112 030103 011001 211001 130001 112000 111110 010102 103201 120002 020001 000001 020002 103210 102210 010012 103200 120000 000002 201000 020000 201101 220110 010100 102001 030101 030001 020101 030100 031101 120110 030123 00000 01000 31200 12100 01100 11020 02000 01120 02101 01000 12300 01102 00200 01220 10200 00100 00200 01100 22301 13100 00300 00030 02200 01200 00100 03200 31200 01000 00300 11120 00100 01100 02001 02001 20300 02000 10001 10220 01000 00000 00000 02000 20100 01000 00100 01010 10100 00100 01000 02000 32300 11000 00100 13032 21301 12000 01000 12000 01011 12101 01111 01011 33212 31000 22100 00000 20000 12000 00100 22130 00302 32000 32030 21001 21020 11010 00301 12021 01201 32010 21120 22000 21020 11201 00202 10020 10202 22000 11000 12100 11020 12100 00100 21011 00101 21000 10000 21000 22010 21000 00101 33021 163 100100 001000 222311 001110 011110 000220 000010 221202 201101 002100 120121 012020 011010 010100 001010 110101 201100 001000 320300 111020 023020 000320 001010 311203 200100 001000 221300 011010 000110 010110 001100 230312 101100 002000 300302 111010 011020 000200 001200 210202 200200 001000 210101 010010 010010 100210 001000 200201 100101 002000 330313 213211912 213211912 213211912 213211912 213211912 213211912 213211912 213221812 213221812 213221812 213221812 213221812 213221812 213221812 213221812 213231811 213231811 213231811 213231811 213231811 213231811 213231811 213231811 213241922 213241922 213241922 213241922 213241922 213241922 213241922 213241922 213251922 213251922 213251922 213251922 213251922 213251922 213251922 213251922 225011811 225011811 225011811 225011811 225011811 225011811 225011811 225011811 225022022 225022022 225022022 225022022 02 00 30 10 00 10 30 32 10 30 20 20 00 01 12 10 20 03 00 00 00 10 12 00 30 03 20 10 10 00 12 00 10 03 00 00 00 00 01 00 30 10 00 00 30 01 00 30 313121 231201 031002 202200 030202 023120 222230 020203 203200 030002 012000 000002 020002 023101 200320 100011 102201 120001 010001 110210 010001 020101 110110 000001 202200 030001 010001 011100 010000 001100 110110 020003 213212 030002 020002 000000 010001 012100 100210 010000 103001 020000 003001 000000 000000 002000 120000 000002 102000 030000 000000 03001 00200 01130 02200 20200 02201 02200 10100 03001 00100 01230 02000 02200 02000 03000 21100 12100 02200 10130 02000 00000 02101 01100 31200 01201 00200 10120 01100 01100 01100 02100 31300 03100 00200 20220 02020 01100 02000 02100 20100 03200 00200 00010 00100 00000 02100 02000 20100 01001 00100 00020 20201 32010 22001 02200 33030 22103 11300 13031 00302 30000 00100 33001 33020 13010 00202 21000 11100 32010 12000 22000 02000 12100 01100 11020 00101 12000 13000 12000 11010 11101 00101 31020 10202 33000 11010 22001 33010 10001 00101 21020 00200 20000 02000 00000 11020 11000 00200 31030 10100 10000 02000 164 023020 021330 000320 020000 311303 302200 013000 210232 032000 000100 001300 000101 320323 312300 013000 210102 010110 010210 000210 020200 200202 210200 001000 110111 020010 011010 010110 000020 110222 200100 001000 310212 021020 020020 000210 000000 200210 100100 000000 310211 010010 010010 010210 000000 200302 100201 000000 310201 010010 010000 000210 225022022 225022022 225022022 225022022 225031922 225031922 225031922 225031922 225031922 225031922 225031922 225031922 225041811 225041811 225041811 225041811 225041811 225041811 225041811 225041811 225051912 225051912 225051912 225051912 225051912 225051912 225051912 225051912 225061821 225061821 225061821 225061821 225061821 225061821 225061821 225061821 225071921 225071921 225071921 225071921 225071921 225071921 225071921 225071921 225081812 225081812 225081812 225081812 225081812 225081812 225081812 03 10 10 10 11 02 00 30 O1 10 10 00 10 01 10 10 03 00 00 00 00 01 00 30 03 10 00 00 00 02 10 10 01 00 00 00 10 01 00 30 03 00 00 00 00 11 00 30 02 10 00 000000 010000 010000 000000 000001 101001 031002 030001 010000 010100 010101 000010 011000 101101 010000 010000 001000 000000 002100 000110 000000 102000 030000 000000 000000 000000 001000 110010 010000 102000 030001 000000 000000 000000 010000 002001 010110 103000 021000 001000 000000 000010 012000 110010 010000 103000 030000 011000 000000 000000 000000 00100 00000 00000 01200 20200 01001 00100 10020 10000 00100 01000 03100 10200 01100 00200 10120 00100 02100 01000 01000 20100 01001 00100 00020 01000 00100 00000 02100 10000 02000 00100 00010 01000 00000 02000 01000 21100 01000 00200 01020 01100 01000 03000 01000 10100 12000 00100 10010 00100 00000 00000 11000 12010 01001 00200 13031 00101 30000 32010 11000 21020 01011 00201 21020 00101 30000 33000 01000 11010 11000 00201 30030 00100 21000 12000 10000 11000 02001 00100 20020 00200 10000 10000 11000 11020 00001 00000 10011 10100 20010 11000 10000 02010 01000 00200 20010 10100 30000 21000 20000 11010 00001 165 000100 100212 100100 000000 210202 010010 010220 010210 000110 210233 100200 001100 110201 010010 010010 000110 000100 100202 100100 000000 320311 010010 010010 010210 000000 200201 100200 000000 300200 010010 010000 000200 010000 310201 100200 000010 310211 100010 010010 000210 000010 100202 100100 000000 110201 010010 010110 000210 000200 200301 100100 225081812 225091911 225091911 225091911 225091911 225091911 225091911 225091911 225091911 225102011 225102011 225102011 225102011 225102011 225102011 225102011 225102011 225111821 225111821 225111821 225111821 225111821 225111821 225111821 225111821 225122021 225122021 225122021 225122021 225122021 225122021 225122021 225122021 225132221 225132221 225132221 225132221 225132221 225132221 225132221 225132221 225141911 225141911 225141911 225141911 225141911 225141911 225141911 225141911 225152211 225152211 00 00 12 00 30 20 00 00 00 02 00 11 03 01 00 00 00 02 00 30 03 10 00 00 00 32 00 30 10 00 00 01 01 00 30 01 00 00 10 01 00 30 10 00 10 20 00 010110 000101 103001 020000 000000 000000 000000 022000 000010 000001 103000 020000 000000 000000 020000 011000 110010 011002 112000 020000 032000 000000 030000 011000 110110 000001 101000 030000 000000 000000 000000 001000 100100 011000 002000 010000 011000 010000 000000 011000 100010 000001 103000 020000 000000 000000 000000 000000 000010 010001 103001 00000 20100 02000 10100 10020 00000 00000 00000 01100 20020 01000 00100 11020 00100 00010 00000 01000 21100 02100 00100 00120 01000 00100 00100 01000 20100 02000 00100 00120 01100 00100 00000 01000 10100 02100 10100 11010 01000 00100 01000 01100 30100 00000 00200 00120 00100 00100 01000 01100 20100 01000 00300 31020 10100 30000 00000 02000 22000 01002 00100 20020 00001 20000 11000 10000 21010 20001 00100 22031 20101 31000 31000 11100 11020 10001 00101 20020 20100 30000 00000 00000 12000 00000 00100 10010 20100 10010 22000 10100 12010 10010 10000 20020 00200 31000 12000 21000 11010 10001 00100 21020 00100 166 001000 310202 010010 010010 000310 000000 200202 100100 001000 201200 110010 010020 000200 000000 100202 100100 000000 310303 110010 010010 010210 000010 210202 200200 001000 210300 010010 020010 010110 000100 100200 100101 000000 110001 121100 110010 000210 000000 100102 100201 001000 210302 010010 010010 000210 000000 200202 200101 001000 210202 010000 225152211 225152211 225152211 225152211 225152211 225152211 225161911 225161911 225161911 225161911 225161911 225161911 225161911 225161911 225171922 225171922 225171922 225171922 ,225171922 225171922 225171922 225171922 225181912 225181912 225181912 225181912 225181912 225181912 225181912 225181912 225191912 225191912 225191912 225191912 225191912 225191912 225191912 225191912 225201822 225201822 225201822 225201822 225201822 225201822 225201822 225201822 225211921 225211921 225211921 225211921 225211921 00 30 03 10 00 00 00 11 00 30 02 00 00 10 10 01 00 10 03 10 10 00 00 01 00 20 02 00 00 00 01 01 00 30 10 00 00 10 01 00 20 03 10 10 00 10 01 00 20 01 020000 010000 000000 000000 001000 100010 000000 103000 020000 022001 000001 000000 001000 000111 010001 012000 030001 020000 000000 000000 011000 100010 000110 102000 030000 000000 010000 010000 011000 110100 020000 102000 020000 000000 020000 010100 001100 000010 001001 103000 020000 000000 000000 000000 012000 000010 010001 202000 010000 010000 000000 00200 00220 00100 00000 00000 00000 10100 01000 00200 00020 00000 01000 00000 00000 20101 01000 00100 10010 00000 00000 00000 02000 30100 01000 00000 00020 00000 00000 00000 01000 11100 11000 00100 00110 01100 00100 00000 01000 20100 01000 00200 11010 00100 00000 01000 01000 20200 02000 00100 01110 01100 10000 00000 10000 22010 00001 00000 20010 00100 21000 11000 10000 11010 01000 00100 20010 10000 11000 11000 11000 11000 00000 00101 22030 10000 30000 12000 00000 13000 00001 00000 20110 10100 31000 12000 11000 11010 21001 00100 30020 10100 20000 01000 00000 12010 01001 00100 11020 00100 10000 11000 00000 167 010010 000310 000000 200101 100100 000000 110102 110010 010010 000220 000000 200211 100100 001000 110202 110010 010010 010210 000000 110212 100200 001000 310103 010020 010110 000210 000000 000203 100100 000000 120202 010010 010010 000110 000010 200212 100100 001000 210201 010010 010010 000210 000000 200201 100100 002000 211202 010010 020010 001110 000000 225211921 225211921 225211921 225221921 225221921 225221921 225221921 225221921 225221921 225221921 225221921 225231822 225231822 225231822 225231822 225231822 225231822 225231822 225231822 225242012 225242012 225242012 225242012 225242012 225242012 225242012 225242012 234011821 234011821 234011821 234011821 234011821 234011821 234011821 234011821 234022022 234022022 234022022 234022022 234022022 234022022 234022022 234022022 234031822 234031822 234031822 234031822 234031822 234031822 234031822 234031822 10 00 00 01 00 00 30 03 10 00 00 1O 00 00 20 03 10 00 00 01 00 00 30 01 10 00 00 10 02 00 10 01 00 00 00 01 22 1O 30 03 10 00 1O 13 01 00 10 03 10 00 00 000000 001000 100010 111002 103100 010000 010000 000000 020000 001000 000200 010100 102000 020000 000001 010000 010000 011000 100100 010001 101000 020000 000000 010000 010100 001100 100100 010100 102100 010010 001001 010000 020100 112101 111020 131023 103001 120011 021011 000001 010001 011000 000000 000001 101000 020000 010000 010000 010001 012000 010100 00100 00100 01000 21100 01000 00100 11020 00000 00100 00000 03000 20200 01000 00200 10030 00100 00100 01000 03000 10000 10001 00200 11110 01100 00100 01000 01000 20200 02000 01201 10120 02000 02100 02000 02000 20100 03100 00200 00130 01200 00000 00000 00000 20300 01000 00100 10110 00100 01000 01000 02000 11010 01000 00100 22030 00100 30000 02000 10000 12010 01000 00200 20020 00101 30000 02000 20000 13010 00101 00300 20100 00100 21000 12000 11000 11010 21101 00100 20020 00201 22001 02000 22101 22111 22102 20201 32133 20303 22000 22101 11000 11020 21102 11000 03030 00001 32000 00010 11000 12010 11000 00101 168 100202 100100 001000 310201 010010 010010 000210 001000 000213 200100 001000 100201 020010 010010 000210 000000 200201 100101 003000 100201 010010 010010 000110 000010 200212 100100 001000 100201 010020 020020 210220 001020 220202 300200 001000 321322 221010 001100 001210 000100 302202 200100 101000 301223 010010 010010 000310 000000 100202 200201 001000 234041812 234041812 234041812 234041812 234041812 234041812 234041812 234041812 234051821 234051821 234051821 234051821 234051821 234051821 234051821 234051821 234061921 234061921 234061921 234061921 234061921 234061921 234061921 234061921 234071811 234071811 234071811 234071811 234071811 234071811 234071811 234071811 234081822 234081822 234081822 234081822 234081822 234081822 234081822 234081822 234092122 234092122 234092122 234092122 234092122 234092122 234092122 234092122 234101812 234101812 234101812 20 12 10 20 12 11 20 11 30 02 00 30 03 10 10 00 3O 02 10 10 01 10 10 00 10 11 10 20 03 00 00 10 13 00 00 10 02 20 00 00 00 02 00 10 12 00 10 00 11 02 10 121001 111101 010001 121001 011200 110001 111100 111120 000001 103000 030001 031001 000001 010001 023002 011221 123001 113100 020000 022000 001001 030000 102100 121110 010002 103000 020203 002000 010001 010000 000000 110010 010021 102001 020000 011000 010000 010001 011001 010010 021001 113001 010000 010000 010000 010001 011000 000010 010101 302002 010001 20100 01000 01200 11210 02100 11200 12201 21200 20100 02000 00200 00020 00000 01100 02000 03100 32300 03100 00300 21031 03100 02000 03002 03100 10100 02100 00300 02130 00200 00200 00200 03000 23210 11000 00200 10110 01100 01100 01100 02100 00200 02000 00100 00110 01000 00000 01000 02000 21100 22200 20200 22010 20100 22010 23010 32000 22021 22011 21211 20020 01100 33000 32000 11001 12001 12002 01303 32020 10310 23000 02000 01010 02010 11002 00300 20020' 10003 33000 23001 22000 33030 11003 00303 33030 10000 11000 02000 11000 12101 11001 10101 11020 21001 21000 01000 11000 00010 12000 00100 30020 30102 31000 169 220202 010120 010010 020220 010110 230213 211202 002100 210201 010010 010020 000120 000000 210302 200200 002000 330303 010010 020010 000320 000000 330313 300201 000000 310201 101020 010000 012310 000000 300202 300100 003000 300300 010010 010010 000210 000100 100202 200101 001000 110101 010010 010010 000100 000010 200101 100100 001000 320202 010021 020210 234101812 234101812 234101812 234101812 234101812 234111821 234111821 234111821 234111821 234111821 234111821 234111821 234111821 234121821 234121821 234121821 234121821 234121821 234121821 234121821 234121821 234131911 234131911 234131911 234131911 234131911 234131911 234131911 234131911 234142022 234142022 234142022 234142022 234142022 234142022 234142022 234142022 234152022 234152022 234152022 234152022 234152022 234152022 234152022 234152022 234161912 234161912 234161912 234161912 234161912 234161912 20 03 00 10 00 21 22 00 30 03 20 32 20 30 02 00 30 12 30 00 20 1O 01 10 11 10 20 10 00 02 00 10 03 00 10 00 10 12 00 10 03 00 10 00 21 11 10 30 01 10 001000 010001 010001 011000 100110 030012 003200 030011 020001 002110 020111 302101 000100 010101 113100 020000 000001 010010 020011 002220 000010 010002 202101 020002 000001 000001 010002 203201 201221 030102 103100 020001 021001 010111 010101 011101 120110 020102 103011 030102 000000 020002 030100 122001 220110 020012 112111 010011 011001 010001 010001 01220 01100 01100 02000 01100 30300 02000 02000 03030 00000 01000 01000 03000 11001 01000 10200 12010 01010 01300 02101 01101 30300 03000 00300 10030 01100 01000 03000 03100 30200 02001 00100 11120 01000 00100 01000 03100 32200 02100 00200 00230 00100 00100 01100 02100 31200 11100 00100 11120 00000 00100 22000 12000 22010 11001 01100 30000 00201 33000 03000 02001 02010 02000 00302 00010 10000 10000 00010 01000 01010 01000 00100 12020 10001 33000 22000 11000 21001 11003 00202 33030 10201 22000 12000 11000 22010 22001 00201 03130 10200 11000 11000 33000 11030 22001 11101 02030 31100 11000 10000 11000 22010 170 020210 000110 310211 200200 000000 330320 030220 030020 000210 001100 010203 300000 003000 010010 110010 010110 200200 000200 302202 310201 001000 300100 000010 020010 000310 000000 200302 120200 001000 320303 000010 110010 000100 000000 100202 200100 001000 320220 210010 021130 000220 000300 320313 301300 021000 321222 110110 010110 010210 010110 100211 234161912 234161912 234172211 234172211 234172211 234172211 234172211 234172211 234172211 234172211 234182112 234182112 234182112 234182112 234182112 234182112 234182112 234182112 234192111 234192111 234192111 234192111 234192111 234192111 234192111 234192111 234201811 234201811 234201811 234201811 234201811 234201811 234201811 234201811 234211921 234211921 234211921 234211921 234211921 234211921 234211921 234211921 234221821 234221821 234221821 234221821 234221821 234221821 234221821 234221821 234232012 011001 000011 100111 203300 020001 001001 000001 030000 000001 220200 020201 103200 020002 030002 010001 020000 010100 110010 032112 111101 001101 102100 010001 010100 111201 110010 020103 202002 030002 030001 010001 020001 002000 000120 102001 113212 030002 010011 010101 010100 111100 120100 010002 102111 010002 020101 010001 010001 012100 111111 030012 00100 01100 22311 02001 03100 01130 02000 01100 00000 03301 10100 02000 10100 10001 00100 00000 00100 11100 22100 12100 01000 01020 01100 01000 03100 03000 31300 02100 00200 00130 00000 00000 00100 02000 11311 13100 10200 00210 00100 01100 01001 01100 20200 01000 00100 11120 01000 00100 01000 02000 30300 22000 10000 33030 11303 13000 11000 01000 11010 10000 00201 33010 30201 22000 31121 02000 22010 22000 00000 10021 00101 11000 02000 22101 03021 22012 00021 32020 00101 12000 22000 23020 21010 21020 00203 12111 01200 11010 01000 01010 11010 12001 01001 32021 10101 12000 22000 21000 21011 21001 10201 30000 171 111100 000010 230212 012001 000110 021120 001000 300102 200200 001000 210101 011010 000010 021220 000100 212201 100200 000000 211213 011020 010010 000110 000110 322203 231013 000000 320302 110020 010010 001210 000000 210201 200101 002000 322212 010010 000010 001110 010010 200312 200101 001010 220211 110010 010110 000310 000110 110112 100001 001010 330320 234232012 234232012 234232012 234232012 234232012 234232012 234232012 234242122 234242122 234242122 234242122 234242122 234242122 234242122 234242122 22 00 20 03 20 32 20 13 33 01 3O 03 20 00 00 003200 030011 020001 002110 020111 302101 010100 100111 203300 020001 001001 000001 030000 000001 220200 02000 02000 03030 00000 01001 01000 03000 22311 02001 03100 01130 12000 01000 00000 03301 00201 33010 03000 02001 02001 02000 00302 33023 11303 13000 11000 01100 11010 10000 00201 172 030220 130020 000210 001100 010203 300000 003000 230212 012001 000110 021120 001001 300101 200220 001000 REFERENCES Abelson, R. P. Scri t Opapcessing in attitude formation and decision making. In J. rol &J W. Payne (Eds. ), Co nition and social behavior. Hillsdale, N. J.: Erlbaum, 19 Atkin, R. 8., & Conlin, E. J. Behaviorally anchored rating scales: Some theoretical issues. Academy of Management Review, 1978, 3, 119-128. Averbach, E., & Coriell, A. 8. Short term memory in vision. Bell Systems Technical Journal, 1961, 49, 309-328. Berman, D. S., & Kenny, D. A. Correlation bias: Not gone and not to be forgotten. Journal of Personality_and Social Psychology, 1977,35, 882—887. Bernardin, H. J. Effects of rater training on leniency and halo errors of student ratings of instructors. Journal of Applied Psychology, 1978,.93, 125-131. Bernardin, H. J. Behavioral expectation versus summated scales: A fair comparison. Journal of Applied Psychology, 1978, 63, 125-131. Bernardin, H. J., Alvares, K. M., & Cranny, C. J. A recomparison of behavioral expectation scales to summated scales. Journal of Applied Payphology, 1976,.61, 564-570. Bernardin, H. J., & Buckley, M. R. Strategies in rater training. Academy of Management Review, 1981, 6, 205-212. Bernardin, H. J., & Kane, J. A second look at behavioral observation scales. Personnel Psychology, 19801.§§3 809-814. Bernardin, H. J., & Pence, E. C. Effects of rater training: Creating new response sets and decreasing accuracy. Journal ongpplied Psychology, 1980, 65, 60-66. Bernardin, H. J., & Smith, P. C. A clarification of some issues regarding the development and use of behaviorally anchored rating scales. Journal of Applied Psychology, 1981, 99, 458-465. Bernardin, H. J., & Walter, C. S. Effects of rater training and diary keeping on psychometric error in ratings. Journal of 173 174 Applied Psychology, 1977, 62, 64-69. Borman, W. C. Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings. Journal of Applied Psychology, 1975, 60, 556-560. Borman, W. C. Consistency of rating accuracy and rating errors in the judgment of human performance. Organizational Behavior and Human Performance, 1977, 29, 233-252. Borman, W. 0. Format and training effects on rating accuracy and rating errors. Journal of Applied Psycholggy, 1979,.95, 410-421. Borman, W. 0. Implications of implicit personality theory and personal constructs for the rating of work performance in organizations. In F. J. Landy, S. Zedeck, & J. Cleveland (Eds.), Performance measurement and theory. Hillsdale, NJ: Erlbaum, 1983. Bousfield, W. A. The occurance of clustering in the recall of randomly arranged associates. Journal of General Psychology, 1953,.42, 229-240. Bower, G. H., Black, J. B., & Turner, J. T. Scripts in text compre- hension and memory. Cognitive Psycholggy, 1979,.11, 177-210. Bransford, J. D. Human cognition: Learning, understanding, and remembering. Belmont, Calif.: Wadsworth Publishing Co., 1979. Bruner, J. S. 0n perceptual readiness. Psychological Review, 1957, _6_4_, 123—152. Bruner, J. 8. Social psychology and perception. In E. E. Maccoby, T. M. Newcomb, & E. L. Hartley (Eds.), Readings in social psychology. New York: Holt, Rinehart, & Winston, 1958. Cantor, N., & Mischel, W. Traits as prototypes: Effects on recognition memory. Journal of Personality and Social Psye chology, 1977, 35, 38-48. Cantor, N., & Mischel, W. Prototypes in person perception. In L. Berkowitz (Ed.), Advances in experimental socialgpsychology, Vol. 12. New York: Academic Press, 1979. Cohen, C. E. Cognitive basis of stereotyping. Paper presented at the Annual Meeting of the American Psychological Association, San Francisco, 1977. 175 Cohen, 0. E., & Ebbesen, E. B. Observational goals and schema activation: A theoretical framework of behavior perception. Journal of Experimental Social Psychology, 1979, 15, 305-329. Cooper, W. H. Ubiquitous halo. Psychological Bulletin, 1981, 29, 210—244. Craik, F. I. M., & Lockhart, R. S. Levels of processing: A frame- work for memory research. Journal of Verbal Learning and Verbal Behavior, 1972,.11, 671—684. Cronbach, L. J. Processes affecting scores on "understanding of others" and "assumed similarity." Psychological Bulletin, 1955, 22, 177-193. DeCotiis, T. A. An analysis of the external validity and applied relevance of three rating formats. Organizational Behavior and Human Performance, 1977, 12, 247-26 . Dunnette, M. D. Personnel selection andyplacement. Belmont, Calif.: Wadsworth, 1966. Dunnette, M. D., & Borman, W. C. Personnel selection and classifica- tion systems. Annual Review of Psychology, 1979, 30, 477-525. Erikson, C. W., & Collins, J. F. Temporal course of selective atten- tion. Journal of Experimental Psychology, 1969,189, 254-261. Estes, W. The cognitive side of probability learning. Psychological Review, 1976, 83, 37-64. Feldman, J. M. Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psycholggy, 1981, 66, Feldman, J. M., & Hilterman, R. J. Stereotype attribution revisited: The role of stimulus characteristics, racial attitude, and cognitive differentiation. Journal of Personality and Social Psychology, 1975,.31, 1177-1188. Frederiksen, C. H. Representing logical and semantic structure of knowled e acquired from discourse. Cognitive Psychology, 1975, l, 37]." 58 0 Glass, A. L., Holyoak, K. J., & Santa, J. L. 00 nition. Reading Mass.: Addison—Wesley Publishing Co., 1979. ’ Hamilton, D. L., Katz, L. B., & Leirer, V. 0. Cognitive representa- tion of personality impressions: Organizational processes in 176 first impression formation. Journal of Personality and Social Psychology, 1980, 32, 1050-1063. Hastorf, A. N., Schneider, D. J., & Polefka, J. Person pgrception. Reading, Mass.: Addison-Wesley Publishing Co., 1970. Heneman, R. L., & Wexley, K. N. The effects of time delay in rating and amount of information observed in performance rating accuracy. Academy of Management Journal, 1983, 26, 677-687. " Hoffman, C., & Mischel, W. Objectives and strategies in the layperson's categorization of behavior. Paper presented at the National Convention of the American Psychological As- sociation, Mbntreal, Canada, September, 1980. Abstracted in Personalityyand Social Psycholggy_Bulletin, 1980,.9, 182-183. Ilgen, D. R., & Feldman, J. M. Performance appraisal: A process approach. In B. M. Staw & L. L. Cummings (Eds.), Research in organizational behavior, Vol. 5. Greenwich, Conn.: JAI Press Inc., 1983. Kelly, G. A. A theory of personality: Thepsychology of personal constructs. New York: Norton, 1955. Landy, F. J., & Farr, J. Performance rating. Psychological Bul- letin, 1980,.87, 72-107. Langdale, J. A., & Weitz, J. Estimating the influence of job information on interviewer agreement. Journal of Applied Psychology, 1973,_51, 23-27. Langer, E. J. Rethinking the role of thought in social interaction. In J. H. Harvey, W. J. Ickel, & R. F. Kidd (Eds.), New direc- tions in attribution research (Vol. 2). Hillsdale, N. J.: Erlbaum, 1978. Latham, G. P., & Wexley, K. N. Behavioral observation scales for performance appraisal purposes. Personnel Psychology, 1977, 30, 255-268. Latham, G. P., & Wexley, K. N. Increasing productivity through erformance a raisal. Reading, Mass.: Addisoanesley Publishing Co., 1981. Latham, G. P., Wexley, K. N., & Purcell, E. D. Training managers to minimize rating errors in the observation of behavior. Journal of Applied Psycholggy, 1975,.QQ, 550-555. 177 Lawrence, D. M. Two studies of visual search for word targets for controlled rates of presentation. Perception and Psychophysics, 1971, _1_(_)_, 85-89. Lingle, J. H., Geva, N., Ostrom, T. M., Leippe, M. R., & Baumgardner, M. H. Thematic effects of person judgments on impression organ- ization. Journal of Personality and Social Psychology, 1979, 37, 674-687. Lord, R. G., Binning, J. F., Rush, M. C., & Thomas, J. C. The effect of performance cues and leader behavior on questionnaire ratings of leadership behavior. Or anizational Behavior and Human Per- formance, 1978, 21, 27-3 . Maier, g. R. F. The appraisal interview. New York: MeGraw-Hill, 19 7. Marcus, H. Self-schemata and processing information about the self. Journal of Personality and Social Psychology, 1977, 35, 63-78. Marques, T., & Howell, W. Intuitive frequency as a func- tion of prior expectations, observed evidence, and individual processing strategies (Tech. Rep. 79-06). Houston, Tex.: Department of Psychology, Rice University, 1979. McArthur, L. 2., & Post, D. L. Figural emphasis and person percep- §§8353 Journal of Experimental Social Psychology, 1977,13, 5. Minsky, M. A. A framework for representing knowledge. In P. H. Winston (Ed.),.Ihepsychology of computer vision. New York: MtGraw-Hill, 1975. Murphy, K. R., Garcia, M., Kerkar, S., Martin, C., & Balzer, W. K. Relationship between observational accuracy and accuracy in evaluating performance. Journal of A Jpliéd Psycholpgy_, 1982, 67, 320-325. Murphy, K. R., Martin, C., & Garcia, M. Do behavioral observation scales measure observation? Journal of Applied Psycholo ogy, 1982,67, 262-267. Naylor, J. C., & Wherry, R. J. The use of simulated stimuli and the "JAN" technique to capture and cluster the policy raters. Educational and Psycholggical Measurement, 1965, 25, 964-986. Nisbett, R. E., & Wilson, T. D. The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality_and Social Psychology, 1977, 35, 250-256. 178 Osburn, H. G., Timmreck, C, & Bigby, D. Effect of dimensional rel- evance on accuracy of simulated hiring decisions by employment interviewers. Journal of Applied Psychology, 1981, 66, 159-165. Picek, J. S., Sherman, S. J., & Shiffrin, R. M. Cognitive organiza- tion and coding of social structures. Journal of Personality and Social Psychology, 1975, 31, 758-768. Potts, G. R. Information processing strategies used in the encoding of linear orderings. Journal of Verbal Learning and Verbal Behavior, 1972,_11, 727-740. Pulakos, E. D. Comparison of two rater training programs: Accuracy training versus error training. Unpublished Master's Thesis, Michigan State University, 1983. Rosch, E. Principles of categorization. In E. Roach (Ed.), Co nition and categorization. Hillsdale, N. J.: Erlbaum, 1 8. Rosch, E., Mervis, C. G., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. Basic objects in natural categories. Cognitive Psychology, 1976, 8, 382-439. Rumelhart, D. E. Notes on schemas for stories. In D. G. Bobrow & A. Collins (Eds.), Re resentation and understandin : Studies in cognitive science. New Yofk:YAcademicPress, 975. Sackett, P. R., & Dreher, G. F. Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology, 1982,_§Z, 401-410. Sackett, P. R., & Dreher, G. F. Situational specificity and assessment center validation strategies: A rejoinder to Neidig and Neidig. Journal of Applied Psychology, 1984, _62, 187-190. Schmitt, N. Social and situational determinants of interview decisions: Implications for the employment interview. Personnel Psychology, 1976, 22, 79-101. Schneider, W., & Shiffrin, R. M. Controlled and automatic informa- tion processing: I. Detection, search, and attention. Psych- ological Review, 1977, 84, 1—66. Schwab, D. P., Heneman, H. G., & DeCotiis, T. Behavioral anchored rating scales: A review of the literature. Personnel Psy- 179 chology, 1975,.28, 549—562. Sentis, K. P., & Burnstein, E. Remembering schema-consistent infor- mation: Effects of a balance schema on recognition memory. Journal of Personality and Social Psychology, 1979, 32, 2200-2211. Shank, R., & Abelson, R. P. Scripts, plans, goals, and under- standing. Hillsdale, N. J.: Erlbaum, 1977. Shiffrin, R. M., & Schneider, W. Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 1977, .8_4, 127-190. Smith, P. C., & Kendall, L. M. Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 1963, 42, 149-155. Spiro, R. J. Remembering information from text: "The state of schema" approach. In R. C. Anderson, R. J. Spiro, & W. E. Montague (Eds.), Schoolipg and the aguisition of knowledge, Hillsdale, N. J.: Erlbaum, 1977. Sulin, R. A., & Dooling, D. J. Intrusion of thematic ideas in retention of prose. Journal of Experimental Psychology, 1974, 3Q, 255-262. Taylor, S. E., & Crocker, J. Schematic basis of social information processing. In E. T. Higgins, C. P. Herman, & M. P. Zanna (Eds.), Social cogpition: The Ontario symposium on personality and social_psychology. Hillsdale, N. J.: Erlbaum, 1981. Taylor, S. E., & Fiske, S. T. Salience, attention, and attribution: Top of the head phenomena. In L. Berkowitz (Ed.), Advances in experimental social psychology, Vol. 11. New York: Academic Fess, 1978. Tesser, A. Self-generated attitude change. In L. Berkowitz (Ed.), Advances in ex erimental social psychology (Vol. 11). New York: Academic Press, 1978. Thorndyke, P. W. Cognitive structures in comprehension and memory of narrative discourse. Cognitive psychology, 1977, 2, 77-110. Treisman, A. M., & Geffen, G. Selective attention: Perception or response? Quarterly Journal of Experimental Psychology, 1967, 12, 1-170 180 Triesman, A. M., & Riley, J. G. A. Is selective attention selective perception or selective response? Journal of Experimental Psychology, 1969,_Zg, 27-34. Triandis, H. 0. Cultural influence upon cognitive processes. In L. Berkowitz (Ed.), Advances in experimental social_psychology (Vol. 1). New York: Academic Press, 1964. Tsujimoto, R. N. Memory bias toward normative and novel trait pro- totypes. Journal of Personality_and Social Psychology, 1978, 16, 1391-1401 . Tsujimoto, R. N., Wilde, J., & Robertson, D. R. Distorted memory for exemplars of social structure: Evidence for schematic memory processes. Journal of Personality and Social Psychology, 1978, _35, 1402-1414. Warmke, D. L. Effects of accountability procedures upon the utility of peer ratings of present performance. (Doctoral dissertation, Ohio State University, 1979). Dissertation Abstracts Interna- tional, 1980, 49, 4011-8. (University Microfilms No. 80-01,853) Wall, S., & Yopp, H. The role of context and inference in the comprehension of social action. Journal of Experimental Social Psychology, 1978,.14, 351-362. weiner, Y., & Schneidermann, M. L. Use of job information as a criterion in employment decisions of interviewers. Journal of Applied Psychology, 1974, 59, 699-706. Winer, B. J. Statistical_principles in experimental design, 2nd ed. New York: McCraw-Hill, 1971. Wexley, K. N., & Klimoski, R. J. Performance appraisal. In K. M. Rowland & G. R. Ferris (Eds.), Research iplpersonnel and human resource management, Vol. 2. Greenwich, Conn.: JAI Press Inc., Wyer, R. S., Jr., & Srull, T. K. Category accessibility: Some theoretical and empirical issues concerning the processing of social stimulus information. In E. T. Higgins, C. P. Herman, & M. P. Zanna (Eds.), Social cognition: The Ontario symposium on personality and social psychology. ’HillsdaIe, :NI J.:fiErlbaum, Zandy, J., & Gerald, H. B. Attributed intentions and information selectivity. Journal of Experimental Social Psychology, 1974, gug, 34-52. 181 Zedeck, S., & Cascio, W. F. Performance appraisal decisions as a function of rater training and purpose of the appraisal. Journal of Applied Psychology, 1982,-67, 752-758. "I711111111111111114111“