' Q Q. g . 9' ‘7“: _ 7 ' .. ‘ LO . Q 0-h\ ‘ . ‘ ' . ."I 'A . ‘ .zak5;}§’ ‘ ,4... 5:“ ',.)._ ”;fiw...3'f.hlcu 0. 2‘ 5 '5\ £3.1‘J 0"!” N I Q ' $1'2‘ °' '. ‘ ‘ - O ‘9‘. (it‘d. # _ ‘f‘iwfzgh ”taunt: ' _ I . ‘~-‘-~- 2. I s, v" ‘ ‘ ‘gJAJQ'im‘d‘ v‘k '4 3 "““d"""“' I . fi _ 4 o . - .2 u. o ._.-'. ' 2- u' 0‘ ' ‘O‘V" :I’n \0 “C ':~.. .‘W -—'~ ‘9— ‘ I.C'~-‘-’¢'~ ~'..' fir-'4‘; ‘.-. .8 . 4...‘r“}.‘:::u_:1~ M.kru- v '4 c-Ifll". _. .n‘, . -,I.O—.I'J‘» rfioA \‘ c \h--.—_ V. I ..H...‘ - ,.-L~-—v.of¢-O-..A ‘ ..qL. _ “fig: .5 w.-« - -o-’- ‘ ' - I J". an T ’- q. .K-‘vv \OOIO- 0' \" a 0 a'r . f- \ o. f 0 ' "1': '. - .5.3‘: .r‘ . In ‘ I .2 J. -. ‘ . -- 0—, av - -—;‘;.-~: 0 ' v. - — v ‘ m 5.?- » ' ‘ ..o ‘1‘ ‘ 5w s“ 3~ ‘ k»: swan-— 'W\rm~.- .9...“ . ~.‘.‘3§\ 2" ‘~“ ‘1‘~= .J ' f. ‘\ ‘ .2 J '.-, 'u‘ \I I ‘ v" ‘0”. Qty“. 4' ~‘ 4 _ "1 V‘ 0 3 . ., 3t...” 5";" I °'\:“"“'1f..i'.¢ 45“" k ' F‘- ' " ' 'AZ‘. .12.: ..‘.u‘:.\" ' h V —q o- " Q.'.: -0 v .1 fl‘g. -—-9 '00-.“ . . . M . o . ._. . - t.‘ c-on..’~a—-A- - K. M J v . '0” ‘ . 0- “0: I‘ fut .0. .L‘ "'&.:I‘ .\.\.§ ‘ i:%::v“toA:-=:i '— mu ‘ h.....“ p "t o‘b‘ho'v‘ r -0. ‘ a c Q -» { ‘ n_o-- - o... ‘5' - V - w o‘ I n-‘Oc. . ~-.~..~‘.Lo .0....\._.... - r ..t§‘ p ,..6 601°. ‘o-,‘.',.’.r p.5-~‘\~.¢.. no . ‘o‘>~U .M Q.. g.. .2 ob h t “A o—- .-oo o 4- - o. .' vs - i‘sL..."2‘.I.~3.°C“ v ' x -. h ‘1 ’o-;x. ‘ Q.--~ . 'L.‘ t. -- .‘ q . .. . '4...‘:O;::‘1 \ 51a~.+‘£mlw fil 4‘:~:; . ~ 2 p q I ._ c. ~:c ”tr...“ -V ' -. -W'(" 0‘ “Q'n‘kabb‘x 3.. .v .u. ‘47:.” rye u “o-‘~v -‘l\.:° -.vm.|_ - , ‘ ‘.“0"‘!: a“ -‘-' ’ Mw-btiw- Q» ‘.._ Q ‘ ‘5‘ ' aha v-v-r. ’-. . o ' - I I Mo<~00~b 6' I". ._.: 4. ;.. .3. -"O‘o.-‘0' - fi - . u H I .- -: ‘- c. hv‘b - . . . .. Irrwaaz. “‘ “.3273 J:J. ..: w- "-1.2..‘2m 2‘11! H‘TL‘A‘- «- m~~O|A2 - ;,1"}\.m'¢}z.'finq “it...“ «Or. nut-unhofilo”. - b. --o . 2. . ..H-L n 'V 0.“ ".~¢ .12.! . 2-;4 Io-~¢ a nm1.'...... :hnuk:~.: ..ot-b~ m Mun-:3 uMcI-‘u .0. a.‘ . -v-—o oa.5a§._‘ nit” ..CDQ“. - amuse-94.. «an .72.. s‘M‘ngi 3;...u. ‘0-“No.“u .k . uooIm~§8M¢t . .04.- .. o‘”\'r m ‘ gm . . a q H.- r»... .‘L .“ .- -3—.-.- v. ~15..-Q..¢L\\ u 3.- '4’..." '4- to... Jiohuu~~.;-w~:§bbmb wt .6. ‘ n... *7} .- ‘2» --.‘s‘. .1» at}. M“" I ”5-. -u - 9‘0““? ‘ ”Qt.- | butt I ou—a can u , .- “an“- be.‘ ;§m}~..t O ~oA-..~..a- ow..- w... . 1. . .. - ~ 2‘;.;. '2...“ A its: ---‘-4 W“ 04' ‘rfizum a“. at»; .-.- -m- 0- 'rv’ .Mo -~a‘.. .-I. ' . . 9v .uu u h... H . .. ".&.3..2.' ‘gguJilfl, _- , r... ._ 0h ‘0. v Q .., .. _ .. z. ,5, . . , '7 .u..~-~x.q1‘£otg‘-8~ohuuflsd ‘13..“ .. v u. : '1‘-.." y“‘0-o.- -"‘”.AIO'. » \K-b £u~gnmsuo ,u—I‘ A cwa‘.‘ -:J .-F~ 53.;- g .‘o O 000' r. -\.A{£‘—- - v z. - sV-o- 'I' -~ .‘t‘u “ ~f-fl‘ - a. ‘ u" - 1‘... ' 7m, §:.-LI:J J~QL..A.-.. .‘ 4 _ _m¢~.M90-0 'u 0"»... -.:.'~\. .Q-Jfisu. 09‘ u’..\;:‘ *-‘¢‘ ' h» a K “"0, “$.90“ “11335:;- m Ito-u tau-- .._.g...A.o-. —:. ,o in“- x :4 g . .. W"“""" ('1. o A N I 5-“; uh 0 - ' -0 \0'\ om.. :2::: «.4. 2:: "*1 .w ‘ 9 ' - m~l~ pin" fir ..§ 32 v-‘mr.0vtntw ’ ts.“ .A 3 0' ' *0 -Q ~:‘l..o:00. '0"?01b0.t‘ A. ~ ‘u-go~d. o. 0‘ 2 0-04 -5s¢‘.-‘ .x—w ..go.'§. s ~5ao.-.o§..¢:¢-‘ ' “ .4 ~:T~‘~-Io'~ : ..1'm..i'§i;1mf°~ not . .u o—c,uw Q n -‘ . 2..- k" can...» \n.%¥f~lo. .d‘LL' Mwsu— :9 .' a... mu~~¢ ' A?“ m Ti“ 0 N‘Uh-Ilk! \-q ram. \‘l h... «4.1.... ~- 5.3-». a ". . 0» fi 0 *fig: ou:‘- 1‘. u...--.: ‘sh «..:w0 "3.3“! it. .. 5~gldoafih$ in Ph' ~J.a:o..‘§s:ov- in“ ~ .. v» man-n32. § «.1; w» o "w sc\-8~‘. v'xl ‘34,“ "on. tn... .5 ml. 0... _~ ”9:... M'— ...Aw . . g ‘3‘} ’ ‘ I 0.2% ALI-LAO. t‘ drbv h; .hTQfQ..’Q.| u * L“... “29ml; i...» ..‘4.-...1‘ Imu.‘ c- 3.9.0.0 ..u 4‘. °....,- «Wm-Iofl— . “angig 4- -. - qu.~~a:‘: .u... ..‘1...’ I a. -...- u 9::‘Jua a“, 3d.‘ a“: Q-ufi—nu -‘.b % q 9mg. 0". O ‘1 2“. s .s.¢fi. ‘- «uq ..I ‘1'. 8- ~O‘I. .tlg‘llu. 1. .‘ d -. QQAO ”0‘“ *~ “"""'~." L... - I .. ~& .1 ‘1‘. -»~ I M u n. “or M .0. Q J. . F..- .‘ Mm...“ “‘3. ‘ finn».v tficm Lt « I O 2-» l ph'MQ ’22-...34Mornaa'” ‘: 1 fly. 5.5”}, q ., 0‘04; I'M-4M ‘,.’U§.U§. s _u aligfinu‘uo-Qm-wg“ " ti >.N " ”q ‘ . . .o,'.;.‘~ L1, . 0.3., .I. .umubu .~vu. .. .2..- .1 . " . ‘ "WW-" ,.-_ 0.0- -__,.‘ .l N...- k m U ldmgmn . Iv. m‘fik. .0 0mm \ U-.. 1... but. “a. ‘0‘.“ 0- n .0. $0! um! I" §~§o It {3" n. u .uuqh-N-ugg v: 09-» 1.. ..s. 6.... 4.1 ‘ h'vnwlu'h oh vtu: cum,» .0 - n - a... a“. -\o v- 0- ‘ a3,..aumo-p ‘7‘. 1"”; u.- “ u‘h\" -.v\ ~ 1.. ‘0 N ' .J*‘~IA%-~ HQJL‘M‘ ‘. M. I O V r..- ‘ - in .. .uyd nun—J. . no. \ ssh-us... a. 3 2:” g} It a - ‘ ‘ ‘ \ -‘u I ' fmtifitxwzar ..:.m': ma: 3).. * 7‘ x :33. 2' ..: ~‘I’t‘1ww22rrrr :.~:“-~*M“§fi 97:51? ..:231 _-. ' ...-.:u'- y .. 0.. .u ..:m ‘3..." ..:..su Ar» bE} Idan‘g 0‘.“ not. ‘H ‘3'“: ‘x fl' Nq’ o.voc~.~:u| ..:-3:2. jg: “3' ‘ .115 :;'§':"‘. tho-t ‘ mu - ‘ . clIewuu-‘ t ; ¢ . . a a c ‘ ~ ' .11;th 2w, ~ mm 2.53. 2.2- 32? :2. :2 V“ 23". ”'2'” 2* “5:32;.“ '2 . o D. ‘ I . - o; MJ'm-au'm-Inu 1.0.4....mwwul...’ .o--:Q.I‘:&Ls ~‘v.‘..m.‘,,, _ ~f‘ . l 2 I 0! .erCf. s .. .II- and. ”7 r . 4:323:39; I'm... . 25:26:)“ I“. ' -.u IA 3. .o—IJ'.< bl :- _: ”m ”n:r:v’ v m XWvJ€Rb\OIo-o.o'~rimh£ J‘ ‘11.- A 5' I I] . ' . \‘U o I ' |nb.i: "‘14 UK s s. ( ..:‘"KQ“ u“ ‘U‘ -xl'~ ’0 "3" ‘2“ -M ' ’0; Ag: [m' I .3 t .‘inw‘i‘h. ‘.'. ..: h. L. #7.: I .I“ . § . o C W ‘0' .‘ -. - ._.. - .- w” 0‘2" “ “z t’:&. .- ':‘?:::;;X. -.:.42 u. 'g)? 3....r4- <0 -.o INKS] :~.:.MEI:‘:.51 gala. “d My-JO-nw.‘ A ' a hrh~ofiv -332...“ u... u—N My. ms~-do-t~ .— ubM d 6‘ I s... .g... ”a -.....u...:...u.-.-QN .. a. . .. c. u «9-0 ... ”L 0:“.93'cw ~ 2.. 0‘0. . o - vo.~’\\oo «and .¥‘.‘66r an...“ «0.52“ 9» dis. ’3‘“- (“no gr- 5.. -\so- g- ..:..g‘ ..:. ..:,w pl ; Not- «1!- - . . . 6-.qu vcfintltk‘uficuyd‘h- g i - t ".33.. ‘ c“ v. 9‘“ .... . , 1 - 0 '9‘ a . - n a gvfibw “Ch I. ' I-b 0-“, I gun’s!» v»- r wag-Sn I lt-Ih IW-rfigil .‘ 2n- 3‘ at 4.2m. IZLLM“ ...._ z'flqz‘m'" "'9 " 'b"-' b—uv-W U. Q up»... '1. -- ‘nflu... 1». “Cm.g ”qt .vt-M J. i. mflI-.;.§‘c “kl--5... !:V M, "“"’.”f0 w... , . .ve. {..: ' -. I. Pk”. \.. s..nc..-\x:0 AJ-r-u. ul‘lszw ~ . ' . , - ;,(. .4 7 . . v“ .3 «that c 9.92;,»— 1 -. I ;v-MAN4 1..- - -- I. u'nh. - Q“... 524‘, «any: ~- “(A .‘A ‘0 . City. £Jn‘..luuh.'1¥1x:uu-viv'3 ‘- 4- ~ .. «2.1.. to uyudvl‘u-u 0° o ‘0‘. . '0“- ' its...“ “3...?" . 1 st nun—n: .‘b ’L “‘“i’?$"~ J2? 'KU.£;J:?. — t' '1“. .0. 01.110-1 y.” i. 42 lM—fl “3"". v4 - c-Mt- b If I. ‘1 ‘ob.‘o'.-o090-.§ 0,“ “2”,; -“. l“. . ~- I‘ c. 'O- ' I .' I'I \ «U .39.!“ a... o .-.I‘ - ‘ ”simple -O’..\A~.. -.~ I It It. I \ O" ‘ J. ' I» “- rhd'O 0“. «vvm ,I . 93 0 "' t u 0'“. an".-. .a. ‘1'!gb gs nu". 0-- OW7$ “a. § 0 “‘4‘ dad ‘& L... 8.23,. HA“0~.§15100':W“\"~ M:t=, (likudtth. 1-!«&4+.--- u...- ‘ -mw _- A. ~- is“. old-v.4-.- .- Ow; h ‘ n Jug; .' ”a ' r ‘ r... '4 a 1' in ‘ ’mv U‘.‘ 8".-.“- {WI-b.0- u .uwv-‘ouoais‘fiohu ..p on 8! Jim}. {Quart “h. ‘ " ..:. .. «7 can"... - ,3 Q a”, ~"' ‘. M“ M‘d ““8.“ ea.._ 0!. v " 3.5.». v 4 o ..v; s 4 t .. '1 .5,“ flaw“. "In whoa-u d that I!“ I‘Q ‘IV 0.2% .I'. ”he- ”Lao. ' f x: A... of I‘d-r -. . . M . o g, .53 ’. an. (raw; .ft' I» j: :74} _ “M ..::s“:-(:’ifi “a“: dz“). '35 :1: - 1 d '- an .6 It...» “4:57 . o.— o-S.u_ .6&\s<"~ 3". t'v‘IO’ . u't’ . r0. . ‘ 't .0 ;;5'."J . ‘ 'JJ q‘rvh ”in. ' -I ah ’I' O ' \"fvhtkyo ' , .10. - .” ::'—.'I‘¥’.’o- ; I a ' .\;d- - . :2“ .~'“:h:.‘“‘ v.50. .upaJ-o. J£.&§ 3:“... '79. I“: - .33.}; "t. m u ,H 2:: ;‘§ A . ' 2 JE- . - 3 . . . . .. &.--‘- a 2 ' «an. '34-;- 'Woo.t:: - \F o 1 ' . o..- ¢c :yr‘h; J—.~‘-:‘. ' w leis-‘3‘! 5"! .. “3"”;45'7’33; ' '5‘“.sz . 3-“3' - '.a'-'-é- ’ -'-t-!'“‘-‘eJ . 0‘ ~"& A. B-“- ' g b -. ".I l . ' g. ‘ ' . . ‘ n . c' ‘ — . -. ‘ . hi 8:- ».‘..2 3" - ot-fkl '.‘:h‘b1_.l_-nu.u.g2..,- £IL m ‘,-$--‘Iu...uu ' ‘ “a k l ‘ : .- - ' d 0o 9‘ var-O nom- -u -._‘ ,y . . }:.a.v‘ ‘5... .4“? ‘J..O‘.I..““ in \- p a. .0 ‘ .1.“ um .5 3,-1.1 . fl... .:‘.:.2‘~ _r§ c-s act. "I ’5“ $Ju... -:5, y. . ‘_ “ .(L .‘3 (..:..h"; 2“,:. '5‘ v ._ :l_. .. h...“ -._, v ”a“. .2 “to": '9. :1As-~bx-o- ' . ¥~1 u'oo-uo;-+. v I ..ss’p. A: 01:“ ._ ~‘ .-0 _ Ia- . a . ..: . n —-¢ .5... n' a: . .N “O. 3. ‘ 0 us- 13 V KJ~.:I 0:... 'o-cM!.OL. v ..o I _. . «4“ 'Isa'."1'hc ~‘l . 9-. ‘:‘~. _, ‘ 0- , . l '\W‘u§d.A §.1:' - . - , ,. ‘ ° 6 . - - ‘1 n nt-huy ‘1”. . - Ah-k- ' I - - ‘ “rt. 0; ' ”A ' W. 3 ° ‘96 KL ‘0 _ . - ,0._* - . (....o...;u:i: . v ‘.;.~~. '.u ‘12. . " ~ . c -. I h... r ' :J .' «I 3 ~‘—- ‘1 -. .qofltvm :%;;fl.‘ . a! '~ -— ..V-I»::.~ J~ : ' ‘0 '0. 6-. way—h: d“. ‘ D l O a 'w ' 3".| git: oIr ~3:\:o-§h:s,. '3'? n .. 4“:- 4‘" .. g . - . -' c - u. .‘ uh" ll "-%,- - 'hvs; ‘- "Id" . '0’ ’ —-‘~ a I .5. -\' ’ k -- t db ' fl - .‘e. - 0:L' -;‘.3‘“‘” VI‘OJ- 33... p - 'L ' ' , - > '7 ' ‘ ‘.& Q‘. 1 ' ("3. 1 ‘1‘ ‘L. -I "'~‘l-'-‘- I.” - ' «d - - s 13v ,_ '«- - ’ ,.. .-‘ ' .r'. it ‘ ’22"‘-'-‘“- V .9 ‘. ‘ IIL;:.‘J.D..":"£. 'l ' ‘ I " ‘ * ‘ ‘ . ' - 19.01 ‘4, . f ‘ .I ' ,——~9 L‘--' a . Q 0- ‘ q . -‘ At: :3“ 5““- “ho-3L! - 23:“ .. ‘s’o -‘ ' - 50-5 ." ' M ~~_-t.-..l’h2" . -.'":"':4:' ~I'.-.;“:e 1'-‘ 7" 1‘ I . . . - _ -- ~ . .- : .- r A - I . t ' "‘5 ‘93” I ;’2 ; ¢ ‘15 ((3.:J‘ ‘ 3 ‘ .15 ' c ° ' ' ~0- 3;? I; 289’ 2') ~ 1‘ 2'. '-van~flfl\gr;o ‘o . 2‘.‘ 0 _.. ..- ._"‘.,._"‘ . ‘2 -h ‘. . .Tr ‘ . - '_ . " . D 3‘ . 53:5;‘A3 ‘.‘2"fi;};'. ,1. -.c'-)3"11.9 t‘ . In. ‘:.i~'rfl;'t h’on _go.;0~ r" 1 I,‘ .0 it . ' .. ‘. . . - :. v .. -li o {n.- ~!_9 ‘ * ~ _ M w b J, I Y ’ -... . *3. NE.“ 7» -.:.Jw-‘T; 5 ,7? f . '4... "..:" o .o' . .0 .- ’3' -‘ . _ ‘ ,_ .. , _ . -. .. fi . . . . . . . ‘ . - . .d . . . . ~ . -, - ’2“ a ' (#V ‘ .v .I l .' . ' . ‘ -__- -. ’ “J ~ .I ..w ' . . o . . ' r- ‘fts‘Io’ésw-s. t 2 - uII-a' : 3‘4 3‘ '1 3“:- ‘i‘ 45'»?- "3 3 5 “" . - .1 ‘n m" 9 -” 3‘ . - .3 --‘A-J~-. .... ~a A ~' O'r’-' q' ‘ ‘ 'c . ..' . - it - .I ' . _ '0 " ‘ ¢ ' “ ‘ ."~£ ._-. ' ’ ' I.’ t K 0 -. o :_ f. . y. . . '| ‘ .0 ’ T' .9.’ ..:!. “: z-l "-- .0 ' ‘- ' ~ "I , :;3;!;i 0' .. -. ° . ..u «2 ’P . . 0 - I» ' 0P. ' o . - .-'. .I...'_ '1.- l' I: .- O . . 5‘ :.'.'I'::'- ... . . . - ‘ ..q . r '. . u 5 . ‘ . _ 3' . " - - " fi" -fi... Jr . ._ . "0923(04; ,‘sw 'ut- 'J-.- . o-x ’\\' .‘ ~. «"9; p .r . 3'. .’ u" I. - 0.. g If}. (g h ‘ I ~.~- 'g‘fi . Q5- oknfoa 'w .. -‘ ~ -0' o .c o o .3...“ . . 0'. - - ’ . ' r ,m -.‘.-(.- .v'u »\',,'° W: ‘.‘;: i‘J‘.."' 'J’plrwJ'“. I11” '2 1 ’1'.” . ‘2 - -su . p... - - 3‘- -' . ,, -:.'-.~ 3"." ' Lift-”we -'-';'. '; ‘ I'VE» : :r.~' GIN-1“ ‘2'"; ‘. .. ii! .3' :I’ ‘ "’. -” ~-I- v .. - -- . - -- I‘D .‘ 1%: D P. . '.~. .. ". .- 2.0%?“ :OJ :flfifi: ; 9 .‘dfi—t ’ ' V'. ."Oc- J .ul . . ' ‘ ’- ' ‘ 0 ,‘K’? '4": '- -{. r . .3, I. h .' ‘. - II :: luftl' . J- .4 :1! ' ' . V > ‘5"J'I.' .0 . . : V“... . «.0 cu - . x w ~. . n. a..." v _ .-' .7 .' .1;~ Ir-.'a.,.fu v 4.”. . \- I ”2-K,- ."c'uv n; . ‘2 '912' m. 2.. 7 - -, . -' , . I; . ' O :-,.:S' f ‘l‘.‘. / . A. .c’:§.:a.c‘ —. ‘1‘). 33r.(.w ‘ or! If“ ..:-r)“. . . 3 ' . .flr’.=. - . g '1“. ...‘-. ‘ t v "— ' ' } '9. tr" a a o «V‘;.132:0 v‘ :7” 2' ~ 5 -. 51;:‘0 ‘ 1.21;- In; :,- ? :~;" .N .60 I." 004’ ' fE' “.‘ . §. «- - - .. - n ‘ ' .5 . I". _. J Jar-2;- -E -r_- ‘55..." .q'w.‘4 "‘ ; v.‘. 0 - ‘I- r- W; 1’ “" "i. .2‘.L‘:'-4 o ". ”o .‘.. n 'a I -' "' ‘\.-'; .yI _hno -f-~o .‘N‘:"‘f" u - .- '_-_.voo 10-“F,n’ .m‘e '_q.'&_. g.",‘a-'pq;pfl-LP&._(,.'.‘“‘ -’J O. '2‘ . I , M . , 1% u" w'. - u- 9.1- . .- 0 - rt:-,;i-'s W 2- -: flew— ' '7» unit" ;4~:.'.-\u!¢Io.-\. ." 'vfi 'r“" “”51 *2 -- ' -‘ - u .- 7' p wool .a‘,;‘ """l' 0“.'I o -_o’-o_‘,;.:I p’u_— _ Ow-'-'l"0 'lU’I¢'Q"'fio-(O$‘: 8“ ‘Ogid'r. ' ~ 0 ‘ o O“. o o '.| '0', . ¢ . w; v ' .539: «,9? 4f. 'Q.‘:L,,'.','.2i'.‘:'.-' . .. 4,.‘!.'t‘aé.‘._"-. :1: :. 12!? $511? 22-.- . 3-... -‘ -' ' ‘ ‘- ‘ . - - : .' .v I . :" . no ..:, . . - - -v. .. I .-_—_'-: u. ' l ‘ ° ' I. . ’ .1“... ~ ' “352:“. 1.2.09. 5'; g... {..‘7'2. I, . 3;,5‘: 2‘22: _ 3 ’1'. .; . “:11“? t - . ’ f :2 ':}£~’-”-:I. .,._f. s L‘, . r .' .' ‘ ‘5' ’. - M J" . a a _...-. 9.....t‘n . %..'.-.QC ' .u.-. c‘+. ‘.n‘ -.I.;, ‘9‘ ..:...‘sV'. :’4' .'...' "\I‘!"!“ 1...."' 4 '92 ° ' '3 " . t. .f ' .v' . - . .1- n I .‘ '41,. y. -" u . ..y ‘9’: “033-.. a-.¢,::., . .D"". , ‘ .. .i e... . M p. .-'.'._v‘ :_.-v.'d-..q g. _. . ’4‘, ‘ .: ‘ _ . “O I o“ ‘n 4". .'\‘ -.. &’~;I 1'8 .'_'“,‘.J¥‘ -'-'.'l~."." «9*. ‘. «c . t, I . 0 --. 3r -_ ~:‘— ,..~$" .. a ’ ‘1 . ' '..."’by |.-.'r.c1.. . .,..‘._'J’-..: gg'n‘o ("..p..' ‘ "I. a l_uI’f -05 ‘7 l - .g " "_&~t. - :~ 0 I ..: 0‘00... '. ‘V' 'I‘ 1-, $.g_o,,uv.’v"_9 doe. “ '- ‘ ~ ,. r. u -3}.‘ "y: rr- foo‘.'s'-“-.‘-----Ov °t:¢.'r'-h .9 II 0' 'I 1.5 1“. - 7‘ _'. ”1"" O _-.z;v..)'_‘_'-a‘,‘,g .vp... ‘2 H."_.\h."0'i‘r9«y.- o}""‘ I ““"l"’.09'.‘ -~.I\¢ofiao c -_Io_o .9070-3: - v0"| 5 a J ‘ '4 ' _ t. o. 2.1“...“ I $2.32!. In a . . . . :32, . ! :oofi.ofi" f‘.‘*o c v . C o O 'O I: 33:5 av: «2 . 0",OI." - ’ ; ..J.:r..fi’. . y .p '. "3 3a. -‘T.'.:.f ‘1‘; —. ' "I" . 'l’. .. I , . . c v ' ' 0. f. :‘o 0" -':Q’o’ O «.3 )6 - ‘. _ o) r. ‘Jno ' '. ' ' '- “0 ‘. 'r' "O 5 . .9- _..‘.' . a . -.fo; ‘I . Q. - I ..p. ' 0o... 1' ..‘l .5. __ . 3’ .‘ ‘01‘04onoD-0 0r.-o -o'-¢O".' .20. 10V -0' o>‘VD-a-oLO’y-‘ "y’fii . n00. vv'.-.‘, v 1: - to V. .o 2 c " . ' r V . O , L". 5 ... .. o,J. 'f'fl'flr.\-:.. .' .'..:::‘ ":':qr‘.-'J:J‘ i6.“ ..:f"; 4"o?¢‘.:. . .‘ '- . f . . .0 ..u'. ”.3 t. _ ..- - _ ._o o u. o . . t ; 9 o a.-. - o - - _ . o n o "‘:o ._. n ' av g. a& 0’ Myf‘: 't'dros.‘ ~ 1"?- u, “1;. 'Align’ir.r“~ 0‘ . ' :3 'v' '1- - o — .-Q 9 J J ft (_ .' .- ‘ .;= rfii‘ "'91)", ’0 'I - u-o 00" I'"'1“‘-Q'. by“ H" II ""- '}' ' " ‘ ' "t" _ "' " ‘ ‘ I""" ...f-. L. n-J- 'vw'y ..'. if», "f. ,.'.'...".. . ....’ f”... .‘pl-o.'.,.>.—. ;~ .'. n c.‘ .r.n . .".", ..(~:,: It! “0":5”.'_-...’ ..4._ , ,...'.".' -.- . . ....,, - rfld'. ‘n‘. n.. _O‘. o.- t ! Jo'o2 . r3 . lf" ‘I‘ -.'1. me .0 ..:-co - - m -. ... H - -l O. ‘1' ' o a, As." a 0' .c’.’ u .2. . - O .. . , '1'; 3.4 . '-_ ‘ - . "I' '0‘, v. ' ' I p ' 0' . fffrf ' c . -co'd" . I o n."' ..-d ‘g... 0' ’ '_r.-"'_... o' '.| 0‘5 0.3.... '— o I g. ..-. . v C PJ’s“... - in ‘ 10 ..N'I . c .I ( '.O¢n-- 1.0- Q . o...’.-..q’ot."". - ....-’|o'noH-v~-0'o o ‘0‘. owo,"-_~¢u:oln- - 0.4 ¢ .. -.,.-.....-¢:oov- c-o,, ‘.... ~, .2. no.0 . - ."L "' in“. I ' 55’... P " ' . ' 0" O .. ‘ t 2,... . ' “t... mm 5. 'f'u "I'? . r; a . - 3‘. r. o. ‘- :w ..‘o {O p- 6 0 , _ . ,'. ”I I I-' , fl...- 73"}:.;~’ .f"‘w loo-op. do, .-.-.u o -o .g . ‘. .oo': o". '9' . "I .0 7°» oon'_.c' "‘Y . I .Cfi.r u . f.... o.oo'rv ‘.0ho'oofgofu...no..—a 0'.L.--!’ v’ ‘g to n 0. cc ‘51:; ‘l ‘ "' .U ,‘ ‘1' 0 :86.” ”D ‘O‘QIQ ~ «.3, ' ‘ .In-vt'L-t ow Jo ‘nl! "’5 vuri-i ,5-‘yr ca 31".!" 3‘. I: r~ .-.'-o --v ‘ql-JIIJ v 9"an ,,I'9'Oo'-‘o '[roo 900‘!" ’ 0' u or II.- o .7 ‘1'- 'Ionrrvo'oovr o1?)- .0...¢~4~00‘."‘c 9.. D- 9 ‘ .‘ '5‘? ' .f.-| (: rugw-fi‘ II 9". o no..— I.‘ I tr— "‘ A‘O 'Of-O — 'J,,.',fl;} .(ng:oi. A :“r‘ur r.v"';‘$'( 9'. -9 _ ,.."'.. ‘r " .,' ' l'.. ..:."--:.'d ':.ovor 'A'.;:4;r ”a.-. .J': .- .7 p c an ‘VOII_ .. - - ’ o - . - ‘ ' Janna; ..:,- , 1‘L7’rg. ounopc-ol‘n.;. ' o ’ 5“. otoooDO->‘ .' 9.. 4222*" “'2" O v-oo~ p 4...: A." -O O | f" ;" -. ." ..: 'I‘?‘.o o l-. [V ' ’ I 0;”! 0.2 0'5"" I v.0 n‘ O; ~."$:l'-'.LC'I —f 3.“ ‘ .I >0... 4 ' 2o . n a . ,'. ' c a “v , *“ft“"-II.-b:'0tva/‘o .." ..IMI' 5.2171,", - -j" 0-... I to '0 on .¢lo- 4-yupl ‘“ 09". y' ' fw- ".D';l. " .; CL- "#2.! I... :0 i o . . .' . puny b. ..,..... ' 2,0 . .. . c .‘ V :0. - (Q _.Q.o’u ’. 4 '.- o o. O o - - . ' g I ' 8‘ ’w a 'pt’."::;:’w‘- 0:610". r. a.mt.' l"!.-v..o .1 .9.' _ a g o 2 . , . Dr‘. p.( ((4 .o .."c ’.OIO‘J J. v _.. - . '. . 5' oil. 1...‘I. .~‘ .‘O‘V. . .'I'~'."- w-O'ysrp' 5' 2* Q //C»~ lé l» g RSITY U3 Ivllfllmlml‘lmmmmum L 3 1293 00761 7909 UE‘uEARY Michigan State University This is to certify that the dissertation entitled Expert and Novice Trainers: Differences in Knowledge and Problem Solving Strategies presented by Margaret A. Maddocks has been accepted towards fulfillment of the requirements for Ph.D. Educational Psychology degree in Major pr r MW MS U 1': an Affirmative Action/Equal Opportunity Institution 0-12771 PLACE IN RETURN BOX to remove this checkout from your record. TO AVOID FINES return on or before date duo. DATE DUE DATE DUE DATE DUE ll “APR 2 a 1996' MW 47' ..- MSU I: An Alfirmdlvo Action/Equal Opportunity lnditutlon cWflnS-DJ EXPERT AND NOVICE TRAINERS: DIFFERENCES IN KNOWLEDGE AND PROBLEM SOLVING STRATEGIES By Margaret A. Maddocks A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Educational Psychology 1991 ii ABSTRACT EXPERT AND NOVICE TRAINERS: DIFFERENCES IN KNOWLEDGE AND PROBLEM SOLVING STRATEGIES By Margaret A. Maddocks Training in industry is becoming one of the major educational activities in the United States. Many organizations rely heavily on their own trainers to deliver a range of courses from executive development to technical training. Yet, not enough is known about the knowledge and skills needed to be an effective trainer and how beginners can become experts. The purpose of this study was to examine the differences between expert and novice trainers in their knowledge and problem solving strategies to uncover key areas on which to focus trainer development programs. Two novice and two expert trainers from varied backgrounds were selected for the study. They were asked to complete a series of problem solving tasks using a "thinking-out-loud" approach. First the subjects were asked to identify problems inherent in three training case studies and to suggest solutions. Next they were asked to complete five sorting tasks. The sessions were audiotaped and later transcribed for analysis. Triangulation was used to collect a variety of data and to analyze them from different perspectives. The amount and type of knowledge was identified, as well as the variety of ideas generated. The number of cues, problems, and solutions that the subjects identified were counted and categorized. Two forms of coding were used and a confirming rater was used to check the reliability. The categories and structures created were analyzed for patterns and similarities and differences iii between subjects and between groups. The consistency of performance across tasks was also analyzed. The research confirmed the results of previous studies in the expert-novice area. The experts' amount and range of knowledge was much greater than novices. The cues they identified and the solutions they suggested reflected a broad range of knowledge including organizational and transfer issues as well as traditional design and delivery concerns. In the sorting tasks, both experts created process structures, again revealing their view of training as part of a larger process model, while the novices categorized by similar techniques and tools. On some tasks one novice appeared to be more of an intermediate, suggesting a developmental sequence that trainers proceed through to develop expertise. The implications of this study are that trainers may develop expertise through a variety of experiences rather than in single classroom train-the-trainer courses. A developmental program of guided learning experiences might be used to train trainers. This approach might be based on a developmental model of competency that considers problem solving and decision making skills as well as classroom techniques. iv ACKNOWLEDGEMENTS There were so many people who supported me throughout this dissertation process and who deserve my gratitude. Thanks first to my colleagues and friends, who never doubted that I would finish, and whose faith kept me pressing on. Thanks to my parents, whose continuous encouragement, especially during the more challenging times, built my confidence and determination. Thanks and love go to my husband, Alan, who was incredibly patient with my late nights and fits of frustration, never once giving up on me and always believing in me 100%. Thanks to my committee members, Larry Alexander, Michael Moore, and Dick Gardner for their willingness to challenge me beyond my own expectations. Most of all, I would like to thank Steve Yelon, my chairman, colleague, and friend, whose enthusiasm first encouraged me to come to Michigan State; who is greatly responsible for what I know about instructional psychology because he not only taught me, but showed me how; and without whose inspiration I never would have fulfilled the dream of completing a Ph.D. TABLE OF CONTENTS Chapter One .............................................................................................................................. 1 Statement of the Problem .......................................................................................... 1 The Purpose .................................................................................................................. 2 Rationale ........................................................................................................................ 3 Trainer Competency ........................................................................................ 3 Teacher Competency ....................................................................................... 5 Rationale Summary ........................................................................................ 8 Study Approach ............................................................................................................ 9 The Research Questions ............................................................................................. 12 Scope and Limitations ................................................................................................ 14 Limitations in the Approach ......................................................................... 14 Scope of the Data Collection and Analysis ................................................. 15 Definition of Terms ..................................................................................................... 16 The Dissertation Plan .................................................................................................. 17 Chapter One Summary ............................................................................................... 18 Chapter Two .............................................................................................................................. 20 Introduction .................................................................................................................. 20 Cognitive Structure ..................................................................................................... 20 Short Term Memory ....................................................................................... 22 Working Memory ............................................................................................ 22 Long Term Memory ........................................................................................ 23 Executive Control ............................................................................................ 23 Representation of Knowledge in Memory ................................................. 23 Expert-NOVice Problem Solving ............................................................................... 29 Thinking as Problem Solving ....................................................................... 29 Expert-Novice Seminal Studies .................................................................... 31 Expert-Novice Research in Various Fields ................................................ 33 Theories of Expertise ....................................................................................... 35 Expert-Novice Summary ............................................................................... 37 Comparing Expert-Novice Teachers ........................................................................ 39 A Theory of Teaching Expertise .................................................................... 39 General Findings .............................................................................................. 41 Routines, Judgment, and Decision Making ............................................... 42 Studies of Teacher Competency .................................................................... 44 Expert-Novice Teachers Summary .............................................................. 45 The Expert-Novice Perspective in the Training Field ......................................... 46 Differences and Similarities between Teachers and Trainers .............................................................................................................. 46 Training Expertise Involves the Use of Techniques ................................ 48 Trainer Competency Studies ......................................................................... 49 vi Expertise in Course Design ............................................................................ 52 Chapter Two Summary .............................................................................................. 53 Chapter Three ........................................................................................................................... 55 Introduction .................................................................................................................. 55 Research Questions ..................................................................................................... 55 Qualitative Methodology ........................................................................................... 57 Why Use a Qualitative Approach? .............................................................. 57 Recommendations for Qualitative Researchers ....................................... 59 Ensuring Reliability and Validity ................................................................. 61 Data Collection ............................................................................................................. 64 Training Problems ........................................................................................... 64 Card Sorting Tasks ........................................................................................... 69 Subjects ............................................................................................................... 72 Procedures ......................................................................................................... 76 Analysis Procedures .................................................................................................... 77 Organizing the Data ......................................................................................... 77 Analysis Procedures for Each Research Question by Task ...................... 79 Limitations of the Study ............................................................................................. 83 Chapter Three Summary ........................................................................................... 85 Chapter Four ............................................................................................................................. 87 Introduction .................................................................................................................. 87 Data Analysis Summary ................................................................................. 88 Training Cases Administration ................................................................................ 91 Sorting Tasks Administration .................................................................................. 93 Individual Findings .................................................................................................... 94 Expertl ............................................................................................................... 94 Expert 2 ............................................................................................................... 96 Novice 1 ............................................................................................................. 98 Novice 2 ............................................................................................................. 102 Comparing Experts and Novices Question One .................................................................................................... 104 Amount of Knowledge ................................................................................... 104 Type of Knowledge .......................................................................................... 109 Categories ........................................................................................................... 119 Organization ..................................................................................................... 124 Comparing Experts and Novices Question Two ................................................................................................... 126 Cues ..................................................................................................................... 126 Sequence ............................................................................................................ 128 Flexibility ........................................................................................................... 130 Time .................................................................................................................... 132 Comparing Experts to Novices Question Three ................................................................................................. 135 Chapter Four Summary ............................................................................................. 137 Chapter Five .............................................................................................................................. 138 Introduction .................................................................................................................. 138 vii Summary of the Study ................................................................................................ 138 Findings, Conclusions, and Recommendations for Future Research ......................................................................................................................... 139 Research Question One .................................................................................. 139 Research Question Two .................................................................................. 146 Research Question Three ............................................................................... 160 Summary of Conclusions .......................................................................................... 164 The Research Approach Conclusions and Limitations ........................................................................ 166 Recommendations for the Field of Training ......................................................... 170 General Recommendations ........................................................................... 170 Specific Recommendations for Training Trainers ................................... 172 Chapter Five Summary .............................................................................................. 174 Appendix A ............................................................................................................................... 175 Training Problems ....................................................................................................... 176 Problem One ..................................................................................................... 176 Problem 2 ........................................................................................................... 179 Problem 3 ........................................................................................................... 182 Appendix B ................................................................................................................................ 185 Card-Sorting -- 30 Key Terms ................................................................................... 186 Part A .................................................................................................................. 186 Part B ................................................................................................................... 187 Part C ................................................................................................................... 187 Training Terms Glossary ............................................................................................ 188 Appendix C ................................................................................................................................ 191 Card-Sorting Task -- 34 Trainer Performances ..................................................... 192 Part A .................................................................................................................. 192 Part B ................................................................................................................... 195 Part C ................................................................................................................... 195 Bibliography .............................................................................................................................. 196 Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 1 1 Table 12 Table 13 Table 14 Table 15 viii LIST OF TABLES Question One: Content and Structure of Knowledge— Training Cases Question One: Content and Structure of Knowledge— Sorting Tasks Question Two: Problem Solving Strategies—Training Cases Question Two: Problem Solving Strategies—Sorting Tasks Inter-Rater Reliability Check Results Amount of Knowledge—Case 1 Amount of Knowledge—Case 2 Amount of Knowledge—Case 3 Amount of Knowledge—Sorting Amount of Knowledge—Number of Competencies in Cases Type of Knowledge—Number of Times Competencies Present, Case One Type of Knowledge—Number of Times Competencies Present, Case Two Type of Knowledge—Number of Times Competencies Present, Case Three Type of Knowledge—Number of Lines Per Competency, Case One Type of Knowledge—Number of Lines Per Competency, Case Two Table 16 Table 17 Table 18 Table 19 Table 20 Table 21 Table 22 Table 23 Table 24 Table 25 Table 26 Table 27 Table 28 Table 29 Table 30 Table 31 ix Type of Knowledge—Number of Lines Per Competency, Case Three Type of Problems / Solutions—Case One Type of Problems/Solutions—Case Two Type of Problems/Solutions—Case Three Type of Problems/Solutions—Sorting One Type of Problems/Solutions—Sorting Two Category Labels—Sorting Task One Category Labels—Sorting Task Two Category Labels—Case One Category Labels—Case Two Category Labels—Case Three Strategies—Cues Flexibility—Number of New Problems/ Solutions Time—Cases Time—Cases Categorizing Time—Sorting CHAPTER ONE INTRODUCTION TO THE STUDY Statement of the Problem Training in industry is becoming one of the major educational activities in this country. According to Training Magazine (1989), $44 billion was spent in 1989 to train employees of organizations in the United States. Of that, $31 billion (70.3%) was spent on salaries for trainers. And yet, not enough is known about the knowledge and skills needed to be an effective trainer and little, if any, research has been conducted to identify these competencies. There are many commercial and academic programs geared to prepare trainers, and many corporations have their own train-the-trainer curriculum. But what knowledge base are the decision- makers in these organizations using to determine the curriculum and measure the success of their "graduates"? Some professional organizations are trying to identify the knowledge and skills that differentiate excellent trainers from less effective ones. The American Society for Training and Development (ASTD) has developed a list of trainers’ roles and key skills based on the knowledge and experience of a committee of experts. The International Board of Standards for Training, Performance, and Instruction (IBSTPI) recently published a list of instructor competencies that details the specific tasks effective instructors perform in business settings. More work needs to be done, however, to validate these lists and cull from them the critical competencies that differentiate the expert trainer from the novice. Teacher effectiveness in educational settings has been studied extensively and in a variety of ways. The process product research, which identified the behaviors of effective teachers, produced rich descriptions and detailed lists of teaching skills, much like the competencies identified by ASTD and IBSTPI for trainers. More recently teaching has been examined from an information processing perspective, with researchers trying to identify the knowledge and strategies expert teachers use to solve the problems that teaching presents to them (Leinhardt and Greeno, 1986). No research of this type has been done to examine the cognitive structure and problem solving strategies of trainers in the business sector. Research on teaching may influence the way teacher education programs are structured, the experiences student teachers are given, and the criteria used to evaluate practicing teachers. In business and industry, similar research is needed to help design appropriate training and development programs for trainers. 50% percent of the training that production workers receive is delivered by in-house trainers, and 58% of all organizations provide train-the-trainer courses (Training, 1989). Those who are attending train-the-trainer courses and those who are providing them could both benefit if more were known about the knowledge and skills needed to be an expert trainer. The P ose The purpose of this study was to examine differences between expert and novice trainers in their cognitive structure and problem solving strategies. By observing and analyzing how they approach a variety of problem solving tasks related to classroom training, the researcher hoped to uncover additional knowledge and skills needed by novices to become experts than those currently identified in the literature. It was also hoped that the study would reveal that there are differences that may not be easily taught in a classroom setting over a short period of time, that may only be acquired through experience in a variety of situations. This type of discovery may indicate a need for industry to re-think its approach to training trainers, which frequently involves only a seminar or workshop on teaching skills. Rationale Searching for ways to improve training in business and industry is justified for many reasons. In the landmark report, Corporate Classrooms (Eurich, 1985), the Carnegie Foundation for the Advancement of Teaching stated that education in industry is here to stay, and warned that without continuous evaluation and improvement of its effectiveness, the success of the entire nation is at stake. In the conclusion of the report, the author suggests that a national strategic council be formed that would "assess the nation's emerging educational needs, identify and review our educational resources, and recommend policies and programs." If such a council were formed, it would most likely use research results to help reach these goals. Since then, as global competition has increased, virtually every business magazine and journal has mentioned the importance of effective and efficient training programs to upgrade the skills of American workers. Training, particularly in the area ofquality improvement and technical skills, is frequently identified as a contributor to the continuous improvement of organizations today. In the view of most business writers, training is here to stay. Trainer Competency The ASTD and IBSTPI competencies are strong foundations from which to start to investigate trainer skills. They were developed from the knowledge of many respected leaders in the field of training and development. The ASTD competencies were generated from the responses of hundreds of practitioners in the field and then validated and categorized by a panel of experts. The IBSTPI standards grew out of the work of the Joint Certification Task Force, which was composed of people actively involved in the Association for Educational Communication and Technology (AECT) and the National Society for Performance and Instruction (NSPI). The publishers of both documents state their desire that the competencies be used as research information, to be investigated further and validated. What needs to be identified is how the competencies are called upon by the expert under various conditions, i.e., when does a trainer use the competencies and for what goals? There is little research on trainer skills, although there is some on successful teaching approaches and techniques used in industry, such as behavior modeling and feedback (Latham, 1988). In their book, Developing and Training Human Resources in Organizations, Wexley and Latham (1981) describe the critical need to carefully select and train instructors because of the tremendous investment companies make in the training function. Training costs include not only the salaries of the trainers, but the salaries of the trainees, who often attend classes during work time. Even more expensive are the costs due to re-training when the original instruction does not achieve intended results or from errors on the job due to poor or incorrect instruction. In their book, Wexley and Latham describe the skills needed by trainers for them to be successful and warn that every training situation is different; there is no algorithm for effective training. There are frequent articles in the trade journals and magazines such as ASTD's Journal of Training and Development, NSPI‘s Performance 6' Instruction journal, and Training Magazine that report expert's opinions about trainer skills and train-the-trainer programs. Few of these authors are able to cite research because it is unavailable, and some have strong, but questionable, recommendations, such as avoiding having novice trainers observe experienced instructors (Broadwell, 1990). These writers, as well as the many consulting firms presenting their own beliefs about instructor competency through their train-the- trainer workshops, have a great influence on the decisions people make about selecting, training, and evaluating trainers. More research needs to be conducted to provide evidence to support these authors' beliefs. Studying the differences between expert and novice trainers may help improve training programs for beginning trainers. Most train-the-trainer courses in industry are taught by expert trainers, and the leader's guides that so many new trainers use are based on what expert trainers say and do while teaching. Although there is much that is good in these programs, there is little reference to the decisions expert trainers make and the way they structure their knowledge of the subject matter as well as their knowledge of teaching. Glaser (1990) describes what can be learned from the examination of expert behavior as a "temporary scaffolding" from which novices can learn to become more expert. Unfortunately, many experts cannot easily articulate what it is they know. After using a strategy enough times it becomes automated, and substeps, decisions, and cues become buried in long term memory. It often takes an outsider's view to identify what the expert teacher does that is different from the beginner. If these skills are articulated, then content of train-the-trainer courses could go beyond mere skill development to guidelines about how and when to use the skills. Teacher Competency Since there is so little written about trainer competency in the industrial literature, many professionals in the field turn to research on teaching for information. While research on teaching contributes much that is useful, it is also limited in its applicability-~the problems encountered while teaching in schools versus teaching in business have some distinct differences. Most important is the fact that classroom training in industry is usually a component of a much larger organizational intervention. Participants are usually learning job-related skills that are valued by their managers. Learning is often reinforced on the job after training, and many other forms of training occur to support classroom learning such as on- the-job coaching and feedback. All of these elements mean that expert trainers' knowledge must go beyond knowledge of teaching techniques into areas of organizational development, politics, and culture. Other key differences include the fact that teachers remain with students over an extended period of time while trainers most often do not, teachers plan their curriculum and lessons while trainers often rely on a program designed by others, and teachers deal with children, while trainers deal with adults, who have past learning histories that can contribute to or interfere with the training. All of these differences imply that the knowledge and strategies of teachers may differ from trainers in significant ways. Although much can be applied from research on teaching to the training field, studies within the field may uncover new, useful information. There are more similarities in the needs of the practitioners in both fields than differences, however, and the research on teaching literature can be of help in justifying the need to investigate differences between novice and expert trainers. Joyce and Hartoonian (1964) were among the first to recognize that we must begin to study why teachers interact the way they do, and not focus so much on what they do. They cited a US. Department of Health, Education, and Welfare report by Turner and Fattu (1960) that described teaching as: the generation, application, evaluation, and reformulation of specific hypotheses about particular learnings. According to Joyce and Hartoonian this report implies that a "teacher faces an educational problem each time he perceives that an educational objective is to be achieved." Berliner and his associates have conducted much research on teaching as problem solving. The primary reason he gives for studying expert and novice teachers is to learn more about the skills that the research community has already identified as important (Berliner, 1986). Under what conditions do teachers activate these skills? What cues signal that certain tasks should be employed? How do teachers know they have applied appropriate skills successfully? What knowledge informs these actions and how is that knowledge organized and retrieved? Studies of teacher/ trainer problem solving may help researchers and curriculum developers get beyond the idea of lists of competencies, refining what is observed into principles that are more broadly prescriptive and less task-oriented. Clark and Yinger (1979) believe that we must come to know how teachers "exercise judgment, make decisions, define appropriateness, express thoughts in their actions" if we want to develop general rules for teaching. Without general rules, novice teachers must prepare themselves by learning possibly hundreds of tactics without a strategy to guide their implementation. Leinhardt and Greeno (1986) characterize teaching as a "complex, cognitive skill" not just a series of behaviors that are executed at predetermined times. They believe teaching ...requires the construction of plans and the making of rapid on- line decisions. The task of teaching occurs in a relatively ill- structured, dynamic environment. Goals and problem solving operators are not specified definitely, the task environment changes in a way that is not always under the control of the teacher's actions, and information appears during the performance that is needed for successful completion of that performance (1986, page 75). Leinhardt and Greeno conclude that teaching is similar to other expert activities that have been studied using an expert-novice framework such as medical diagnosis, chess, and solving physics problems and should be studied using those models. Teaching, like these fields, includes not only advanced knowledge of the subject matter, but also of the task of teaching. deGroot (1966) and Chase and Simon's (1973) early studies of expert chess players and Chi, Glaser, and Rees' (1982) studies of expertise in physics uncovered the importance of subject matter knowledge--its content and structureuas an integral feature of the competence of experts. They focused their studies on the process of solving problems, not the output. Studies in teacher education and the popular writing in training and development often focus on behaviors and activities that are effective rather than the underlying cognitive structure of the instructor. There has been little research about the problem solving approach of instructors or the impact of their subject matter knowledge on their effectiveness (Leinhardt and Smith, 1985). Uncovering these hidden skills may lead to the development of new theories of teaching that include principles, general rules, and a new form of competencies to use for training, developing, and evaluating teachers and trainers. Rationale Summary In summary, there are many reasons to study the cognitive structure and problem solving approaches of expert and novice trainers. Such research may help uncover key differences that may not be known to practitioners in the field and may help to verify those that are known. The results of such research can contribute to the development of more effective and efficient train-the-trainer programs and other longer-term instructor development activities. It may also reveal that it is not a simple task to teach in industry and that a more thorough approach to preparing trainers is needed. This knowledge may also help make expert needs analysis methods, instructional decision-making, and delivery techniques more explicit in leader's guides and may contribute to better designed classroom instruction, especially when courses must be taught by non-experts who depend heavily on the instructional design of the materials. This research can contribute to the knowledge base used to design expert instructional systems such as those used in computer- assisted instruction and interactive video. Finally, these results can help in the selection and evaluation of trainers, whose ranks are increasing every day. By viewing trainer skills as more complex than simply exhibiting a series of behaviors, decision-makers in the field of training and development may be convinced of the importance of investing more time and effort in carefully preparing those who are responsible for so many others. Study Approach If training is considered different from simply being a set of skills, but is rather a complex planning, decision-making, and execution activity, how should the cognitive structure and problem solving strategies of trainers be studied? The problem solving and decision-making strategies of trainers in industry has not been studied. Even in teacher education literature, there has been a limited amount of research in this area. Until recently, researchers in education focused their attention on product process correlational approaches, which yielded much knowledge about skills effective teachers exhibit. There was little examination of teacher's problem solving strategies or subject matter knowledge during this phase (Leinhardt & Smith, 1985). During that stage of research on teaching, experts were searching for general laws of teaching that could combine to form prescriptions for teaching (Clark & Yinger, 1980). Although this research approach has identified many important skills that teachers can apply, it has not uncovered any general laws. 10 Since that time, there has been more focus on the cognitive aspects of teaching in an effort to uncover the reasons why teachers employ certain skills at certain times. The primary goal of this research is to understand the task of teaching, rather than to prescribe specific behaviors. Research that has focused on teachers' problem solving approaches and cognitive structure includes that of Clark and Yinger (1980), Berliner (1988), Leinhardt and Smith (1985), Leinhardt and Greeno (1986), and Ropo (1987). The Clark and Yinger studies focused on teacher planning as a complex cognitive activity that is a critical component of teacher thinking. Their approach was to use teachers' self-reports of their thought processes obtained through interviews, questionnaires, journal keeping, and "think aloud" procedures. They studied one teacher at a time rather than comparing expert teachers to beginners. Berliner's studies compared expert, novice, and postulant teachers (peeple from industry who wanted to teach but did not want to go through traditional teacher training programs) in a variety of ways. Berliner and his colleagues looked at various aspects of teacher thinking and problem solving including interpreting classroom phenomena, using routines, and predicting classroom phenomena. His methods included having small groups of subjects interpret videos of lessons and describe slides depicting classroom scenes to see what cues they attended to. He also conducted studies in which teachers predicted student behavior by reviewing records. In all cases, Berliner compared the thinking of experts to novices to isolate their differences. Leinhardt and Greeno examined the schemata of expert and novice teachers by the way they taught lessons. These studies were conducted by observing mathematics teachers in their classrooms. Leinhardt and Smith examined the differences between expert and novice math teachers by using such methods as classroom observation, videotaping of lessons, interviewing about teaching and 11 planning, and card sorting tasks. They focused on analyzing the relationship between subject matter knowledge and lesson structure. Ropo studied differences between expert and novice teachers in the structure of their thinking and their teaching behavior during lessons. His methods included clinical interviews and classroom observation. Ropo's studies focused on a wide range of cognitive processes including goal—setting, decision-making, the structure of questioning activities, and thoughts during guiding and prompting activities. The goals and approaches of studies of expert and novice teachers are based on the extensive research comparing experts and novices in other fields. According to Follman (1985), the expert-novice literature focuses on two areas of inquiry: quantitative differences and qualitative differences. The studies investigating qualitative differences have been much more fruitful in terms of uncovering specific differences common to many, though not all, fields. Studies included examination of fields such as chess, medicine, physics, bridge, reading, mathematics, music, and art. The areas of interest included amount and structure of knowledge, skill, speed, accuracy, memory, recall, strategies, and reaction time. Studying the qualitative differences between experts and novices can also reveal what changes take place as expertise is acquired and can give clues as to how it is acquired, i.e., is it through practice, experience, or the ability to build accurate mental representations (Lesgold, 1983)? A final study that is of interest to this researcher is one conducted by Lovell (1987), which examined differences in knowledge organization as displayed in problem solving performance by expert and novice instructional developers. Although the research on school teaching and other fields cited above is helpful in suggesting a starting point for the current study, Lovell's work may contribute more due to the similarity in the context of her study to the current study. She examined expert and novice instructional developers who were educated and experienced in 12 developing training for business and industry situations as well as education. Research questions and approaches similar to those used in that study will be used as the basis for this study. Duplicating her approach in a different skill area (training versus instructional development), but similar context (business and industry) may contribute to a body of knowledge that decision-makers, policy planners, training consultants, and academicians may use to develop a more comprehensive and valid approach to conducting training and development activities. The Research Questions The researcher examined three general questions. The first question focused on the content and structure of expert and novice trainers' knowledge. The second question focused on the problem solving strategies of expert and novice trainers. Each of the first two questions were elaborated in a series of sub-questions. The third question examined the consistency of the subjects across tasks. Question One: Do novice and expert trainers differ in the content and structure of their knowledge of training? The sub-questions are: 1a. Do expert and novice trainers differ in the amount of information they generate during selected tasks? 1b. Do expert and novice trainers differ in the type of information they generate during selected tasks? 1c. Do expert and novice trainers differ in the way they categorize information generated during selected tasks? 1d. Do expert and novice trainers differ in the way they organize the information and categories they generate during selected tasks? 13 Question Two: Do expert and novice trainers differ in the strategies they use to solve training problems? The sub-questions are: 2a. Do expert and novice trainers differ in the cues they attend to when solving selected problems? 2b. Do expert and novice trainers differ in the sequence they use to solve selected problems? 2c. Do expert and novice trainers differ in their flexibility in solving problem variations? 2d. Do expert and novice trainers differ in the time it takes to work through each of the selected problems? Question Three: Do experts and novices differ in the consistency of their knowledge content and structure and problem solving strategies across tasks? The questions on the content and structure of knowledge were proposed because the research on experts and novices in every field pinpoints the knowledge base of the expert as a primary source of differences in approaches to problems and tasks. In general, expert knowledge is more detailed and is organized into more abstract categories. The problem solving questions are based on the belief held by researchers in the field of teaching and other areas of expertise, that most complicated tasks are a form of problem solving and that problem solving tasks reveal important differences between experts and novices. The most frequently mentioned differences are those involving the cues experts attend to, the sequence they use to solve problems, and the elapsed time of the problem solving session. These observations about cognitive structure and problem solving strategies are discussed in numerous studies, which will be described in Chapter Two of this dissertation. 14 This study was approached by asking four subjects (two expert and two novice trainers) to respond verbally to the same problems. Three of the problems simulated what trainers must do to prepare to teach a training program. The other two problems were card-sorting tasks. Subject responses were audiotaped and then translated into typed transcripts. The content of these transcripts served as the research data. Analysis consisted of coding the data, generating and asking questions that might lead to answers to the research questions, and then searching the data for relevant clues. The research method and analysis process will be described in more detail in Chapter Three of this dissertation. Scope and Limitations Limitations in the Approach This study is limited in its approach in that it focuses on the differences between expert and novice trainers as revealed during simulations and sorting tasks, rather than what might be observed while they are involved in real-life situations. It is possible that the subjects' performance on the problem solving tasks would be different if they were involved in actual training situations. And it is possible that the sorting tasks are an artificial way to discover the knowledge of trainers. Some may argue that it is not important what trainers can talk about, but what they can actually do. The problems presented to the subjects were designed to be as realistic as possible. They were derived from information gathered from the training literature about typical problems trainers encounter. However, because these problems only simulate reality, the subjects may not have perceived them in the same way they perceive actual teaching problems. In addition, the subjects' responses may not match what they would actually do if these situations were to occur. In actual training situations, the subjects would be acquiring cues and information from 15 various clients, students, peers, and their own managers, rather than from a written case study. Also, they would most likely be able to ask questions about the information before making decisions. During this research study, the subjects were not allowed to ask questions of the researcher during the completion of the tasks in order to maintain consistency across tasks and subjects. The study also refrained from focusing on trainers' attitudes, theories, and beliefs about teaching, except as they may be inferred from the data. There was no attempt to identify the subjects' rationale behind their answers and solutions, except as required to complete the tasks. The examination of trainers' philosophies and beliefs is a different focus than that proposed in this study, which is targeted at trainers' cognitive structure and problem solving strategies, rather than their affective characteristics. Scope of the Data Collection and Analysis The research methods and analysis of this study were similar to that used in other studies of experts and novices in its qualitative approach. The primary goal of the study was to understand the thought processes of a small set of subjects in order to begin to describe how expert and novice trainers may be different. As descriptive research, the results of this study may provide focus for those who wish to study trainers in a more quantitative manner. In Chapter Five, the researcher has translated the results into hypotheses for other researchers interested in this area. Generalization of the results to the larger population of trainers in business and industry was done with great caution. Rather than use an experimental approach to quantify thought processes and activities, the study examined a limited number of subjects in an in-depth manner as they performed several cognitive tasks. The research approach generated a large data set on each subject, and these data sets underwent multiple analyses to identify 16 consistency across tasks. The researcher has found some results similar to those of other expert-novice studies and in this way is contributing to the knowledge base about the nature of expert performance. Definition of Terms The following terms are used throughout the study. The definitions are based on definitions of similar terms in other studies and literature from the training and development field. They are somewhat limited in scope for the purpose of this study. Classroom Training: A classroom event that is directed at improving job performance. There are many other forms of training such as on—the-job training, computer-assisted instruction, and job aids, which will not be considered in this study. Trainer: (or instructor) One who teaches, guides, coaches, and motivates a group of learners and who engages in preparatory and evaluative activities about the instruction and the learners (IBSTPI, 1988). Course: (or workshop, program) A prepared program of information and strategies designed to produce specified outcomes and with the intention of affecting performance in a specified way. The structure of the course can vary from very informal to extremely structured (IBSTPI, 1988). Novice: A trainer who has taught more than one and fewer than five sessions of any course and fewer than three courses in total, has received average to above average ratings on post-class student evaluation forms, and is judged by clients, superiors, and peers to have a high probability of success conducting training. Expert: A trainer who has conducted training in industry for at least five years, with a minimum of ten courses per year or three hundred hours (e.g., l7 twelve three-day classes), has received consistently high ratings on post-class student evaluation forms, and is judged by clients, superiors, and peers to be a highly effective and successful trainer. Verbal protocol: A transcribed record of audiotaped data elicited from subjects who have been instructed to "think aloud" (Lovell, 1987). Cognitive Structure: The content and organization of a person's knowledge and the processes used to encode and retrieve that information. Problem solving: A cognitive process that follows this general sequence: first, a problem situation is perceived, usually as a goal to be reached; then a series of strategies is activated to reach the goal; and finally, the result is evaluated to see if the goal is attained. If the goal is not achieved, the process may repeat until it is. (Meijer & Reimersma, 1986, Chi, Glaser, & Rees, 1982). The Dissertation Plan Chapter One of this dissertation presents a statement of the problem and the purpose of the study. In addition, this section explains the rationale for the study and presents the research questions. The section closes with a description of the scope and limitations of the study and definitions of key terms. Chapter Two of the dissertation contains a review of the literature. Literature relating to the following topics is summarized in this section: cognitive structure, including information processing and knowledge representation; studies of expert- novice problem solving; studies of expert-novice teachers; and the expert-novice perspective in the training field. Chapter Three of the dissertation explains the research approach in detail. This section includes a detailed description of methodology, including background information about the benefits and limitations of using a qualitative approach. 18 Next, the training problems and card sorting tasks will be described. Also included in this section is a description of the procedures and analysis methods to be used. Chapter Four of the dissertation summarizes the results of the study and presents an analysis of the data. Individual findings are reported, then data comparing the experts to the novices are presented. Group data are first reported for Question One, followed by Question Two and Three results. Chapter five presents the conclusions of the study, hypotheses for further study, the limitations of the results, and recommendations. Chapter One Summary Training is becoming a major industry in this country and much of it is being conducted by peOple new to the training profession. Yet not much is known about the skills to be an effective trainer, and what is known focuses on static competencies, not dynamic strategies that can be used in varying situations. Also, the existing competency literature focuses on observable behaviors and not necessarily the cognitive structure and problem solving strategies of the expert trainer. Research in the teacher education and cognitive psychology fields has revealed a new perspective on expert performance that can be used to uncover the differences between expert and novice trainers. No research of this type has been conducted in the training field, and such a study may help to identify the thought processes expert trainers use when solving training problems. A qualitative approach will be used to study these differences to generate information for future studies. In addition, the results of this study may support the researcher's belief that conducting training is a complex skill, developed over a long period of time. This observation may help change the perspective of human resource decision-makers, 19 who frequently provide only minimum training to new instructors before giving them maximum responsibility for improving employee performance. 20 CHAPTER TWO REVIEW OF THE LITERATURE Introduction A review of the training and development literature revealed no research on the differences between expert and novice trainers, nor was there any reference to the cognitive strategies of trainers. Yet, there were several articles about techniques for conducting classroom training and a few that discussed train-the-trainer programs, implying needed competencies. The expert-novice/ cognitive strategies View is being studied increasingly in teacher education and other fields. Clearly it is time to generate some research on this topic for human resource professionals in business and industry. Since this study focuses on a new perspective of trainer competency, i.e., training delivery as a cognitive skill, a broad review of related literature was done. The literature in the field can be clustered into four topic areas: Cognitive structure and information processing; studies of expert-novice problem solving; studies of expert-novice teachers; and the trainer competency literature. Each area contains several sub-topics relevant to this study that will be discussed as appropriate. Cognitive Structure There has always been a debate about how to study and explain human performance. The behaviorist view has led to the specification and quantification of human actions. This perspective has led decision-makers in the training and development field, among others, to focus on observable behaviors as evidence of competency. What has happened as a result, is that long lists of competencies have been developed that can apply to the various situations trainers face. This activity is 21 a "never-ending story," because there could be potentially hundreds of actions a trainer could take, depending on the demands of each situation. In fact, the researcher developed such a list containing over two hundred individual skills an instructor could and should exhibit under a variety of circumstances. These "circumstances" were not described, however. As expected, the trainers and managers who tried to use this list for development and evaluation purposes eventually discarded it, asking for something less "dense," easier to remember, and more prescriptive in nature. Searching for instructor competency using the behaviorist view has not been entirely satisfying. Cognitive psychologists take the view that human performance, although observable, is related to activities occurring in the mind that generate actions. The starting point for these researchers is the structure of the mind--it's components and their functions. These psychologists believe that by understanding the structure of the mind we can study human performance in a more realistic and comprehensive manner. Using this perspective, decision-makers in the human-resource field can begin to view trainer competency as a more complex skill and trainer training as more than modeling and transferring a set of classroom behaviors. The first task for cognitive psychologists is to describe the components of the mind and their functions. One way to do this is to represent the structure of the mind with a model of its functions. A general information processing model found in the literature includes four major components: perception and short term memory, working memory, long term memory, and an executive control function (Gagne, 1965; Greeno, 1980; Shavelson, 1974). One should not assume, however, that these components represent physical sections of the brain. 22 Short Term Memory When we perceive sensory stimuli in the world, these perceptions are stored in short term memory. Short term memory (STM) has been studied extensively and has been found to have limited capacity both in the quantity of information that can be held and the length of time information is stored. It is generally believed that STM can hold only seven, plus or minus two, "chunks" of information at a time (Miller, 1956). It is also believed that these chunks remain in STM for only a few seconds before they must be transferred to working memory or long term memory, or they will be lost (Simon, 1974). However, it is possible to rehearse the content of short term memory to preserve it for as long as the rehearsal continues (Shavelson, 1974). Working Memory Working memory (WM) can store more information than STM for longer periods of time, maybe hours or even days (Shavelson, 1974). In WM "are built internal representations of stimulus objects being learned" (Feigenbaum, 1970). Working memory performs the function of organizing incoming information into hierarchies and networks to facilitate connecting this new information to existing information in long term memory. According to Greeno (1980), working memory performs an important function during problem solving by storing the representation of the problem. This representation includes the givens and the unknowns of the problem and leads to finding a means to move from the givens to the goal state. In more recent literature, working memory as a separate concept appears to be missing from discussions about information processing. Greeno (1980) discusses a "short term sensory storage" that receives information and holds it for a fraction of a second, then "short term memory" identical to that described earlier, and an 23 "intermediate memory" that matches the description of working memory above. Most writers currently view working memory as a component of short term memory (Ericsson, 1985). No matter what it is called, there seems to be an intellectual function that transforms raw incoming data into a relational network to ease entry into long term memory. Long Term Memory Long term memory, by all accounts, is a permanent component of the mind that has unlimited capacity and is well organized. Much has been written about the structure of long term memory and this literature will be discussed in the next segment of this section, Representation of Knowledge in Memory. However, it is generally agreed that long term memory is organized into nodes, networks, or schemata that make it easier to input and retrieve information. Executive Control A final component of the information processing model is the executive control function. This function is a decision making component that helps in the retrieval and storage of information. The control function determines where to retrieve information from and checks to see if the information is appropriate (Collins and Quillian, 1972). It also determines where incoming information should be encoded and manages the process of recalling existing, related knowledge and assigning the new knowledge to it. Representation of Knowledge in Memory Schema Theory One view of the structure of knowledge is that of the schema. Kant (1787), then Bartlett (1932) first described how memories are stored in the form of schemata, 24 but only recently have researchers begun to define more precisely what schemata are. Several current theories share the View that schemata are data structures or procedures that are used to organize the components of specific experience and to expand the representation of an experience...to include components that were not specifically contained in the experience but that are needed to make the representation coherent and complete in some important sense (Greeno, 1980). Schemata are a representation of concepts in the form of a network of interrelationships of the concept's attributes. According to Rumelhart and Ortony (1977), schemata have the following characteristics: 1) they have variables, 2) can embed within one another, 3) represent generic concepts which vary in their levels of abstraction, and 4) represent knowledge rather than definitions. Because schemata have variables, different values or instances of these variables can be recorded. For example, the concept "give" may have three variables: a giver, a gift, and a recipient. These variables become more delineated as new examples ‘of them are presented from the environment to the information processor. By’viewing the components of schemata as variables, one can imagine a rich network of instances describing the schemata as well as the existence of a logical procedure for encoding new information through linkages to specific variables. Since schemata embed, a more limited amount of variables exists than one might imagine. The hierarchical relationship between some concepts allows them to Share variables at different levels of generalization. Rumelhart and Ortony (1977) describe three levels of schemata: dominating schemata, subschemata, and primitives. Schemata also vary in their level of abstraction and are not only C<>Inposed of semantic memory. Tulving (1972) described two forms of memory: eF>isodic memory that encodes knowledge as specific events and semantic memory 25 that allows retrieval from a general base of knowledge and language. Rumelhart and Norman‘s theory goes further to describe memory that is more than episodic or semantic, that is, memory that is not linked to specific events or language. Schemata can be action sequences or plots of stories. They can be interpretations of meaningful events within a particular context, not just static concepts. Schemata are representations of knowledge, not just words. They represent the very meaning behind words and can hold different configurations for different people. This is a critical characteristic of the structure of memory when observing differences and similarities between subjects such as experts and novices. In what ways should the experts and novices be similar in their cognitive structure and in what ways can they be different? This is not an easy question to answer. Stages of Knowledge Acquisition Another View of cognitive structure is presented by Anderson (1982). This View focuses on the acquisition of cognitive skills and thus, their structure in memory. He discusses three stages of skill acquisition: the declarative stage, the knowledge compilation stage, and the procedural stage. He prefers to view the representation of knowledge in this way because this perspective helps to explain more about how learning affects the nature of knowledge over time. Declarative knowledge of a skill is merely a set of facts about the skill encoded during initial learning experiences. Some form of mediation is required during this Stage to help generate the desired behaviors. The knowledge compilation stage is a transitional step that occurs after practice. During this stage the facts are transformed into procedural knowledge, which generates actions directly without the need for interpretation or "thinking about what to do." The procedural stage is baSed on the idea that knowledge continues to develop once a skill is learned. 26 "Tuning" and automaticity occur and the performance of the skill is often speeded up. The difference between declarative knowledge and procedural knowledge is crucial to understanding the theory. In Anderson's theory, declarative knowledge is represented as a propositional network and procedural knowledge is represented as productions. A large component of this theory is the ACT Production System, which consists of productions that operate on declarative information to specify when an action should take place. In a sense, production systems represent decision or rule schemata and declarative information represents concept schemata. The contribution of Anderson's view is that it provides a framework for understanding how knowledge changes over time, how it gets encoded correctly or incorrectly, and how skilled performance is different from novice performance in its execution. Anderson's theory includes a discussion of the changes that occur as skilled behavior develops. When first learning a skill, through interpreting knowledge in declarative form, the learner must rely heavily on working memory. This initial stage of learning, because of its dependency on the limited capacity of working memory, results in errors and a slow-down in processing. However, as learners become more familiar with a subject—matter, they begin to create productions- procedures for completing thinking tasks. This begins the process of knowledge compilation. Compilation is actually two processes: composition, in which a series Of productions or steps to complete a task are collapsed into one production; and Proceduralization, which "builds versions of the productions that no longer require the domain-specific declarative information to be retrieved into working memory" (P. 382). As a result of compilation, thinking, especially problem solving, requires “0 verbal rehearsal of the problem facts, is faster, and avoids a piecemeal approach. Once compilation has occurred, the novice is well on the way to becoming an expert—able to think more quickly and with fewer errors. 27 Three Modes of Learning Another view of how knowledge changes with experience is presented by Rumelhart and Norman (1978). They discuss three modes of learning: Accretion, tuning, and restructuring. Accretion is the common form of everyday learning, and occurs as we accumulate information. Tuning occurs when we modify existing schemata to accommodate new information related to them. Restructuring is not often used, and involves creating new structures to accommodate significantly different information. Restructuring also occurs when we significantly re-organize existing structures into entirely new ones. Restructuring helps us encode and retrieve information easier and quicker and is the result of the accumulation of a "critical mass of information." This theory of knowledge acquisition explains why even experts seem to show improvement after years of experience with a subject. As more information is encoded, more complex structures can be created, which in turn can help the learner interpret and remember more advanced knowledge. Accretion, tuning, and restructuring can be thought of as a spiral form of learning, with each restructuring leading to a higher level of understanding allowing for more accretion and tuning. Two Forms of Information Processing Shiffrin and Schneider (1977) describe two qualitatively different forms of information processing: automatic and controlled. Controlled processing is a serial form of search and retrieval that requires full attention and uses both processing and memory resources. Automatic processing is a parallel process that frees the learner to use other processing systems. Controlled and automatic processing occur Simultaneously and are present in all complex intellectual tasks. According to Shiffrin and Dumais (1981), there are few purely automatic tasks, except for some motor actions like walking. The importance of this theory is its view that different 28 processing tasks require different levels and types of processing resources. This conclusion is based on the assumption that "the information processing system contains capacity limitations that cannot be removed by training or practice" (Shiffrin and Dumais, p. 118). The implications of such a view are that, although skilled performance reflects increased speed and accuracy in the automatic components of a task, even experts are limited by the controlled processes involved. Many studies of information processing expertise involve asking subjects to complete memory or search-and-detection type tasks that are somewhat artificial. In these experiments researchers observe the change in the subjects' performance over time in terms of speed, accuracy, and other quantitative measures. The literature that exposes more qualitative differences between skilled and unskilled performance takes a different approach. This body of research is based on the view that thinking is a form of problem solving and therefore compares the performance of experts and novices on familiar, meaningful, problem solving tasks. Anderson (1982) and Newell (1980) view problem solving as the basic operating approach in all information processing. All of the theories cited above are conducive to this view, but studies that relate specifically to the problem solving approaches of experts and novices reveal more about the specific differences between skilled and unskilled behavior. To summarize the discussion of cognitive structure, the following general points are presented: 0 Cognitive structure consists of short term memory, working memory, long term memory, and executive control. 0 Theories of how knowledge is imbedded in long term memory include: — the representation of knowledge as schemata -—declarative, compilation, and procedural stages of knowledge acquisition 29 —accretion, tuning, and restructuring modes of learning -—automatic and controlled processing Expert-Novice Problem Solving Thinking as Problem Solving The study of expert-novice problem solving skills evolved out of research in the field of artificial intelligence and computer simulation of thinking, as well as the frontier studies of a few cognitive psychologists. Previous to these studies, expertise had been studied as an isolated skill, e.g., the ability of a subject to learn a memory task in the laboratory and recall the information quickly and accurately. Along with other studies of intelligence, for example, comparing genius' performance with low ability subjects, age differences, and different levels of learners, cognitive psychologists concluded in general that intelligence relates to speed of thinking, memory span, and use of cognitive strategies to approach new situations. With more accessible computer technology and the language of programming, researchers began to explore not only what differences exist between high and low level cognitive performance, but why they exist. Computer and cognitive scientists try to replicate intelligent thinking and simulate human problem solving on computers. Psychologists became interested in this area because of the writing of researchers like Newell and Simon, who, using the programming model, asked such questions as, "what are the processes that cause varying levels of Performance?" With the publication of Newell and Simon's (1972) theory of PI‘Oblem solving, a new paradigm for studying information processing evolved. Although philosophers have been studying the mystery of problem solving for th(Nisands of years, Newell and Simon's theory lured cognitive psychologists into Viewing thinking as problem solving rather than simply an accumulation of skills ancl knowledge. 3O Newell and Simon's model of problem solving includes these components: first, a problem is represented in terms of a problem space. During this phase, the problem solver envisions the initial state ("where I am") of the problem and the goal state ("where I want to be"). Then the problem space is broken down into a series of goals and subgoals, that is, steps to transform the initial state into the goal state. If the task is familiar, the problem solver simply recalls past experience to solve the problem. If the task is unfamiliar, then a heuristic search process must be employed to find the strategies that will lead to the goal state. Throughout the process, the problem solver continually evaluates progress through a test procedure to be sure the subgoals selected are helping progress toward the goal state. A final test compares the actual goal state to the desired goal state to check for a match. If there is no match, the process is repeated until there is one. If this is the general problem solving model, then how does it apply to differences between experts and novices? It used to be believed that the differences were mainly quantitative, that experts in a field have better memories and are able to acquire and remember more information. In this case, novices need only learn all of the facts, no matter how long that takes, and they too will be experts. Much of our current education and training approaches seem to operate on this belief. With Miller's (1956) classic research on the "magical number seven," came a breakthrough theory indicating that no matter how skilled a person is, their immediate or short term memory capacity is somewhere between five to nine "Chunks" of information. The difference between novice and expert performance On memory tests relates to the size of the chunk that is memorized. Miller, and later Simon (1974), reported that skilled performers group data into larger chunks and, thus, are able to remember much more. But Simon goes beyond discussing simply the size of the chunk, because, according to both Miller and Simon, chunk size appears to be variable and 31 sometimes quite large. What is important is the way the chunk is formed in short term memory for storage into long term memory. A chunk can be of almost unlimited size, as long as there is a relationship that is meaningful to the learner between the "bits" (Miller) or "vocabulary" (Simon) in a chunk. Simon reports the results of studies of chess players who could remember board positions shown to them only briefly because they were able to chunk the pieces into a familiar pattern. In fact, these studies revealed a remarkable discovery: ...quantitative estimates were made of the 'vocabulary' of familiar chunks in a master's memory. The estimates obtained by several different procedures all fall in the range of 25,000 to 100,000 chunks-- that is, a vocabulary of roughly the same size as the vocabulary of an educated adult in his native language. (p.487) Expert-Novice Seminal Studies The seminal studies of master and novice chess players by researchers like deGroot (1966) and Chase and Simon (1973) inspired a series of expert-novice problem solving studies in a variety of fields. These studies have yielded interesting results--new information about the differences between novices and experts that seems to apply no matter what the field of expertise is. deGroot and his colleagues discovered that masters were able to remember chess positions on the board only if they were meaningful, i.e., actual plays that an expert would see. As a result of these studies, it became apparent that meaningful "Chunking" was critical. If short term memory is limited to five to seven chunks and there are 20 pieces on the board, then the experts must have been able to chunk the pieces into four or five categories or sets. The novices were unable to do this. B0th expert and novice performed the same (poorly) when the positions on the board were random. 32 Chase and Simon examined the chunking process of experts while they encoded and recalled information by looking at the boundaries between chunks and the relations among the components of a chunk. They studied the boundaries by timing the speed with which the chess master placed pieces while reconstructing a board. Longer pauses were interpreted as boundaries between chunks. They studied the content of the chunks by seeing how the masters reproduced a game board layout, that is, what positions they replicated in what chunks and with what accuracy. They were surprised to find that masters apparently recalled not only larger chunks, but more chunks. This seemed to contradict Miller's and others' earlier conclusions. Their hypothesis is that experts chunk the chunks into larger, more abstract categories that go beyond simple chess relationships. For example, a series of patterns may be grouped in hierarchical form by a strategy or game type. These abstract relations are beyond what a novice would perceive. If the results of these studies are applied to the problem solving model, then we can conclude that experts are able to recall more of the sub-goal steps necessary to reach the goal state. But how do they choose the appropriate sub-goals and the order in which they are called into play? In other words, are the experts' search heuristics different from novices? Charness (1979) studied expert and novice bridge players to see if deGroot's and Chase and Simon's conclusions would apply. He found the model does apply to the bridge field, and he also discovered more information about the general problem solving approach of master players. Like earlier studies, his Study revealed that experts had a large "vocabulary" of patterns of cards. But he further concluded that experts actually saw the problem differently and recognized SOlutions more accurately and more quickly than novices. Novices tended to use a hit or miss, trial and error approach and took longer to solve problems. The experts I‘tifpresented the problem accurately, were able to reduce the problem space because of 33 their knowledge of patterns and appropriate strategies, reached a solution quicker, and had fewer errors. Reitman (1976) reported similar results from her earlier study of Go players. She also concluded that experts chunk their "vocabulary" in different ways depending on the problem at hand. In other words, the expert's problem solving strategies, i.e., their heuristics, determined the chunking of the information needed to solve the problem. Expert-Novice Research in Various Fields Problem solving / expert-novice studies have been conducted in other areas with findings similar to those described here. Johnson (1988) studied the troubleshooting performance of expert and novice technicians using a thinking-out- loud approach. He observed that the experts created a more accurate problem space and more quickly reduced the problem space by generating effective hypotheses. The novices generated random hypotheses and used a hit-or-miss approach to the problem. In their studies of baseball experts, another group of researchers discovered that high knowledge individuals recalled more details of a half-inning of play by chunking the events into abstract categories, while the low knowledge individuals tried to recall unrelated details (Voss, Vesonder, and Spilich, 1980; Spilich, Vesonder, Chiese, and Voss, 1979). The high knowledge individuals 1' ecalled information that was more relevant to the game and even elaborated beyond given information, while low knowledge subjects remembered more irrelevant information such as details about the weather. It also appeared that the experts recalled information by using a hierarchical structure-recalling more general information first, then filling in the details. Meijer and Riemersma (1986) studied the problem solving strategies of CI‘ildren as they solved word problems in math. They concluded that the children's 34 difficulty with the problems were due to 1) their inability to represent the problem accurately and completely, 2) their inability to select the appropriate search strategies or heuristics, and 3) their lack of knowledge. The researchers concluded that even when taught heuristics, as students are taught in typical math classes, a lack of subject matter knowledge gained from practice prevents students from constructing an accurate problem space and selecting effective strategies. Simon and Simon (1978) examined the problem solving approaches of a skilled and non-skilled student while solving simple physics problems. They examined the protocols of the subjects, which were constructed using a thinking- out-loud approach. Many of their observations corresponded to what other studies show. For example, the expert solved the problems quicker and with fewer errors. In addition to looking at quantitative differences, however, the researchers used a production system to describe the processes the subjects went through to solve the problems. Both the expert and novice processes were similar in structure. They both used a production that checked the conditions to determine whether the values of the independent or dependent variables were known, found an equation, and filled in the data. However, upon determining the conditions, the expert used a "working forward" approach and the novice used a "working backward" approach to find the solution. That is, the expert simply looked for all of the "givens" in the problem and solved for the unknowns, not really paying attention to the goal of the problem. The novice, surprisingly, appeared more goal-directed, searching for ways to reach the main goal. The researchers concluded that the expert's performance was due to his confidence with the simple problems as a result of extensive practice, and that a more complex problem would induce more of a means-end analysis. They also Concluded that the expert had a "physical intuition" (a higher-level physics skill) that helped him transform the numerical problem into a physical representation to 35 solve it. According to Simon and Simon, the experience, confidence, and physical intuition of the expert contributed to his superior performance. Theories of Expertise Clearly the amount and structure of the knowledge of experts contributes greatly to the effectiveness of their performance. Chi, Glaser, and Rees (1982) summarized the findings from their studies of physics experts and novices into the beginnings of a theory of expertise. They focused on the expert's representation of the problem, because this representation is the foundation for all subsequent actions. They concur with others that the representation of the problem and the resulting heuristic search is different for the expert than the novice, because of the expert's extensive knowledge base. These authors conclude that experts can activate the correct schemata to solve problems. The schemata of the expert and novice contain both declarative and procedural knowledge, which both help to solve problems, but in different ways. Declarative knowledge helps the problem solver construct the problem space. Procedural knowledge specifies the conditions under which a solution should be applied. Experts' schemata contain much more of this procedural knowledge, because of their experience, so they can select the appropriate solution quickly and with little trial-and-error. This theory explains so much about the difference between novices and experts who seem to have equal intelligence, general problem solving or 'thinking' ability, and even an apparently equivalent grasp of the terminology and "school"- type knowledge of a subject. For example, why don't novice physicians, architects, and scientists, who all have extensive schooling in their areas, have the same ability to solve problems as those Who have been working in the field for years? Is there a way to capture and condense the experience experts have so that their procedural knowledge can be transmitted to the new members of their professions? In some 36 way we must capture the entire picture the expert sees, including the relationship between conditions and solutions, not just solutions in isolation. Another question of interest is how, in light of the results of these studies, is the knowledge of the expert organized in memory? If the information used to solve problems is encoded and retrieved in chunks, as many studies report, then is it organized in memory that way? Or is experts' knowledge organized in the form of cognitive maps in which the concepts are connected by their relationships in a network format? Most of the studies report that experts tend to retrieve information in a series of chunks that represent abstract, general categories that apply to the particular step in the problem being solved. Once a chunk is retrieved, the expert automatically recalls the specific information associated with that chunk or reconstructs the details to fit the current situation. This procedure implies a hierarchical organization of knowledge in memory, not a relational one. Chase and Chi (1981) support this view by stating that ...people do not have anywhere near the capacity to conjure up a complete image of a cognitive map. Certainly chess masters cannot imagine a whole chess board at once--they do it pattern by pattern. (p. 132) Their study of map drawing confirmed this belief. Their subjects recreated the layout of a campus, and they did it section by section. They also discovered that the subjects, especially those who were architectural students, organized their drawing by higher level patterns that were conceptual or functional. Because of their experience with spatial environments, the expert subjects were more accurate in the location of buildings and the total layout as well. An interesting study of a planning process revealed even more about how information is organized for retrieval (Hayes-Roth and Hayes-Roth, 1979). In this 37 study, a verbal protocol for planning a series of errands was analyzed. The researchers found that the subject did not sequentially generate a sequence that was organized and logical, as a hierarchical structure would imply. Instead the subject made plans, then revised them as he proceeded through the problem solving process. He first planned the general steps he would go through, but as he examined the specifics of each stop, i.e., what the stop was for, where the stop was, what would follow the stop, he revised his plan. Thus his planning was an iterative process, with each pass being executed at a different level of abstraction. In a sense he generated a problem space, then applied rules or hypotheses to determine the best route from the initial state to the goal state, testing the hypotheses as he proceeded, and revising his plan. This study illustrates very well how thinking that is not automatic can appear 'messy' but is essentially purposeful. The more a novice practices and the more experience he or she gains, the more accessible the appropriate strategies and knowledge to solve a problem or reach a goal become. This study of planning is interesting for another reason. The planning process analyzed here is similar to the kind of thinking teachers and trainers go through to plan a class or make decisions while teaching. And, although they provide relevant findings, many of the other expert-novice studies (in which subjects reproduce physical patterns as in the games of chess, Go, and bridge) involve a more tangible organization of knowledge. Research has been done in the teaching field comparing expert and novice problem solving that explores more abstract thinking processes. Expert-Novice Summary Before turning to research on expert and novice teachers, here is a summary of the differences between experts and novices discussed so far: 0 Experts have a tremendous amount of knowledge of a field 0 Novices' knowledge is less, quantitatively, lacks depth, and is disorganized 38 Experts' knowledge is gained through experience and practice with a variety of situations Experts' knowledge contains much procedural knowledge, which helps them know when to apply specific strategies, resulting in an organized approach with fewer errors. Experts' knowledge is organized in memory in a hierarchical manner, rather than in the form of a cognitive map or network. This facilitates retrieval a chunk at a time. Experts' knowledge is retrieved in chunks, more quickly than novices, more accurately, and with more detail. Chunks can be of almost unlimited size because experts can "chunk within chunks" by finding relationships between concepts. Experts' chunks are abstract categories, relating to functions, principles, rules, and theories of a subject matter. Chunks represent patterns of information. Chunks, once retrieved, can help experts remember or reconstruct details associated with that category. Experts retrieve the appropriate information because they create an accurate representation of the problem or situation. This is because they attend to only the relevant cues. Novices attend to irrelevant cues and form incomplete and inaccurate problem representations. Experts can elaborate more, provide more details, and enhance descriptions with information not found in original data. Experts have as many errors as novices when trying to recall meaningless information or random patterns. When a problem is simple, experts can solve it without using an analytical approach; their actions are automatic and their approach may involve "working-forward." When a problem is complex, experts use a "working- 39 backward" approach, similar to novices, but they are able to create a limited problem space and solve the problem in fewer steps. Novices use a hit-or- miss, trial-and-error approach. ngpering ExpeQ-Novice Teachers A Theory of Teaching Expertise In recent years, researchers in the field of teacher education have been studying teaching from a problem solving perspective and have uncovered findings similar to those in other areas of expertise. Leaders in the field include David Berliner and Gaea Leinhardt and their colleagues. Both researchers have compared expert and novice teachers and have developed theories of teaching as problem solving. But viewing teaching as problem solving has its origins years ago, as reported by Joyce and Hartoonian (1965). They describe a study conducted by Turner and Fattu in 1960 in which the researchers analyzed teaching as a series of intellectual activities including "goal setting, diagnosis, selecting materials, eliciting a set from pupils, and utilizing behavioral outcomes" (p. 421). Joyce and Hartoonian called for a change in the research perspective from observing what teachers do with children to studying the thinking processes that lead to those actions. Leinhardt and Greeno (1986) have constructed a similar theory of teaching as problem solving. They feel that teaching is as complicated an intellectual activity as other areas studied in the expert-novice arena such as solving physics problems, diagnosing illnesses, and playing chess, if not more complicated. Teachers must have a extensive knowledge base, of both the content they are teaching and the skill of teaching, they must be able to plan for a variety of conditions, they must be able to make quick, effective decisions, and they must adapt to a continuously changing environment. As in most complex activities, the expert teacher must be able to 4O perceive new information, revise plans and actions, and evaluate progress toward goals. Berliner's (1988) theory of expertise in teaching is the most comprehensive and is based on the work of brothers Hubert and Stuart Dreyfuss, philosopher and computer scientist, respectively. Berliner views the development of expertise in teaching as consisting of five stages of development: Stage 1. Novice: The novice teacher is learning the rules and basic activities of teaching. The tasks are viewed in isolation, not relative to specific conditions. The novice's behaviors are inflexible and focus on the logical, objective rules to follow. However, the novice seeks real-world experience to practice applying these rules. Stage 2. Advanced beginner: The advanced beginner is applying the rules to a variety of conditions and beginning to see similarities between situations. Labels and categories are starting to build. However, the advanced beginner teacher cannot tell which actions are more important and why, because he or she has not made overt decisions. This is still a reactive rather than proactive stage. Stage 3. Competent: The competent teacher can plan actions based on a knowledge of what is important and not important. At this stage, the teacher can learn from experience more efficiently by attending to only relevant details. But the competent teacher is not "fast, fluid, or flexible," and cannot respond automatically as more advanced performers can. Stage 4. Proficient: The proficient teacher can categorize experiences into higher level schemata and view situations more globally. This abstract view allows the proficient teacher to perceive quickly, intuitively recognizing patterns and connecting information in ways that less expert teachers wouldn't be able to. But the proficient teacher cannot respond instinctively and automatically, still analyzing what to do when an unfamiliar condition arises. 41 Stage 5. Expert: Finally, automaticity in perception and response is achieved. The expert responds without thinking, in a confident, non-analytical way that is almost always effective. In fact it is at this stage that many expert teachers can't articulate what they do or why they do it, they simply act in ways that are second nature. General Findings Berliner (1986, 1988) and his colleagues have conducted numerous studies of expert and novice teachers to confirm what has been found in other fields. Beginning with teacher's perceptions of classroom situations, he studied teacher's interpretations of classroom scenes flashed on a screen for a few seconds. He found that experts' descriptions were rich, detailed, and more accurate than novices, whose descriptions were confused and sparse. He also discovered that when experts and novices viewed videos of classroom situations, the experts were able to remember, describe, and evaluate events quickly and on a more abstract level than novices. Novices gave a sequential, literal description that lacked inferences and conclusions. Experts in Berliner's studies were able to attend to relevant cues and discern the importance of events. After viewing a slide for only a few seconds, experts recognized a pattern of classroom behavior and drew conclusions from it about the type of lesson going on and the children's reactions to it. Novices reported isolated facts that were obvious but irrelevant such as students' clothing, color of hair, and the furniture arrangements. In another study, teachers were asked to predict student performance based on records. Once again expert's perceptions were organized into patterns, only relevant information was attended to, and the teachers were able to predict future performance easily. These conclusions correspond directly to the studies of expert chess players, who can reconstruct board positions 42 after a brief observation and also predict future winning moves because of their ability to organize perceptions into familiar patterns already embedded in memory. Berliner also reports findings that confirm some of the characteristics of expertise described in the information processing and artificial intelligence literature. He discovered that expert teachers use routines that allow them to automatically respond without thinking. He observed experts and novices teaching unfamiliar lessons, and found that the experts employed routines that they used in similar situations. Novices had no such routines and approached the lesson in a random and disjointed manner. Their disorganized approach created classroom management problems, while the experts' routines signalled stability and predictability to their students. Routines, Judgment, and Decision Making Leinhardt and Greeno (1986) also discovered the presence of teaching routines while observing math teachers. During a lesson, the expert teacher in their study employed a variety of routines to collect homework, discover who needed extra help, take attendance, and start and finish the lesson. These routines resulted in a quick, efficient pace, yet provided for the students' needs and accomplished the teacher's goals. The novice, on the other hand, was not able to check attendance or the homework, confused the students about what they were expected to do, and even lost control of the class. This was all due to the fact that she lacked an organized, predictable approach to the lesson. Clark and Yinger (1979), Housner & Griffey (1985), Peterson and Clark (1978), and Ropo (1987) all report similar findings related to teacher routines. Expert teachers appear to have a repertoire of actions that are used based on the conditions at hand. Their planning and their responses to classroom situations reflect their 43 need for routines. They act quickly and effectively during normal conditions or in a "business as usual" (Clark and Peterson, p. 561) manner. If conditions change, experts quickly activate problem solving strategies to identify appropriate actions. Expert teachers are able to make quick and correct judgments and decisions. This is because they are able to perceive situations accurately, relate the relevant cues and patterns to previous experiences, recall effective strategies, and monitor their own performance while executing the strategies. Clark and Yinger (1979) reviewed studies of teacher thinking and found that judgment helps teachers predict student performance and to assess the effectiveness of future instructional activities. It also helps them assess their own performance and revise approaches as needed. One critical component in teacher judgment is the ability to recognize relevant cues. Clark and Yinger report that uncovering exactly what those cues are is sometimes difficult, however. Another aspect of judgment mentioned in the review by Clark and Yinger is "interactive decision-making," which involves decisions teachers make while teaching. The major finding in these studies is that teachers consider alternative strategies only when unusual conditions arise. This confirms the results of studies in physics, where expert problem solvers used a "working backward" approach on simple problems and employed a true problem solving, means-end analysis on difficult or new problems. A study by Ropo (1987) reports an in-depth analysis of teacher thinking during questioning exercises with students. Ropo discusses the number and type of questions asked, as well as the use of teaching strategies he calls scaffolds. Scaffolds are routines similar to guiding and prompting techniques and are used when students misunderstand or are unable to answer a question. Advanced forms of scaffolds require a mastery of the subject matter as well as the ability to call upon the appropriate strategy at the correct time. 44 Experts in Ropo's study used scaffolds twice as much as novices to help students find the correct answers. He describes five types of scaffolds: repetition (repeat the original question), example (give a concrete example), partition (break the question into several sub-questions), alternatives (provide answer alternatives), and additional questions (ask different, but related questions). He found that novices repeated the same question more than experts, who rarely used that approach. Experts used the more challenging approaches (partition, alternatives, and additional question) more than novices, with alternatives, a coaching or socratic approach, being used most frequently. Experts were also able to adapt their approach depending on the student answering the question. This study reveals how decision- making skills and knowledge of students, content, and teaching contribute to the expertise of teaching. Studies of Teacher Competency Lavely et a1. (1986) did an extensive review of studies of expert-novice teachers and found the following results of studies reporting students' ideas about excellent versus incompetent teachers. These findings, although they are summaries of characteristics as perceived by students, and not observations of problem solving strategies, confirm much of what has been discovered in the problem solving studies. According to Lavely's review, characteristics of excellent teachers include knowledge of the subject matter, communication ability, interest in students, encouraging classroom participation, and organization, among others. These characteristics evolve from a grasp of subject matter as well as confidence in teaching routines. One cannot, for example, involve all students without being able to understand what their needs are and then vary the content to meet their needs. Another study in Lavely‘s review reported on the reasons for dismissal of incompetent teachers in California. These included the inability to maintain 45 discipline, impart subject matter, and treat students properly. These characteristics may relate to competence in subject matter, assessing conditions using relevant cues, and using effective teaching routines. Expert-Novice Teachers Summary Summarizing the expert-novice teaching literature, the following general findings are frequently reported: Teaching can be compared to a problem solving process progressing from initial perception of the conditions and goals, selection of appropriate strategies, test of the strategies, revision as necessary, and assessment of achievement of the goal state. The development of teaching expertise occurs in stages similar to the development of expertise in other fields: novice, advanced beginner, competent, proficient, and expert. At first the teacher acts in a random fashion, unable to perceive patterns or act in routine ways. As an expert, the teacher can quickly recognize patterns in every situation and respond in an automatic and effective manner. Expert teachers attend to relevant cues, particularly student characteristics and behaviors, both while planning and while teaching. Expert teachers generate more detail, concrete examples, and elaborate more when planning and describing classroom events. Expert teachers use routines throughout the teaching process related to planning, classroom management, lesson execution, questioning, and evaluation of themselves and their students. Routines provide automatic responses to familiar situations. Even though expert teachers use routines, they are able to act in a flexible manner to adapt their behavior during lessons, based on information 46 gathered during self-assessment and evaluation of students. This interactive decision-making is accurate and quick. 0 Novices lack a repertoire of routines, attend to irrelevant cues, and generate random strategies. They tend to base judgments on general principles or goals of teaching. They are unable to vary their actions based on occurrences in the environment, even to the point of losing control of the classroom. Novices cannot generate detailed descriptions of classroom situations or strategies. The Expert-Novice Perspective in the Training Field Differences and Similarities between Teachers and Trainers The expert-novice teacher literature is quite relevant to the expertise of the industrial and organizational trainer. In general, the roles and tasks of teachers and trainers are similar. Both the teacher and trainer must plan teaching strategies and become familiar with the content and sequence of instruction, they must manage the classroom environment--adapting instruction to individual needs, they must present information clearly and provide opportunities for learner involvement, and they must evaluate the effectiveness of instruction and adapt strategies when necessary. However, there are a few critical differences that may impact the definition of expertise in the training field. Teachers generally prepare their own lessons using a textbook as a reference and their own experience, while trainers often use a course designed previously, and often by someone else. The instructional design of training programs uses a very systematic approach and, if done well, relates all content and activities to learner's performance on the job (Eurich, 1985). This lack of involvement in course design may affect the amount of knowledge trainers have about instructional planning. 47 Teachers are able to stay with the same learners for several months, while trainers usually meet with learners for a few hours, or at most a week or two, then move on to another group. Teachers have more time than trainers to recognize patterns in learner behavior before they must react to them. Trainers must "reach" learners as soon as possible. Out of necessity, trainers may develop these learner assessment skills quicker than teachers. Learner maturity and motivation is also different for teachers and trainers. Teachers who deal with elementary and junior high children generally have an easy time motivating and involving them. In industry, learners are usually paid to come to class, often have negative memories of education, and bring organizational and personal issues into the classroom. Learning how to motivate and manage adult learners becomes a primary development goal for most beginner trainers. A larger difference involves the basic nature and goals of teaching in the two settings. Teachers teach as part of a societal goal to educate all its citizens. The object is to increase and broaden a child's knowledge to prepare him or her for the world of work. Students in training programs are already a part of the world of work and usually attend training to improve job skills. Most training programs result from organization goals to improve productivity, quality, or other business targets. The training class is often part of a larger intervention that may involve reorganization, job redesign, new staffing, new equipment and processes, and new reward and reinforcement systems. The trainer is a member of a larger team and must relate classroom events to the larger environment in which learning takes place. This difference may put more emphasis on trainers' abilities to accurately identify needs, to build organizational support for training, to market training to managers, and to troubleshoot barriers to successful learning and transfer of skills to the job. These skills are not necessarily relevant in a school teaching situation. 48 Therefore, while the expert-novice teacher literature will be helpful in this study and serves as the foundation for the approach, the study may also reveal differences between teachers and trainers in terms of areas of expertise. Literature from the training field would be helpful to point to these differences, but there is a lack of research in this area. This researcher is unaware of any studies comparing expert and novice trainers in business and industry. There are a few articles and books that discuss the importance of hiring and training trainers, critical characteristics of trainers, and the importance of subject matter knowledge. The ASTD (1989) and IBSTPI (1988) instructor competency studies are the most detailed source of information about instructor competencies and some of the details of these studies will be discussed. Training Expertise Involves the Use of Techniques Most literature on training related to teaching presents the advantages and disadvantages of various teaching techniques. Few articles relate the techniques to specific types of students or conditions, except to relate all approaches to adult learning. The primary literature that relates techniques to strategies and conditions is the instructional design literature. There is so little on effective trainers that one would think that anyone could teach effectively in industry if the course were designed well and the right tool kit of techniques was used. In fact, in the most recent review of human resource training and development in the Annual Review of Psychology by Latham (1988), selection, training, and evaluation of trainers is not even mentioned. This omission may reflect either the lack of studies on this topic or a view that trainer competency is unimportant. Designing courses well and employing effective techniques is not enough. The literature on expertise in every field has uncovered the importance of experience and knowledge in helping the performer assess the conditions of a 49 situation and select the appropriate strategies. Although good instructional design can set the stage for a successful training program, in the end it is the trainer who must make decisions about varying the design for the particular audience. The skills of expert trainers, like the skills of experts in every field, differentiate them from the merely competent. It is the experts who produce efficient, effective, and exciting training. The paucity of research in industry in this area may indicate a lack of appreciation for this difference. Trainer Competency Studies Two professional organizations do recognize this difference and have begun a search for the competencies that relate to expertise in training. Both the American Society for Training and Development (ASTD) and the International Board of Standards for Training, Performance, and Instruction (IBSTPI) conducted surveys of experts in the training field to identify key competencies. At this time, however, neither has designated which skills are indicative of the expert. The lists include basic competencies for all trainers. Also, both lists focus on specific teaching skills more than general strategies such as dealing with the reluctant learner, although the IBSTPI list borders on this type of information. There is little, if any, discussion of specific conditions under which certain strategies should be used. The discussion of conditions within these documents is general and not prescriptive. The IBSTPI list of competencies is extensive and meaningful, however, and will be reported here. The competencies include the following: 0 Analyze course materials and learner information 0 Assure preparation of instructional site 0 Establish and maintain instructor credibility 0 Manage the learning environment 0 Demonstrate effective communication skills 50 0 Demonstrate effective presentation skills 0 Demonstrate effective questioning skills and techniques 0 Respond appropriately to learners' needs for clarification and feedback 0 Provide positive reinforcement and motivational incentives 0 Use instructional methods appropriately 0 Use media effectively 0 Evaluate learner performance 0 Evaluate delivery of instruction 0 Report evaluation information This list reflects an appreciation for the need to assess the environment and adapt performance accordingly. Within the standards document, the specific performances under each competency are listed. Many of these performances include making judgments and adapting strategies to meet learners’ needs, although they are stated rather generally. In fact, the authors of the study list judgment as one of three critical functions of a trainer (in addition to being able to perform and being able to provide rationale.) Instructors must be able to judge the adequacy of their own performance and the performance of others. They must be able to monitor and assess that performance; judge the extent to which it is functional and appropriate; select strategies and tactics; monitor the learning climate and environment, and assess the impact of these on learners' progress toward achieving learning objectives. (page 3) This description closely parallels the general problem solving model, making it clear that professionals in the training field recognize the complexity of the teaching process. However, further work needs to be done by IBSTPI and other groups to describe specific training situations and student characteristics to help the novice understand when to apply certain strategies. 51 Another list of standards published by Powers (1984) identifies critical competencies for trainers that focus on interactive skills such as questioning and discussion skills; training techniques that cannot necessarily be controlled by the instructional design of the program. Other authors such as Knowles (1985), Pfeiffer and Jones (1972-1990), and Goad (1982) suggest that teaching in industry depends heavily on the ability of the trainer to involve the audience, maintain interest, and spontaneously adapt the learning experience to the individual learners' needs. Goad presents the trainer's role as a composite of facilitation skills that includes 1) subject matter expert, 2) counselor, 3) leader / motivator, 4) learner, 5) psychologist, 6) manager, 7) and human being. Many trainers in industry are selected because of their content expertise, especially in—house trainers. Hillman (1989) lists other expertise that trainers should have to be effective including the knowledge and skills to understand various instructional methods and put them into practice when needed. According to Hillman, the key to success is using the appropriate method in the appropriate situation. Methods are usually built into training programs during the instructional design phase, but trainers also need to be able to interpret each training situation's unique demands. Some writers discuss training expertise as characteristics or abilities, but do not mention how they are acquired, for example, through experience and practice. Wexley and Latham (1981) list the following core characteristics that effective trainers in industry must have: 0 expertise in the skills and knowledge to be imparted 0 knowledge of, and ability to apply, learning principles and training methods 0 mastery of organization, task, and person analyses to identify training needs 52 0 possession of personal qualities that facilitate learning These authors recommend that trainers be selected who already have these abilities, but also agree that training the trainer and providing various teaching experiences are necessary to develop expertise. In general, then, in the the training literature, there is little or no discussion of the differences between expert and novice trainers, although there are lists of competencies available. Most of the discussion in the training literature about training effectiveness appears to revolve around the assumption that proper instructional design will ensure successful training. The discussions of trainer skill focus heavily on speaking and group facilitation skills, and not necessarily problem solving abilities. While both of these factors are important (instructional design and facilitation skills), they create an incomplete picture of the expertise of trainers. Clearly, research is needed similar to that conducted in the teaching field to uncover the differences between expert and novice trainers in industry. Expertise in Course Design A study “relevant to expertise in the training field and this researcher's study examined instructional design skills (Lovell, 1987). Lovell compared expert instructional designers to novices while they performed several problem solving tasks. The subjects were asked to complete three instructional design case problems and two sorting tasks. The results included the following differences: 0 Experts generated more information, including details, examples, and abstract categories. 0 Experts systematically worked through complex problems while novices used a random approach. 0 Experts worked within a general sequence of steps, working out the details of each step as they went, signaling a chunking process. 53 0 Experts were distracted by specific instructions about how to complete the sorting task, implying that they preferred to develop their own approach based on their experience. 0 Novices appeared to know about instructional development techniques, but were unable to apply them to the complex problems. Lovell's study confirms much of what has been found in the problem solving studies in other areas of expertise. Expert developers have more applied knowledge of the field, use a chunking process to retrieve information, use a systematic approach to solve familiar problems, and are able to apply their knowledge to new situations by selecting appropriate strategies. Lovell also examined the times to complete tasks and the amount of information produced during the problem solving and sorting activities. She found that experts took longer to solve the case problems. During the sorting tasks, experts took slightly longer to begin the task, but there were no differences in the length of time to complete the task. These results concur with other studies that show that experts sometimes take longer to begin tasks because of the time it takes to construct an accurate problem representation during unfamiliar tasks. Chapter Two Summagy This literature review was presented to describe the background for the study. The first part of the chapter focused on cognitive structure and the way information is processed and stored. The information processing view of thinking provides the framework from which all expert-novice studies are built. The second part of the literature review described the range of studies conducted in the problem solving, expert-novice research area. The studies that were cited examined expertise in a variety of fields, including chess, physics, and baseball. A discussion of expert-novice studies in the area of teaching followed the 54 report of studies from other areas. There have been numerous inquiries into teaching as problem solving, as this literature shows. Finally, a review of the literature in the training field concluded the chapter. There is a lack of research in the area of expert-novice trainers in industry, although there is some literature on competencies and instructor standards, which was presented. The next chapter of this dissertation describes the research methodology implemented in the study. 55 CHAPTER THREE DESIGN OF THE STUDY Introduction This design of this study was directed at examining differences in the cognitive structure of expert and novice trainers and their approach to problem solving tasks. This chapter includes the research questions, a discussion of the selection of a qualitative research approach, a description of the data-gathering tools and procedures, and a definition of proposed subjects. The chapter ends with a description of the analysis techniques. This study approach is based on the qualitative research methodology of those who have investigated expert-novice problem solving performance, but primarily on the study conducted by Lovell (1987) for her dissertation. Many of the data— gathering and analysis techniques were modeled after her study. LeCompte and Goetz (1982) describe the need for qualitative researchers to clearly specify their methods so that others can use their data collection and analysis techniques. Duplicating successful methods can begin to build a research tool kit for those Studies that do not use quantitative methods and experimental design. Lovell's design is logical and clear, and her study yielded results that contributed to the body 0f knowledge about the nature of expertise. Therefore, it is an appropriate study to emulate. Wong The researcher examined three general questions. The first question focused on the content and structure of expert and novice trainers' knowledge. The second queStiOn focused on the problem solving strategies of expert and novice trainers. 56 Each of these questions was elaborated in a series of sub-questions. A third general question examined the consistency of the subjects' performance across tasks. Question One: Do novice and expert trainers differ in the content and structure of their knowledge of training? The sub-questions are: 1a. Do expert and novice trainers differ in the amount of information they generate during selected tasks? 1b. Do expert and novice trainers differ in the type of information they generate during selected tasks? 1c. Do expert and novice trainers differ in the way they categorize information generated during selected tasks? 1d. Do expert and novice trainers differ in the way they organize the information and categories they generate during selected tasks? Question Two: Do expert and novice trainers differ in the strategies they use to solve training problems? The sub-questions are: 2a. Do expert and novice trainers differ in the cues they attend to when solving selected problems? 2b. Do expert and novice trainers differ in the sequence they use to solve selected problems? 2c. Do expert and novice trainers differ in their flexibility while solving problem variations? 2d. Do expert and novice trainers differ in the time it takes to work through each of the selected problems? Question Three: Do experts and novices differ in the consistency of their knowledge content and structure and problem solving strategies across tasks? 57 To answer these questions, subjects were asked to verbally respond to three training problems and to two sorting tasks. The training problems related to three challenging instructional situations that a trainer might face. The problems are common in the training field, but each one is different from the others. The sorting task involved defining and categorizing training terms and trainer competencies. Qualitative Methodology Why Use a Qualitative Approach? Why is a qualitative approach to these research questions being used? The most influential factor in this decision was the body of literature on which this study is based. The initial studies of expertise by deGroot (1966) and Chase and Simon (1973), and the later studies that span various fields, all use a form of qualitative research. These researchers most often used a "think aloud" approach, asking their subjects to trace their thoughts as they solved problems. Some used quantitative approaches such as timing performance, measuring length of pauses, and counting errors, but not experimental or quasi-experimental designs. Most used only one or two subjects in each study, who were selected because of their novice or expert status in a field. According to Chi, Glaser, and Rees (1982), the study of expert-novice problem solving focuses on qualitative differences because these are the characteristics that differentiate them the most. Much of what has been learned would not have been discovered using a quantitative approach. The debate about whether experts have better memory capacity or whether they simply structure their knowledge differently would not have been resolved without qualitative research methodology. When examining the patterns that experts see, the structure of their 58 memory, and their methods of retrieval, there is a need to focus on the process and content of thinking rather than simply on the end result. Clark and Yinger (1980) take the position that qualitative research "is not a search for general laws" (p. 4). According to these researchers, descriptive methods further the understanding of specific situations, and are not intended to predict or generalize across a field. Qualitative research is exploratory, while quantitative research is confirming or disconfirming. The results of descriptive research can be used by practitioners to stimulate their thinking rather than prescribe their actions with a high probability of success. Clark and Yinger further argue that the purpose of qualitative studies is not to find variables and hypotheses that can be later used in correlational and experimental designs, although others disagree with this view. Berliner (1986) believes that observational, correlational, and experimental research on teaching has benefited the field in identifying competencies needed by all teachers. What these studies have not revealed, precisely because they operate in ' a controlled environment, are the conditions under which these skills should be applied. He suggests that the study of expertise is in its initial, exploratory phase and that the results of current studies will direct researchers where to look further. In fact, as one reads through the expert-novice literature, it can be seen how researchers have focused on the fruitful areas of others to extend and expand the knowledge base in a meaningful way. LeCompte and Goetz (1982) summarize the reasons for using a qualitative approach. Descriptive research helps to: 0 describe systematically the characteristics of variables of interest 0 generate and refine conceptual categories 0 discover and validate associations among phenomena 0 compare constructs and postulates generated from phenomena in one setting with comparable phenomena in another setting 59 0 generate hypotheses and confirm constructs They also state that the results are meant to be translated for comparable situations, not generalized across an entire population. Qualitative researchers "choose phenomena to study because they are similar or because they differ systematically along particular dimensions" (p.34). Recommendations for Qualitative Researchers Miles and Huberman (1984a) agree that qualitative research is necessary to understand "context-embedded, qualitative, more interpretive" areas of inquiry (p.20). But they believe that a blending of quantitative and qualitative will yield the most fruitful results. They commend the qualitative researchers who use more systematic methods of data collection, storage, and analysis and recommend that procedures be well documented so others can reproduce them. The goal is to "interpret and explain (though not predict) these phenomena, and have confidence that others, using the same tools, would arrive at the same results" (p.23). Still, these writers feel that the weakness in descriptive research is the analysis phase, and that there is an insufficient amount of reliable, valid, minimally agreed-upon analysis procedures for qualitative data. Miles and Huberman suggest several systematic methods for data reduction and analysis. They believe data reduction is necessary in order to make sense of the data collected during the study. They list several ways to reduce and focus the data before, during, and after the data have been collected. Among these suggestions are: 0 Form a conceptual framework for the inquiry 0 Develop research questions 0 Do purposive, rather than random, selection of samples 60 Decide if instrumentation should be minimal to emphasize construct and context validity or pre-planned to emphasize internal validity, generalizability, and manageability of data 0 Use contact summary sheets to record events 0 Code data based on research questions or inductively 0 Use memos during data collection to record reactions, ideas, and insights 0 Display data using tools such as context charts, scatterplots, flow charts, causal networks, and descriptive or explanatory matrices Several of these techniques were used in this study, including the use of research questions and pre—planned instrumentation. Another set of recommendations for qualitative researchers comes from the structured observation studies of managers, popularized by Mintzberg's study of chief executive officers (1968). Structured observation is differentiated from other observational approaches because it involves observation by someone other than the subject, uses a category system, and does not used randomized sampling procedures. According to Martinko and Gardner (1985), structured observation has problems, which can be managed by the use of a variety of techniques. The problems include a high degree of variability in interpretation of the data, lack of documentation of unobservable activity involved in managerial tasks, and the focus on quantitative rather than qualitative aspects of behavior. To minimize these and other problems, Martinko and Gardner suggest researchers use: 0 multiple category coding rather than mutually exclusive coding 0 cross-tabular analysis, analyzing the data from a variety of perspectives 0 complementary observation methods, including unstructured observation 0 methods to capture cognitive processes, such as verbalization while subjects complete tasks 61 0 methods to analyze cognitive process protocols 0 varied coding schemes 0 micro-behavioral analyses 0 large sample sizes (+40) stratification of subjects by performance levels. This dissertation study used a variety of these approaches including cross-tabular analysis, verbal protocol capturing and analysis, varied coding schemes, and stratification of subjects by performance levels. Ensuring Reliability and Validity Reliability and validity are problems with any research approach, but particularly with qualitative research because it is often less systematic and structured than more scientific forms. LeCompte and Goetz recommend that researchers enhance external reliability by 1) carefully identifying the subjects, 2) selecting the appropriate context for observation, 3) precisely specifying analytic constructs and premises, and 4) presenting complete and clear descriptions of data collection and analysis methods. To reduce threats to internal reliability, they suggest using low-inference descriptors of observations and audio equipment for recording data, among other things. In this study, the subjects' responses were recorded on audiotape for later transcripting, coding, and interpretation. LeCompte and Goetz feel that validity may be the qualitative researcher's greatest strength because there is so much documentation of the event and so little translation. They suggest that attention be paid to Campbell and Stanley's (1963) and Cook and Campbell's (1979) recommendations for controlling internal validity, especially observer effects. Observers can change how people respond and what they say by their perceived status and role, and they can affect the results by the way they 62 participate in the tasks performed by the subject. In this study, the observer played a passive role, except when giving instructions. For this reason, Ericsson and Simon (1980), suggest guidelines for using verbal reports as data, since this is the most common format for data collection. They propose a model that predicts how types of verbalization and observer interaction affect the validity of the information verbalized during thinking—out- loud sessions. The best form of verbalization is that which requires the subjects to state thoughts that are already available to them as a requirement of the task being performed. In other words, the course and structure of cognitive processes are not changed because of the necessity to verbalize thoughts. The verbalized thoughts match the internal ones. Going beyond that type of task by asking subjects for their reasons for making certain decisions or to report specific types of information, may alter the actual process of thinking. They recommend using general probes and avoid asking for rationales during or after the observation. In the proposed study, the subjects will be asked to verbalize while performing a problem solving task. They will be given only task instructions, with no further explanations or comments except to clarify, if necessary. Their verbalizations should match their thinking process because they will not be asked to draw conclusions or provide reasons for their approach. . External validity in qualitative research relates to the degree to which a study is comparable and translatable (LeCompte and Goetz, 1982). The most serious threats to external validity are construct effects. Construct validity is "the extent to which abstract terms, generalizations, or meanings are shared across times, settings, and populations" and "how the effects of observed phenomena are construed" (p. 53). The concern here is that researchers may not be able to translate or duplicate the results of studies to other areas. Also, subjects may not perform consistently across tasks because of poor instrumentation or instructions. To control for this, these 63 authors suggest, once again, precisely documenting the data collection and analysis procedures, standardizing and simplifying instructions across tasks, and using triangulation. Triangulation is a method of confirming conclusions by looking at different sources. Miles and Huberman (1984a) suggest triangulating across different data sources, data collection methods, and analysis techniques to verify conclusions. Triangulation was a major consideration in this study, as it was in Lovell's. Data were collected across five problems of two types: three case studies and two sorting tasks. The data were coded in different ways, and were analyzed from a variety of perspectives including differences in amount and type of information, categories of data, cues attended to, and sequence, flexibility, and organization of information. The final research question of this study, regarding consistency of findings across variables (such as problem solving time and sequence) and problems, relates directly to the concern about validity. According to Lovell, The more sources consulted, the more confidence can be placed in the explanation. Validity to the qualitative researcher involves checking and rechecking to be sure that what is first seen is really there. (p. 59) In summary, the data collection and analysis procedures for this study incorporated several of the recommendations found in the literature about qualitative research. There are always barriers to eliciting and accurately describing the cognitive structures and problem solving strategies of experts and novices, but some of these barriers were avoided by careful planning and duplication of others' successful methods. 64 Data Collection Two different types of instruments were used to gather data for this study: three case problems related to challenging training assignments and two card~ sorting tasks involving training terminology. The data were in the form of verbal protocols, which were elicited from the subjects as they completed the tasks. The verbal protocols were audiotaped, then translated into typewritten form, coded, and analyzed. In the case problems, the subjects were asked to describe what problems they anticipated for each situation and how they would handle them. They were also asked how their conception of the problems changed when certain characteristics of each problem were varied. Later the subjects were shown their problems and solutions in the form of transcripts and asked to categorize their problems and solutions. Finally, they were asked to arrange the categorized data into a conceptual structure to illustrate the relationships between ideas. In the sorting tasks, the subjects were asked to define the training tools and techniques, tell what training problems these tools and techniques might solve, and then categorize them. The subjects were also asked to arrange the categorized terms into a conceptual structure. The instruments are described below. Training Problems Three common problems encountered by trainers in industry were constructed. The problems differ in context, audience, and training content, but are identical in structure and type of information provided. The latest industry report by Training Magazine (1989) was used to select the different contexts for the problems. The highest-ranked industries, audiences, and subject matter were considered, but the final choices were selected to provide three different scenarios. 65 Expert trainers should be able to generalize their skills and knowledge across most contexts and types of training problems. Content The training problems content was based primarily on Reigeluth's framework for analyzing instructional-method variables and conditions (1983). His model includes three types of variables that combine to form an instructional program. The first type of variable is instructional conditions, such as subject- matter characteristics and student characteristics. Environmental conditions and constraints can also be included in this category. The second type of variable is instructional methods, which includes subject matter organization and delivery as well as management strategies. The third type of variable is instructional outcomes. These variables are considered by instructional designers during needs analysis as prescriptions for the appropriate type of program design. They are also considered by competent trainers as they prepare to teach, and used as cues about potential problems and barriers to success. Typical needs analysis questions include: 0 What is the organization like? 0 Who is the client? 0 Who is the audience for the training program? 0 Why is the training being requested? 0 What are the expected outcomes of the program? 0 What is the training program content and delivery method? 0 Under what environmental and scheduling conditions will the training be offered? 0 What materials and media will be used? With the answers to these questions, trainers can identify potential problems and plan to avoid or minimize any that might occur. 66 The questions above were answered by the information provided in the three training problems. The information was viewed by the subjects as cues about the challenges that the trainer will face in each case. The general categories of information given in each case and their components are listed below: 0 Audience: the learners in the program Audience characteristics: job function, educational level, previous experience with the program material, attitude about the training 0 Client: the person or persons who commissioned the training 0 The organization: the type of company or organization the learners work for 0 Program background: description of why the program is being offered, what learners are expected to be able to do after the training, the content, the length of the class, the location of the training, and class size 0 Program materials and media: general description of the materials the trainer and learners will use, media to be used, and furniture and other equipment provided The Tasks The training problem tasks consisted of four steps for each problem. The directions for the three problems were identical to maintain consistency across tasks. Different sets of instructions imply different types of tasks and can lead to different sets of findings. First, the subjects imagined themselves to be trainers who have been given a classroom teaching assignment. They were asked to use the information given to identify potential problems that could occur. Then they suggested strategies they would use to prevent or minimize the problems. Second, once they identified all perceived problems and solutions, a few variables in each training problem were changed. The subjects were asked to again identify potential 67 problems and solutions. Third, and at a later date, the subjects viewed a transcript of their problems and solutions, transferred to slips of paper. The subjects were asked to sort the cards into categories. Fourth, the subjects constructed a conceptional framework with the categorized cards. The problems and the directions for the tasks are described in general below and in detail in Appendix A. Problem 1 You will be the trainer of a quality improvement workshop for production operators in a large aircraft parts plant. You are an outside training consultant and have not worked for this company before. The three-day workshop is somewhat technical, and the learners have not had any previous technical coursework like this. They have varying abilities and had a bad experience a year ago with a team- building quality course. There is a concern that some of their supervisors may not "buy into" the program. The training will be conducted in the back of the plant in a training room. You will be given a leader's guide, overheads, videos, and corresponding participant materials. Problem 2 You will be the trainer for a procedures workshop for administrative and office staff for the state Department of Health, Education, and Welfare. You are a member of the personnel staff. A new record-keeping system is being implemented and the staff must learn how to complete the new forms. There have been many programs like this, as the procedures change with every new director. The workshop is four hours long and there is a lot of information to cover. The workshop will take place in a large conference room in the building where the participants work. You will be given a course outline, overheads, and example 68 completed forms. Participants will have c0pies of blank forms and detailed guidelines on how to fill them out. Problem 3 You will be the trainer for an executive training program for officers of a large bank in your area. You are an outside training consultant who works for a prestigious training company. The five day workshop is a strategic planning and management development program that includes some self-analysis activities. The officers have never used a strategic planning process, nor have they been asked to do any self-assessment during training programs. In fact, they are used to selecting off- site seminars of their own choosing. The training will be held in a training room in the building where most of the participants work. You will be given a detailed, scripted leader's guide, which you must follow closely. You will also have overheads and the participants will receive a workbook with worksheets. Relationship of Problems to Research Questions These problems were selected to reveal information about the cognitive structure and processes of trainers while they prepare for teaching. Although the tasks the subjects were asked to perform did not involve spontaneous reaction to participant behavior, which is a major instructor role, they did provide cues about perceived potential problems and their solutions. These problems provided data for all of the research questions. In regards to the questions (major question one) about the content and structure of the subjects' knowledge of training: 0 Data were obtained about the amount and type of information experts and novices generated by analyzing the number and types of problems and solutions the subjects identified. 69 0 The categorization question was answered by having the subjects categorize the problems and solutions. 0 The subjects' organization of the index cards was used to identify their cognitive organization of the concepts. In regards to the questions (major question 2) about the subjects' problem solving strategies: 0 The cues they attended to were identified when they told what information led them to identify each problem. 0 The transcripts provided evidence about the sequence of completing the problem solving tasks. 0 Flexibility was analyzed from the subjects' responses when key variables were changed in each problem. 0 The time it took to begin and finish each problem was measured. The third major question about the differences in consistency of the subjects' knowledge and problem solving strategies was examined by looking at their performance across the three training problem tasks. Further discussion of the analysis procedure can be found at the end of this chapter. Card Sorting Tasks The Tasks Two card-sorting tasks were used to confirm the findings in the case study tasks and to gain more information about the research questions. The format of the two sorting tasks was the same to maintain consistency across them. Subjects were asked to review index cards with training information on them and complete seven sub—tasks for each set of cards: 70 Define the terms or phrases Tell what training problems the techniques might prevent or solve Sort the cards into piles Label the piles 0 Construct a conceptual network with the categories of cards The subjects verbalized their thinking during the sorting tasks, while the observer audiotaped them. Content Several sources were used to design these tasks. The first task, which asks the subjects to define how they would use specific training tools to solve teaching problems, was generated from a list of terms gathered from the training literature. The researcher has developed glossaries for several train-the-trainer courses and a trainer's guide, based on a variety of resources including journals and books listed in the references. This list was augmented by the glossary in the IBSTPI (1988) competency study and a review of the past year's issues of Training Magazine and the Training and Development Journal. The list originally consisted of fifty-six terms. It was reduced by omitting items that appeared to be covered by other items, e.g., "competency-based training" was omitted in favor of "competency tests." Another reduction method was to omit terms that were very general such as "motivation," "learning," and "trainer." Terms were also omitted if they referred to very obvious concepts ("leader's guide"), were not really classroom training techniques ("instructional design"), or were, in the researcher's opinion, too obscure ("results evaluation"). An attempt was made to reduce the list to about thirty terms because of an observation made by Lovell (1987) during pilot tests of her key terms list. She found that "expert subjects could complete sorting of thirty techniques in about an hour 71 without becoming either confused or bored. Novices took considerably longer and became bored" (p. 66). The final list of key training techniques has thirty items. It can be found in Appendix B. The second sorting task involved defining, sorting, and arranging training techniques or competencies. The competencies were taken from the IBSTPI instructor standards study. The study results are organized in this fashion: there are fourteen competencies; these competencies are broken down into performances (usually four to six items per competency); the performances are further broken down into conditions, behavior, and several criteria. For example, a competency is ”Establish and Maintain Instructor Credibility," a corresponding performance is "Demonstrate content expertise," and a criteria for that performance is "Instructional content is mastered." The criteria for the performances became the original list of seventy techniques for this study. Again the goal was to reduce the list to thirty items. The list was reduced in a similar manner to the tools list. Items that didn't make sense out of context from the standards book were removed. There were two items for every performance that related to judging the effectiveness of the rest of the strategies and giving a rationale for the judgment. While I believe that these are important competencies, they seem difficult to interpret in isolation. Items that seemed redundant were combined. Several items were slightly reworded to make them more consistent with the format of the rest of the items. The final list consisted of thirty-four techniques. The list for this sorting task can be found in Appendix C. Relationship of Sorting Tasks to Research Questions Both of these sorting tasks helped uncover the knowledge and understanding experts and novices have about the field of training. They also helped answer the research questions that relate to the subjects' problem solving strategies. Further, 72 the results of these tasks were compared to each other and to the results of the first tasks to identify the consistency of subjects' performance across types of tasks. Specifically, in regards to the first set of research questions about subjects' knowledge content and structure: 0 Data about the amount and type of knowledge was revealed by subjects' definitions and the problems they listed. 0 Categorizing was measured by analyzing the categories the subjects formed with the terms. 0 The subjects' cognitive organization of the terms was identified when they created a conceptual network of the categorized cards. The second major question was answered in the following manner: 0 The sequence with which the subjects selected, categorized, and organized the cards was analyzed. 0 The subjects' flexibility was measured by the number of different problems they identified for each technique. 0 The time to begin and complete the tasks was recorded. The third major research question regarding consistency was analyzed by comparing the results of the two sorting tasks to each other and to the results of the first three problems. Subjects Description of Subjects Four subjects were selected for this study from a list of potential subjects. Two were novice trainers and two were experts. Defining "expert" trainer and "novice" trainer is a difficult task. Anderson (1982) states that "it requires at least 100 hours of 73 learning and practice to acquire any significant cognitive skill to a reasonable degree of proficiency" (p.369). How many hours does it take to learn the network of complex skills required of trainers? Berliner (1986) identifies the experts for his studies through a combination of methods. He looks for teacher reputation among peers, supervisors, parents, and students; classroom observation by independent observers; and performance in laboratory tasks. He always uses novices as those beginning teachers, with some teaching experience in student teaching, who have the highest potential to become excellent teachers. He also warns that one should not confound experience with expertise--some experienced teachers have still not mastered their craft, while some who are inexperienced develop competency quickly. In most of the studies of expert-novice problem solving, such as studies of chess and physics, the definition of expertise rests with the years of practice the subjects have had in a variety of situations. The essential difference between experts and novices is the amount and organization of the knowledge they possess and how they use this knowledge to solve familiar and new problems. The depth and breadth of the knowledge of experts is a direct result of the number and variety of experiences they have had. The definitions for expert and novice in the study being proposed here are based on that distinction. The following definitions were used for the purpose of selecting two novice trainers and two expert trainers: Novice: A trainer who has taught more than one and fewer than five sessions of any course and fewer than three different courses in total, has received average to above average ratings on post-class student evaluation forms, and is judged by clients, superiors, and peers to have a high probability of success conducting training. 74 Expert: A trainer who has conducted training in industry for at least five years, with a minimum of ten courses per year or three hundred hours (e.g., twelve three-day classes), has received consistently high ratings on post- class student evaluation forms, and is judged by clients, superiors, and peers to be a highly effective and successful trainer. The researcher identified a pool of qualified trainers from which to select the subjects. Some were internal trainers in corporations and some were external consultants. Novice trainers that were available included graduate students just entering the training field and newly appointed internal trainers. Novices who previously have been teachers were not considered for the study if their experience was extensive. Once selected the subjects were invited to participate, starting with the first two selections on each list. Those who agreed to participate voluntarily will be selected. Expert 1 has a Bachelor of Science degree in Business Administration and has also complete studies for Ordination in the Catholic Church. He has held various supervisory and management positions outside and inside the training function. He has taught a variety of courses to a variety of functions and levels in a large Midwestern telephone company. He has been a training staff developer and is currently a facilitator and consultant, sometimes doing outside work for his own company. Expert 2 has a Bachelor of Science degree in art education and a Master's degree in Instructional Technology. She has worked as a high school art teacher, but has spent most of her career working for training consulting companies in instructional design and training delivery. She is currently Director of Training for a small import-export company. As such, she serves as corporate trainer as well as manager. 75 Novice 1 has a Bachelor of Arts in Comparative Literature and a Master's degree in television production. She is currently pursuing a doctorate in Organizational Wellness. She has worked in media production for a large, automotive company for most of her career, and is currently the manager of experiential learning. She teaches experiential learning courses that take place in outdoor settings and nonformal classroom situations. Novice 2 has no college degree, although while attending college, he majored in language and business administration. He also has a strong background in math. He also works for a large, but different automotive company, and is currently a quality consultant and trainer. Previously he has held positions as an operator and a supervisor in a plant. Small Sample Size The small sample size is a concern in most, if not all qualitative studies, yet most studies have small samples. This is because the goal is to collect extensive information from one or a few specially selected subjects, rather than a bit of information from a large, random sample. The design of this study resulted in a large quantity of information about each subject through the use of recorded verbal protocols. Presenting problems that were similar in structure showed the consistency of performance across tasks. Also, the data collected were analyzed in several different ways to uncover relationships and patterns that may not be obvious. Studying these questions with a large sample would require limiting and constraining the tasks, yielding much less information. Since this was an exploratory study, not an attempt to predict or prescribe, using a small sample is the most logical approach. 76 Procedures The tasks were first tested with a pilot subject to ensure that the directions were clear. The study subjects then met with the researcher individually to complete the tasks. The tasks were done on separate occasions to avoid the possibility that ideas generated in one task would influence the next. Separating the tasks over several days or weeks might also have prevented the subjects from becoming bored or frustrated. Each subject met with the researcher for a total of approximately eight hours. The distribution of time and tasks per subject is as follows: 0 Hour one: Complete first training problem 0 Hour two: Complete second training problem 0 Hour three: Complete third training problem 0 Hour four: Categorize results from first training problem 0 Hour five: Categorize results from second training problem 0 Hour six: Categorize results from third training problem 0 Hour seven: Complete first sorting task 0 Hour eight: Complete second sorting task The task problems were written, including the instructions. The sessions were audiotaped, and the tapes were transcribed later. Several techniques were used to control bias during the administration of the tasks. First, a "warm-up" discussion unrelated to the tasks was conducted to relax the subjects. Second, a practice problem was completed before the first case and the first sorting task. The subjects were encouraged to ask questions during the practice to ensure that they understood and felt comfortable with the procedures, especially "thinking-out-loud." To minimize influence over the subjects during the tasks, the researcher limited the prompts given during the session to questions and reminders about the task requirements, e.g., "...and what are the problems there?" or "yes, go on..." While the 77 subjects "thought-out-loud" the researcher avoided non-verbal reactions such as smiling, raising eyebrows, or nodding. Most of the time the subjects were completing the tasks the researcher took notes, looking down at a binder to avoid eye contact. The sessions were timed. Two sets of times were recorded for each subject and for each task. First, the length of time to begin the task was recorded. This was measured by starting when the researcher handed the subjects the instructions and materials and ending when the subjects started verbalizing. The second set of times measured the length of time to complete each task. This timing began when the subjects started verbalizing and ended when they said they were finished. The times to begin the sorting and categorizing tasks were inconsequential and are not reported in Chapter Four. Analysis Procedures Organizing the Data The first step in the analysis of the data was to organize the verbal protocols into manageable form. To do this, typed transcripts of the audiotaped protocols were made. The transcripts were typed in the form of paragraphs, with each paragraph representing a complete thought. A change in topic represented the beginning of a new paragraph. The transcriber identified the paragraphs, and then the researcher checked and verified them. The paragraphs were used to code and quantify the data. For both the training problems and the card-sorting tasks, the data were also quantified by counting the number of lines. A line was considered any line (not sentence) in the typed transcript, including those with only one word. For example, this paragraph has five lines. Lines were used during the analysis to determine the amount of information given. 78 The next step in organizing the data was to code the typed protocols for the training cases. The coding was done in two ways. First, the paragraphs were coded using the IBSTPI key competencies. Second, the problems and solutions identified by each subject were coded inductively, based on the nature of the responses by the subjects. The coding procedures are described in more depth during the discussion of the research questions that follows. Whenever coding was done, a reliability check was completed. Miles and Huberman (1984b) suggest using the following formula for checking reliability: number correct Reliability = number correct + number incorrect This procedure is completed by coding the data, then re-coding them and checking for agreement. The "number correct" equals the number of items coded the same way and the "number incorrect" equals the number of items coded differently. The goal is to have the reliability as close 1.00 as possible. The coding was completed in three ways. First, a training expert coded a set of data that had been coded by the researcher. The resulting inter-rater reliability was low (.32). Because of this, the researcher re-coded one set of data three weeks after initial coding. Since the intra-rater reliability was still inadequate (.75), a more elaborate coding procedure was completed. A different expert was trained to code the data using the IBSTPI competencies and then later trained to code the same data using the inductive categories created by the researcher. These categories are described in more detail in Chapter Four. The rater training consisted of a discussion of the meaning of each category and the generation of typical examples that might be elicited by subjects. Once the training was completed, the "confirming rater" coded the first ten randomly selected 79 paragraphs based on the predominant theme in each. The raters (confirming rater and the researcher—the head rater) discussed any discrepancies until they reached at least 80% agreement. More training was included if the confirming rater was still unclear as to the meaning of each category. Once the initial training and rating was completed, the confirming rater coded ten percent of the paragraphs in each case for each subject. These paragraphs were selected randomly. The raters checked for discrepancies after rating each case (four transcripts—one per subject). They discussed the discrepancies until they achieved a minimum of 80% agreement. For the inductive categories coding, the same procedures were used, but the units coded were the problem/ solution sets identified by the subjects. The results of the reliability check can be found in Chapter Foun Analysis Procedures for Each Research Question by Task Training Problems The three training problem tasks were analyzed individually and then the results were compared across tasks for each subject, across groups of subjects (experts compared to experts, novices compared to novices), and between groups of subjects (experts compared to novices). The analysis procedures for each research question are described below. Question One: Do novice and expert trainers differ in the content and structure of their knowledge of training? 1a. Amount of knowledge: To determine the amount of knowledge each subject possesses, the number of lines and paragraphs of uncoded data was counted. Then the number of problems and solutions was counted. For a more meaningful quantification, the data were coded using the IBSTPI key 8O competencies. The fourteen general competencies were used to code the data. The coding procedure consisted of assigning a competency to each paragraph in the transcript. The number of competencies was then counted as well as the number of times each competency appeared. In addition, the number of lines associated with each competency was counted. This provided information about the amount of detail each subject provided. 1b. Type of knowledge: The coded data were analyzed to determine which competencies were present in the protocols and if there were any patterns or similarities within and between subjects. The uncoded data were analyzed to identify types of problems and solutions the subjects identified. Problems and their solutions were then categorized inductively by the researcher. The number of problem / solution sets assigned to each category was counted and patterns were identified. 1c. Categories: The uncoded data were analyzed to determine the number of categories generated, the number of items in each category, and the type of categories generated as reflected in their labels. 1d. Organization: The conceptual networks of the data created by the subjects was documented on paper by the researcher. The organization was analyzed in terms of the nature of the structure (e.g., is it hierarchical, sequential, or relational?), the number of equal categories (placed at the same level), the number of levels, and the number of items under each major level. Question Two: Do expert and novice trainers differ in the strategies they use to solve training problems? 2a. Cues: The uncoded data were analyzed to determine the number of cues identified and the type of cues. The researcher also noted where no cues were identified for particular problems. 81 2b. Sequence: The order in which the problems and solutions were identified was listed from the uncoded transcript. Also, the sequence in which the subjects categorized their problems and solutions was recorded. These sequences were analyzed to determine any similarities and differences between and within subjects and groups. 2c. Flexibility: The subjects' responses to the variations in the problems were analyzed by identifying the number of new problems and solutions they identified and the degree of difference in the solutions. This degree of difference was going to be rated on a zero to five scale, with zero representing "no difference" and five representing "a great deal of difference." This rating system did not work effectively on the study data and the results were not reported. 2d. Time: The times for the subjects to begin the tasks were compared and the times to complete the tasks were compared. Also, the total times (combining both sets of times) was compared. Question Three: Do experts and novices differ in the consistency of their knowledge content and structure and problem solving strategies across tasks? This question was answered by comparing the performance of the two groups across the three training problem tasks on each of the sub-questions for the major research questions. These results were also compared to the results of the sorting tasks for each question. Sorting Tasks The two sorting tasks were analyzed individually and then the results were compared across tasks for each subject, across groups of subjects (experts compared to experts, novices compared to novices), and between groups of subjects (experts to 82 novices), as in the training problem tasks. These protocols were not coded. The analysis procedures for each research question are described below. Question One: Do novice and expert trainers differ in the content and structure of their knowledge of training? 1a. Amount of knowledge: To analyze the amount of knowledge, the number of 1b. 1c. 1d. items that the subject could define was counted. Also, the number of problems identified was counted. To determine the depth of knowledge, the number of paragraphs and lines for each term was counted. Type of knowledge: The terms that were defined were analyzed to determine if they fit some pattern or categorization of knowledge. For example, a subject may recognize and talk at length about motivational issues, but know little about media uses. Categories: The data were analyzed to determine the number of categories generated, the number of items in each category, and the type of categories generated as reflected in their labels. Organization: The conceptual networks of the data created by the subjects was documented on paper by the researcher. The organization was analyzed in terms of the nature of the structure, the number of equal categories, the number of levels, and the number of items under each major level. Question Two: How do expert and novice trainers differ in the strategies they use to solve training problems? 2a. 2b. Cues: Cues could not be identified for the sorting tasks. The sorting tasks were too varied and long to add this additional requirement for the subjects. Sequence: The order in which the terms were selected and defined was listed from the transcript. Also, the sequence of categorizing the terms was 83 documented. These sequences were analyzed to determine any similarities and differences between and within subjects and groups. 2c. Flexibility: The variety of problems that the subjects identified for any training tool or technique were recorded. The problems for a particular term were going to be analyzed by comparing how different they were. This difference rating approach did not effectively work for the data in this study. 2d. Time: The times for the subjects to begin the tasks were compared and the times to complete the tasks were compared. Also, the total times (combining both sets of times) were compared. Question Three: How do experts and novices differ in the consistency of their knowledge content and structure and problem solving strategies across tasks? This question was answered by comparing the performance of the two groups across the two sorting tasks on each of the sub-questions for each major question. In addition, the results for each question from both the training problem tasks and the sorting tasks were compared. Limitations of the Study There were several limitations of the study based primarily on the data collection and analysis procedures. The first set of limitations relates to the fact that the researcher is an expert in the subject matter of the study and that the subjects both knew the researcher and recognized her expertise. These facts could have influenced the study in the following ways: 0 The researcher might have inadvertently reacted positively or negatively to the subjects' responses during the tasks. Interactions were minimized to reduce this risk. 84 0 The subjects may have been self-conscious about their knowledge and may have responded differently than if they had been "thinking-out-loud" in front of a non-expert. To reduce this risk, attempts were made to relax the subjects. Value judgments about their performance were avoided, except to consistently thank and reinforce them at the completion of each task. 0 The researcher's knowledge and beliefs may have influenced the coding of the data since blind coding was not used. Since the researcher was present during the administration of the tasks, blind coding may not have effectively masked the identity of the subjects and that is why it was not used. Although this limitation is an important one, it is unlikely that coding could have been done by a non-expert. Coding by another rater was used to reduce the threat of this limitation on the reliability and validity of the results. Another set of limitations results from the approach of the study and the conclusions drawn from it. 0 The study produced only a "snapshot" of a thinking process. The V" completion of the tasks and the subjects thinking-out-loud" probably represent only a small portion of the problem solving and analysis process that experts and novices complete when faced with real situations. The use of multiple cases and sorting tasks and the triangulation of the data analysis hopefully reduced this threat. 0 Drawing the conclusion that verbal protocols represent the thinking of the subjects may be dangerous. We can only assume that what a person says represents what he or she is thinking. However, the "thinking-out-loud" approach to data collection is generally accepted by researchers in the area of expert-novice problem solving as one of the most direct, albeit limited, 85 methods to "observe" thinking. The third research question regarding consistency of performance across tasks may help to reduce this threat. 0 The task structures may have influenced the responses of the subjects. As reported earlier in this chapter, structuring tasks in descriptive research can aid reliability, but the structure may also influence the subjects' performance. For example, one of the variables analyzed was the sequence of problem solving and the sequence in selecting items to sort and categorize. The subjects' sequence may have been pre—determined by the sequence in which the information was presented. More discussion about this item is provided in the final chapter under Limitations and Their Implications. The final set of limitations relates to the analysis procedures. 0 Using pre-existing categories may be a "force-fit." Once again, the experts in the area of structured observation recommend using categories to increase validity. However, using pre-existing categories that are derived for purposes other than the study at hand may limit the coding procedures. In this study, for example, some the IBSTPI categories did not apply to the data generated, and there were some data that did not appear to fit any of the categories. These conclusions were corroborated by the confirming rater. Because of this, the researcher derived a set of categories inductively by examining the data, and then re-coded the same transcripts using the new categories. This additional coding approach was a better "fit" for the data. Chapter Three Summary This chapter presented an overview of the research design and the analysis procedures used. A discussion of the rationale for using a qualitative approach, and some of the limitations of such an approach were presented. Several techniques 86 were listed to ensure a systematic approach to the study was taken, even though it was a qualitative, not experimental, design. These suggestions helped support the reliability and validity of the study. The instrumentation, subjects, and procedures were described. Then the general approach to analyzing the data was given. It was hoped that if these design and analysis procedures were followed, this study of expert-novice trainers would yield the same level of interesting results as other studies of its type. Finally, the limitations of the data collection and analysis procedures were discussed. Chapter Four presents the analysis of the data. 87 CHAPTER FOUR DATA ANALYSIS Introduction The results of the data analysis are discussed in this chapter. The study focused on three major research questions and several sub-questions. The following questions were examined: Question One: Do novice and expert trainers differ in the content and structure of their knowledge of training? The sub-questions were: 1a. Do expert and novice trainers differ in the amount of information they generate during selected tasks? 1b. Do expert and novice trainers differ in the type of information they generate during selected tasks? 1c. Do expert and novice trainers differ in the way they categorize information generated during selected tasks? 1d. Do expert and novice trainers differ in the way they organize the information and categories they generate during selected tasks? Question Two: Do expert and novice trainers differ in the strategies they use to solve training problems? The sub-questions were: 2a. Do expert and novice trainers differ in the cues they attend to when solving selected problems? 2b. Do expert and novice trainers differ in the sequence they use to solve selected problems? 88 2c. Do expert and novice trainers differ in their flexibility in solving problem variations? 2d. How do expert and novice trainers differ in the time it takes to work through each of the selected problems? Question Three: Do experts and novices differ in the consistency of their knowledge content and structure and problem solving strategies across tasks? This chapter presents the data analysis summary, a brief description of the administration of the tasks, and a presentation of the results. First the individual findings will be presented, then the differences between the novices and experts will be summarized. Data Analysis Summary The case data were analyzed in coded and uncoded form. The data elements that were analyzed for questions one and two are listed in Tables I through 4 below. Question three was examined using the same data but by comparing each subject's performance across all tasks. 89 Table 1 Question One: Content and Structure of Knowledge—Training Cases Question Form of Data la. Amount of knowledge lb. Type of knowledge 1c. Categories 1d. Organization number of lines and paragraphs number of problems & solutions number of IBSTPI competencies competencies present patterns of competencies patterns of problems/ solutions number of categories during sort number of items / category types of categories describe conceptual networks count equal categories count number of levels count number of items / level 90 Table 2 Question One: Content and Structure of Knowledge—Sorting Tasks Question Form of Data 1a. Amount of knowledge 1b. Type of knowledge 1c. Categories 1d. Organization number of items defined number of problems identified number of paragraphs and lines identify patterns in terms defined number of categories number of items / ca tegory types of categories describe conceptual networks count equal categories count number of levels count number of items / level Table ,3 Question Two: Problem Solving Strategies—Training Cases Question Form of Data 2a. Cues number of cues identified type of cues identified note absence of cues 2b. Sequence order of problems and solutions 2c. Flexibility 2d. Time sequence of categorizing number of new problems/ solutions during case variations degree of difference in solutions time to begin task time to complete task total time 91 13.13114. Question Two: Problem Solving Strategies—-Sorting Tasks Question Form of Data 2a. Cues not applicable 2b. Sequence order of terms selected/ defined sequence of categorizing 2c. Flexibility new problems identified during review degree of difference in solutions 2d. Time time to define time to categorize time to create structure Training Cases Administration The case data were collected from the subjects on six separate occasions. During the first three sessions, the subjects read the case studies, then listed the potential problems and solutions they perceived. There were also four variations on each of the cases, which the subjects reviewed and then discussed. Once again, the subjects identified the potential problems and suggested solutions for each variation. On the last three occasions, the subjects reviewed slips of paper containing their suggested training problems and categorized them. They then arranged the categories into a structure on the table. The sessions were audiotaped, then transcribed for data analysis. The subjects were instructed how to complete the task and were given a practice case ("Summer Vacation") to analyze before the actual tasks were begun. During the tasks, the researcher provided no coaching, except to occasionally remind the subjects of the components of the task, e.g., "...and what are the solutions to the problems you see?" 92 The data were analyzed in coded and uncoded form. No data were missing. Most of the analysis involved uncoded data, e.g., number of problems, number of solutions, timing, etc. The IBSTPI competencies were used to code the paragraphs in the transcripts. The researcher assigned a competency (one of fourteen major competencies) to each paragraph. A list of these competencies is found in Appendix C. In addition, the problems and solutions the subjects identified were coded using an inductive approach. The reliability of this coding was checked in three ways. First, another expert (besides the researcher) coded the data. The outside rater's coding was compared with the researcher's and the resultant inter-rater reliability was calculated as .32. The low reliability between raters may be a result of three factors: 0 There was no training or discussion about the meaning of the competencies. 0 Four of the competencies seemed to overlap: #1: assess learners and materials; #4 (manage the learning environment); #9 (provide positive reinforcement and motivational incentives); and #10 (use instructional methods appropriately). 0 There seemed to be two competencies missing: adapting the instructional design before the training begins and ensuring application and transfer opportunities and support. The second method to check reliability was an intra-rater method, in which reliability was calculated by comparing two sets of coded data. The researcher coded one set of data on separate occasions and the reliability was calculated as .75. Because of the low reliabilities found in the first and second methods, a third, more extensive approach to inter-rater reliability was used. A second expert was trained to code the data using the IBSTPI competencies and the inductive categories. 93 This procedure is described in detail in Chapter Three. The resulting inter-rater reliability indexes are listed in Table 5 below: Table 5 Inter-Rater Reliability Check Results Reliability Approach Reliability Index Coding Paragraphs Coding during training .50 (5/10) Coding during training w/ discussion 1.00 (10/10) Agreement / independent .69 (25 / 36) Agreement including second choice .81 (29/36) Agreement after discussion .92 (33/36) Coding Problems/Solutions Agreement / independent .84 (21 / 25) Agreement after discussion .92 (23/ .25) The reliability indexes for the third coding approach were satisfactory for drawing conclusions with confidence about the coded results, and no further checks were completed. fleeting Tasks Administration The sorting tasks were administered on two separate occasions. The first task involved 30 training terms and the second task involved 34 trainer competencies from the IBSTPI Instructor Standards document. On both tasks, the subjects were instructed to define the items, categorize them, and organize them into a structure. They were told to tell the researcher if they did not know the definition of a term, 94 but to make an attempt at defining it anyway. In most cases, the subjects were able to define the terms and explain the competencies. The subjects used the "thinking-out-loud" technique to complete the tasks. They were instructed to first define the items, categorize them, and then structure them. In some cases, the subjects combined the last two activities, simultaneously categorizing and forming a structure. The sessions were audiotaped and transcribed. The researcher drew the structures on a piece of paper to record the configurations. The data were analyzed in an uncoded form to answer the research questions. No data were missing. Individual Findings Expert 1 In most tasks, Expert 1 generated a great deal of information and demonstrated the widest range of ideas. He was particularly fluent when it came to solutions to problems. He expressed a positive attitude throughout the interviews and seemed to view each challenging situation as an opportunity. In fact, he seemed to get more excited when he faced the most difficult training problems. His greatest area of interest was in defining the training terms and competencies. Part of his job requires him to train trainers, and he frequently referred to how what he teaches in his course related to the information the study presented to him. In case one (production operators receiving quality training), Expert 1 identified 29 problems and generated 67 solutions. In case two (record-keeping training for state employees) he identified 21 problems and 47 solutions, and in case three (strategic planning training for bank officers—the most complex case) he identified 26 problems and 69 solutions. Not surprisingly, he also took longer (nine minutes) to read this case before beginning to discuss it. The other subjects only took three to four minutes to read this case. Most of his other times were similar to 95 Expert 2, although frequently a bit longer. During sorting task one (30 key terms), Expert 1 generated 505 lines of information and 76 problems. During sorting task two (34 competencies) he generated 775 lines and 90 problems. The types of knowledge that he generated in terms of problems ranged from concerns about the learner to concerns about the instructor's credibility, depending on the case. His solutions reflected an ability to suggest a range of ideas from instructional design and materials adaptations to motivational and organizational strategies. The competencies he most often referred to and elaborated upon were #1 (analyze learners and materials), #3 (instructor credibility), #4 (manage learning environment), #9 (positive reinforcement/ motivation), and #10 (instructional methods). When identifying problems and solutions during the cases his approach was to identify the general problem related to a particular cue, then state a general solution. He would then test the solution against the data, generating alternative, specific solutions. He frequently identified the advantages and disadvantages of each sub-solution, and he even suggested further problems that the solution would pose. But, through it all, he remained upbeat about the situation, making remarks like, "Piece of cake" and "Oh yeah! we can handle this!" During most categorizing and structuring tasks, Expert I insisted on creating more than one set of categories and more than one structure. He usually started with a general relational set of categories and structures, then moved to one or more sequential structures related to design and delivery of training programs. His category labels changed from task to task, yet they often related to specific instructional considerations such as "preparing the learner" and "discussion approaches." The general format of his structures was consistent—usually a horizontal series of categories with sub-categories as horizontal off-chutes—but the nature of the structure really related to the task content. In terms of the cases, he 96 had very different labels and structures depending on the nature of the case information. For example, for case two he was concerned with pre—planning with the legal and technical staff and his membership on the design and delivery team. There was no mention of a "team" in the case, but his perspective on the problem dictated that he view this as a team approach. In general Expert 1 was confident, was able to elaborate at length during each task, varied his ideas based on each case, and drew from a wealth of experience gained over the years in a variety of training situations. And, although he does not consider himself an instructional designer, he seemed to view the tasks from a macro view, suggesting pre- and post-training strategies as well as strategies that would occur in the classroom. He did not take the case situations as given, but rather explored alternative solutions to the clients' problems. Expert 2 Expert 2 generated about as much information as Expert 1, and in the cases, more. For case three she generated 666 lines, 20 problems, and 58 solutions. She spoke very quickly and confidently. Like Expert 1 she found the cases stimulating and "interesting" rather than problematic. She felt positive about the solutions she suggested, but she did "test" them against the cues found in the cases. Expert 2 was the most creative in terms of solutions and problems. For example, in the bank training case, the officers were accustomed to going to a resort to be trained, so she suggested holding a cocktail party after each daily session and also listed several ways to build team spirit and a sense of identity, even though those issues were not listed in the case as goals of the program. Expert 2 generated about the same number of problems and solutions as Expert 1. For case one she identified 29 problems and 69 solutions. For case two she 97 identified 21 problems and 47 solutions, and for case three, 26 problems and 69 solutions. Both experts generated more solutions than the novices. The types of knowledge that Expert 2 generated varied depending on the case and the sorting task. For case one, she related her discussion to six competencies, for case two she focused on only four competencies, but for case three, she related to six again. Instructional methods (#10) was a strong competency for this subject, referring to it five times in case one, 12 times in case two, and 16 times in case three. Expert 2 has about equal experience in instructional design and training delivery and was able to naturally generate instructional design solutions to training problems, even when the problems were more motivational. For example, when the bank officers were threatened by having to analyze their personal values during training, she suggested several instructional strategies to help them feel more comfortable while still completing the assignment. Expert 2 struggled with the categorizing and structuring tasks. Although it was easy for her to define the terms and competencies, she couldn't make up her mind about categories. Her time for sorting task one was 26 minutes, compared to Expert 1, who took only 16 minutes. The two experts' times for sorting task two were 58 minutes and 7 minutes respectively. Unlike Expert 1, who made a clear decision at the outset to re-categorize several times, Expert 2 searched for mutually exclusive categories that logically included only a distinct group of items. Her frustration built as she tested the category name against the attributes of the items in the category. She was able to describe many attributes for each item and kept moving items around. In fact, she decided during two of the categorizing tasks to "clone this item" and put it in two places, even though she only had been given one slip of paper per item. Interestingly her times for sorting the problems she had identified for the case studies were close to those of Expert 1. The case sorting tasks posed fewer problems, perhaps because they were her own ideas, they were more 98 specific, and they revolved around a case with unique features. It is possible that the case "theme" provided an advance organizer that informed her categories and structure, while the key terms and competency sorting provided no such "scaffolding." Expert 1 may have had less of a struggle because he imposed a sorting rule at the beginning of the task, e.g., "These are are my pre-planning categories, then I'll do a delivery categorization." Why Expert 2 did not impose such rules at the beginning is unclear. The discrepancy may relate more to individual personality differences than expertise. Expert 2 enjoyed participating in the study and expressed her excitement about the new ideas she was gaining from her participation. Her experience in large-scale performance interventions, instructional design using a variety of mediums, and her sensitivity to the teaching task allowed her to draw from a wide range and depth of experience. Both Expert 1 and Expert 2 did not hesitate to expand their problem solving beyond the classroom, questioning the decision-making in the cases that led to the development of the training program and incorporating non-training human performance problems and solutions in their definitions of terms and competencies. For example, in the bank training case, Expert 2 suggested that if the Bank President was not totally committed to the training, then the program should not be run. During the sorting tasks, she suggested how tools such as job aids can be used in lieu of training to improve job performance in a more efficient manner. Her flexibility of thinking was evident throughout her participation in the study. Novice 1 Novice I appeared right at the outset to be more of an intermediate in her range and depth of knowledge, compared to Novice 2. She was confident in identifying problems and solutions, although frequently her solutions were not the 99 best approach for the situations. She had little or no trouble identifying problems and solutions in the cases, and there was little hesitancy while she defined the terms and competencies. However, she did not appear to be as positive and excited as the experts were while completing the tasks. Instead, she seemed to view the cases as "unrealistic" and wrought with "poor planning." She was critical and evaluative of the decision- makers in the cases, while the experts accepted the situation as reality and moved on from there. Novice 1 also approached the cases with a heavy motivational perspective, focusing on how the situation would affect the learner's attitude. In fact, she even "read into" the cases, adding details that were not present, and elaborating on problems and solutions based on those assumptions. For example, in case one, there was a statement about previous quality training around teamwork that had not been well-received. She assumed from this that the participants did not like teamwork and would not want to work in teams during this training. Because of this she suggested a series of unnecessary, alternative grouping arrangements for classroom exercises to avoid conflicts. The experts made no such assumptions about the teamwork issue, focusing rather on the fact that there wasn't any organizational support for the earlier program. Novice 2 spoke slowly and with many pauses. This resulted in a discrepancy between her times and the amount of knowledge she generated, compared to the other subjects. In case three her time was 36 minutes compared to expert times of 48 and 52 minutes. The number of lines generated was 294 compared to 497 and 666. The number of problems she identified was close to that of Expert 2, except on case three, in which she generated only 31 solutions compared to 69 (Expert 1) and 58 (Expert 2). Again, case three was the most complex case, with more subtle and sophisticated potential problems. These results reinforce the caution needed when 100 task time alone is used as a quantitative measure of expertise, and that the collection and analysis of multiple types of data that includes qualitative data is critical. When identifying problems and solutions for the case situations, Novice 2 focused on three competencies: #1 (assess learners and materials), #4 (manage the learning environment), and #9 (positive reinforcement and motivation). She did not suggest instructional methods problems and solutions as much as the experts did. In fact, where the experts often suggested instructional solutions to motivational problems, Novice 1 almost always suggested motivational solutions. And the quality of her solutions was not as high as the experts. For example, in case two, the participants were going to "suffer through" another in a series of new record-keeping programs. Novice 1 suggested spending a great deal of class time explaining why the new system was there (even though it was obviously a whim of the new department director) and convincing the participants to use it. In the experts' opinion, this approach had a high probability of backfiring. The experts suggested avoiding a lengthy discussion about the "why" of the program because there was nothing to be done about it—there was no choice in implementation. Instead, they discussed how to make the learning easier, quicker, and less painful than past programs might have been. Novice 1 works in a strong negative environment, and this culture may influence her perspective on training issues, spotting only the negative aspects of a situation and viewing herself as a helpless worker rather than a powerful decision-maker. Novice 1 did not seem to test solutions as the experts did. She chose a solution or two, then moved on to the next problem. Her flexibility of solutions was not as great as the experts, as reflected in the range of competencies she exhibited, particularly in case three where she used only five competencies compared with seven competencies used by the experts. She did not often suggest other problems that her solutions might generate, and she did not seem to take a 101 macro view of the situations, suggesting alternatives to the program approach, the instructional design, or ways to gain organizational support for the learners. Novice 1 was more passive, accepting the situations as given, and focusing problem solving on the classroom experience alone. Her background in instructional media was evident as she defined, sorted, and structured the training terms. She elaborated more on the media terms and used more specific jargon than with other training terms and competencies. For example, she explained technical aspects of interactive video such as level I and level 11 media. She had little trouble categorizing and structuring the terms and competencies, as reflected in her times. For sorting task one, categorizing and structuring only took 20 minutes as compared with the 58 minutes of Expert 2. Her time for sorting task two was nine minutes to categorize and 58 seconds to structure her categories. The category names and components were not always logical. It appeared that she sometimes put an item in a category simply because she didn't know where else it could go. Novice 1 had a varied and interesting approach to structuring the categories in the sorting tasks. Her structures ranged from a "solar system" view of the training terms with "learner-centered" positioned in the middle, to a sequential/ hierarchical configuration. She formed two relational structures and three sequential structures, with one of those also being hierarchical. Her categories and structures were more sophisticated than those of Novice 2, but not as developed as those of the experts. She appeared more confident than Novice 2 during these tasks and seemed to enjoy them more than any other task she was asked to do. In general, Novice 1 produced much more information than Novice 2, and in many ways was more like the experts. However, it was the quality and quantity of solutions, the competencies she exhibited, and the type of categories she formed that differentiated her from the experts and associated her more with Novice 2. 102 Novice 2 Novice 2 was the least confident of the group. He viewed the training cases as very problematic, although he was not always able to identify a wide range of problems. Most often his approach was to focus on a few key factors and elaborate on them at length. Most of his discussion related to specific, personal experiences he has had related to doing or observing training programs. Novice 2 had a great concern for the learner, both from an ability perspective and a motivational perspective. The motivational issues revolved around learner confidence and self- esteem rather than attitude about the training program, in contrast to Novice 1. This may reflect the situation in which Novice 2 works, which is a heavy industry factory environment, in which the workforce is less educated than in many white collar or high-tech work environments. Novice 2 varied greatly in the amount of knowledge he generated compared to the others, although not necessarily in terms of the number of lines, because his "stories" were not always brief. The difference exists in the number of problems and solutions he identified for the cases. In case one, he generated 15 problems and 20 solutions compared to 29 and 67 for Expert 1 and 25 and 35 for Novice 1. In case three the difference was even greater. In this case, Novice 2 identified 12 problems and 13 solutions, compared to 20 problems and 31 solutions generated by Novice 1. His flexibility in coming up with multiple solutions to a problem was very low. Where experts sometimes had a two-to-one ratio of solutions to problems, Novice 2 had only a one-to-one ratio. He did not test his solutions or identify how a solution might cause another problem as the experts did. Novice 2 performed better in the sorting tasks than in the case studies. He was able to define 25 of the 30 training terms correctly and all of the 34 competencies. His definitions were not always specific, but they were not incorrect. He generated more lines in the sorting tasks than Novice 1, who was more succinct. He generated 103 436 lines compared to 248 lines for Novice 1 in sorting task one and 604 lines compared to 421 for sorting task two. He identified 54 problems compared to her 55 in sorting task one and 66 compared to the 71 of Expert 2. In terms of timing, Novice 2 always had the shortest completion time. It took him 17 minutes, 14 minutes, and 23 minutes to discuss the cases, compared to 24, 25, and 48 minutes for Expert 1. His short task time was most obvious during the categorizing and structuring tasks, which took seven minutes and three minutes for sorting task one and nine minutes and three minutes for sorting task two. In these tasks, the novices' times were close to each other. In terms of competencies present, Novice 2 was not that different from the rest of the subjects. He had 6 competencies, 5 competencies, and 5 competencies present in cases one, two, and three. These figures are close to what the others had. Novice 2 focused on competencies #1 (assess learners and materials) and #9 (positive reinforcement and motivation). He did not suggest instructional methods (competency # 10) as solutions often. In fact in case three, he did not have one instructional method coding as compared with 229 lines for Expert 2 in the same case. It may be that instructional approaches to training problems are the most sophisticated and complex solutions available and not readily accessible to the new trainer who focuses more on motivational and presentation issues. Novice 2 created relational or hierarchical conceptual structures rather than the sequential or process-oriented structures of the experts. He frequently used the words "this category will affect this category and this category" while creating his structure. This may reflect a beginning awareness of the inter-relationships between the groups of training terms and the competencies as he builds his cognitive net. As with Novice 1, some of the categories that Novice 2 formed did not "make sense" to the researcher, and it seemed that he may have been placing items in categories 104 because he wasn't sure where they went. In fact, he would say "I'm not sure why this goes here, but I'll put it here anyway." Despite his lack of confidence, Novice 2 was always cooperative and willing to try all tasks. Although he appeared intimidated while discussing unfamiliar situations ("Oh no, I don't know what I'm supposed to do here, this is too difficult!"), he appeared very confident when discussing topics that were within his realm of professional experience. For example, case one was a situation that was familiar to him. He focused on the learners' educational abilities and skills, rather than on the motivational or attitude issues on which Novice I focused. In fact, in this case, the problems and solutions Expert 2 suggested were more like those of the experts than like those of Novice 1. His solutions focused on instructional and classroom management approaches and, in general, his whole approach to the case was balanced. He did not make the assumptions that Novice 1 made and the solutions he suggested were ones that would work. In contrast, the bank training case boggled his mind, and he felt very frustrated as he tried to work through it. He said, "Well, I really don't know what bank officers are like, but if I guessed, I would say..." In spite of his limited range and depth of knowledge, Novice 2 appears to be building a strong, principle-oriented cognitive structure from those situations he has experienced. Cemparing Experts and Novices: Question One The first question in the study related to the quantity and quality of knowledge that experts and novices generate. The four sub-questions related to a) amount of knowledge, b) type of knowledge, c) categories, and d) organization. Amount of Knowledge 0 In general, experts generated more knowledge than novices, although Novice 1 was sometimes equivalent to the experts. 105 0 There were inconsistent results from the amount of knowledge generated for the sorting tasks. Novice 2 generated more than Novice 1 and almost as much as the experts. This may be due to the fact that he frequently told stories from his experience to define the terms. It is evident from the total number of problems and solutions he identified that he did not generate more qualitative information than Novice 1 even though her transcript was shorter for the sorting tasks. 0 If paragraphs represent separate "chunks" of information, then there were consistent and obvious differences there. Experts have more "different" chunks or categories of information than novices, while novices can elaborate on a particular chunk which with they are familiar. For example, while defining one competency, Novice 2 generated 49 lines (his average number of lines per competency was 24), because he told a story to define the term. 0 The real differences are revealed by the number of solutions the experts generated while solving the cases, compared to the number of problems identified. Although the number of problems was similar for the subjects (except Novice 2), the number of solutions per problem was greater for the experts. Table 6, Table 7, and Table 8 summarize the findings related to amount of knowledge generated during the case study tasks. 106 Amount of Knowledge—Case 1 # lines #paragraphs #problems #solutions Expert 1 269 36 29 67 Expert 2 322 33 25 41 Novice 1 262 25 25 35 Novice 2 I81 18 15 20 T l 7 Amount of Knowledge—Case 2 # lines #paragraphs #problems #solutions Expert 1 432 35 21 47 Expert 2 443 35 26 47 Novice 1 234 27 26 41 Novice 2 181 17 15 14 11112.8 Amount of Knowledge—Case 3 # lines #paragraphs #problems #solutions Expert 1 497 42 26 69 Expert 2 666 49 20 58 Novice 1 294 29 20 31 Novice 2 220 18 12 13 107 Table 9 summarizes the amount of knowledge generated in the sorting tasks. 0 The subjects, except Novice 2, were able to identify all of the terms (task one) and all of the subjects were able to define the competencies to some degree of correctness. The novices lacked confidence in their definitions saying things like, "I'm not really sure what this means, but I guess it could mean..." The definitions the subjects gave were compared to definitions derived from training literature (ASTD, 1989; Goad, 1982; IBSTPI, 1988) and the researcher's experience. This glossary in located in Appendix B. 0 The experts did not tell stories or give specific examples of the use of the terms and competencies. Rather they elaborated on the terms and competencies regarding why they were important or useful, what advantages and disadvantages they have, and when to use them. Novice 1 had succinct, clear, and most often quite accurate definitions for the terms and competencies, but was not as capable of describing the variety of situations in which an item could be used. Novice 2 told stories. Table 9 Amount of Knowledge—Sorting Sorting Task 1 Sorting Task 2 # defined # lines # problems # defined # lines # problems Expert 1 30/30 505 76 34/34 775 90 Expert 2 30/30 456 58 34/34 552 71 Novice 1 30/30 248 55 34/34 421 95 Novice 2 25/ 30 436 54 34/ 34 604 66 108 0 The experts and novices varied only slightly, if at all, in terms of the number of competencies that they discussed. Only eight of the fourteen competencies were identified in the protocols. The missing competencies are: 5. Demonstrate effective communication skills 6. Demonstrate effective presentation skills 7. Demonstrate effective questioning skills 8. Respond to learners' needs for clarification and feedback 13. Evaluate delivery of instruction 14. Report evaluation information 0 The missing competencies may be a result of the nature of the cases and the components included in each. The cases provided information that could be discovered before teaching during the planning stages. 0 Another explanation for the missing competencies is that four of the missing competencies related more to live teaching and are more specific than the others. They are in a sense sub-categories of #10, instructional methods. With a different task, such as viewing a videotape of an actual classroom situation, these competencies might have surfaced. 0 The missing competencies could also result from consistent rater error. However, the outside rater did not identify the presence of these missing competencies. 109 Table 10 Amount of Knowledge—Number of Competencies in Cases Case 1 Case 2 Case 3 Expert 1 7 8 7 Expert 2 6 4 7 Novice 1 6 5 5 Novice 2 6 5 5 Type of Knowledge The second sub-question under Research Question One related to the differences between the experts and novices in terms of type of knowledge generated. Tables 11, 12, and 13 summarize the number of times a competency was present in a protocol of the training cases. Tables 14, 15, and 16 show the number of lines generated for each competency. At first glance, these data illustrate more individual differences between the subjects than distinct differences between the experts and novices. It is clear from these data, that if amount and type of knowledge is one way to categorize expertise, then Novice 1 may be more like the experts than like Novice 2. However, case three clearly separates the experts from the novices. In this more complex case, the experts included seven competencies and elaborated on each at length. The lowest number of lines was for Expert 2 on competency #3 (instructor credibility). The interesting difference between the experts in this case is on competencies #4 (manage learning environment) and #10 (instructional methods). The contrast between the two experts on these items (117 lines for Expert 1 compared to 70 lines for Expert 2 and 27 lines compared to 229 lines) may reflect an individual bias toward a set of strategies in dealing with specific situations. Both sets of 110 strategies are viable, and when used comprehensively can solve the same problems. For example, in the bank problem, one of the greatest challenges to the experts was getting the officers to focus on the workshop rather than their work concerns, since the training was being held on-site. To resolve this, Expert I focused on ways to help the bank executives get their messages, contact their staffs, and still actively participate in the workshop—classroom management approaches. Expert 2 suggested ways to present the information, conduct discussions, and administer exercises to solve the same problem—instructional methods. It is interesting to note that both experts discussed competency #9 at length, consistent with each other. Table 11 Type of Knowledge — Number of Times Competencies Present Case One Competency E1 E2 N1 N2 1. Analyze learners & materials 6 3 7 4 2. Prepare instructional site 5 5 — 3. Instructor credibility — — 2 — 4. Manage learning environment 7 6 2 4 9. Positive reinforce./ motivation 13 13 11 7 10. Instructional methods 3 5 2 1 11. Media 1 1 1 1 12. Evaluate learner performance 1 - — — 111 Table 12 Type of Knowledge — Number of Times Competencies Present Case Two Competency E1 E2 N1 N2 1. Analyze learners & materials 2 8 — 3 2. Prepare instructional site 1 — — — 3. Instructor credibility 4 - 3 1 4. Manage learning environment 5 8 5 4 9. Positive reinforce/motivation 7 7 9 6 10. Instructional methods 14 12 9 3 11. Media 1 — 1 — 12. Evaluate learner performance 1 — — — Table 13 Type of Knowledge — Number of Times Competencies Present Case Three Competency E1 E2 N1 N2 1. Analyze learners & materials 10 5 5 5 2. Prepare instructional site 2 3 1 2 3. Instructor credibility 5 1 — 3 4. Manage learning environment 8 6 7 3 9. Positive reinforce./ motivation 14 16 11 5 10. Instructional methods 2 16 5 - 11. Media 1 2 - — 12. Evaluate learner performance — — — — 112 Dbl—e14 Type of Knowledge — Number of Lines Per Competency Case One Competency E1 E2 N1 N2 1. Analyze learners & materials 48 33 70 38 2. Prepare instructional site 25 44 — 9 3. Instructor credibility — — 13 - 4. Manage learning environment 54 55 16 27 9. Positive reinforce./ motivation 113 133 136 78 10. Instructional methods 20 51 17 13 11. Media 2 6 7 3 12. Evaluate learner performance 7 — — - T l 1 Amount of Knowledge — Number of Lines Per Competency Case Two Competency E1 E2 N1 N2 1. Analyze learners & materials 31 95 27 37 2. Prepare instructional site 8 - — — 3. Instructor credibility 61 — 27 12 4. Manage learning environment 64 103 70 40 9. Positive reinforce./ motivation 84 97 70 72 10. Instructional methods 157 148 73 20 11. Media 15 — 7 - 12. Evaluate learner performance 12 — — — 113 Table 16 Amount of Knowledge — Number of Lines Per Competency Case Three Competency E1 E2 N1 N2 1. Analyze learners & materials 119 66 51 42 2. Prepare instructional site 36 40 19 40 3. Instructor credibility 50 18 — 28 4. Manage learning environment 117 70 99 28 9. Positive reinforce./ motivation 186 215 86 18 10. Instructional methods 27 229 40 — 11. Media 13 28 — — 12. Evaluate learner performance — — — — Additional observations can be made about the data in these tables: 0 Competency #9 (provide positive reinforcement and motivational incentives) was by far the the most commonly discussed competency. This is evident in both the number of times the competency was mentioned and the number of lines associated with it. Competency #10 (instructional methods) was a major subject for both experts in case two. Other frequently and extensively discussed competencies were #1 (analyze course materials and learners) and #4 (manage the learning environment). 0 It is clear that the subjects consistently emphasized particular competencies based on the specific characteristics of each case. For case one, which dealt with training plant operators in quality concepts, the big issue was motivation (competency #9) and assessing learners and 114 materials (competency #1). Instructor credibility was not viewed as a major issue in this case. For case two, teaching state employees how to use a new record system— the fifth in as many years, the issues revolved around managing the learning environment (competency #4), motivation (competency #9), and instructional methods (competency #10). In this case, all four of the subjects agreed that the subject matter was potentially boring and confusing. For case three, training bank executives in strategic planning—their first on-site training program, the issues centered on analyzing learners and materials (competency #1), managing the learning environment (competency #4), and motivation (competency #9). If we look at Novice 2 separately from the rest, some major differences appear. He provided very little information on instructional problems and solutions and focused mainly on motivation and learner and material considerations. He consistently placed last in terms of instructional methods, and in case three, he didn't discuss these strategies at all, while all three others did. Another way to look at type of knowledge is to examine the types of problems and solutions the subjects identified, both in the cases and the sorting tasks. After analyzing the data from the cases and the sorting tasks, the following categories of problems and solutions emerged: Concerns about the trainer Attitude or motivational issues Content, design, or materials issues Classroom management, class size, and timing concerns Learner ability 115 0 Location and logistics 0 Application or transfer issues 0 Media These categories resulted from inductive conclusions based on the data the subjects generated. The researcher coded the problems and solutions that the subjects identified to search for patterns. These results were confirmed by another expert rater. Tables 17, 18, and 19 summarize the number of times each type of problem/ solution set was discussed in cases one, two, and three. Tables 20 and 21 illustrate the number of times each type of problem/ solution was discussed during the sorting tasks. Table 17 Type of Problems/Solutions — Case 1 Problem / Solution Type E1 E2 N 1 N 2 Trainer 2 Attitude / Motivation 8 Content/ Design / Mat. 2 Classrm Mgmt. 3 FWD)!" Learner — Location 7 Application — Media 3 WHO\O\Nt—‘Ul NHUJDJt-limup- I 905199. 116 Table 18 Type of Problems/Solutions — Case 2 Problem / Solution Type E1 E2 N1 N2 1. Trainer 5 2 - — 2. Attitude/ Motivation 4 5 8 4 3. Content/ Design/ Mat. 5 12 13 - 4. Classrm Mgmt. 4 6 3 6 5. Learner — 1 — 5 6. Location - — 2 — 7. Application 1 — 1 1 8. Media 2 — 1 — Table 12 Type of Problems/Solutions — Case 3 Problem / Solution Type E1 E2 N1 N2 1. Trainer 2 — 1 4 2. Attitude/ Motivation 10 7 10 3 3. Content/ Design/ Mat. 3 2 6 — 4. Classrm Mgmt. 6 2 2 1 5. Learner — 3 — 2 6. Location 6 2 1 2 7. Application — - — 1 8. Media 3 4 — - 117 Table 20 Type of Problems/Solutions — Sorting 1 Problem / Solution Type E1 E2 N 1 N2 1. Trainer 13 — 1 4 2. Attitude/ Motivation 20 9 16 10 3. Content/ Design/ Mat. 19 14 2 14 4. Classrm Mgmt. 4 1 2 4 5. Learner 19 22 22 19 6. Location — — — 2 7. Application 2 6 3 1 8. Media 3 6 — - 212111.121 Type of Problems/Solutions — Sorting 2 Problem / Solution Type E1 E2 N1 N2 1. Trainer 17 9 11 4 2. Attitude/ Motivation 21 13 26 12 3. Content/ Design/ Mat. 15 15 9 12 4. Classrm Mgmt. 10 9 15 8 5. Learner 17 18 28 21 6. Location 5 7 3 1 7. Application - 1 4 4 8. Media 2 3 7 3 Some observations about the data on these tables are listed below. 0 Expert I consistently discussed problems and solutions around the trainer. He frequently made remarks such as, "You'll really be in good favor if you pull this one off." His trainer issues revolved around understanding the 118 program content and the audience, and not upsetting the client. This may reflect his role as a corporate trainer in a utility organization. Once again, motivation and attitude are important issues across the four subjects. There seem to be few consistent differences between the two groups regarding this topic. In fact, Expert 1 and Novice 1 seem to be more similar than the two experts in this regard. In terms of content, design, and materials, Expert 1 and Novice 2 appear to be similar. Novice 2 discussed content and design issues more in the sorting tasks than in the case. It appeared that in the cases, he felt he had little power to change the materials and was more concerned about how to make the learner feel comfortable with what was given. Media was an issue for the experts and not the novices in four out of five of the tasks. Although it was not discussed as much as the other competencies, it appears to be a relevant problem that surfaces in varied training situations. The categories of problems and solutions listed here are similar to the IBSTPI competencies, except for #3 (content, design, and materials) and #7 (application). These two categories may reflect missing categories on the IBSTPI standards. Competency #1 deals with analyzing learners and materials, but not the design. Competency #9 discusses motivation and competency #10 talks about adapting instruction, but not in terms of revising the design. None of the competencies talk about pre- and post- training on-the-job support and training transfer, which all four subjects mentioned. Even the two novices mentioned organizational support and transfer issues several times during the study. Except for these missing competencies, however, the results in this analysis tend to correspond to the results presented and discussed earlier. 119 0 Looking at Novice 2 separately from the rest of the subjects, assuming that Novice 1 may be more of an intermediate, there are some differences. Across all of the tasks, classroom management seems to be an area that he didn't discuss in terms of problems and solutions. Also, content, design, and materials is a weaker area, except in the sorting tasks, which included some content, design, and materials terms. 0 It is interesting to note how "learner-focus" emerged in the sorting tasks. This category seemed to be important to both groups, and it was discussed more by the novices in sorting task 2. The sorting tasks were less structured than the case studies and may have provided more opportunities for the subjects' values and beliefs about learner ability and background to surface. Categories The next sub-question under research question one, differences in the knowledge of the experts and novices, related to categories generated by the subjects. The data were generated in two ways. First, the sorting tasks provided two sets of categories (one from each sorting task). Second, the training cases provided three sets of categories (one from each case). The problems the subjects identified during the initial case study discussion were given to them at a later date to be categorized. There were no real differences in the number of categories into which the subjects separated the items. In fact, an interesting observation occurred. Out of 20 Opportunities for sorting, aggregating all subjects, the subjects created seven categories seven times. This may relate to the "magic number seven" phenomena first identified by Miller (1956) and later confirmed as researchers studied the size and quantity of "chunks" of information (Simon, 1974). There were no differences between the subjects in terms of numbers of categories identified, with each subject 120 at some time creating seven categories. The number of categories ranged from three (four occurrences) to eight (two occurrences). The number of categories did not seem to be influenced by the nature of the cases or the sorting terms. The number of items per category did not appear to discriminate between the groups, except in terms of the type of categories, which will be discussed shortly. All of the subjects tended to have categories with the typical number of items around four to seven. Occasionally a category would contain ten items or only one item. There appear to be no patterns here or differences between the groups. The differences lie in the types of categories the subjects formed. The category labels for the five tasks are listed in Tables 22, 23, 24, 25, and 26. A discussion follows the tables. The items are listed in the order they were given by the subject. Table 22 Category Labels — Sorting Task 1 Expert 1 Eyert 2 Novice 1 Novice 2 learner - centered logistics logistics communication logistics learning audience tools teaching aids presentation ID process philosophy training progress discussion elements of training approaches to trng. unique concepts feedback evaluation discussion tools goals 8: objectives media interactive lecture learning activities addit. trng. methods evaluation media learning supports 121 T_alie_2§ Category Labels — Sorting Task 2 Expert 1 Expert 2 Novice 1 Novice 2 learner success phys. arrangement outside the instruction assess. 8: course eval. behaviors of trainer est. rapport w/ lrnr inside the instruction managing course obj. communica. techniques assess the trainer things to both in. 8: out learner needs media / mechanics assess the content teaching techniques evaluation managing content media/instruc. mat'ls. trainer adapting delivery/ presenting profess. acceptability demonstrating managing group T l 24 Category Labels — Case 1 Expert 1 Expert 2 Novice 1 Novice 2 pure consulting location find client expectation attitude design parameters employee attitude needs analysis time learner-centered prep. class size/ timing pre-training mat'ls trainer confidence content considerations material delivery issues/sched. educa. lvl. & ability videos value to company motivation logistics environmental support of organization trainer preparation map to center transfer 122 Table 25 Category Labels — Case 2 Expert 1 Expert 2 Novice 1 Novice 2 prepare self mechanics/ hurdles content iss. to clarify trnr know. of subject prepare mechanics course info. / develop. trnr. motiv. issues attitude prepare learner cognitive pre-iss.: work environ. results program delivery affective instructor prep class size post—1mg. environment time element media training center Table 26 Category Labels — Case 3 Expert 1 Expert 2 Novice 1 Novice 2 pre-plng contract mtg. attitudes aff. training root issues for lrnrs. personality / attitudes meet with legal/ tech. 1mg. gap/trng. probs. concemszcntnt/approa. instructor confidence my part on the team materials core business issues logistics leftovers:incl in design environmental issues trainer credibility script implementation issues An analysis of the category data reveals the following information: 0 The category labels and the subjects' rules for creating them varied greatly from task to task. The task appears to dictate the category labels, especially for the experts and Novice 1. For example, Expert 1 gave a very different set of categories for each case, especially case three. 123 The experts appeared to focus more on instructional design issues regarding the content and the approach. They labeled categories using instructional design terminology such as "design parameters" and "course information and development." The experts appeared to be concerned with the needs of the client, while the novices seemed to be more concerned with the learner. The experts mentioned "professional acceptability" and "pre-planning contract meeting," while the novices' labels included "root issues for learners" and "educational level and ability." The experts' categories were more cohesive as a group for each task. There is a pattern of category labels more so than with the novices. For example, comparing Expert 1 to Novice 1 on the first sorting task, it appears at first glance that the categories are similar, but the novice has categories that could easily mean the same thing, e.g., tools, discussion tools, and learning supports. When the novice did not know what to call a category, the label given was very general and ambiguous, or the item being placed in the category became the category name. For example, Novice 2 did not know what to call a group of instructional techniques, so he labeled them "unique concepts." Novice 2 labeled a category "script," but it included several items relating to teaching techniques. Novices also put items in a category that didn't make sense. For example, Novice 2 put teaching approaches and techniques in a category called "goals and objectives." In general, the experts, and sometimes Novice 1, labeled their categories at a consistent level of abstractness and frequently used training "jargon." Yet their categories were not so general as to be meaningless or not 124 mutually exclusive. Their categories also reflected their sense of control beyond the classroom. Organization The final sub-question under Research Question One relates to the organization of the subjects' knowledge. This question was answered by examining the conceptual structures that the subjects constructed after forming categories for both the training case tasks and the sorting tasks. For the training cases, the subjects created a structure from the problems they identified and then categorized. For the sorting tasks, the structure was created by arranging the training terms and then the competencies into a network on the table. In order to analyze and compare the structures, the researcher created labels for types of structures: hierarchical, sequential, and relational. The following definitions were used for the three types of structures: 0 Hierarchical: a structure that is arranged from more general at the top to more specific as items move downward. Items are grouped by attributes that relate them to the general term under which they are categorized. ° Sequential: a structure that is arranged in left-to-right fashion using a time sequence or chronological process flow. 0 Relational: a structure that appears more "free-form, " resembling a concept map (Rumelhart and Ortony, 1977). Items are arranged in clusters or connected to nodes based on their relationships. Words such as "influences" and "is affected by" are often used to describe the nature of the relationship between items. Items can belong to more than one cluster. The results of the analysis of the structures the subjects constructed is summarized below. 125 Both experts tried to construct a sequential structure on every task. In fact, they attempted to categorize using a sequential or process flow approach, but the problems, terms, and competencies didn't always allow for that. The structures were easier to construct using a process approach, and so the experts' structures were all sequential, with some relational sub- sections. This may reflect their knowledge of the field and their view of training as a process, rather than a set of tools and techniques. The novices' structures varied from task to task, but were mostly relational. Novice 1 had three of five structures as relational, while all of Novice 2's structures were relational. Novice 2 described the categories in his networks "influencing" or "affecting" other sections. Novice 1 described the structure for the training terms as an inner core, a middle section and an outer ring. The inner core was made up of non-negotiable items and the outer core had items that the trainer could choose to use or not. The relational structures may represent the beginnings of a formation of a more sophisticated structure in which a training process and a relational network are evident as in the experts' structures. None of the subjects created a pure hierarchy. Novice 2 created a hierarchy and relational structure, putting some items in a sub-category. The experts' sequential categories were sometimes divided into sub- categories and branches, but not true hierarchies. The experts' structures were "deeper" in terms of number of items in each category, but not necessarily in terms of number of levels. In fact, Novice 2 had the most levels (four—on two tasks) and had three and two levels on the other tasks. Expert 1 had only one level on three tasks and three levels on the other two. Expert 2 had only one level on four tasks, and 126 two levels on the remaining task. Novice 1 had four, one, one, two, and three levels on her structures. 0 The number of items in the major categories and sub-categories and the number of equal categories varied inconsistently across the subjects. Expert 1 had as many as 20 items in one major category and as few as one item in another. These data seem to be generated more from the case material and the training terms and competencies than the subjects‘ existing knowledge base. Also, it was sometimes difficult to tell what the major categories were in the relational structures, because the subjects described the categories as equally important. Comparing Experts and Novices: Question Two The second question in the study related to the problem solving strategies of the experts and novices. The sub-questions related to a) cues, b) sequence, c) flexibility, and (1) time. Cues The cues that led the subjects to identify problems and solutions in the three cases were analyzed. There were no cues for the sorting tasks, as the items themselves served as cues. Table 27 summarizes the number of cues the subjects identified in the cases. A discussion of the data and the types of cues the subjects identified follows the table. 127 11m Strategies—Cues # cues/ case one # cues/ case two # cues/ case two Expert 1 22 24 38 Expert 2 23 30 39 Novice 1 21 24 23 Novice 2 12 11 10 0 The number of cues identified for the first two cases was similar for Expert 1, Expert 2, and Novice 1. However, case three, which was more complex and subtle, shows a marked difference between the experts and Novice 1. Novice 2 differed greatly from the other subjects on all three cases. This may highlight an important difference. It is possible that the experts, and in this case an intermediate, are able to "see" things in an environment that the novice cannot see, nor use. 0 It is interesting to note that the number of cues was greater for the experts than the novices, but the number of problems identified was similar. Keeping in mind that the number of solutions was greater for the experts than the novices, and that the experts tended to use an ends-mean analysis and test of the solutions, this may make sense. It is the cues against which the experts test their solutions. 0 In most cases the subjects identified cues for the problems they discussed. However, on a few occasions, both novices "jumped to conclusions" about the cases that were unsubstantiated by the case facts. In the operator training program, Novice 1 was convinced that the participants would not like team approaches in the training session. This was not supported by the information. Since Novice 2 seemed concerned with learner needs 128 and abilities, he elaborated at length about the operators not being able to use calculators and the math materials. While there was a cue about the need to use calculators, he assumed the learners would not be able to use them and identified solutions to that problem. In this situation, the experts suggested that a sensitive approach be used to identify those with problems and then deal with the situation as it arises. They did not focus on specific strategies for that problem. 0 The types of cues identified was consistent for all subjects. The cues for the experts corresponded to the key information that the author of the cases included. The experts did not overlook many cues. Their range of cues spanned all of the variables included in the cases from learner information, to logistics, to materials, to organizational support. The novices range of cues corresponded for the most part to the case information. What they missed were the details around each major set of cues. Novice 2, especially, would read a new section of the case, pick up a cue, elaborate on it, then proceed to the next section. Novice 1 looked at depth for some cues, particularly those around learner attitude and organizational support issues, but overlooked details on others such as leaders guide descriptions and logistics. Sequence Sequence, the second sub-question under Research Question Two, did not become a major source of data in terms of the order of the sequence of items identified or selected. The data collected were influenced by the nature and structure of the tasks. The subjects identified problems and solutions in the sequence the case information was presented, for the most part. Occasionally, especially with the experts, a topic was reviewed when a later problem linked to the 129 earlier one. But, generally, the problems were identified in the order in which they were presented. The terms and competencies were presented to the subjects in a random pile of index cards. Again, the subjects simply picked up the next card to define it and to categorize it. The categorizing task for the cases had similar results. The subjects selected the next problem, read it, and put the slip of paper on the table to begin categorizing. There were some interesting findings related to the sequence of categorizing, however. 0 During the categorizing tasks, all of the subjects moved items around several times to form different groups, until they were satisfied with their categories. The experts always tried to categorize in a sequential configuration, but sometimes had trouble with the traditional models they had in their mind of training design and delivery. They would then attempt a relational structure, creating categories, then testing the items against the category label. If the category label and the items included were‘n't cohesive, then the subject would re-arrange the categories again. For Expert 2, this sequence occurred during every sorting task. Her times reflect the frustration and confusion she felt about these tasks. The interesting observation about the experts is that in the end, most of their structures ended up being sequential. Through re-arranging and creative labeling they were able to fit the problem space to their goal. 0 The novices began categorizing by identifying similar words on the cards and slips of paper. If the term started with "manage," then it went in the "manage" category. However, they found that this categorization didn't work, so they chose categories by selecting a familiar item and building other items around that one. Sometimes they couldn't think of a title for 130 the category that would include several diverse items, so they created a very general term such as "tools." Novice 2 rarely tested his items and categories once they were formed. Novice I tested sometimes, especially when asked to read her category items at the end of the task. But neither tested the categories to the extent that the experts did. In general the experts used a deductive approach, identifying a category label from their experience, then fitting the terms into it, while the novices used an inductive approach, reading off items until they found those that seemed to belong together. Flexibility Flexibility is the third sub-question under Research Question Two. Flexibility relates to the variety of information the subjects provided. One source of flexibility data was the number of new problems identified during the case variations. The other source was the degree of difference between the new problems and the original problems. The researcher tried to rate the degree of difference, but was unable to do so. It was difficult to determine the degree of difference when the problems and solutions related to the same variables in a case or the same items in a sorting task. For example, if the variation in the case study related to the materials or the content, then the subject would suggest new strategies, but still they were still related to the problem of materials. As a result, there appeared to be only a small degree of difference in most situations. It appears the pertinent data relate to the number and type of different problems and solutions—the range, rather than the degree of difference. As a result, the data were analyzed in that manner. The number of new problems and solutions identified in the three cases when the variations were presented are listed in Table 28. 131 Table 28 Flexibility — Number of New Problems/Solutions Case 1 Case 2 Case 3 # Prob # Solu # Prob # Solu #Prob # Solu Expert 1 7 24 9 16 10 29 Expert 2 10 18 14 26 7 25 Novice 1 10 16 9 15 8 16 Novice 2 12 11 9 8 8 9 It's clear that once again, the differences lie in the number of solutions, rather than the number of new problems. Except in case one for Expert 2 and Novice 1, and in case two for Expert 1 and Novice 1, the experts identified many more solutions than the novices. The range of solutions was greater for the experts as well. For example, on variation three of case one, the problem was that the operators had never had a training program before. Expert 1 suggested to: ——build in excitement (instructional approach) —--have some fun with it (motivational approach) —talk to the supervisors (organizational support) —have a high concern for people (trainer behavior) —bring in their experiences (learner involvement) —listen for their experiences (trainer behavior) —go for buy in (motivational approach) Expert 2 suggested to: —orient the participants to what training is all about (pre—learning) —emphasize the need-to-know aspect (motivation) —introduce the concept of quality (content) 132 —start from the beginning, where they are (learner ability) ——help them make a plan so it can work (learner involvement) Novice 1 suggested to: ——create a positive atmosphere (motivational) —encourage questions (instructional approach) —show how different experiences help (learner involvement) —make things fun (motivational) —share the objectives (instructional approach) —show how it's company-wide (motivational) Novice 2 suggested to: —build in a long introduction to justify the program (motivational) —plan to teach a longer time (classroom management) The experts were able to generate ideas that went beyond the learner or the trainer, into areas such as content and supervisor support. Time The final sub—question under Research Question Two is that of times to complete tasks. Tables 29, 30, and 31 summarize the finding regarding task time. 0 Times to begin the tasks, complete the tasks, and total times for the cases show a marked difference between Novice 2 and the rest of the subjects. Novice 1 spoke longer than Expert 1 and Expert 2 in case one and took almost the same amount of time on case two. In case three, the experts exceeded the novices noticeably. 0 An important consideration here is rate of speech and time spent talking (versus pauses) during the elapsed time. Novice 1 was a very slow speaker and paused at length frequently during all tasks. Expert 2 was a very quick speaker. Expert 1 was a moderately fast-paced speaker. This bears out if 133 you compare the amount of information produced by Expert 1 and Expert 2 in approximately the same span of time as Novice 1 (e.g., case three: Expert 1 = 497 lines, Expert 2 = 666 lines, Novice 1 = 294 lines). There was very little difference in the amount of time the subjects took to discuss the variations on the case studies (not shown). Once again, Novice 1 took more time than anyone else to discuss the variations, due to her slow rate of speech and frequent pauses. There were some differences in the time to define, categorize, and structure the key terms (sorting task 1) and competencies (sorting task 2). The two novices took less time to categorize and structure the terms and competencies, although Expert 1 took a much shorter time than Expert 2. On the cases, all of the subjects took less time categorizing. In most situations, there were fewer items to categorize. Also, the case problems that were being categorized were more specific to a situation than the training terms and competencies. The subjects were familiar with the items, since they had generated them during their analysis of the case studies. 134 Table 29 Time—Cases Time to Begin Time to Discuss Total Time Case 1 Case 2 Case 3 Case 1 Case 2 Case 3 Case 1 Case 2 Case 3 E 1 4 * 5 9 20 30 38 24 25 48 E 2 3 2 4 27 34 48 30 37 52 N 1 2 3 3 35 32 33 37 35 36 N 2 1 2 3 15 13 20 17 14 23 *number of minutes Table 30 Time—Cases Categorizing Time to Categorize Time to Structure Total Time Case 1 Case 2 Case 3 Case 1 Case 2 Case 3 Case 1 Case 2 Case 3 E 1 18 14 22 ——* 25 13 18 39 35 E 2 11 18 12 8 9 9 19 27 21 N1 13 14 14 ——* ——* —-* 13 14 14 N 2 8 6 6 4 2 — - * 1 2 8 6 "Categorizing and structure occurred simultaneously T11 Time—Sorting Sorting Task 1 Sorting Task 2 Define Categorize Structure Define Categorize Structure E 1 43 16 -— 68 17 -- E 2 42 26 32 49 58 7 N 1 28 12 8 50 9 1 N 2 37 7 3 38 9 3 135 Qamparing Experts to Novices: Question Three Research Question Three examined the consistency of knowledge content and structure and problem solving strategies across the three cases and the sorting tasks. In some areas there were consistent differences between the two groups. However, Novice I appeared at times to be more like the experts than like Novice 2. The consistency results are summarized below. 0 Amount of knowledge: Experts consistently generated more knowledge than novices. The difference was particularly noticeable in the number of solutions they generated for the problems listed. Experts elaborated more on specific topics and provided a greater range of topics. 0 Type of knowledge: The experts discussed a greater range of problems and solutions. They included instructional and classroom management strategies in addition to the motivational and learner needs analysis strategies of the novices. Experts tended to think beyond the training classroom and questioned the parameters around the situation. They did not accept the status quo, but felt empowered to suggest changes in training design and delivery. The novices felt helpless and blamed the difficult situations on the clients rather than identifying ways to try to meet their needs given the constraints. During the sorting tasks, the experts' knowledge tended to be more abstract in the definition, yet very detailed in the problems given. The novices' definitions were either brief and too general or too specific, including stories and examples. 0 Categories: Experts' categories were logical and usually related to a training process. The categories were formed using a deductive approach based on a preconceived model of teaching. The experts adapted the category labels to the specific task at hand. Novices' categories were in some cases too general to be meaningful and mutually exclusive. In other 136 cases the category consisted of one item and that item became the name of the category—very specific. Novices used an inductive approach to categorization. They did not appear to be using a model to form categories. Novice 1 sometimes provided categories that might fit a model, but then included categories that were on a different level of abstraction or were formed by some other decision rule. Organization: Experts' structures were mostly sequential, relational structures. The categories were arranged in sequence of the training process. Sub-categories and structures were formed. Novices' structures were relational, basing the relationships on categories that influenced or affected others. For Novice 1 there was some attempt to create a sequential structure for the two sorting tasks. There was inconsistency in the number of levels and number of items in each level for all subjects. Cues: Experts used most, if not all, of the cues in the cases to identify problems and solutions. They checked their solutions against the cues. They reviewed the cues to identify additional problems and solutions. NoviCes tended to notice one or two cues in a paragraph of information, thenidentify problems and solutions, without testing them against the case data. They then moved on to the next set of cues, skipping further details around a particular variable, e.g., materials and media. Sequence: All subjects followed the sequence in which the information was presented to complete the tasks. The sequence of categorization became more variable for the experts, who re—arranged categories several times before settling on a final form. Novices tended to categorize as the terms and problems were given. Flexibility: Experts demonstrated more flexibility across tasks. They were able to generate more ideas and vary them across a range of competencies 137 and themes. They viewed challenges as opportunities and adapted their knowledge to fit the situation. They used different types of approaches depending on the situation. The novices had a narrower range of approaches, focusing frequently on learner attitude and ability concerns and suggesting motivational strategies to solve most problems. 0 Time: Time was inconsistent across the groups of subjects, except in the categorizing task. Novice 1 took more time than the experts on different tasks, most likely because of her slow rate of speech. The categorizing tasks took much longer for experts as they re-formed categories and tested their organization rules. Chapter Four Summary This chapter focused on the analysis of the study results. It began with a summary of the data sources used to answer the research questions. Then individual findings were reported for the four subjects across all tasks. Next, the results for Research Question One were summarized and analyzed, followed by the results for Research Questions Two and Three. Chapter Five focuses on the study conclusions based on the findings, limitations of the study, and recommendations for further research. 138 CHAPTER FIVE SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS Introduction Chapter Five provides a summary of the previous chapters and conclusions and recommendations. Each question is presented, and is then followed by findings, a discussion of the relationship of the findings to past literature, conclusions and implications, and recommendations for hypotheses and future research questions. The discussion of findings includes supporting as well as contradictory findings from the literature review. The chapter also includes general conclusions related to the research design used in this study as well as a discussion of limitations. The chapter ends with a discussion of general recommendations for the field of training and specific suggestions for training and developing trainers in organizations. Summary of the Study In this study two expert trainers and two novice trainers were asked to verbally respond to three training case studies and two sorting tasks. The case study tasks involved identifying potential problems and solutions, then creating categories and structures with the identified problems. The sorting tasks involved defining, categorizing, and structuring training terminology and training competencies. The interviews were audiotaped and then transcribed for analysis. The analysis sought to determine if there were any differences between the novices and experts related to the content and structure of their knowledge and their problem solving strategies. Specifically, the analysis focused first on the amount and type of knowledge, categories formed, and structure of the organization of ideas. Secondly the analysis focused on cues used, sequence of tasks, flexibility in solving 139 problems, and the time it takes to complete tasks. A final analysis was completed to determine the consistency of performance between the two groups across tasks. The qualitative research approach included strategies to ensure reliability and validity in interpretation of the data. First, the items in the case studies and sorting tasks were derived from literature in the field of training. Second, the data were coded using standards developed by the International Board of Standards for Training, Performance, and Instruction. The data were also coded inductively, using categories that emerged from the subjects' protocols. All of the coded data were checked for reliability. Finally, triangulation was used to compare multiple sources of information about the same research questions and to analyze the data in a variety of ways. The triangulation approach to qualitative research was used to ensure that the results were reliable and independent of the problem type and content. Fin in 5 Conclusions and Recommendations for Futur Research This section describes the findings for each research question, supported by information from the literature review. In addition, recommendations for additional research questions and hypotheses will be presented. Research Question One Research Question One focused on differences in the content and structure of the knowledge of expert and novice trainers. The specific question and sub- questions are listed below: Do novice and expert trainers differ in the content and structure of their knowledge of training? The sub-questions were: 1a. Do expert and novice trainers differ in the amount of information they generate during selected tasks? 140 1b. Do expert and novice trainers differ in the type of information they generate during selected tasks? 1c. Do expert and novice trainers differ in the way they categorize information generated during selected tasks? 1d. Do expert and novice trainers differ in the way they organize the information and categories they generate during selected tasks? Finding I—Amount and Type of Knowledge Experts appear to generate more accurate information than novices when solving problems and defining technical terminology. They are particularly fluent in terms of the number of solutions proposed for each problem suggested. Their solutions are accurate and cover a range of instructional variables. Relation to Past Literature This finding supports many of the expert-novice studies, particularly those by deGroot (1966) and Chase and Simon (1973), who found that experts had a wealth of information based on their experience in the field. In addition, the finding supports other studies that show that experts can elaborate and provide many details on a variety of topics in their field of expertise (Voss, Vesonder, and Spilich, 1980; Charness, 1979). Both sets of studies also found that experts were able to chunk information into meaningful categories that were rich with sub-categories. This chunking ability was a consistent finding, whether the area of expertise was knowledge of a game or knowledge in a professional activity. 141 Conclusions and Implications Expert trainers have a wealth of knowledge that they can call on, both in routine and new situations. This knowledge base has breadth and depth, allowing expert trainers to identify a variety of problems in a given situation and to find a range of detailed solutions for them. In general, experts focused on organizational and transfer considerations and instructional design concerns, as well as the delivery and motivational issues identified by the novices. Their perspective of the trainer's role was broader and their varied experiences and backgrounds enabled them to consider not only the learner, but the trainee's supervisor and the organizational culture. As in most studies of expertise, the experts for this study were chosen based on their years of successful practice in the field. Since experience is a common denominator among expert-novice studies, these conclusions about the amount of knowledge experts possess may lead one to believe that only through experience can novices become experts, and that formal education may not be productive. Rather than abandoning formal education, however, the implication may be that it be adapted to include guided experiences in a variety of situations. This is similar to the skilled craftsman's apprenticeship and the medical resident's internship. The critical factor here may be the ability to structure learning activities that simulate the range and depth of an expert's lifelong experience, in an attempt to accelerate the learning process. Another informal observation, noted in Chapter Four, was the experts' confidence level with every problem they faced and the sorting tasks they performed. It is possible that experts' degree of knowledge gives them confidence in solving problems and allows them to generate many solutions for any situation, even those with which they have little experience. The implication here is that 142 training for trainers must go beyond mere competency and reach into the realm of confidence. This requires multiple practices in a variety of situations, not just one successful trial. Recommendations for Future Research The following hypothesis and research questions are generated from these conclusions and implications: Hypothesis: Differences in expert and novice trainers' amount of knowledge and the details provided are due to their experience in a variety of training situations. Research Questions: This study, as others in a variety of fields, has clearly identified that amount and type of knowledge is a foundation variable in the study of expertise. The question that arises relative to this issue is one that has been asked for many years in every area of professional activity: Can expertise be gained only through experience? Other related questions include: 0 What kinds of experiences are necessary to develop expertise? 0 Can effective experience be gained from simulated events or case studies? 0 Are there different levels of experience related to different levels of expertise? 0 How long does it take to become an expert trainer? 0 What kind of guidance do novices need while experiencing situations in their field in order to benefit from them? Finding 2 — Categories 8 Organization Experts categorized and organized information into some sequence or process flow rather than simply by common attributes or into a 143 hierarchy. These sequences were similar to instructional design and delivery models that are common in the field of training. Relation to Past Literature This finding confirms the research in the areas of expertise related to patterns of knowledge and the cognitive structure of the expert. Chase and Simon (1973) reported that chess players recalled chunks of information according to a larger pattern of strategies, given the characteristics of the game as evident by the board configuration. Additional support comes from research on teaching in which routines have been identified that influence teacher planning and decision making (Leinhardt and Greeno, 1986; Housner & Griffey, 1985; Clark and Yinger, 1979; and Peterson and Clark, 1978). The routines are retrieved based on cues in the situation and the teacher's goals. In the same way, the expert trainers in this study established a "rule" for organizing the items to be categorized and structured, based on their past experience, then proceeded to create a structure that corresponded to the rule. The sequential or relational nature of the experts trainers' structures appears to contradict, however, some of the research concluding that experts appear to retrieve information in hierarchical form (Chase and Chi, 1981; Voss, Vesonder, and Spilich, 1980; Spilich, Vesonder, Chiese, and Voss, 1979). In these studies, when experts were asked to recall facts, they discussed general, abstract ideas first, then recalled more specific ideas under each area. In the present study, subjects were asked to arrange concepts given to them into a structure, rather than merely to recall the concepts, and the results appeared more sequential, or even relational, than hierarchical. The difference in the study approaches, i.e., recalling information versus organizing given information, may account for the differences in results. Since studies of problem solving are attempting to derive implications about the way 144 people store information, process it, and retrieve it, then the study approach should be considered when analyzing results and drawing conclusions. Perhaps information is stored in a relational form, organized for retrieval into hierarchies, and manipulated into other structural forms depending on the task requirements and experience of the person. Training is an action-oriented area of expertise and its skills and knowledge are used in processes rather than in conceptual problem solving, such as in chess or physics. This process orientation may influence the way novices learn and increase their knowledge base until they become experts. It may also influence the way they "see" the environment in which they work and the way they view problem solutions. A comprehensive study that asks the same subjects to store, retrieve, and use information may begin to answer these questions. Conclusions and Implications Expert trainers categorize information about their field into sequential or process flow structures rather than simple relational networks or hierarchies. Even when concepts and techniques are given to them rather than generated by them, expert trainers struggle to impose a sequential structure on them, labeling categories as process steps and inserting new terms to fill conceptual gaps where no terms have been given. Problems and their solutions are also categorized into process structures much like those that are generated in sorting tasks. Expert trainers use similar H H' categories across tasks, e.g., "needs analysis, instructional design, delivery," and "evaluation." The categories that expert trainers use relate to common process models prevalent in the training field. Novice trainers do not consistently develop a process structure, but occasionally they use process-type category names, an indication that process-oriented thinking may be evolving. Is knowledge stored in memory in a different structure from the pattern that emerges when it is retrieved or manipulated? Specifically, is information stored in 145 a relational or sequential format, yet retrieved in a hierarchical manner? It may be that increased knowledge in an area, and thus expertise, may facilitate the creation of a "relational data base," much like those found in computer information storage systems, that can be restructured depending on the task—or problem—at hand. In a relational data base, information nodes are linked into a network based on common attributes and relationships. It is also possible that experts can form the appropriate structure for a particular subject matter. For example, one subject matter, such as math, which is conceptual in nature, may lend itself to a hierarchical structure, while another, such as training, which is action-oriented, is more easily organized as a process or into inter-related nodes. The implications of these conclusions and questions relate to the way training information is organized for novice trainers so they can learn it. If information is presented as a series of topics organized in a hierarchical fashion, novices may not be able to go beyond the imposed structure to use the information in situations other than classroom tests. Training information should be organized and presented as steps in a process, since that is how the experts see it, with related sub- steps, concepts, and principles. And novices should be given the opportunity to apply the process, concepts, and principles to problems and situations, testing the sequence and the related details to achieve some goal. In fact, trainer competency models also should be organized in a process structure, facilitating the building of an action-oriented, problem solving set of skills and knowledge. Recommendations for Future Research The following hypothesis and research questions can be generated from these conclusions: Hypothesis: Expert trainers' knowledge is organized into sequential flow diagram, relational patterns rather than hierarchical structures. 146 Research Questions: This finding stimulates further questions about the nature of the knowledge of expert trainers, as well as approaches to developing expertise. 0 Is there a clear training process model in the cognitive structure of trainers? 0 Is it a general model, adaptable to specific situations, or is it specific? 0 Can novices be taught a process model and use it as a perceptual template for gathering new information? 0 Will the use of a sequential training model accelerate the transition from novice to expert? 0 Why do novices appear to focus on relational models while experts focus on sequential models? 0 Is there a natural sequence in the development of knowledge structures that proceeds from relational to sequential? 0 When is a hierarchical structure generated? 0 Do structures of knowledge exist in the minds of experts or are they constructed depending on the task demands or the nature of the subject matter? 0 Are existing structures relational and then formed into sequential patterns as needed to solve problems? Research Question Two Research Question Two focused on the problem solving strategies of the experts and novices. Specifically the question and sub-questions were: Do expert and novice trainers differ in the strategies they use to solve training problems? 147 2a. Do expert and novice trainers differ in the cues they attend to when solving selected problems? 2b. Do expert and novice trainers differ in the sequence they use to solve selected problems? 2c. Do expert and novice trainers differ in their flexibility while solving problem variations? 2d. Do expert and novice trainers differ in the time it takes to work through each of the selected problems? Finding 3 — Cues Experts identified more cues than novices and they used their cues to generate and test solutions to problems. Relation to Past Literature This finding supports the evidence from the studies of teacher expertise conducted by Berliner (1986, 1988). Berliner found that expert teachers were able to perceive appropriate cues even when presented with a teaching situation only briefly. They were able to use the cues to describe the situation in more detail and then recommend strategies to use to reach goals and solve problems. Other studies show that cues help experts create an accurate problem space, which can be used to test strategies to achieve the goal state. Simon and Simon (1978) and Chi, Glaser, and Rees (1982) found that experts' representation of the problem space was more accurate and detailed than novices. Experts could separate relevant cues from irrelevant cues and create a detailed problem space by elaborating on the given information. 148 Conclusions and Implications Expert trainers view situations with a different set of "goggles" than novices. In other words, they are able to "see" information in a situation (in the form of cues) that informs their actions. Expert trainers collect information using a broad perspective and therefore are able to recognize and solve a broad range of problems. Since they collect so many cues from a situation, they can generate and test a variety of solutions or approaches to achieve goals. In addition they can test these solutions in an abstract way before taking actions that may fail. Even though expert trainers problem solve using so much information, they are efficient because they collect only the relevant cues to help them achieve their goals or to confirm their hypotheses. The literature on problem solving across many fields, including this study on trainers, suggests that experts view situations as problems to solve, while novices view them as snapshots. Experts perceive only the relevant cues, while novices focus on others. Experts perceive patterns of cues, while novices identify isolated cues. These patterns may represent a "variable set" that experts use as a template to solve problems. If situations are viewed as problems to solve, and expert trainers can represent these problems accurately, how do we equip novice trainers to do the same? We must present information about training—the training process and its related concepts and principles—in a context. We need to point out the relevant cues in a situation that indicate a pattern and that narrow the choice of strategies to use. We also need to help novice trainers broaden their tool kit from one all- purpose strategy to many strategies, depending on the pattern of cues in a situation and their hypotheses about a solution, enabling them to test strategies against the given information. 149 Recommendations for Future Research The following hypotheses and research questions can be generated from these conclusions: Hypothesis: Experts identify all of the relevant cues to solve a problem, given a training situation. Hypothesis: Experts create an accurate problem space based on the cues perceived in a training situation. Research Questions: The issue of cues stimulates interesting questions relative to trainers' perceptions while gaining experience. 0 Why do expert and novice trainers perceive different cues in the environment and in training case studies? 0 What different cues do experts and novices attend to? 0 How can novices learn to perceive the appropriate cues and patterns of cues? 0 What are the key variables that experts use as a template to form patterns of cues? 0 How can novices learn to use the appropriate cues to create an accurate problem space? 0 How can novices learn to use cues to test solutions to problems? Finding 4 — Sequence Experts used a working-forward analysis to solve familiar training problems and to categorize problems, terms, and competencies. 150 Relation to Past Literature This finding supports findings by other researchers relative to the order in which experts solve problems (deGroot, 1966; Charness, 1979; Reitman, 1976, Simon and Simon, 1978). In familiar situations, experts tend to "work forward," constructing their problem representation, stating their goal, and then generating one or two strategies based on the goal they want to achieve. They test these strategies against the goal and problem space to determine if they will be effective. This is called means-ends analysis in contrast to novices' hit or miss, ends-means problem solving strategies. Novices tend to create an inaccurate and incomplete problem space, then generate strategies that may or may not help them reach the goal. The goal is not necessarily clear to them, although it may become clearer as they apply different strategies and observe the results. The expert trainers in this study followed the same sequence, generating several effective strategies, testing them against the cues, collecting more cues, generating more strategies, and testing them. They even identified further problems that could result from the solutions they suggested. What was not obvious was the goal state they were trying to achieve. Expert trainers did not explicitly state their expected result. General statements such as, "This will enhance learning" or "This will make them feel more comfortable" were frequently made by the experts, and perhaps these represented their goal states. Conclusions and Implications Expert trainers, as with experts in other fields, view new training situations as problems to solve. In familiar situaitons, their problem solving processes can be viewed as a working-forward approach. That is, expert trainers are able to identify a goal based on given information, then generate a limited number of possible strategies to achieve that goal. In unfamiliar situations, expert trainers use a 151 working-backward approach, much the same as novice trainers in every situation. In a working backward approach, the goal is somewhat unclear (for novices, it may be entirely unclear), and the problem solver must generate a variety of strategies or solutions, test them against the given information, then refine the goal and focus the strategies, until an adequate solution path is found. Both of these approaches relate to the typical problem solving sequence suggested by many researchers in the field of expert-novice thinking: 1) create a problem representation in the form of a goal state and an initial state, 2) generate strategies to move from the initial state to the goal state, 3) test the strategies against the problem representation until an adequate solution is found. Training expertise, as with other areas of expertise, then, combines problem solving skills with a rich knowledge base to produce effective and efficient actions. In some way training experts are able to quickly assess problem situations, organize the cues, and then retrieve (in familiar situations) or construct (in unfamiliar situations) the solutions. In familiar situations, these solutions or strategies may represent routines that they have used successfully in the past. In unfamiliar situations, the strategies may be a hybrid of familiar routines. Given the same information, novice trainers seem to grope for any strategy that seems to relate, testing it against an assortment of cues and an unclear goal. Until novice trainers can experience the results of their actions in a variety of situations, they may be unable to generate either routines or new, effective strategies, except on a random basis. Novice trainers should be given opportunities to "see" situations through experts' eyes, not only recognizing the cues, but understanding how they relate to the desired goal. Novice trainers should be taught to articulate general goals such as "learner comfort" or "learner attention." They should be taught principles to help them select the appropriate strategies, given the cues. In other words, they should 152 be helped to see the big picture of a situation—its complete goals, its characteristics, related rules and principles, and the strategies, processes, and activities required to achieve the goal. Recommended Research The following hypotheses and research questions can be generated from these conclusions: Hypothesis: Expert trainers use a means-ends strategy to solve training problems. Hypothesis: Novice trainers use an ends-means strategy to solve training problems. Research Questions: This finding presents challenges to the researcher and practitioner in the area of expert-novice differences because it is so little understood. Researchers have found that when novices are taught heuristics, they still cannot apply them if they do not have the knowledge base from which to draw strategies (Meijer and Riemersma, 1986). Some research questions include: 0 When do experts use a means-ends analysis for training problems? 0 Are means-ends analyses a way that experts use the routines they have acquired over their years of experience? 0 Will teaching novice trainers routines in context help them build a set of problem solving strategies? 0 Which routines do expert trainers use most often? 0 Can a means-end analysis strategy be learned and used by novices? 0 How can novices be shown the "big picture" of a situation? 0 What instructional techniques can be used to help novices combine cues, goals, principles, and strategies to solve a problem? 153 Finding 5 — Flexibility Experts were able to generate a wide range of solutions for problems and discuss issues related to a wide range of trainer competencies. Relation to Past Literature This finding supports the research cited for Finding 2. Expert chess players win games because they think of correct and comprehensive strategies, no matter what the situation (Chase and Simon, 1973). Experts are confident in new situations, partly because they can see the similarities between the current situation and their past experiences and partly because they have confidence in their problem solving abilities. In fact, the expert trainers in this study relished the challenges presented in the most difficult case and in the more obscure training terms and competencies. It is interesting to note that flexibility was not a primary focus of the researchers in the studies of expertise related to games such as chess and bridge. Flexibility is discussed more in the literature about professional expertise such as medicine and teaching. In fact, for Berliner (1988), Clark et al (1978, 1979), and Leinhardt and Greeno (1986), expert teachers' use of routines, combined with their ability to be flexible allows them to be successful in any situation, no matter how unfamiliar. Conclusions and Implications Across a variety of problems and tasks, expert trainers are able to generate relevant solutions. They often suggest multiple solutions and elaborate on them. Sometimes they create a "system" of solutions, drawing on different variables and situation-specific factors. Novice trainers tend to suggest a few solutions that are heavily influenced by a personal "theme" such as motivation issues or learner 154 ability. Novice trainers tend to suggest similar solutions in all problems and their solutions do not range across a broad spectrum of alternative approaches. Flexibility is a characteristic of experts that surfaces in many studies. Yet it indicates an obscure and unpredictable expert skill. What makes the expert trainer choose a certain strategy when several others are viable? Why does the novice trainer continue to apply the same, limited set of strategies, and then fail when taking a risk to try something new? Of all of the findings reported so far, this one appears to be the most related to the specific experiences the subjects have had, as well as their values. Although expert trainers are able to identify a range of problems and solutions, they still seemed to focus their ideas around a personal "theme," as discussed in the individual findings for the subjects in Chapter Four. Even novice trainers appear to be driven by a theme or set of values that shaped their perception of the problems as well as their suggested solutions. One person in this study was concerned with learner confidence, another with organizational support, another with learner academic ability, and the last with learner motivation. These themes may have evolved from both the context and the problems these trainers have experienced in their careers, and may be representative of their attitudes and values about what constitutes important goals in training situations. So, guided by these themes, expert trainers may build their knowledge base, being flexible across a spectrum of factors, but influenced by their values. In addition, in terms of novice trainers, the personal theme may serve as an organizer for the variety of information bombarding them as expertise is developed. The implications from this finding of flexibility of thinking are twofold. First, trainers should experience a variety of situations. The organizational cultures, the level of the audience, the subject matter, and the instructional methods should all be varied during trainers' "apprenticeships" so they can begin to generalize across 155 situations, and also generate various solutions depending on the cues. There will be routines that novice trainers learn that apply to most situations, but they may vary in delivery or style, depending on the context. Secondly, it appears that trainers should be encouraged to articulate their values and beliefs about learning and human behavior, as well as the goals of training, so that they can begin to develop a personal theme to help shape the targets they are reaching for, no matter what the context of a situation. There may be a set of basic goals that all trainers strive for such as learner confidence, interest, and achievement; organizational support of the training program; transfer of training to the job, etc. If these goals are presented to the novice trainers, and then reinforced through experience, they may be more likely to see the "big picture," then focus their perceptual goggles, and finally, select appropriate strategies. Recommended Research The following hypothesis and research questions can be generated from these conclusions: Hypothesis: Expert trainers are more flexible in their approach to training problems, generating a range of solutions that consider several variables. Research Questions: The issue of flexibility relates to the routines discussed by researchers in the teaching area. It also relates to trainers' values and beliefs about learning and learners. Several questions can be proposed: 0 Does flexibility of approach depend on knowledge acquired through experience? 0 How does flexibility in problem solving relate to trainers' routines? 0 Can novice trainers learn to be flexible without years of experience in different situations? 156 0 Is there a template or model that can be used to coach novices to be more flexible in their problem solving strategies? 0 Do trainers come to the field with values and beliefs that shape their perceptions and inform their problem solving strategies? 0 How are trainers' values and beliefs shaped? 0 Are different, viable strategies selected because of values and beliefs? 0 Why do different trainers select different strategies? Finding 6 — Time Times to complete tasks were not consistent in most of the tasks. However, one finding was discovered: Experts took longer than novices to categorize the training terms and competencies. Relation to Past Literature Time to complete tasks has not been found to be as important a differentiator in many of the problem solving studies as some of the more qualitative factors. Some studies showed that experts were able to reach solutions quicker than novices, especially in routine situations, but these situations involved solving a common problem such as those found in bridge and chess (deGroot, 1966; Chase and Simon, 1973; Charness, 1979). These tasks are quite routine compared to the categorizing task presented to the expert trainers in this study. Expert trainers may not be required to organize their knowledge into a comprehensive structure very often, and therefore this task was more formidable than that of solving case studies. But why did the novice trainers take less time than the experts? Psychologists who theorize about the way information is processed may provide an answer to this question. Anderson (1982), Shiffrin and Dumais (1981), 157 and Shiffrin and Schneider (1977) discuss "automatic" and "controlled" processes that vary depending on the different levels and types of processing resources required for a particular task. For the expert trainers, the sorting tasks constituted a controlled thinking process requiring the use of their entire training knowledge base, which, as this study has shown, is quite large. This complex processing task would inevitably take more time for the expert trainers than the novices. The issue of the sorting and categorizing tasks requiring a more complex, controlled process may explain the V055, Vesonder, and Spilich (1980) finding that contradicts this study, in which they report that experts quickly recalled information. This contradiction may be explained by the way in which the experts in these studies created a hierarchical structure, forming higher-order categories of their own choosing first, then trying to fit the terms they generated into the categories. In the present study, the subjects did not form hierarchical structures. They formed sequential, and sometimes relational, structures, which may take more time to generate. Also, the nature of the sorting tasks in this study may have made the categorizing process more difficult and time-consuming because the experts were not allowed to generate additional items to put into the categories, but rather had to work with what they were given. They were also not given general category names, but were asked to form them, inductively, for the information given. Another consideration here is that the expert trainers tended to move items from category to category, frequently dissatisfied with the "fit." The novice trainers very often placed items in a category and left them there. This difference in approach may relate to the fact that experts perceive several defining attributes for each concept, allowing it to fit into a variety of categories (Greeno, 1980; Rumelhart and Ortony, 1977). This difference may also indicate that experts do, in fact, store information in a relational data base and then form a sequential structure when required to organize it during a sorting task. The same experts may form 158 hierarchical structures from their relational data base when asked to recall information from memory rather than form categories with information presented to them. Or, as discussed earlier, expert trainers may recall and organize information based on the task demands. Conclusions and Implications It is difficult to draw conclusions from this finding. All we can really say is that expert trainers take more time to categorize information related to their field. There is no apparent difference between novice and expert trainers in the time it takes them to solve training problems or define training terms. Time, it appears, is not an important differentiator between expert and novice trainers in terms of their knowledge and problem solving strategies. However, two interesting implications can be drawn related to time to complete the categorizing tasks. First, because expert trainers take longer to organize information, they may have difficulty forming structures that are acceptable to them in all situations. They may vary the members of categories, depending on how they define the task ("is this a training process or is it a set of skills?") If this is true, how do expert trainers coach an apprentice in the basic skills of training? Would the conversation be filled with "it depends" and "yes and no" as novice trainers ask for clear strategies and goals? This issue may relate to the observation that subject matter experts often have trouble teaching novices, making the subject appear more complex than beginners can comprehend. A second implication is that expert trainers may have trouble categorizing because they have not been asked to think about categories, tending to solve problems unconsciously. For expert trainers, recalling or re-constructing their knowledge to complete a routine activity or to solve a problem may be second- nature. This supports the concept of automaticity that information processing 159 theorists describe (Anderson, 1982; Shiffrin and Dumais, 1981; Shiffrin and Schneider, 1977). If experts' knowledge is automatic, how can they explain to novices what they are doing and why they are doing it? Further implications related to the difficulty expert trainers have in categorizing their knowledge involves the transformation from novice trainer to expert trainer. Some organized, even if relational, structure of knowledge needs to be presented to novices as they learn the process, skills, concepts, and principles of training. Expert trainers may not be the appropriate persons to generate this structure, however, unless someone helps them translate and commit the knowledge to some stable categories. This structure should include the attributes of the items in the categories, and these attributes should be added to as novice trainers experience more and varied situations. As they learn, novice trainers should be encouraged to form their own structures, based perhaps on a general, sequential instructional development and delivery model, which seems to be consistent across expert trainers. The structure may transform into a complex web of related concepts as the novice trainer applies the model and builds new ideas through experience. In terms of using expert trainers as coaches and mentors for novices, there are other implications. Expert trainers, to be mentors for novices, should be able to categorize and verbalize their knowledge—even if it is an artificial structure—so that novices can see abstract, connecting themes. The categories can be re-arranged as the contexts change so that novice trainers can learn flexibility in applying a variety of strategies. But the rules for moving information from one category to the next need to be clear, consistent, and communicated. Recommended Research The following hypothesis and research questions can be generated from these conclusions: 160 Hypothesis: Experts take longer to categorize a large set of training information given to them when they have not generated the information themselves. Research Questions: There are several research questions that can be generated from this finding and the lack of findings. Are there tasks in which expert and novice trainers are similar or different relative to the time it takes to complete the task? Do the times to complete tasks vary more because of individual differences than because of levels of expertise? What is happening in the mind of the expert while categorizing training terms and competencies? Do experts struggle with categorizing because their knowledge is imbedded at an unconscious level or because it is stored in a complex, relational data base rather than categories? Why do experts have difficulty categorizing while they have no difficulty recalling or structuring strategies to solve problems? Can expert trainers serve as mentors and coaches for novice trainers without creating some stable knowledge structure to use as a guide? Research Question Three The final research question analyzed the consistency across tasks for the two groups. Specifically the question asked was: Do experts and novices differ in the consistency of their knowledge content and structure and problem solving strategies across tasks? 161 Finding 7— Consistency Experts and novices tend to be consistent in the amount and type of knowledge they generate, the way they categorize and structure information, the cues they identify, their sequence of problem solving, and their flexibility. Relation to Past Literature This finding was expected, considering the findings for the individual research questions and the studies cited earlier. The one inconsistency was some of the performances of Novice 2, who appeared to be an intermediate, and almost an expert in some areas. This may reinforce Berliner's (1988) model of teaching expertise in which he identified five stages of proficiency. Novice 1 in this study might be classified as competent (stage 3) because she was able to generate more effective solutions than Novice 1, but was limited in her range of solutions and did not seem to have a broad view of the role of the trainer. Conclusions and Implications Expert and novice trainers are qualitatively and quantitatively different in a consistent manner in terms of their knowledge and problem solving strategies. Expert trainers consistently identify more problems in case situations and generate more solutions. Their problems and solutions consistently vary across a wide range of factors and include much detail. Expert trainers consistently identify more cues and these are relevant to the problems they identify. They are also consistent in their sequence of problem solving and in the way in which they categorize training information, preferring a sequential organizational structure to a hierarchical one. Novices consistently generate less information, which does not cover a range of 162 factors. Their organizational structures for information tend to be more relational or random. Because the performances of one of the novices in this study were inconsistent, and because she had more formal education than the other novice, there is an indication that there may be intermediate stages in the development of expertise related to experience. This was especially evident in the amount and type of information she generated. It is possible that development from novice to expert, at least in terms of professional expertise such as teaching, may be uneven with some skills developing before others. For example, there may be a learning sequence that proceeds from developing routines to developing flexibility. Another sequence may be the ability to satisfy multiple goals in a training situation. Development may depend on the novice trainer's opportunity and ability to perceive relevant data in the environment, store them meaningfully, and use them to form and test strategies. The strategies that are effective across situations and whose results are predictable may become routines, while more situation-specific strategies may evolve more slowly. Because of this finding and these conclusions, a developmental model of expertise should be used as a scaffolding for trainer training. Berliner's five stage model, which is based on the work of Stuart and Herbert Dreyfuss, appears to support the findings of this study and others in the area of professional expertise (including his own). The stages are: 0 Stage 1: Novice Stage 2: Advanced beginner Stage 3: Competent 0 Stage 4: Proficient Stage 5: Expert 163 According to Berliner, the skills developed at each stage appear to build on the previous stage as represented by this hierarchy: 0 Stage 1: rules 0 Stage 2: rules, routines, labels 0 Stage 3: rules, routines, labels, relevant details, overt decisions 0 Stage 4: rules, routines, labels, relevant details, overt decisions, global view, abstract categories, quicker decisions 0 Stage 5: rules, routines, labels, relevant details, overt decisions, global view, abstract categories, quicker decisions, automaticity If this is the developmental model, then training for novice trainers can be designed to support this sequence, just as developmental programs for children are geared to expected stages of progress. Using this model as a template, you would not expect novice trainers to be able to make quick and accurate decisions early in their careers unless they had achieved the prerequisite skills or unless they were provided with a job aid or coaching. If an accelerated learning program for trainers were designed, it would provide information and experiences corresponding to each stage of development, with competency tests administered to measure the trainer's readiness for the next stage. Obviously this approach would put a damper on the current, commonly accepted train-the-trainer approach, which usually involves an intensive forty-hour workshop followed by total immersion into a classroom situation. It appears that if a more reasonable training sequence is not implemented, novice trainers will not be able to make decisions with accuracy and automaticity for quite a long time. Recommended Research The following hypotheses and research questions can be generated from these conclusions: 164 Hypothesis: Experts and novices behave in a consistent manner, no matter what the problem type or content is. Hypothesis: Inconsistent performance on tasks indicates a person in transition between novice and expert. Research Questions: Research questions related to this finding revolve around the development of expertise and intermediate skills. Are there stages of development that trainers proceed through from novice to expert? What are the stages of trainer development and what are their corresponding skills and knowledge? Do different performance levels represent different stages of development for trainers? How does experience in a range of situations affect the consistency of performance across tasks? Why are novices and experts consistent across tasks, while intermediates are not? Can trainer development be accelerated? Summary of Conclusions The conclusions from this study are summarized below. Expert trainers have much more information about their field than novices that they can use in routine and new situations. This large amount of information has breadth and depth. Expert trainers generate detailed, effective solutions to a wide range of problems that they identify given case study information. Novice trainers generate simplistic, and sometimes ineffective, solutions to problems. 165 Expert trainers categorize information about their field into sequential or process flow structures rather than simple relational networks or hierarchies. These process structures resemble action steps described in common models relative to planning, implementing, and evaluating training. Novice trainers develop random or relational structures that do not appear to correspond to a cohesive model. Expert trainers perceive cues in training case study information that novice trainers do not see. These cues vary across a broad spectrum of factors, enabling expert trainers to generate solutions to a variety of problems. In addition, expert trainers use these cues to test their solutions, refining and elaborating on them until a satisfactory solution "set" is obtained. Novice trainers do not revisit their cues while developing solutions. Expert trainers view training challenges as problem solving tasks, using a working-forward approach when solving familiar training problems. They analyze the given information, then identify a goal to reach for, and develop one or two strategies, refining them by checking against cues. Novice trainers use a working-backward or hit-or-miss approach, suggesting random strategies, not testing them against cues, and not clearly articulating a goal. Expert trainers use a working-backward approach in unfamiliar situations. Expert trainers are flexible in the amount of information they generate about their field and in the number of terms they can define. This demonstrates their range of knowledge. Novice trainers generate information from a limited range of factors. Both expert and novice trainers generate information around a personal "theme," which they use 166 to develop ideas. The experts' themes are visible across the wide range of information they generate. 0 Expert trainers take longer to categorize training terms and problems. They struggle with categorization tasks, striving for a "perfect fit." Novice trainers do not struggle with categorization. There are no other consistent differences between novice and expert trainers regarding time to complete tasks. 0 Expert trainers are consistent in performance in all areas of this study: amount and type of knowledge, organization of information, cues, sequence of problem solving, flexibility, and time to categorize. Novice trainers are inconsistent, especially in the area of amount and type of knowledge. This may reflect the existence of intermediate stages of development. The Research Approach: Conclusions and Limitations In addition to the findings listed and discussed above, additional conclusions can be drawn about the research approach and limitations of the study. 0 Some of the findings may relate to individual differences. Expert 1 exceeded Expert 2 in some areas including number of solutions and number of competencies present. On the other hand, sometimes Expert 2 exceeded Expert 1, e.g., amount of knowledge. The novices differed greatly. Some of this may relate to the formal education and "world experience" of Novice 1 that Novice 2 lacks. Each subject seemed to have a model or paradigm through which they viewed the training problems and terms. These seemed to relate to their experience in their training jobs and the culture in which they currently work. In spite of these paradigms, they still acted consistently in some tasks. 167 A study of expert and novice trainers should probably include trainers at an intermediate stage considering the inconsistencies found in Novice 2's performance on the tasks. Having two subjects representing each of the three categories may help discover transitional stages of development. Also, the subjects came from varied backgrounds and training situations, and that helped validate the differences. Novice 2 had the most different background and job situation—more blue-collar—and it may be that additional studies should focus on expertise in a specific type of training situation and culture to determine if there are different sets of skills needed for each environment. Also, studies should be conducted in specific training formats, e.g., classroom trainers, on-the-job trainers, technical trainers, management trainers, etc. The sorting tasks were frustrating and tiring for the subjects. There were too many terms to define. By the time the subjects got to the categorizing and structuring tasks, they were already tired. They might not have put as much effort into these tasks as they would have under different circumstances. However, the subjects didn't seem to vary their times during the sorting of the case problems, which were done on a separate occasion. The other frustration related to the sorting tasks, especially for the experts, was that the set of terms was not "comprehensive." In other words, when they tried to build their model, they could not find all of the terms they needed. It might be more interesting to let the subjects tell all that they know about training and observe the sequence in which they recall information. They could also be asked to draw a structure from memory. Since most of the data are qualitative, they are subject to rater bias. The researcher is an expert in training and may have viewed the protocols 168 through a personal paradigm, filtering the subjects' actual meaning. Further work needs to be done to capture and analyze qualitative data in a reliable, yet accurate manner. There was much more information in the data that were not analyzed for the purposes of this study and that would have been difficult to categorize. But the information that can be learned from these data is important in furthering the training and development of trainers. Paragraphs were used in this study to denote new "chunks" of information. These chunks were counted to determine the amount and type of knowledge. This is the approach taken in many of the expert- novice studies. However, it did not appear that the paragraphs in themselves represented new information. Rather it appeared that the problems and solutions that the subjects gave were the distinct sets of information. Problem/ solution sets were used as a measure of amount and type of knowledge, as well as a measure of flexibility to account for this possibility. In future studies it would be wise to carefully select the units of analysis before coding and counting the data. The IBSTPI competencies were used to code the data in this study, upon the recommendation of experts in the field of qualitative research who suggest using existing categories, yet they were not always a good "fit" for this study. There were three categories "missing." That is, there were data that seemed to form additional categories—they did not fit into any of the IBSTPI categories. These categories might be labeled: 1) performs a learner needs analysis 2) makes adjustments in the instructional design (not just the materials as stated in the IBSTPI list) 169 3) adapts strategies based on organizational support and transfer opportunities. _ To compensate for this problem, the researcher recoded the data using inductively-derived categories. However, it is interesting and of some concern that the IBSTPI group did not include these tasks in their list. Further investigations into trainer competency, such as this research study, may help broaden and validate the existing documents. A final conclusion relates to the nature of expertise and the focus of this study. Many of the expert—novice seminal studies analyzed the knowledge and skills of experts in non-professional activities, such as chess, go, and bridge playing. The cognitive processes of these experts may be easier to pinpoint than those of the expert professional such as a doctor, physicist, teacher, or trainer. There are many more variables to inform decisions and the breadth and depth of knowledge is greater. Chess strategists consider the pieces, the board, and the opponent's moves, while trainers must consider the learner, the organization, the subject matter, the materials, the instructional design, the location, and their own knowledge and abilities, among other variables. Perhaps better methods for studying the expert professional should be developed, with researchers depending less on the quantitative analytical approaches of the seminal studies, e.g., counting paragraphs. The work of Berliner (1988) and Clark and Yinger (1979) in the study of teachers includes examples of more appropriate techniques, such as showing slides or videotapes to subjects, having them experience simulations, asking them to analyze student records, and examining subjects planning methods. This study used realistic training case studies to simulate real problems in an attempt to deal with this difference. And, in fact, there were more interesting and rich results from 170 the case studies than there were from the sorting tasks, which correspond to the more traditional expert-novice research approaches. Recommendations for the Field of Training This study confirmed for the researcher many personal hypotheses about how training is conducted in organizations. It also underscored the frustration one feels as a professional trainer when faced with unreasonable expectations about transforming people from novices to experts in a few short training sessions. It is not only in the training of trainers that this expectation exists—new supervisors are expected to complete a few basic workshops and a brief tour of duty in various departments and then perform like a veteran. New operators are expected to learn a complex piece of equipment in a few days and then are pressured to meet production quotas and quality goals like a seasoned performer. The question becomes, "when will decision-makers in organizations recognize that expertise isn't something one learns overnight?" The medical profession seems to understand this, and even the teaching profession is beginning to focus more on guided experience than textbook learning about teaching. One training process in organizations that seems to take this ’learning curve' into consideration is skilled trades certification, which often requires apprentices to complete four years of classroom and on-the-job experiences before becoming fully certified. If this and similar models could be applied to other training situations, learners may be less frustrated and the transition from novice to expert may be accelerated and, certainly, less painful, both for the trainee and for the organization. General Recommendations Based on findings in this study and in other, similar studies of professional expertise, several general recommendations can be made about training. 1) 2) 3) 4) 5) 171 Competency models should be developed for every job that include principles and concepts that can be translated into strategies and tactics, rather than just focusing on job tasks. The models should include a set of variables to be considered and should illustrate the link between the variables and desired goals or outcomes. Process models that can be used as templates for decision-making should also be included. The competency models should be used to develop a series of structured, evolving experiences for trainees. Trainees should be introduced to common, simple experiences until basic routines are recognized and developed, then given the opportunity to make decisions in more complex situations. Classroom learning experiences should be brief and followed immediately by application experiences. Trainees should have opportunities to "fail safely" to see the results of their decisions and actions. Then they should be provided with opportunities to discuss their perceptions and reasons for their performance. A "spiral learning" approach can be used, exposing trainees to elaborations on the same basic knowledge by adding more complex variations to basic situations. Methods should be derived to help trainees "see" through the eyes of the expert, comparing their own perceptions to those of the experts. Experts should be helped to make their knowledge and skills explicit, or they should be discouraged from taking the responsibility for teaching novices. Trainees should be provided with the "big picture" of how their knowledge, decisions, and actions are affected by and affect other parts of the organization. This may contribute to the development of a broad and deep network of related principles and concepts that can be built upon as 172 the trainee experiences the variety of situations that occur day-to-day on the job. 6) Checklists, job aids, and coaching should be provided to extend training beyond the traditional classroom approach. 7) Trainees should not be expected to perform at top levels upon completion of a single training experience. Rather, they should proceed through a series of developmental experiences and checkpoints, with each level demanding higher performance. Specific Recommendations for Training Trainers In addition to the general list of recommendations above, several specific suggestions can be directed at the development of trainers in organizations. 1) Rather than depending on competency models that are static lists of techniques, generate a model that is dynamic, considering the developmental nature of learning how to teach, as well as the variety of situations in which trainers can find themselves. Berliner's five levels of competency is an appropriate starting point for this approach. Reigeluth's (1983) model of instructional design also can be used to build basic and advanced training strategies, principles, and concepts. This model states that instruction must consider a) training conditions, e.g., the learner, the trainer, the organization and culture, the subject matter and expected job performance, and the training design and materials, b) delivery strategies, including environmental, content, and motivational strategies, and c) evaluation strategies such as assessment of the instructional design and delivery, learner achievement and attitude, and extent of transfer to the job. 2) 3) 4) 5) 6) 173 Trainers should be viewed as more than a delivery medium, in the way that computer software or videodisc training programs are viewed. Trainers are needs analysts, instructional designers, and evaluators, as well as presenters. As such, they make decisions and need to be empowered to make those decisions by being provided with the knowledge and guided experience to do so. The task of training should be viewed as problem solving, with skill development that includes practice in recognizing and differentiating cues in the environment, employing routines, recalling and generating appropriate strategies, and analyzing the results of actions taken. Trainers can have a tremendous positive or negative impact on the organization and should be carefully selected and trained. New trainers should be paired with seasoned trainers who can articulate their knowledge and coach with patience and specific suggestions. The trainers with whom beginners apprentice should be able to consistently model all of the decision-making and problem solving skills that are expected of an expert. Train-the-trainer learning experiences should include more case studies and simulated teaching events, rather than simply focusing on presentation skills. Novice trainers should be encouraged to view expert trainers in a variety of situations, and then to practice skills in similar situations. Videotape can be used to capture not only the trainers' actions, but the learners' reactions in simulated and real teaching situations. Apprentice trainers should co-teach and be limited to easier teaching assignments until they can demonstrate the mastery of basic routines. Then their experiences can be varied to enhance their ability to be flexible, creative, and successful in any situation. 174 7) Training is a challenging professional activity because it requires expertise in two dimensions: mastery of the subject being taught and mastery of the science and art of teaching. This dual role of trainers further suggests the difficulty of developing into an expert. Organizations should recognize this challenge and be willing to invest the time and resources to enable novices to become experts without losing them somewhere along the way because of frustration, stress, and failure. Chapter Five Summary This chapter summarized the study and presented the findings. Hypotheses and recommended research questions were presented. Some of the key findings included the differences between expert and novice trainers in the amount of knowledge they bring to bear on training situations and their approaches to problem solving. Individual differences may account for some of the findings, but there seem to be clear differences in some areas. Many of the findings support studies of expertise conducted in other fields, especially in the area of teacher education. What is needed now is research into the model of training expertise and its development, in a similar fashion to the Berliner studies. Training in organizations, and specifically trainer development, must become more than a 40 hour workshop and instead be viewed as an ongoing professional development process. Continued research and creative implementation of new approaches in this area will help achieve this objective. 175 APPENDIX A 176 Training Problems Problem One Part A Imagine you will be the trainer of a quality improvement training program for a large manufacturing plant (10,000 employees) that makes aircraft parts. You are an outside consultant and have not worked for this company before. Listed below is the information given to you about the assignment. Think about the potential problems you foresee and the strategies you would use to prevent or minimize them. The training program is a three-day workshop in quality improvement concepts and skills. There are several technical tools to be taught to the group, but practice exercises are built into the design to help the participants learn. The training program will be delivered to all production operators. These operators have an average educational level of tenth grade; some with high school and college degrees, some who cannot read or do simple math operations, and the rest somewhere in the middle. The classes will consist of mixed ability groups as there is no feasible way to separate those of lower ability. Class size will be about 20 to 25. The operators have not had a technical course like this before, but they did attend a quality circle workshop about a year ago to learn team-building skills. That program was ill-received because it was poorly taught, and the learners were never given the opportunity to use the skills on the job afterward. There is little interest in the new workshop because of this bad experience of a year ago. 177 The person who is running the program, the Manager of Human Resources, has worked closely with the Director of Quality to develop a workshop that is designed well, but both agree there is a lot of material to be covered. They expect that with a skilled trainer, the learners will enjoy the program and be able to use the skills on the job afterward. One concern they have is that some of the operators' supervisors, who attended a similar program, do not "buy in" to the program content. The class will be taught in a training room that is located at the far end of the factory. You will be given a leader's guide and overheads to use. There are also several videos that present the more complex statistical concepts. These were purchased from a quality institute affiliated with a local university. The learners will be given a workbook and worksheets for the problem-solving tools they will learn. You will be given calculators for the learners to use for any math problems. You can request any other equipment you may want to use. What potential problems could occur while you are teaching this program? What would you do to prevent the problems from occurring or to minimize them if they do occur? Part B Listed below are variations on the training assignment above. Read them, and then state any other problems that might occur based on this new information. Be sure to identify your strategies to prevent or minimize any problems you identify. Variation 1: You will be teaching homogeneous groups. The operators will be grouped for your classes according to their math and reading abilities. They will be given a pre-course screening test to determine their ability levels. 178 Variation 2: The workshop will be offered over several weeks instead of three days in a row. You will see each group for one full day, three weeks in a row. Variation 3: The operators have never had any training programs before, in the quality area or on any other topic. Variation 4: The class will be taught off-site in a training center located about five miles from the plant at which the operators work. 179 Problem 2 Part A Imagine you are a trainer on the personnel staff of your state's Department of Health, Education, and Welfare. You will be teaching a training program to employees of the department. Listed below is the information given to you about the assignment. Think about the potential problems you foresee and the strategies you would use to prevent or minimize them. The training program is a four hour workshop on a new record-keeping system. The system will be totally on computer in about a year, but for now the staff will need to complete all forms by hand or on typewriters. The program will cover the reasons for the new approach, the main components of the system, and how to use new forms. The training program will be delivered to all office and administrative employees and their supervisors. The average educational level of the target audience is two years of college; some having advanced degrees, some having trade school backgrounds, and the rest somewhere in the middle. The learners will attend the classes as work teams, with supervisors and all of their employees attending together. Class size may vary from as small as five people to as large as 30. The staff has had several courses like this before. Almost every time a new department director is hired, there is a major revision of procedures. There have been five new directors in the past eight years. Most of the past courses have been taught by internal training personnel, were several days long, and were considered rather boring, although necessary. The person who is running the program, the Personnel Manager, has worked closely with the Manager of Information Processing to make sure the course design is effective, but both agree that there is a lot to cover in four hours. They expect that 180 with a skilled, internal training consultant, the learners will be able to get through the content and remember how to use the forms on the job after the class. The class will be taught in a large conference room located in the department office building. You will be given a course outline and overheads of all of the forms. In addition you will be given examples of completed forms. The learners will be given copies of all of the forms and very detailed guidelines on how to complete them. You can request any equipment you may want to use. What potential problems could occur while you are teaching this program? What would you do to prevent the problems from occurring or to minimize them if they do occur? Part B Listed below are variations on the training assignment above. Read them, and then state any other problems that might occur based on this new information. Be sure to identify your strategies to prevent or minimize any problems you identify. Variation 1: The program is on a complex record-keeping system related to a new law passed in the legislature. The participants will need to learn a series of new procedures, formulas for budgets, and documentation systems for the tasks they perform. The group has never had a system like this before. Variation 2: All previous workshops have been taught by outside consultants from the firms who designed the new procedures for the department. The consultants were also the salespeople for the companies that received the contracts from the department director for the procedures projects. Variation 3: The program was designed by technical and legal staff members who are familiar with the new law and the procedures. There isn't enough time to have the personnel staff review the training materials before you begin teaching. 181 Variation 4: The materials include a summary of the new law, materials describing the procedures that must be followed and drafts of the forms to be used (the legal and financial staffs have not finished designing the final forms). There will be no completed forms to use as examples. 182 Problem 3 Part A Imagine you work for a prestigious training firm and have been assigned to teach a training program at one of your city's large banks. Listed below is the information given to you about the assignment. Think about the potential problems you foresee and the strategies you would use to prevent or minimize them. The training program is a five day workshop on strategic planning. A new planning method is being implemented to focus goals and strategies on the bank's new customer service policy. It is a complex, detailed approach that includes examination of each officer's own personal values, management style, and goals. The bank has not had a strategic planning process and has never incorporated any "soft" skills concepts in discussions of management ability. The training program will be delivered to all upper level managers of the bank (30 peOple), most of whom are vice-presidents. The average educational level of the target audience is some coursework beyond college, and many of the participants have advanced degrees. The learners will attend the classes in cross- functional groups. Class size will be about 10 to 15 peOple per class. Attached is a profile of the officers that summarizes their educational and training experiences. It also groups them by how many people report to them. The bank has never done any in-house training for their officers. In the past, this level of management always attended seminars of their own choosing. These seminars always took place off-site, often in resorts or at graduate schools of business. The officers enjoyed these courses because of the variety of businesses represented, the expertise of the instructors, and the opportunity to travel and explore a different environment. 183 The person who is running the program, the Vice-President of Human Resource Management has worked closely with the Vice-President of Planning and Development to make sure the course design is effective, but both agree that there may be some resistance to the personal approach the workshop takes. They expect that with a skilled, external training consultant, the officers will accept the new way of thinking and operating and implement it without resistance. Of course, the officers will be required to implement the process, even if they are uncomfortable with it. The class will be taught in a management training room located on the top floor of the main branch of the bank, where many of the officers work. You will be given a seminar leader's packet. It will include a step-by-step teaching guide, with a script that you will need to follow very closely. Also, you will receive overheads covering every major point and activity, videos, and sample completed worksheets. The participant workbook is a copy of all overheads and the worksheets they will use during exercises. You can request any additional equipment you may want to use. What potential problems could occur while you are teaching this program? What would you do to prevent the problems from occurring or to minimize them if they do occur? Part B Listed below are variations on the training assignment above. Read them, and then state any other problems that might occur based on this new information. Be sure to identify your strategies to prevent or minimize any problems you identify. Variation 1: This is a two-week workshop on strategic planning. The officers attending the program will have to come to work early and stay late if they want to 184 complete any of their regular work during the training. They will be encouraged to delegate their responsibilities to their staffs or delay completion of projects. Variation 2: The managers have never attended a training program, on- or off-site. They have had speakers come to meetings now and then, and most have attended graduate programs in finance or business. They have not really ever asked for training and feel that, at this point in their careers, there is little else to learn, except to keep updated on economic, legal, and financial issues. Variation 3: The course is a packaged program that was designed by a large training company. It is the standard version that is used with clients ranging from banks to department stores. Variation 4: The seminar will be taught in a variety of places. There is no training room available for the whole two weeks. You will use conference rooms located on various floors of the bank. At least one of the training days you will have to meet in a large office, where the desks will be pushed aside and replaced with tables and chairs. 185 APPENDIX B 186 Card-Sorting - 30 Key Terms Part A Directions A training tool or technique is written on each card. For each technique, complete the following tasks: 1. Explain what the tool or technique is. 2. Describe what training problem or problems it might be used to solve. 3. Review the pile again and describe any additional problems each technique might solve. 4. If you do not know what a term means, say so and put that card aside. Training Tools and Techniques Active listening Closed questions Competency tests Customizing Debriefing Discussion Evaluation Experiential learning Feedback WWNQWFWN?‘ b—3t—It—lt—3t—JHh—l oxtniiswpi—sp Flipchart . Humor Ice-breakers . Interactive lecture . Interactive video . Job aids . Learner-centered training 187 17. Learning objectives 18. Logistics 19. Media 20. Modular training 21. Non-verbal communication 22. Open-ended questions 23. Overheads 24. Pilot test 25. Practice exercises 26. Prerequisite skills 27. Role plays 28. Simulations 29. Team teaching 30. Worksheets Part B Directions Now review all of the cards with terms you were able to define, and sort them into piles or categories. Label the categories. Part C Directions Now take your categories and arrange them into some network, picture, or structure that shows the relationships between them. Be sure to include all of the cards you put into each category. If you feel you need to re-categorize them to complete this task, go ahead. 10. 11. 188 MW . Active listening: Listening to someone by participating in the conversation, helping the person clarify and solve their problems. Closed questions: A question asked by the trainer that has only one correct answer. Closed questions often require only a "yes" or "no" answer. Competency tests: Assessments of the learner's ability to perform the expected behaviors at the end of the training class. Customizing: The process of adapting a class to meet the needs of the audience. Customizing can involve re—designing the activities in the class and the materials. Debriefing: A discussion that follows a practice or application exercise to help learners recognize the main points of the experience. . Discussion: An instructional method in which the trainer leads the group to discuss a topic, analyzing and building on each other's ideas. . Evaluation: A process to determine the effectiveness of a workshop or curriculum. . Experiential learning: An instructional technique in which the participants discover the main points to be learned through a simulation or other type of activity. Feedback: Information that tells people how they are performing. Flipchart: A chart pad and stand that can be used to present ideas in a class or record ideas as the class proceeds. The charts can be hung on the walls. Humor: Any technique that encourages laughter and a lighter view of the subject matter. Humor is used to relax learners and increase motivation. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 189 Ice-breakers: Techniques usually used at the beginning of a class that help participants feel more comfortable and relaxed. Interactive lecture: An instructional method in which the trainer presents information to learners in a way that combines explanation with discussion. This is usually done by asking and answering questions. Interactive video: A type of media in which a computer and video screen are combined to teach the learner information and skills. The learner can answer questions and control the topics being presented. Job aid: Any tool that helps learners use skills and knowledge on the job such as tables, diagrams, matrices, illustrations, and charts. Learner-centered training: Training that is specific, action-oriented, job-related, and interesting to the learner. Learning objectives: The goals of a workshop, stated in terms of observable actions that the learners will be able to perform as a result of the training. Logistics: The scheduling, materials, equipment, and environmental needs to consider when delivering a training class. Media: Any methods used to facilitate communication with an audience including overheads, Whiteboards, handouts, workbooks or guides, games, equipment, videos, audiotapes, and computers. Modular training: Training consisting of one set of specific skills or one topic presented in a brief block of time, for example, two hours. Often several modules are offered as a series. N on-verbal communication: A form of communication that occurs through gestures, body movement, body location, and tone of voice, rather than through words. 22. 23. 24. 25. 26. 27. 28. 29. 30. 190 Open-ended questions: Questions asked by the trainer that allow for more than one correct answer. Overheads: Transparent film containing words and graphics, that can be projected on a screen to highlight key points being presented. Pilot test: A test-run of a workshop to determine its effectiveness. Pilot tests are usually done with a typical group of participants and some experts who can help give feedback. Practice exercises: Activities that allow learners to use the skills presented in a workshop using a meaningful example, but not necessarily a real job example. Prerequisite skills: Skills needed to be mastered before learning new skills. Role plays: Learning activities in which participants act out parts to help them apply skills related to behaviors. Role plays are often used when attitudes interfere with a person's willingness to use new skills. Simulations: Learning activities in which participants are asked to simulate an actual situation. Simulations are often detailed and accurate representations of activities the learner will perform on the job. Team teaching: A teaching approach in which more than one trainer teaches a class. Worksheets: Step—by-step instruction sheets for using the skills taught in a workshop. Usually they include space for completing the steps using a real example. 191 APPENDIX C 192 Card-Sorting Task -- 34 Trainer Performancee Part A Directions A training strategy is written on each card. For each strategy, complete the following tasks: 1. Explain what the strategy is. 2. Describe what training problem or problems it might be used to solve. 3. Review the pile again and describe any additional problems each strategy might solve. 4. If you do not know what a strategy means, say so and put that card aside. Training Strategies (IBSTPI Competencies) (Competency 1: Assure preparation of the instructional site) 1. Confirm the physical arrangement of the instructional site, materials, equipment, and furniture (Competency 2: Establish and maintain instructor credibility) 2. Demonstrate acceptable personal conduct and social practices 3. Demonstrate content expertise 4. Demonstrate flexibility in response to learner needs and interests (Competency 3: Manage the learning environment) 5. Involve learners in establishing an appropriate level of learner comfort 6. Adapt delivery to account for learner characteristics 7. Manage the time available for the course 193 8. Provide opportunities for learner success 9. Manage group interactions and participation 10. Resolve learner behavior problems (Competency 4: Demonstrate effective communication skills) 11. Use appropriate verbal and non-verbal language 12. Use frames of reference familiar to the learners (Competency 6: Demonstrate effective presentation skills) 13. Use the voice effectively 14. Use eye contact effectively 15. Use gestures, silence, movement, posture, space, and props effectively 16. Organize content effectively 17. Use anecdotes, stories, analogies, and humor effectively (Competency 7: Demonstrate effective questioning skills and techniques) 18. Use appropriate question types and levels 19. Direct questions to learners appropriately 20. Use active listening techniques 21. Repeat, rephrase, or restructure questions 22. Allow time for learners' questions, comments, concerns, and answers (Competency 8: Respond apprOpriately to learners' needs for clarification or feedback) 23. Identify learners with clarification and feedback needs 24. Provide prompt, timely, and specific feedback (Competency 9: Provide positive reinforcement and motivational incentives) 25. Use introductory activities appropriate to developing learner motivation 194 (Competency 10: Use instructional methods appropriately) 26. Manage the group dynamics associated with instructional methods (Competency 11: Use media effectively) 27. Use media and hardware properly 28. Substitute for, add to, switch, or create media as required (Competency 12: Evaluate learner performance) 29. Monitor learner progress during instruction 30. Evaluate attainment of end-of-course objectives (Competency 13: Evaluate delivery of instruction) 31. Evaluate the instructional design and modify during delivery 32. Evaluate the effects of other variables, including the instructional environment, on learner accomplishments (Competency 14: Report evaluation information) 33. Report evaluation and end-of—course information 34. Recommend revisions and changes to existing materials and suggestions for new programs and activities 195 Part B Directions Now review all of the cards with strategies you were able to define, and sort them into piles or categories. Label the categories. Part C Directions Now take your categories and arrange them into some network, picture, or structure that shows the relationships between them. Be sure to include all the cards you put into each category. If you feel you need to re-categorize them to complete this task, go ahead. 196 BIBLIOGRAPHY Anderson, R. C. (1982). Acquisition of cognitive skill. Psychologal Review _82, 369-406. American Society for Training and Development. (1989). Models for HRD practice. Baltimore, MD: ASTD. Bartlett, F. C. (1932). RememberinLA study in experimental and socialpsycholgy; Cambridge, England: Cambridge University Press. Berliner, D. C. (1988). The development of expertise in pedagogy. Washington, DC: AACTE Publications. Berliner, DC. (1986). In pursuit of the expert pedagogue. Educational Researcher, 8, 5-14. Broadwell, M. (1990). Ten myths about instructor training. TraininLMagazine, May, 81-83. Campbell, D. T., 8: Stanley, J. C. (1963). Experimental and quasi—experimental designs for research. Chicago: Rand McN ally. Charness, N. (1979). Components of skill in bridge. Canadian journal of Psychology, 33, 1-13. Chase, W. G., 8: Chi, M. T. H. (1981). Cognitive skill: Implications for spatial skill in large-scale environments. In J. Harvey (ed.) Cpgnition, social behavior, and the environment. Hillsdale, NJ: Lawrence Erlbaum Associates. Chase, W. G., 8: Simon, HA. (1973). Perception in chess. Cognitive Psychology, 4, 55-81. Chi, M.T.H., Glaser, R., 8: Rees, E. (1982). Expertise in problem solving. In R.]. Sternberg (ed.), Advances in the psychology of human intelliflce. Hillsdale, NJ: Lawrence Erlbaum Associates. 197 Clark, C. M., 8: Yinger, R. J. (1980). The hidden world of teaching: Implications of research on teacher planning. East Lansing, MI: Institute for Research on Teaching. Collins, A. M., 8: Quillian, M. R. (1972). Experiments in semantic memory and language comprehension. In L. W. Gregg (ed.) anition in learning and memory. New York: Wiley. Cook, T. D., 8: Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. deGroot, A. D. (1966). Perception and memory versus thought: Some old ideas and recent findings. In B. Kleinmuntz (ed.) Problem solving. New York: Wiley. Ericsson, K. A. (1985). Memory skill. Canadian Iournal of Psychology, Q, 188-231. Ericsson, K. A., 8: Simon, H. A. (1980). Verbal reports as data. Psychological Review 8_7, 215-251. Eurich, N. (1985). Corporate classrooms: the learningbusiness. Princeton, NJ: The Carnegie Foundation for the Advancement of Teaching. Feigenbaum, E. A. (1970). Information processing and memory. In D. A. Norman (ed.) Models of memory. New York: Academic Press. Follman, I. (1985). Excellence in public school teaching. Paper presented at the Florida Educational Research Association, Miami, FL. Gagne, R. M. (1985). The conditions of learniig and theory of instruction. New York: Holt, Rinehart, and Winston. Glaser, R. (in press). On the nature of expertise. In Proceedings of the In Memorium Hermann Ebbinghaus Symposium. Amsterdam, the Netherlands: Elsevier-North Holland Publishers. Goad, T. W. (1982). Delivering effective training San Diego, CA: University Associates. Greeno, I. G. (1980). Psychology of Learning, 1960-1980. American Psychologist, _3_5, 713-728. 198 Hayes-Roth, B., 8: Hayes-Roth, F. (1979). A cognitive model of planning. Cognitive Science 3, 275-310. Housner, L. D., 8: Griffey, D. C. (1985). Teacher cognition: Differences in planning and interactive decision-making between experienced and inexperienced teachers. Research Quarterly for Exercise and SportJ 5_6_, 45-58. International Board of Standards for Training, Performance, and Instruction. (1988). Instructor competencies: the standards. Evergreen, CO: IBSTPI. Johnson, S. D. (1988). Cognitive analysis of expert and novice troubleshooting performance. Performance Improvement (Luarterly, 1, 39-54. Joyce, 8., 8: Harootunian, B. (1965). Teaching as problem solving. lournal of Teacher Education, Q, 420-427. Kant, E. (1963). Critique of pure reason. (lst ed. 1781, 2nd ed. 1787, translated by N. Kemp Smith). London: Macmillan. Knowles, M. (1985). Self-directed learning. Chicago: Follett. Latham, G. (1988). Human resource training and development. Annual Review of Psychology, 3_9, 545-82. Lavely, C., Berger, N., Bullock, D., Follman, J., Kromrey, J., 8: Sawilowsky, S. (1986). Expertise in teaching: Expert pedagogues. Tampa, FL: Institute for Instructional Research and Practice: University of South Florida. LeCompte, M., 8: Goetz, J. (1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, a; 31-60. Lee, C. (1990). Industry report. Training Magazine, 27, 29-32. Leinhardt, G., 8: Greeno, J. (1986). The cognitive skill of teaching. |ournal of Educational Psychology, Z8, 75-95. Leinhardt, G. 8: Smith, D. A. (1985). Expertise in mathematics instruction: Subject matter knowledge. Journal of Educational Psychology, 2, 247-271. 199 Lesgold, A. M. (1983). Acquiring expertise. In JR Anderson 8: S. M. Kosslyn (eds.) Tutorials in learning and memory. San Francisco: W. H. Freeman. Lovell, P. (1987). Expert and novice instructional developers: A study in how organization of knowlgge/ experience is displayed in problem solving performance. Unpublished doctoral dissertation. Michigan State University. Martinko, M.J. 8: Gardner, W.L. (1985). Beyond structured observation: Methodological issues and new directions. Academy of Management Review E, 676-695. Meijer, J. 8: Riemersma, F. (1986). Analysis of solving problems. Instructional Science _13, 3-19. Miles, M. B., 8: Huberman, A. M. (1984a). Drawing valid meaning from qualitative data: Toward a shared craft. Educational Researcher, 3, 20-30. Miles, MB, 8: Huberman, A. M. (1984b). Qualitative data analysis. Beverly Hills: Sage Publications. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological RevievL 33, 81-97. Mintzberg, MB. (1968). The manager at work—determining his activities and programs by structured observation. Unpublished doctoral dissertation. MIT Sloan School of Management. Newell, A. (1980). Reasoning, problem solving, and decision processes: The problem space as a fundamental category. In R. Nickerson (ed.) Attention and performance VIII. Hillsdale, NJ: Lawrence Erlbaum Associates. Newell, A., 8: Simon, H. A. (1972). Human problem solving. New York: Prentice- Hall. Peterson, P. L., 8: Clark, C. M. (1978). Teachers' reports of their cognitive processes during teaching. American Educational Research Journal, E, 555-565. Pfeiffer, J. W., 8: Jones, J.E. (1972-1990). The annual handbook for group facilitators. San Diego, CA: University Associates. 200 Powers, B. (1984). Trainingysystem guide. Three Bridges, NJ: Bob Powers and Associates. Reigeluth, CM. (1983) Instructional design: What it is and why is it? In C.M Reigeluth (ed.) Instructional deggn theories and models. Hillsdale, NJ: Lawrence Erlbaum Associates. Reitman, J. S. (1976). Skilled perception in go: Deducing memory structures from inter-response times. Cognitive Psychology, 3, 336-356. Ropo, E. (1987). Teachers' conceptions of teaching and teaching behavior: Some differences between expert and novice teachers. Paper presented at the American Educational Research Association, Washington, DC. Rumelhart, D. E., 8: Norman, D. A. (1978). Accretion, tuning, and restructuring: Three modes of learning. In J. W. Cotton 8: R. Klatzky (eds) Semantic factors in cognition. Hillsdale, NJ: Lawrence Erlbaum Associates. Rumelhart, D. E., 8: Ortony, A. (1977). The representation of knowledge in memory. In R. C. Anderson, R. J. Spiro, 8: W. E. Montague (eds.) Schooling and the acquisition of knowledge, Hillsdale, NJ: Lawrence Erlbaum Associates. Shavelson, R. J. (1974). Methods for examining representations of a subject-matter structure in a student's memory. Journal of Research in Science Teaching, 11, 231-249. Shiffrin, R. M., 8: Dumais, S. T. (1981). The development of automatism. In J. R. Anderson (ed.) Cognitive skills and their acquisition. Hillsdale, NJ: Lawrence Erlbaum Associates. Shiffrin, R. M. 8: Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, &, 127-190. Simon, H. A. (1974). How big is a chunk? Science, 183, 482-488. 201 Simon, D. P., 8: Simon, H. A. (1978). Individual differences in solving physics problems. In R. Siegler (ed.) Children's thinking: What develops? Hillsdale, NJ: Lawrence Erlbaum Associates. Spilich, G. J., Vesonder, G. T., Chiesi, H. L., 8: Voss, J. F. (1979). Text processing of domain-related information for individuals with high and low domain knowledge. Lournal of Verbal Learning and Verbal Behavior, 13, 275-290. Tulving, E. (1972). Episodic and semantic memory. In E. Tulving 8: W. Donaldson (eds). Qrganization of memory. New York: Academic Press. Turner, R., 8: Fattu, N. (1960). Solvingproficiency amongelementary teachers: The development of criteria. Bloomington, IN: Institute of Educational Research, Indiana University. Voss, J. F., Vesonder, G. T., 8: Spilich, G. J. (1980). Text generation and recall by high-knowledge and low-knowledge individuals. burnal of Verbal Learning and Verbal Behavior, 3, 651-667. Wexley, K., 8: Latham, P. (1981). Developing and trainingiuman resources in organizations. Glenview, IL: Scott, Foresman 8: Company. 31293007617909