g": 44 . . 44:44 “4.9.; in?! W449") 4.41m“ 44 *3: ..,...u— .141 SL4. ERIK-:4”. 2.244. 24». ..,. 2.3 3..- .4‘9..44, ‘14. 44. _ u I 2 . 43.94:. 44 .4144“. .rot‘h: 4 M944 .Lifi4f U449¢m444tx444m4 3492.19. 4.9... 4...“ “3.9814 “339... lhnquuHJm... .4.“..“43404 wr.,.41.....4 .44-n44...394.48.:4, 23.44:. 94 .mbMU 24.92 . 22 “H.234... ..v 4.4442 _ 2 494,45le] 4 2 4 24.49- ....4 49 43994 84“...“ {M33 444‘.“ 4:. .../44.92 4.94 0.4.» .2 $14. 449:... 458.444-3‘4n44’449u4 .94... . .. 4 . . 2. 2: 4 .. .2. 24:4...42 .444 4 . 2 2. .. 24: 8 ., 9.3.8.14 4244.349422244 . 24.4 424... x ..,”...2 .... 4.2....2z..4.4,...2._21.. 22.. .2. 2.2 ..2 4 24 . 4..... . 222......232 . 2 ......2..- ..2 2. _ .2... ...2»... 22.2142. .2 4 . 4'9- 4 ,u . c. 4 94 .42 4 9: 2 4 4 4 4 4 44U9h9442. .2433“. .4299- :8 ..44¢ 4.1., 444 8484944494 :41. .94.,4P4.4.4.:44 4 42.. 2 4,1444 4.1 .. .2 4449 s42 42. 42; .14 44 .44. 2f! 4.1.4444. 845?. 44“. 9 49\ 4488.444ka 42 2 293.48.. 4. ,‘8 . . , 2 39.9.48 n 44 . 4 .4 . :49,.4|.2. 4. 2 . . 8’44 . L4»: 444 . .2.... .94. 4.4.4244.- .44...:.4| . l9. .9 . .94. 4 44.17%»49'45.” wo- I4r.: 4 . , 44 494 : 2, . . . . :44 4.442. 9.4 4.4.. :9 424‘. . l. 9.1 to 4... 4 , 448.324.4244.) .4 41%.» 4484 .8.. ”48M”? 4 .3444: 4. . . ., : . 4. 094?”...42 . :44 .4 ,444,4,I4. 2.4.2. .. 4. . 444.24....22.::n: . 4.9.“.9444ufl4..342 4.4. ..w...9444.444u4.94 4..... 9449, 4 4 4 . . . 2 . 4 .2 2 . 44: . 4 4 . 4494. , 2 2 . : : I . ’4 . 4 .49 285442 . 8.43:. .r‘ifi.‘ 2 4, 3.94:2): .. . . .. ... ..44.4.4l. 4“ 42.4.4 . . 484 . 4 4 44.9.4}... .4 42.28.24. . 4. 4 4 4928.44! 4444.49.44“ «4L82444.4344.499 4 4.1.2... 4. .34.. 4. _ . 1.4.844. 44 .4 4 44.4.4 4 _ 2 2.. 92. ... . .... 4.444224414932949 49.. 2.4. .4... 44:49... It 49.41.514.344“. ..4... I . p 490. 4 . .. _ 9a 4 Co. .4 . . , . . 09 2. . 9 . . 2 . 4 .4'4484 4: O 2 . I 8 v 9 3342444.. .2499. 49p)“. 4 4 4.9- . .2 . .2 4.3,.C\ 44.49.441n-49 :4. .00.... 4 .4 . . . 2 .4444. 4 92!. 4. .. 4. 4. : 2 :2 4 : 94 9 .2 .48 ...4 4 45f? “444. «4 4~41J4Y49 9 4 44 r.» p) 4 45"? 4. c. 844. 4a.: , . . 693.4. . .89 .444... . .. 2444.2843444444944: 34. 4.: 4.4. . , 2, I‘f‘.9 4. 30...”..44894944 8912399444249L44 . £1.24tofl I 4. 4 2.9448414...” . 2 .49 439.444... 499 4, 4.44.4439. . 48. 222.949 . ._ . .43.. 94 4 . 2 2 4 .4444: 4 2 .944849. ...-I843! .4 8 4 p 44.4.1442. 989 98: '4. ..4 .94 4 .4894 2 . Jar-C. 41.4. 4'24 .48.. .PJ:\. 4: .423 :9 J..... 4444 . 4:! 4 ... . . 2 2 : 3444.44 .44 . 444.4. ..4423 4499 .... . 4| 2.. 44 144 4 4.41.4J2 . .4 4422849444.... . 4944 . , 83: O 4 «34434 4.4.. 3 .4 444,.” ... O4 94’. 8.99. 4 ..4. 2.9? ,v::. .27..“ .944 4.3 . 2 4 4. .244 94289499 44 34448.. 3. 6. -9.. 4 4 .412... 4 9434.44,... 44 ‘04 (4 44.. .4.. 104.4 9444 2.3 44.44... .4944...VI :.9 4.44 9.2.. .. 441.44....2. 2 2.42.9: ..4 ...... .22422. 2. 94 _ 444,. 44.544442441524244. 4.9 “I 4’3... 444's..4944 29419.4 4.339299%..9‘3818 2.44.9 1444.44.41.44... . 44.42....r 4. 2 . 2 4:944 ..4. 24.42. 4. 2 14:402. ..44..:..9 . 4.4:. ..4. :24. 24 .44.. 89 “.32.?4OMD-u9WO 9494848 4 :14 . . 444.44.94.840fi482940 29:4-u.. .144 4 . 2. L... .9994944:"4 . 34F: :4....8‘44..-.o .2.. ......4.4. ..4“..n4.446 4. 4.94.48.48 4.3 9:938. 4444 . 4C4 . . . ,. 2 . . . 4 2 2 2 , . .44 .34.: .4 . . 4. 4 .814 o 3. . 44:4 44 4». ”0'.qu 2 I944 :48“. 44, I94.44 44,... 40.9. . . , , 444 4'. 4.9. ,Ffl, 44. .4 314948494 .\44 4. 4 444.44. 4‘ 4, .4 o 4! 4.....F99.4J44l\4.914.442J~n.93\8-'.4 J:L-2..44089‘444444...449 . , . . 9 . '4. 4. : 2 ....49444-1949224 44.-9 .4423”. 494224.549. 7 9414412244.». J..3:l 842 4 444 492.322,. 4924.4 ..4. 4 .4”. 934.134.1849.. 4 404nu44442. 44,414... 44. 44440....“ 4.444.... ... 4. 5 ., .44 4 2.948 2 , .444414 4 :44. 4.4.. 494481.44. .9448944.4444244 149494 ‘49. .1994? 24. 4 ”9.44 ..., ,.44 44. .,...4 2 3. . ,.4 .... , . 22 79.4 49 32.9. 4: ..8443, 4L4: 3H4..24 4. .4. 44.84... .LP'CJM News 4 . .94. .. .4u‘44948. .9643444..4 64,4. 9.4:4 4 .44 .22 . 2 2 ,9). . 244.4 92.. . . 2 , Joni... . 4 294'... '394444. .4 4 4444. 4 48.44.19 494424. 34...: . .4444 349 4u4946 4 . .4 . . 844.44. 444,? 4344 a . \. J 4. . 494 8 : . . 2 4 .... .2 3 O. . 9 14.4..- 4 0'9. 4 49799239 4,, :4493 4 49.94.4494\94”. .39 .444 4.2.4 94.)! 4.4. ““4088; 46949428’4939 .3" .4 I. . 4. 34. . , AH",Y 4944.9434:‘4w~49.4.h049044 29808.!(47 ‘42 .849”... 94‘4298 c.1489r. I ,.. 4. . 2 2 .. 2 4&24442 .. , .9,.3434 84“ 449... .2 . , .. . 2 , _ 2 . . 4 44 .489 .42-48444.49. a 2 94?“. 9. .4. ”mg . . 4 999‘ 94. 2 A (*0\ WV! .444. 4 . 2 2 . 2 49:44 ...Vur‘. h” .99I..4¥49u4.,.ns.ho?4.4m\4WI4.r4.15}. 25.. .4 . . .24444' 2. 4.9 94.4.. 99 . .494. 4 .4 .. . 3&4”?! v 4 ”0mm. 2,4. 9.4. “4‘49! ». .9 ?~ 2 . . 22 Q. 49“.. .4439- 9.... _ 3. 4. ,... _. 4 4.4.1.2894 .- 84.4... 4 494.4: :4 .3 442 .4 .1 .44. 4 . 4. ..2191444..94,...J9. 4 (44 244:2“: 4.49.4...H_94r.44 . .2 . .2 4.4.44.4. 1 .24 .2, 44?..2294 ,. .4 4.444 4 2.: 4 .4 . .2. . ..2 . . 2. 2 4. 4 .4 .444. 4:: 49 2 ’4 1&3”, «chum 43:88.1 2‘4 944:: "2 v 894 .9 . 944 . . 43.4999 .4424. 24.44924. 9 2. 44.94.4244... 9494848444. 4494 94 2 4.443444 4.. n. o III-9'4 . 44. . 4 9.1 . 4 4.4 9.44 44 4.. 2443.494 28244944144834I4 249.4 4.44.... . 4.. . 49 2. .4 2 :9 499 .4-4 .' 2 . 3 2 4 . ,4 44.14:! 4 4 2 9... “1.4.2“ 4.2, 42:4”4 .- 945..» J..: 2 2.9.44. 14-249.929.84. ..4... .2789 ..4. 444”: . ..4 44 .9 44.22. . 2 44444....4144 41 42w..”"1.89ru;4a49§~ 5%,:‘1 ...4.~89. .4.. .... 22 . . .4.. . 4‘ 4.44 ’4 4. 4 n .99 ‘- 1.4.2.824..4:9 4 4.9.2.4532 4 8 1.9.4.3, 4 4 L88'4424‘44838v4‘ 4. 40’ 9." 4, 4, 8.448“. . 4 444 9,4. “.2. o2 44. 4.434444 4.44449...94444.. 894.. .2 9:4... 4. 49 ...... 4.. . 44. 44. 22444.2 .4, 4:4 . ,4494441....r 4.4.9.4442... :4 .4 484.24.)...”5.‘ 4‘4. .894». :4 ,4 444.42.4n144v4 git-u ,8 . 28. .49 , 42 ...2 84 4. .r 2 .3184! 44 . ,4 I: 4 994.414.4944. 84 8444.4 4. 23446424, 892.4 .LL4I . L .4.. 444-4... 4.. 44:4. 414 4 4 p29... 4.44.9.4 4 44 9.4 4 n9 v.” .49 . Wild. ..2 29’. 2 44er44rfl4u4n J,4 9.49:9...43 «44 44.498 .r..‘2. .2 .449 4.8:. 44.4.4. 39 .44. 44 .4 4. 4 . .... ..94. :44 . 4 4 44.4 4.: 41.44.33.398. 2. 2 4 .4944 842 .... ...9. 4.4.4 r44.h 9. .49l8.4:4.481 .4 8.... 9 .94 MR§W9.. ..4 4104q44§44m24m3 49932944...’ LL “W94. .04.“. 4.9 ou-Iu' ..4. 44,8. . 4 .4 . 494.44.. 4,4444. 44499 48 4.4.9.9 .294444r44444. | .4. .., 3. 4494842 2.9:. 499.. ..94 .444.- . H4 4 9 44.9.4189. Iv . a 443.844.944.43884494 44 498994444184 44. 3 144o(:4 3‘44 .9422: ”I. . :. 4 4P4}? 9'84“.“4 ..uufit. .5 44 4 . 2.44 4.4444 .. ..2. . 4. 4. 444424.... .44 :44. .2444. . 29.4,. 3 4.: 4. . 1 .28. 444441. .52 24L4V.8 99.4444. 4. 2) «9. 94,. 2 ... 9 ..4 ..uam) 2 449.994.144.343!"- 44. 8*. 9424.43,. iétw , . . :43. .4944 .4.)......! 4 4... 444.9 4 4 .. 4, 2 .4424 4 .64.. .344, 9449. 44- 289: 4.8 ”41.944.44.49 8449438344. 7929/ 3594,8424 41 4 9 .4 9:49qu 42.49. 93:94:44: 9.244934onv13u24.948o 2‘3 .44 ... 24.4., 4. 4.. . .. 42 2.44. .4. 9 4 .6 . ...4. 4. 08.9.4 4 ..4. .4.. 4, .2 8944 I4: .2 .4464. 4499 .4 44' 4‘48 “0.94““. a 4 444 2;“!- 2494484044. .44 994.391.44.44 .4. . . .4 ,, 2.494., 9444., 2.4‘ 2 4. .‘444. .4993 4.4.8.2” . :3: . 9 2 42.4444 .4: 4.4.9.4144. 4.. ..nfli. 4.11.44: 2. 4. 4.4.... 4244.4..4. . . 22.4: .. 2424:... . , 9 .. 2 19?. 4.: 3492..) u: 1 . psali 9: 3.3449519 .. “on"..uv... .4 . 44:34.44. 44 2 2 .. :4... . 2.4244 ..r. .4 4 .9«. 4. 4:94.484234279449: r... 9 .4 4444.. 4., 44444k ”$991143. 49". 2.4, 42 .4.. 2.449 :44. 2. . .4 ..2. . 4. 4:42.44. 22.4 .444. ,... .448; 4I 4.48.444 , 3, R4849 ..4 .44 4 .32‘1938 130' 4.01.. ..4 44448444» .44» 2 . . . . . 2 .4....2.4.4_,l444 2. 4489. 44994 4:9.» 4.26. 4., 4.4.84.8...44 44.. .2. .32.! .4 .2. LL34 4. 4. 4444 4. .44 .92“?! .242. .44. ..44444! .4.... “444a ..:. 44 8.4.4, 4 4 4&9 . .344. 22: .4 . 242:4! 4 94. 4, ..4 2 J 4.4.4445”: .. .9 w 4 72:4" 2.4. M 34462 4424 . n4 . 44 4 4r :2 1 .9). ,244 4.4. 4444/2444... 441 1494.43 24. 48.4.. 2“.4P4..444 3 44. .4 1.24.9442: rte: , ..444 .. ......4244324492919 .4 4. 4 4 4.94 .9“: .443 2 .4... «44 44. 4.4 1‘49" 4.». I. .43. 34:44.. 4 4'4. 2 .42 9.. 4,481. «..,-4444.. 8."..J44m .J. ...). ..9 .2 4.42 .4. .42. 2.... ..2... ..4 .9 .4.. 2. . 4..: 4 . 2 49241:?! 4 3.9. 2 444?. 44 4: :44- 4,444.44... 24:9 .9! 44 :v 99 444389 4.: I 444 WWW ..flunr4 4:924 .4 94 .485... 994.4442 2 o”. . 4:. 4 . .4.: 4». 44 .4 44484439849 4 2 ~...~43.o.49.’:4l44 .4. W454. 4. , 94 114.34%”..42 444 9.4 . 44.2: .33.!!944-241432‘4 4 .2944 “.r1443440h49u? 4.. 44243.. 44.44.44 4124} .. 4 2 2 2 . . . . . . ...9 4. . 4 ... . 2. ,4 4 4.4 3.494. . 4 4 . 4442.249. 2 . :54 . ..44,2.24V 4.44 . .44. 4449.4. .9 49122494444. 4.449. ..4.,.4 . 429. 44 4.. 4421. .244 1.142.294.4433. “2.24.14. ,4... . . . _ 2 . . .2....«424; . 94.... 444.44 .4.. 4. .44. .4949 2..... 4mm9 44.9, 4.44.9 4: 94244,..49 4 93.2. 24.48 2 9 4.4.42.3, ..4. 44.4 4.2449. 4.94 4.4.0 4. 498.431.34.24 3449.: “4.1999933: 4 1444:4148“ . la, 3:444 4. 4.4.9. . 2 2 2 2 2 . . 2 . . 9944 , 4v 2 V44 . ... 3'42 . D . 2:4: 244.84» 4 “4.4499. 2.44.44 ...L 4 44 41.4.4144 o 8:234 4499 44?? 4.4 28.4.9 44.44494J4444r..449424 .9 44.44494. I. 94 .fl... 24“. 9 . . 2 . . , 2 .. . I 2: 4 2 4.444. 444:. .4 . 8 1.4948 .4.-‘4 4:4 9 .245 . 4 H4. . .4 4 94.44. 91.44440: . 4“ gnaw? .24. burs? 4.14.4 £2,444 48 94 4 4 4.4.4 444. , 28 2. 4 4 9.... . 4. .4. .... 24 94:4. 2 94?. £4 4:4 42. 4H4}. .284 .9444. . 4 ..4 4. 442,4 \ 344449.: 44. a". 94.4"“. .I 2:4. 44. 1.94....214:4.44 4442.944 .-4 44 4484.. ....29 4. .2 , . .4494. 24... 2. 9 . ,...4224: 1.1.444 411,1. 44“ 449g. 44.4.49 4 4 .. «wt 244.: :44 :4,::.44 4:29.93: '94. .2 44M! 44 ..2 4:9 44(4- ..2 .4...9.224 4.442 .4 .2 4. 4.... 4 . . 9 ...:4....4.....9..4.:444442244.1..4 . 4944"... a 2 42....41‘r134 2.4%.. .44 4 ..4. ...4 442...,244445401442141..4....4449 $94.... .4: 4am“. iii: .4 . 43,4449 2.4. . . do 2... ,4. ...4. 4:. . .42I42 4.44 .42 . .24 .2. 2942444444. 4... .4.. 447919.... 44 4.. 44.... 44 4 ..9 4 434 .43 innit 4 .44“... . . 84V. 4 4 :_ 9 $4.4 4442444.. 4944.9 (9.348 , :3 4. 92.298! .94 444.. .8 .. .44 22 342 24 .. .4 911......: 41. .4149. .4.. 4.2 4.4.. 444:4..4844. .9 4.4442814. .4 :3. 74.9.8 44 L .944 4. 484m”! 4.424.," 4 .244..uJ 4 4454.”; .9444.4o4_4.4.44494:49,‘. .49 44, 6:. 13.4”»! 2 .4 49.29.4444 J51}... 2 2 ...4. b .. 42. ..2... 4 2' 44.42 4. :2 4.4 .44 49942.4. 444.44, 484. 4: 4442 4.2444 .., ,4 441449.34. 9 4.4491181344499944}... 4.: 83.. .9 4:41.444} .2344 .4.4: . .424 2h9n4uon4 Hula. 9.644.474. 4.74 24.44449. 44.“ 941444.432: .4. 2 4.49. ..44. .9) . . 44.. 4. 2 :9. ,.9 9 2:424. 4. .44.. .2 ,4... , 4... 4 2. 94. 2: .4 .922 .4411... .944 44.4 42:94 3.4.4.4239 2 184.8448). .4.” ~Up. 28.?qu 4 4. 4 9.34.23 I :44 .4484H44 4 ..I.444n42.9 474244 .4 3‘8). 44”. 2.34.9439: 3.:2 .4474...- 24 4944 .498“r 4 4 23'14 444.834.! 9.49. 2. . 44 .92 .444 44.4 ,..p 9444, . 2 :9: 9994483844 .8._.4..9N44.4.4, 44.4 95.93‘III 83494 2 2H4. 1”. ”444.4. 44.119439, .48- 4933944484.. .4l4,8194w".242. 8 9. 9. 84 4 ....33448’i4i4r3 9.449244 Prmrfi...‘ 2 944444. .244 44......242312 22.44 4... ..,4 44:: . ...2 . 4.. 44 4.9.4.5444 4. c 94. ..4444.1444..442 0.944294 244,.“ n 3282 44! 8 ,44.4 44. .4 44 3:44 9.: 442 3494.44 4,441 .4444. . .4 4. .4 92.2944. 4“. 4‘ 4 2 4:42.424. 4 I. .2. . 44 4. 2,44. 9 .9. 4444.84 44444., 4.2.392... 489 44. .2224. 44 :44 9 4.4. 2 ....445 4: ..m . 4 43.4 .44. .4}... 444444.. 4. 4.444 4:..Vr9124843r 89: 38.4889 .4! 4. .444: .. 44.2«W34v: .4 39.448 1:: 4.14.. .. u 2. ...: 4. 93¢“... 4.44.4.5 42 :9. 442:: 4.14.5697 .444, ,dl‘: 98 99 4 9 19.1. t.9’494.94.44l4 4 9 .44 .4. .4... 4 .4.. 42 4 4. n: 4.., 4.48 4 4.. 4. 3.. 9 44.{4...729r.9429444.401493. 989.4494: "4 ,L 9 . 9 W24 4 .. 84 x 9 2 . 4.84...4049l 4 9. s 4444 4088?, 43493.4... .4 (N2. . ' 0.. .4 .24184'99444 :k4r4f29‘4f4. . 4. 4 2 .49.- 4 9. 49 :44. .. : 4 4. ..4 .3 . , 5,289.42! .0 34.919...” 44.44942 4494 4.. 4:4...999f4419’3'9244’ 4.. 9438999389. .833: 94. 442. 4:.94944:J. 2 42 44.49 44 4 . fa». rfi‘ . 9 . .1. 44 4 .2 .9 44 9:4. 444. 9414442., 94!. 44 ...9. 244 4.. 4. 2 4 4. .. .4.... 434,419....{4 4.82. .4. 2 , . .324 2 . 3.144.493.9114 .494 2:94:97 79.444444. 983524999“ ”$204139“qu IMO! 4341.4»! ‘grvlirill !.§¥t4= 494. r8942...“ a39348499 “4.8994,. 49¢ 3929,98... 9843.492. 84 4 994... o 92 2. I .4. 4 9‘. .. . ..2 . ““4.,44J99444 4 )JJ...314‘.4448.24444,€49L. 4.284.983 9‘,..'88‘83’h8h4¥943 .43 4 2 u. 4 , . 2 . 4 2 v 9 . 4’8: 484 9 8 44 4. 5 48 : 2 ,2. 2 . . . 2 . 2 2. , .2 . '24 4.4. 4 o .4 4 9.44444 .. 44:44. .4. 1984' god-8944 ’4 .2....“44.. 4 4W444 424. :J9492444.:.. n”:hpflL~.§.-944443up48#..8744.44 4'. 3w... ”any“? 8&7 332‘34393 .4439! : 4 4 44 . 941 94.: 4.44.444. 2 . ...944... 4242.4 .9... 444. . 2.2.4.442 u... “44944.... fink: 4 24 1.: 449.4. .9994 .9334. 4.49.44 444944.: .244 .40.»! 43 4244 4 423.419. . :44 4 94 449404014924... 4E... .44.. 9.9449 4.8 44 .8. I 4 9.4.4.9983: 4 84. 9 44 993. 2.4 2...94 . .44-2 .’44.4.u848.4‘.4 4“... 94.. 9M .4! 144919.“ . . 44?"! . K: 4. 14.... 2. 94245424943344. 3942. 4482844 #44949422949443 44.3 I n 2 9.. 43.44.4443... 2 . ..,.4 9.9.33}... .2 2:94 4 43.2.1. in”??? 42 4.43:4 39.44944 4.... 2.4....24: 2934. o, 924‘” 44 8498. 448 ‘53 4: ml" 4 .2944 5.94:... .93 44l'.\:4 4. ,..}4 .4. 8494., 44444.3(. 4 v9 944 44449. 4.4.34: 8 444494.29 o .. 4.2342 .4 4 44 4. . .44493.44.4.429 4 49.4. 4,. .44.... 4 f 4.99.1944 ..,...Lliaul . 4204444434... 2144!: :4 4 ..2 fir.“ - .4444 .2... 44.2.... 22....4. I. 4.249 :, 4.4 :2 .44. I. .4 . .2944. 3.2.9.4. ... 4 ..424. 444.4: .. .2 :4 4 . .4 , 19.2.12: .4....4. .9..4...4....2..4., .1. ....92 4 4.1.»...4. 94.4 L 2.2 2.241.444 4.492.432: v , 69.". 2 898.8 8... 4.44952- 93: . 94. 14.8. .. 4'44gc49443454 r49. 4 44 4. 94. 4 . 4:9. .. 494 4 . 43.4. It 4 , ..V. ,44444.4.9:9484 . ..9. 394.949.3494 . ..444m 444 .5 4| 4 344.33.. 4.9 44494.9... .444ou2419l. .9 49.,.‘. 4 4 .5. 4 492... 41.4 .«.4.4444,4444.4.9.8 4.. . 34.4.1.4”. “"4949. 44 4 . 9 44.44...“ 4 8.44.49.24 494 .4 3993‘ J’u‘ v 2 . 4.4 2 .44 .43449924; 4294. 44 4 4 44, 43:4. 4 .24 :4 4.9 .44 .4,. 2.94 4444494.. ..2 .444... 4 . 99.1 44.92. .444 :4. .4 49 9 4.. 2 .. I\4.:.44o... 2. 2 4.44 74,448.44 2 4. .98 44 94:24 .49....4224 _. ...»..43 44.4. ..2. 4. 4.4.4.4... .4444 44.4428 .2 , .95 4. 4 . 4424444449. .44. 2 . 14.4.. 4 2.4.2.4...V441899z141. 2.44.12.49.94. 4. . $4411 . 8.. 4o 4 949 2449885.. 4948.: “449.4 344‘: 43.444 8 4 .3 (I. .3498 44.4 4 . I48 .2.!- 494.' 4.69 F. 144393. . 4 ..4 4. .944 944-..'. 94:9 ‘43 24484494394 4.843... .4 .0244 9.4. 413.4444... 4.4 62. a4: 4 2. . )4 4 .4284. 9444:. 2 . . :24: 2 892.3418. 4 4 43444 .4.. ..2-I944 9494 3. . 4.. .4.? 4 4:42 443 4 4,4 4.43:. 4 4.44. . 4.9.4384 . 2.4.‘3. 1.4 .9 449399. 2 .4 94... 2 .JK¢.43 1440.3. 9 Mr. . .9. 4. 4 4444,2344: 4' .1234. 2 4.4 .4.. . 2 . 4 9 44.499. :9 .AW' 2. 4n... 4.: 3. :29 3, .444} 3.94... , . 4 4» 29.3. .I. 448. .9. . 4 9913 4. 4,:44 I) +4.9. 3.4449”... 4 44 4.4.3.490. .. 93 .4.-.994 .44 49 48.144 . .94 4.4.4 949. .um; ..4. . 9.424... .3. 9.: v9.4, .4434: 49 4 ... a ...9 2 4, 44 ..4 .3 3.4944424. :34 4: 4494444. 4 . I. 42 .ui8.4.4. J.. .: 4..,2244 4 b.4244 9... 494.34 £44384 ..44.42. ...4 4, 4:44:42 ..4 ,4 4 ... 4 . 4.4., I392.-. .- 4.4444 .4444. .. 44 4.44.424. 4... .224 _ 4.2.44 4. 2.4 4... .. 9.242.422”... ... . 4.2.43 934 44:49:, : 5 . 2 2 .449 .2. I. 443 2 . 4 . 4N4.“ .9.42 344.444 «42, 4.9.4043! 4. ..44..n9.4494.2. .29.. 34:88: 4 . ...... .4 2, 44484494.. 4 44944.4 4. 4 .44. 42.42 . o. . 344444;. ..4-4 94.4 9 394‘ 14 94 . 448“ .9444- 2Hr j—l A level of significance was pre-determined as equal to or below 0.05 for the test of variability (Q-statistic). A Q—statistic value having a probability level exceeding 0.05 was not regarded as significant, hence the variability of effect sizes within the group would be regarded as adequately homogeneous. When a group of effect sizes are relatively homogeneous an overall grand mean effect size for this group is more easily interpreted as representing this group of studies. If on the other hand a Q-statistic value has a probability level equal to or below 0.05, this indicates a significant amount of heterogeneity among effect sizes within the group suggesting interpretation of an overall grand mean effect size for this group would be too difficult to make. Too much heterogeneity suggests there are other variables impacting effect size results among some studies that are not adequately accounted for by this grouping. In this latter case, the meta-analysts job then becomes one of investigating possible sources of difference among effect Sizes for this group of studies to uncover sources of too much heterogeneity (Borenstein et al., 2009; Lipsey & Wilson, 2001a; Cooper & Hedges, 1994). The hypotheses for this study served as a means of testing multiple variables identified a priori for purposes of uncovering unique relationships within the accumulated data. 55 When the meta-analyst finds comparable characteristics behave or operate similarly across various studies of the same intervention, this provides accumulated evidence about what really matters relative to producing the most desirable results from an intervention (Dunst et al., 2004). 56 CHAPTER 4 RESULTS Findings from Systematic Selection of Research Studies To locate a representative set of research studies for potential inclusion in this meta-analysis, I utilized multiple strategies including computerized searches of several databases, manual inspections of reference lists as well as reviewing websites listing research by authors who have published in this area of study (Durlak, 2003; Lucas & ’ Cutspec, 2005). Figure 1 provides the steps followed for this search. A comprehensive electronic search of the professional literature used PsycInfo, ERIC, MEDline and GoogleScholar using the term, “Incredible Years”. These combined methods yielded 2,525 citations (495 from PsycInfo, ERIC, MEDline and 2,030 from GoogleScholar). Abstracts were reviewed electronically to locate studies for possible inclusion in this meta—analysis. Where there was not a specific parent training program mentioned in the abstract, the article was reviewed to determine if IYPTP was the intervention studied. This procedure yielded 85 studies for further inspection. Cross-referencing was also employed by hand searching lists of references in articles and book chapters and also checking the Incredible Years website (www.incredibleyears.com) where there are published and unpublished studies available. The latter procedure resulted in 6 more studies. A total of 91 studies were reviewed to determine whether they met criteria for inclusion in this meta-analysis. Inspection of the 91 studies led to exclusion of 52 studies (57 %) and the inclusion of 39 studies (43%). Studies were excluded from this meta-analysis for seven primary reasons. Fifteen studies ( 16%) did not provide the needed scores for this meta-analysis (e.g. means, 57 standard deviations and sample sizes). Another fifteen studies (16%) provided outcome measures not within the scope of this meta-analysis (e. g. parent self-efficacy and child autonomy). Nine of the studies (10%) were excluded because IYPTP effects were confounded by the inclusion of additional interventions (e. g. individual child intervention, summer programs, or school interventions). Five studies (5%) were excluded because they were single-case study designs. Four studies (4%) were excluded because they reported on data already included within a prior, included study. Three studies (3%) were excluded because they were qualitative studies. One study (1%) was excluded because the treatment was significantly diluted (only provided two sessions of IYPTP). This summary of reasons for exclusion provides some confidence that the sampling for this meta-analysis did not exclude any one type of study at a significant magnitude. 58 Figure 1: Literature search process for locating and selecting studies: STEP 1: Electronically Searched for Terms “Incredible Years” V Electronic database searches (PsycINFO; ERIC, MEDline) = 495 citations I WWW Search Engine (Scholar.Google) = 2,030 citations STEP 2: Examined 2,525 Abstracts to identify studies using Incredible Years Parent Training Program (IYPTP) Note: Abstracts not naming a specific parent intervention led to inspection of the entire article to either rule-in or rule-out IYPTP STEP 3: Selected and reviewed 85 studies for possible inclusion according to pre-defined criteria Step 4: Cross-Reference Search: Included electronic search by author names associated with IYPTP (i.e. Webster-Stratton, Reid, Hutchings, Scott, etc,. . .); hand search of references listed by located studies and within book chapters written by Webster-Stratton; reviewed www.incredibleyears.com listings; This yielded 6 additional studies not previously identified / STEP 5: Excluded 52 studies for one of seven different primary reasons: 1. Score type not provided (n = 15) 2. Outcome measures not within scope of this study (n = 15) 3. Treatment effects confounded by other treatments added (11 = 9) . Qualitative or narrative study (n= 3) . Single-case studies (n = 5) . Signif. dilution of intervention (n=l) . Study duplicates data from other studies already included (n = 4) \IO\UI-b 59 l Excluded Primary Studies Replication Studies STEP 6: 39 studies appropriate for inclusion within meta-analysis /\ Included 14 Primary Studies Included 25 Replication Studies Coding of Study Variables Each study accepted for this meta-analysis was reviewed using a coding form for purposes of gathering data relevant to a priori hypotheses. This coding form may be reviewed within Appendix A. Examination of Possible Bias in Results for Between-Group Studies Studies that report larger effect sizes are more likely to be submitted and accepted for publication than studies producing small to moderate effects (Kratochwill & Shemoff, 2004). This trend results in a less than random selection process whereby studies included in a meta-analysis likely overestimate the true effect of the intervention studied. It is important to assess the extent to which results of a meta-analysis may have publication bias and subsequently consider the potential impact on conclusions drawn as a result of this consideration. Before conducting this analysis it is important to recognize that effect sizes generated from between-groups and within-group studies are generated quite differently, thus requiring not only separate meta-analyses but also separate tests of publication bias. Two methods were employed to evaluate possible publication bias for both types of studies. First, a funnel plot was constructed as a graphic means of examining the distribution of effect sizes found in each study relative to their standard error. When publication bias is absent, the plot of studies should be distributed symmetrically about the combined effect size (depicted as a perpendicular line) such that the plot resembles a proportioned funnel shape. If there is a higher concentration of studies plotted on the side of the combined effect size representing favorable change, this would reflect the presence 60 of some publication bias. When bias is present smaller studies reporting statistically significant findings are published and included in the meta-analysis, while there is a presumption that other smaller studies that did not yield statistically Significant results were not published and are therefore not included within the study. Sampling is supposed to be random and include the population of all studies, not just those reporting larger effect sizes. For this study desirable effects are negative, reflecting reductions in child conduct behavior problems. Examination of the funnel plot for between-group studies (Figure 2) found most included studies fell within a moderate size range as depicted by the clustering in the middle of the graph. At the very bottom of the graph there is more representation of favorable effects among some smaller studies perhaps suggesting some undue bias of publication effect. However the impact of these studies appears negligible in relation to the weight of the symmetrical bulk of larger studies falling in the middle of the graph as well as a larger study falling toward a less desirable effect. 61 Figure 2: Funnel plot for between-groups studies: Precision (1lStd Err) 10 -2.0 -1.5 -1.0 Funnel Plot of Precision by Hedges's g -0.5 0.0 0.5 1.0 1.5 2.0 62 To further consider possible publication bias a second method of analysis was employed. Rather than speculating about the potential impact of missing, unpublished studies remaining in a “file drawer,” Rosenthal (1979) recommends a method for calculating the number of missing studies that would lead to nullifying the found effect in the analysis. If this number is quite small there is more reason for concern given their potential to change the overall outcome of the study, making conclusions based on the current results more tenuous. If on the other hand a large number of studies would be required to nullify the effect, then conclusions using the studies already obtained may be made more confidently even though there may be some slight inflation incurred by using only published studies. The between-group studies included within this meta-analysis yielded a Z-value of — 10.24 and corresponding p-value of 0.0000. The fail-safe N calculation resulted in a figure of 657. This means we would need to locate 657 ‘null’ studies in order for the combined 2-tailed p-value to exceed 0.050 significance. It seems unlikely there are that many studies finding no effects. The fail-safe N supports the interpretation of the funnel plot, suggesting that publication bias is not a problem in this current study. It should also be noted that the traditional fail-safe N algorithm uses probability levels (p-values) for each study, and then combines those values. The more accepted method that was used for this research, calculates effect sizes for each study, combine these, and then compute the p-value for the combined effect. Examination of the funnel plot for within—group studies (Figure 3) included in this meta-analysis shows six of the seven studies produced very similar benefit for reductions in conduct problems, at a moderate level. The outlier score showing a more significant, 63 desirable result is actually an unpublished study which would contraindicate the hypothesis of there being publication bias. 64 Figure 3: Funnel plot for within-groups studies Precision (1lStd Err) -20 Funnel Plot of Precision by Hedges'sg -1.5 -1.0 0.5 0.0 0.5 1.0 15 Hedges'sg 65 2.0 The within-group studies used in this meta-analysis included two unpublished studies, thus two of the four replication studies were unpublished (Clondalkin, n.d. & Rogers, 2007). The seven within-group studies in this meta-analysis yielded a Z-value of -6.81915 and corresponding p-value of 0.0000. The fail-safe N calculation resulted in a figure of 78. This means we would need to locate 78 ‘null’ studies in order for the combined 2-tailed p-value to exceed 0.050 significance. It seems unlikely there are that many studies finding no effects. The fail-safe N supports the interpretation of the funnel plot, suggesting publication bias in this study is not a significant problem. It should also be noted here that the traditional fail-safe N algorithm uses probability levels (p-values) for each study, and then combines those values. Whereas the more accepted method that was used for this study is to calculate effect sizes for each study, combine these, and then compute the p-value for the combined effect. Primary Hypotheses Test Findings For this meta-analysis statistical tests were conducted separately for studies using between-group comparisons and studies using within-group comparisons, since the effect sizes were calculated differently for each type of study and would not be comparable. Lipsey and Wilson (1993) found that effect sizes for studies using within-group comparisons yielded significantly larger benefits than those using between-group comparisons where control groups were used. This led them to conclude that one group pre/post within-group studies may inflate true effects from an intervention compared with those studies using a control group. Maughan and colleagues (2005) meta-analysis of behavioral parent training intervention found similar results, further supporting the parceling of studies according to design. 66 Research Question 1: Are significant differences between reductions in child conduct problems for primary and replication research studies using the Incredible Years Parent Training Program (IYPTP) previously found in the preliminary meta-analysis (Sougstad et al., 2008) retained with the addition of a broader range of studies produced over the last 2.5 years? Hypothesis la: It was hypothesized for studies measuring effects from IYPTP that the grand mean effect size representative of a group of primary studies conducted by inventor Dr. Carolyn Webster-Stratton and her colleagues at the University of Washington would Show significantly greater reductions in child conduct behavior problems than the grand mean effect size representative of replication studies, at a 0.05 level of statistical significance. Results for Hypothesis Test la: Given the principal hypothesis for this study that primary and replication studies would yield significantly different effect sizes, a test of homogeneity was calculated for each group (primary and replication studies) separately to ensure that the grand mean effect size depicted adequately represents each group of effect sizes. The Q-statistic tests whether the observed variance among effect sizes within a group are larger than expected from sampling error. The effect size for each study estimates the true value of a grand mean effect size for a group of studies. To report a grand mean effect size as representative of a group of studies, the effect sizes for studies within that group should display an acceptable level of homogeneity. Statistically-significant heterogeneity contraindicating the reporting of a grand mean effect size for a group of studies will be indicated by a Q-Statistic below the pre-defined cut-off of a p-value of 0.050. 67 Table 2 provides combined effect sizes (Hedges g) and associated statistics for each within-groups study of IYPTP included in this meta-analysis. Studies were grouped according to whether they were a primary or a replication study. A test of homogeneity was conducted for each group of primary and replication within-group studies using a Q- statistic. For the primary within-group studies of IYPTP the Q-value of 4.613 with 2 degrees of freedom yielded a non-significant p-value of0. 100. Similarly the Q-value of 2.172 with 5 degrees of freedom yielded a non-significant p-value of 0.825 for the replication studies of IYPTP. Since the grand means representing each group of studies appears to represent a fairly homogeneous set of findings, these may be compared statistically to determine their degree of similarity. Borenstein and colleagues (2009) advocate use of a Mixed-Effects Model for comparing subgroup means, which uses the Random-Effects Model within subgroups and a Fixed-Effect Model across subgroups. Using Mixed-Effects Analysis to compare the grand mean effect size for the primary and replication studies yielded a Q—value of 0.261 with 1 degree of freedom and a non-significant p-value of 0.609. Additionally, for illustration purposes the confidence interval for the Hedges’s g effect Size -0.727 (-1.115 to -0.338 with 95% confidence) for primary studies overlaps with the confidence interval for the replication studies, Hedges’s g effect size -0.616 (- 0.787 to -0.445 with 95% confidence). This significant amount of overlap of about one- half standard deviation further demonstrates the non-significant p-value showing no statistically-significant difference in reductions for child conduct problems between primary and replication within-groups studies of IYPTP. The grand mean effect size of - 0.634 (-0.791 to -0.478 with 95% confidence) indicates a moderate benefit accrued from 68 IYPTP, with child conduct problems reduced by about two-thirds of a standard deviation for within-group studies. Table 2: Within-groups IYPTP studies, immediate reductions in child conduct problems Hedges’s g Stud. Variance Lower Upper z-value p-value Error limit limit Primary Studies: W-S & Shoecraft 2009 -0.488 0.230 0.053 0.940 -0.037 ~2.1 18 0.034 W-S, 1982b -0.526 0.307 0.094 -1 .128 0.076 -l.713 0.087 W-S, 1994 -1.026 0.160 0.026 -1.340 -0.712 -6.401 0.000 Primary,Random Model: -0. 727 0.198 0.039 -I.115 -0.338 -3. 667 0. 000 Replication Studies: Axberg, 2007 -0.596 0.186 0.034 -0.960 -0.232 -3.210 0.001 Clondalkin, 2004 —0.984 0.359 0.129 -1 .689 -0.280 -2.740 0.006 Fergusson et al., 2009 -0.67 7 0.140 0.020 -0.952 -0.402 -4.829 0.000 Manby, 2005 -0.564 0.279 0.078 -l.1 11 —0.017 -2.022 0.043 McIntyre, 2008 -0.328 0.382 0.146 -l.077 0.420 -0.860 0.390 Roars, 2007 -0.502 0.204 0.041 -0.901 -0. 103 -2.464 0.014 Replication Random Model: -0.616 0. 087 0.008 -0. 787 -0. 445 -7.061 0. 000 Overall Primary & Replication. -0. 634 0.080 0. 006 -0. 791 -0.4 78 -7.940 0.000 Random Model: Table 3 provides a listing of effect sizes (Hedges’s g) and associated statistics for each between-groups study included within this meta-analysis, along with the summary grand mean effect size for each group of studies (primary versus replication). Given the principal hypothesis for this study, that primary and replication studies would yield significantly different effect sizes a test of homogeneity was calculated for each group separately to ensure that the grand mean effect size depicted in Table 3 adequately 69 represents each group of effect sizes. The Q-value for primary studies of 42.940 with 8 degrees of freedom yielded a significant p-value of 0.000. This finding indicates this group of primary studies has too much variance to use a grand mean effect size to represent these studies. Similarly the Q—value for replication studies of 28.487 with 14 degrees of freedom yielded a significant p-value of 0.012. This finding also indicates too much variance to use a grand mean effect size to summarize these studies. Given these results for between-groups studies of IYPTP, further meta-analysis is required prior to concluding whether there is a significant difference between the benefits of primary and replication studies for IYPTP. 70 Table 3: Between-Groups Studies for IYPTP Immediate Reduction in Child Conduct Problems I Hedges’s g I Stud. I Variance I Lower Upper z-value p-value Error limit limit Primary Studies: Goss, Fogg, 0.104 0.171 0.029 - 0.231 0.439 0.61 1 0.541 Web-St. 2003 Kim, Cain, Web- - 0.228 0.374 0.140 - 0.962 0.505 - 0.610 0.542 St. 2008 Webster—Stratton - 0.528 0.327 0.107 - 1.169 0.114 - 1.611 0.107 19823 Webster-Stratton - 0.871 0.396 0.157 - 1.647 - 0.094 - 2.198 0.028 1984 Webster-Stratton - 0.210 0.107 0.012 - 0.420 0.001 - 1.950 0.051 1998 Webster-Stratton - 0.160 0.132 0.017 - 0.418 0.099 - 1.212 0.226 2001 Webster-Stratton - 1.009 0.193 0.037 - 1.388 - 0.631 - 5.227 0.000 et al 2004 Webster-Stratton - 0.827 0.206 0.042 - 1.230 - 0.423 — 4.018 0.000 et a1 1988 Webster-Stratton - 1.189 0.230 0.053 - 1.640 - 0.738 - 5.168 0.000 et a1 1997 Random Model: -0. 524 0.155 0. 024 -0. 827 -0. 221 -3.389 0. 001 Replication Studies: Brotman, Gouley 0.357 0.207 0.043 - 0.049 0.763 1.723 0.085 eta12005 . Brotman, Klein -0.701 0.351 0.124 - 1.389 - 0.012 - 1.993 0.046 et a1 2003 Bywater et al In - 0.066 0.292 0.085 - 0.637 0.506 - 0.225 0.822 Press Gardner, Burton, -0.652 0.251 0.063 - 1.143 - 0.161 - 2.602 0.009 Klimes 2006 Hutchings, - 0.656 0.183 0.034 - 1.016 - 0.297 - 3.576 0.000 Bywater et al 2007 Jones, Daley et - 0.713 0.234 0.055 - 1.171 - 0.255 - 3.050 0.002 a1 2007 Larsson, Fossum - 0.558 0.184 0.034 - 0.918 - 0.198 - 3.038 0.002 et a1 2009 LeTarte et al - 0.459 0.367 0.135 — 1.179 0.261 - 1.250 0.211 2010 McIntyre 2008 -0.246 0.290 0.084 - 0.813 0.322 - 0.848 0.396 Patterson, 0222 0.200 0.040 - 0.613 0.170 - 1.109 0.268 Barlow et al 2002 Scott, O’Connor - 0.010 0.164 0.027 - 0.331 0.31 l — 0.061 0.952 eta12006 Scott, Spender et - 0.238 0.232 0.054 - 0.692 0.216 - 1.027 0.305 a1 2010 Scott, Sylva, et a1 - 0.417 0.202 0.041 - 0.813 - 0.020 - 2.059 0.039 2010 Taylor et al 1998 - 0.775 0.342 0.1 17 - 1.446 - 0.104 - 2.263 0.024 Random Model: - 0.362 0.089 0.008 - 0.537 - 0.188 - 4.075 0. 000 71 Primary and replication studies of IYPTP can only be compared fairly when studies using different types and usages for IYPTP are matched. IYPTP between-groups studies contain one of two IYPTP forms (group-administered and self-administered) whereas the within-groups studies only included group-administered IYPTP. The group- administered IYPTP in its Basic form includes roughly 10-12 sessions with a group of parents conducted by co-leaders (Webster-Stratton, 1998). The self-administered form of IYPTP has parents view the same video vignettes that are used in the group form, but to facilitate integration of content each parent completes self-administered workbooks without any group intervention or regular contact with a therapist or group leader (Webster-Stratton, 1988). There are studies included within this meta-analysis that examined the effects of the self-administered form of IYPTP. Table 4 includes the 3 primary studies of the self- administered form of IYPTP. No replication studies met inclusion criteria for this study. Kratochwill, Elliot, Loitz, Sladeczek & Carlson (2003) reported on the use of the self- administered form of IYPTP but their child outcome variable was an omnibus measure that included both externalizing and internalizing problems. The current meta-analysis is restricted to only those child outcome measures having to do with externalizing conduct problems. The only other two studies located employing the self-administered form of IYPTP, were single-case studies (Ogg & Carlson, 2009, Walcott, Carlson & Beamon, 2009). Case studies were excluded from this meta-analysis. Examination of the references listed within each of these studies found no additional citations for studies using the individually-administered IYPTP, not already considered for this meta-analysis. For the primary studies using the self-administered form of IYPTP the Q—value of 0.568 72 with 2 degrees of freedom and a p-value of 0.753 showed good homogeneity among these studies results. The resulting effect size (Hedges’ g) for the primary, individually- administered IYPTP is -0.531 (-0.737 to -0.324, with 95% confidence). This would indicate that the average child benefiting from self-administered parent training improved their behavior at a level superior to roughly 70 % of those in control groups who did not receive this intervention. Table 4: Between-flups, Self-Administered IYPTP (primary studies only available) I Hedges’s g I Stnd. I Variance Lower I Upper z-value p-value Error limit limit Primary Studies: W-S, 1990 -0.531 0.201 0.040 -0.925 -0.137 -2.642 0.008 W-S, 1992 -0.604 0.157 0.025 0912 -0.296 -3.841 0.000 W-S et a1, 1988 -0.412 0.201 0.040 -0.805 -0.018 -2.052 0.040 Primary, Random -0. 531 0.105 0.011 -0. 737 -0.324 -5. 037 0. 000 Model: The absence of independent replication of between-groups studies of the self- administered form of IYPTP prevents any comparison with an effect size representing primary studies of self-administered IYPTP. Finding no published independent between- group comparisons of the self-administered form of IYPTP was surprising given the Si gnificant benefits reported by the primary studies, and the likelihood that the self- administered form may be more easily replicated without needing therapist intervention. In addition to recognizing two forms of IYPTP (self-administered and group- administered) Webster-Stratton (1998b) also distinguishes between two primary uses of IYPTP. There is a clinical treatment of families who have a child exhibiting significant conduct problems and a second form used as community prevention to improve parenting and child social functioning (e. g. Head Start families). Both forms of IYPTP include four primary components: 73 1.) Interactive play and involvement 2.) Reinforcement techniques like praise and rewards 3.) Limit setting 4.) Discipline (e. g. nonviolent time-out, ignoring, logical and natural consequences) A close inspection of all between-groups studies using the group—administered form of IYPTP included in this meta-analysis (Table 3) led to a determination that the group-administered form of IYPTP is actually used for not just two purposes as was suggested by Webster-Stratton in 1998, but has actually been studied for three purposes. Studies of IYPTP fit within a public-health model inclusive of three-tiers of intervention. The prior, preliminary meta-analysis of IYPTP (Sougstad et al., 2008) only distinguished between the 2 original uses but Figure 4 illustrates the three-tiered usage of IYPTP. First, studies examine IYPTP as a form of Tier One, primary, universal- prevention (e. g. within Head Start and other preschool environments) to potentially “inoculate” youth and families against future conduct problem development. Second, studies examine IYPTP as a form of Tier Two selective-interventions provided to youth and families showing risk factors known to be associated with the development of conduct problems. Third, studies examine IYPTP as a Tier Three indicated-intervention to address cases where clinically-significant conduct symptoms are present requiring the most intensive intervention. 74 Figure 4: Three-Tiered Usage of Incredible Years Parent Training Program: Tier I: Indicated Treatment Severe CP Tier II: Selective Intervention Targeting Known Risk Factors and entire range of Conduct Problems Tier [11: Universal Prevention Inoculation to Prevent Potential Risk Factors from having Deleterious Effects; Level of Conduct Problems Not Directing Intervention Table 5 summarizes the primary and replication studies where IYPTP was used as a Tier One intervention. These studies were grouped together because the selection of intervention recipients for these studies was not determined by a risk factor uniquely associated with the development of conduct problems, or according to a score on a 75 clinical measure of conduct problems. For example, Webster-Stratton, Reid & Hammond (2001) studied delivery of IYPTP with 272 mothers having a child attending Head Start. Yet Webster-Stratton & Hammond (1998b) found a sample of 394 Head Start families in the Northwest region of the United States where their studies were conducted, consisted of only 35% possessing at least three or more risk factors (e.g., single parenthood, poverty, depression, life stress, psychiatric illness, parent history of drug abuse, child abuse and spouse abuse), and between 40-45% having mothers who display high rates of harsh or physically negative parenting. Since it is estimated that the majority of the subjects in these two Head Start studies did not display risk factors at the aforementioned magnitudes, and IYPTP was provided at the Head Start Center level (not selected toward any group of at-risk students or families identified by a number of risk factors or conduct symptoms within the Head Start population) it was concluded this type of study is actually a universal prevention study. Similarly, G053 and colleagues (2003), including Dr. Webster-Stratton provided IYPTP to parents of children at day care centers where there was again poverty and other risk factors evident, but many of these are risk factors for not only conduct problems but also many other forms of child and family dysfunction. Scott, O’Connor and Futh (2006) provided IYPTP across multiple settings identified for their impoverished circumstances in the United Kingdom (UK), similar to the Head Start studies in the USA. The UK. study deployed a gating procedure for making sure that the most high-risk students/families received intervention before those in less need. However, once these higher risk subjects were ensured of immediate intervention they were mixed into groups with lower risk subjects. While this step ensured an ethically supportable expediency to treatment delivery for those most in need, 76 the intervention effects were measured by overall changes by groups that were not defined by risk factors unique to conduct problems in children. It is reasonable to assume primary prevention studies included some higher risk students/families but in all of these studies the unit of analysis was change over time from treatment (versus control groups) at a level not unique to any particular characteristic other than circumstances of poverty entitling them to a preschool education program. Included along with these primary prevention studies was research by Kim, Cain and Webster-Stratton (2008) and McIntyre (2008b), who delivered IYPTP as demonstration projects with populations not known to be at high risk for conduct problems, but nevertheless may benefit from parent training (i.e. Korean mothers; Parents of children with a developmental disability such as Autism or Mental Retardation). The primary studies’ Q-Statistic of 2.541 with 3 degrees of freedom yielded a non-significant p-value of 0.468 indicating a roughly homogeneous set of effect sizes for this group. Similarly the Q-Statistic of 0.502 with 1 degree freedom yielded a non- significant p-value of 0.479 indicative of an absence of excessive heterogeneity. These results suggest it is reasonable to statistically compare the grand mean effect Size representing each group, according to the predicted hypothesis of there being a significant benefit from primary over replication studies. Comparison of the grand mean effect sizes (Hedges’s g) for primary and replication studies using IYPTP as a tier one prevention program using Mixed-Effects Analysis yielded a Q-statistic of 0.191 with 1 degree of freedom and a non-significant p- value of 0.662. Therefore there is no statistically-significant difference between the benefits reported by these two groups of studies. 77 The obtained Hedge’s g effect size of -0. 122 suggests a very small benefit in the reduction of conduct problems for IYPTP used for primary prevention. For this group of studies, the unknown, true effect size lies between -0.250 and 0.006, ninety-five times out of one-hundred (95% confidence interval). Considering the Tier One, universal-primary prevention group of studies includes subjects where conduct problems were not identified as being highly problematic, this small effect size is not at all surprising. Table 5: IYPTP Tier One, Primary-Universal Prevention @etween-Igroups studies) I Hedges’s g I Stnd. I Variance Lower I Upper z-value p-value Error limit limit Primary Studies: Goss, Fogg, 0.104 0.171 0.029 -0.231 0.439 0.611 0.541 W-S, 2003 Kim, Cain, -0.228 0.374 0.140 -0.962 0.505 -0.610 0.542 W-S, 2008 W-S, 1998 -0.210 0.107 0.012 -0.420 0.001 -l.950 0.051 W-S, 2001 -0.160 0.132 0.017 -0.418 0.099 -1.212 0.226 Primary, Random Model: -0. I3 7 0. 0 73 0. 005 -0. 281 0. 00 7 -1. 885 0. 062 Replication Studies: McIntyre, 2008 -0.246 0.290 0.084 -0.813 0.322 0.848 0.396 Scott et al., 2006 -0.010 0.164 0.027 -0.331 0.31 1 -0.061 0.952 Replications, Random Model: -0. 06 7 0.142 0. 020 -0.346 0.212 -0.4 70 0.638 Combined Primary & Replications, -0. 122 0.065 0.004 -0. 250 0. 006 -I.873 0. 06 I Random Model Table 6 contains those studies examining the use IYPTP as Tier Two selective intervention targeting groups Specific to a particular risk factor(s). Inclusion in the Tier Three indicated intervention group required either a minimum cut-off score for clinically- significant conduct problem severity or clinic referral to treat high levels of conduct problems which the Tier Two group of studies does not contain. Tier Two studies are also differentiated from the aforementioned Tier One prevention studies because IYPTP is being used to specifically target groups where parenting and/or child functioning are 78 known to be at least somewhat problematic, suggesting an at-risk for conduct problems Status. The studies by Brotman and colleagues (2003 and 2005) studied IYPTP effects on children having a formal record of antisocial behavior in their immediate family history (e.g. adjudicated sibling, etc,. . .). Bywater and colleagues (In press) studied IYPTP effects on children served by foster care parents, where they cite about four times the rate of conduct disorder is found among this population than in the general population of the United Kingdom. Nilsen (2007) also studied the effects of IYPTP on foster parents. LeTarte and colleagues (2010) studied the effects of IYPTP in parents known to be neglectful and were already being served within the child welfare system in Canada. Patterson and colleagues (2002) studied IYPTP delivered to parents of children known to be above the 50th percentile on a behavior inventory measuring conduct problems. Because this cut-off score is well below the clinically-significant range of conduct problems (falls within normal limits) this study was regarded as a form of selective intervention where children/families were selected because of an at-risk status rather than a clinical level of severity separate from most in the general pOpulation. There were no primary studies regarded as falling within a Tier Two level of selective intervention. 79 Table 6: IYPTP Tier Two, Selective Intervention Targeting At-Risk Youth/Families (between-groups studies) Hedges’sgI Stnd. I Variance I Lower I Upper I z-value I p-value Error limit limit Primal-y Studies: NONE Replication Studies: Brotman et 0.357 0.207 0.043 -0.049 0.763 1.723 0.085 al., 2005 Brotman et —0.701 0.351 0.124 -1.389 -0.012 -l.993 0.046 al., 2003 Bywater et -0.066 0.292 0.085 -0.637 0.506 -0.225 0.822 al., In Press Letarte et al., -0.459 0.367 0.135 -l.l79 0.261 -1.250 0.21 l 2010 Nilsen, 2007 -0.495 0.380 0.145 -1 .240 0.250 -1 .302 0.193 Patterson et -0.222 0.200 0.040 -0.613 0.170 -1.109 0.268 al., 2002 Replications, Random -0.1 95 0. 165 0.02 7 -0. 519 0.129 -1.1 79 0.239 Model: Examination of Table 6 found the Brotman et a1. (2005) study yielded a Hedges’s g effect size that is a substantial outlier to all other studies in this group (and that of other groups as well). The only measure comprising this effect size was based on a tool developed by the researchers called Observed Peer Play in Unfamiliar Settings (OPPUS) which is quite different from the other parent rating scale and independent observation measures included within other studies examined for this analysis. Because this measure accounted for 22% of the group mean effect size this study was removed. 80 Table 7: IYPTP Tier Two, Selective Intervention Targeting At-Risk Youth/Families with Outlier Removed (between- roups studies) Hedges’s g I Stnd. Variance I Lower Upper I z—value p-value Error limit limit Primary Studies: NONE Replication Studies: Brotman et -0.701 0.351 0.124 -1.389 -0.012 -l.993 0.046 al., 2003 Bywater et -0.066 0.292 0.085 -0.637 0.506 -0.225 0.822 al., In Press Letarte et al., -0.459 0.367 0.135 -1.l79 0.261 -1.250 0.21 l 2010 Nilsen, 2007 -0.495 0.380 0.145 -l.240 0.250 -1.302 0.193 Patterson et -0.222 0.200 0.040 -0.613 0.170 -1.109 0.268 31,2002 Replications, Random -0.318 0.130 0.01 7 -0.5 73 -0. 063 -2.445 0. 014 Model: Table 7 more accurately represents the effects for this group of Tier Two selective intervention studies with the previously identified outlier removed. The Q—value of 2.531 with 4 degrees of freedom yielded a non-significant p-value of 0.639. This non- significant p-value at the 0.05 level indicates an acceptable level of homogeneity to use a grand mean effect size to represent this group of studies. The Tier Two selective intervention studies using IYPTP yielded a Hedges’s g effect size of -0.318 with a 95% confidence interval between -0.573 to 0063. On average this represents about a one- third standard deviation benefit from IYPTP Tier Two. The average child benefiting from IYPTP as a Tier Two intervention improved their behavior at a level superior to roughly 66 % of those in control groups who did not receive this intervention. Table 8 shows Tier Three indicated intervention between-group studies in which IYPTP was used to treat clinically-significant child conduct problem symptoms. These studies involved intervening with parents of children demonstrated to show a clinically- significant magnitude of conduct problem symptoms based on standardized measures (e. g. roughly above the 90th percentile on the Intensity scale of the Eyberg Child 81 Behavior Inventory and/or a minimum number of problems such as 10 reported on the Problems scale of this same instrument). Table 8: IYPTP Tier Three, Indicated Intervention Treating Clinically-Significant Child Conduct Problems (between-groups studies) I Hedges’s Stnd. Variance Lower I Upper I z-value I p-value g Error limit limit Primaq Studies: Webster-Stratton - 0.528 0.327 0.107 - 1.169 0.114 - 1.611 0.107 1982a Webster-Stratton 1984 - 0.871 0.396 0.157 - 1.647 - 0.094 - 2.198 0.028 Webster-Stratton et a1 - 1.009 0.193 0.037 - 1.388 - 0.631 - 5.227 0.000 2004 Webster-Stratton et al — 0.827 0.206 0.042 - 1.230 - 0.423 - 4.018 0.000 1988 Webster-Stratton et a1 - 1.189 0.230 0.053 - 1.640 - 0.738 - 5.168 0.000 1997 Primary, Random Model: -0. 93 7 0. 109 0. 012 -1. 149 -0. 724 -8. 634 0. 000 Replication Studies: Gardner, Burton, -0.652 0.251 0.063 - 1.143 - 0.161 - 2.602 0.009 Klimes 2006 Hutchings, Bywater et - 0.656 0.183 0.034 - 1.016 - 0.297 - 3.576 0.000 al 2007 Larsson, Fossum et al - 0.558 0.184 0.034 - 0.918 - 0.198 - 3.038 0.002 2009 Scott, Spender et al - 0.238 0.232 0.054 - 0.692 0.216 - 1.027 0.305 2010 Scott, Sylva, et al 2010 - 0.417 0.202 0.041 - 0.813 - 0.020 - 2.059 0.039 Taylor et a1 1998 - 0.775 0.342 0.1 17 - 1.446 - 0.104 — 2.263 0.024 Replications Random Model: -0. 533 0. 089 0. 008 -0. 707 -0.359 -6.005 0. 000 Primary& Replications Combined, Random -0. 695 0.069 0.005 -0.830 -0.560 40.115 0.000 Model: Tests of homogeneity were performed on the Tier Three studies primary and replication groups separately. The primary studies Q-value of 3.134 with 4 degrees of freedom yielded a non-significant p-value of 0.536. The replication studies Q-value of 3.149 with 5 degrees of freedom yielded a non-significant p-value of 0.677. These results suggest an acceptable level of homogeneity to report one grand mean effect size representing each group of studies. 82 However, visual inspection of the primary studies found a wide variation for obtained effect sizes spanning over one standard deviation (-l.189) to about one-half of a standard deviation (—0.537) thus making an interpretation of one overall effect size representative of this group quite difficult given this wide level of dispersion. Borenstein and colleagues (2009) note that a non-significant p-value may be indicative of low power (p. 113). Examination of statistics for this group indicates within-study variance was non-significant with a p-value of 0.71 1. The low number of studies combined with a low number of subjects in these studies seems to be the most likely explanation for the non- significant finding for heterogeneity. These observations warrant firrther examination of the primary Tier Three studies, since it is likely these do not comprise one group of homogeneous studies that would be adequately represented by one grand mean effect size. Close inspection of the primary Tier Three studies found dosage (number of sessions) of IYPTP varied considerably. Differences in the dosage of IYPTP treatment was not hypothesized as a tested variable for this meta-analysis a priori. However given the wide variation in dosages across primary studies, this was tested first prior to a statistical comparison between primary and replication Tier Three studies, and before any a priori hypotheses are tested. Variations in IYPTP dosage clearly need to be accounted for prior to considering if other hypothesized variables differentially impact on IYPTP study effect Sizes. T est for Dosage Effect for I YPT P Webster-Stratton (19823) studied an early form of IYPTP employing only 4 weekly sessions of two-hours, and this produced the lowest Hedges g effect size for 83 between-groups primary studies. Since this dosage level is less than half the established dose for IYPTP this study was removed from firrther analysis. It was firrther noted that two of the other between-groups primary studies (Webster-Stratton & Hammond, 1997; Webster-Stratton, et al., 2004) reported using 22-24 two-hour, weekly parent group sessions. These two studies were found to have the largest Hedges g effect sizes and their relative weights account for 60% of the mean effect size for the primary between- group studies after the 1982a study was removed. Within the “Handbook of Parent Training” (Schaefer & Briesmeister, 1998) Webster—Stratton and Hancock (1998b) reported IYPTP in its original, BASIC form consisted of 12 weeks using 10 videotapes with more than 250 vignettes. By the third edition of this handbook (Briesmeister & Schaefer, 2007) Webster-Stratton’s.chapter (2007) again reported the original BASIC program of 12 sessions was developed and found effective across several studies for young children diagnosed with Oppositional Defiant Disorder. However, at this point ten years after the first book chapter summarizing IYPTP, Dr. Webster-Stratton was recommending a combination of IYPTP BASIC and ADVANCED programs for this same population of children that takes between 20 to 24 weeks to complete. Neither the Webster-Stratton and Hammond (1997), or the Webster-Stratton, Reid and Hammond (2004) studies included in this meta-analysis described the addition of the ADVANCED content to comprise their 22-24 weeks of intervention. Data from the Webster-Stratton (1994) that tested the addition of the ADVANCED program was not included in this meta-analysis because it was regarded as an addition to the BASIC program. The BASIC program was employed within all studies contained in this meta- 84 analysis. Because the two studies using 22-24 sessions of IYPTP indicate usage of only the BASIC program and not the ADVANCED program, they were retained. To test whether dosage accounts for a statistically-significant greater benefit for IYPTP Tier Three studies, a comparison was made between the effects from the two studies using 22 - 24 two-hour sessions, with that of other studies (primary and replication) which all reported using 9 - 16 two-hour sessions. The range of sessions among studies within this meta-analysis suggests (with the exception of the two studies using larger dosages) studied group-administered IYPTP generally includes a dosage range of 12 sessions either minus 3 or plus 4. Table 9 shows study dosage comparison. 85 Table 9: Contrast IYPTP Tier Three Dosages (22-24 versus 9-16 two-hour sessions) I Hedges’s g I Stnd. I Variance I Lower Upper I z-value p-value Error limit limit Studies using 22-24 Two-Hour Sessions: W-S et a1 - 1.009 0.193 0.037 - 1.388 - 0.631 - 5.227 0.000 2004 W-S et al - 1.189 0.230 0.053 - 1.640 - 0.738 - 5.168 0.000 1997 22-24Session Random -1. 084 0. I48 0. 022 -.I.3 74 -0. 794 - 7. 326 0. 000 Model: Studies using 9-16 Two-Hour Sessions: Gardner, -0.652 0.251 0.063 - 1.143 - 0.161 - 2.602 0.009 Burton, Klimes 2006 Hutchings, - 0.656 0.183 0.034 - 1.016 - 0.297 - 3.576 0.000 Bywater et a1 2007 Larsson, - 0.558 0.184 0.034 - 0.918 - 0.198 - 3.038 0.002 Fossum et a1 2009 Scott, - 0.238 0.232 0.054 - 0.692 0.216 - 1.027 0.305 Spender et al 2010 Scott, Sylva, - 0.417 0.202 0.041 - 0.813 - 0.020 - 2.059 0.039 eta12010 Taylor et a1 - 0.775 0.342 0.1 17 - 1.446 - 0.104 - 2.263 0.024 1998 Webster- - 0.871 0.396 0.157 - 1.647 - 0.094 - 2.198 0.028 Stratton 1984 Webster- - 0.827 0.206 0.042 - 1.230 - 0.423 - 4.018 0.000 Stratton et a1 1988 9—16 Session Random -0. 5 91 0. 080 0. 006 -0. 74 7 -0. 435 - 7.403 0. 000 Model: For the 22-24 sessions group a test of homogeneity yielded a Q-Statistic of 0.3 59 with 1 degree of freedom and a non—significant o-value of 0.549. For the 9-16 sessions group the test of homogeneity yielded a Q-statistic of 5.384 with 7 degrees of freedom and a non-significant p-value of 0.613. The relative homogeneity of each group warranted a comparison of the grand mean effect size representing each group of studies. Mixed-effects analysis yielded a Q-Statistic of 8.593 with 1 degree of freedom and a Significant p-value of 0.003. This finding indicates a statistically-significant effect for a 86 higher dosage of 22-24 two-hour sessions of IYPTP over that of more commonly encountered dosages of 9-16 two-hour sessions within a Tier Three usage. The dosage of 22-24 sessions of IYPTP Tier Three yielded a Hedges’s g effect size of -1.084 (-1 .374 to -0.794 with 95% confidence). This result suggests about a one- standard deviation reduction in child conduct problems. The average child benefiting from 22-24 sessions of IYPTP as a Tier Three intervention improved their behavior at a level superior to roughly 84 % of those in control groups who did not receive this intervention. The dosage of 9-16 sessions of IYPTP Tier Three yielded a Hedges’s g effect size of -0.591 (-0.747 to -0.435 with 95% confidence). This result suggests about two-thirds a standard deviation of benefit. The average child benefitting from 9-16 sessions of IYPTP as Tier Three intervention improved their behavior at a level superior to roughly 73% of those in the control groups who did not receive this intervention. This latter finding is a result of combining both primary and replication studies that all used dosages between 9- 16 sessions of IYPTP as a Tier Three intervention. In order to equitably compare primary to independent replications of the Tier Three use of IYPTP the two primary studies that used a significantly greater dosage (22-24 sessions) were removed from further comparison yielding Table 10. 87 Table 10: IYPTP between-groups studies using 9-16 two-hour sessions as a Tier Three Intervention to Treat Clinically-Significant Child Conduct Problems Hedges’s g I Stnd. Error I Variance I Lower limit Upper limit z-value p-value Primary Studies: Webster- Stratton 1984 - 0.871 0.396 0.157 - 1.647 — 0.094 -2.198 0.028 Webster- Stratton et a1 1988 - 0.827 0.206 0.042 - 1.230 - 0.423 -4.018 0.000 Primary, Random Model: -0. 83 6 0.183 0. 033 -I.I94 -0.4 78 -4. 5 79 0. 000 Replication Studies: Gardner, Burton, Klimes 2006 -0.652 0.251 0.063 -l.l43 -0.l6l - 2.602 0.009 Hutchings, Bywater et al 2007 - 0.656 0.183 0.034 - 1.016 - 0.297 - 3.576 0.000 Larsson, Fossum et al 2009 - 0.558 0.184 0.034 -0.918 -0.198 - 3.038 0.002 Scott, Spender et a1 2010 - 0.238 0.232 0.054 - 0.692 0.216 - 1.027 0.305 Scott, Sylva, et a1 2010 -0.417 0.202 0.041 -0.813 - 0.020 - 2.059 0.039 Taylor et al 1998 - 0.775 0.342 0.117 - 1.446 -0.104 - 2.263 0.024 Replications, Random Model: -0.533 0. 089 0. 008 -0. 707 -0.359 -6. 005 0. 000 Combined Primary & Replications Random -0. 591 0. 080 0. 006 -0. 74 7 -0.435 -7.403 0. 000 Table 10 illustrates the most equitable comparison of primary versus replication studies of Tier Three usage of IYPTP to treat significant child conduct symptoms, where dosages are roughly similar. The test of homogeneity for primary studies yielded a Q- Statistic of 0.010 with 1 degree of freedom and a non-significant p-value of 0.921. The test of homogeneity for replication studies yielded a Q-statistic of 3. 149 with 5 degrees of freedom and a non-significant p-value of 0.677. The within-group variance Q-Statistic of 3.159 with 6 degrees of freedom yielded a non-significant p-value of 0.789. These 88 findings suggest an absence of significant heterogeneity within each group making it appropriate to compare the grand mean effect sizes (Hedges’s g) representing each group. The Mixed-Effects Analysis yielded a Q-Statistic of 2.225 with 1 degree of freedom and a resulting non-significant p-value of 0.136. Hence there is no statistically significant difference between the benefits of primary and replication studies for the Tier Three usage of IYPTP to treat significant symptoms of child conduct problems using between 9 and 16 two-hour sessions. The Grand Mean Effect Size (Hedges’s g) for Tier Three usage of 9 to 16 two- hour sessions of IYPTP to treat significant child conduct problems across both primary and replication studies is -0.591 (-0.747 to -0.435 with 95% confidence). This suggests about two-thirds of a standard deviation benefit from Tier Three IYPTP (9-16 sessions). The average child benefiting from 9-16 sessions of IYPTP as a Tier Three intervention improved their behavior at a level superior to roughly 73 % of those in control groups who did not receive this intervention. Hypothesis 1b: It was hypothesized that the grand mean effect size for reductions in child conduct problems would be significantly greater for parent rating scale outcome measures than for direct observation of parent-child behavior, at a 0.05 level of statistical significance. It has been generally found that parent rating scale results showing parental perceptions of child behavior show more benefit from intervention than independent observations of child behaviors. This was most recently noted within the meta-analysis of Parent-Child Interaction Therapy and Triple-P Parent Training studies (Thomas & Zimmer-Genbeck, 2007). 89 Results for Hypothesis Test lb: Table 11 provides a summary of study effect Sizes (Hedges’s g) according to only parent-rating scale results, contrasted for primary and replication studies. The test of homogeneity for primary studies yielded a Q-Statistic of 0.567 with 1 degree of freedom and a non-significant p-value of 0.451. The test of homogeneity for replication studies yielded a Q-statistic of 2.791 with 4 degrees of freedom and a non-significant p-value of 0.593. The comparison of these two adequately homogeneous groups using the Mixed Effects Model yielded a Q-Statistic of 3.570 with 1 degree of freedom and a non-significant p-value of 0.059. This non- significant p-value and the considerable overlap between the confidence intervals for primary and replication grand mean effect sizes for each group suggests adequate homogeneity to interpret one overall, grand mean effect size as representative of this entire group of studies. The Hedges’s g grand mean effect size of -0.726 (-0.894 to - 0.558 with 95% confidence) is an appropriate measure to compare with the results listed in Table 11 showing effect size outcomes for independent observations of child conduct problems. 90 Table 11: Parent rating scale results (no independent observations) for IYPTP between-groups studies usi "3 9-16 sessions (contrastin g primary vs. replications) Hedges’s Stnd. I Variance Lower Upper z-value p-value g Error limit limit Primary Studies: Webster-Stratton - 1.316 0.413 0.171 -2.126 -0.507 -3.l86 0.001 1984 Webster-Stratton - 0.968 0.209 0.044 -l.377 -0.559 -4.653 0.000 etall988 Primary Random Effects -1. 039 0. 186 0. 035 -1. 404 -0. 6 73 -5.5 74 0. 000 Model: Replication Studies: Gardner, Burton, -0.721 0.253 0.064 -1.217 -0.225 -2.846 0.004 Klimes 2006 Hutchings, -0.835 0.185 0.034 -l.197 -0.473 -4.521 0.000 Bywater et a1 2007 Larsson, Fossum -0.558 0.184 0.034 -0.918 -0.198 -3.038 0.002 et312009 Scott, Sylva, et a1 -0.417 0.202 0.041 -0.813 -0.20 -2.059 0.039 2010 Taylor et al 1998 -0.775 0.342 0.1 17 -1.446 -0.104 -2.263 0.024 Replications Random Effects -0. 642 0.096 0. 009 -0. 831 -0. 453 -6. 659 0. 000 Model: Total Primary & Replications -0. 726 0. 086 0. 00 7 -0. 894 -0. 558 -8. 4 76 0. 000 Random Effects: Table 12 displays effect sizes (Hedges’s g) and associated statistics for only those studies reporting independent observations of child conduct behavior in relation to parents. The test of homogeneity for primary studies yielded a Q-Statistic of 0.067 with 1 degree of freedom and a non-significant p-value of 0.796. The test of homogeneity for replication studies yielded a Q-statistic of 0.501 with 1 degree of freedom and a non- significant p-value of 0.479. The comparison of these two adequately homogeneous groups using the Mixed Effects Model yielded a Q-Statistic of 0.358 with 1 degree of freedom and a non-significant p-value of 0.549. The latter result supports using the grand mean effect size for this entire group of studies, to compare with the grand mean effect size for the studies reporting only parent rating scale results listed in Table 11. 91 Table 12: Independent observations results (no Parent rating scale) for IYPTP between-groups studies using 9-16 sessions (contrastinglprimary vs. replications) I Hedges’s I Stnd. Variance Lower Upper I z-value I p-value g Error limit limit Primary Studies: Webster-Stratton -0.425 0.378 0.143 -1. 166 0.317 - l . 123 0.261 1984 Webster-Stratton -0.535 0.199 0.040 -0.926 -0. 145 -2.687 0.007 etall988 Primary -0. 511 0. I 76 0.031 -0. 85 7 -0. 166 -2. 901 0. 004 Random E fleets Model: Replication Studies: Gardner, Burton, -0.514 0.245 0.060 -0.994 -0.034 -2.099 0.036 Klimes 2006 Hutchings, -0.298 0.181 0.033 -0.653 0.056 -1.649 0.099 Bywater et a1 2007 Replications Random Effects -0.3 74 0. 146 0. 021 -0. 660 -0. 089 -2. 5 73 0. 010 Model: Total Primary & Replications -0.430 0.112 0.013 -0. 650 -0.210 -3.831 0. 000 Random Effects: The grand mean effect size of -0.430 (-0.650 to -0.210 with 95% confidence) for independent observations outcomes is not significantly different at the pre-set level of 0.05 than the grand mean effect size of -0.726 (-0.894 to -0.558 with 95% confidence) representing parent rating scale outcomes. These confidence intervals overlap slightly though this non-significant outcome may also be attributable to the small sample of studies representing these outcomes. Hypothesis 1c: It was hypothesized that for studies measuring effects from IYPTP that the grand mean effect size representative of a group of primary studies conducted by inventor Dr. Carolyn Webster-Stratton and her colleagues at the University of Washington would show significantly greater reductions in negative parenting than the grand mean effect size representative of replication studies, at a 0.05 level of statistical significance. 92 Results for Hypothesis Test 1c: Table 13 illustrates measured changes in negative parenting across primary and replication studies. The primary studies’ Q-Statistic of 0.155 with 1 degree of freedom yielded a non-significant p-value of 0.694. Similarly the replication studies Q-Statistic of 4.453 with 2 degrees of freedom yielded a non- significant p-value of0. 108. These results suggest acceptable levels of homogeneity within each group, where the Q-statistic of 4.607 with 3 degrees of freedom yielded a non-significant p-value of 0.203. A Mixed Effects Analysis yielded a Q-Statistic of 1.516 with 1 degree of freedom and a non-significant p-value of 0.2 1 8. The latter finding supports interpretation of a grand mean effect Size for both primary and replication studies of -0.491 (-0.714 to —0.269 with 95 % confidence) for reductions in negative parenting. This represents about one-half a standard deviation reduction in negative parenting. The average parent benefiting from IYPTP as a Tier Three intervention reduced their negative parenting behaviors at a level superior to roughly 70 % of those in control groups who did not receive this intervention. 93 Table 13: Reductions in Negative Parenting for IYPTP between-groups studies I Hedges’s g I Stnd. I Variance Lower Upper I z-value I p-value Error limit limit Primary Studies: W-S et al., 0660 0.202 0.041 -l.066 -0.265 -3.270 0.001 1988 W-S et al., -0.545 0.214 0.046 -0.964 -0.126 -2.549 0.01 l 1997 Primary RandomEffect -0. 606 0.14 7 0.022 -0.893 -0.318 -4. I28 0. 000 Model: Replication Studies: Gardner et al., 0692 0.247 0.061 -1.177 —0.207 -2.797 0.005 2006 Hutchings et -0.060 0.171 0.029 -0.395 0.276 -0.349 0.727 al., 2007 Scott et al., 0314 0.210 0.044 -0.726 0.098 -1.492 0.136 2010 Replications RandomEffect -0. 321 0. I 79 0.032 -0. 6 71 0.030 -1. 794 0.073 Model: Combined Primary & -0.491 0.113 0.013 -0. 714 -0.269 -4.329 0.000 Replications RandomE/fect: Hypothesis ld: It was hypothesized that for studies measuring effects from IYPTP that the grand mean effect size representative of a group of primary studies conducted by inventor Dr. Carolyn Webster-Stratton and her colleagues at the University of Washington would Show significantly greater improvements in positive parenting than the grand mean effect size representative of replication studies, at a 0.05 level of statistical significance. Results for Hypothesis Test 1d: Table 14 summarizes the Tier Three primary and replication studies reporting data amenable to this meta-analysis, about changes in positive parenting via self-report of parents and/or independent observations. It was noticed that the reporting of data on changes in parenting behaviors appeared less consistent even for the same researcher (e. g. Webster-Stratton) where variables were reported separately in one study and aggregated 94 into larger categories in others as well as being reported using different types of scores. This likely makes meta-analysis of these data a bit more challenging to ensure equitable comparisons across studies. Table 14: Increases in Positive Parenting for IYPTP between-groups studies Hedges’s g I Stnd. I Variance Lower Upper z-value p-value Error limit limit Primary Studies: W-S et al., 2.375 0.496 0.246 1.403 3.347 4.790 0.000 1984 W-S et al., 1.194 0.213 0.046 0.776 1.613 5.594 0.000 1988 W-S et al., 0.874 0.220 0.048 0.443 1.306 3.971 0.000 1997 Primary 1.331 0.318 0.101 0. 707 1.955 4.182 0.000 RandomEffect Model: Replication Studies: Gardner et al., 0.464 0.242 0.059 -0.01 1 0.940 1.916 0.055 2006 Hutchings et 0.448 0.171 0.029 0.1 12 0.783 2.615 0.009 al., 2007 Scott et al., 0.285 0.21 1 0.045 -0. 129 0.699 1.349 0.177 2010 Replications RandomEffect 0.402 0.11 7 0. 014 0. 1 74 0. 631 3.448 0. 001 Model: Combined Primary & 0.512 0.110 0. 012 0.297 0. 72 7 4.6 76 0.000 Replications RandomEffect: The replication studies reporting changes in positive parenting yielded a Q- statistic of 0.444 with 2 degrees of freedom and a non-significant p-value of 0.801. This suggests a fairly homogeneous set of data for this group. The primary studies reporting changes in positive parenting yielded a Q-statistic of 7.717 with 2 degrees of freedom and a significant p-value of 0.021, suggesting an unusual amount of heterogeneity. Visual inspection of the data found one study (Webster-Stratton, 1984) produced an overall effect size (Hedges’s g) of 2.375 which is over double that of the other two primary studies and 4 to 5 times greater than the replication study effect sizes. Table 15 shows 95 changes in positive parenting effects minus the outlier study. With this study removed the Q-statistic of 1.089 with 1 degree of freedom yields a non-significant p-value of 0.297. With both primary and replication groups showing adequate homogeneity, a Mixed Effects Analysis was performed comparing the mean effect sizes from each group. This yielded a Q-statistic of 10.343 with 1 degree of freedom and a significant p-value of 0.001. The obtained primary studies Hedges’s g effect size of 1.039 (0.725 to 1.352 with 95% confidence) is significantly greater than replication studies the Hedges’s g effect size of 0.402 (0.174 to 0.631). This greater degree of improvement in positive parenting for both primary and replication studies was primarily based on independent observations of parenting behaviors. Table 15: Increases in Positive Parenting for IYPTP between-groups studies (Outlier removed) Hedges’s g I Stnd. I Variance Lower I Upper I z-value p-value Error limit limit Primary Studies: W-S et al., 1.194 0.213 0.046 0.776 1.613 5.594 0.000 1988 W-S et al., 0.874 0.220 0.048 0.443 1.306 3.971 0.000 1997 Primary 1. 03 9 0. I 60 0. 026 0. 725 1.352 6.494 0. 000 RandomEjfect Model: Replication Studies: Gardner et al., 0.464 0.242 0.059 -0.01 1 0.940 1.916 0.055 2006 Hutchings et 0.448 0.171 0.029 0.1 12 0.783 2.615 0.009 al., 2007 Scott et al., 0.285 0.21 1 0.045 -0.129 0.699 1.349 0.177 2010 Replications RandomEffect 0. 402 0. 11 7 0.014 0.1 74 0. 631 3.448 0. 001 Model: Combined Primary & 0.623 0.094 0. 009 0.438 0.808 6.612 0. 000 Replications RandomEffect: 96 Secondary Hypotheses Testing Findings The second stage of meta-analysis was intended to further examine what factors moderate an anticipated greater benefit from primary studies over that of independent replication studies for IYPTP. However, this principal hypothesis was not supported. While there were no significant differences found for child conduct problem outcomes between primary and replication studies, several planned hypotheses tests offer additional information regarding moderators of IYPTP benefits across both study types, and are discussed below. Moderators are variables that are present at baseline and differentiate under what conditions and for whom an intervention is effective. (Beauchaine et al., 2005; Shadish & Sweeney, 1991). Research Question 2: What variables moderate greater benefit from IYPTP for primary and replication research as separate groups? Are they similar or different? Moderators are variables that are present at baseline and differentiate under what conditions and for whom an intervention is effective. (Beauchaine et al., 2005; Shadish & Sweeney, 1991). Hypothesis 2a: The grand mean effect size for a group of studies treating the most severe child conduct problems using IYPTP will be significantly different from the grand mean effect size for a group of studies treating the least severe child conduct problems using IYPTP, at a 0.05 level of statistical significance. Numerous studies have found that the greater the severity of a problem at the onset of treatment, the greater benefit (Beauchaine, et al., 2005; deGraff et al., 2008; Spirito et al., 2009), although there have been exceptions to this trend (N owak & Heinrichs, 2008; Weisz et al., 2006). Results for Hypothesis Test 2a: Fortunately the vast majority of the Tier Three intervention studies used the Eyberg Child Behavior Inventory (ECBI) as a measure of 97 child conduct problems, allowing for comparisons of child conduct severity between studies. The ECBI Intensity Scale measures the frequency of child disruptive behavior as rated on a scale from 1 (never) to 7 (always). The ECBI Problem Scale measures how problematic a child’s behavior is perceived based on yes/no answers to statements about whether a behavior is present (Butler, Brestan & Eyberg, 2008). The norms for the ECBI indicate Intensity scores at or above 127 have been regarded as clinically-significant while Problem scores at or above 11 have been regarded as clinically-significant (Eyberg & Ross, 1978). These cut—off scores have been regarded as being at the 93rd percentile within the norm sample. However a more recent re-standardization of ECBI norms has suggested these cut-off scores should now be 132 for the Intensity Scale and 15 for the Problems Scale (Colvin, Eyberg & Adams, 1999). These have not been published, however, in a peer-reviewed journal or validated independently as was the earlier norms. Five out of the six replication Tier Three studies reported ECBI scores. All five cited ECBI Intensity scores, and 4 out of 5 reported ECBI Problem scores. Of the Tier Three primary studies three out of four studies also reported ECBI scores. All three cited ECBI Intensity scores and two out of three reported ECBI Problem scores. These rates of score reports suggest reasonably good representative data across the primary and replication study samples for comparisons to be made. Mean scores were reported separately for experimental and control samples as well as frequently reported separately for mothers and fathers. The mean ECBI Intensity, Problem Scale scores and their standard deviations for each study, and then for groups of studies were calculated by hand. Each reported mean and standard deviation within a study was multiplied by the size of the sample on which this was based upon 98 (experimental or control). These products were then added together for all scores reported for a scale (Intensity or Problem) in each study. The total of all these products were then divided by the total number of respondents who provided these scores (i.e. all sample group members). This method accounted for different sample sizes from which means and standard deviations were reported. In this way one mean score from a smaller sample would not have equal influence as a larger sample on the overall mean for a scale. Results from these calculations are listed in Table 16. For the Tier Three independent replication studies the ECBI Intensity mean score was 141.4 and the ECBI Problem mean score was 18.5. The one study (Scott et al., 2010) not reporting a Problem score was also noted to report a Intensity mean score of 117.6 that was significantly below the mean for the other three studies (149), hence the latter likely best represents the mean for Tier Three studies. For Tier Three primary studies the ECBI mean Intensity score was 153.85 and the Problem mean score was 19.94. Combining these results yielded a mean ECBI Intensity Scale score of 151.4 and an ECBI Problem mean score of 19.22 for primary and replication Tier Three studies. For the Tier Two independent replication studies (there were no primary studies at this tier) the mean ECBI Intensity Scale score was 119.97 and the ECBI Problem mean score was 12.39 based on three studies. For the Tier One primary studies (no ECBI scores listed for replications) the mean ECBI Intensity Scale score was 90.83 and the ECBI Problem Scale score was 9.9. 99 Table 16: Severity of Initial Conduct Problems b1 r Intervention Tier ECBI Intensity Score ECBI Problem Score TIER 111 Primary Studies = 159.15 Primary Studies = not reported (22-24 sess) ESg= -l.084 (n of individuals = 305) (n of individuals = 305) Replication Studies = None Replication Studies = None TIER 111 Primary Studies = 151.2 (26.4) Primary Studies = 19.4 (6.7) (9-16 sess) ESg = -0.591 (n of individuals = 202) (n of individuals = 202) Total n of Replication Studies = 149 (29.06) Replication Studies = 18.5 (7.03) individuals = 587 (n of individuals = 385) (n of individuals = 385) TIER 11 Primary Studies = None Primary Studies = None ESg = -0.318 . Replication Studies= 119.7 (31.79) Replication Studies = 12.39 (8.25) Total n of (n of individuals = 197) (n of individuals = 197) individuals = 197 TIER I Primary Studies = 90.83 (26.35) Primary Studies = 9.9 (8.0) ESg=-0.122 (n of individuals = 435) (n of individuals = 435) Total n of Replication Studies=none reported Replication Studies=none reported individuals = 435 Note: Mean Scores reported (standard deviations in parentheses) Several findings are notable relative to the data presented within Table 15. First, for ECBI Intensity and Problem scores are each significantly different in magnitude between the three tiered levels according to One-Way ANOVA (Fisher F statistic of 564.544 with 2 degrees of freedom and 191.614 with 2 degrees of freedom respectively) that yielded statistically significance (p = 0.000). This data supports the initial hypothesis that more severe conduct problems initially, would be associated with larger effect sizes 100 (listed in first column to the left in table 16). However, the severity of Intensity and Problem scores are essentially the same for Tier Three studies using either 9-16 sessions or 22-24 sessions, where the latter is shown to have a much larger effect size. This suggests that beyond initial severity of conduct problems other variables also have influence upon overall effects from the intervention. It is likely that the increased dosage has an additional, favorable impact on the outcome’s effect size, although there may be additional variables not yet identified that affect this larger outcome. It is important to note that the separation of the IYPTP studies according to usage (three tiers) is validated by the data contained within Table 16. Finally, the data in Table 16 lends further support to the more recent re-standardized norms for the ECBI that suggest using a cut-off for the Problems score of 15 (rather than the former 11) to distinguish clinical significance (Colvin, Eyberg & Adams, 1999). One type of study not included within Table 16 is the primary studies using the self-administered form of IYPTP. All three of these studies reported ECBI scores for the Intensity scale and two of the three reported scores on the Problems Scale. The overall mean ECBI Intensity score was 152.74 which is essentially the same as Tier Three studies. Similarly, the mean ECBI Problems score was 19.70 which are also comparable to the Tier Three studies. When comparing the obtained effect size (Hedges’s g) of -0.531 (-0.737 to -0.324 with 95% confidence) for the Self-Administered IYPTP with that of the Tier Three (9-16 sessions) obtained effect size (Hedges’s g) of -0.591 (-0.747 to -0.435 with 95% confidence) there is no statistical evidence of an improved effect from the more labor intensive and time-consuming group-administered form of IYPTP over that of the self-administered form. This lack of difference is not due to a differing 101 level of severity treated by each type of IYPTP. Additionally, comparison of the group- administered IYPTP at an increased dosage of 22-24 sessions with a Hedges’s g effect size of -1.084 (-1.374 to -0.794 with 95% confidence) with that of the Self-Administered outcome (-0.531; -0.737 to -0.324 with 95% confidence) results in an unlikely, statistically-significant difference from the 22-24 session group-administered form over that of the self-administered form. The lack of overlap between these two different sets of confidence intervals suggests there is a significantly greater benefit from intervening with the most severe forms of child conduct problems when the group-administered form is at twice the typical dosage for the BASC IYPTP (22 to 24 sessions). Certainly the low number of primary studies and no replication studies for the 22-24 sessions suggests these findings are preliminary. Hypothesis 2b: The grand mean effect size for child conduct problems will be significantly better for each group of studies, according to the amount of training therapists delivering IYPTP received, at a statistically significant level of 0.05. Lochman and colleagues (2009) looked at training in a unique and highly understudied manner by considering training on a relative scale of 3 conditions involved in transporting an EBI into a school setting. Benefits were significantly affected by whether therapists had a level of no training, basic training or training plus ongoing feedback. More benefits in terms of reducing behavioral problems in youth were also recently associated with adherence to treatment protocol as facilitated by ongoing supervision (Schoenwald, Sheidow & Chapman, 2009). For this meta-analysis it is anticipated studies would be grouped according to Lochman’s distinctions of no training, basic training and training plus ongoing supervision. 102 Results for Hypothesis Test 2b: (see “Results for Hypothesis Test 2c”) Hypothesis 2c: The grand mean effect size for a group of studies reporting use of methods to ensure treatment fidelity will be significantly greater at the 0.05 level of significance, than the grand mean effect size for a group of studies not reporting use of methods to insure treatment fidelity. Results for Hypothesis Test 2c: It was hypothesized that studies using some measure of treatment fidelity and those studies using more highly trained therapists (group leaders) would have greater benefits. While collecting data from the articles included within this meta-analysis it became evident that methods for insuring treatment integrity and the level of therapist training were inextricably tied together. The Tier Three primary studies (with one exception) listed several methods for insuring treatment integrity. These are listed below: 1.) Therapists co-led their first group with a supervisor. 2.) Sessions followed content directed from a treatment manual. 3.) Therapists kept detailed notes for each treatment session, including weekly protocol checklists for standards to be covered within each session (e.g. agenda, number of vignettes, role-plays to be completed and themes to be discussed). These were monitored weekly to ensure delivery of all components. 4.) Sessions were videotaped for review and feedback during weekly supervision meetings; and these were randomly selected for integrity checks which were always reported to be very high. Examination of the Tier Three independent replications studies found that all of these contained Similar methods of insuring treatment fidelity but at a lesser degree of 103 comprehensiveness. All of the replication studies made mention of ensuring that group leaders had multiple prior experiences with delivering the program. There were also some studies mentioning the use of the IYPTP manual where it is probably reasonable to assume all did this at some level. Most of the replication studies did not make mention of reviewing videotaped sessions, using checklists, or independent, random checks of integrity. There was consistent mention of weekly supervision meetings to ensure treatment integrity and ongoing training of therapists. Both primary and independent replication studies using IYPTP as a Tier Three intervention reported high levels of therapist (group leader) training. All studies reported therapists had received specialized training in IYPTP. Four out of the six independent replications reported therapists had received 3 — 4 day trainings from either Dr. Webster- Stratton or another certified trainer endorsed by Dr. Webster-Stratton. Two of the six replication studies also indicated therapists were certified by Dr. Webster-Stratton. 104 CHAPTER 5 DISCUSSION Common criticisms of meta-analysis include the assertion that combining different studies is like comparing apples to oranges (Borenstein et al., 2009) and that meta-analytic aggregation of studies yields claims about a swath of interventions too broad to provide meaningfirl conclusions (Beutler, 2009). This meta-analysis avoided these potential criticisms by including several features. First, this study focused on studies for one particular intervention. Second, the principal hypothesis for this study was evaluated using one type of outcome (child conduct problems) measured with only two methods (rating scales and independent observations) where many studies used the same or similar standardized measures. Third, this study combined IYPTP studies separately according to type of research study (between-groups versus within-groups) since effect sizes were calculated using different forms of data (pre-post changes for groups in isolation and compared with a control group). Fourth, studies were grouped by the particular use of IYPTP described (i.e. prevention, selective and indicated). These separations were further validated by divergent levels of conduct problem severity within each tier. Another potential criticism of meta-analysis is that it ignores important data. This study used methods to acknowledge and avoid this potential problem. There was a careful, thorough and systematic search for data pertaining to the focus of this study. The nature of studies not included in this meta-analysis was tabulated to inform readers about areas of potential inadequate representation. Examination of the reasons studies were excluded could inform future research. Methods to evaluate for potential publication bias 105 and missing studies deployed in this meta—analysis suggested insufficient reason to suspect these were a significant problem for this meta-analysis. Additionally, the literature search found and included two pieces of unpublished research. While this could potentially cause a problem in terms of the lack of peer-review and quality assurance for these studies, these did not present a unique influence on the findings for this study. It should also be noted that relying on only narrative reviews of the professional literature suffers from the same potential publication bias as meta-analysis, but it is easier to ignore this with the former, and can at least be considered and evaluated within the latter (Borenstein et al., 2009). Narrative reviews of studies are susceptible to weighting evidence non- systematically, thereby potentially ignoring valid data and/or validating problematic data. Interpretations of results for multiple variables, reported by a mixture of statistics (e.g., effect sizes, p-values, F -statistics), within and across studies, makes it difficult to form any consolidated conclusions for a set of studies (Olejnik & Algina, 2000). This is especially true when studies produce a mixture of results within and across similar research domains. Interestingly, the most recent primary study included in this meta- analysis (Webster-Stratton et al., 2004) did not cite each of the large number of outcome variables used separately, but did report composite scores comprised of the same two types of measures reported in this meta-analysis (e. g. ratings and observations) for the same three dependent variables of interest in this meta-analysis (i.e. child conduct problems, negative and positive parenting). This meta-analysis weighted studies to account for, and minimize variances within and between studies so that conclusions could be based on homogeneity of data produced by systematic and replicable procedures. This 106 study avoided limitations of earlier meta-analyses of parent training that did not account for sample size and did not analyze significant sources of heterogeneity (Cedar & Levant, 1990; Lundahl et al., 2006; Serketich & Dumas, 1996). This study also adds to earlier meta-analytic findings touting the benefits of behavioral parent training as a general category of intervention (Maughan et al., 2005), by more focused evaluation of a specific program that could be transported into practice. This study also provides a systematic means of comparing meta-analytic results for IYPTP with the meta-analytic results from two recent studies examining other parent training programs intended to address child conduct problems (deGraaf et al., 2008; Thomas & Zimmer-Genbeck, 2007). The principal hypothesis of this meta-analysis study was that primary research effect sizes (Hedges’s g) would be greater than those for independent replications of IYPTP at a pre-set 0.05 level of statistical significance. Preliminary, significantly greater reductions of child conduct problems found by primary inventor-based research studies over that of independent replication studies (Sougstad et al., 2008), were not found within this more comprehensive meta-analysis. For the with-in groups studies of IYPTP there was no statistically-significant difference found between primary and replication study reductions in child conduct problems. For these within-groups studies the grand mean effect size (Hedges’s g) of -0.634 (-0.791 to 0478) represents a benefit of about two- thirds of a standard deviation. Examination of the within-groups studies suggested diverse uses of IYPTP (prevention, selected and indicated uses), but not enough studies for any particular use to support further grouping and statistical analysis. This makes final interpretations of findings for the within-groups studies difficult. Because there was a larger sampling of between-groups studies (includes control group) and this type of 107 study is regarded as the gold standard for demonstrating effects from an intervention, the remainder of this investigation concentrated on the between-groups data. Figure 5 provides a summary for results from between—groups primary and replication studies of IYPTP across all uses and types examined within this meta- analysis. For between-groups Tier One Universal Prevention studies of IYPTP there were no statistically-significant differences between primary and independent replication effect sizes yielding a grand mean effect size (Hedges’s g) of -0.122 (-0.250 to 0.006 with 95% confidence). This is a quite small effect, but not surprising considering subjects may not have shown any conduct problems to begin with. For between-groups Tier Two selective intervention studies there were no primary studies that met inclusion criteria for this meta-analysis. The between-groups replication studies using IYPTP as a Tier Two selective intervention yielded a grand mean effect size (Hedges’s g) of -0.318 (-0.573 to -0.063 with 95% confidence). This represents about one-third a standard deviation of benefit. Tier Two studies focused on groups specific to a particular risk factor(s) (e.g. parenting known to be neglectful, youth placed in foster care, conduct problems including those below a clinically-significant level, etc,. . .). 108 Figure 5: Summary Table for IYPTP Between-Groups Studies Effects (Hedges’s g) Tier 111: Primary 22-24 two hr. grps. -1.084 (-1.374 to -0.794) Tier 111 Primary + Replication 9-16 two hr.grps. -0.591(-0.747 to -0.435) I Primary: Self-Administered -0.531 (-0.737 to -0.324) I Tier 11: Primary + Replication -0.318 (-0.573 to -0.063) Tier 1: Primary + Replication -0.122 (-0.250 to 0.006) -1.00 -0.50 0.00 Use of IYPTP as a Tier Three indicated intervention to treat the highest levels of clinically-significant child conduct problems yielded similar benefits for primary and replication studies when 9-16 two-hour sessions were used. These studies provided a grand mean effect size (Hedges’s g) of -0.591 (-0.747 to -0.435 with 95% confidence) representing about two-thirds of a standard deviation reduction in child conduct problems. The primary studies using 22-24 two-hour sessions of IYPTP as a Tier Three indicated intervention yielded a statistically—significant higher Hedges’s g mean effect size of -1.084 (-1.374 to -0.794 with 95% confidence) representing about 1 standard deviation of reduction in child conduct problems. No replication studies of the self-administered form of IYPTP met inclusion criteria for this study. The primary studies of self-administered IYPTP yielded a grand 109 mean effect size (Hedges’s g) of -0.531 (-0.737 to -0.324 with 95% confidence) which represents about one-half of a standard deviation of benefit. The overall effect size for IYPTP Tier Three intervention is generally comparable to the measured benefit found from meta-analysis of the Triple-P Parent Training Program Level 4 that is used to address the highest levels of conduct problems (de Graff, et al., 2008), though the increased dosage studies of IYPTP (22-24 two-hour sessions) grand mean effect size (Hedges’s g) appears to be significantly greater than the benefits reported for Triple-P Level 4. The absence of significantly different benefits from the self-administered form of IYPTP with that of the BASIC group-administered form of IYPTP at the 9-16 two-hour dosage level is not consistent with a similar comparison made by deGraff and colleagues (2008) for Triple-P. They found significantly greater benefits for a self-directed form of Triple-P over that of group administered. However Nowak and Heinrichs (2008) for Triple-P study did not find significant differences between the self-directed and group-administered Triple-P interventions. While the number of studies is small, those IYPTP studies that used double the BASIC IYPTP program dosage (22-24 two-hour sessions) yielded twice the measured benefit from this more intensive dosage. Webster-Stratton and Herman (2010) recently mentioned a similar finding embedded within their recent article. They recently found significantly greater benefit from 20 weeks of two-hour IYPTP over that of 10 weeks IYPTP for families of children dually diagnosed with Oppositional Defiant Disorder and Attention Deficit Hyperactivity Disorder. They also cited a Similar finding by Henggeler, Schoenwald, Liao, Letoumea and Edwards (2002), that 20 sessions or more was 110 associated with significantly higher benefits treating a similar, older population of youth with conduct problems receiving Multi-Systemic Therapy. Summary Overall, there were no statistically-significant differences in reductions for child conduct problems between primary and replication research studies found in this meta- analysis when comparisons were made fairly, based on the usage and dosage of IYPTP across a three-tiered public health model of prevention and intervention. An initially encountered difference between primary and replication studies in the preliminary meta- analysis study (Sougstad et al., 2008) and also encountered in this current study for Tier Three IYPTP were spurious, due to the effect of an intervening higher dosage used in two primary studies. Once these two studies were removed there was no significant difference between effect sizes for primary and replication studies of IYPTP using similar dosages, for the same levels of child conduct problems. The severity of conduct problems was found to be higher for each tier of intervention at a statistically significant level. Greater intensity and frequency of conduct problems coincided with a higher tier use of IYPTP, and a greater overall mean effect size from IYPTP intervention. This finding coincides with the findings of several other studies (Beauchaine, et al., 2005; deGraff et al., 2008; Spirito et al., 2009) and contradicts others reports on this issue (Nowak & Heinrichs, 2008; Thomas and Zimmer-Gembeck, 2007; Weisz et al., 2006). Additional review of primary and replication Tier Three studies found these were more similar rather than different in terms of the level of training held by therapists 111 (group leaders) delivering IYPTP. The studies were also quite similar in terms of a pervasive use of the IYPTP Manual, prescribed videotapes, role-plays and coverage of content for each session, as well as use of weekly supervision meetings sometimes involving review of videotaped sessions. The latter form of supervision was most prevalent among primary studies as was the use of random checks of therapist completed checklists and submitted videos of sessions to verify treatment fidelity. The incidence and discrimination of differences in these forms of training and fidelity for each study were so Small and difficult to detect differentially across this small sample of studies that statistical analysis was not appropriate. Finding these elements generally present across both primary and replication research studies seems to further help explain no significant differences between these two types of studies. The presence of these features for the primary and replication studies further reinforces the initial premise for this study, that IYPTP contains many features that make it a more likely candidate for effective replication and transportation beyond primary researchers. The finding of significant benefits for studies containing treatment agents highly trained and regularly supervised for the integrity of deployment is consistent with the recent findings of Lochman and colleagues (2009) as well as Schoenwald and colleagues (2009). Dr. Webster-Stratton, her colleagues, and other researchers have invested a great deal of thought and resources to ensure successful transportability of IYPTP with good fidelity and integrity beyond the primary research team. Perhaps the distinction made for this study, between primary and independent replication of research studies according to authorship oversimplifies a complex set of issues involved with transporting, disseminating and evaluating an evidence-based intervention. Future examination of this 112 differentiation has important ramifications for current systems of evaluating the evidence- base for interventions. Currently, the highest levels of evidence are reserved for “independent” verification of benefits. Yet when primary researchers and colleagues are involved with training, ongoing supervision of EBI deployment and reviews of treatment integrity for certification in the IYPTP does this refute independence? Using Chorpita’s typology (Table 1) this meta-analysis found that beyond the primary inventor’s efficacy studies (Type I) there is a range of studies showing significant and robust benefits from IYPTP as a transported (Type II) and disseminated (Type III) EBI, while efforts to ensure the highest levels of training and fidelity of intervention deployment may “muddy the waters” for determining whether there is absolute independence. Data across three levels of severity for child conduct problems collected within this meta-analysis were supportive of more recent, proposed alteration of norms for determining clinical- significance on the Eyberg Child Behavior Inventory (Colvin, Eyberg & Adams, 1999). The absence of no independent replications of between-group studies of the self- administered form of IYPTP prevented any comparison with an effect size representing primary studies of self-administered IYPTP. The finding of no between-groups or within-group studies using the self-administered form of IYPTP is quite surprising given the significant benefits reported by primary research, and the more plausible likelihood that the self-administered form would be more easily replicable without needing to ensure therapist integrity for group intervention. For the most severe child conduct cases the group-administered form of IYPTP requiring a substantial investment of time and resources was only found to produce significantly higher benefits (about one standard deviation) over that of the self- 113 administered fornr of IYPTP (about half-standard deviation) when the dosage of two- hour group sessions was doubled to between 22 to 24. The latter finding has not been reproduced by independent replication studies but was recently reported by Webster- Stratton and Herman (2010) for youth diagnosed with Oppositional Defiant Disorder and Attention Deficit Hyperactivity Disorder. There was some preliminary evidence collected for this meta-analysis demonstrating an improvement in parenting behaviors with IYPTP. Primary and replication studies showed statistically significant, moderate to large changes in reductions of negative parenting and increases in positive parenting. The latter finding was significantly greater for primary over that of replication studies. Because parenting measures varied greatly within different studies it was not possible to test to what degree initial measures of positive and negative parenting relate to found benefits from IYPTP. Limitations A potential limitation of this meta-analysis is that it was conducted on a fairly small number of studies. However, it is now possible to systematically compare these results to meta-analyses of other parent training programs (de Graff et al., 2008) and to conduct future systematic research that may also be compared to these findings. These cross-checks across similar research over time can produce meaningful conclusions for practice and future research. While a small number of studies included in this meta- analysis may be a limitation, this did allow for a careful analysis of dispersion across studies that might otherwise be obscured by larger data sets (6. g. larger data sets become more homogeneous by virtue of larger sampling). Small sample size is also not unique to 114 this meta-analysis. Weisz and colleagues (2005) noted small sample sizes within their meta-analysis of youth psychotherapy research studies from the past four decades and this has been noted for the studies across the larger field of psychological intervention (Kazdin, 2008). Effect Sizes calculated for studies included within this meta-analysis tell how much benefit was accrued from IYPTP intervention, avoiding the pitfalls of limitations inherent with null-hypothesis testing (Beutler, 2009; Hinshaw & Park, 1999; Kazdin, 2008; Kehle & Bray, 2006; Kehle et al., 2007; Olejnik & Algina, 2000; Sanabria & Killeen, 2007; Schmidt, 2009; Swaminathan & Rogers, 2007; Volker 2006). The meta- analytic procedures conducted for this study exceeded the rigorous standards of calculating effect sizes in isolation of one and other prescribed by the Procedural and Coding Manual for the Identification of E vidence-Based Interventions (Task Force on Evidence Based Interventions in School Psychology, 2003) by further synthesizing various effect sizes relative to their levels of common homogeneity across multiple primary and replication studies of IYPTP. This methodology sought to maintain the integrity of the core mechanism of change for IYPTP while systematically considering what variables relate to outcome differences for whom and under what conditions (Bhattacharyya et al., 2009; Fixsen et al., 2009; Ingraham & Oka, 2006; Sheridan 2005). While the magnitude of an effect from an EBI study (small, medium or large) holds some significance (Cohen, 1988), comparison of measured benefits relative to those previously obtained in the same research area or type of outcome as was done for this study, provides useful information about the actual value of an EBI (Durlak, 2002). 115 The bulk of meta-analysis was performed on between-groups studies of IYPTP while the within-groups studies lacked adequate numbers to perform additional analyses. The effect sizes used in the between-groups meta-analysis (Hedges’ g) were calculated using the most rigorous methodology available, whereby differences between pre- and post- means were divided by the pooled pre-intervention standard deviations (Becker, 1988) and were adjusted for small sample sizes (Carlson & Schmidt, 1999; Hedges & Olkin, 1985). These methods produced the most conservative effect size value for studies with smaller sample sizes. Nevertheless, separation of studies across different forms and uses of IYPTP produced comparisons between small groups of studies. Considering the small number of studies and the small number of subjects within a group of studies (summarized in Table 16) one should use caution in interpreting the results from these comparisons. One way to do this is by interpretation of effect sizes within the context of their confidence intervals as was done for this study. Finding no significant differences between confidence intervals for grand mean effect sizes representing two groups of studies (primary and replication) may be at least partially an artifact of smaller sample sizes, warranting further confirmation as more studies may be added to address this primary research question. One other potential limitation of this study is that effect sizes representing each study were calculated by combining parent rating scale scores with independent observations of behavior for both child and parent dependent variables while excluding interview data that was only available within some studies. Further analysis of this excluded data would clarify to what degree this restriction of data may have affected results from this study. 116 Careful review of the between-groups studies resulted in conducting meta- analysis for three tiers of IYPTP intervention. Separating between-groups studies into three tiers of intervention supported equitable comparisons between primary and replication studies of IYPTP where unique benefits would be anticipated for differing uses (e. g. universal prevention, selective and indicated intervention). Webster-Stratton and Herman (2010) refer to IYPTP distinguish between universal prevention, selective and indicated interventions somewhat differently than was determined for this meta- analysis. For example, they consider IYPYP in Head Start Centers as a form of selective intervention whereas this meta-analysis categorized these as universal prevention since poverty is a risk factor for many child and family dysfunctions not just conduct problems. The distinction between three levels of tiered service delivery lies at the heart of the National Association of School Psychologists “Blueprint for Training and Practice, Third Edition” (2006). Evidence-based interventions Should be viewed according to the purpose(s) they serve across tiers. This meta-analytic summary of a roughly thirty year history of IYPTP intervention studies supports deployment at all three tiers of intervention, where significant benefits were found in ameliorating child conduct problems and improving parenting behaviors. Future Directions for Research Several recommendations for future research are offered based on the findings of this meta-analysis. First, the absence of replication between-groups and within-groups studies of the self-administered form of IYPTP points to a notable gap within the research literature on IYPTP needing to be filled by researchers independent of the primary 117 researchers. There would seem to be a good likelihood that replication studies of the self- administered form of IYPTP would find significant benefits in reducing child conduct problems and that this form may be quite amenable to reproductions with good fidelity in settings beyond the. primary researchers. Surprisingly the self-administered form of IYPTP has not been evaluated with sub-clinical child conduct problem populations where this may be found beneficial pointing to another area of needed research. Second, some child conduct problem cases may be more amenable and adequately treated by the self-administered form of IYPTP than others, thus research investigating how to make this determination may be beneficial (e. g. What benchmark can be used to increase the level of intervention from a tier two to a tier three?) Case study research as well as group-level research could add helpful information in this regard. Research studies driven by a tiered model of intervention described within this research study may offer clearer methods of measuring response to intervention, and subsequent delivery of greater resources based on demonstrated needs. Third, Webster-Stratton and colleagues have more recently developed two additional intervention components for the Incredible Years (IY) brand, adding child and teacher training to IYPTP. Subsequent meta-analyses may be beneficial in collecting primary and replication studies that include these additional components once there are more replication studies available. Since these additional treatment components represent a large amount of additional investment of time and resources a similar analysis to the one performed for this study will be beneficial in ascertaining what degree of benefit is found from these additional investments, and whether benchmarks set by the primary studies can be replicated. 118 Fourth, Dr. Webster-Stratton initially argued and has subsequently provided empirical support along with many others for the effective treatment of young children’s conduct problems using a parenting group format as an economical alternative to having therapists intervene with each family separately. Additional replication studies directly comparing the benefits of the group-administered IYPTP to that of other parent training interventions implemented with families separately would provide additional data regarding to what degree benefits may be similar or different from group- versus individual-family treatment of child conduct problems. These findings would also hold significant implications for deployment of limited resources based upon predicted benefits from each type of intervention. Fifth, IYPTP in its original, BASIC form consists of about 12-14 weekly, two- hour sessions (Webster-Stratton, 1996; Webster-Stratton, 2001; Webster-Stratton & Reid, 2010; Weisz, 2004). This meta-analysis found studies using 9-16 two-hour group sessions produced significant, immediate, moderate reductions in child conduct symptoms whereas 22-24 two-hour sessions of group-administered IYPTP accrued a large effect, roughly doubling the immediate benefit. These greater benefits from increased dosage still need to be verified by independent replication beyond primary researchers. Additional research is needed to demonstrate whether the significant benefit for 22-24 session dosage remains greater than that of the 9-16 session dosage for long term outcomes beyond immediate effects. A recent systematic review of child psychotherapy research studies found roughly fifty percent reported differential effects from varying dosage level, suggesting the presence of an active treatment ingredient (Jensen, Weersing, Hoagwood & Goldman, 2005). Further independent replication 119 research studies of IYPTP at an increased dosage would provide additional verification of the active treatment agent for IYPTP and its greater benefits with the most severe forms of child conduct problems and their parents at increased dosages. Sixth, more research is needed to identify the mechanism(s) of change within an intervention such as IYPTP (Kazdin, 2008). Jensen and colleagues (2005) asserted studies generally lack adequate controls to account for nonspecific therapeutic factors of positive expectancies, therapeutic alliance and attention which likely play a role in intervention effectiveness. Future IYPTP primary and replication research studies that measure these variables would provide additional information regarding possible IYPTP moderators of change. Eames and colleagues (2009) recently developed the “Leader Observation Tool” (TOL) to study implementer’s process skills using IYPTP. They used this research tool during the collection of data for one of the studies included in this meta- analysis (Hutchings et al., 2007). They found larger changes in positive parenting measured by independent observations were related to group leader’s process skills. Among leader behaviors including listening, empathy, physical encouragement, and negative behaviors the category of “positive behavior” was found to be Significantly related to differences in parent-reported parenting styles, while empathy skills showed a similar trend approaching statistical significance. Of particular relevance to the aforementioned discussion about needing to locate mechanisms of change, Eames and colleagues (2009) found changes in positive parenting predicted improvements in child conduct behaviors. This demonstrates changes in parenting from IYPTP do serve as a mechanism of change for improvements in child conduct problems. Further research delving into IYPTP group-leader process skills using the TOL or other similar measures 120 would add significant and important information toward additional successful replications of IYPTP by practitioners within authentic community settings. Additional measures of more nonspecific factors such as the favorability of therapist-parent relationship would also add to our knowledge about what works best and for whom. Webster-Stratton and Herman (2010) place a premium value on not only adequate fidelity of treatment content, but also on the meaningful interpersonal processes achieved within the training milieu. The latter qualities go beyond checklists for coverage of prescribed content and delve into the therapist’s adaptation of the intervention to meet the unique needs of various treatment recipients. At this time, these qualities are only judged by those certified as mentors in the IYPTP, using reviews of videotaped session delivery. What qualities are evaluated at this level of clinical implementation and treatment adaptation should be further identified and studied using controlled research to further define what matters most in treatment deployment beyond the number of role-plays and video vignettes presented. Seventh, the number of studies that report on observed changes in positive and negative parenting were quite small for the group included within this meta-analysis pointing to a need for these measures to be included within future research on parent training addressing child conduct problems. Only a handful of studies have already demonstrated changes in parenting mediate changes in child conduct problems (Beauchaine et al., 2005; DeGarmo et al., 2004; Eames et al., 2009). This meta-analysis found significant, moderate effects for reductions in negative parenting and increases in positive parenting consistent with these findings. Additionally, increases in positive parenting were significantly greater for primary versus replication studies, although the 121 small number of studies this finding was based on preclude any meaningful conclusions. Eames and colleagues (2009) finding of a significant correlation between positive group- leader behavior and that of improvements in positive parenting that are associated with changes in child conduct problems strongly points to the importance of further research examining the role of positive behaviors and their benefits toward altering undesirable child behaviors. Kazdin (2005) noted IYPTP goes beyond the typical behavioral, operant l conditioning methods of many other parent training programs, and similarly the manual .. for IYPTP places a high value on working to establish positive relationships between parents and children before proceeding with more behaviorally-oriented methods of child management (Webster-Stratton, 2001). It is hypothesized for future research, that the magnitude of change in positive parenting plays a significant role in achieving higher benefits from IYPTP. Unfortunately for the small number of studies included in this meta-analysis there was not enough data to support research methods such as meta- regression to test this hypothesis. Eighth, recognizing studies in psychology are often small in size would seem to warrant calculation of effect sizes using methods such as those contained within this meta-analysis that account for smaller sample sizes (Hedges’ g). If the sampling of studies found for this research is any indication of the larger reporting of effect sizes within the profession, one should maintain caution and reservation regarding effect sizes that do not fully account for sample size, especially where samples are lower than twenty. Additionally, where effect sizes were reported in studies it was rare to also find reporting of confidence intervals for these effect sizes. In the absence of the latter, reporting of effect sizes as absolute values is just as problematic as reporting of significant p-values. 122 To properly interpret effect sizes and to compare them, confidence intervals are needed to estimate the accuracy of an effect size estimate relative to the larger group this statistic is believed to represent. Ninth, meta-analyses are most productive when theory is integrated with empirical findings across studies such as was done for this study. Future meta-analyses must go beyond removal of outliers to obtain adequately homogeneous sets of studies supportable by current theories. The most stable, reliable and valid estimates of intervention potency and robustness can be demonstrated best when evidence contained within multiple studies of an intervention are weighted and systematically examined using meta-analysis rather than using vote counting and narrative reviews to determine an evidence base for an intervention. Tenth, an emphasis on evidence-based intervention requires careful attention to many complex issues, not the least of which is answering, “What is an evidence-based intervention?” Many professional organizations within psychology and allied professions have organized thoughtful sets of criteria by which interventions are bestowed the “evidence-based” badge (Task Force on Evidence Based Interventions in School Psychology, 2003). These procedures often distinguish between higher and lower levels of evidence such as “promising” and “well established” (Eyberg et al., 2008). However, many of the methods used for evaluating research studies of an EBI amount to “vote counting” (Borenstein et al., 2009) whereby the credibility for an EBI is determined by the accumulated presence of several statistically significant results. This is highly problematic from a statistical perspective given the undeserving equal weight each p- value is afforded using this methodology. Null hypothesis tests reporting p-values tel] 123 nothing about the true effect from the intervention (Hinshaw & Park, 1999; Kazdin 2008; Kehle & Bray 2006; Sanabria & Killeen, 2007). Borenstein and colleagues (2009) acknowledge the intuitive appeal, yet unsupportable notion of adding up the number of p- values (i.e. vote counting) to provide more accurate decisions than those based on isolated significance test results. However, obtained p-values are largely a function of study power. Non-significant p-values do not necessarily mean an effect is absent, and significant p-values may represent a small effect with little practical significance. Without the systematic weighing of evidence for an intervention as was done by this study, so called evidence-based practice decision-making may likely resemble the world of politics (e. g. vote counting) or popularity contests (e. g. cherry picking), which may be more about factors other than truth (McNeil, 2006). The latter undesirable practices make the profession more vulnerable to decision-making based more on “fads and the bandwagon effect’ rather than cumulative scientific knowledge (Kehle & Bray, 2007; Trachtman, 1981). The methodology used to downgrade IYPTP from a “well- established” to a ‘probably efficacious’ intervention (Eyberg et al., 2008) essentially amounts to vote-counting of significant p-values and relied on a criterion of finding only fifty percent of reliable and valid outcome measures needing to show superiority for intervention versus control groups. However, this meta-analysis weighted findings for both IYPTP primary and replication research studies meeting inclusion criteria for this study using all parent rating scale and independent observation data reported. This procedure yielded strong and compelling reasons to assert IYPTP has a “well established” evidence-base across both primary and independent replication studies. For combined parent rating scale and independent observation outcomes (Table 10) and for 124 each of these outcomes separately (Tables 11 and 12 respectively) there are statistically significant benefits found for both primary and replication studies amounting to small to moderate to large effects depending on use. Differential outcomes relative to varying dosages as was found from this meta-analysis suggests future reviews of interventions need to account for dosage levels when ascribing a level of evidence base. Eleventh, this meta-analysis relied on quantitative studies of IYPTP. These findings would be supplemented by additional qualitative research. For example, combining quantitative and qualitative research methods to study IYPTP implementation within a “real world community prevention practice” identified several barriers practitioners encountered relative to treatment fidelity (Stern, Alaggia, Watson & Morton, 2008). The researchers found a group of community based practitioners implemented IYPTP with a high degree of adherence to the treatment manual with three exceptions. Practitioners deviated from prescribed treatment in their use of role plays, the dosages of videotaped modeling and their use of “buddy calls.” These findings have important implications relative to insuring treatment integrity and replication of IYPTP. Another qualitative study of a different parent training program implementation highlighted important, unintended consequences from the intervention not identified by quantitative methods (Mockford & Barlow, 2004). These researchers noted many mothers participating in the intervention experienced difficulties gaining the support of their partners in using new methods taught in the training, and that this led to increased discrepancies in parenting between partners, and increased parent conflicts. This finding has important implications to consider relative to a variable that may need to be accounted for relative to measuring benefits from an intervention such as IYPTP. 125 Twelfth, related to the aforementioned statistical support for IYPTP across primary and replication research studies, what methods are most appropriate to determine the level of evidentiary support for an EBI should be reviewed. Alternatives such as the methodology used for this meta-analysis might be considered for those interventions amassing multiple replication studies. While this method is much more time consuming, practice-based decision-making about EBI adoption should be made from “usable knowledge” (Rosenfield, 2000) and recipients deserve decisions based on the best evidence available (F ixsen et al., 2009). To make research usable by practitioners requires more studies like this one, that take a “Consumers Report” approach to weighing accumulated studies of an intervention using systematic, reproducible and transparent methods. Where there has been a recent proliferation of intemet web sites purportedly describing levels of evidence-base for interventions, the professions of psychology and other allied mental health providers would be most accurately guided by research when this is accumulated and summarized regularly using accepted and transparent methods of meta-analysis. There is a frontier of research still to be done in order to make sense of the substantial number of research studies accumulating within a vast and ever-expanding research literature. More research is not necessarily better or more helpful unless we can make sense of, and integrate new information with what has already been studied. Thirteenth, statistics differentiating IYPTP Tier Two studies from Tier Three studies revealed statistically significant differences for ECBI scores between these two groups. These findings were consistent with more recent, suggested rc-standardization norms for the ECBI. Colvin and colleagues (1999) have suggested using a cut-off score of 15 and above to define clinical significance on the Problems Scale rather than the 126 former norm of 11 and above. The findings for this meta-analysis provide further evidence for this suggested change. Implications for Practice This meta-analysis study offers several important implications for practice. First, practitioner’s selection of an intervention should rely on more than just whether it has produced a level of statistical significance unlikely produced by chance. Practitioners should make informed decisions to adopt an intervention based on answers to several important questions, including but not limited to these: 1.) What is the anticipated level of benefit for a particular use (i.e. universal prevention, selected or indicated intervention) and for whom? 2.) Have these benefits been replicated by others outside of the primary researchers, and do they achieve similar benefits? 3.) What level of problem severity is to be addressed? 4.) What level of dosage is necessary to achieve the desired benefits? 5.) What level of resources will be needed to implement with adequate fidelity? Second, specific to this meta-analysis, IYPTP is found to be an intervention with a substantial evidence base well beyond the primary researchers that warrants adoption by practitioners in authentic community-based practice settings to address young children either at-risk or currently exhibiting child conduct problems. It has been successfully transported and disseminated by replication research studies outside the realm of primary research, across a three-tiered model of prevention and intervention. Several of these replications have used practitioners in authentic community settings to deliver the 127 intervention versus highly-controlled university research contexts. When use and dosage of IYPTP are accounted for, IYPTP produces similar, significant levels of benefit in reducing child conduct problems for both primary and replication research studies. The findings from this study suggest inventor-based benchmark levels of benefit can be achieved beyond the primary researcher when there is a high degree of therapist training and qualifications along with weekly supervision specific to the delivery of IYPTP. These findings suggest practitioners maintaining fidelity of treatment using IYPTP can L expect similar, significant benefits in reductions for child conduct problems, diminution of negative parenting and increases in positive parenting. For clinically-significant child conduct problem cases practitioners should find the average benefit from BASIC IYPTP at a dosage of 9—16, two-hour sessions to be about a half-standard deviation reduction in child conduct problems while an increased dosage of between 22—24 two-hour sessions may lead to an average benefit of one standard deviation benefit in child conduct problems. These benefits would be according to combined parent rating scale and independent observations of child behaviors. Third, findings from this meta-analysis suggest practitioners implementing IYPTP should likely find average benefits of about one half-standard deviation in reductions of negative parenting. Replication studies suggest about the same level of benefit in increased positive parenting while primary research suggests much greater, one standard deviation changes may be possible. Eames and colleagues (2009) recent research suggests positive and empathic group-leader behaviors during IYPTP delivery serves a particular benefit for increasing positive parenting behaviors and subsequent reductions in child conduct problems. Practitioner deliveries of group-administered IYPTP are 128 encouraged to emphasize positive interactions with parents that include empathy for their experiences of parenting a child with significant conduct problems. Considering these findings practitioners implementing IYPTP are strongly advised to not lose sight for how important it is to devote initial and ongoing efforts toward the development of strategies for encouraging positive relationships between group leaders and participants, as well as between parents and their children prior to engaging in implementation of more aversive strategies for child behavior management. Fourth, the more time-consuming and high demand of resources needed for the group-administered form of IYPTP over that of the more economical self-administered form only appears warranted for the most severe cases of child conduct problems. The feasibility of investing resources in practice necessary for the group-administered form of IYPTP over that of the self-administered form for treating clinically-significant child conduct problems should be determined by the degree to which resources are available to ensure therapists are highly trained and supervised by qualified mentors of IYPTP, and treatment integrity is ensured by high levels of weekly expert supervision. The self- administered form of IYPTP may serve as an initial gateway procedure for determining a need for the consumption of much greater resources to deliver the group-administered form. In other words, initial treatment may be provided by use of the self-administered form for those families amenable to this level of intervention. Those not responding adequately would then be afforded the group—administered form. Foster and Roberts (2007) found about one-third of parents receiving a videotape parent training (not IYPTP) to handle clinic referred disobedient preschool children needed therapist assistance for initial compliance training. Sougstad and colleagues’ (2008) meta-analytic comparison 129 of primary studies using self-administered IYPTP with that of self-administered IYPTP along with therapist consultation did not find significantly better outcomes from the additional provision of therapist consultation to the self-administered program. While the self-administered form of IYPTP appears to be an attractive intervention it also remains limited. Benefits are about half that of an intensive 22-24 week, two-hour deployment of the group-administered form and there is a large number of cases that do not adequately benefit from the self-administered form even with therapist consultation. 130 Appendix A IYPTP Meta-Analysis Dissertation (Sougstad, 2010) Title of Study: Publication Name: Authors Names: Pub Date: 1.) [:1 Incredible Years Parent Training Interven. & [J Child Conduct Problems Outcome 2.) [:1 Primary Inventor-Based OR Cl Independent Replication 3.) [:1 Random Assignment (exp vs. control grp) [:1 Other: 4.) Cl Measures of treatment fidelity included (Type: 5.) Cl Paid Participants OR El Unpaid Participants 6.) Cl Recruited Participants OR El Referred for Treatment 7.) Parents: [:1 unskilled OR El semi-skilled OR [:1 skilled 8.) Cl 2-parent family OR Cl Single-parent family 9.) Cl Conduct Probs with ADD/Inattention, [:1 Conduct Probs with ADHD, Combined 1:1 ADHD Only [:1 Conduct Problems Only 10.) [J United States, [:1 Canada, Cl Denmark, 1:] Norway, Cl Sweden, 13 Ireland, [:1 Wales, 1:1 England, D South America: Venezuela, other: C] S. Korea, Cl 1:] El l 1.) Therapist Characteristics: Therapist Training Background: El Non-professional (para-pro, etc,. . .), [:1 Graduate stds. [:1 Profess. (MA, MSW, PhD) 131 I YPT P level of Therapist Training: [:1 No formal training reported, [:1 Some training not from credentialed IY staff El 3-day training from credentialed IY staff Cl 3-day training from credentialed staff PLUS additional supervision Cl Experts (certified in IYPTP, affiliated with IY and/or U. of Wash; Par Clinic) 12.) Mother Report: El, Father Report: 1:], Mother & Father Combined Rpt.: [:1 l3.) Equivalence of control and experimental groups measured: 14.) Initial mean conduct problems score based on rating scale score: 15.) Initial mean conduct problems score based on indep. observer: 16.) Initial mean Negative parenting score (self-report rating): 17.) Initial mean Negative parenting score (indep. obs.): 18.) Initial mean Positive parenting score (self-report rating): 19.) Initial mean Positive parenting score (indep. obs.): 132 References * = Primary Studies # = Independent Replication Studies Aarons, G. A. (2005). Measuring provider attitudes toward evidence based practice: Organizational context and individual differences. Child and Adolescent Psychiatric Clinics of North America, 14, 255—271. Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service Provider Perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34, 411-419 Aarons, G. A., & Sawitzky, A. (2006). Organizational culture and climate and mental health provider attitudes toward evidence based practice. Psychological Services, 3, 61—72. Achenbach, T. M., & Edelbrock, C. S. (1991). Manual for the child behavior checklist and revised child behavior profile. Burlington, VT: University Associates in Psychiatry. Achenbach, T. M., Dumenci, L., & Rescorla, L. A. (2003). DSM-oriented and empirically based approaches to constructing scales from the same item pools. Journal of Clinical Child and Adolescent Psychology, 32, 328-340. American Psychological Association. (2002). Criteria for evaluating treatment guidelines. American Psychologist, 57, 1052-1059. American Psychological Association’s Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61, 271-285. American Psychological Association Task Force on Evidence-Based Practice for Children and Adolescents. (2008). Disseminating evidence-based practice for children and adolescents: A systems approach to enhancing care. Washington, DC: American Psychological Association. Ang, R. P., & Hughes, J. N. (2002). Differential benefits of skills training with antisocial youth based on group composition: A meta-analytic investigation. School Psychology Review, 31, 164-185. Anthonysamy, A., & Zimmer-Gembeck, M. J. (2007). Peer status and behaviors of maltreated children and their classmates in the early years of school. Child Abuse & Neglect, 31, 971-991. 133 Arnold, M. E., & Hughes, J. N. (1999). First do no harm: Adverse effects of grouping deviant youth for social skills training. Journal of School Psychology, 3 7, 99—115. #Axberg, U., Hansson, K., & Broberg, A. G. (2007). Evaluation of the Incredible Years Series — An open study of its effects when first introduced in Sweden. Nordic Journal of Psychiatry, 61, 143-151. Bates, S. L. (2005). Evidence-based family-school interventions with preschool children. School Psychology Quarterly, 20, 352-370. Beauchaine, T. P., Webster-Stratton, C., & Reid, M. J. (2005). Mediators, moderators, and predictors of l-year outcomes among children treated for early-onset conduct problems. A latent growth curve analysis. Journal of Consulting and Clinical Psychology, 73, 37 l—-3 88. Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41, 257-278. Berkowitz, S. J. (2003). Children exposed to community violence: The rationale for early intervention. Clinical Child and Family Psychology Review, 6, 293-302. Beutler, L. (2009). Making science matter in clinical practice: Redefining psychotherapy. Clinical Psychology: Science and Practice, 16, 301-317. Bhattacharyya, O., Reeves, 8., & Zwarenstein, M. (2009). What is implementation research? Rationale, concepts, and practices. Research on Social Work Practice, 19, 491-502. Bickman, L., Lambert, E. W., Andrade, A. R., & Penaloza, R. (2000). The Fort Bragg continuum of care for children and adolescents: Mental health outcomes over 5 years. Journal of Consulting and Clinical Psychology, 68, 710-716. Bloomquist, M. L. & Schnell, S. V. (2002). Helping children with aggression and conduct problems, best practices foriIntervention. New York: The Guilford Press. Borenstein, M., Hedges, L. V., Higgins, J ., & Rothstein, H. (n.d.). Comprehensive Meta- Analysis (Version 2. 0) [Computer software]. Englewood, NJ: Authors. Borenstein, M., Hedges, L. V., Higgins, J ., & Rothstein, H. (2009). Introduction to Meta- Analysis. Chichester, West Sussex, United Kingdom: John Wiley and Sons Ltd. Publications. *Brotman, L. M., Gouley, K. K., Chesir—Teran, D., Dennis, T., Klein, R. G., & Shrout, P. (2005). Prevention for preschoolers at high risk for conduct problems: Immediate outcomes on parenting practices and child social competence. Journal of Clinical Child and Adolescent Psychology, 34, 724-734. 134 *Brotman, L. M., Klein, R. G., Kamboukos, D., Brown, E. J ., Coard, S. I., & Sosinsky, L. S. (2003). Preventive intervention for urban, low-income preschoolers at familial risk for conduct problems: A randomized pilot study. Journal of Child and Adolescent Psychology, 3, 246-257 Brestan, E. V., & Eyberg, S. M. (1998). Effective psychosocial treatments of conduct- disordered children and adolescents: 29 years, 82 studies, and 5,272 kids. Journal of Clinical Child Psychology, 27, 180-189. Bronson, MB. (2000). Self-regulation in early childhood, nature and nurture. New York: Guilford. Brophy, J. (1995). Elementary teachers perceptions of and reported strategies for coping with twelve types of problem students. East Lansing, MI: Institute for Research on Teaching, Michigan State University. ED 389 390. Retrieved from the World Wide Web October 30, 2007: http://www.eric.ed.gov/ERICDocs/data/ericdocsqul/content_storage_0l /0000019b/ 80/ l 4/44/bf.pdf Burns, G. L., & Walsh, J. A. (2002). The influence of ADHD-hyperactivity/impulsivity symptoms on the development of oppositional defiant disorder symptoms in a 2-year longitudinal study. Journal of Abnormal Child Psychology, 30, 245-256. *Bywater, T., Hutchings, J ., Linck, P., Whitaker, C. J ., Daley, D., Yeo, S., & Edwards, R. T. (In Press). Incredible Years Parent Training support for foster carers in Wales: A multi-centre feasibility study. Retrieved from http://incrediblevears.corn/Library/paper.asp?nMode:1&nLibrarvID=6l4 Cancelli, A. A., Lange, S. M., Lopez, E., Kerwin, C., Matalon, T., Perlman, L., . . . Hanson, J. (1989). A study of guidelines for the conduct and analysis of simultaneous clinical replications. Professional School Psychology, 4, 295-305. Capaldi, D., lDeGarmo, D., Patterson, G. R., & Forgatch, M. (2002). Contextual risk across the early life span and association with antisocial behavior. In J. B. Reid, G. R. Patterson, & J. Snyder (Eds.) Antisocial Behavior in Children and Adolescents: A Developmental Analysis and Model for Intervention. (pp. 45-64) Washington, DC: American Psychological Association. Carlson, K.D., & Schmidt, FL. (1999). Impact of experimental design on effect size: Findings from the research literature on training. Journal of Applied Psychology, 84, 851-862. Castro, F. G., Barrera, M., & Steiker, L. K. H. (2010). Issues and challenges in the design of culturally adapted evidence-based interventions. Annual Review of Clinical Psychology, (Vol. 6, pp. 213-239). Cedar, B., & Levant, R. F. (1990). A meta-analysis of the effects of parent effectiveness training. The American Journal of Family Therapy, 18, 373-384. 135 Chafouleas, S. M., & Riley-Tillman, T. C. (2005). Accepting the gap: Introduction to the special issue of bridging research and practice. Psychology in the Schools 42, 455- 458. Chambless, D. L., & Hollon, S. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology. 66, 7-18. Chambless, D. L. & Ollendick, T. H. (2001). Empirically supported psychological interventions: Controversies and Evidence. Annual Review of Psychology, 52, 685- 716. Chorpita, B. F. (2003). The frontier of evidence-based practice. In A. E. Kazdin and J. R. Weisz (Eds.) Evidence-based psychotherapies for children and adolescents (pp. 42- 59). New York: Oxford. Chorpita, B. F ., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566-579. Chorptita, B. F., & Regan, J. (2009). Dissemination of effective mental health treatment procedures: Maximizing the return on a significant investment. Behaviour Research and Therapy, xxx, 1-4. Chronis, A. M., Chacko, A., Fabiano, G. A., Wymbs, B. T., & Pelham, W. E. (2004). Enhancements to the behavioral parent training paradigm for families of children with ADHD: Review and future directions. Clinical Child and Family Psychology Review, 7, 1-27. *Clondalkin Pamership (2006). Incredible Years Basic Parenting Programme: The Clondalkin Partnership Study tn Ireland. Retrieved from: htt ./ /www.mcred1ble ars.com/L1bra / . PnMode=18£nLibra D=594 Cohen, J. (1988). Statistical power for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates. Cohen, M. A. (1998). The monetary value of saving a high-risk youth. Journal of Quantitative Criminology, 14, 5-33. Colvin, A., Eyberg, S. M., & Adams, C. D. (1999). Restandardization of the Eyberg Child Behavior Inventory. [On-line]. Available: mod/neitphhpufl.edu/Measures/ECB1%20Restandardization.pdf Last accessed: 11/06/10. Conduct Problems Prevention Research Group (1996). Parenting Practices In ventory. Unpublished manuscript. University of Washington: Seattle. 136 Cooper, H., & Hedges, L. V. (1994). Potentials and limitations of research synthesis. In H. Cooper & L. V. Hedges, (Eds.). The handbook of research synthesis. (pp. 521- 528). New York: Russell Sage Foundation. DeGarmo D. 8., Patterson G. R., & F orgatch M. S. (2004). How do outcomes in a specified parent training intervention maintain or wane over time? Prevention Science, 5, 73-89. deGraaf, I., Speetjens, P., Smit, F., de Wolff, M., & Tavecchio, L. (2008). Effectiveness of the Triple-P Positive Parenting Program on behavioral problems in children. Behavior Modification, 32, 714-735. Dishion, T. J., McCord, J ., & Poulin, F. (1999). When interventions harm: Peer groups and problem behavior. American Psychologist, 54, 755-764. Dodge, K. A. (2001). The science of youth violence prevention; Progressing from developmental epidemiology to efficacy to effectiveness to public policy. American Journal of Preventive Medicine, 20, 63-70. Dodge, K. A., Dishion, T. J ., & Lansford, J. E. (2006). Deviant peer influences in programs for youth: problems and solutions. New York: Guilford. Duncan, R. D. (1999). Peer and sibling aggression: An investigation of intra- and extra- familial bullying. Journal of Interpersonal Violence, 14, 871-886. Dunst, C. J ., Hamby, D. W., & Trivette, C. M. (2004). Guidelines for calculating effect sizes for practice-based research syntheses. Centerscope, 3, l -— 10. Durlak, J. A. (2002). Evaluating evidence-based interventions in school psychology. School Psychology Quarterly, 1 7, 475-482. Durlak, J. A. (2003). Appendix B: General guidelines for conducting a literature review. In Procedural and Coding Manual for the Identification of Evidence-Based Interventions. Retrieved July 1, 2007 from Task Force on Evidence-Based Interventions in School Psychology Web site: http://wwwsp- ebi.org/documents/_workingfiles/EBlmanual 1 .pdf Eames, C., Daley, D., Hutchings, J ., Whitaker, C. J ., Jones, K., Hughes, J. C., & Bywater, T. (2009). Treatment fidelity as a predictor of behaviour change in parents attending group-based parent training. Child: care, health and development, 35, 603-612. Eddy, J. M., Reid, J. B., & Curry, V. (2002). The etiology of youth antisocial behavior, delinquency, and violence and a public health approach to prevention. In M. R. Shinn, H. M. Walker & G. Stoner (Eds.) Interventions for Academic and Behavior Problems 11: Preventive and Remedial Approaches. (pp. 27-52) Bethesda, MD: National Association of School Psychologists. 137 E g ger, H.L., & Angold, A. (2006). Common emotional and behavioral disorders in preschool children: Presentation, nosology and epidemiology. Journal of Child Psychology and Psychiatry, 4 7, 313-337. E yberg, S. M., Boggs, S. R., & Algina, J. (1995). Parent child interaction therapy: A psychosocial model for the treatment of young children with conduct problems behavior and their families. Psychopharmacology Bulletin, 31, 83-91. Eyberg, S. M., Nelson, M. M., & Boggs, S. R. (2008). Evidence-based psychosocial treatments for children and adolescents with disruptive behavior. Journal of Clinical Child & Adolescent Psychology, 3 7, 215-237. E)’ berg, S. M., & Ross, A. W. (1978). Assessment of child behavior problems: The validation of a new inventory. Journal of Clinical Child Psychology, 7, 113-116. F eI‘gnsson, D. M., Horwood, L. J ., & Ridder, E. (2005). Show me the child at seven: The consequences of conduct problems in childhood for psychosocial functioning in adulthood. Journal of Child Psychology & Psychiatry, 46, 83 7-849. * F ergusson, D., Stanley, S. L., & Horwood, J. (2009). Preliminary data on the efficacy of the Incredible Years Basic Parent Programme in New Zealand. Australian and New Zealand Journal of Psychiatry, 43, 76-79. F it“Ikelhor, D., Ormrod, R., Turner, H., & Hamby, S. L. (2005). The victimization of children and youth: a comprehensive, national survey. Child Maltreatment, 10, 5-25. F i)(sen, D. L., Blase, K. A., Naoom, S. F ., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19, 531-540. Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., Ji, P. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, xxx, 1-25 Forman, S. G., Fagley, N. S., Steiner, D. D., & Schneider, K. (2009). Teaching evidence- based interventions: Perceptions of influences on use in professional practice in school psychology. Training and Education in Professional Psychology, 3, 226-232. F ortin, L. (2003). Students’ antisocial and aggressive behavior: Development and prediction. Journal of Educational Administration, 41 , 669-688. Foster, B. T. & Roberts, M. W. (2007). Training with videotapes: Recognizing limitations. Child & Family Therapy, 29, 21-35. Frick, P. J. (2004). Developmental pathways to conduct disorder: implications for serving youth who show severe aggressive and antisocial behavior. Psychology in the Schools. 41, 823-834. 138 *Gardner, F ., Burton, J ., & Klimes, I. (2006). Randomised controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: Outcomes and mechanisms for change. Journal of Child Psychology and Psychiatry, 4 7, 1123- 1132. Gerler, E. R. (2004). Handbook of school violence. New York: Haworth Press. Gilliam, W. S. (2005). Prekindergartners Left Behind: Expulsion Rates in State Prekindergarten Programs. New York: Foundation for Child Development. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-Analysis in social research. Newbury Park, CA: Sage Publications. Glicken, M. D. (2004). Violent young children. Boston, MA: Pearson Education, Inc. Goldstein, S., & Rider, R. (2006). Resilience and the disruptive disorders of childhood. In S. Goldstein & R.B. Brooks (Eds.). Handbook of resilience in children. (pp. 203- 222). New York: Springer. Goodman R (1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581-586. #Gross, D., Fogg, L. Webster-Stratton, C., Garvey, C., Julion, W., & Grady, J. (2003). Parent training with families of toddlers in day care in low-income urban communities. Journal of Consulting and Clinical Psychology, 71, 261-278. #Kim, B, Cain, K. C., & Webster-Stratton, C. (2008). The preliminary effect of a parenting program for Korean American mothers: A randomized controlled experimental study. International Journal of Nursing Studies, 45, 1261- 1273. Graczyk, P. A., Domitrovich, C. E., Small, M., & Zins, J. E. (2006). Serving all children: An implementation model framework. School Psychology Review, 35, 266-274. Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J ., Caswell, W. & Robinson, N. (2006). Lost in knowledge translation: Time for a map? Journal of Continuing Education in the Health Professions, 26, 13-24. Grimes, C. L., Klein, T. P. & Putallaz, M. (2004). Parents’ relationships with their parents and peers: Influences on children’s social development. In J .B. Kupersmidt & K.A. Dodge (Eds.) Children ’s peer relations. (pp. 141-158). Washington, DC: American Psychological Association. Grolnick, W. S., & Apostoleris , N. H. (2002). What makes parents controlling?. In E. L. Deci & R. M. Ryan (Eds.), Handbook of self-determination research (pp. 161-181). Rochester, NY: University Of Rochester Press. Grolnick, W. S., Deci, E. L., & Ryan, R. M. (1997). Internalization within the family: The self-detennination theory perspective. In J. E. Grusec & L. Kuczynski (Eds.), 139 Parenting and children 's internalization of values: A handbook of contemporary theory (pp. 135-161). New York: Wiley. Hartman, R. R., Stage, S. A., & Webster-Stratton, C. (2003). A grth curve analysis of parent training outcomes: Examining the influence of child risk factors (inattention, impulsivity, and hyperactivity problems), parental and family risk factors. Journal of Child Psychology and Psychiatry, 44, 388-398. Hedges, L.V. (1994). Statistical considerations. In H. Cooper & L.V. Hedges, (Eds.). The handbook of research synthesis. (pp. 29—3 8). New York: Russell Sage Foundation. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Toronto, Ontario, Canada: Academic Press. Henggeler, S. W., Schoenwald, S. K., Borduin, C. M., Rowland, M. D., & Cunningham, P. B. (2009). Multisystemic treatment of antisocial behavior in children and adolescents (2nd ed.). New York: Guilford Press. Hinshaw, S. P. (1987). On the distinction between attention deficits/hyperactivity and conduct problems/aggression in child psychopathology. Psychological Bulletin, 101, 443-463. Hinshaw, S.P., & Park, T. (1999). Research problems and issues: Toward a more definitive science of disruptive behavior disorders. In H.C. Quay & A.E. Hogan (Eds.) Handbook of disruptive behavior disorders. (pp. 593-620) New York: Kluwer Academic/Plenum Publishers. Hoagwood, K., Burns, B.J., & Weisz, J .R. (2002). A profitable conjunction: From science to service in children’s mental health. In B]. Burns & K. Hoagwood (Eds.) Community treatment for youth, evidence-based interventions for severe emotional and behavioral disorders. (pp. 327-338) New York: Oxford University Press. Hoagwood & Johnson (2003). School psychology: a public health framework 1. From evidence-based practices to evidence-based policies. Journal of School Psychology. 41, 3-21. Hughes, T.L., Crothers, L.M., & J imerson, SR. (2008). Identifying, assessing and treating conduct disorder at school. New York: Springer. Hughes, T. L., Kaufman, J ., & Miller, J. A. (2010). Is everything old new again: School psychology training past, present, and future. In T. L. Hughes & Kaufman, J. (Eds.) Handbook of education, training and supervision of school psychologists in school and community. Hunsely, J ., & Lee, C. M. (2007). Research-informed benchmarks for psychological treatments: Efficacy studies and beyond. Professional Psychology: Research and Practice, 38, 21-33. 140 *Hutchings, J ., Gardner, F ., Bywater, T., Daley, D., Whitaker, C., & Jones, K. (2007). Parenting intervention in sure start services for children at risk of developing conduct disorder: Pragmatic randomised controlled trial. BMJ: British Medical Journal, 334 (7595) Ingraham, C. L., & Oka, E. R. (2006). Multicultural issues in evidence-based interventions. In B. K. Nastasi (Ed.) Multicultural issues in school psychologr New York: Haworth Press. Jaffee, S. R., Belsky, J., Harrington, H., Caspi, A., & Moffitt, T. E. (2006). When parents have a history of conduct disorder: How is the caregiving environment affected? Journal of Abnormal Psychology, 115, 309-319. Jaffee, S. R., Caspi, A., Moffitt, T. E., & Taylor, A. (2004). Physical maltreatment victim to antisocial child: Evidence of an environmentally mediated process. Journal of Abnormal Psychology, 113, 44-55. Jensen, P. S., Martin, D., & Cantwell, D. P. (1997). Comorbidity in ADHD: Implications for research, practice, and DSM-V. Journal of the American Academy of Child and Adolescent Psychiatry, 36, 1065- 1079. Jensen, P. S., Weersing, R., Hoagwood, K. E., Goldman, E. (2005). What is the evidence for evidence-based treatments? A hard look at our soft underbelly. Mental Health Services Research, 7, 53 — 74. *Jones, K., Daley, D., Hutchings, J ., Bywater, T., & Eames, C. (2007). Efficacy of the Incredible Years Basic parent training programme as an early intervention for children with conduct problems and ADHD. Child: care, health and development, 33, 749-756. Kamphaus, R. W., & Frick, P. J. (2002). Clinical Assessment of child and adolescent personality and behavior (2nd ed.). Boston, MA: Allyn and Bacon. Kazak, A. E., Hoagwood, K., Weisz, J. R, Hood, K., Kratochwill, T. R., Vargas, L. A., & Banez, G. A. (2010). A meta-systems approach to evidence-based practice for children and adolscents. American Psychologist, 65, 85-97. Kazdin, A. E. (2005). Parent management training; treatment for oppositional, aggressive, and antisocial behavior. New York: Oxford University Press. Kazdin, A. E. (2008). Evidence-based treatment and practice, New Opportunities to bridge clinical research and practice, enhance knowledge base, and improve patient care. American Psychologist, 63, 146-159. Kazdin. A. E., Bass, D., Ayers, W. A., & Rogers, A. (1990). Empirical and clinical focus of child and adolescent psychotherapy research. Journal of Consulting and Clinical Psychology, 58, 729-740. 141 Keenan, D., & Shaw, D. S. (2003). Starting at the beginning: Exploring the etiology of antisocial behavior in the first years of life. Causes of conduct disorder and juvenile delinquency. In B. B. Lahey, T. E. Moffitt, & A. Caspi (Eds.) Causes of conduct disorder and juvenile delinquency. (pp. 153-181). New York: Guilford Press. Kehle, T. J ., & Bray, M. A. (2007). On such a full sea and we still are not afloat: An introduction to statistical reform in school psychology. Psychology in the Schools, 44, 415. Kehle, T. J ., Bray, M. A., Chafouleas, S. M., & Kawano, T. (2007). Lack of statistical significance. Psychology in the Schools, 44, 417-422. Kendziora, K. T. (2004). Early intervention for emotional and behavioral disorders. In R. B. Rutherford, M. M. Quinn & S. R. Mathur (Eds.) Handbook of research in emotional and behavioral disorders. (pp. 327-351) New York: Guilford. Kogan, K. L. & Gordon, B. M. (1975). A mother-instruction program: Documenting change in mother-child interactions. Child Psychiatry and Human Development, 5, 189-200. Kokko, K., & Pulkkinen, Lea. (2000). Aggression in childhood and long-term unemployment in adulthood: A cycle of maladaptation and some protective factors. Developmental Psychology, 36, 463-472. Kratochwill, T. R., Elliot, S. N., Loitz, P. A., Sladeczek, I. & Carlson, J. S. (2003). Conjoint consultation using self-administered manual and videotape parent-teacher training: Effects on children’s behavioral difficulties. School Psychology Quarterly, 18, 269-302. Kratochwill, T. R., & Stoiber, K. C. (2002). Evidence-based interventions in school psychology: Conceptual foundations of the procedural and coding manual of Division 16 and the Society for the Study of School Psychology Task Force. School Psychology Quarterly, 17, 341-389. Kratochwill, T. R. & Shemoff, E. S. (2003). Evidence-based practice: Promoting evidence-based interventions in school psychology. School Psychology Quarterly, 18, 389-408. Kratochwill, T. R. & Shemoff, E. S. (2004). Evidence-based practice: Promoting evidence-based interventions in school psychology. School Psychology Review, 33 , 34-48. Kumpfer, K. L. & Alvarado, R. (2003). F amily-strengthening approaches for the prevention of youth problem behaviors. American Psychologist, 58(6/7), 457-465. La Greca, A. M., Silverman, W. K., & Lochman, J. E. (2009). Moving beyond efficacy and effectiveness in child and adolescent intervention research. Journal of Consulting and Clinical Psychology, 77, 373-382. 142 Lahey, B., Miller, T. 1., Gordon, R. A., & Riley, A. W. (1999). Developmental epidemiology of the disruptive behavior disorders. In H. C. Quay & A. E. Hogan (Eds.) Handbook of disruptive behavior disorders. (pp. 23-48) New York: Kluwer Academic/Plenum Publishers. Lahey, B. B. Moffitt, T. E., & Caspi, A. (Eds.). (2003). Causes of conduct disorder and juvenile delinquency. New York: Guilford. Lahey, B., & Waldman, I. D. (2003). A developmental propensity model of the origins of conduct problems during childhood and adolescence. In B. B. Lahey, T. E. Moffitt & A. Caspi (Eds.) Causes of conduct disorder and juvenile delinquency. (pp. 76- 117) New York: Guilford. *Larsson, B., Fossum, S., Clifford, G., Drugli, M. B., Handegard, D. H., & Morch, W. T. (2009). Treatment of oppositional defiant and conduct problems in young Norwegian children. European Child and Adolescent Psychiatry, 18, 42-52. *Letarte, M., Normandeau, S., & Allard, J. (2010). Effectiveness of a parent training program “Incredible Years” in a child protection service. Child Abuse & Neglect, 34, 253-261. Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment; Confirmation from meta-analysis. American Psychologist, 48, 1181-1209. Lipsey, M. W., & Wilson, D. B. (2001a). Practical meta-analysis. Thousand Oaks, CA: Sage. Lipsey, M. W., & Wilson, D. B. (2001b). The way in which studies have “personality” and why it is important to meta-analysis. Evaluation & the Health Professions, 24, 236-254. Lochman, J. E., Boxmeyer, C., Powell, N., Qu, L., Wells, K., & Windle, M. (2009). Dissemination of the C0ping Power Program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology, 7 7, 397—409. Loeber, R., & Farrington, D. P. (2000). Child delinquents: development, intervention, and service needs. Thousand Oaks, CA: Sage Publications, Inc. Loeber, R., F arrington, D. P., & Petechuk, D. (2003). Child Delinquency: Intervention and Prevention. Washington, DC: US. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention. Lucas, S. M., & Cutspec, P. A. (2005). The role and process of literature searching in the preparation of a research synthesis. Centerscope: Evidence-Based Approaches to Early Childhood Development, 4, 1-26. 143 Lundahl, B., Risser, H. J ., & Lovejoy, M. C. (2005). A meta-analysis of parent training: Moderators and follow-up effects. Clinical Psychology Review, 26, 86—104. *Manby, M. (2005). Evaluation of the impact of the Webster-Stratton parent-child videotape series on participants in a midlands town in 2001-2002. Children & Society, 19, 316-328. Maughan, D. R., Christiansen, E., Jenson, W. R., Olympia, D., & Clark, E. (2005). Behavioral parent training as a treatment for externalizing behaviors and disruptive behavior disorders: A meta-analysis. School Psychology Review 34, 267-286. McCabe, K. M., Hough, R., Wood, P. A., & Yeh, M. (2001). Childhood and adolescent onset conduct disorder: A test of the developmental taxonomy. Journal of Abnormal Child Psychology, 29, 305-316. McCart, M. R., Priester, P. E., Davies, W. H., & Azen, R. (2006). Differential effectiveness of behavioral parent-training and cognitive-behavioral therapy for antisocial youth” A meta-analysis. Journal of Abnormal Child Psychology, 34, 527- 543. *McIntyre, L. L. (2008a). Adapting Webster-Stratton’s Incredible Years parent training for children with developmental delay: Findings from a treatment group only study. Journal of Intellectual Disability Research, 52, 1176-1192. *McIntyre, L. L. (2008b). Parent training for young children with developmental disabilities: Randomized controlled trial. American Journal of Mental Retardation, 113, 356-368. McFarlane, J. M., Groff, J. Y., O’Brien, J. A., & Watson, K. (2003). Behaviors of children who are exposed and not exposed to intimate partner violence: An analysis of 330 Black, White and Hispanic children. Pediatrics, 112, 202-207. McHugh, R.K., & Barlow, DH. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73-84. McMahon, R. J ., Wells, K. C., & Kotler, J. S. (2006). Conduct problems. In E. J. Mash & R. A. Barkley (Eds.) Treatment of childhood disorders (3rd ed.). (pp. 137-268) New York: The Guilford Press. McNeil, T. (2006). Evidence-based practice in an age of relativism: Toward a model for practice. Social Work, 51, 147—156. Michael, K. D., Huelsman, T. J., & Crowley, S. L. (2005). Interventions for child and adolescent depression: Do professional therapists produce better results? Journal of child and Family Studies, 14, 223-236. 144 Mihalic, S., Irwin, K., Elliot, D., Fagan, A., & Hansen, D. (July, 2001). Blueprints for violence prevention. Juvenile Justice Bulletin. Washington, DC: United States Department of Justice, Office of Juvenile Justice and Delinquency Prevention. Retrieved from the World Wide Web on July 1, 2007 from: http://www.nc_irs.gov/pdffiles1/oiidp/l 87079.pdf Mockford, C., & Barlow, J. (2004). Parenting programmes: Some unintended consequences. Primary Health Care Research & Development, 5, 219 - 227. Moffitt, TE. (2005). The new look of behavioral genetics in developmental psychopathology: Gene-environment interplay in antisocial behaviors. Psychological Bulletin,131, 533-554. Moffitt, T. E., Caspi, A., Harrington, H., & Milne, B. J. (2002). Males on the life-course- persistent and adolescence-limited antisocial pathways: Follow-up at age 26 years. Development and Psychopathology, 14, 179—207. Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent group designs. Psychological Methods, 7, 105-125. Mullen, E. J ., & Bacon, W. F. (2003). Practitioner adoption and implementation of evidence-based effective treatments and issues of quality control. In A. Rosen & E. K. Proctor (Eds.), Developing practice guidelines for social work intervention: Issues, methods, and a research agenda. New York City: Columbia University Press. Muntz, R., Hutchings, J ., & Edwards, R. T. (2004). Economic evaluation of treatments for children with severe behavioural problems. Journal of Mental Health Policy and Economics, 7, 177—189. National Association of School Psychologists (2006). Blueprint for training and practice (3rd ed.). Silver Springs, MD: Author. *Nilsen, W. (2007). Fostering futures: A preventive intervention program for school-age children in foster care. Clinical Child Psychology and Psychiatry, 12, 45-63. Nock, M. K., Kazdin, A. 13., Hiripi, E., & Kessler, R. C. (2006). Prevalence, subtypes, and correlates of DSM-IV conduct disorder in the National Comorbidity Survey Replication. Psychological Medicine, 3 6, 699—710. Nowak, C., & Heinrichs, N. (2008). A comprehensive meta-analysis of Triple-P Positive Parenting Program using Hierarchical Linear Modeling: Effectiveness and Moderating variables. Clinical Child and Family Psychology Review, 11, 114-144. 145 .I Nutley, 8., Walter, L, & Davies, H. T. O. (2009). Promoting evidence-based practice: Models and mechanisms from cross-sector review. Research on Social Work Practice, 19, 552—559. Ogg, J. A., & Carlson, J. S. (2009). The self-administered Incredible Years Parent Training Program: Perceived effectiveness, acceptability, and integrity with children exhibiting symptoms of Attention-Deficit/Hyperactivity Disorder. Journal of Evidence-Based Practices for Schools, 10, 143-166. Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286. Ollendick, T. H. (2005). Evidence-based parent and family interventions in school psychology: A commentary. School Psychology Quarterly, 20, 512-517. Ollendick, T. H., & Davis, T. E., III. (2004). Empirically supported treatments for children and adolescents: Where to from here? Clinical Psychology: Science and Practice, 11, 289—294. Ollendick, T. H., & King, N. J. (2000). Empirically supported treatments for children and adolescents. In P. C. Kendall (Ed.), Child and adolescent therapy: Cognitive behavioral procedures (2nd ed.). (pp. 386—425). New York: Guilford Press. Olweus, D., Limber, S., & Mihakic, S. (1999). Bullying prevention program. In D.S. Elliott (Ed.) Blueprints for violence prevention: Book Nine. Boulder, CO: Center for the Study and Prevention of Violence. Ortego, C., & Botella, J. (2010). The hard but necessary task of gathering order-one effect-size indices in meta-analysis. Psicologica, 31, 291-315. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157-159. Pardinini, D. A., Lochman, J. E., & Powell, N. (2007). The development of callous- unemotional traits and antisocial behavior in children: Are there shared and/or unique predictors? Journal of Clinical Child and Adolescent Psychology, 3 6, 319- 333. Patterson, G. R. (2002). The early development of coercive family process. In J. B. Reid, G. R. Patterson, & J. Snyder (Eds.) Antisocial behavior in children and adolescents: A developmental analysis and model for intervention. (pp. 25-44) Washington, DC: American Psychological Association. Patterson, G. R., & Yoerger, K. (2002). A developmental model for early- and late- onset delinquency. . In J. B. Reid, G. R. Patterson, & J. Snyder (Eds.) Antisocial behavior 146 in children and adolescents: A developmental analysis and model for intervention. (pp. 147-172) Washington, DC: American Psychological Association. *Patterson, J ., Barlow, J ., Mockford, C., Klimes, I., Pyper, C., & Stewart-Brown, S. (2002). Improving mental health through parenting programmes: Block randomised controlled trial. Archives of Disease in Childhood, 871472—477 Popper, K. R. (1954). Degree of confirmation. The British Journal for the Philosophy of Science, 18, 143-149. Quay, H. C. (1999). Classification of the disruptive behavior disorders. In H. C. Quay & A. E. Hogan (Eds.) Handbook of Disruptive Behavior Disorders. (pp. 3-21) New York: Kluwer Academic/Plenum Publishers. Quintana, S. M., & Minami, T. (2006). Guidelines for meta-analyses of counseling psychology research. The Counseling Psychologist, 34, 839-877. Radder, H. (2003). The philosophy of scientific experimentation. Pittsburg, PA: University of Pittsburg Press. Reid, J. B. (1993). Prevention of conduct disorder before and after school entry: Relating interventions to development findings. Journal of Development and Psychopathology, 5, 243-262. Reid, J. B., Patterson, G. R., & Snyder, J. (2002). Antisocial behavior in children and adolescents: A developmental analysis and model for intervention. Washington, DC: American Psychological Association. Reid, M. J ., Webster-Stratton, C., & Baydar, N. (2004). Halting the development of conduct problems in head start children: The effects of parent training. Journal of Clinical Child and Adolescent Psychology, 33, 279-291. Rhule, D. M. (2005). Take care to do no harm: Harmful interventions for youth problem behavior. Professional Psychology: Research and Practice, 3 6, 618-625. Robinson, E. A., & Eyberg, S. M. (1981). The dyadic parent-child interaction coding system: Standardization and validation. Journal of Consulting and Clinical Psychology, 49, 245-250. Robinson, E. A., Eyberg, S. M., & Ross, A. W. (1980). The standardization of an inventory of child conduct problem behaviors. Journal of Clinical Child psychology, 9, 22-28. *Rogers, R. D. (2007). Evaluating the eflectiveness of the Incredible Years Parenting Education and Support Program (Unpublished doctoral dissertation). James Madison University, Harrisonburg, VA. 147 Rosenfield, S. (2000). Crafting usable knowledge. American Psychologist, 1347-1355 Rosenthal, R. (1979). The “file-drawer problem” and tolerance for null results. Psychological Bulletin, 63 8-64 1. Rosenthal, R. (1991). Meta-A nalytic procedures for social research. Newbury Park, CA: Sage. Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59- 82. Rutter, M. (2005). Environmentally mediated risks for psychopathology: Research strategies and findings. Journal of the American Academy of Child and Adolescent Psychiatry, 44, 3-18. Sanabria, F., & Killeen, P. R. (2007). Better statistics for better decisions: Rejecting null- hypotheses statistical tests in favor of replication statistics. Psychology in the Schools, 44, 471-481. Sanders, M. R., Markie-Dadds, C., Tully, L. A., & Bor, W. (2000). The Triple-P Positive Parenting Program: A comparison of enhanced, standard, and self-directed behavioral family intervention for parents of children with early onset conduct problems. Journal of Consulting and Clinical Psychology, 68, 624-640. Satcher, D. (2001). Youth violence: A report of the Surgeon General. Washington, DC: US. Public Health Service, US. Department of Health and Human Services. Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90-100. Schoenwald, S. J., Sheidow, A. J ., & Chapman, J. E. (2009). Clinical supervision in treatment transport: Effects on adherence and outcomes. Journal of Consulting and Clinical Psychology, 7 7, 410-421. *Scott, S., O’Connor, T., & Futh, A. (2006). What makes parenting programmes work in disadvantaged areas? The PALS trial. Downloaded from: http://ssiacvmruorg.uk/media/doc/3/i/What_makes_parenting_pr%s_effectivemdf *Scott, S., Spender, Q., Doolan, M., Jacobs, B. & Aspland, H. (2001). Multicentre controlled trial of parenting groups for childhood antisocial behaviour in clinical practice. British Medical Journal, 323, 1-6. *Scott, S., Sylva, K., Doolan, M., Price, J., Jacobs, B., Crook, C., & Landau, S. (2010). Randomised controlled trial of parent groups for child antisocial behavior targeting multiples risk factors: The SPOKES project. Journal of Child Psychology and Psychiatry, 51, 48-57. 148 Serketich, W. J ., & Dumas, J. E. (1996). The effectiveness of behavioral parent training to modify antisocial behavior in children: A meta-analysis. Behavior Therapy, 27 , 171-186. Shadish, W. R., & Sweeney, R. B. (1991). Mediators and moderators in meta-analysis: There’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59, 883-893. Shaw, D., Lacourse, E., & Nagin, D. S. (2005). Developmental trajectories of conduct problems and hyperactivity from ages 2 to 10. Journal of Child Psychology and Psychiatry,45, 1—12. Sheridan, S. (2005). Commentary on evidence-based parent and family interventions: Will what we know now influence what we do in the fiiture? School Psychology Quarterly, 20, 518-524. Sheridan, 3., & Kratochwill, T. (2007). Conjoint behavioral consultation: promoting family-school connections and interventions (2nd ed.) New York: Springer — Verlag. Shirk, S. & McMakin, D. (2008). Client, therapist, and treatment characteristics in EBT’s for children and adolescents. In Handbook of evidence-based therapies for children and adolescents (pp. 471-484). New York: Springer. Silverman, W. K., Kurtines, W. M., & Hoagwood, K. (2004). Research progress on effectiveness, transportability, and dissemination of empirically supported treatments: Integrating theory and research. Clinical Psychology: Science and Practice, 11, 295—299. Snyder, J ., & Stoolmiller, M. (2002). Reinforcement and coercive mechanisms in the development of antisocial behavior: Peer relationships. In J. Reid, G. Patterson & J. Snyder (Eds.) Antisocial Behavior in Children and Adolescents: A Developmental Analysis and Model for Intervention. (pp. 65-100). Washington, DC: American Psychological Association. Sougstad, J. R., Oka, E. R., Carlson, J. S., & Tomac, R. (2008, August). Meta-Analysis of the Incredible Years Parent Training Intervention Program: Synthesis of primary and replication research across the frontier from efficacy to effectiveness. Poster Presentation to the 116th Annual Convention of the American Psychological Association; Boston, MA. Southam-Gerow, M. A., Marder, A. M., & Austin, A. A. (2008). Transportability and dissemination of evidence-based manualized treatments in clinical settings. In R. G. Steele, T. D. Elkin & M. C. Roberts (Eds.). Handbook of evidence-based therapies for children and adolescents (pp. 447-469). New York: Springer Southam-Gerow, M. A., Ringeisen, H. L., & Sherrill, J. T. (2006). Integrating interventions and services research: Progress and prospects. Clinical Psychology.- Science and Practice, 13, 1-8. 149 Spaccarelli, S., Cotler, S., & Penman, D. (1992). Problem-solving skills training as a supplement to behavioral parent training. Cognitive Therapy and Research, I 6, 1-18. Spirito, A., Abebe, K. Z., Iyengar, S., Brent, D., Vitiello, B., Clarke, G.,. . . Keller, M. (2009). Sources of site differences in the efficacy of a multisite clinical trial: The treatment of SSRI-resistant depression in adolescents. Journal of Consulting and Clinical Psychology, 7 7, 439—450. Sprague, J ., & Walker, H. (2000). Early identification and interventions for youth with antisocial and violent behavior. Exceptional Children, 66, 367-379. Stahmer, A. C., & Aarons, G. A. (2009). Attitudes toward adoption of evidence-based practices: a comparison of Autism early intervention teachers and children’s mental health providers. Psychological Services, 6, 223-234. Steele, R. G. , Elkin T. D., & Roberts M. C. (Eds.). (2008). Handbook of evidence-based therapies for children and adolescents. New York: Springer. Stern, S. B., Alaggia, R., Watson, K., & Morton, T. R. (2008). Implementing an evidence-based parenting program with adherence in the real world of community practice. Research on Social Work, 18, 543-554. Stoiber, K. C. (2002). Revisiting efforts on constructing a knowledge base of evidence- based intervention within school psychology. School Psychology Quarterly, 1 7, 533- 546. Swaminathan, H., & Rogers, H. J. (2007). Statistical reform in school psychology research: A synthesis. Psychology in the Schools, 44, 543-549. Task Force on Evidence Based Interventions in School Psychology. (2003). Procedural and coding manual for the identification of evidence-based interventions. Retrieved July 1, 2007 from Task Force on Evidence-Based Interventions in School Psychology Web site: http://wwwspebi.org/documents/_workingfiles/EBImanual1.pdf Taylor, T. K., Schmidt, F., Pepler, D., & Hodgins, C. (1998). A comparison of eclectic treatment with Webster-Stratton’s parents and children series in a children’s mental health center: A randomized controlled trial. Behavior Therapy, 29, 221-240. Thomas, R., & Zimmer—Gembeck, M. J. (2007). Behavioral outcomes of Parent-Child Interaction Therapy and Triple-P Positive Parenting Program: A review and Meta- Analysis. Journal of Abnormal Child Psychology, 35, 475-495. Trachtman, G. M. (1981). On such a full sea. School Psychology Review, 10, 138-181. Tremblay, R. E., Nagin, D. S., Seguin, J. R., Zoccolillo, M, Zelazo, Boivin, M., Perusse, D., & Japel, C. (2003). Physical Aggression During Early Childhood: Trajectories and Predictors. Pediatrics, 114, 43-50. 150 van de Wiel, N., Matthys, W., Cohen-Kettenis, P. C., & van Engeland, H. (2002). Effective treatments of school-aged conduct disordered children: recommendations for changing clinical and research practices. European Child & Adolescent Psychiatry, 11, 79-84. Volker, M. A. (2006). Reporting effect size estimates in school psychology research. Psychology in Schools, 43, 653-672. Waas, G. A. (2002). Identifying evidence-based interventions in school psychology: Building a bridge or jousting with windmills? School Psychology Quarterly, 17, 508- 517. Walcott, C. M., Carlson, J. S., & Beamon, H. L. (2009). Effectiveness of a self- administered training program for parents of children with ADHD. School Psychology Forum: Research in Practice, 3, 43-61. Walker, H., Ramsey, E., & Gresham, F. M. (2004). Antisocial behavior in school: Evidence-based practices, Second Edition. Belmont, CA: Wadsworth. Watkins, M. (2009). Errors in diagnostic decision making and clinical judgment. In T. B. Gutkin & C. R. Reynolds (Eds.) Handbook of School Psychology (4th ed.). Hoboken, NJ: Wiley & Sons. Watson, M. W., Fischer, K. W., Andreas, J. B., & Smith, K. W. (2004). Pathways to aggression in children and adolescents. Harvard Educational Review, 74, 404—430. #Webster—Stratton, C. (1982a). The long-term effects of a videotape modeling parent training program: Comparison of immediate and 1 year follow-up results. Behavior Therapy 13, 702-714. #Webster-Stratton, C. (1982b). Teaching mothers through videotape modeling to change their children’s behaviors. Journal of Pediatric Psychology 7, 279-294. #Webster-Stratton, C. (1984). A randomized trial of two parent training programs for families with conduct-disordered children. Journal of Consulting and Clinical Psychology 52, 666-678. Webster-Stratton, C. (1989). Dyadic parent-child interaction coding system — revised. Unpublished manuscript. University of Washington: Seattle. #Webster-Stratton, C. (1990a). Enhancing the effectiveness of self-administered videotape parent training for families with conduct-problem children. Journal of Abnormal Child Psychology 18, 479-492. #Webster-Stratton, C. (1992). Individually administered videotape parent training: “Who benefits?” Cognitive Therapy and Research, 16, 31-51. 151 Webster-Stratton, C. (1994). Advancing videotape parent training: A comparison study. Journal of Consulting and Clinical Psychology, 62, 583-593. Webster-Stratton, C. (1996). Early intervention with videotape modeling: Programs for families of children with oppositional defiant disorder or conduct disorder. In M. J. Guralnick (Ed.). The Effectiveness of early intervention: second generation research. (pp. 435-4 73). Baltimore, MD: Paul Brookes Publishing. #Webster-Stratton, C. (1998). Preventing conduct problems in Head Start children: Strengthening parent competencies. Journal of Consulting and Clinical Psychology 66, 715-730. Webster-Stratton, C. (2000). The incredible years training series. Juvenile Justice Bulletin. Washington, DC: United States Department of Justice, Office of Juvenile Justice and Delinquency Programs. Webster-Stratton, C. (2001). The parents and children series, BASIC program leader ’s guide. Seattle, WA: Author. Webster-Stratton, C. (2007). Tailoring the incredible years parent programs according to children’s developmental needs and family risk factors. In J. M. Briemeister & C. E. Schaefer (Eds.). Handbook of parent training, helping parents prevent and solve problem behaviors (3rd ed.). (pp. 305-344). Hoboken, NJ: Wiley & Sons. #Webster-Stratton, C., & Hammond, M. (1997). Treating children with early-onset conduct problems: A comparison of child and parent training interventions. Journal of Consulting and Clinical Psychology 65, 93-109. Webster-Stratton, C., & Hancock, L. (1998). Parent training of young children with conduct problems: Content, methods, and therapeutic processes. In C. E. Schaefer & J. M. Briesmeister (Eds.). Handbook of parent training, New York, NY: John Wiley & Sons. pp. 98 — 152. Webster-Stratton, C., & Herman, K. C. (2010). Disseminating Incredible Years series early—intervention programs: Integrating and sustaining services between school and home. Psychology in the Schools, 47, 36-54. #Webster-Stratton, C., Kolpacoff, M., and Hollinsworth, T. (1988). Self-administered videotape therapy for families with conduct-problem children: Comparison with two cost-effective treatments and a control group. Journal of Consulting and Clinical Psychology, 56, 558-566. Webster-Stratton, C., & Lindsay, D.W. (1999). Social competence and conduct problems in young children: Issues in assessment. Journal of Clinical Child Psychology, 28, 25-43. Webster-Stratton, C., Mihalic, S., F agan, A., Arnold, D., Taylor, T., & Tingley, C. (2001). The Incredible Years: Parent, Teacher And Child Training Series: 152 Blueprints for violence prevention, book eleven. blueprints for violence prevention series. Boulder, CO: Center for the Study and Prevention of Violence, Institute of Behavioral Science, University of Colorado. Webster-Stratton, C., & Reid, M. J. (2010). The Incredible Years Parents, Teachers, and Children Training Series, A multifaceted treatment approach for young children with conduct problems. In J. Weisz & A. Kazdin (Eds.). Evidence-based psychotherapies for children and adolescents (2nd ed.). (pp. 224-240). New York: Guilford Publishing. #Webster—Stratton, C., Reid, M. J. and Hammond, M. (2001). Preventing conduct problems, promoting social competence: A parent and teacher training partnership in head start. Journal of Clinical Child Psychology, 30, 283-302. #Webster—Stratton, C., Reid, M. J ., & Hammond, M. (2004). Treating children with early- onset conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child and Adolescent Psychology, 33, 105-124. #Webster—Stratton, C. & Shoecraft, K. (2009, unpublished). Delivering of Incredible Years Parenting Program. Washington State Child Welfare Report. Retrieved from: http://incredib[gears.com/Librarv/items/washington-state-child-welfare-report_6-25- 09%20.pdf Webster-Stratton, C., & Taylor, T. (2001). Nipping early risk factors in the bud: Preventing substance abuse, delinquency, and violence in adolescence through interventions targeted at young children (0 to 8 years). Prevention Science, 2, 165- 192. Weersing, V. R., & Weisz, J. R. (2002). Community clinic treatment of depressed youth: Benchmarking usual care against CBT clinical trials. Journal of Consulting and Clinical Psychology, 70, 299—3 10. Weiss, B., Catron, T., Harris, V. & Phung, T. M. (1999). The effectiveness of traditional child psychotherapy. Journal of Consulting and Clinical Psychology, 67, 82-94. Weisz, J. R. (2004). Psychotherapy for children and adolescents: Evidence-based treatments and case examples. Cambridge, UK: Cambridge University Press. Weisz, J. R., Donenberg, G. R. Han, S. S., & Weis, B. (1995). Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. Journal of Consulting and Clinical Psychology, 63, 688-701. Weisz, J. R., Hawley, K. M., & Doss, A. J. (2004). Empirically tested psychotherapies for youth internalizing and externalizing problems and disorders. Child and Adolescent Psychiatric Clinics of North America, 13, 729-815. 153 Weisz, J. R., Huey, S. M., & Weersing, V. R. (1998). Psychotherapy outcome research with children and adolescents: The state of the art. In T. H. Ollendick & R. J. Prinz (Eds.), Advances in clinical child psychology (Vol. 20). (pp. 49—92). New York: Plenum Press. Weisz, J. R., & Jensen, A. L. (2001). Child and adolescent psychotherapy in research and practice contexts: Review of the evidence and suggestions for improving the field. European Child & Adolescent Psychiatry, 10, 12-18. Weisz, J. R. Jensen-Doss, A. & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist, 61 , 671-689. Weisz, J. R. Jensen-Doss, A. & Hawley, K. M. (2005). Youth psychotherapy outcome research: A review and critique of the evidence base. Annual Review of Psychology, 56, 337-363. Weisz, J. R., Weiss, B., Alicke, M. D., & Klotz, M. L. (1987). Effectiveness of psychotherapy with children and adolescents: A meta-analysis for clinicians. Journal of Consulting and Clinical Psychology, 55, 542-549. Weisz, J. R., Weis, B., & Donenberg, G. R. (1992). The lab versus the clinic; Effects of child and adolescent psychotherapy. American Psychologist, 4 7, 1578-1585. Weisz, J. R., Weiss, B., Han, S. S., Granger, D. A., & Morton, T. (1995). Effects of psychotherapy with children and adolescents revisited: A meta-analysis of treatment outcome studies. Psychological Bulletin, 11 7, 450—468. White, J. L. & Kratochwill, T. R. (2005). Practice guidelines in school psychology: Issues and directions for evidence-based interventions in practice and training. Journal of School Psychology, 43, 99-115. Wilkinson, L. & the American Psychological Associatiation Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 549-604. Wilson, D. B., & Lipsey, M. W. (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods, 6, 413-429. Wilson, S. J., Lipsey, M. W. & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71, 136-149. Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Newbury Park, CA: Sage. 154 ”11111111111111![113111211 11111111 ll“