. 1 1:36......,..... x... V.‘Av€<(...d11(\.\wkgr..
.11. .1312... 1.13 ~3§
{3.14.33}. “an . 11th ......
b.3332 vflliivsiqi .Y
2.34.0.5. 223...}... 5:
\ I . }.I‘~}’ $v1§1l¢11| gavlfl
.13....“ .42:..5.)I-.1..i§xl.l...y3§

  

                          

. . _ .... ... ... . I a.,.\.\u......y.rx.n. .\.. . hi 1.47:1 \«
...: . .. .... .. 3.3.3 .3. . ..3..... 13.33....5. .
“.3” sémy .... .... ..M. .3..»... 3...... . 3...... ...... .. $39.3...
...... yawn. .. .... ... .. ..fimuim. . .... . . .

.33.“... ....m. ...... .3. .35333333333 ..3..
... ...... ...... 33.33.... .

.... .... 3.3.- .... . .... .............zn...wr.......§ .
. Inna—Von. . .. «5.31.. . a .. tofi .h.....ir)~¢fiyxl ...)..u. .
....“ «.3 W . ...3 . .33... .... .u. .. 3 4mm. gamma .RfiaimhghybL.

2) 2.12.: [fix-534.532 [.2

.‘\y..,.fl..3.1.h...3s\).. 1.1.2.03. 1...... ..n.
3...:13... --Iy] .
3132:2933:

  
    

  
    

 

  

  

v. . 1 5.51.5.1): 1...... ... u
.... . ...q . .... x. ......m»... ....n» "5...... ... 1.. wave? nun. ..mmnqblmwwwm...
sully... 3...). ....Q. LE." ...... .52.... . h». . a uv ..3..... 2.1.13. . 134.95}: ..
...J..-.....L...«..a .. «.... a. .. 3M... . «3...... . . .1333. .. .....ppanah. ..-. .135:
G. . ma? ififimv .. h». . Lynn 3 h. 3. tr (yr! lb.- vx 3mxtm1hnrfl. .. 9.“qu I ..3.....«133715asa
nu. . .. _
32%....”me .. . ...s . ...
. x: .

  

«S... .... .13

rJuVW..WH.Hh.3wNW.MmWW . .. .
Va > n .

     

r

 

...) .

v . ...! ... .
. .... 3.... m..v...§..;mo.....,v ...a....n«.....vs... .13).... ......fl. . _. . ..
...-v.3. .... ...... .. umnmoh3xurjsxw. 32. x .....Ia:;.......z\$1\§¢vlsws . . . . .
.. ... z . . 23.3.3. ._ .... .....33....i.-.......3..3 .
Z r... . ... an... . 1 ”1.5.3... . hwfinrhfifi5whn...g‘1 . x
. .

1 y . 1
ha a .. . .2" .... 3 x. i3!
. {baabmfl ..3..“.

            
    

  

7. .
. .3312: 7).... 5.1.2.3.}... (.3 31!:.J\¢\|n. lu1lyisi€uad ...
51.13311..- .4... ..3..!” 11.1)»)...3‘! ....lzc‘lly

 

. .. . -
... .
. -95....

..

. _ \t x . .. ({3113... i.vl...unni~l..iil1..x
.. 1. . .. ... . ...... .. .... . 338$- .asnwhflnnfihu..3t?.?.33f1§3.13.3153?

. . . .... .5.» . 3. 1.3.3.... 213... .i .9... .1193]... ...
.. .... anrsxzmufi .... .. .. . . 1.x}... . .. miluza. Wlfa..z).323 :3h..i(zthfi.lz‘isyyi>hxytle‘ahflt
. 11.. ...... .. .....a y. can". ...u... 3.. ..-“ ..lw

 

.13. . a... 1.5. 1.4.2.1....
.. a... 338.13.. y “0.19%???3r3shiialififi. 3.5.5."... runs.
1 4’1 ‘ in i‘. .|\-’\\. 7".K‘s’it‘Vl

.. . ...? ... ) 311-23.}...

. V \ ’3 \‘i‘ \ ...... 809‘ ‘71.. x...

. s s ......

$.13: 11

 

.....(«i‘x .. .
1:17. . ....4
...h\1I$}.4.vvi\lil

   

  
       

         

.- . 7 . ifsskvxxf
..3..... ..3... 2...! v w 3% 5.985.331)...
lung...» .35.}... n. ... .55.. .... ...... kn»... WWW
. .. . . . q .
. . . w . ..(.
. ~ I,

   

    

 

   

                                    

           

  
     

. .
... v~yI Q: 1 . 11 .s «v 1 i . Isa . iizykh§x1~lfiis ‘1
“influbuawaght .sltfi ....V . {$fl#¢a$)\b$ isl- ighu.‘ ifixgg.5flrb§}!flflsli .\
fin“. ..wn o ... JUL. . a... ...W.‘ .J gafilsy‘. . é.§n.§§l .. 1n..9¢:1.!.l..\(..).\)i.)\i:l 1)....‘Iui
.. 5.4.531: . E .. .. . .. um . 3:33. . 5‘2.) .. NE. ...
..fi.ya...,.f-- . fi 4.. .. . . . if: . .-.
{.53.}. in... . . ..M .. o . 1. ... .

 

_. ... Wigwmmx a“??? .. 2 . h o. -. J
. 3..... .. “WE. . um... mammfidklimiinzzi $31“.
..3..»... w. .3..»... _. ..Jvasfiumefiw . £53.32
...... $433.3... ._ , .. ...nuvfih....a. .33 ..
cm“ H. 5 ... y . . known . .v:t..$1u..) .1. . 4m.“

.wm . mm .515 .. s. .43.. 53.... .u.-. :fim.
.Aww%..w..§m ... a... .. an...” . @«mw. ..
.. .. . a . .... . :

.... .. . fizfl .. .3...“ .. 2.... ......L......Shu.....:..+.u...y vumfiantfifi
. 61...»: .. .. "...: “......IL 81
2...... ....mmdh. avgdrfihwem} . a. 39.3 ...L a... - 2

a l: 92.. ... A E.
1 r. it} ._ 5 .
. . a .2223; 3...... ... L
.....z a. .3323. . . . y
.. . 32...... .
..

. . . . . . . ... ii}... .313“... .
. 3...... ......3... 3.33.3.3... ...... ..3..”. 2.33.... ......
. . . . .... . 1.3..- .

)

     

      

  

    

     

       

     

         

3.21.. 1.54}. .

. 1! ilrirruflpgfl .11 it"
u 1
v .

          

 

      

:5 .‘&~
2...... .. . .-.
rt... .. .3.. w. ..
{Tin-Anziufiurh 737.51.“.WWI ..

. . In»... :mmqunééautiw ... ....wuwrfi . 1.. . .. T:
.. $1.13.. . ..fi... ...... .933. . . $5.... .. .1;
.151...) ...WJ.E...W.>..:¢...«I&.0? . . . . z.

. I II. F . o I . 1... $33.94. w\: ...: 3!. .... l
“1.. a!!! :leI airli..Dl-PVC .II. .

. .gnmfiyne!0i!hhwfidfi.3§1.ifin.
935.199 ....r1' ll ..vl.N.o 99 “I”!
fifch.3.¥uuvo:3r- whine.

. $17 I: ..
11.319.153.9(11 is}...
...-$2611.}... 34%;, . ...HA 1:.

1 it!!!) 3.. )1...
gain-37.53,: 51.1.31!!! ..
.81.. 3" )4. £33! ....Iiylizlifliuv

a... \ . 3:213. ..D.:I1O:\11\{h.v1.l(|ll. ....le-snful
fizlg x iéiutfiu. wit.
.... 143.11! 105.1%! ’Gi§\rsixyzo.yi
. 3i ‘ \i’I-l)‘ .J . ‘1')“ .

.. )X!v¥..\....§¢!. v
4-: ) 4)»... iv! _ ‘1 .2 .
. t . “I b 3.1.}..11: in”..l.V\§ (17‘1.l..|..1!zv:.l¢|
. 03:4 . . VV.\I..I. iii; ‘11.“),
. 3g 33% 11141 I

c
9.92): J.

v

.00.!

   

   

      

. z ....
t. . f.‘ \
. ...ufirflg cryiuufluhvt. ....m
. ...... ...... ...bfi...»€rvf+hm». 1nd.
. . 12:! M57 tub. (a
...»..gdfifiu... $1,...fiph. ...
.uuufiaiywmmwmmfl ”...th b nun, . «1
.......AA ‘11! fii'llfl‘b:.ihhflfl .1 . 5-0.1“... . 1“. 6|) . .
new... .1..1..+e1......!2. ......r... 39...}. 3 .2... .45.. July...
. “.233. 53‘... stiiomr. SR»? 3%. 11.35...)
..vt.-:...h.. L}... t... 3.552....H3isfii 38...»?! it v

n I! t I . . .....D‘Q. . V. . . . .
...f..nu...;......:T ......fimwutfinhgrt §P§._.Kt . $34.... .
x 1’...
k

        

Ii

   

f:

                                                         
                  

    

     

                 

ll‘sdlfi. ‘

A».
» 2.33.3153. 235.3.
E .
N.“ u . u . .9? n9 .0!»
N 5.51....T..1.19~b Qat. Enid. :1 5..
...uu.....~!....,zxw. .3.....3. ...i....Zt ..3.....3....... 4 1323...? .. , .. ..
{brunummuhgfzgulnfivré 5 .. .Ezkngmzmfinififi. .
3. .. .. h u. - - . .
....yr ..Iafi.....:....LI 3.13:3“... .2 ....

. z.
. kafiri. Tflfisfius.§£fifiu.

. 4%...
u. 71 V

39......»va ir.fl.uisl‘ .

. rafitél.!.nufl.

 

 

 

              

  

   

 

 

                 

 

. ......E’ v3.3.3.1... 1 .
.41.... V. ..v‘

3 1.:
. .. 3.2.1. “8.335.? 1.. LE
. < .A o ,L. (I :I J. C
. . . .. _ . . .241. .lgtnt3nnnhuflifitiflmnvnfltn.
.. . . .. $72.3!3-E5‘211y4ifiJifi

.. .. -.....lrrt at Bunk?
.. . ...yr9II y . 2.0.93.6... £3“. .

   

..I...
.. 1. .123... 3.3.5.; ..tarz.rlrill. . fzséuflgii

No.31. 13?...th .15....331h .lifiyflgtrtzkftlufiufz .. £3.31:

...:tfifi.-§tr§hd.xt s r .. 3.15.2... ft..1rvuatt.flx$...2}§|
(1.133.113.3332-..‘ .....- .... (31.512.15.33
£323.... ..3... ..tpln.iblit>eftt}g1§ .81.: . 3..

.. . .-.!....1¥Eftfi¢tvlni£u.i$ .... 3321...! g .....lfl
\*J.I.4..u.tfd.€al)‘|.lx.5¥.}ff (2.3.1 .. Ill!.§|..»...l..ttf. ...... .1)“.
0; Jv.£.lavtflt.§.io|.lltx ... ‘ttnll’coDVer9? {Quite} \

.. .. 1:333:52?! . 9.3.6 ill... 3 . .
gianflhxzés . Sailfisltuinrtlflitéi

.15..“ .lqu}’ [.nn?ynn$i..~hf.fl.:£! . ..

             

. . . s...u..:.¢....|u\ 1....
.Islt.£\..§.lut!.ls..:al .I.l..tv lb.-.
it..£.|k.u\~(‘ .17. f
{it‘ll-iitt... . . ...

 

? Ill};|l|lllfl||fllflllll1lllljflfllfll «will e

 
 
     

 
 

"LIBRAR Y
Moan 5W
1;.» u . o .

This is to certify that the
thesis entitled

THE CONSTRUCT VALIDITY OF A SELF—REPORT
INSTRUMENT IN MENTAL HEALTH
EVALUATION

presented by

Lorraine LaFerriere

has been accepted towards fulfillment
of the requirements for

Ph . D. Agree in Psychology_

/L/Mflwvwmjm

Major professor

Date August 7, 1979

0-7 639

   
  
  

OVERDUE FINES ARE 25¢ PER DAY
PER ITEM

Return to book drop to remove
this checkout from your record.

 

 

 

 

THE CONSTRUCT VALIDITY OF A SELF-REPORT
INSTRUMENT IN MENTAL HEALTH
EVALUATION

By

Lorraine LaFerriere

A DISSERTATION

Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY

Department of Psychology

1979

 

ABSTRACT

THE CONSTRUCT VALIDITY OF A SELF-REPORT
INSTRUMENT IN MENTAL HEALTH
EVALUATION
By

Lorraine LaFerriere

Research in mental health evaluation has evolved primarily
during the last twenty years, with attention focused on issues of
methodology and design (Campbell and Stanley, l963; Kenny, 1975) and
more recently on issues of measurement (Waskow and Parloff, l975;
Erickson, l975; Ellsworth, l975). An emerging consensus suggests
that evaluating mental health programs requires more than traditional
length-of—stay and rehospitalization measures; measures of how the
patient is functioning socially and psychologically are necessary
to determine whether patients truly benefit from treatment. Less
consensus appears around what measures should be used, and whether
they should be based on the client's own report of his functioning,
or a clinician's or significant other's report.

This study is an examination of the validity of a client's
self-report instrument in mental health evaluation. The study was
part of a larger effort to pilot test a set of procedures and instru-
ments in outcome evaluation, jointly sponsored by the Michigan

Department of Mental Health and the Urban Institute. In l977-l978,

 

 

Lorraine LaFerriere

the pilot was implemented in five community outpatient units and two
inpatient units. Several hundred outpatients and inpatients were
asked to complete pre- and post—treatment questionnaires about their
psychological and social functioning, and to name a ”significant
other" who would complete similar questionnaires. Clinicians rated
the clients' symptomatic disturbance at intake. A random sample of
several hundred community residents also completed the self-report
instrument.

A substantial portion of the eligible populations, especially
inpatients, did not complete questionnaires at intake and/or at
follow—up. Data loss analysis indicates the subjects completing
questionnaires were less disturbed and were of a higher socioeconomic
status than subjects not completing questionnaires. Thus, the
generalizability of the findings are limited to the healthier
portion of inpatients and outpatients.

The primary instrument in this study, the Brief Symptom
Inventory (BSI), was selected for its ease of administration, poten—
tial applicability for outpatients as well as inpatients, and the
well established psychometric properties of its parent instrument,
the SCL-90 (Derogatis, l977). In examining the construct validity
of the 851 for evaluation purposes, three criteria were established
--whether the scores (a) differentiated between inpatients, out-
patients and community residents; (b) were congruent with other
measures of symptom disturbance; and (c) were sensitive to pre- and

post-treatment changes in symptom disturbance.

 

Lorraine LaFerriere

Results of the study indicate the 831 has construct validity
for women, but not for men. Most of the validity criteria were
satisfied for female outpatients and inpatients. By contrast, male
inpatients did not report significantly more symptoms than male out-
patients, nor did either sample of men report a lessening of
symptoms after treatment. The correspondence between the 831 and
reports of symptoms by clinicians and significant others was greater
for outpatients than for inpatients, with the lowest correspondence
being for male inpatients.

Investigation of the validity of the 851 is confounded with
other possible factors, including possible inherent sex differences
in the experience and/or reporting of symptoms; methodological ques-
tions regarding measurement of change; and some evidence that
symptomatic disturbance may be more salient factors for women
seeking and benefitting from treatment than men. Further research
is necessary to examine these issues in more depth.

The evidence from this study suggests that while the 851 may
be a valid instrument for use with outpatients, it is not valid
for use with inpatients. The construct validity of this scale is
strongest for female outpatients, and weakest for male inpatients.
The combined results of a high percentage of data loss and the lack
of instrument validity for males render this self-report model
inappropriate for evaluating the effectiveness of mental health

services.

 

ACKNOWLEDGMENTS

I would like to acknowledge and thank the following people
who contributed to the successful completion of this study:

First, I would like to thank the staff members in the
Clinton-Eaton-Ingham Community Mental Health Center, the St.
Lawrence Inpatient Unit, and the Michigan Institute of Mental Health.
They gave many hours out of their busy schedules to gather data and
have clients complete questionnaires. The willingness of several
hundred clients from these agencies to complete questionnaires is
also appreciated.

I owe the greatest thanks to my staff on the project.

Cathy Comins worked with me from the beginning of the study, and
handled a wide range of tasks with great competence and sensitivity.
Judy Pfaff was responsible for most of the computer analyses in the
project, which she completed with much patience and skill. -Both
Judy's and Cathy's support, good humor and friendship throughout the
project will always be remembered.

This study was made possible by the Budget and Evaluation
Section of the Michigan Department of Mental Health. I sincerely
thank Ron Uken, Administrator of this section, for his support of
this project and mental health evaluation in general. I would also

like to thank Rick Spates, Susan Lawther, Sandy Herman and Stu

 

 

Halgren for the many hours they spent discussing the project and
assisting with the data analyses.

I would also like to thank Harry Hatry and Al Schainblatt
from the Urban Institute. Their original proposal for the study,
and their vigilance in implementing and critiquing the project were
important to its successful development.

My appreciation is also extended to the chairman of my
dissertation committee, Don Grummon, and its members, Bertram
Stofflemayr, Bob Zucker, and Bill Crano. I am grateful to Don
and the other members for their sustained interest in the study,
and for their helpful suggestions and support during its two years
of fruition. Although not members of my dissertation comnittee,

I would like to thank Bob Calsyn for being my first and best
teacher at MSU of methodological issues in evaluation research, and
Bob Ellsworth for being so supportive and informative about these
issues as well.

Finally, I would like to thank my husband, Barry Wright.

He gave me the greatest support possible by encouraging me during
the discouraging times, and by being willing to think through the
difficult parts of the study. His intellect, wit and sense of the

absurdity of it all was invaluable to me.

 

 

TABLE OF CONTENTS

Page

LIST OF TABLES ......................... vi
INTRODUCTION .......................... '1
LITERATURE REVIEW ....................... 6
The Brief Symptom Inventory ................ 7
History and Development ................ 7
Psychometric Properties ................ 9

Factor Structure of the SCL-9O ............. l2
Construct Validation .................. l3
Designing an Evaluation System .............. l6
Instrument Selection .................. l6

' Selecting an Informant ................. l9
Selecting a Follow-Up Period .............. 23

Data Loss ....................... 24
Methodology ...................... 25

Previous Outcome Studies ........ - ........ 29

METHOD ............................. 36
Description of the Study ................. 36
Subjects ........................ 37
Instruments ...................... 39

Data Collection .................... 42
Hypotheses ........................ 43
RESULTS ............................. 47
Subject Participation Rates. ............... 47
Representativeness of Intake Sample ............ 49
Representativeness of Follow—Up Sample .......... 53
Validity: Differentiation of Groups ........... 6O
Validity: Concurrence Between Measures .......... 64
Sensitivity of 851 to Change ............... 72

iv

 

 

Page

DISCUSSION ........................... 79
Generalizability of Results ................ 79

Tests of Construct Validity ................ 80
Alternative Theoretical Explanations ........... 84
Interpretation of Sex Differences ........... 84‘
Measuring Change .................... 87
Treatment Effectiveness ................ 89
Concurrence Among Informants ..... . . .’ ...... 89

The Evaluation System ................. 91
Recommendations for Further Research ........... 94
Conclusions ........................ 95
BIBLIOGRAPHY .......................... 97
APPENDICES ........................... l03

A PARS Scores of Outpatients and Inpatients at Intake . . lO4
B Client Satisfaction with Service ............ l06

C Consent Forms and Instruments ............. lO8

 

Table

TO.

IT.

12.

13.

LIST OF TABLES

Page

Demographic Characteristics of Community Residents

and Clients in Tri-County Study ............. 38
Number of Subjects With BSI, PARS and HPRS

Questionnaires ..................... 44
Demographic Characteristics of Subjects Completing and

Not Completing an Intake Questionnaire ......... 50
HPRS Scores of Subjects Completing and Not Completing

A BSI at Intake ..................... 52
Characteristics of Outpatients Returning and Not

Returning Follow-Up Questionnaires ........... 54
Characteristics of Inpatients Returning and Not

Returning Follow-Up Questionnaires ........... 56
Effects of Data Loss on Tri-County Study .......... 59
Brief Symptom Inventory Intake Scores for Inpatients

and Outpatients ..................... 6l
Brief Symptom Inventory Mean Intake Scores for

Inpatients, Outpatients and Community Residents ..... 62
Correlations Between Inpatient Scores on Symptomatic

Disturbance Scales on the B51 and PARS ......... 65
Correlations Between Outpatient Scores on Symptomatic

Disturbance Scales on the B51 and PARS ......... 67
Mean Scores and Correlations Between Corresponding

HPRS and 851 Scales ................... 7O
ANOVA of BSI Scores--At Intake, Three and Six Months. . . . 75

vi

 

INTRODUCTION

State and community mental health treatment programs are
serving an increasing number of people experiencing a variety of
emotional and behavioral problems. In order to provide these serv-
ices, state and federal agencies have been increasing the funds avail-
able to these treatment programs, with the expectation that effective
services will be provided to the treatment recipients. Many of these
agencies, however, are not able to measure the effectiveness of their
programs. Objective and quantifiable evidence of the impact of
treatment is not routinely available to persons administering,
funding and evaluating the programs.

A well-designed evaluation system can provide information for
a variety of decisions. At a minimum, the type and degree of disturb-
ances and problems characteristic of the population served can be
known. Hypotheses regarding the impact of certain types of treatment
for certain types of client problems can be generated. Successive
tests of these hypotheses will bring the evaluator closer to measuring
the "true" effectiveness of the treatment program. This information
can help identify which programs appear to be successful and not
successful in treating certain types of clients. When the evaluator
has identified those program variables influencing the less desirable

outcomes, the programs can be re-designed to improve an evaluation

system. If the instruments are not appropriate for the population
being assessed, if their psychometric properties are questionable, or
if completing them is too time consuming or confusing, the entire
evaluation system is in jeopardy. The selection of the instruments
is one of the most important steps in designing an evaluation system.

Current research in outcome evaluation (Waskow and Parloff,
l975; Erickson, l975; Ellsworth, l975) suggests that outcomes should
be measured on a variety of dimensions, primarily including the social
and psychological functioning of the clients. No one singleinstru-
ment has emerged as a "best" measure for outcome evaluation; often,
the use of several measures completed by different informants familiar
with the client's functioning is ideally recommended.

The recommended composite of evaluation instruments typically
includes a report from the client about his or her emotional state.
Client self-report instruments, especially symptom checklists, are
seen as easy to complete and score, and as allowing clients to report
systematically on a variety of psychiatric problems (Derogatis, l974;
Schainblatt, l978). Questions remain, however, as to which instru-
ments are most appropriate for what types of clients; research efforts
to date have not systematically addressed this issue.

Given the lack of consensus regarding which instruments are
most useful in measuring the effectiveness of mental health programs,
the instruments selected for a system should be pilot tested and then
assessed according to criteria relevant to their intended use. Unless

this step precedes the actual utilization of the evaluation data, the

 

evaluator will not be able to determine whether the results reflect
the strengths or weaknesses of the instrument or the actual outcome
of treatment.

The usual approach to assessing an instrument is to investi-
gate its psychometric properties. If reasonably high reliabilities
are established for an instrument and it meets some validity criteria,
the instrument becomes commercially available and may be used for a
variety of purposes. Its use may or may not relate to the original
purpose(s) for which it was designed. This approach to selecting an
instrument is not sufficient to insure the instrument is used in an
appropriate context.,

This study is an assessment of the validity of an instrument
selected for use in a pilot project evaluating mental health treatment
programs. The instrument, the Brief Symptom Inventory (BSI)
(Derogatis, 1977) is a 53—item questionnaire measuring the self report
of the respondent's psychological symptoms. The 851 was selected for
this study for several reasons. First, it incorporates many of the
advantages of its parent instruments, the SCL-9O (Derogatis, l977)
and the HSCL (Derogatis et al., T974), which have established relia-
bility and validity properties, and have been used extensively in
psychological assessment research. The 881 has the advantage of being
shorter than the SCL-90 and includes more symptoms manifested by both
outpatients and inpatients than the HSCL. Finally, the 831 is an
instrument similar to those being used or recommended by many state
and federal agencies developing mental health evaluation systems. As

such, it is representative of an approach to mental health evaluation

that could be incorporated into the evaluation of hundreds of pub-
licly funded mental health programs.

In assessing the BSI, this study addresses several questions
relevant to establishing the instrument's construct validity. Does
the BSI accurately assess the client's psychological symptoms at
intake and follow-up? Does it differentiate among groups of people
with inpatient, outpatient or "normal” status? Is the symptomatic
disturbance measured by the BSI congruent with the clinician and a
significant other's report of the patient's symptomatic distrubance?
Does the BSI reflect change that would be predicted on the basis of
clinical experience and prior research?

This study is one step in assessing an evaluation system being
pilot tested by the Michigan Department of Mental Health. Other
aspects of the pilot-~the social functioning and client satisfaction
measures, the feasibility and cost of administering the system and
the efficiency of the procedures--are being assessed in separate
studies. The evaluation model used in this pilot was tested in out—
patient and inpatient units in Clinton, Eaton and Ingham counties to
determine whether it is an appropriate model to implement on a state-
wide basis. Several hundred inpatients, outpatients and non-patients
were tested at pre— and post-treatment intervals. The system incor-
porated many of the design features recommended in national reports
of evaluation strategies; as such, it parallels several other states'
efforts to develop statewide evaluation systems.

The results of this inquiry will further refine the informa—

tion available to evaluators regarding the selection of appropriate

 

instruments for evaluating treatment programs. It will explore the
usefulness of the instrument with regard to particular aspects of the
evaluation system and identify the types of patients and symptom
disturbance measurable by this instrument. The large sample of out-
patients, inpatients and non-patients available to test these issues,
as well as the multiplicity of instrumentation in this study, should
provide uniquely valuable information in the design of an evaluation

system.

LITERATURE REVIEW

The literature relevant to this study is drawn from several
different areas of inquiry. In order to test one aspect of an eval-
uation system, the design of the entire system must be theoretically
and practically sound. The validity of the BSI in measuring treat-
ment outcome is intricately related to the other aspects of the
system--in particular, the selection of a multiple core battery of
instruments and the methodology used to assess the impact of treat-
ment.

The results of previous psychotherapy outcome research are
also relevant to this project. An awareness of outcomes of other
studies assessing the effectiveness of both outpatient and inpatient
programs places this study in better context. If one assumes treat-
ment programs help some people and do not help others, the question
becomes, "How do you design a system which measures that change?"
Reviews of previous studies measuring treatment effectiveness are
helpful in the formulation-of hypotheses and subsequent analysis of
the results of this study.

The literature review begins with a discussion of the Brief
Symptom Inventory, its history, development, and psychometric proper-
ties. The approach to establishing the validity of the BSI is then
discussed. Following the discussion of the validation of the BSI is
a review of literature relevant to designing an evaluation system,

6

particularly selecting the instruments and controversial methodologi-
cal issues in measuring change. Finally, the results of other out-
come studies, including psychotherapy research and inpatient treat-

ment programs, are reviewed.

The Brief Symptom Inventory

 

History and Develgpment

 

The Brief Symptom Inventory has developed over a number of
years in conjunction with efforts to establish a reliable and valid
method of assessing psychological symptomatology through a patient
self-report mechanism. These efforts can be traced to the Cornell
Medical Index (Wider, l948) and the "Discomfort Scale" (Parloff,
1953). The Clinical Psychometrics Research Unit at John Hopkins has
extended this work into the development of the Hopkins Symptom Check-
list and the SCL¥90-R (Derogatis, l97l, 1972, l974). Each of these
instruments has refined and improved the self-report of psychological
symptoms as an assessment tool.

The Brief Symptom Inventory (BSI) is a brief form of the SCL—
90-R. The subject is instructed to indicate on a five-point scale
how much he was bothered by a list of 53 symptoms during the last
week. The responses range from "not at all" to "a great deal.“ The
53 items on the BSI are those items which had the highest factor
loadings on the nine primary symptoms scales of the SCL-90-R. The
BSI takes approximately ten minutes to complete, and was designed for
use with the debilitated patient who cannot complete the 90-item

SCL-90-R and for use in a mental health clinic where the number of

 

clients seeking treatment precludes a more lengthy psychological
assessment.

The BSI measures symptomatic disturbance along the same nine
primary dimensions and three global indices as the SCL-90-R. The
primary dimensions are: Somatization, Obsessive-Compulsive, Inter-
perSonal Sensitivity, Depression, Anxiety, Hostility, Phobic Anxiety,
Paranoid Ideation, Psychoticism. These symptom dimension scores pro-
vide a profile of the nature and intensity of the patient's symptoms
along standard, clinically recognizable classifications of psycho-
pathology. The scores also provide discrete information about
significant details of the patient's condition, including his suici-
dal tendencies, experience of guilt, and aggressive or violent
thoughts.
~ Three global indices of symptomatic disturbance are also
derived in scoring the BSI. The General Severity Index (631) is the
average symptom score and is the best single indicator of the dis-
turbance level. The Positive Symptom Total (PST) equals the number
of symptoms endorsed. The Positive Symptom Distress Index (PSDI)
measures the intensity of symptom distress when all zero responses
(no disturbance) are eliminated.

Derogatis (l977) reports several advantages to using a psy-
chological self-report instrument. The primary advantage is its
reflection of the patient's experience of psychological distress.
This subjective report of the patient's perception of his inner-state
is unique and valuable information for the clinician and researcher.

Other advantages of the self-report symptom checklist include ease

of administration, the minimal professional time required for assess-
ment, and actuarial methods of scoring and interpretation.

The self-report symptom inventory has a variety of potential
uses. The SOL-9O has been used as a screening instrument to deter-
_ mine need for mental health service (Craig and Abeloff, l974) and as
a standard source of information regarding the clinical status of a
patient. Derogatis also reports that the self-report inventory has
become a frequent means of operationally defining “normality" versus

"abnormality" (Derogatis, l977).

Psychometric Properties

 

Although the reliability and validity of the BSI has not been
formally tested, these properties of the SCL-9O have been established.
Because the BSI is a subset of the most critical items in the SCL-
90-R, the psychometric testing of the parent instrument is relevant
'to establishing reliability and validity properties of the BSI. This
relevancy is supported by evidence of correlations ranging from .92
to .98 on the similar symptom dimension scores of the BSI and SCL-
90-R (Derogatis, l977).

Both the internal consistency and test-retest reliability of
the SCL-9O for outpatients have been established with coefficients
ranging from .77 to .90 (Derogatis, Rickels and Rock, l976; Derogatis,
1977; Edwards, l977). The high internal consistency coefficients
indicate the homogeneity with which the items selected to represent
each symptom construct actually reflect the underlying factor. The
stability of the report over time was established by the high cor-

relations between the reports of 90 outpatients assessed twice during

TO

a one-week interval. A recent study assessing the reliability of
five self-report symptom inventories for a nonpatient and outpatient
sample concluded that the SCL-9O approaches perfect reliability in
terms of internal consistency and test-retest reliability (Edwards,
l977).

Several studies have explored different aspects of the
validity of the SCL-90. A major study of the concurrent validity
compared scales of the "90" with similar scales of the MMPI (Derogatis,
Rickels and Rock, 1976). MMPI clinical scales, as well as Wiggins
content scales and Tryon cluster scales correlated highly (range from
.40 to .75 with a sample of 2l9 symptomatic volunteers) with compar-
able SCL-9O scales. Similarly high correlations were found in a con-
current validation study of the "90" with the Middlesex Hospital
Questionnaire (Boleloucky and Horvath, l974).

. The discriminative validity of the "90“ has been investigated
in two studies of medical patients judged to be likely to manifest
psychological symptoms. Both Craig and Abeloff (l974) and Abeloff
and Derogatis (l977) found that cancer patient's symptom profiles on
the "90" were similar to these of psychiatric patients. A study of
clinical depression among participants in a methadone maintenance
program found the scales of the "90" to discriminate between the
depressed and non-depressed groups (Weissman, et al., l976).

Construct validity of the "90" was investigated to determine
the correlation between the operations of measurement and the theo-
retical constructs which they purport to measure. Derogatis and

Cleary (l977) completed a study in which factor scores on the nine

 

ll

primary dimensions (sample of l,000 outpatients) were rotated and
compared with hypothesized structure of the "90". The match between
the theoretical and empirical structures was quite high, thus estab-
lishing supportive evidence for the construct validity of the SCL-
90-R.

Unfortunately, the reports on the validity and reliability
of the "90" have been limited primarily to studies using outpatient
and nonpatient samples. One report (Derogatis, 1977) found SCL-9O
scores gathered on 200 inpatients were two standard deviations above
the nonpatient sample, but did not systematically assess the discrim-
inant validity of the "90" with inpatients. The validity of the "90"
in measuring treatment effectiveness also has not been established.
Although Derogatis (l977) reported that inpatient scores fell about
one standard deviation (measured against community norms) from admis-
sion to discharge, similar reports on change scores for outpatients
have not been reported.

These studies indicate substantial reliability and validity
of the SCL-90 in assessing outpatients' and nonpatients' self-report
of symptomatic disturbance. Derogatis' initial reports suggest the
SCL-90 is valid for inpatients, but this has not clearly been estab-
lished, particularly as an instrument sensitive to pre- and post-
treatment change in symptomatic disturbance. Because these investi-
gations of the psychometric properties of the "90" serve as a proxy
measure for the reliability and validity of the BSI, they bear on
this study's construct validation of the BSI. Particular attention

must be paid to whether the BSI is valid for inpatients, and is

 

l2

sensitive to pre- and post-treatment differences in the inpatient

and outpatient populations.

Factor Structure of the SCL-9O

Investigations of the factor structure of the SCL-9O began
with factor analysis of the original 58-item HSCL (Derogatis, Lipman,
lCovi, Richels, l97l, l972). These studies revealed that depression
and anxiety factors accounted for the bulk of the variance, and had
factorial invariance across social class and psychiatric diagnosis.

The SCL-9O was created by the addition of 32 items concerned
with symptoms of more serious psychopathology to the HSCL. Derogatis
and Cleary (l977) found the "90" to have factorial invariance across
sex on all nine of its primary symptom dimensions. Lipman, Covi and
Shaprio (l977) identified eight similar factors in a study involving
chemotherapy of depressed patients.

A recent study by Hoffman and Overall (l978) was prompted by
their concern that previous factor studies of the "90" had been con-
ducted on patients selected for symptoms of anxiety and/or depression.
Using a clinic population unselected for diagnosis and representative
of an outpatient population, they found the rotated factors of the
”90" to be quite similar to the responses of anxious or depressed
patients. They found the Depression, Somatization and Phobic Anxiety
factors to be the most clearly defined, and most consistent with other
work on the HSCL and SCL-90. However, the large portion of the
variance accounted for by the first unrotated factor and the high

intercorrelations among factors suggest that the "90" measures more

 

13

of a general complaint or general discomfort dimension than distinct
dimensions of psychopathology. Therefore, they recommend limiting
the use of the SCL—90 to its measurement of global disturbance level,
rather than individual Symptom scales.

Investigation of the factor structure of the SCL—90 has
direct bearing on the present study of the validity of the BSI.
Because no studies of the factor structure of the BSI have been

reported, the structure must be assumed to be no better defined than

that of the "90". Therefore, the global measures of disturbance on
the BSI will be the primary focus of this analysis. Symptom scale

scores will be reported for information purposes, but their validity

will not be considered in-depth.

Construct Validation

The validity of the BSI must be established with respect
to the purpose for which the BSI is being used. As Nunnally (l967)
described in his basic text on pSychometric theory, ”a measuring
instrument is valid if it does what it is intended to do. . .one
validates not a measuring instrument, but rather some use to which
the instrument is put” (p. 75). Cronbach (T970) reiterated this
practical approach to validation in suggesting the essential question
is ”How valid is the interpretation I propose for this test?"

Several different types of validity are described by Cronbach
(1970), each of which asks a different question:

(l) Criterion validity: how do measures of some valued

performance (criterion) relate to the test score?

T4

(2) Content validity: do the observations truly sample

 

the universe of tasks or the situations they are claimed
to represent?

(3) Construct validity: how can scores on the test be
explained psychologically? Does the test measure what
it is said to measure?

Whereas criterion-validity is often established through single exam-
inations of the correlation between test scores and certain outcome
criteria (e.g., later performance on a job), construct validity is
established through a long-continued interplay between observation,
reasoning and imagination.

Construct validity is described by Anastasi (l968) as a
comprehensive concept of validity that includes content and
criterion-oriented validity; it is a broad, enduring and abstract
kind of behavioral description and, as such, requires the gradual
accumulation of information about the test from a variety of sources.
The following description of construct validity by Anastasi (l968)
is applicable to the current investigation of the BSI as a tool for
evaluating mental health services:

The theoretical construct measured by a test can be defined
in terms of the operations performed in establishing the
validity of the test. Such a definition would take into
account the variables with which the test correlated sig-
nificantly, as well as the conditions found to affect its
scores and the groups that differ significantly in such
scores. . . It is only through the empirical investigation
of the relationships of test scores to other external data
that we can discover what a test measures (p. l22).

Three common methods of establishing construct validity cited

by Magnusson (l966) are also highly applicable to the BSI:

 

l5

l. The study of differences between groups which should differ
according to the theory for the variable;

2. The study of how the test results are influenced by changes
in individuals or environment which, according to the theory,
should respectively influence or fail to influence the
individual's positions on the continuum; and,

3. The correlation between different tests which are assumed
to measure the same variable (p. 131).

These three aspects of establishing construct validity are
utilized in the present investigation of the validity of the BSI.
Using Magnusson's criteria to determine whether the BSI is a valid
instrument for assessing pre- and post-treatment symptomatic dis-
turbance of outpatients and inpatients, the BSI should:

l. Discriminate among inpatients, outpatients and nonpatients
who are likely to have different levels of symptomatic
disturbance;

2. Be sensitive to pre- and post-treatment differences in
symptoms of outpatients and inpatients; and,

3. Bear a direct relationship to other measures of sympto-
matic disturbance.

Thus, the construct validity of the BSI will be assessed by
testing a series of hypotheses about symptom disturbance of recip-
ients and non-recipients of mental health services, rather than
according to a single measure of correlations between test scores
and criterion scores. The hypotheses described above are further
developed in reviewing research findings regarding pre- and post-
treatment clients' reports of symptoms and their relationship to

other measures of symptom disturbance. Methodological issues in

 

16

evaluation research, as well as the results of psychotherapy outcome

research also provide guidelines for the current investigation.

Designing an Evaluation System

 

The issues inherent in evaluating the outcome of mental
health treatment programs have received increased attention in the
psychological literature in the past several years. An increasing
number of academics are addressing themselves to the issues of how
to design a system that is sensitive to the objectives of the treat-
ment program and also produces valuable information to the planners

and policy-makers for mental health programs.

Instrument Selection

 

Several authors have recently discussed the need for develop-
ment of more relevant outcome measures for evaluating hospital
treatment programs than traditional statistics measuring length of
treatment and return to treatment or relapSe (Erickson, l975; Speer,
l976; Lick, l973). The objective of mental health treatment programs
is not to treat people for a short or long period of time, or to
prevent them from returning to treatment at a later time. Measures
must be used which report whether the patient's psychological and
behavioral functioning improved as a result of treatment.

As Erickson (1975, p. 526) suggested, outcome measures should
reflect the objectives of the treatment program:

In summary, it appears that patient movement statistics

are so full of fallacies and are so difficult to interpret
meaningfully, especially in brief-stay settings, that they

must be regarded as useless or misleading if taken by them-
selves. Efficiency and effectiveness depend on the time

17

needed to reach goals. The optimal treatment period depends

on the needs of the patient and the resources of the program

in question. At this point, patient movement statistics are

as dated as bed census statistics as a basis for computing

cost effectiveness. We can no longer bypass the direct assess-
ment of improvement in the psychosocial functioning of patients
admitted to our care.

A report by the National Institute of Mental Health recently
recommended that outcomes of mental health programs be measured with
a battery of instruments focused primarily on client distress and
social functioning in the commUnity (Waskow and Parloff, 1975). A
group of NIMH consultants noted for their contributions to psycho-
therapy research were asked to select a core battery of instruments.
In doing so, they stressed the importance of selectinginstruments
that could be used for comparing outcomes across a variety of treat-
ment approaches and with a variety of patient populations. Included
in their recommendations for outcome measures were the Hopkins
Symptom Checklist (HSCL) (Derogatis, et al., 1974) and the Personal
Adjustment and Role Skills Scales (PARS) (Ellsworth, 1975). Regarding
the HSCL,Waskow said "Clearly a battery must include a measure of
symptomatology from the perspective of the patient" (p. 253); he went
on to point out that the PARS offers "an extremely important per—
spective to tap, especially if one is interested in the community
adjustment and in the overt social behavior of the patient" (p. 258).

The HSCL is a symptom cheCklist very similar to the BSI,
the major difference being that the BSI measures symptoms in a wider
range of areas (e.g. the BSI includes paranoid and psychotic symp-

toms). This difference makes the BSI more relevant for an inpatient

population; the NIMH report was concerned primarily with measuring

18

outcomes for an outpatient population. In a comparable vein, the
report suggested the PARS might be more appropriate for former
inpatients than for an outpatient neurotic population. Ellsworth
(1975) reported, however, wide use of the PARS for both outpatient
and inpatient populations. Thus, the selection of the BSI and PARS
as core instruments in this study is congruent with the NIHM recom—
mendations, although somewhat adjusted for use with an inpatient and
outpatient population.

Measures of client satisfaction with service have also been
recommended (McPhee, 1975) as an essential part of an evaluation
system. These measures provide information about the client's per-
ception of the impact of treatment on their problem, and a general
measure of their overall satisfaction with the treatment program.

In addition, assessing the family burden arising from the patient's
illness has been included in recommended evaluation instruments
(Arnoff, 1975). As an increasing number of patients are being
treated and maintained in the community rather than hospitals, it is
necessary to assess the impact of this shift on the patient's family.

Measures of patient satisfaction with service and of family
burden are included in the present study. The full complement of
outcome measures, including symptomatic disturbance, role functioning,
satisfaction with service and family burden reflects the complexity
of treatment objectives of mental health programs. Although these
measures may not inter-correlate in a way that offers a simple

definition of treatment success, the support for using a multi-

19

variate outcome measure is convincing (Erickson, 1975; Waskow and

Parloff, 1975; Speer and Tapp, 1976).

Selecting an Informant

The choice of informants, the person providing the informa-
tion about the patient, is an important factor in the design of the
evaluation system. This choice is often intertwined with, orimbedded
in, the selection of the instrument, as the instrument is designed
to measure a certain person' perspective on the patient's function-
ing. There are arguments for and against the choice of the patient,
therapist, significant other (patient's friend or relative), or
research interviewer as the informant. Empirical analysis of these
alternatives offers some guidance, but the ultimate choice is more
determined by the question "Who wants to know what information at
what cost and for what purpose?" (Maguire, 1977).

The client may be the best provider of information regarding
his subjective state and feelings of distress or well being, but the
validity and reliability of their reports of behavioral adjustment
are questionable (Ellsworth, 1968; Paul, 1966; Carr and Wittenbaugh,
1969). Advantages of having the client provide information regarding
his subjective discomfort include the low expense in obtaining this
information (especially in a brief, self-report measUre) and its
relevancy in determining whether the client felt better as a result
of treatment (Schainblatt, 1977). ‘Disadvantages include the possible
distortion of reporting due to a patient being too distressed to

complete the questionnaire or excessively defending against

20

acknowledging his problems. Others argue that patients seeking
treatment should not be burdened or alienated by having to complete
a questionnaire that is designed to evaluate the agency rather than
directly assist the patient (Maguire, 1977).

In traditional settings,the therapist is regarded as the
most qualified to evaluate the outcome of treatment. Ideally, the
therapist is the most sensitive and objective observer of the patient
and the extent to which he/she improves over the course of treatment.
Therapists' ratings of improvement have frequently been used as
measures of outcome, as evidenced by their use in 65 percent of the
165 studies reviewed by Luborsky, et al., (1971).

Arguments against therapists' ratings include the high cost
of using professional time to eValuate treatment, especially in public
mental health agencies where therapists are excessively burdened with
record-keeping related to accountability and evaluation requirements
(Maguire, 1977). A strong bias against using therapists' ratings
also exists among administrators who believe therapists will inflate
their ratings in order to present a better picture of their efforts.
This problem is highlighted in Garfield et a1.'s report (1971) that
therapists' ratings show more improvement than pre- and post-
difference scores on client self-report inventories, such as the
MMPI. Finally, a strong argument against using therapists' ratings
at discharge was raised by Ellsworth (1968) when he found them to be
unrelated to measures of post-hospital adjustment.

Significant other ratings are being used more frequently as

their reliability and validity has been demonstrated, particularly

 

 

21

in assessing patient adjustment in the community (Ellsworth, et al.,
1968). Depending on the data collection procedures, this approach
can be less expensive and burdensome than collecting the same infor-
mation from patients and/or therapists. Disadvantages of this
approach center primarily around the high data loss which occurs
either because the patient has no significant other to designate, or
because the significant other fails to respond to the questionnaire.
Especially with mailed questionnaires, data loss has been as high as
40-60 percent of the respondents (Ellsworth, 1975).

Using a research team to conduct evaluations, usually by
direct patient interviews, is an expensive but highly desirable
approach. Trained interviewers can get highly reliable, objective
and in-depth information about patient functioning. The high costs
of hiring and training research interviewers makes this an infeasible
option unless a significant portion of an agency's budget is allo-
cated for evaluation purposes.

Empirical comparisons of the different perspectives offers
limited guidance in selecting an informant. .Carr and Wittenbaugh
(1969) found little agreement among patients, families and therapists
about treatment effectiveness. Other research (Ellsworth, 1975)
suggests patients' and their significant others' ratings of sympto-
matic disturbance are positively correlated at a statistically sig-
nificant level at intake and at fellow-up. The correlations between
the ratings were higher at follow-up than at intake.

Garfield, Prager and Bergin (1971) systematically examined

the relationships among eight different outcomes, including

22

therapists, client and supervisor global ratings of improvement, and
clients, therapists and supervisors pre- and post-treatment ratings
of client disturbance. The global ratings of change by the different
informants tended to correlate with one another, and to reflect more
change than the pre- and post—ratings of client disturbance. The
authors recommended as sensitive indexes of change, the client's
self-report of depression on the MMPI, and the therapist's global
rating of improvement. They also suggested that difference scores-
may be more objective measures of change than global improvement
ratings, as the latter requires a retrospective judgment about change
over time which is highly susceptible to measurement error.

in a critique of the Garfield et a1. (1971) study, Fiske
(1971) and Luborsky (1971) argued that the use of raw change scores
may have unduly influenced their results. They suggested adjusting
the difference scores to control for the influence of the pre-test
score on the difference score. Luborsky (1971) also defended the
use of global improvement scores as having strong face validity; the
client or therapist may recognize that the client has changed in ways
that are quite significant, but this appears as a minor and insig-
nificant change on a questionnaire tapping may different dimensions
of behavior. Clearly, the instrument used for the evaluation must
be sensitive to patient change; the time period encompassed by the
MMPI is probably too broad to pick up the uneven fluctuations in
patient functioning.

Each of the authors reviewed in this section came to the

same conclusion regarding the selection of an informant--multip1e

 

 

23

measures offer more information than measures from a single informant.
Given the lack of intercorrelations among measures by different
informants, however, the conclusions drawn from the outcome studies
must be specific to the outcome criteria employed. Thus, one measure
might indicate a positive outcome while others do not. While using
multiple outcome measures does not simplify the task of the evaluator,
it is a realistic reflection of the complexity of outcome measurement

and the developing technology in evaluation research.

Selecting a Follow-up Period

 

Measuring outcome at a point after completion of treatment
is essential to determining whether treatment had a lasting impact.
Few studies of outpatient psychotherapy describe how outcome varies
according to the timing of the follow-up (Luborsky, 1971; Meltzoff
and Kornriech, 1970). Erickson (1975, p. 529) addressed this issue
more directly in his review of outcome studies in mental hospitals,
when he wrote:

The shorter the follow-up period, the easier it is to

get complete data but the more uncertainty there is as to
whether the patient has had an adequate trial period. As
the follow-up period grows longer, it becomes harder to
determine whether a relapse is due to a deficiency in the
hospital treatment program or to unforeseen crises.
The impact of following up inpatients at different points in time is
not clear from the research. In general, following up patients from
one month to one year after discharge indicates chronic patients are

typically restored to a marginal pre-morbid level of functioning

(Davis, Diritz and Pasamanick, 1972; Ellsworth, et al., 1968). When

 

 

n. *_;_‘___.‘

- unw.".‘_s-- ..

 

24

patients have improved, symptom reduction is more apparent than
improvement in social functioning (Ellsworth, et al., 1968).

As Schainblatt (1977) noted, there appears to be no rational
basis for selecting a follow-up period in designing an evaluation
system. Including change measures for both inpatients and outpatients
in the present study further complicates the issue, as outpatient
treatment usually does not have as definite an ending point as the
inpatient's discharge._

The follow-up times selected for the present study were three
and six months after treatment was initiated. This allowed for a
comparison of adjustment levels of inpatients and outpatients at two
points in time after intake. Given a short average length of treat-
ment in the outpatient and inpatient units in the study, most patients
should have completed treatment by the first or second follow-up

times.

Data Loss

Related to the follow-up issue is the loss of data due to the
difficulty of contacting patients at follow—up. Ellsworth (1975)
described this problem as one of the most serious threats to the
validity of evaluation research, especially with a mailed-
questionnaire follow-up. When a significant percentage of patients
are not included in the follow-up, the conclusions regarding outcome
are limited by the extent to which the "lost subjects" differ from

the entire subject population.

25

Schainblatt (1977) has proposed several procedural techniques
to minimize data loss. They include repeated contact of subjects who
fail to respond to initial mailed questionnaires, including telephone
interviews of subjects. These recommendations have been incorporated
in the present study inorder to maximize the return rate and mini-

mize the data loss.

Methodology

 

If patients' conditions are asSessed at intake and at a
selected follow-up period, the study has incorporated a "one group
pretest-posttest design." Campbell and Stanley (1963) concluded
that while this design is better than no evaluation at all, it is
subject to erroneous conclusions, as it fails to control for threats
to internal validity. Any measure of change between the pre- and
post-test may be due to history, the occurrence of other change-

producing events; or maturation, the biological or psychological

 

processes which systematically vary with time; or testing, the effect
of subjects taking a test more than one time.

Another major threat to internal validity which has been
discussed extensively in evaluation methodology literature is

statistical regression. This is the tendency of persons who have the

 

most deviant scores (high or low) from the mean to regress toward the
mean on second measurement. Presumably, the more deviant the score,
the larger the error of measurement it contains. Campbell and

Stanley (1963) suggested that regression effects are more apparent

 

 

 

26

for groups selected for their extreme scores than for groups selected
for independent reasons which happen to have an extreme mean.

The strongest methodological correction to the "one group
pretest-posttest design" is to centrol for threats to internal valid-
ity by randomly assigning patients to a treatment (experimental) and
no-treatment (control) group. With random assignment, the effects
of maturation, history, testing, and statistical regression are con-
trolled. However, this technique is often unacceptable to adminis-
trators of public mental health programs who feel responsible for
providing service to all persons requesting it. While true experi-
ments might be possible for evaluations of one-time, innovative pro-
grams, they are infeasible for evaluation of ongoingprograms.

The non-equivalent control group design has been recommended
as an alternative to randomly assigning subjects to an experimental
and control group (Campbell and Stanley, 1963; Speer and Tapp, 1976).
Although this design does not provide as strong a control for threats
to internal validity, it provides a reasonable and feasible alterna-
tive to the more stringent random-assignment design. This strategy
involves taking pre- and post-test measures on subjects from an
available, intact group which may or may not be equivalent to the
treatment group on relevant variables, such as socioeconomic status,
initial level of disturbance, etc. For example, Speer and Tapp

(1976) recommended comparing test scores established for non-patient

 

27

population with a patient population to assess the impact of treat-
ment for the patients.1

Critics of this design (Campbell and Erlebacher, 1970;
Campbell and Boruch, 1975) have argued that using a non-equivalent
control group biases the measure of a treatment effect. They con-
tend that different rates of maturation and regressionto the mean
(presumable "better" rates for the non-equivalent group) will under-
estimate the gain to the experimental group resulting from the
treatment.

Kenny (1975, p. 346) described the problem as follows:

Given these pre-treatment differences on the dependent

variable, interpreting a difference between the experimental
and the control group on the_post-treatment measure becomes
problematic. To interpret a post-treatment difference one
must examine the magnitude of the pre-treatment difference,
and that difference or perhaps some adjusted difference can

be used as a baseline to judge the post-treatment difference.
At issue is whether pre—treatment differences should increase,
decrease, or remain stable if there is no treatment effect.

In order to measure the treatment effects in the non-
equivalent control group design, Kenny recommended one of four sta-
tistical analyses: (a) analysis of covariance; (b) analysis of co-
variance with reliability correction; (c) raw change score analysis;
or (d) standardized change score analysis. Alternatively, Nunnally
(1975) suggested that using analysis of covariance partials out the

effects of pre-test differences or post-test differences. Instead,

 

1This type of comparison may be possible with the current
data set, as measurement on a nonpatient sample will be taken at
two points in time concurrent with pre- and post-treatment measures
on the patient sample.

28

he recommended examining the interaction of the two groups with pre-
test and post-test measures.

Measuring change, whether among treated or untreated subjects,
is a problematic area which has stirred great controversy among
methodologists. Cronbach and Furby (1970) argued against the use of
gain scores however they might be adjusted or refined because "such
scores are systematically related to any random error of measure-
ment" (p. 68). They point out that if raw difference scores are used,
the difference between the "pre" and "post" test scores is related
to the "pre-test" score. The portion of change that is a function of
the initial level of adjustment must be removed. Different statis-
tical techniques are recommended for this purpose. Meltzoff and
Kornreich (1970) have suggested the transformation of initial and
final scores into standardized scores. Ellsworth (1975) recommended
using residual change as a means of removing the correlation between
the pre- and post-test scores with the difference score. Others con-
tend that residualized gain scores use "observed scores" rather than
"true scores" in the residualization, and argue in favor of a "true
score change" formula for measuring change (Luborsky, 1971; Cronbach
and Furby, 1970). For the type of design used in the current study,
however, Cronbach and Furby (1970) have recommended using a signifi-
cance test on the difference between sample means to describe the
magnitude of the treatment effect.

Unfortunately, few real life examples comparing the alterna-
tive methodological approaches described above appear in the evalua-

tion literature. The controversy surrounding the use of

29

nonequivalent control groups has been discussed entirely with respect
to evaluation of the Head Start Project (Campbell and Erlebacher,
1970), rendering the above arguments applicable only to the evalua-
tion of an educational program fordisadvantaged children. The
methodological issues may be quite different in evaluating a mental
health program; for example, the concept of maturational rate of
subjects with respect to learning is dissimilar to the fluctuations
in psychological distress of mental patients.

The controversy surrounding the measurement of change also
has failed to produce applied studies of how outcomes differ when
various statistical techniques are applied to the analysis.

Ellsworth (1975) offered a rare demonstration of the high correlations
between pre-hospital adjustment and uncorrected outcome and gain
scores, and how residual change scores give "base free" measurements
of change. He pointed out that if two treatment programs are equally
effective but one has initially more disturbed patients assigned to
it, gain score analysis will favor that treatment program. If out-
come scores are used, the treatment program with better adjusted
patients would be favored. Using residual change scores would

reflect treatment differences rather than group differences in

initial adjustment scores.

Previous Outcome Studies
Studies testing psychotherapy outcomes are reviewed by
Luborsky (1971), Meltzoff and Kornreich (1970), and Erickson (1975).

These reviews provide information concerning the results that might

30

be expected from an evaluation of mental health treatment programs.
As such, this information should be valuable in structuring and
critiquing the design of an evaluation system.

Concerning the effectiveness of treatment programs in general,
Meltzoff and Kornreich (1970) concluded:

Far more often than not, psychotherapy of a wide
variety of types and with a broad range of disorders has
been demonstrated under controlled conditions to be accom-
panied by positive changes in adjustment that significantly
exceed those that can be accounted for by the passage of
time (p. 175) . . . Reviews of the literature that have
concluded that psychotherapy has, on the average, no
demonstrable effect are based on an incomplete survey of
the existing body of research and an insufficiently
stringent appraisal of the data . . . 0n the contrary,
controlled research has been notably successful in
demonstrating significantly more behavioral change in
treated patients than in untreated controls. In general,
the better the quality of the research, the more positive
the results obtained (p. 177).

Luborsky (1971) was concerned primarily with what factors,
including attributes of the patient, therapist, and treatment modal-
ity, influence the outcome of psychotherapy. He reviewed 166 studies
of the outcomes of outpatient psychotherapy with adult patients under-
taken from 1946-1969. In his introduction, he suggested that

patients, as a group, will improve in treatment, but that individual

 

patients may or may not improve, depending on certain factors. Iden-
tifying those factors is the focus of his article.

The level of pathology prior to treatment was reported in
14 out of 28 studies to be inversely related to outcome. In these
14 studies, the healthier the patient at intake, the greater the
change reported at the end of psychotherapy. 0f the other 14 studies,

13 reported no significant relationship between initial level of

31

disturbance and change, and one study indicated that sicker patients
changed more in short-term treatment. Meltzoff and Kornreich (1970)
reported a similar discrepancy in studies relating initial disturb-
ance level to outcome. An equal number of studies found a positive,
negative and no relationship between pre-morbid state and outcome.
These results were applicable in studies of schizophrenia and
inpatients, as well as psychoneurotics and outpatients. They offered
the following explanation of this diversity of findings:

Severity of maladjustment can be looked at clinically in

terms of symptom intensity, duration, pervasiveness or
extent of interference with contemporary life; or it can be
viewed more broadly as the balance of functional assets and
liabilities in an individual's life compared to some esti-
mate of his potential. Some investigators simply define it
operationally in terms of psychometric performance. It is
easy to see why different research orientations to the
meaning of severity can lead to different results and dif-
ferent approaches to prognosis (Meltzoff and Kornreich, 1970,
p. 218).

Luborsky (1971) and Meltzoff and Kornreich (1970) reviewed
several studies which indicated the presence of strong affect, pri-
marily anxiety and depression, was related to change. Similarly,
the number of complaints endorsed on problem checklists was a posi-
tive factor. Both of these findings suggest the patient who reports
more affect or complaints is asking for help, and thus is probably
more ready to change than patients reporting no problems.

Luborsky (1971) reported other patient factors associated
with positive change were patient‘s intelligence, motivation for
treatment, younger age and higher educational achievement. Meltzoff

and Kornreich (1970) reached a different conclusion regarding the

relationship between patient factors and outcome. They pointed out

32

the lack of adequate controls in these studies and argue that studies
demonstrating a positive relationship between patient factors (age,
IQ, marital status, education and social class) and outcome of treat-
ment have confounded the variables. They suggested further research,
controlling for intervening variables affecting these relationships,
should be conducted before definitive conclusions are drawn.

Luborsky's review concluded that treatment factors generally
were not strongly associated with outcome, except that combined forms
of treatment (e.g. group plus individual therapy, pharmacotherapy
plus psychotherapy) were more effective than single forms of therapy.
Again, Meltzoff and Kornreich's review concluded that these rela-
tionships have not been sufficiently demonstrated. They argued that
the methodological weakness in studies comparing individual and
group treatment precludes drawing conclusions regarding their rela-
tive effectiveness.

Luborsky acknowledged that the outcome criterion being
reported affects the results of the study. Most studies he reviewed
tended to use only therapist's gross improvement ratings; these
studies tended to show greater improvement than those using pre- and
post-treatment difference scores on client self-report inventories.
While therapist's rating of improvement has been criticized often as
a biased measure of change (Garfield, Prager and Bergin, 1971), it
is one of the few criterion measures which tends to have consistent
significant correlations with other criterion measures (Fiske, et

al., 1964; Garfield, et al., 1971).

33

In his review of outcome studies in mental hospitals,

Erickson (1975) highlighted the importance 6f the type of measure
used as the criterion for success. He criticized the use of tradi-
tional length-of—stay and return—to-hospital statistics as outcome
measures, as they are more reflective of hospital administrative
policy than successful or unsuccessful patient adjustment. Similarly,
he warned against equating discharge with successful adjustment.
The lack of correlation between measures taken in the hospital and
post-hospital adjustment (Ellsworth, et al., 1968) is evidence of
the fallacy in using discharge rates as an outcome measure. The
patient's functioning at discharge is not necessarily predictive of
how he will be functioning several months after discharge. This‘
indicates that neither the flag; of discharge nor the level of func-
tioning at discharge are satisfactory outcome measures.

Psychosocial measures of adjustment when the patient has
returned to the community are recommended as outcome measures of the
effectiveness of treatment (Ellsworth, 1975; Erickson, 1975). If
made in addition to discharge measures, follow-up measures give
information about whether the patient maintained or built on gains
made in the hospital.

Erickson's review (1975) concluded the improvement in patient
functioning after discharge is usually limited to regaining a mar-
ginal, pre-morbid level of functioning. Chronic patients had poorer
outcomes than acutely disturbed persons without prior hospitalizations.
Some studies found patients who were married and employed steadily

prior to hospitalization also had better outcomes.

34

Inpatient outcome studies indicate improvement in symptomatic
disturbance is more likely than improvement in role skills or psycho-
social behaviors (Ellsworth, et al., 1968; Pasamanick, et al., 1967).
Employment following hospitalization is rarely improved over pre-
morbid functioning, and most discharged patients will not be working
full time at follow-up (Anthony, et al., 1972).

Erickson (1975) concluded that most treatment programs,
regardless of the type of interventiOn, have not been able to
resotre marginally functioning patients to a normal, community level
of psychosocial functioning. Meltzoff and Kornreich (1970) also
pointed out that in several experimental studies demonstrating posi-
tive outcomes, the gains of the experimental groUp were either tem-
porary or later matched by advances in the control group. This
suggests hospital treatment programs either fail to produce long-
term change in patients or the illnesses which repeatedly bring a
patient to the hOSpital are a continuous phenomenon with recurring
elevations in symptomatic and behavioral disturbances. While both
explanations undoubtedly have merit, the long-term follow-up of
patients suggests the latter explanation has more implications for
the evaluation of outcomes. If the most successful and innovative
inpatient programs have been unable to produce long-term improvement
in patient functioning, expecting traditional inpatient programs to
effect long-term patient gains is unrealistic.

Drawing generalizations from the outcome literature is com-
plicated by the lack of uniform, intercorrelated outcome measures

and the variability in patient characteristics prior to treatment.

35

Outcomes clearly reflect interactions between the type of hospital
treatment program, aftercare services and patient factors such as

diagnosis, chronicity and sex. Erickson (1975) recommended future
studies become more specific in relating the type of treatment and
patient characteristics necessary to achieve the desired treatment

objectives within a certain time period.

METHOD

Description of the Study

This study involved testing a series of hypotheses relevant

to determining the validity of the BSI. The data were collected as
part of a pilot evaluation project Sponsored by the Michigan Depart-
ment of Mental Health, in conjunction with the Clinton-Eaton-Ingham
County Community Mental Health Board and St. Lawrence Hospital.
The study design originated in an Urban Institute proposal (Schain-
blatt, 1977), and was subsequently modified by local and state par-
ticipants in the project. The purpose of the pilot project was to
assess the feasibility, cost and utility of a system to monitor the
outcomes of state-supported mental health treatment programs.

The pilot evaluation project was conducted in seven local
inpatient and outpatient units. The pilot was designed to collect
pre- and post-treatment information about a patient's psychological
symptoms and social functioning, family burden and patient satisfac-
tion with service. Overlapping information was provided by three
different informants: the patient, his or her clinician, and a
"significant other" designated by the patient. A non-patient sample
was tested with identical self-report instruments and procedures,
thus creating a quasi-experimental design with a nonequivalent

control group.

36

37

Subjects

All adults beginning outpatient or inpatient treatment from
August 1977 through October 1977 in several outpatient and inpatient
units were eligible to participate in the study. The agencies par—
ticipating in the study were: Ingham Community Mental Health Center,
Clinton County Counseling Center, Eaton County Counseling Center,
Mason Mental Health Center, Capitol Area Counseling Center, St.
Lawrence Inpatient Unit, and New Riverside Treatment Center. In
addition to the outpatient and inpatient samples, a random sample
of 825 cemmunity residents in theClinton, Eaton and Ingham counties
were asked to complete the BSI in October, 1977 and again in April,
1978. Approximately 80 percent of these residents completed the BSI
in 1977; information is not currently available on the second
administration in 1978.

The demographic characteristics of the inpatient, outpatient
and nonpatient populations in the study are shown in Table 1. All
three populations had more women than men participating, with the
majority of each group under 40 years of age. Most people had been
or were currently married.

The outpatient population had a racial composition similar
to the residents (8 percent nonwhite), but the inpatient population
had a greater percentage (16 percent) of nonwhites. Outpatients
and inpatients generally had less education, poorer employment status
and lower incomes than the resident population. The greatest

differences among the groups was in their prior use of mental health

l-“"

38

Table l.--Demographic Characteristics of Community Residents and
Clients in Tri-County Study.

 

 

§2$?§2;§§1 Outpatients Inpatients
n = 825 n = 615 n = 198

Sex:

Male 37% 34% 43%

Female 63% 66% 57%
Age:

18 - 29 40% 52% 38%

3O - 39 24% 29% 26%

4O - 49 14% 12% 17%

50 - 59 11% 5% 6%

60 - 69 6% 2% 7%

3. 70 5% 1% 6%
Marital Status:

Never Married 24% 26% 30%

Married or Remarried 62% 40% 38%

Widowed, Separated

or Divorced 14% 35% 32%

Race:

White 94% 92% 84%

Nonwhite ‘ 7% 8% 16%
Education:

High School (grades 0-12) 53% 64% 80%

College (grades 13-16) 35% 33% 18%

Graduate (grades 17 and

above) 12% 4% 3%

Employment:

Currently Employed 56% 51% 30%

Unemployed 2% 15% 28%

Not in Labor Force 42% 34% 27%

Other 0% 0% 15%
Income: .

$0 - $3,999 11% 37% 31%

$4,000 - $7,999 15% 15% 18%

$8,000 - $11,999 18% 15% 17%

$12,000 and above 56% 32% 34%
Previous Mental Health Serv:

Previous Outpatient 5% 41% 18%

Previous Inpatient 1% 11% 47%

Current Outpatient 2% - -

None 92% 48% 35%

 

1Includes residents of Clinton, Eaton and Ingham counties who
agreed to participate in Community Norm Survey conducted by Depart-
ment of Mental Health in October, 1977.

39

services, 6 percent of the residents, compared to 52 percent of the
outpatients and 65 percent of the inpatients, had had prior mental

health treatment.

Instruments

 

The symptomatic disturbanCe of patients was measured by the
Brief Symptom Inventory (Derogatis, 1977), the short form of the SCL-
90-R, a self-report inventory for symptom patterns of psychiatric
and medical patients (Derogatis, Rickels, and Rock, 1976). The
major reliability and validation studies have been done on the SCL-
90-R, although the correlations on the identical symptom dimensions
of the two scales range from .92 to .99 (see Literature Review for
more information on the SCL-90).

The BSI consists of 53 items that the respondent is asked
whether he/she was bothered by during the past week. Responses are
on a five-point scale, ranging from "not at all'I to "extremely".

The BSI measures psychological symptoms along nine primary dimensions:
Somatization, Obsessive-Compulsive, Interpersonal Sensitivity;
Depression, Anxiety, Hostility, Phobic Anxiety, Paranoid Ideation,
and Psychoticism. Three global indices of disturbance are calculated:
the Global Severity Index (GSI), which combines information on the
numbers of symptoms and intensity of perceived distress; Derogatis
cited the GSI the best single indicator of the current level or
depth of the disorder. The Positive Symptom Distress Index (PSDI)
is a pure intensity measure, corrected for the number of symptoms.

It functions as a measure of response style, indicating whether the

40

patient is "augmenting" or "attenuating" symptomatic distress in his
style of reporting. The Positive Symptom Total (PST) is the number
of symptoms the patient reports as positive (Derogatis, 1977).

The Brief form of the Hopkins Psychiatric Rating Scale
(B-HPRS) was used to measure the clinician's perception of the
patient's symptomatic disturbance. The long form of the HPRS was
designed by Derogatis (1977) to be the "psychiatrist's version" of
the SCL-90-R. The first nine symptom dimensions of the HPRS are an
analogue to nine dimensions on the BSI and SCL-90-R. The B-HPRS was
designed by Derogatis for use in the present study; the changes were
some minor alterations in the item-descriptors and an elimination of
eight additional dimensions on the HPRS: Thus, the B-HPRS includes
the same nine symptom dimensions as the BSI, and one Global Pathology
Index scale ranging from “absent" (0) to "extreme" (8) pathology.

The HPRS is designed to be used by psychiatrically sophisti-
cated clinicians (e.g., psychiatrists, psychologists, psychiatric
nurses). Each dimension of the scale is defined on the form and
represented on a seven-point scale ranging from "none“ to ”extreme".
Three of the seven points on each dimension are further defined by
brief clinical descriptors.

Because the HPRS has only recently been developed and is not
yet commercially available, formal psychometric properties are not
developed. The inter-rater reliability of the B-HPRS was not formally
tested; however, two hour training sessions on the instrument were

held with all clinicians participating in the study. This allowed

41

clinicians to discuss the scale and their orientation to completing
it, and to rate several sample clients.

Client social functioning was measured on the Personal
Adjustment and Role Skills (PARS) questionnaire (Ellsworth, 1975).
The PARS is a 61-item questionnaire completed by the patient's
designated significant other (usually a family member or friend).
Although the same form is administered to males and females, factor
analyses indicate the items load differently for men and women.

Thus, there are nine primary dimensions for men: Interpersonal
Involvement, Depression-Agitation, Anxiety, Confusion, Alcohol—Drug
Abuse, Household Management, Relationship to Children, outside Social,
and Employment. For women, the Depression-Agitation and Anxiety
dimensions are collapsed into one dimension, Depression-Anxiety, thus
creating eight dimensions for women. The respondent is asked whether
the patient has, during the past month, been a certain way or per-
formed a certain function. The responses are on a four-point scale,
ranging from "rarely" to "always", except on the Social and Employ-
ment dimensions, in which the responses describe a certain type of
activity, such as "stayed at home this paSt month" to "often involved
in outside activities."

The PARS has been developed over the past ten years, and
has been refined in five amendedversions. It has been used with
hundreds of inpatients and clinic clients in extensive research
projects. The psychometric properties of the PARS are well estab-
lished. The reliability estimates for the PARS test-retest stability

range from .66 to .95. The internal consistency (alpha) estimates

42

ranged from .67 to .94 for hospital patients and .73 to .91 for clinic
clients. Validity properties have been established indicating high
agreement between ratings by significant others and patients; demon-
strations of the instrument's ability to reflect different rates of
change for female and male patients and to predict whether the

patient is likely to have been hospitalized or seen at a clinic have
also been made as part of the validation of this instrument (Ellsworth,

1975).

Data Collection

 

The data collection was handled by the service agency and
the Department of Mental Health (DMH). At the community mental
health centers, the secretaries and receptionists were trained to
administer the consent forms and BSI questionnaire to incoming clients
prior to their first intake session. At the inpatient units,
psychiatric nurses and aides administered the consent forms and BSI
questionnaires normally within 48 hours of admission. This informa-
tion was then transmitted to the DMH data coordinator, who logged
the data, and sent the PARS questionnaire to the designated signifi-
cant other. The DMH data coordinator was also responsible for
mailing questionnaires to the significant other, clients and former
inpatients at the three and six-month follow-up.

The B-HPRS was completed at the time of the patient's intake
by the clinician in the outpatient units, and the psychiatric nurses

and social workers in the inpatient units.

43

The study of non-patient residents in Clinton, Eaton and
Ingham counties was contracted to a private research firm, Westat,
Inc. in Rockville, Maryland. They provided trained interviewers who
made telephone contact with 1,000 randomly selected residents. The
respondents were asked if they were willing to receive a mailed
questionnaire (identical to that completed by the patients) and to
provide socio-economic information on the telephone about themselves.
Eight hundred and seventy four persons agreed to participate; 75
percent of those people returned the completed questionnaire. A
second questionnaire was sent to these residents at a time concurrent
with the six-month follow-up of the patient sample.

Most of the procedures for collecting data were developed by
the Urban Institute consultants to the project. These procedures
were designed to maximize the response rates and minimize the costs
of data collection. In order to test the impact of a three and a
six-month follow-up on response rates and costs, a random sample of
half the outpatients were not followed up at three months. All
inpatients were followed up at three months; all outpatient and
inpatient subjects were followed up at six months. Table 2 shows
the number of subjects who returned questionnaires at each point in

the data collection process.

Hypotheses

 

In examining the BSI's validity in measuring the effective-
ness of mental health treatment programs, several hypotheses are

proposed as criteria for establishing construct validity, and are

44

Table 2.--Number of Subjects With BSI, PARS and HPRS Questionnaires.

 

 

Inpatients Outpatients

Intakes During Study Period 202 617
Subjects with BSI

Intake 122 518

Three Month Follow Up 49 160

Six Month Follow Up 48 287
Subjects with PARS

Intake 41 247

Three Month Follow Up 29 84

Six Month Follow Up 18 161
Subjects with HPRS

Intake 167 326

Discharge 105 N/A

 

tested. These hypotheses reflect previous research findings on how
a valid self-report measure of symptoms should function with a
psychiatric and non-psychiatric population.

The first question regarding the BSI's validity is whether
the self-report scores differentiate groups of individuals who, on
the basis of independent criteria, would be expected to report sig-
nificantly different levels of disturbance. The three groups par-
ticipating in this study are differentiated by their inpatient,
outpatient or nonpatient status. Each group should report severe,
moderate or mile symptomatic disturbance respectively. If the BSI
is a valid self-report instrument, its scores should reflect these

group differences.

 

Hypothesis 1:

Hypothesis 2:

 

Hypothesis 3:

Hypothesis 4:

45

Inpatients will report significantly greater
symptomatic disturbance than outpatients and
nonpatients at intake.

Outpatients will report significantly greater
symptomatic disturbance than nonpatients at
intake.

At intake, inpatients will score higher on the
psychoticism scales (7-9) than outpatients
and nonpatients.

At intake, outpatients will score higher on
the neuroticism scales (1-6) than the non-
patients.

The second major area Of construct validation of the BSI

\ is its relationship to other measures of symptom disturbance. The

Personal Adjustment scales on the PARS (completed by the significant

other) and the Hopkins Psychiatric Rating Scale (completed by the

clinician) provide independent measures of the patient‘s symptomatic

disturbance.

Hypothesis 5:

Hypothesis 6:

Hypothesis 7:

Hypothesis 8:

At intake and follow-up, inpatient self-
report of symptomatic disturbance will have
a significant, positive correlation with the
significant other's report of the patient's
symptoms.

At intake and follow-up, outpatient self-
report of symptomatic disturbance will have
a significant, positive correlation with the
significant other's report of the patient's
symptoms.

At intake and follow-up, the correlation
between the significant other's rating and
patient's self-report of symptomatic dis-
turbance will be higher for outpatients
than inpatients.

At intake, the self-report of outpatient
symptomatology will be positively correlated
with the clinician's rating of symptoms.

Hypothesis 9:

Hypothesis 10:

46

At intake, the self-report of inpatient
symptomatology will be positively correlated
with the clinician's rating of symptoms.

At intake, the correlation between the clin-
ician's rating and the patient‘s self-
report of symptomatic disturbance will be
higher for outpatients than inpatients.

The last area of investigating the construct validity of the

BSI is determining whether the scores differentiate between pre-

and post-treatment levels of Symptomatic disturbance. The instru-

ments sensitivity to changes in patient's symptoms is an important

factor in establishing its validity as a tool for evaluating treat-

ment effectiveness.

Hypothesis 11:

Hypothesis 12:

Inpatients will report significantly less
symptomatic disturbance at follow-up than
at intake.

Outpatients will report significantly less
symptomatic disturbance at follow-up than
at intake.

RESULTS

Subject Participation Rates

 

The participation rates for subjects completing BSI ques-
tionnaires and other informants completing PARS and HPRS question-
naires are given in Table 2 (page 44). Many of the patients who
began treatment at the participating outpatient and inpatient centers
did not complete questionnaires at intake and follow-up. This data
loss raises the question of representativeness of the data collected
for the sample of subjects who did participate in the study. Of
concern is Whether the study gathered data from a biased sample of
subjects who were substantially different from patients not parti-
cipating in the study.

As shown in Table 2, 202 inpatients and 617 outpatients
began treatment during the intake period of the study. About 60
percent of the inpatients and 85 percent of the outpatients completed
a BSI at intake. Of the patients pgt_completing an intake question-
naire, most simply refused to do so. A smaller portion were unable
to complete the BSI because they were too disturbed to do so, or
were inadvertently omitted from the study because they were over-
looked by the staff handling the intake procedures.

After the intake BSI was completed, subjects were asked to
Sign a consent form giving permission to be sent a follow-up ques-

tionnaire and designating a significant other to be sent a PARS

47

48

questionnaire at intake and follow-up. Most of the subjects com-
pleting the intake questionnaire agreed to receive a follow-up
questionnaire. Only about half of these subjects were willing or
able to name a significant other to receive a PARS.

The actual follow-up return rates are also shown in Table 2.
For the three-month follow-up, about 160 outpatients and 49 inpatients

' The six-month fellow-up yielded 287 out-

returned a completed BSI.
patient and 48 inpatient BSI questionnaires. Thus, of the subjects
eligible to participate in the study, 24 percent of the inpatients
and 47 percent of the outpatients completed intake and six-month
follow-up questionnaires. A smaller percentage of inpatients (13
percent) and outpatients (18 percent) had completed BSI question-
naires at intake, three and six months.2

Data loss was also significant for the PARS and HPRS (see
Table 2, page 44). Many patients were unable or unwilling to desig-
nate a significant other, and the PARS questionnaire return rates
for significant others was also low. Clinicians completed HPRS
questionnaires for about 80 percent of the inpatients and 50 percent
of the outpatients. The low percentage of completed HPRS's for out—
patients was due to poor staff cooperation at one community mental

health center; other centers had about 75 percent completion rates

for the HPRS.

 

1For reasons outside the control of the author,_a random
sample of only one half of the outpatients was followed up at three
months.

2The Urban Institute report on the project (Schainblatt,
1979) gives a more detailed description of response and consent rates.

49

There are two different samples of subjects in this study--
the sample completing intake questionnaires and the sample completing
intake gpg_follow-up questionnaires. The extent to which each sample
is representative of the entire incoming population of inpatients

and outpatients is discussed below.

Rgpresentativeness of Intake Sample

 

The first issue--whether the intake data is generalizable to
all outpatients and inpatients beginning treatment at the partici-
pating facilities--was examined with a series of chi-square tests
on the demographic and diagnostic variables of subjects completing
and not completing a BSI at intake (see Table 3). When comparing
the demographic characteristics (sex, marital status, employment,
education, annual income, age, ethnic status, etc.) of the sample,
significant differences emerged on one variable, annual income. Out-
patients completing an intake questionnaire had significantly higher
incomes than outpatients not completing a questionnaire. For the
inpatient sample, a larger percentage of patients completing an
intake questionnaire were not married than among the non-
participating patients.

Demographic variables are usually considered in determining
generalizability because they have an assumed relationship to the
mental health of the patient sample. More direct measures of disturb-
anCe level collected on the sample include previous mental health

1

service, diagnosis, PPB objective, and the clinician's rating of

 

1This variable is information routinely collected by the
Department of Mental Health and reflects the clinician's assessment
of the primary treatment objective at intake.

50

Table 3.--Oemographic Characteristics of Subjects Completing and Not Completing an Intake

 

 

Questionnaire.
OUTPATIENTS INPATIENTS
CHARACTERISTICS Intake Questionaire was Intake Questionnaire was
COMPLETED NOT COMPLETED COMPLETED NOT COMPLETED
(n=506) (n=82) (n=110) (n=72)
Sex: x2 = .05 x2 = .98
Male 34% 36% 40% 49%
Female 66% 64% 60% 51%
Ethnic Status: x2 = 1.34 x2 = .79
White 93% 89% 86% 79%
Non-White 7% 12% 14% 21%
Marital Status: x2 = 3 05 x2 = 6 44*
Married 41% 31% 30% 48%
Not Married 59% 69% 70% 52%
Employment: X2 = .21 X2 = .01
Employed 51% 54% 35% 36%
Unemployed or Not in Labor 49% 47% 65% 64%
Force
Education: X2 = .07 X2 = .01
High School 64% 63% 80% 80%
College 36% 37% 20% 20%
9
Previous Mental Health Service: 1‘ - 6.02* X2 - 6.36*
None 50% 37% 29% 48%
Previous Inpatient 9% 1 % 52% 39%
Other (outpatient, etc.) 40% 48% 19% 14%
Annual Income: x2 = 5.89* x2 = 2.303
Less than $4,000 or
Public Assistance 36% 45% 33% 32%
$4,000 - 11,999 30% 34% 38% 29%
$12,000 or more 34% 21% 29% 39%
PPB Objective: x2 = 7.82*
Psychosocial Adjustment 81% 70%
Crisis Resolution 9% 15% Not
Rehabilitation/Habilitation 3% 10% Available
Maintenance 2% 5%
Diagnosis: x2 = 4.33 x2 = .393
Mentally retarded or
Organic Brain Syndrome 1% 3% 9% 1%
Psychosis 10% 16% 32% 25%
Neurosis 89% 81% 59% 64%
Age: x2 = 5.36 x2 = 4.99
18-39 80% 72% 68% 61%
40-59 16% 18% 25% 21%
60 or older 4% 10% 3% 18%

 

aIn these categories, missing data resulted in the sample size being reduced to
n=55 and n=35 for the Completed and Not Completed groups respectively.

*
p <.05.

51

symptomatic disturbance on the HPRS. Statistical tests were con-
ducted on these measures to determine whether the participating sub-
jects were more or less disturbed at intake than the entire popula-
tion.

For the outpatient sample, a chi-square test revealed sig-
nificant differences between subjects completing and not completing
an intake questionnaire on two of the four initial disturbance level
variables-eprevious mental health service and PPB objective (see
Table 3). The differences indicate that the group not completing a
questionnaire had more previous mental health treatment and had a
treatment objective of crisis resolution, rehabilitation/habilitation
or maintenance more frequently than the group completing a question-
naire. The "not completed" group also had a higher percentage (16.0
percent compared to 9.8 percent) of persons with a diagnosis of
psychosis than the "completed" group, although this difference was
not statistically significant. (Unfortunately, too few outpatients
not completing a questionnaire were rated on the HPRS to compare the
differences on this measure of intake disturbance level (see Table 4).

In summary, the outpatients participating in the study had
less previous mental health service, different PPB objectives and
lower incomes than outpatients not participating in the study. These
findings suggest that the outpatients participating in the study at
intake were from a higher socio-economic class and were less dis-
turbed than those outpatients not participating. Feedback from the

staffs administering the questionnaires further supports this

52

Table 4.--HPRS Scores of Subjects Completing and Not Completing A
BSI at Intake.

 

 

BSI Completed BSI Not Completed t-Test
Outpatients V
Males 3.6 (n=105) 3.6 (n=5) -
Females 3.4 (n=202) 3.5 (n=l3) __
Inpatients 4.44 (n=116) 4.72 (n=57) - -o.94
(p=.35)
Males 4.04 (n=45) _ 4.86 (n=28) -1.78
(p<.05)
Females 4.69 (n=71) 4.59 (n=29) 0.26
(p=.79)

 

conclusion; they reported often not administering the intake ques-
tionnaire to patients who seemed particularly upset or in a crisis
state. ‘

The information regarding the representativeness of the
inpatient sample at intake is contradictory. Although the staff
administering the questionnaires reported that the inpatients par-
ticipating in the study were the better adjusted, less disturbed
portion of the inpatient sample and were not representative of the
entire inpatient population, the measures of initial disturbance
level are ambiguous.

One measure of previous mental health service revealed sta-
tistically significant differences between the inpatients partici-
pating and not participating in the study (see Table 3). The par-

ticipating group had a greater percentage of persons with previous

53

hospitalizations. This finding is somewhat questionable, however,
as there was a high percentage of missing data (about 50 percent)
for the treatment objective and diagnosis variables.

The HPRS rating (see Table 4) of inpatients indicates the
males not participating were more disturbed than those participating.
This difference was statistically significant t(72) = -1.78, p <.05 .
Among the female outpatients, however, there were not significant
differences in the HPRS ratings. 8

On the basis of the intake measures of disturbance level of
inpatients, it appears that the male inpatients in the study were
less disturbed than the male inpatient population in general. The
female inpatients in the study do not appear to be a biased sample
of inpatients. However, the high percentage of missing data and the
statements of staff administering the questionnaires cast some doubt

on the validity of this conclusion.

ngresentativeness of Follow-Up Sample

 

The second issue is whether the subjects who provided follow-
up data were a representative sample of all outpatients and inpatients
beginning treatment at the participating units. Chi-square tests
were performed on the demographic and intake characteristics of out-
patients and inpatients who completed and did not complete follow-up
questionnaires (see Tables 5 and 6). Separate analyses were com-
pleted for the subjects completing three and six-month question?

naires.

 

 

54

,Table 5.--Characteristics of Outpatients Returning and Not Returning
Follow-Up Questionnaires.

 

3-MONTH FOLLOW-UP 6-MONTH FOLLOW-UP

 

CHARACTERISTICS Returned Did Not Returned Did Not
MHQ Return MHQ MHQ Return MHQ
Number of subjects 152* 465* 264* 351*
Sex: Male 26% 37%** 27% 39%**
Female 74% 63% 73% 61%

Annual Income:
Less than $4,000 or

Public Assistance 36% 38% 35% 40%
$4,000 - 11,999 25% 32% 31% 28%
$12,000 or over 40% 30% 34% 31%
Ethnic-Status:
White 93% 91% 92% 92%
Non-White 5% 9% 7% 8%
Marital Status:
Married 46% 38% 42% 38%
Not Married 54% 62% 58% 62%
Education:
High School or Less 58% 65% 62% 64%
College or Graduate 42% 35% 37% 35%
Employment:
Employed 57% 50% 53% 50%
Unemployed or Not
in Labor Force 43% 50% 46% 50%
Other 1% 0% 0% 0%
Age: 18 - 39 86% 80% 84% 80%
4O - 59 13% 18% 15% 19%
60 or Older 2% 3% 3% 1%
Previous Mental Health
Service:
None 55% 46% 50% 47%
Previous Inpatient 4% 14% 8% 13%

Other 41% 41% 42% 40%

55

Table 5.--Continued.

 

3-MONTH FOLLOW-UP 6-MONTH FOLLOW-UP

CHARACTERISTICS Returned Did Not Returned . Did Not
‘ MHQ Return MHQ MHQ Return MHQ

 

Clinician Rating of
Symptoms at Intake: (a) (n=93) (n=232) (n=152) (n=173)

None 13% 6% 9% 8%
Mild 42% 47% 43% 47%
Moderate 37% 40% 40% 39%
Severe 9% 7% 9% 7%

Client Report of
Symptoms at Intake: (b) (n=151) (n=367) (n=264) (n=254)

None 20% 18%** 14% 23%**
Mild 49% 41% 45% 41%
Moderate 30% 32% 36% 28%
Severe 1% 9% 5% 9%
PPB Objective:
Psychosocial 87% 77%** 80% 80%
Crisis Resolution 5% 12% 12% 9%
Rehabilitation 6% 9% 6% 9%
Maintenance 1% 2% 1% 3%

 

* . .
Sample sizes are lower where indicated.

* .
2 * Statistically significant differences between categories

(X , p <.05).

(a)Ratings are Global Pathology Index on HPRS (0-1 = none,
2-3 = mild, 4-5 = moderate, 6-8 = severe).

(b)Reports are General Severity Index on BSI (O-.99 = none,
1.00-1.99 = mild, 2.00-2.99 = moderate, 3.00-4.00 = severe).

56

Table 6.--Characteristics of Inpatients Returning and Not Returning
Follow-Up Questionnaires.

 

3-MONTH FOLLOW-UP 6-MONTH FOLLOW-UP

CHARACTERISTICS Returned Did Not Returned Did Not
MHQ Return MHQ MHQ Return MHQ

 

Number of subjects 44 157*. 43 158*
Sex: Male 30% 47%** 40% 44%
Female 71% 53% 61% 56%
Annual Income: (n=l8) (n=67) (n=20) (n=65)
Less than $4,000 or
Public Assistance 6% 40%** 20% 37%
$4,000 - 11,999 55% 28% 40% 33%
$12,000 or over 39% 31% 40% 31%
Ethnic Status:
White 86% 71%** 86% 72%
Non-White 5% 18% 9% 17%
Marital Status:
Married 35% 37% 34% 38%
Not Married 65% 62% 66% 62%
Education:
High School or Less 95% 75%** 82% 79%
College or Graduate 6% 25% 18% 22%
Employment:
Employed ' 27% 31% 37% 28%
Unemployed or not
in Labor Force 59% 55% 56% 56%
Other 14% 15% 7% 17%
Age:
18-39 62% 64% 75% 61%
40-59 27% 22% 19% 25%
60 or older 12% 14% 7% 15%
Previous Mental Health
Service:
None 27% 37% 28% 36%
Previous Inpatient 49% 48% 46% 49%

Other 24% 16% 26% 15%

57

Table 6.--Continued.

 

3-MONTH FOLLOW-UP 6-MONTH FOLLOW-UP

CHARACTERISTICS Returned Did Not Returned Did Not-
MHQ Return MHQ MHQ Return MHQ

 

Clinician Rating of

Symptoms at Intake: (a) (n=39) (n=129) (n=37) (n=l31)
None 5% 4%** 8% 3%**
Mild 5% 23% 8% 22%
Moderate 64% 37% 62% 38%
Severe 26% 36% 22% 37%

Client Report of

Symptoms at Intake: (b) (n=42) (n=72) (n=39) (n=82)
None 7% 18% 10% 16%
Mild 33% 32% 31% 33%
Moderate 31% 30% 33% 29%
Severe 29% 20% 26% 22%

 

* .
Sample sizes are lower where indicated.

**
2 Statistically significant differences between categories

(X , p <.05).

(a)Ratings are Global Pathology Index on HPRS (0-1 = none,
2-3 = mild, 4-5 = moderate, 6-8 = severe).‘ .

(b)Reports are General Severity Index on BSI (0-.99 = none,
1.00-l.99 = mild, 2.00-2.99 = moderate, 3.00-4.00 = severe).

For the outpatient follow-up sample (Table 5), significant
differences were found between outpatients returning and not return-
ing a questionnaire at three months on Sex, Previous Mental Health
Service, Client Report of Symptoms at Intake (the General Severity
Index on the BSI) and PPB Objective. These differences indicate
the three-month outpatient sample underrepresented males and persons

with (a) previous inpatient service, (b) a self-report of severe

58

symptomatic disturbance and (c) a PPB objective of crisis resolu-

tion or rehabilitation. Tests on intake characteristics of the six-
month outpatient sample revealed statistically significant differences
on only one variable--the client's report of symptoms at intake (the
General Severity Index on the BSI). This difference indicates the
six-month outpatient sample underrepresented persons reporting no
symptoms and severe symptoms and overrepresented those with mild

and moderate symptoms.

For the inpatient sample (Table 6), significant differences
were found between inpatients returning and not returning a question-
naire at three monthson Sex, Annual Income, Ethnic Status, Educa-
tion and Clinician Rating of Symptoms at Intake (Global Pathology
Index on the HPRS). These differences indicate the three-month
inpatient sample underrepresented men, persons with incomes below
$4,000, non-white persons, and persons with mild and severe symptoms
at intake, and overrepresented college-educated persons. For the
six-month inpatient sample, only one characteristic--Clinician
Rating of Symptoms at Intake--was significantly different for
responders and non-responders. This difference indicates the six-
month inpatient sample underrepresented persons rated by clinicians
as having mild and severe symptoms.

As summarized in Table 7, the sample of outpatients and
inpatients who completed pp§H_intake and follow-up questionnaires
does not appear to be entirely representative of the entire incoming
population. Both the outpatient and inpatient samples underrepre-

sent men and persons with severe symptomatology at intake. The

59

.mo. Va .pmmp mcmzcmiwco saw: umcwacmpmn we: moceope_cmmm .nouo: mmwzcmguo mmmpcaa
.Ampcmwueacw mom .mpcmPAmauzo “Fwy auspm meg ace mFQmepm cowacngoa m:PEoo:w asp we age momepemocmam

 

weepaazm mcm>mm

ace cFPE m:_>m; mm
mcmwowcwpo zn copes meow
-cma co “smegma cmPPmEm

As_mv we

mom

excuse pm wsopasxm
wem>mm use mEopaszm o:
mcwpcoame mcomcma new
mmFee eo pcmocma cmppmsm

cowpeozum mmmppoo ewe:
mcomcma can .mxmucw um
wsopasam mem>wm use GFAE
mcw>ec mm mcmwow:__o >3
vogue m:0mcma wee .mmwu
teeocwe .ooo.¢a zopmn
mmeoocw saw; mcomcoa

.mmpcs eo pcmocmg coppesm

Aamwv es

mom

mm>wpomnno

mas apcmEpmzmne Pewoom
tosoxmaa ages can excuse
pm macaqsxm mem>mm eo
mcwpcoamc mcomsma .mmow>
-cmm spFMm; qucms msoe>
twee saw; mcomema use
mmpme we accosmg smPPMEm

we; mmFme mmcomcma new;
uses eo accuses ngPeEm

muw>com cupcm; Fmpcms
mzow>mca wees mm:e_ow
i:_Fo an mmcwpmc moan
inesym_u sepqexm cmZoP

aux_p:cowewcmwm

we; Amweweccowu

immza mcwccapmc po:
mpomnnsm gee: emceasoo
cmgzv mmewm::o_pmmac
mcwcszpmc mpomwnzm

”moceeccowpmwac
vmmeQEoo sue: muomnazm

NON meeenesm e_eeme_m La embasz
"mezmce<azc

afiemmv o__

tux_ucmo_ew:mwm

we; Amme_mccowp

immza mcwcszpms no:
muommazm ;p_z cmemano
cmcsv mmcweccowpmmzc
mcwzespmc mpomnnzm

mm>wpomwno

mas epcmEpmanum choom
-osozmae mcoe .mo_>ewm
cupmm; Pmpcme mzow>mea
mmwp .mmsoo:_ cmcmw:

 

Aemev emu Aemmv Nm_ whammy mom ”mee_aeea_cme=e
cmwopaaoo saw; mpomnasm
A_e Nee Ape accenaam apneae_m Le cease:
“mezmHeaaeso

ececz-xem eceozieeeee exacec

 

.zczam xucsooiwep co mm04 apes eo mpomeem--.m mpnee

60

six-month sample shows less bias on a variety of demographic vari-
ables than the three-month sample. For the outpatients, this may
have been due to the reduced sample size at three months resulting

from a follow-up of only half of the consenting outpatients.

Validiry: Differentiation of Groups

 

The first four hypotheses deal with the extent to which the
BSI reflects the differences in symptomatic disturbance expected for
self-differentiated groups, i.e., inpatients, outpatients, and non-
patients.

Hypothesis 1: Irpatients will report greater symptomatic
disturbance than outpafTents at Tntake.

 

Analysis of variance tests confirmed that inpatients scored
significantly higher at intake on all global and dimension scores
than outpatients (see Table 8). A main effect was also found for
sex, with females scoring significantly higher than males. Signifi-
cant interaction effects between sex and inpatient/outpatient status
were found on three dimension scales (Somatization, Obsessive-
Compulsive and Psychoticism) and on two global scales (General
Severity Index and Positive Symptom Distress Index).

These results indicate that inpatients do report signifi-
cantly more symptomatic disturbance at intake than outpatients, but
this finding is stronger for females than male inpatients. Table 9
shows the breakdown of BSI intake scores for males and females. A
t-test comparing inpatient and outpatient scores by sex reveals that

female inpatients have significantly higher scores at intake than

61

Table 8.--Brief Symptom Inventory Intake Scores for Inpatients and

 

 

 

 

 

 

Outpatients.
F-TEST SCORES FOR
. . MAIN EFFECTS/
Inpatients Outpatients
n=121 n=518 INTERACTION
a Status/
Status Sex Sex
BSI Dimensions:
1. Somatization 1.18 .81 20.9* 11.0* 5.45*
2. Obsessive- ‘ ‘
Compulsive 1.79 1.37 17.8* 8.3 6.9*
3. Interpersonal '
Sensitivty 1.71 1.45 6.7* 19.8* 2.3
4. Depression 2.04 1.69 11.5* 17.8* 3.3
5. Anxiety 1.90 1.62 9.0* 22.9* 2.2
6. Hostility 1.44 1.17 8.2* 12.1* 2.1
7. Phobic Anxiety 1.18 .78 22.7* 11.2* 3.1
8. Paranoid Ideation 1.64 1.24 16.3* 7.5* 2.5
9. Psychoticism 1.65 ' 1.23 21.5* 13.2* 6.2*
10. Additional Items 1.92 1.41 25.1* 10.8* 2.9
BSI Global Scores:
1. General Severity
Index 1.63 1.27 23.7* 21.2* 5.8*
2. Positive Symptom
Total 32.93 29.46 8.8* 17.7* 0.4
3. Positive Symptom
Distress Index 2.42 2.11 23.6* 16.7* 3.7*
*
p .<.05

aStatus refers to inpatient versus outpatient status.

62

xpwczeeoo cage mucmwpmapzo ace Amo.uvav mmgoom canoe; >_p:mowewcmwm empem>me pmmpie
k.

.mpcmwpmgpzo can» mpcmwumacw soc Amo.uvav mmcoom segue:

.mucmuwmmc

x.
xpucmowewcmvm empmm>mc “mouth
1*

 

 

 

 

mm._ e~._ aae_.~ aaam.. aoo.~ e..m xueec macabm_o eoeaesm e>_peaea .m
m.ep N.ep aam.om aam.em am.mm e.a~ Payee eacaesm e>_cemoa .N
_m. mm. aaem.~ aaNP._ amm.P AN._ xeeec seeee>em _eeeeeu ._
"mmmoum u<mose Hmm
em. om. aamN.. tep_.P ama._ eN._ smec_caecsma .m
mm. mm. eeom._ aam_.~ aem._ mm._ eecbeeec eeoeaaaa .w
mm. 0.. item. same. awm.~ am. seeexe< cecaea .5
me. me. aamN._ aamo.P see.F m_.F see__pmoz .e
co. me. «ame._ aamm.. «N_.N _m._ see_xe< .m
mm. Ne. samk._ sawe.P amm.~ Ne._ ecwmmecaee .e
me. we. 44cm._ «amm.P amm.P _m.. sc_>weemeam Paeamebaeeeec .m
me. am. tame._ «amm._ a~o.~ mm._ e>em_=a5eo-e>emmemao .N
am. a_. item. may“. a_e._ mm. eeeeaneeeEOm .P
”monmzmzHo Hmm
Aspeuev Acmmuev Aeemuev Aekeuev Apkuev Aomuev
mmpmsmd mmpmz mm_msoe mmpmz mmpoemu mmpmz
mezmoemmm >eHz=zzou mpzmHe<ae=o mezmHe<azH

 

zpvczseou new mpcmwumgpao .mp:m_pma:H

.mpemcemom

so; moeoom wxmucH cam: agoacm>cH Ecuaezm ewwemi-.m epoch

63

female outpatients, but that male inpatients do not have signifi-
cantly higher scores at intake than male outpatients. This finding
was true for all global and dimension scores.

Hypothesis 2: Outpatients will rgport greater_§ymptomatic
disturbance at intake than non:pafTents.

 

 

This hypothesis was confirmed. As illustrated in Table 9,
outpatients scored higher on each of the BSI global and dimension
scores than the nonpatient community sample. These differences
were statistically significant (t-test, p<<.05) for both males and
females.

Hypothesis 3: At intake, irpatients will score higher on the

p§ychoticism scales (7-9) than ourpatients
and norpatients.

 

 

 

As shown in Table 9, inpatients scored significantly higher
than outpatients and nonpatients at intake on the scales reflecting
psychotic symptoms (Phobic Anxiety, Paranoid Ideation and Psy-
choticism). For the Psychoticism scale (#9), there was a signifi-
cant interaction effect between in/outpatient status and sex. This
interaction indicates that the difference between inpatients and
outpatients was greater for female than male inpatients.

Hypothesis_4: At intake, outpatients will score higher
on the neuroticism scales (1-6) than the
nonpatients.

This hypothesis was confirmed for both males and females.
Outpatients scored significantly higher on the dimension scales
reflecting neurotic symptoms (Somatization, Obsessive-Compulsive,

Interpersonal Sensitivity, Depression, Anxiety and Hostility) than

the nonpatients. Outpatients also scored higher than the

64

nonpatients on scales 7-9, the psychoticism scales. These differ-
ences were statistically significant (t-test, p‘<.05) for both males

and females on all dimension scales.

Validity: Concurrence Between Measures

 

Hypotheses 5-10 also address the issue of the BSI's con-
struct validity by examining the concurrence between the BSI and two
other measures of symptomatic disturbance, the Hopkins Psychiatric
Rating Scale (Brief Form) and the symptom scales of the Personal
Adjustment and Role Skills questionnaire.

Hypothesis 5: At intake and follow-up, inpatient self-report
of symptomatic disturbance will have a signifi-
cant, positive correlation with the significant

other's rgport of the patient's symptomatic
adeStment.

 

The correlation between inpatient scores on corresponding
symptom scales is shown in Table 10. The General Severity Index on
the BSI was compared with the Overall Symptomatic Disturbance score
on the PARS. Each of the dimension scales on the BSI that has a
comparable measure on the PARS was compared with its corresponding
scale on the PARS.

At intake, correlations between inpatient scores were gen-
erally quite low. None of the correlations were significant at the
.05 level. The most important relationship, that between the global
measures of symptomatic disturbance on the PARS and BSI (line 1)
was .23, indicating a positive but non-significant relationship.

At follow-up, the correlation between measures was higher

than at intake. Four of the seven correlations were statistically

 

65

.mo.v.a .c comcmmm
*

 

 

vac. mm. 4N. eaemmeeaeo
icowpeuwm< xnmexc< ownoga .N
mow. «mm. oo. cowmmmcawo
-:o_amppm< xpmwxc< .o
avm. mam. om. cowmmmcamo
icowpmp_m< eowmmmsaoo .m
mm.i kwm. m_.i gemEm>Po>cH xpw>wpwmcmm
chomcchmch FecomsmqemucH .v
ace. om. No. ecemzceeu Emecwbaeuxma .m
AMNCV AMHCV fiNPHCV
mm. mm. eo.- Az_eo mc_azv
xpmwxc< memex=< .N
cc. tam. mm. mocenczpmwo A.H.m.ov xmecH
ovumEouaezm Preco>o xuwcm>mm chmcmw ._
Amucv Asyncv Ammucv
mpcmepeacH mucmwumacH mycmwpmacm mm<a Hmm
a212OA4om azuzoobou m¥<pz~ mmpmom mcwvcoqmmceou
:ezozixfim :Hzozimmmzp

 

one so mmpmum mucmngsumwo owgwsooasxm :o museum pcmvumacH cmmzumm meowpepmccouii.op w—nep

.mm<a ucm Hmm

66

significant at the three-month follow-up (n=17). At the six-month
follow-up, in spite of a sample size decrease1 (n=8), four of the
seven correlations were significant. The global measures of sympto-
matic disturbance (line 1) were significantly correlated at three
months (r = .58, p <.05). The same correlation at six months was
positive but not significant (r = .40, p = .18), perhaps due to the
small sample size.

Correlations between intake and follow-up scores on the PARS
symptom scores were generally low, but significant (r = .38, p<<.05
for outpatients; r = .36, p<:.05 for inpatients). The low correla-
tions indicate significant others reported different levels of
symptoms at follow-up than at intake.

In summary, the inpatients' report of symptomatic disturbance
was not strongly related to their significant others' report of
symptomatic disturbance. The correspondence between reports was
greater at follow-up than at intake.

Hypothesis 6: At intake and follow-up, outpatient self-
report of pymptomatic disturbance wiTl have
a STgnificant, ppsitive correlation with the

significant others' report of the patients'
pymptomatic adjustment.

 

In general, this hypothesis was confirmed at intake and
follow-up. As shown in Table 11, the global measures of symptoms on
the BSI and PARS had significant, positive correlations of r.= .34,

r_= .29 and r_= .30 at intake, three and six months respectively.

 

1Sample size decreases resulted from the small number of
patients with completed BSI and PARS questionnaires.

 

67

.mo.v.a .L cemcmma
.1

 

 

saw. me. see. eaemmeeaaa
icowumawm< Aymec< ownogm .n
amm. 40m. ems. ee_mmecaeo
icowpmpwm< xpmexc< .o
amm. op. ame. cowmmmsamo
icowucp_m< co_mmmsama .m
mo. N_. 49.- eeesc>Po>eH spe>epemeem
chomcwasmucH pmcomcchmch .v
mmm. my. emu. cowmzecou Emwowpocuama .m
Avmncv Ammncv Ammucv Aapco mmpmzv
4mm. so“. aem. speexe< sce_xe< .N
«om. mom. aem. mocecczpm_c A.H.m.wv xmcem
owpmsopasxw ppmcm>c prcm>mm Poemcmw .F
Amppucv Aonncv Ammmucv
mucm_umap:o mpcmwumguzo mucmepeauzo mm<a Hmm
QDIZOAAOE nauseous» mx<pz~ magnum mcwvcoammesoo
:Hzozime :Hzoztmumzp

 

,.mm<m use

Hmm one :o mopeum mocmnezpm_o ovumsopaezm :o mmsoom ucmwpmapzo cmmzamm meowpmpmccou--.~_ m—ncp

68

At intake, six of the seven correlations were statistically
significant. At three months, three of the seven were significant,
and at six months, five of the seven were significant.

Hypothesis 7: At intake and follow-pp, the correlation
between the significant other's ratingyand
patient‘s selfjrgport of symptomatic dis-
turbance will be higher for outpatients
thangimpatients.

This hypothesis was confirmed at intake, when there was a
stronger statistical significance for the correlations between out-
patient scores than inpatient scores (see Tables 10 and 11). At
intake, almost all of the outpatient correlations were statistically
significant, whereas none of the inpatient correlations were sig-
nificant.

At follow-up, the correspondence between PARS and BSI
measures was about equally strong for outpatients and inpatients.
The global measures were significantly correlated for both inpatients
Lr(17) = .58, p_<<.01] and outpatients [r(70) = .29, p < .01] at
three months. At the six month follow-up, the global measures of
PARS and BSI were: [r(1l9) = .30, p < .01] for outpatients, and
[r(8) = .40, p_= .17] for inpatients. Although the correlation
was higher for inpatients, the difference failed to reach signifi-
cance, possibly due to the small sample size (n=8) for inpatients.

Hypothesis 8: At intake, the self-report of outpatient

pymptomatology will be positively correlated
with the clinician's rating of symptomatology.

 

This hypothesis was generally confirmed. The corresponding

global measures of symptomatology--the General Severity Index on

 

69

'the BSI and the Global Pathology Index on the HPRS--were correlated
at statistically significant levels for both males Lr(106) = .35,
p <<.05] and females [r(215) = .40, p_< .05].

As Table 12 illustrates, the dimension scores on the HPRS
and BSI were significantly correlated for males on all the dimensions
but one, Obsessive-Compulsive, which was correlated at the .07 level.
For females, six of the nine dimensions were significantly corre-
lated; the Obsessive-Compulsive, Paranoid Ideation and Psychoticism
scales had positive, but not significant, correlations.

Hypothesis 9: At intake, the self-report of inpatient

symptomatology will be positively correlated
with the clinician‘s rating of symprpmatology.

 

This hypothesis was generally confirmed for female inpatients,
but not for male inpatients. (The global measures on the HPRS and
BSI were significantly correlated for females [r(66) = 21, p_:<.05],
but not for males [r(44) = .16, p <<.05].

The dimension scores showed a somewhat erratic pattern of
correlations for male inpatients (see Table 12). Three of the nine
scales (Somatization, Depression and Anxiety) had significant posi-
tive correlations. Hostility had a significant negative correlation
Lr(44) = -.33, p_:<.051; the other scales had low positive and
negative correlations.

For female inpatients, six of the nine scales had signifi-
cant positive correlations, with Paranoid Ideation, Psychoticism

and Phobic Anxiety correlating around the p = .10 level.

7O

.mc.v a

a.

.oxcu:_ um upco.m_:.scc mo_comc

 

 

II III TIIIIIII I 1.11111 11 I

 

4.. m.. 4.. ... 444:. a..eb>bm
~cemcoui_mm
..N. ..4 4.. 3.4 .am. m.m .mm. 4.4 x444. 44c.ae.as
.aaa.e-mea: .o.
4.. m.. N.. . ... .we
4.. 4.. 4a.- 4.. ea. 4.4 .m.. 4.4 mza: 54.6..azbsme .4
4.. 4.. N.. ... .ma ee.eaae_
... ... 4o. m.. mo. 4.4 .mm. ... mes: 4.aeacaa .4
4.. m.o w.o ..o .m: seu.xe<
N.. o.. N.. ..o .mm. m.o .om. 4.0 mes: c.2aea ..
... N.. N.. o.. .mm
.mm. 4.. .mm.- ... «4.. 4.. 44.. m.. wee: so....ma: .4
N.N 4.. ... 4.. .me
.34. 4.4 «mm. e.~ .mm. m.~ «an. m.~ mes: see.xe< .m
4.N ... ... m.. .ma
..m. ..m ..m. ..N .... 4.N .44. ..N was: ea.mmacabo .4
o.~ m.. . m.. m.. .mn 43.4.3.4:am
..m. 4.N 4.. e.~ «mm. 4.N .Nm. 4.N wee: .aeencbacbce. .m
..N 4.. 4.. m.. .me b>.m.saeao
.em. 4.. ... 4.. 4o. ~.. 4.. ... mes: -b>.mmcmao .N
4.. 4.o m.o ..o .mm
.mm. 4.. «mm. ... .O4. 4.. ,mm. ... was: eo.ca~.basam ..
u=w4owmemou «Loom u:o_o_eumou wcoom u:m_o_eemou mcoom a:m_u_4emou ocoom
copua_occou cam: :a_um_mgcou com: comuapmecou :coz cowaopoccou cam:
mm—com .mm uzm
.eene. .44nev m.~uev .mo.ue.
mm4c5ou mm.az Mm—cEmm mm.cz mam: a:.e:oammceou

mpzu_h<mh:o

l." . 1:41 l‘ul.rl|l

 

..lil . 1|N111 . l.-. . u 11.. I i .I .1 1.5!..- 11» I'll! I11

m42m_h<mz_

II II-

c.mm_aom .mm ace mxe: a=.v:oammccou coagumm m=o_ua.mccou vac

”ocean :eozii.m. b.2e4

71

Hypothesis 10: At intake, the correlation between the clini-
cian's rating and the patient's self-rpport
of symptomatic disturbance will be higher for
outpatients than inpatients.

 

 

 

 

 

This hypothesis was confirmed for the male subjects in the
study. As described above, the correlations between the two scales
were consistently positive and statistically significant for male
outpatients. However, male inpatients had erratic positive and
negative correlations which generally were not statistically signi-
ficant.

For female subjects, the correlations between the General
Severity Index:(BSI)and the Global Pathology Index (HPRS) were
slightly higher for outpatients Lr(215) = .29, p_:<.051 than for
inpatients Lr(66) = .21, p_< .051.‘ However, this difference was not
statistically significant (p = .72). This finding, and that of an
equivalent number of dimension scales being significantly correlated
for both inpatients and outpatients, suggests that the hypothesis
was not confirmed for female subjects.

Thus far, the BSI appears to satisfy several of the validity
criteria for outpatients and nonpatients. Outpatients had signifi-
cantly higher scores than nonpatients on all of the BSI scales.
Outpatient reports of symptomatology were significantly correlated
with clinicians' ratings of symptomatology at intake. They were
also significantly correlated with the significant others' report
of clients' symptoms at intake, three- and six-month follow-up.

The BSI meets some, but not all, of the validity tests for

inpatients. Inpatients had higher scores on all of the BSI scales

72

than outpatients, but the difference was statistically significant
only for females. The self-report of symptoms was significantly
correlated with clinicians' ratings for females but not for males.
Inpatient BSI scores were not correlated with significant others'
ratings at intake, but had many significant correlations at three-
and six-month follow-up. The BSI appears to meet certain validity

for female but not male inpatients.

Sensitivity of BSI to Change

 

Hypotheses 11 and 12 examine the relationship between BSI
scores at intake and follow-up. The central question is whether the
BSI is sensitive to change in an outpatient and inpatient population
between intake, three and six months.

A balanced design Analysis of Variance program1 was used to
examine these hypotheses in order to provide simultaneous tests of
three main factors--time, patient status, and sex. The analyses
required that all subjects have BSI Scores at intake, three and six
months. Thus, the sample size is significantly reduced from that
in previous hypotheses where only BSI scores at intake were examined.
The sample size for the predictive validity hypotheses was 27
inpatients and 113 outpatients.

Hypothesis 11: Ippatients will rpport greater symptomatic

disturbance at intake than atTIthree or six
monthsleollow-up,

 

 

1BALANOVA User's Manual (Frankmann, l978).

73

Hypothesis 12: Outpatients will report greater symptomatic
disturbance at intake than at (three or six
months) follow-up.

 

These hypotheses were confirmed for inpatients and outpatients.
Significant main effects between scores at intake, three and six
months were found for inpatients and outpatients on all global and
dimension scores except the PST, Positive Symptom Total (see Table
13). As Table 13 illustrates, inpatients and outpatients had
significantly lower BSI scores at three- and six-month follow-up
than at intake. The reduction in symptom disturbance was most sub-
stantial between intake and three months, although symptoms did
continue to decrease between the three and six-month follow-ups.

When male and female scores are analyzed separately for out-
patients and inpatients, the women's scores changed more than the
mens. Females reported more symptoms at intake than males, and also
reported more change at follow-up than males. The different rate of
change for females was reflected in the finding of significant inter-
action effects (p <.05) between sex and time on three of the four
global scales (the General Severity Index, Grand Total and Positive
Symptom Total) and interaction effects approaching significance
(p<<.10) on the dimension scores for Interpersonal Sensitivity,
Depression, Anxiety and Psychoticism. These findings indicate that
the BSI is more sensitive to change among females than males,

rpgardless of their patient status. Most of the BSI scales reflected

 

a pre- and post-treatment difference in symptoms for females. This
finding was not true for males, for whom three of the global scales

do not reflect significant reduction in symptoms after treatment.

74

No significant main effects were found between inpatients
and outpatient scores for this sample of inpatients (n=27). This
finding is in contrast to the significant difference between inpatient
and outpatient scores at intake in testing Hypotheses 1-4. A larger
sample size (121 inpatients and 518 outpatients) was used to test
Hypotheses 1-4. The larger sample size included more symptomatic
inpatients (GSI = 1.63 for 121 inpatients, compared to GSI = 1.49
for this sample of 27 inpatients). As discussed in the section on
Data Loss, when the inpatient sample is limited to only those sub-
jects with a 851 at intake, three and six months, some of the more
disturbed inpatients are eliminated from the sample. Thus, the sample
of inpatients available for this analysis is only a small portion of
the original inpatient population; the generalizability of these

findings are, therefore, quite limited.

75

Table 13.--ANOVA of BSI Scores--At Intake, Three and Six Months.

 

INPATIENTS OUTPATIENTS

Intake Three Six Three Six u

Months Months Intake Months Months

 

General Severity Index

 

Males 1.17 .91 .71 1.10 .95 .84
(n=11) (n=27)

Females 1.81 .81 .60 1.22 .84 .74
(n=16) (n=86)

Total 1.49 .86 .65 1.16 .89 .79
(n=27) (n=113)

Significant Main and Interaction Effects: Time (

r(2)=14.9o, p=.OO)
Sex/Time (f(2)=

3.00, p=.05)

 

Grand Total

 

Males 61.0 48.00 37.64 58.19 49.96 44.48
Females 94.31 43.25 31.63 63.86 43.98 39.06
Total 77.66 45.63 34.63 61.02 46.97 41.77

Significant Main and Interaction Effects: Time (f(2)=14.67, p§.00)
Sex/Time (f(2)=2.94, pf.05)

 

Positive Symptom Total

 

Males 26.36 26.00 23.33 23.85 27.14 25.56
Females 35.00 24.17 24.44 27.20 22.70 23.64
Total 30.68 25.08 23.89 25.52 24.92 24.60

Significant Main and Interaction Effects: Sex/Time (f(2)-2.85, p=.05)

 

Positive Symptom Distress Index

 

Males 1.96 1.65 1.41 1.96 1.75 1.78
Females 2.49 1.82 1.55 2.02 1.68 1.56
Total 2.23 1.73 1.48 1.99 1.72 1.67
Significant Main and Interaction Effects: Time (fl2)=14.23, p?.00)
Sex/Time (E(l)=4.56, p?.03)

 

76

Table 13.--Continued.

 

 

 

INPATIENTS OUTPATIENTS
Three Six Three Six
Intake Months ‘Months Intake Months Months
Somatization

Males .61 .53 .42 .56 .53 .36
(n=11) (n=27)

Females 1.11 .56 .36 .72 .56 .49
(n=16) (n=86)

Tota1 .86 .55 .39 .64 .54 .42
(n=27) (n=113)

Significant Main and Interaction Effects: Time (f(2)=5.20, p?.01)

 

Obsessive-Compulsive

 

Males 1.19 .93 .91 1.24 1.14 .90
Females 1.91 .90 .73 1.33 1.00 .89
Total 1.55 .91 .82 1.29 1.07 .89

Significant Main and Interaction Effects: Time(§(2)=7.82, p?.00)

 

Interpersonal Sensitivity

 

 

 

 

 

Males 1.20 1.14 .80 1.29 .99 1.09
Females 2.11 .09 .81 1.42 .98 .86
Total 1.66 1.06 .80 1.35 .99 .98
Significant Main and Interaction Effects: Time (f(2)=8.28, p?.00)
Depression
Males 1.61 1.41 .88 1.56 1.28 1.28
Females 2.45 1.20 .80 1.64 1.10 .93
Total 2.03 1.30 .84 1.60 1.19 1.11
Significant Main and Interaction Effects: Time (f(2)=13.28, p=.00)
Anxiety
Males 1.46 1.00 .89 1.36 1.10 .99
Females 2.12 .89 .73 1.65 1.01 .95
Total 1.79 .94 .81 1.50 1.05 .97

Significant Main and Interaction Effects: Time (f(2)=l4.39, p;.00)

 

77

Table l3.--Continued.

 

 

 

INPATIENTS OUTPATIENTS
Three ' Six Three Six
Intake Months Months Intake Months Months
Hostility
Males 1.30 1.16 .76 .99 ' .97 .91
(n=ll) (n=27)
Females 1.43 .74 .59 1.15 .78 .71
(n=16) (n=86)
Total 1.37 .95 .68 1.07 .87 .91
(n=27) (n=113)

Significant Main and Interaction Effects: Time (f(2)=6.03, p?.00)

 

Phobic Anxiety

 

 

 

 

 

Males .68 .49 .40 .76 .59 .54
Females 1.36 .55 .38 .68 .41 .44
Total 1.02 .52 .39 .72 .50 .49
Significant Main and Interaction Effects: Time (f(2)=9.68, Ef.00)
Sex/Time (5(1)=4.27, pe.04)
Paranoid Ideation T
Males 1.22 .75 .76 1.00 1.02 .75
Females 1.71 .69 .63 1.20 .95 .79
Total 1.47 .72 .69 1.10 .98 .77
Significant Main and Interaction Effects: Time (f(2)=8.76, p?.00)
Psychoticism
Males 1.27 .95 .75 1.21 .97 .90
Females 1.85 .76 .38 1.20 .77 .67
Total 1.56 .85 .56 1.21 .87 .79

Significant Main and Interaction Effects: Time (f(2)=14.37, pf.00)

 

78

Table 13.--Continued.

 

 

 

INPATIENTS OUTPATIENTS
Three Six Three Six
Intake Months Months Intake _ Months Months
Additional Items
Males 1.23 .80 .55 1.10 .91 .77
Females 2.19 .95 .63 1.30 .86 - .69
Total 1.71 .87 .59 1.20 .89 .73

Significant Main and Interaction Effects: Time (f(2)=16.19, p;.00)

 

 

DISCUSSION

0n the basis of prior research, three criteria were estab-
lished as the basis for examining the construct validity of the BSI:

1. scores will differentiate between inpatients, out-

patients and community residents;

2. scores had concurrence with other measures of symptoms;

and,

3. scores were sensitive to pre- and post-treatment.

differences in symptoms.

The self-report symptom scores collected on a sample of out-
patients and inpatients prior to treatment and at follow-up were
measured against these criteria. The criteria were largely satis-
fied for the outpatients, but the findings were more ambiguous for
the inpatient sample. These results suggest the BSI has construct
validity for an outpatient, but not an inpatient population. This
section reviews these findings, discusses alternative theoretical
explanations for them, and finally relates them to the larger issue
of designing an evaluation system for measuring treatment effective-

ness of mental health programs.

Generalizability of Results
In considering the validity tests in this study, it is

important to clarify the extent to which these findings are

79

8O

generalizable to the range of inpatients and outpatients typically
seen at public mental health facilities. Clients participating in
this study were a sample of all outpatients and inpatients beginning
treatment during a three month period at several outpatient and
inpatient units. However, the sample of subjects completing BSI
questionnaires was not entirely representative of the inpatient

and outpatient populations at these centers. As described in the
RESULTS section, the subjects participating in this study were less
disturbed at intake and had higher socioeconomic characteristics
than the agencies' populations. These biases resulted from the more
disturbed and lower socioeconomic persons not completing question-
naires at intake and follow-up. Thus, the results of this study

are not generalizable to the participating agencies‘ ”typical“ out-
patient or inpatient populations. Rather, these findings regarding
the 881's validity apply to a somewhat healthier and higher socio-
economic class population than is usually seen at an outpatient or

inpatient unit.

Tests of Construct Validity
For the outpatient population, the BSI appears to meet
several of the criteria for construct validity. Outpatients scored
significantly higher on all BSI scales than the community residents.
The outpatients' self-report of symptoms was significantly corre-
lated with the clinicians' rating of symptoms on an analogue scale,

the HPRS. In addition, the BSI ratings were significantly

81

correlated with the significant others' rating of symptoms on the
PARS. This finding held up at intake and both follow-up periods.

Although the BSI had strong validity properties for an out-
patient p0pulation, these findings did not hold up for an inpatient
population. For the inpatients, validity of the BSI was established
in part for females, but not for male inpatients. When outpatient
and inpatient BSI scores were compared, BSI scores were significantly
higher for inpatients than outpatients. When sex was controlled for,
however, this finding held up for women and not men. Thus, female
inpatients reported significantly more symptomatology than female
outpatients, but the male inpatients did not report significantly
higher symptoms than male outpatients.

A similar sex difference was found in the relationship
between inpatients' BSI scores and clinicians' ratings. The scores
were significantly correlated for female inpatients but not male
inpatients. Thus, the correspondence between clinicians' ratings
of symptoms and patients' self-reports was greater for female
inpatients than male inpatients. These two findings suggest the BSI
is a valid instrument for female inpatients, but not male inpatients.

The correspondence between the inpatients' self-report of
symptoms and their significant others' ratings was quite low,
especially at intake. The comparisons of global BSI and PARS
symptom scales revealed positive but nonsignificant correlations
at intake and follow-up. Only the correlation at three-month follow-
up achieved statistical significance. The small sample size for this

analysis undoubtedly contributes to the low Significance level of

82

these correlations. Unfortunately, the small sample size precluded
separate analyses for male and female inpatients. Thus, it is not
known whether the lack of validity established earlier for male
inpatients was also true in comparing inpatients' BSI and PARS
scores.

To determine whether the BSI is sensitive to pre-post treat-
ment change, the difference between scores at intake and follow-up
was analyzed for those outpatients and inpatients who completed an
intake, three and six-month follow-up questionnaire. When analyzed
together, inpatients and outpatients had significantly lower scores
at follow-up than intake. When sex and patient status were con-
trolled for, however, the analysis revealed different patterns of
change for males and females, outpatients and inpatients.

Although inpatients and outpatients as a group had signifi-
cantly lower symptomatology at follow-up than at intake, there was
a significant interaction effect between sex and time on most of the
global scales. Females had a greater pre-post treatment reduction
in symptoms than males, regardless of patient status. This finding
may have been due to the women having higher intake scores than men,
thus allowing for a greater reduction in symptoms at follow-up.
These results suggest once again, however, that the BSI is a more
valid instrument for females than males.

The inpatient sample1 used in this analysis did not have

significantly higher intake scores than the outpatient sample,

 

1Due to the requirement that only subjects with complete data
sets (i.e., intake, three and six-month questionnaires completed) be

83

indicating this inpatient sample was an even less disturbed sample
than that used in the first two tests for construct validity. Given
this, the finding of insensitivity to pre-post treatment differences
is generalizable only to the least disturbed portion of the inpatient
population, i.e., those inpatients most closely resembling outpatients
on intake measures of symptomatic disturbance.

In summary, the BSI meets moSt of the established tests for
construct validity for the outpatient sample, thus suggesting it
may be an adequate instrument for measuring outpatients' symptoms
before and after treatment. The BSI does not appear to be sensitive
to pre- and post-treatment differences in symptoms for male out-
patients, however, and thus may not be valid for this population.

If the BSI is used to evaluate outpatient program effectiveness,
caution should be used in generalizing any findings to an entire
outpatient population. The most disturbed members of the population
may be unwilling or unable to complete the BSI, thus limiting the
external validity of the evaluation results.

This study indicates the BSI is not a valid instrument for
measuring inpatient symptoms before and after treatment. The major
problem in using the BSI is the high percentage of data loss,
resulting in only 20 percent of the population completing pre- and
post-questionnaires. This sample appears to be the less disturbed

members of the inpatient population.

 

used for this analysis, the sample size was significantly reduced
from that used in the previous analyses. This was especially true
for the inpatient sample. which was reduced to 11 males and 16
females.

84

For those inpatients who do complete the BSI, it appears to
be a valid measure of symptoms for females but not males. Males do
not report greater symptoms than their outpatient counterparts, nor
do they report a reduction in symptoms after treatment. They also
report themselves as being less symptomatic than do their clini-
cians. Thus, the BSI fails all of the tests for construct validity
for male inpatients. By contrast, it meets most of the tests for
female inpatients. The only exception to this was a lack of cor-
respondence between self and significant others ratings at intake.
The small sample size of inpatients available for this analysis
limits the strength of these findings, and suggest the need for

further testing with a larger sample size.

Alternative Theoretical Explanations
This study found clear differences in men and womens'

responses to the BSI. Females consistently report greater symptoma-
tology than males; this occurred among all three groups of subjects--
inpatients, outpatients and community residents. The BSI failed to
meet any of the criteria for construct validity for male inpatients,
and yet, satisfied most of the criteria for female inpatients. This
section considers alternative theoretical explanations for these

findings.

Interpretation of Sex Differences
Other studies of self-report inventories have also found sex

differences, usually in the direction of women reporting more

85

symptoms than men. Many scales, including the SCL-90 and the MMPI,
have incorporated different norms for men and women in their scoring
systems (Derogatis, 1977; Welsh and Dahlstrom, 1956). Several
studies on different MMPI scales have shown women (including clients
and normals) reporting more symptoms than men, particularly anxiety
and depression (Welsh and Dahlstrom, 1956). In their extensive
review of psychological studies of sex differences, Maccoby and
Jacklin (1974) found consistent evidence of greater anxiety among
females than males; these studies were conducted on non-patient
adults and children. ‘

Maccoby and Jacklin (1974) question whether women are
actually more symptomatic than men. They point out that the scales
used in these studies were not validated against behavioral or phy-
siological measures of anxiety. They also cite some evidence that
males tend to be more defensive than females, and have higher "lie"
scale score (Sarason, 1960), and suggest that girls may be condi-
tioned to admit weakness more readily than boys.

Clearly, more research is needed to determine whether the sex
differences found in this study and others reflect true differences
in the level of symptomatology between men and women. To the extent
this study has used independent, objective measures of symptoms (see
Discussion section, Concurrence Among Informants), the evidence on
this question is mixed. For the outpatient sample, no significant
differences occurred between men and womens' symptoms either in the
clinicians' or significant others' ratings. And yet, outpatient

females rated themselves as more symptomatic than did the male

86

outpatients. Among inpatients, females rated themselves as more
symptomatic than males, as did also the clinicians and significant
others.

Several problems arise with this analysis. First, the com-
parisons are made between ratings on different sets of subjects (see
Tables 10, 11 and 12). Secondly, inter-rater reliability of HPRS
ratings was not established across clinicians participating in this
study. And, finally, the lack of concurrence among different raters
and the low absolute values of the correlations between measures
must be considered. Definitive evidence that women are truly more
symptomatic than men would have to be shown in a study designed
more particularly to address this issue than the current study.

If women seeking treatment are more symptomatic than men,
the issue becomes whether an instrument such as the BSI should be
used to evaluate the effectiveness of treatment programs. Men may
be seeking treatment for reasons other than severity of symptoms;
indeed, results of this study indicate male inpatients report sig-
nificantly more social functioning problems (e.g., handling their
occupations) than either male outpatients or female inpatients (see
Appendix A). Social functioning problems may be the more salient
factors for men seeking inpatient treatment, and symptomatic dis—
turbance more salient for women. More research is necessary to
address this issue, but current information indicates the BSI alone

is not an adequate evaluation instrument, particularly for men.

87

MeasuringHChangg_

 

Analysis of the change in symptoms from intake to follow-up
indicates women change more than men. 'This was true for outpatient
women, as well as inpatient women. The question arises whether this
finding was a statistical artifact resulting from the higher intake
scores of women. Greater intake scores would allow for more change;
this might occur as a regression to the mean, the tendency of deviant
scores to move closer to the mean on second measurement.

Campbell and Stanley (1963) described regression to the mean
as a serious threat to validity of an evaluation study not using
equivalent control groups. They suggested, however, that regression
to the mean is more likely to occur when groups are selected for
analysis on the basis of their deviant scores.

Regression effects are thus inevitable accompaniments of
imperfect test-retest correlation for groups selected for
their extreme scores whenever encountered. If a group
selected for independent reasons turns out to have an
extreme mean, there is less a priori expectation that the
group mean will regress on a second testing, for the
random or extraneous sources of variance have been
allowed to affect the initial scores in both directions
(p. 11, Campbell and Stanley, 1963).

In this study, groups were not selected for analysis because
of their extreme scores, and thus, the expectation of a significant
regression-to-the-mean factor should be lower. There is substantial
evidence, however, that the amount of change reported is directly
related to the initial scores. Several studies have reported this

and recommend adjusting for the portion of change that is a function

of the initial level of adjustment (Ellsworth, 1975; Meltzoff and

88

Kornreich, 1970). The "best" method for this adjustment is still
being debated (Kenny, 1975;Nunnally,l975; Cronbach and Furby, 1970)
and is discussed more thoroughly in the Literature Review.

One frequently recommended technique for controlling for
initial scores is analysis of covariance (COVARAN). This study did
not use COVARAN to partial out the effects of pretest differences on
posttest differences. Nunnally (1975) recommended this technique
not be used in quasi-experimental designs because

(a) COVARAN does not control for variables which correlate

with both the pretest and posttest variables;

(b) a cardinal assumption regarding equivalent within-

groups regression slope is frequently violated; and

(c) there is disagreement about how to adjust statistical

parameters of COVARAN to take account of measurement
error.

The approach recommended by Nunnally (1975) is to examine
the interaction of treatment conditions with comparison groups,
using a within subjects, repeated measurement factor. Statistical
tests are then performed on all main and interaction effects to
determine the significance of the treatment effect and how it inter-
acts with different groups of subjects over time. This approach was
utilized in the present study, although other methods of examining

change should be undertaken in future studies.

89

Treatment Effectiveness

 

Related to the issue of measuring change is whether or not
change actually did occur. A plausible alternative explanation for
the findings of less change among males than females is that treat-
ment was less effective for men than women.

In their review of psychotherapy research, Meltzoff and
Kornreich (1970) concluded

There is no clear relationship between the sex of the
patient and the outcome of psychotherapy. Most studies
report no differences. It is typically an incidental
finding rather than one upon which research has focused.
Much of the data comes from surveys with fallible criteria
measures and uncertain experimental conditions (p. 236).

On other measures of change in the present study, there
is conflicting evidence whether males change less than females. In
responding to a client satisfaction questionnaire at follow-up,
males and females had similar responses to questions about how much
they had changed as a result of treatment (see Appendix B). Ratings
on measures of social functioning indicate a substantial portion of
men improved (4O - 50 percent, depending on the informant) in their
social functioning after treatment. Further analysis of this issue
is needed, using composite measures of change (e.g., analyzing
change on symptomatic disturbance and social functioning and client

satisfaction measures) to determine if men seek treatment and benefit

from treatment in different ways than women.

Concurrence Among Informants
Evidence on the correspondence among different raters (self,

clinician and other) indicates greater agreement exists among

90

measures of symptoms for outpatients than inpatients. Although
statistically significant, the correlations were low and once again
suggest more correspondence between the BSI and other measures for
females than males.

The ambiguous results of this study are not surprising,
given that other studies of the relationship between patient, thera-
pist and significant other ratings have had mixed results. Carr and
Wittenbaugh (1969) found little agreement among patients, families
and therapists about treatment effectiveness. By contrast, two
other studies (Ellsworth, 1975; Fontana and Dowds, 1975) found sig-
nificant correlations between patients' and significant others'
ratings at intake and follow-up. In both studies, the correlations
were significantly higher at follow-up (range from .50 to .69) than
at intake (.20 - .22). It is interesting to note the similarity
between these studies and the present in their findings of stronger
relationships between ratings at follow-up than at intake. This
indicates patients and their significant others have more discrepant
views of the patients' symptomatology when the patient enters the
hospital than when he has completed treatment.

There were two main differences between the present study
and those by Ellsworth and Fontana and Dowds. Their studies had over
200 subjects, thus possibly rendering the low intake correlation
statistically significant (an equivalent correlation in the present
study was not significant). In addition, their studies used the
same scale (PARS) for both the patient and the other. Their use

of the same scale, and the possible similar response set this would

91

engender, might lead to a greater correlation between patient and
others' report of symptoms than was found in the present study.

In summary, several theoretical arguments may be raised as
alternative explanations to the findings in the present study. Sex
differences on the BSI may be due to mens' tendencies to experience
and/or report fewer symptoms than women. Findings of greater change
among women may occur because the measurement of change has not con-
trolled for womens' higher intake scores. On the other hand, men may
change less because the treatment is less effective for them, or
because social functioning problems are more salient factors in their
treatment than symptoms. Different measures and different informants
also raise questions about which are most appropriate for what types
of analyses.

Further research is needed to answer definitively the above
questions. The evidence in the present study is that the BSI meets
tests of construct validity for outpatients but not for inpatients,
particularly for male inpatients. The reasons for this may be
embedded in the above theoretical positions; the issues are con-
founded with one another and need further exploration. For whatever
reasons the BSI fails certain tests of validity, its use as an

evaluation instrument with inpatients must be seriously questioned.

The Evaluation System
The adequacy of a state-wide evaluation system based on a
self-report instrument should be assessed separately from the

validity of the questionnaire. The data collection procedures and

92

instruments used in this pilot study together comprise a gygpgm

for evaluating treatment outcomes. Other procedures and instruments
have been pilot tested in other counties in Michigan (Clifford,
1978;Ihlevich, 1979) and will be compared with results of this
pilot to develop a possible state-wide strategy for evaluating
mental health services in Michigan. .

The extent of data loss in this study raises questions about
the adequacy of an evaluation system built on a self-report instru-
ment. About 40 percent of the inpatients and 20 percent of the out-
patients did not complete intake questionnaires. One report on this
project by the Urban Institute (Schainblatt, 1978) suggests the con-
sent rates might be improved by having better trained data collectors
in the outpatient units and by having more staff resources committed
to collecting data in the inpatient units. Rates might also be
improved if agencies had more of an incentive to have clients com-
plete questionnaires. In the current study, many of the staff par-
ticipating in the study did not believe the information being col-
lected would be useful to them, and cooperated with the study pri-
marily because it was mandated by the Department of Mental Health.

These modifications might result in somewhat higher question-
naire completion rates at intake. The fact remains, however, that
large portions of the client populations, especially among inpatients,
will be unable or unwilling to complete questionnaires about their
psychological and social functioning prior to beginning treatment.
The experience in this pilot also indicates many clients are unwilling

or unable to complete follow-up questionnaires as well. Only

 

93

one-fourth of the inpatients and one-half of the outpatients eligi-
ble for this study completed both intake and follow-up question-
naires. The additional problem of lack of instrument validity for
those inpatients who do complete the questionnaire argues against
the sole use of a self-report model to evaluate mental health ser-
vices, especially for the inpatient programs.

The self-report model for evaluating mental health programs
for outpatients does appear to have validity for the healthier, less
disturbed portion of the outpatient population. If care is taken
not to generalize the results of the evaluation to the entire out-
patient population, the information provided with the self-report
model does appear to be useful. In selecting a state-wide eval-
uation system, however, it is important to be able to compare the
effectiveness of different types of programs with the entire range
of clients using the services. When the primary objective of public
mental health programs is to find cost-effective alternatives to
inpatient hospitalization (i.e., outpatient, aftercare and partial
hospitalization programs), the evaluation system should allow com-
parisons across these programs. Because of its limited applicability
to more seriously disturbed inpatients and outpatients, the self-
report model would not provide for these inter-program comparisons.

An evaluation system based on a clinician's rating of psy-
chological and social functioning constitutes a viable alternative
to a self-report model. Having clinicians complete standardized

assessment instruments at intake and discharge would insure complete

94

coverage of both an inpatient and outpatient popUlation. In a
separate paper, the clinicians' rating system is recommended as the
"preferred alternative" for a state-wide evaluation system (LaFerriere,

1979).

Recommendations for Further Research

 

In examining the validity of the BSI, several important
issues arose concerning the development of a system to evaluate
mental health services. In addition, the finding of sex differences
in the self-report of symptoms raised several interesting questions
for further research. Some of these issues might be explored with
the current data set; others would require additional data to be
collected.

First, the question around measuring change and treatment
effectiveness needs to be explored further. When the "follow-up"
data for the community norm sample (i.e., the repeat administration
of the BSI at a time concurrent with the six-month follow-up of
clients) becomes available for analysis, pre-post treatment differ-
ences for clients can be compared with change rates for a normal
sample. This quasi-experimental design utilizing a non-equivalent
control group will provide a more rigorous test of treatment versus
non-treatment effects.

Further analysis is needed to determine whether women are
more symptomatic than men, and how these differences are related to
seeking and benefitting from mental health services. A factor anal-

ysis exploring which of the social functioning and symptom

95

disturbance variables are most highly associated with male and
female treatment outcomes would be very useful.

Finally, the relationship among different informants'
ratings of symptoms and social functioning should be re-analyzed
using additional data collected during a second pilot phase of this
project at the Michigan Institute for Mental Health. The increased
sample size from this data set might permit analysis using only
those outpatients and inpatients who had self-ratings as well as

clinician and significant other ratings.

Conclusions

 

In pilot testing an evaluation system based on a client
self-report instrument, this study found the Brief Symptom Inventory
to be a valid instrument for use with female outpatients and
inpatients. For these populations, the BSI met three essential
criteria for construct validity--differentiation between groups of
inpatients, outpatients and nonpatients; congruence with other
measures of symptomatology; and sensitivity to pre- and post-
treatment differences in symptoms. By contrast, the BSI did not
meet the essential validity criteria for males. Male inpatients
did not report significantly more symptoms than male outpatients,
nor did either sample of men report a lessening of symptoms after
treatment. The correspondence between the BSI and reports of
symptoms by clinicians and significant others was greater for out-
patients than for inpatients, with the lowest correspondence

being for male inpatients.

 

96

The evidence from this study suggests that while the BSI may
be a valid instrument for use with outpatients, it is not valid for
use with inpatients. The construct validity of this scale is
strongest for female outpatients, and weakest for male inpatients.
The combined results of a high percentage of data loss and the
lack of instrument validity for males render this self-report
model inappropriate for evaluating the effectiveness of mental

health services.

BIBLIOGRAPHY

97

BIBLIOGRAPHY

Abeloff, M.D. and Derogatis, L.R. "Psychological Aspects of the
Management of Primary and Metastatic Breast Cancer."
Proceedings of the International Conference on Breast Cancer.
New York: ‘ATR. List, Inc., 1977.

Anastasi, A. P§ychological Testing, 3rd Edition. London:
MacMillah‘Company, ColTier-MacMillan Ltd., 1968.

Anthony, W.A., Buell, G.J., Sharrett, S., and Althoff, M.E.
"Efficacy of Psychiatric Rehabilitation.“ prchological
Bulletin 78 (1972): 447-456.

 

Arnoff, Franklin. "Social Consequences of Policy Towards Mental
Illness." Science 188 (1975): 1277.

Boleloucky, L., and Horvath, M. '"The SCL-90 Rating Scale: First
Experience with the Czech Version in Health Male Scientific
Workers." Activitas Nervosa Superior 16 (1974): 115-116.

Campbell, D.T., and Boruch, R.F. "Making the Case for Randomized
Assignments to Treatments by Considering the Alternatives:
Six Ways in Which Quasi-Experimental Evaluations in
Compensatory Education Tend to Underestimate Effects."
In Evaluation and Erperiment: Some Critical Issues in
Assessing Social Programs. Edited by C.A. Bennett and A.A.
Lumsdaine. New York: Academic Press, Inc., 1975.

Campbell, D.T., and Erlebacher, S. "How Regression Artifacts
in Quasi-Experimental Evaluations Can Mistakenly Make
Compensatory Education Look Harmful." In Compensatory
Education: A National Debate, Vol. 3 of The Disadvantaged
Child. Edited by J. Helmuth. New York: Brunner/Mazel,
1970. -

 

 

Campbell, D.T., and Stanley, J.C. Erperimental and Quasi-
Erperimental Designs for Research. Chicago: Rand McNally,
1963.

 

Carr, J.E., and Whittenbaugh, J. "Sources of Disagreement in the
Perception of Psychotherapy Outcomes." Journal of Clinical
ngchologyyZS (1969), 1: 16-21.

 

98

 

99

Clifford, D.L. "Implementing Outcome Evaluation in a Community
Mental Health Center." Paper presented at 86th Convention
of American Psychological Association. Toronto, 1978.

Craig, T.J., and Abeloff, M. "Psychiatric Symptomatology Among
Hospitalized Cancer Patients." American Journal of
Epychiatry 131 (1974): 1323-1327.

 

 

Cronbach, L.J. Essentials of Psychological Testing. Third Edition.
New York: Harper.and Row, 1970.

Cronbach, L.J., and Furby, L. "How We Should Measure 'Change' --
0r Should We?" _Egychological Bulletin 74 (1970), 1.

Davis, A.E., Dinitz, S., and Pasamanick, B. "The Prevention of
Hospitalization Schizophrenia: Five Years After an
Experimental Program." American Journal of Orthopsychiatry
42 (1972): 375-388. ‘

Derogatis, L.R. SCL-9O Revised Version Manual-I. John Hopkins
University School of Medicine, 1977.

Derogatis, L.R., and Cleary, P.A. "Factorial Invariance Across
Gender for the Primary Symptom Dimensions of the SCL-90."
British Journal of Social and Clinical Psychology, 1977
(in presS)?

Derogatis, L.R., Lipman, R.S., and Covi, L. "SCL-90: An Outpatient
Psychiatric Rating Scale - Preliminary Report." Psycho-
pharmacology Bulletin 9 (1973),1: 13-17.

Derogatis, L.R., Lipman, R.S., Covi, L., and Rickels, K. "Neurotic
Symptom Dimensions as Perceived by Psychiatrists and Patients
of Various Social Classes." Archives of General Psychiatry
24 (1971): 454-464.

"Factorial Invariance of Symptom Dimensions in Anxious
and Depressive Neuroses." Archivesof General Psychiatry
27 (1972): 659-665.

 

Derogatis, L.R., Lipman, R.S., Rickels, K., Uhlenhuth, E.H., and
Covi, L. "The Hopkins Symptom Checklist (HSCL): A Measure
of Primary Symptom Dimensions." In Psychological Measure-
ments in Psychppharmacology. Edited by P. Pichot. Basel:
Karger, 1974.

 

 

Derogatis, L.R., Rickels, K., and Rock, A.F. "The SCL-90 and the
MMPI: A Step in the Validation of a New Self-Report Scale."
British Journal of Psychiatry 128 (1976): 280-289.

lOO

Edwards, D.W., Yarvis, R., Mueller, D.P., Zingale, H., and Wagman,
W.J. "Test-Taking and the Stability of Adjustment Scales:
Can We Assess Patient Deterioration?" Evaluation_guarterly
2 (May 1978), 2: 275-291.

 

Ellsworth, R.B. "Consumer Feedback in Measuring the Effectiveness
of Mental Health Programs. " In Handbook of Evaluation
Research. Edited by M. Guttentag and E. L. Stovening.

London: Sage Publications, 1975.

 

. PARS Scale Manual. Roanoke: Institute for Program
Evaluation, 1975.

 

. "An Evaluation of Mental Health Services. " Presented at
4th International Symposium of the Kittay Scientific
Foundation, New York, 1976.

Ellsworth, R.B., Foster, L., Childers, B., Arthur, G., and Kroeker,
0. "Hospital and Community Adjustment as Perceived by
Psychiatric Patients, Their Families and Staff. " Journal
of Consulting and Clinical Psychology_32 (1968), 5, Part 2,
Monograph Supplement.

 

Erickson, R.C. "Outcome Studies in Mental Hospitals: A Review."
Psychological Bulletin 82 (1975), 4: 519-540.

Fiske, D.W. "The Shaky Evidence Is Slowly Put Together. " Journal
of Consulting and Clinical Psychology 37 (1971), 3: 314-315.

Fiske, D.W., Cartwright, 0.5., and Kirtner, W.L. "Are Psycho-
therapeutic Changes Predictable?" Journal of Abnormal and
Social Ppychology 69 (1964): 418-426.

 

 

Fontana, A. F. , and Dowds, B. N. "Assessing Treatment Outcome.'
Journal of Nervous and Mental Disease 161 (1975), 4. 221- 238.

 

Forsyth, R.P., and Fairweather, G.F. "Psychotherapeutic and Other
Hospital Treatment Criteria: The Dilemma." Journal of
Abnormal and Social Psychology 62 (1961), 3: 589-604.

 

Garfield, A. L. ,Prager, R. A. , and Bergin, E. "Evaluation of Outcome
in Psychotherapy." Journal of Consulting and Clinical
prchology_37 (1971), 3: 307- 313.

Helmstadter, G.C. Principles of Psychological Measurement. New
York: Appleton-Century-Crofts, 1964.

 

Hoffman and Overall. "Factor Structure of the SCL- 90 in a Psy-
chiatric Population. " Journal of Consulting and Clinical
Ppychology 46 (1978), 6.

101

Ihilevich, D. and Glaser, G. Manual for the Progress Evaluation
Scales. Owosso, Michigan:- Shiawassee County Community
Mental Health Services Board, 1979.

 

Kenny, D.A. "A Quasi-Experimental Approach to Assessing Treatment
Effects in the Nonequivalent Control Group Design."
Psychological Bulletin 82 (l975), 3: 345-362.

LaFerriere, L. "Recommendations for Developing a Statewide Mental
Health Evaluation System." Lansing, Michigan: Department
of Mental Health, 1979.

Lick, J. "Statistical vs. Clinical Significance in Research on
the Outcome of Psychotherapy." International Journal of
Mental Health 2 (1973), 2: 26-37.

 

 

Luborsky, L., Chandler, M., Auerbach, A.H., Cohen, J., and Bachrach,
H.M. "Factors Influencing the Outcome of Psychotherapy:
A Review of Quantitative Research." Psychological
Bulletin 75 (1971), 3: l45-185.

 

Magnusson, David. Test Theory. Massachusetts: Addison—Wesley Co.,
1969.

 

Maguire, L. "Evaluating Treatment Outcome." Ann Arbor, Michigan:
Washtenaw County Community Mental Health Center, 1977.

McPhee, C., and Zusman, J. anlity Evaluation of Mental Health
Services. Washington, D.C.: National Institute of Mental
Health, 1976.

Meltzoff, J., and Kornreich, M. Research in Psychotherapy. New
York: Atherton Press, Inc., 1970.

 

Mikawa, J.K. "Evaluation in Community Mental Health." In Advances
in Psychological Assessment, Volume 3. Edited by P. Reynolds.
San FranciSco: Josey-Bates Publications, 1975.

Nunnally, J.C. Psychometric Theory, New York: McGraw-Hill, 1967.

"The Study of Change in Evaluation Research: Principles
Concerning Measurement, Experimental Design, and Analysis."
In Handbook of Evaluation Research. Edited by E.L. Stovening
and M. Guttentag. London: Sage Publications, l975.

 

Pasamanick, B., Scarpetti, F., and Kinitz, S. Schizophrencis in
the Community: An Esperimental Study in the Prevention of
Hospitalization. New York: Appleton-Century-Crofts, 1967.

 

Paul, G.L. Insight vs. Desensitization to Psychotherapy. California:
Stanford university Press, 1966.

102

Sarason, S.B., Lighthall, F.F., Davidson, K.S., Waite, R.R., and
Ruebush, B.K. Anxiety in Elementary School Children. New
York: Wiley, 1960.

Schainblatt, A.H. Monitoring the Outcomes of State Mental Health
Treatment Programs: Some Initial Suggestions. Washington,
D.C.: The Urban Institute, 1977.

"Interim Report: What Happens to the Clients? Monitoring
the Outcomes of State and Local Mental Health Services.”
Washington, D.C.: The Urban Institute, 1978.

Speer, D.C., and Tapp, J.C. "Evaluation of Mental Health Service
Effectiveness: A 'Start Up' Model for Established Programs."
American Journal of Orthopsychiatry 46 (1976), 2: 217-228.

Waskow, I., and Parloff, M.B. Psychothergpy Change Measures.
Rockville, Maryland: National Institute of Mental Health,
1975.

 

Weissman, M.M., Slobetz, F., Prusoff, B., Mezritz, M., and Howard,
P. "Clinical Depression Among Narcotics Addicts Maintained
on Methadone in the Community." American Journal of
Psychiatry 133 (1976): 1434-1438.

 

 

Wider, A. The Cornell Medical Index. New York: Psychological
Corporation, 1948.

 

Wise, T.N., and Fernandez, F. "Psychological Profiles of Candidates
Seeking Surgical Correction for Obesity." Psychosomatics,
1977 (in press).

 

APPENDICES

103

 

APPENDIX A

PARS SCORES 0F OUTPATIENTS AND INPATIENTS
AT INTAKE

104

 

PARS Scores of Outpatients and Inpatients

 

at Intake
FEMALES
Average PARS Scores: Outpatients Inpatients
(n=157) (n=24)

Symptom Score

(range = 5-20*) 11.8 15.5
Role Skills

(range = 3.8*-15) 11.3 8.6
Alcohol/Drug Abuse

(range = 4-16*) 5.4 7.4

 

*These scores indicate the most gysfunctional score.

 

 

MALES
Average PARS Scores: Outpatients Inpatients
(n=81) (n=12)

Symptom Score

(range = 4.7-18.7*) 10.8 13.3
Role Skills

(range = 3.5*-l4) 7.9 6.8
Alcohol/Drug Abuse

(range = 5-20*) 7.4 8.5

 

*These scores indicate the most dysfunctional score.

105

APPENDIX B

CLIENT SATISFACTION WITH SERVICE

106

Client Satisfaction with Service

 

OUTPATIENTS INPATIENTS
Males Females Males Females
Number of Subjects 41 1.0 12 30
Overall Satisfaction
(range: l-5*) 4.3 4.0 3.7 3.3
Reported Change
(range: 1-5*) 4.0 3.8 4.2 3.6

 

*Most satisfied or greatest change reported.

107

APPENDIX C

(CONSENT FORMS AND INSTRUMENTS
Brief Symptom Inventory

Personal Adjustment and Role
Skills Questionnaire

Client Satisfaction Survey
Form

108

 

109

9111-18424!

CLIENT CONSENT FORM
MENTAL HEALTH STUDY

He would like to know how the people who use our services are feeling and how they are
functioning in their daily lives. He also would like to know if the services provided
are helpful in dealing with problems. In order to collect this information. we are
asking all clients using our services to participate in a study. This study should
help us improve our programs over the next several years. There will be no direct.
inuediate benefits and no risks to you if you agree to participate.

All information gathered in this study will be kept confidential. Your name will not
be connected with the infomation you provide, nor will this ,information oe shown to
anyone outside the evaluation staff in our agency and the Michigan Department of Mental
Health. The data collected in this study will be used for statistical pursoses in the
Mutant of Mental Health and destroyed within five years of its collect: on.

The study will involve two parts. You may participate in org or both parts:

(1) A questionnaire to be sent to you at one or two later times in the year
Mich you would complete and return to the Oepartnent of Mental Health.
This questionnaire is similar to the Mental Health questionnaires you just
coauleted; and

(2) A questionnaire to be sent to a person you feel knows you well and
could say how you are doing on a daily basis. This person may be a
relative (spouse. parent or child if over 14 years) or a close friend.
The questionnaire would be sent to this person now and at one or two
later times in the year. If you wish to see a copy of this question-
naire. please ask the secretary. .

We sincerely hope you will agree to participate in both parts of this study, but there
W111 130 no Denalty to you if you do not. You will receive the same services whether you
agree to participate or not. You may withdraw your consent to participate at any tine
during the study by notifying any staff wiper at this agency. If you have any questions
about the study, or your participation in it. please ask the secretary or your counselor.
Thank you for your cooperation.

. O 0

PLEASE GiECK THE BOX 0R BOXES BELOW THAT INDICATE YOUR CONSENT 0R NON-COPSENT AND THEN
SIGN YOUR NAME AT THE BOTTOM OF THE PAGE. THANK YOU.

D I agree to participate in the study described above and have had an

opportunity to have my questions answered. You may send me a question-
naire at one or bro later times in the year at the following address:

Address

 

City Zip Code

Telephone No.

 

D You may send a questionnaire to this person who knows me well:

 

 

Name Is this person
here with you

Address today?

City Zip Code

 

Telephone No.

 

U I do not wish to participate in this study because

 

 

""OUR SIGNATURE DATE

 

110

B.S.I.*

INSTRUCTIONS: Below is a list of problems and complaints that people sometimes have. Read each one carefully and
decide how much that problem bothered you during the past 1295' including today. Circle the number under the
column heading that best describes how much that problem bothered you. For example. the first problem is
"Nervousness or shakiness inside." If you have been bothered by that problem a little bit during the last week,
you w0uld circle 2 under the second column. Please do not skip any items. If yOu change your mind. erase the
first mark completely. If y0u have any questions. please ask the secretary.

During the past week, how much During the past week, how much

were you bothered by: \\ were you bothered by: <\
x. 9‘0; P v31 r 9:“ ‘- "\\\\\g ‘29“ ‘61,“
. 3 ~. -’I . .
\\°" ‘; tobc‘mi‘itdv <9" i 0: Pbc‘ts5\‘:*oe
l. Nervousness or shakiness inside 0 1 2 3 4 27. Difficulty making decisions 0 1 2 3 4
2. Faintness or dizziness 0 1 2 3 4 28. Feeling afraid to travel on O l 2 3 4
' buses, subways. or trains
3. The idea that someone else can 0 1 2 3 4
control your thoughts 29. Trouble getting your breath O l 2 3 4
4. Feeling others are to blame for O 1 2 3 4 30. Hot or cold spells O l 2 3 4
most of your troubles
31. Having to avoid certain things, 0 1 2 3 4
5. Trouble remembering things 0 1 2 3 4 places or activities because
they frighten you
6. Feeling easily annoyed or O 1 2 3 4
irritated 32. Your mind going blank 0 1 2 3 4
7. Pains in heart or chest 0 l ,2 3 4 33. Numbness or tingling in parts 0 l 2 3 4
of your body
8. Feeling afraid in open spaces 0 1 2 3 4 ,
. 34. The idea that you should be 0 1 2 3 4
9. Thoughts of ending your life 0 1 2 3 4 punished for your sins
10. Feeling that most people 0 l 2 3 4 35. Feeling hopeless about the future 0 1 2 3 4
cannot be trusted -
36. Trouble concentrating 0 1 2 3 4
11. Poor appetite 0 1 2 3 4
12 S dd 1 d f 0 1 2 3 4 37. Feeling weak in parts of yOur body 0 l 2 3 4
. u en y scare or no reason
38. Feeling tense or keyed up 0 1 2 3 4
13. Temper outbursts you could not 0 1 2 3 4
control 39. Thoughts of death or dying O 1 2 3 4
14. Feeling lonely even when you 0 1 2 3 4 40. Having urges to beat, injure or O 1 2 3 4
are with people harm someone
15. Feeling blocked in getting 0 1 2 3 4 41. Having urges to break or smash O l 2 3 4
things done things
16. Feeling lonely O 1 2 3 4 42. Feeling very self-conscious with 0 1 2 3 4
others
17. Feeling blue 0 1 2 3 4
43. Feeling uneasy in crowds O 1 2 3 4
18. F elin o i t t i th’ 1 2 3 4
e g n n eres n ings 0 44. Never feeling close to another 0 1 2 3 4
19. Feeling fearful 0 1 2 3 4 person
20. Your feelings being easily hurt O 1 2 3 4 45' Spells 0f terror or panic 0 1 2 3 4
21 F 11 th t 1 .0 1 2 3 4 46. 'Gctting into frequent arguments O 1 2 3 4
. ee ng a pee e are
unfriendly or d1§11ke you 47. Feeling nervous when you are 0 l 2 3 4
c a one
22. Feeling inferior to others 0 l 2 3 4 48. Others not giving you proper 0 1 2 3 4
23. Nausea or upset stomach 0 1 2 3 4 credit for your achievements
24 F eling that o are t h d 0 1 2 3 4 49. Feelinglso restless you couldn‘t O l 2 3 4
. e y u we c e s sti
or talked about by others _
SO. Feelings of worthlessncss O 1 2 3 4
25' Trouble falling asleep 0 I 2 3 4 51. Feelirw that pepple will taFe O 1 2 3 4
I . " I. " . ‘ " t I
26. Having to check and double- 0 1 2 3 4 OGVJHIOJC o ’(u 1f you 1L then
check what you do 52. Ftolings of stilt C 1 2 3 4
S3. Idea something is wrong with your 0 1 2 3 4
*COpyright 1975 Ly Leonard Derogatis. Ph.D. mInd '

Ill

DMH FORM N0. 1841R

PERSONAL ADJUSTMENT AND ROLES SKILLS CUESTIONHAIRE*

 

INSTRUCTIONS: Please describe the person‘s community adjustment during the past
month by answering each question telow. Hark your answer to each question by
circling the number under your answer choice. For exempl_, in question :1, if
the person you are rating has shown consideration'fkfifyou “often” during the
last month. you would circle 3 under the heading "often."

Please answer each statement below.

fiAgency Use Only;

L_I_1_LJ

A.N.
C.C.N. L_l_J__L_L_l_J
N.Q.
NIA.

W—

Date Comp. UJLLILLJ

 

 

 

 

 

 

 

 

 

 

 

 

 

DURING LAST—MONTH, HAS HE/SHE . . . DURING LAST MONTH, HAS HE/SHE . . .
Some- Almost Some- Almost
Rarely times Often Always Never times Often Always
1. Shown consideration for 16. Had difficulty eating
you. 1 2 3 4 (poor appetite, indi-
gestion. etc.) 1 2 3 4
2. Felt close to members
of household. 1 2 3 4 17. Been nervous. l 2 3 4
3. Discussed important 18. Acted restless and tense. 1 2 3 4
matters with you. 1 2 3 4
19. Had difficulty sleeping. l 2 3 4
4. Been able to talk it
through when angry. l 2 3 4 DURING LAST MONTH, HAS HE/SHE . .
Some-
5. Cooperated (gone along) Never Rarely, times Often
when things asked of
him or her. 1 2 3 4 20. Jumped from one subject
to another when talking. l 2 3 4
6. Shown interest in what
you say. 1 2 3 4 21. Just sat and stared. l 2 3 4
7. Shown affection toward 22 Forgotten to do
you. 1 2 3 4 important things. 1 2 3 4
8. 'Gotten along with 23. Been in a daze or
other family members. 1 2 3 4 confused. 1 2 3 4
DURING LAST MONTH, HAS HE/SHE . . . 24. Needed supervision or
Some- guidance. 2 3 4
Never Rarely» times Often
25. Lost track of time. 1 2 3 4
9. Said peeple don't care
about him/her. 2 3 4 26. Seemed to be off in
another world. 1 2 3 4
10. Said people treat him/
her unfairly. 1 2 3 4 DURING LAST MONTH, HAS HE/SHE .
Some-
11. Complained or worried Never FGCEJX times Often
about problems. 1 2 3 4 _ _—-__'
27. Been drinking alcohol
12. Said people try to push to EACESS. l 2 3 4
him/her around. 2 3 4
28. Been using drugs
13. Said life wasn't worth excessively. l 2 3 4
living. 2 3 4
29 Become drunk on alcohol
14. Said things looked or high on drigs. 1 2 3 4
disc0uraging or hopeless. l 2 3 4
30. Had a drinking or drug
15. Talked about being problem that upset
afraid. l 2 3 4 family. 1 2 3 4
31. Had a drinking or drug
problem that interfered
with working. 1 2 3 4

 

'Copyright 1977 by Institute for Program Evaluation
Special printing by permission of IPEV 6/77

 

112

DMH FORM N0. 1841R

 

 

 

 

 

 

 

DURING LAST MONTH, HAS HE/SHE . . . DURING LAST MON1H. HAS HE/SHE . .
Almost Some- Almost
Never times Often Always 45. Attended meetings of civic. church or other
organizations? (Mark one)
32. Done chores around (1) Did not attend any meeting this
house. 1 2 3 4 past month.
(2) Rarely attended meetings.
33. Done household cleaning. 1 2 3 4 (3) Sometimes attended meetings.
(4) Often attended meetings.
34. Prepared meals for the
family. 1 2 3 4 46. Participated in recreational activities Outside
the home? (Mark one)
35. Done laundry, ironing (1) No recreational activities outside
or mending. l 2 3 4 home.
(2) Rarely participated in outside
36. Done grocery shopping. 1 2 3 4 recreation.
(3) Sometimes participated.
(4) Often participated.
37. Are there usually children in the home? (Mark one)
1 No (If "no,” skip to question 44) 47. Been employed outside the home? (Mark one)
2 Yes (If “yes." answer questions 38-43) (1) Unemployed last month
(skip to question 51)
DURING LAST NONTH. HAS HE/SHE . . . (2) _ Employed part time last month.
Almost Some- Almost (3) Employed full time last month.
Never times Often Always Note: If employed part or full time,

 

_ please answer questions 48-50.
38. Spent time with the

children. 1 2 3 4 48. About how much tgkg_hore pay_did he/she earn fr'
working last month? ‘Tbo not include money from
39. Shown affection toward pension or welfare)
the children. 1 2 3 4 (1 Earned little or no money last month.

 

(2) Earned less than SlCO per week.

 

 

 

 

40. Kept promises to the (3) Between $100 and 3200 per week.
children. 1 2 3 4 (4) Over $200 per week from work‘ng.
41. Been consistent in re— 49. From working, did he/she earn an adequate amount
acting to the children. 1 2 3 4 of money last month?
(1 Earned no money by working last month.
42. Known right thing to (2) Earned enough to take care of
do when disciplining personal needs.
children. 1 2 3 4 (3) Earned enough to partially support a
family.
43. Had children show (4) Earned enough to adequately support a
respect for him/her. 1 2 3 4 family.
DURING EAST MONTH. HAS HE/SHE . . . SO. Oid he/she look forward to going to work each
day? (Mark one)
44. Been involved in activities outside the home? (1) __m____Not eralOyed last month.
Mark one) 52; Rarely looked forward to work.
1 Stayed at home this past month. 3 Sometimes looked forward to work.
2 Rarely involved outside the home. (4) __._‘__ USually looked forward to work”

 

3 Involved in some outside activities.
(4) Often involved in outside activities.

 

euuceeuv -
NOTE:
CERTAIN AREA

PERSOIT
BE SURE TO AIS HER EACH QUESTION BELCu.

113

IC41R

 

CLLSlIOAS il-u§ ASA THAT it; {LTICAlE HHETAIR
8 Of AO‘LST”ZTT CALSED PAOF_ENS FOR THE
YCU ARE RATI' : T‘Ri'u TPE PAST' 'uNTH. PLEASE

 

DURING THE PAST FONT“,

51.

52.

S3.

54.

55.

56.

57.

58.

59.

d.

HAS hE/ShE FAD PROBLEMS .

Talking and relating to you and people close
with him or her?
(1) No problems.

(2) '—: Some problems.
(3)

Serious problems.
If this is a problem,

is it . . .
Getting worse ___ Getting better

t Self or teing angry with others?

Feeling bao atoo

(I) __ li'o prc‘JIEi'S.
(2) ___ Some :0 W15 5.
(3) SCFICJED HO ES

If this is a pioolc-m. is 5it . .
Getting worse ____Getting better

Being nervous, not sleeping or eating well?

1) No problems.
2) Some uicblems.
(3) Serio .s Voblems.

If this is a pr: ole ., is it .

Getting worse *Cetting better __No change

Forgetting things, being confused?
(1) ____No problcns.

(2) __"_Some problems.

(3) __¢ Serious problems,

If this is a problem. is it . .
Getting worse ___ Getting better _No change
Using alcohol or crops to excess?

(1) _"_ No problems.

(2) _Some protileT s

(3) _Serious proble s.

If this is a grobfem is it .

Getting worse "_n 'Eetting better _No change

Doing reuserc r crores, laundry, cooking,
cleaning, shopping?

(0) ___ Not expected.

(I) ___ No problem.

(2) ___ Scme problems.

(3) Sevious'problems.

If this is a problem is it .

Getting worse __ _Cett ing better ___ No change
Relating t: children in the home?

(O) __ No children home.

(1) __H.Noinoblem.

(2) __ Sore aroblems.

(3) ____Serirss problems.

If t.is is a problem, is it . . .

Getting worse ___ Getting better____ No change
Getting involved in outside social activities?
(0) ___ hot errected.

(l) ___ Nc prcblem.

(2) {are problems.

(3) Se‘ieus problems.

If this is a C’Cblem, is it .

Getting worse __W_Getting better ____NO change

Earning mOhev from working?

(0) Not expected.

(1) :TT'Nn problem.

(2 —— Some problems.

(3 __T Serious :rct ems.

If this is a probl e.n. is it . .
Getting worse '*__Getting better

No change

No change ___

_No change

HAS
FROELEHE

THL PERSON IK‘: FF:

RATING RAISED THE FOLLONI NG
FER THE FAHILY DURING THE LAST MONTH?

(Circle the number under yOur answer)

60.

61.

62.

63.

64.

65.

6b.

67.

Not at A little Quite
all bit a bit

A great
deal

Has she/he been
irritable or
angry? I 2 3 4

Have you been

worried he/she

might hurt

thanself? l 2 3 4

Have his/her

problems CAUSED

a drain on the

family's finances? l 2 3 4

Have any children

in the family been

upset BECAUSE of

his/her problems

(e.g.. angry,

frightened. sad)? 1 2 3 4

Have any children

in the family not

gotten endugh

attention EECAUSE

OF his/her

problems? 1 2 3 4

Have you or any

other adult in the

family been more

upset than usual

BECAUSE of his/her

problems? 1 2 3 4

Have you or anyone
else in the family
taken cvev extra

duties BECAUSE cf
hiS/her problems? 1

A)
w
A

Have his/ncr prob-

lems INTEPFERED

with your family‘s

activities? 1 2 3 4

 

 

PLEASE COMPLETE THE FOLLOWING EACKGROUNO QUESTIONS:

68.

69.

Today's Date.

ch often did y0u see this person during the last
month?

(1) ___ Notat all.

(2) ‘___ Once or twice during past month.

(3) ___ About once a week.

(4) {___ About 3 to 5 times a week.

(5) ____ Saw daily.

(6) ___ Saw daily for 5 or more hours daily.

Rr.at is y0ur relationship to the person you are

___ SpouSe or mate.
(2) ___ Parent.
(3) ___ Other relative (sister. aunt. etc.)
(4) ___ Friend.

 

(OVER)

ll4

CLIENT SATISFACTION QUESTIONNAIRE

The following questions ask you to rate the services you received at our agency. Please circle those answers which
best describe your experience at our agency. Your counselor will not see the answers you give here. He will add
up the answers all clients give us to help us plan for future services.

l. Did your counselor seem to understand you and your problems?

 

l 2 3 4 5
Never Seldom Sometimes Usually Always
understood understood . understood understood _ understood

2. Here you satisfied with how often you saw your counselor to get help with your problem?

 

l 2 3' 4 5
Very dissatisfied Somewhat Undecided Somewhat Very satisfied
dissatisfied . satisfied

3A. You came to our agency to get help with certain problems. Has there been any change in those problems since you
first came to our agency?

'

 

I 2 3 4 5
A great deal Horse Same Improved A great deal
worse improved

38. How important were the services you received at our agenCy in helping you with those problems?

 

l 2 3 4 5
Not important Not very important Undecided Somewhat Very important
at all important

0n the next few questions, please place a check (15:) beside those answers which best describe your experience at
our agency.

4 For each of the following, please check whether you received the service at our agency and how helpful the service
' was if you received it:

 

 

 

 

Not Received Not at all Not too Very

(Check all that apply) At this Agency Helpful Helpful Undecided Helpful Helpful
a. Group therapy I. _ 2. __ 3. __ 4. _ 5. __ 6. _
b. Individual therapy l. __ 2. __ 3. __ 4. __ S. __ 6. __
c. Family therapy I. _ Z. _ 3. __ 4. __ S. __ 6. __
d. Medication l. __ 2. __ 3. __ 4. __ 5. __ 6. __
a. Emergency services I. _ 2. _ 3. __ 4. __ S. _ 6. __
f. Marital therapy I. _ 2. __ 3. _ 4. __ S. _ 6. __
9. Activity therapy I. _____ 2. _____ 3. _____ 4. ____ S. ____ 6. ____
h. Respite center services l. _____ 2. _____ 3. _____ 4. _____ S. _____ 5. _____
1. Residential services I. 2. 3. 4. 5. 6.

DHH Form l843-R (PLEASE TURN OVER TO PAGE 2

V

HS

-2-

SA. Here yOu referred to another agency by our staff?
l. YES. The agency recommended to me was

 

 

2. NO.

SB. If "YES," did you receive services at the agency referred to you?
1. YES 2. N0, because: (Please check all that apply)

a. The agency is too far away.

D. My problem is solved.

c. I wasn‘t eligible for services there.

d. I had to wait too long for an appointment.
e. I haven't gotten around to going there yet.
f. The fees are too expensive.

9. Other (Please explain)

 

 

6A. Are you still being seen at our agency? 1. YES 2. N0

68. If "N0," what were the reasons you stopped coming to our agency? (Please check all that apply)

—J

0 . My counselor suggested I stop. (Reason:
02. I felt I should stop. but my counselor didn’t.

 

03. My caunselor and I both felt that goals were met.

3

. My counselor left.

05. No counselors were available at the time(s) . needed to be seen.

06. I felt my therapist was unable to help me.

O
N

. I was sent to another agency for services.

08. The fees were too expensive.

0
O
D

I got tired of coming.
l0. Other reaSOns for stopping. (Please explain)

llllllll

 

 

Please circle the answers below vhich best describe your feelings about our agency. as you did in the first section.

7. Would you use our services again if yoo felt the need?
l 2 3 4 5

No, definitely not No, I don‘t Undecided Yes. I think so Yes, definitely
think so

 

8. In general. how satisfied or dissatisfied were you with the services you received at our agency?

 

l 2 3 4 5
Very dissatisfied Sotewhat Undecided Somewhat Very satisfied
dissatisfied satisfied

Please use the soace belOw to share with us any other suggestions. comments or observations that mignt help us plan our
services in the future. Thank you for your cooperation.

DHH Fonn lS43-R(lC/77)

"I7'11llllllllllllflillllllli