— — = — * h — —. —— — — _— —_ _‘. — — (Nam: lllllalll lllllllllllllllllllllllllllllll 1293 00104 7509 LIBRARY Michigan State University This is to certify that the thesis entitled PREDICTORS OF RETURN OF FOLLOW-UP DATA BY FORMER PSYCHOTHERAPY CLIENTS presented by RALPH JOHN TOBIAS has been accepted towards fulfillment of the requirements for MA , PSYCHOLOGY degree 1n (lbw M Major professor Date :1’911787 0-7639 MSU is an Affirmative Action/Equal Opportunity Institution MSU LIBRARIES RETURNING MATERIALS: Place in book drop to remove this checkout from your recordl FINES will be charged if book is returned after the date stamped below. PREDICTORS OF RETURN OF FOLLOW-UP DATA BY FORMER PSYCHOTHERAPY CLIENTS BY Ralph John Tobias A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of MASTER OF ARTS Department of Psychology 1987 ABSTRACT PREDICTORS OF RETURN OF FOLLOW-UP DATA BY FORMER PSYCHOTHERAPY CLIENTS BY Ralph John Tobias The assumption that follow-up assessment is needed for the evaluation of psychotherapy was questioned by Nicholson and Berman (1983) who concluded—that information available at follow-up added little to information gained at the end of treatment, In order to study bias in the rate of return, the present study attempted to predict return rate of mailed questionaires using socioeconomic status (SES), client satisfaction, perceived symptom change, and therapist variables. None of these variables produced statistically significant correlations with return rate. The variables were then grouped into clusters of SES, change in symptom severity, and client's perceptions. This model identified return behavior correctly in 60% of the 56 cases examined (0.lO>p>0.05). Implications for performing follow-up studies were disussed. ii ACKNOWLEDGMENTS This work benefitted from the encouragement, support, and interest of many people. In particular I would like to express appreciation to Dr. Norman Abeles, my chairperson, for his helpful suggestions and endless patience. I also thank my committee members: Dr. Ralph Levine for his time and energy in suggesting and explaining useful statistical techniques; and Dr. Bertrum Karon for his encouragement and good humor. In addition, I would like to recognize the contribution of my parents, Siegfried and Martha Tobias, and my brother, Edmund. This work was unquestionably benefitted by their support, curiosity, and unmatched interest. ' iii TABLE OF CONTENTS LIST OF TABLES INTRODUCTION REVIEW OF THE LITERATURE FOLLOW-UP DATA RESPONSE BIAS IN THE SURVEY LITERATURE RESPONSE BIAS WITH THERAPY CLIENTS PURPOSE OF THE STUDY STATEMENT OF HYPOTMESES HYPOTHESIS HYPOTHESIS HYPOTHESIS HYPOTHESIS HYPOTHESIS HYPOTMESIS OU'lubWNP STATISTICS METHOD SUBJECTS INDEPENDENT VARIABLES DEPENDENT VARIABLE INSTRUMENTS PROCEDURE RESULTS HYPOTMESIS HYPOTHESIS HYPOTHESIS HYPOTHESIS HYPOTHESIS HYPOTHESIS COMBINING CORRELATED VARIABLES OiU'anNH PREDICTING RETURN DISCUSSION BIBLIOGRAPHY iv iv @wNN H 14 15 15 15 16 17 17 17 18 22 22 23 24 24 28 29 29 31 31 32 32 32 35 38 46 Table LIST OF TABLES Pearson Correlation of Predictors with Return Correlations between Predictor variables Component Loadings of Predictor variables Discriminant Function Analysis Beta Weights for Clusters Predicted Return vs. Actual Return Page 30 30 34 34 36 INTRODUCTION When completed and returned, mailed questionaires are a valuable research methodology. However, the presence of unidentified bias in the return rates of questionaires may introduce threats to both the reliability and validity of the findings (Campbell and Stanley, 1963). Mailed questionaires have been used to assess the efficacy of psychotherapy with these same threats to validity remaining. The results then become in doubt; it remains unclear whether the findings are a result of a testing effect or whether an actual finding has been identified. Data available on therapy clients throughout therapy can be used to assess the questionaire as a research instrument. Only a small number of explanations are proposed that might explain why a former therapy client would or would not decide to return a follow-up questionaire. This study contributes to the examination of possible biases in mail questionaires by examining client characteristics and client perceptions of therapy as variables that are influential in the rate of return of questionaires. The identification of unrecognized sources of error refines the methodology while the adjustment for the effect of the bias increases the reliability of the findings. REVIEW OF THE LITERATURE Follow-up Data Nicholson and Berman (1983) present the argument that follow—up research in therapy patients may be neither cost- effective nor particularily helpful. The results summarized 67 follow-up studies for some form of psychotherapeutic intervention. The influences of psychotherapy were concluded to be sufficiently robust to show little change between termination and follow-up. The authors make reference to the concluding remarks in the literature review done by Smith, Glass, and Miller (1980) where the authors stated: Psychotherapy is beneficial consistently so and in many different ways. Its benefits are on par with other expensive and ambitious interventions, such as schooling and medicine. The benefits are not permanent, but then little is. (p.183) ' On the basis of their own results Nicholson and Herman (1983) reply that "our findings indicate that for a broad range of disorders this improvement stands the test of time." (p.275) The findings that the effects of psychotherapy are durable over time are used to suggest that follow-up evaluations be used in a more limited way so that resources may be conserved. Follow-up evaluations require additional energy, time, and money, resources that might be needed elsewhere. The 2 3 design of follow-up instruments, their reproduction, distribution, collection, tabulation, and analysis may require a sizable amount of a project's resources. Examining follow-up procedures is a means to a more efficient research methodology and a means to a better understanding of those variables that produce change in psychotherapy (Garfield, 1983). Response gigs in the Survey Literature Survey research data may be biased if some subjects respond and others do not. Mausner and Bahn (1974) suggest that non-compliance may be related to the variables under study. The extent and direction of the bias must be evaluated empirically, and the bias cannot be estimated before the study begins. Several techniques have been employed by researchers in the data analysis to account for non-responding. Bennett and Hill (1964) found that non-response was significantly related to only one of the twenty-three variables examined, and it was concluded that non-response was not an important confounder. Kivlin (1965) argues that response rate bias does not disturb the relationships among variables as all variables involved are equally effected and, relative to one another, remain unchanged. Rather than discount or minimize the possible differences between responders and non-responders, Robins (1963) describes the calculation of a range within which the true result should fall. Non-respondents are assumed to have opinions within either extreme of the distribution. 4 The upper and lower limits can then be calculated. In another approach, Lehman (1963) contends that it is a mistake to assume non-response is randomly distributed among the variables being examined and suggests to analyze the grouped data as if the returns represented a probability sample. Still another method requires the use of statistical weightings based upon response rates of various identified subgroups within the survey population (Fuller, 1974). Each of the above approaches assumes that each of the important discriminating variables are already identified. Mandell (1974) contends response rate is not uniform for each of the possible subgroups that exist in a sample and response rate differences exist. Each subgroup would respond differently depending on the strength of those variables present that either encourage or discourage. responding. The absence of an identifiable biasing variable with the available groupings does not prove that a consistent bias is not present. An empirical investigation about who does and does not respond to a particular questionaire is a means of uncovering the possibility of bias. Bebbington (1970) broadly categorized possible confounders into four areas; the attitude of the would-be subject, the topic of the survey, the impression of the interviewer, and the kind of survey instrument. The study used a 1951 cohort of then thirteen year old boys from nine previously selected schools each chosen due to the social class of the population which 5 it served. Seventy-three percent of the 614 subjects contacted replied after a ten year follow-up. School records were used to compare various kinds of "responders" to ”refusers.” Those that replied after the initial request were labelled "stallers." Individuals that were difficult to locate were labelled ”elusives,” and those that left the area were labelled "emigrators." The author was not able to identify characteristics that withstood statistical tests for significance for each of the categories. Never the less, non—responders were still assumed to consist of several distinct types of individuals. Their combined effect is theorized to bias the results in an unpredictable manner. In an attempt to account for non-responders, some researchers have advocated the use of ”waves" of questionaires to those that have not responded to previous requests (Lehman, 1963). Early responders are compared with late responders, and late responders are hypothesized to be similar to non—responders on major variables (Larson and Catton, 1959). Schwirian and Blaine (1966-67) hypothesized that initial responders to a survey of blue collar United Auto Workers would be more extreme than later responders. Half of the 46% returned responded to the initial survey while the other half responded to a second request. The subjects who returned the questionaire later had similar, but less extreme views then the initial group. The views of the two groups combined created a view different from the one had a second survey request not been done. 6 Linsky (1965) describes the two typical areas of research examining bias in mail questionaires. One approach is to compare different techniques and characteristics of mail questionaires. Those cited by Linsky (1965) include; different types of mail delivery, different color of stamps, number and type of stamps, supply of return envelopes, special letterheads, printed forms, questionaire length, and different appeals encouraging response. Other researched characteristics include; cash rewards (Blumberg, Fuller, and Rare, 1975), research affiliation with a status institution (Mauser and Bahn, 1974), how important the research topic is to the respondent (Heberlein and Baumgarten, 1978; wallace, 1954), and follow—up reminders (Hinrichs, 1975; Kanuk and Berenson, 1975). This approach details those methods that provide differences in response rates but does not aid in accounting for those differences. Kanuk and Berenson (1975) also discuss the difficulties in mail questionaire research. The authors describe a lack of systematic knowledge, conceptual framework, and efforts to replicate previous findings as the reasons for the inability to correlate certain response behaviors to specific questionaire techniques. The other method described by Linsky (1965) is to examine the differences between those that did and those that did not respond. Response rates are thought to increase if the topic is of interest to the potential respondent and the organization conducting the research is perceived as being prestigious (Blumberg et al, 1975; 7 Hausner and Bahn, 1974). The perceived importance of the topic to the subjects when rated by the therapist on a three point scale of 'saliency' was reported to account for an average response rate of 77% while a perceived unimportant tOpic was responded to by an average of 42% of the sample. How important the topic was perceived to be to the subject accounted for 50.5% of the variance in response rates (Reberlein and Baumgartner, 1978). The demographics of the sample population have been examined for differential response rates. The variables frequently examined include: Intelligence or IO, income, education, or a measure of socioeconomic status (SES) (Macek and Miles, 1975; Kanuk and Berenson, 1975). Cough and Hall (1977) found no significant response bias among physicians by either gender or SES. Vincent (1964) collected familial information from San Francisco Bay high school students and found those that cooperated more likely had fathers that had only completed high school; non-responders were found to have fathers that had high educational achievements. These results were also received by Reuss (1943) in a six year follow-up of the 1936 freshman class of State College of Washington. However, Kivlin (1965) found that dairy farmers who responded to a questionaire on the adoption of improved techniques were of higher SES and had more formal education than non-respondents. Similarily, Ognibene (1970) found higher educational achievements and more professionals among responders. These results were replicated with a follow-up of a child guidance clinic (Robins, 1963). Within the same 8 job classification, Gannon, Northern, and Carrol (1971) found that grocery checkers who had more than 15 years of education were more likely to respond to questionaires. Increased income, a variable correlated with high educational achievement, was found to be more heavily represented among responders than non-responders (Brown, 1984). Personality factors have been hypothesized to influence the decision to respond to a particular questionaire (Bennett and Hill, 1964; Lubin, Eugene, and Zuckerman, 1962; Ognibene, 1970). Wallace (1954) found SES did not differentiate responders from non-responders but that some subjects had a ”tendency to reply" to questionaires beyond what a probability distribution would predict. The research was conducted with a random sample of male Time magazine subscribers and a random sample of males in Akron, Ohio. They found that in both samples, about 50% of the questionaires returned, regardless of the topic, were consistently completed by the same individuals. In an - attempt to identify personality variables that would distinguish between responders and non-responders, Robins (1963) found no significant results in their study of records of a child guidance clinic. Response gig; gigp Therapy Clients Few studies have examined the possible influences of response bias with psychotherapy clients. In addition to personality variables and demographics, variables unique to psychotherapy have been examined to account for differences 9 between responders and non-responders to follow-up questionaires. Pilkonis, Imber, and Rubinsky (1984) examined the influence life events had on the outcome of psychotherapy for 64 patients followed-up an average of seven months after the end of psychotherapy. In addition to providing demographic information, subjects completed a 98-item events checklist and a Hopkins Symptom Checklist at intake. Each client was rated by the intake worker on the Global Assessment Scale. The life events checklist was completed again at follow-up. The best fitting multiple regression model included the variables of age, sex, socioeconomic status, a categorization of the presenting problem, prior treatment, and chronicity. Of the 12 possible regression equations that resulted from the research project's design, social class was included in six of those models as being significant. Typically, lower social class status was associated with poorer self-rated adjustment at follow-up than clients of other social classes. Correlations were reported that supported the results of Nicholson and Berman (1983) where outcomes at follow-up were strongly related to those at termination. Marziali (1984) received total compliance for forty-two clients and eight clinicians in her study of therapeutic alliance, and the perspectives of the clinician, the client, and an objective observer. Clients, clinicians, and nonparticipant judges attained inter-rater agreement correlations of 0.60 to 0.83 on the study variables. 10 Assessment instruments included Derogatis' then labelled Behavior Symptom Index, Beck's Hood Scale, and Weissman's Social Adjustment Scale. Measures were obtained prior to therapy, and three months after termination. After termination, both clients and therapists completed Therapy Evaluation Ouestionaires. The authors concluded that the best predictors of outcome were first the ratings of patients' perception of their own contributions, and then, other's positive and negative contributions to the therapy relationship. Clinician's perceptions of their own or the other's positive contributions to the therapy relationship were also significant in predicting outcome. The best predictors of positive outcome were client's and clinician's positive contributions. Kirchner and Hogan (1982) surveyed clients who were seen at a community mental health center within a one year period. Of the 190 eligible former clients, a response rate of 52% was obtained using two follow-up contacts. Non- response was not adjusted for. Thirty-three of 35 subjects who indicated they felt "extremely helped" thought that the therapist was ”very interested" in them. Forty of 58 respondents who felt their therapist was ”very interested" in them sought no further help. Of those questionaires returned, 36‘ of the former clients indicated that therapy was "extremely helpful." Sixty-one percent of those responding indicated that they felt their therapist was ”very interested” in them. Therapists' ratings of their clients' improvement 11 appeared to be more critical of therapeutic outcome. Staff rated 48% of the clients who responded as "unimproved.” The authors suggest that perceiving that help is being given "results” in improvement. No mention is made of the alternative explanation of clients not wishing to be critical of the agency, or their former therapist. Cognitive dissonance may have occurred; clients who invested their time, energy, and money in therapy may be difficult to convince that little improvement occurred. This explanation could also account for the results obtained. Compliant behavior was found to increase following the experience of guilt (Konske, Staple, and Graf, 1979). While the guilt-compliance phenomenon has not been detailed, it is suggested that at least some of the clients who respond may have been prompted to comply out of guilt concerning some aspect of the psychotherapeutic experience. Rosen (1978) cites Milgram's "obedience" experiments to explain the willingness of clients approached by an authority figure in a community mental health center to sign an information release form when requested. When clients were given the opportunity to refuse, compliance dropped from 100% to 20%. Compliers were more likely to display more serious mental health problems, have less income, and less education. Crutchfield (1955) examined subjects believing they were being assessed because of their superior professional functioning. Conformity to perceived group norms was related to submissiveness, and being overly accepting of authority. There was no significant correlation found 12 between conformity and neuroticism in scales of the Minnesota Multiphasic Inventory. Cooley and Lajoy (1980) examined the perception of the therapeutic relationship by therapist and former clients at a community mental health center in Oregon. Complete data was obtained on 54 of 110 randomly selected clients who had used the mental health center over a three month period. Therapists' ratings of each client's improvement and client's self-ratings of improvement were positively correlated with the dimensions of perceived therapist's ”understanding" and ”accepting” as rated by clients. When therapists rated the therapeutic relationship, the dimension of ”independence-encouraging” was correlated with total client improvement ratings. In addition, with increased agreement in client and therapist ratings of perceived ”understanding" and "accepting," greater improvement was reported by both clients and therapists. Disagreements on ratings between therapist and client were associated with poorer outcomes. Sirles (1984) examined responses of 329 former clients who received counseling. A 44% return rate was obtained and differences between responders and non-responders were examined. Response rates were found to range between 20%- 75% dependent on the counselor seen. Those that terminated according to a pre-made plan were more likely to respond than those that suddenly withdrew. Those that were seen six or more times responded more often than those that were seen 2-5 times. Those clients that were assessed by the 13 counselor as having "improved" were significantly (p<.05) more likely to respond than those assessed as having made "no change." variables related to psychotherapy and its outcome appear to influence whether or not responses are made to survey requests. This study examines several psychotherapy related variables to explore the impact on responding to psychotherapy follow-up survey questionaires. PURPOSE OF THE STUDY This retrospective study investigates the use of client demographic characteristics and client's perception of therapy as possible predictors for the return of mailed follow—up questionaires. Bias in the return of follow-up questionaires has been researched little. According to Kanuk and Berenson (1975) the number of research studies designed to evaluate the effectiveness of mail surveys is ”so sparse as to make conclusive results almost impossible to reach." As might be expected, those studies which examine the methodology of psychotherapy outcome surveys are even fewer. Specifically, this study investigates the use of client demographic and client perceptions about therapy obtained pre- and post- therapy through self-report measures. These measures are used as a means of predicting whether or not a client will return follow-up questionaires. The measures administered include the Symptom Checklist-90 (Revised version) (Derogatis, 1977), a two factor index of social position (Hollingshead, 1957), and a Post Therapy Client Questionaire (PTCQ). The PTCQ asks clients about their opinions and evaluations about therapy. 14 STATEMENT OF HYPOTHESES Hypothesis 1 Those clients that are classified according to Hollinghead's Two Factor Index of Social Position as being higher in social class will be more likely to return follow- up data. Research with subjects from the general population have found that higher Socio-economic (SES) individuals typically have higher levels of responding (Kivlin, 1965; Ognibene, 1970). Research examining the behaviors of therapy clients have tended to support the findings of those of the general population (Robins, 1963; Schramski, Beutler, Lauver, Arizmendi, and Shanfield, 1984). The Two Factor Index combines the correlated variables of education and occupational status. Subjects are then assigned to one of five groups, I-v. Social Class I is the lowest through Social Class v which is the highest (Hollingshead, 1957). SES was found to be a significant variable in those studies that stated they had used the classification system (Pilkonis, Imber, and Rubinsky, 1984; Schramski et al, 1984). Hypothesis ; Those individuals that have a significant decrease in symptom severity or frequency as measured by the SCL-90-R 15 16 pre- and post— therapy will be more likely to return follow— up questionaires. The overall score of the SCL-90-R has been validated as a summation of current psychological functioning (Derogatis, 1977). The difference between pre- and post— scores on the self-report questionaire will be a measure of change in psychological functioning. Clients who have had a significant decrease in reported distress would be expected to return follow-up questionaires (Howard, Millham, Slaten, & O'Donnell, 1981; Nelson, 1981; Sargent, 1960; Sirles, 1984). Hypothesis 3 Those individuals who have a statistically greater perception of positive symptom change will be more likely to return follow-up questionaires. Clients who retrospectively perceive improvement in their symptoms are perhaps more likely to return follow-up quesioniares due to a connection between the change and psychotherapy (Cooley and Lajoy, 1980; Sargent, 1960). The Post Therapy Client Ouestionaire contains five questions that ask the client to make a retrospective evaluation. Kirchen and Hogan (1982) found that clients rate therapeutic outcome more favorably than therapists do. The questions are not designed to identify how much change actually occurred, but to estimate how much change clients believe has taken place. 17 Hypothesis‘i Those clients that indicate a high level of satisfaction as compared with low satisfaction on the Post Therapy Client Questionaire will be more likely to return a follow-up questionaire. Clients that indicated they were ”highly” or "extremely” satisfied would be predicted to more likely return questionaires at follow-up perhaps due to the investment they have developed through the experience of therapy (Kirchen and Hogan, 1982). Jacobs, Aronson, and Nystrom (1983) suggest that if clients are given the opportunity to decide to respond based on their satisfaction with the treatment outcome, than ”randomization is no longer present," and a biasing influence may be discovered. Hypothesisqg Those clients who experienced the therapist as being interested in them as a person would be more likely to return follow-up questionaires. Those individuals who believed the therapist to be genuinely interested in them would be more likely to decide to return follow-up questionaires. Actual change may not have taken place, nor must the client necessarily have been satisfied with the therapy. Hypothesis‘g Those individuals that experienced the therapist as being interested in them as "just another patient" would be significantly less likely to return follow-up questionaires. 18 This hypothesis is a corollary to the preceding one. Clients who perceive their therapist as being less personable, and seemingly less interested in them may develop a reduced investment in therapy, or perhaps, a reduced desire to respond for the therapist's sake. The feeling of not being special to the therapist may have no correlation with actual change attained. Statistics Following are the statistics proposed for the data analysis prior to the beginning of the study. Each of the hypotheses will be tested by initially obtaining the Pearson product—moment correlation coefficient. This statistic describes the strength of the association between each variable involved in the hypothesis to the return of follow- up data. The coefficient "r" will then be used in a t-test to examine if the association between the two variables is statistically significant. Each of the hypotheses described in the previous section will be tested against the null hypothesis that no significant association will be found. Hypothesis 1 examines the socioeconomic status of the client to be influential in the willingness of an individual to return follow-up questionaires. In particular, a positive correlation is expected to be found. Clients that are assigned by their reported income and educational attainment to higher socioeconomic classes will be more likely to return the requested information. This result is expected because of past cited reports in the available literature. 19 Hypothesis 2 proposes that individuals who report a change in severity between pre and post test will be more likely to return the follow-up questionaires. This result will translate into a positive correlation. As the change score obtained between pre and post test increases, the likelihood of finding that a client has returned the follow- up guestionaires will also increase. The statistical test will evaluate whether the association between the variables is statistically significant. Hypothesis 3 proposes that clients who retrospectively report that they perceive symptoms as having improved will be more likely to return follow-up questionaires. The Post Therapy Client Questionaire contains a number of questions that evaluate symptom change. The expected result of the calculations is a positive correlation. Those individuls who report greater perceived symptom change would indicate this by marking the responses assigned low numerical value by the questionaire. Hypothesis 4 proposes that clients who report a high level of satisfaction with their therapy will be more likely to return the follow-up questionaires. A negative correlation is expected. Hypothesis 5 proposes that clients who have had an experience in therapy in which they felt that the therapist was interested in them ”as a person" will be more likely to return the follow-up questionaires. Again, a positive correlation is expected as increased values for the questions evaluating this hypothesis are indicative of increased levels of agreement with the 20 hypothesis. Hypothesis 6 proposes that individuals who experienced their therapist's attitude toward them as ”just another patient" would be less likely to return follow-up questionaires. A negative correlation is expected for support of this hypothesis. After examining each of the six proposed hypotheses, the collected data will be examined by ”cluster analysis." Variables that are strongly correlated are combined into variables that account for as much variance as possible while reducing the correlation between variables. A principle components analysis will be done first to obtain the loadings between all of the variables. A multiple group factor analysis will then be performed to identify each of the clusters. Factor analysis is utilized to combine correlated variables so that as few variables as are necessary are created to explain as much variance as possible. A discriminant function analysis will then be performed using the cluster variables in place of the variables used to test the six hypotheses. This statistic will provide Beta weights on each of the variables that can be used to predict the appropriate categorization of individuals into those that either did or did not return follow-up questionaires. An evaluation of the validity of the Beta weights obtained becomes possible. The raw data on individuals will then be used in the formula to obtain the statistical 21 assignment to either the group that did return information or to the group that did not. The statistical predictions will then be compared to the actual outcome. Correct predictions can then be a measure of the strength of the variables examined in predicting the return of follow-up data. Generalizability beyond the 56 cases examined in this study will then be examined. Rather than attempt another study to cross validate the findings, Schmitt, Coyle and Rauschenberger (1977) advocate the use of a statistic to test the change in weights that might be anticipated if the weights obtained in this sample were applied to an entirely new sample. The Wherry formula for estimating the squared population cross-validity will be used to evaluate how effective the sample regression equation might be in future samplings. METHOD Subjects The subjects for this study were all clients in psychotherapy at the Psychological Clinic at Michigan State University. The clinic serves the community at large and the adult clients range in age from 25-75 years old. The clients are typically from the lower middle class with some middle class clients. To be eligible, individuals may not be full-time students at Michigan State University at the time of intake. The clinic is a research and training site for the Clinical Psychology Graduate Program for the Department of Psychology. The psychotherapists at the Psychological Clinic are all advanced graduate students in clinical psychology all of whom have had prior clinical experience. Clients in this study were all adults who signed a consent form after the psychotherapy research program had been explained to them. No one who declined to participate was denied psychological services. Socio-economic status is routinely gathered at intake so that fees can be appropriately assessed. Prior to therapy, subjects complete the Symptom Checklist-90-R. At the end of therapy this same instrument is completed as is ‘the Post Therapy Client Questionaire. Clients who did not: 22 23 agree to participate, complete the pre- and post SCL-90-R, and complete the Post Therapy Client Questionaire were not included in the study. Independent variables Six independent variables have been created to evaluate the ability to predict the return of follow-up questionaires. Those variables are: socioeconomic status; change in global symptomatology; perceived symptom change; satisfaction with therapy; and two kinds of interest the client may have experienced from the therapist, ”interested in them as a person," and as "just another client." Level of education and occupation is routinely collected at intake. This information will then be converted to an Index of Social Class from I through v as outlined by Hollingshead (1957). The SCL-90-R has been validated as an instrument that can assess overall symptomatology. The changes in pre- and post- scores can be used to obtain a change score in psychological functioning. The Post Therapy Client Ouestionaire (PTCO) contains five questions that request the client to retrospectively evaluate the amount of "change" or ”benefit” they perceive as having occurred over the course of therapy. Response selections range from ”a great deal” to ”not at all,” or from ”very adequately" to "very inadequately." Client satisfaction on the PTCO is measured by one question that asks directly about their level of "satisfaction" concerning the outcome of therapy. The 24 response choices range from ”extremely dissatisfied" to "extremely satisfied” on a seven point scale. Each variable measuring the client's perception of the therapist's attitude of the client as "just another client," or ”interested in them as a person” have multiple questions on the PTCQ, six and four questions respectively. Client's responses will be averaged for each variable represented by multiple questions. Dependent variable The dependent variable is the return of the client's follow-up questionaires. A behavior was chosen as it was thought to be concrete and easily identifiable. Each of the independent variables will be assessed as to how much each can account for predicting a client's behavior. Instruments .The hypotheses described earlier were tested by examining the results of pre-therapy SCL-90-R, post-therapy results of the SCL-90-R, and the Post Therapy Client Questionaire. In addition, Hollingshead's Two Factor Index of Social Position was used to classify subjects into social class. SCL-SO-R The SCL-90-R is a self-report instrument requiring the subject to indicate those symptoms, thoughts, and feelings that have been recently experienced. The subject is also required to indicate the extent of distress experienced ranging from "none at all" to "extremely." This measure assumes the subjects are aware of their symptoms, thoughts, 25 and feelings and are willing to report them. A subject's level of awareness has been identified as a confounder in using the SCL-90-R as outlined in the differences between inpatient responses and outpatient responses (Holcomb, Adams, and Ponder, 1983). In addition, social desirability has been identified by Derogatis (1977) as a distortion present in the SCL-90-R. The SCL-SO-R measures general discomfort in psychological functioning and has been tested; for its clinical utility (Derogatis, Lipman, Covi, Rickels, and Uhlenhuth, 1974); on general outpatients (Derogatis, Lipman, and Covi, 1973); and in particular for factor invariance on anxious and depressed patients (Derogatis, Lipman, Covi, and Rickels, 1972). Nine scales have been developed to provide a profile of an individual's level of functioning along each of the dimensions: (a) Somatization, (b) Obsessive— Compulsiveness, (c) Interpersonal Sensitivity, (d) Depression, (e) Anxiety, (f) Hostility, (g) Phobic Anxiety, (h) Paranoid Ideation, and (i) Psychoticism. The precursor to the SCL-90-R was the Hopkins Symptom Checklist (HSCL). This early version contained 58 items and with factor analysis was found to contain five identifiable factors (Derogatis, Lipman, Rickels, Uhlenhuth, and Covi, 1974) and was reported to have face and concurrent validity (Winokur, Guthurie, Rickels, and Nael, 1982). The HSCL was expanded into the SOL-90 and the current version is the SCL- 90-R. Little change has occurred between the SCL-90 and the revised version; Derogatis (1977) cites studies using either 26 version as support for his claims of reliability and validity. Studies on reliability have focused on measures such as internal consistency and test-retest (Derogatis, 1977). Holcomb, Adams and Ponder (1983) assessed the reliability of each set of items which define each factor. Reliability was calculated using co-efficient alpha. A high degree of consistency was found among the items which make-up each of the nine factors and results suggest that each factor is measuring a different construct. Clark and Friedman (1983) reported that in their study of 442 Veterans Administration psychiatric outpatients only five of the nine factors emerged. Hoffmann and Overall (1978) reported a similar difficulty in using a sample of 358 subjects from a general outpatient clinic population. In a study of 327 St. Louis State Hospital readmissions to the outpatient services, 12 factors emerged with one factor, "agitated depression,” being significantly stronger than any of the other eleven (Evenson, Holland, Mehta, and Yanta, 1980). While each of the above studies presented evidence that question the reliability concerning the nine scales, each author found evidence that supported the use of the SCL-90-R as a "measure of general discomfort" (Holcomb, Adams, and Ponder, 1983), or a "global distress factor" (Clark and Friedman, 1983). The diversity of the study populations provides support for the reliability and validity of the SCL-90-R questionaire as an overall measure of symptomatology. 27 ' This instrument for assigning social class standing was originally developed for the New Haven Connecticut Study (Hollingshead and Redlich, 1958). In the original instrument, three variables were used to assign social position; ecological area of residence, occupation, and education. In the revised version, only education and occupation are used to make the group assignments (Hollingshead, 1957). This instrument has been used in the reliability and validity studies for the SCL-90-R (Derogatis and Cleary, 1977; Derogatis, Lipman, Covi, and Rickels, 1971). Egg; Therapy Client Questionaire This instrument is subject to the similar biasing influences as other self-reporting techniques, such as social desirability. However, as this study wishes to examine how clients present themselves and describe the utility of therapy, generalizability of the findings need not go beyond what clients report. Whether what clients report represents how they actually perceived things is not important; Nicholson and Berman (1983) discourage the use of follow-up techniques in psychotherapy outcome studies based upon self-reports of progress. The use of the PTCQ is to obtain responses to questions of psychotherapy, regardless of truth in reporting. The same biases that influenced the responses of studies that led Nicholson and Berman (1983) to their conclusion are recognized to be occurring in the PTCO. 28 Procedure Clients seeking psychotherapy at the Psychological Clinic are provided an opportunity to participate in an evaluation of services. The request was done formally by letter or by the intake worker in person. Those who agreed to participate completed the SCL-90-R before therapy began but after a therapist had assumed reponsibility for the case. Other self-report measures, some for the therapist, were also completed at this time, but are irrelevant to this study. Audio tapes of certain sessions were also collected and retained. At termination, both client and therapist again complete these instruments and in addition complete a post therapy questionaire. The data will be analyzed as previously described. RESULTS Hypothesis 1 The first hypothesis proposed that: Those clients that are classified according to Hollingshead's Two Factor Index of Social Position as being higher in social class will be more likely to return follow-up data. Social class as calculated by Hollingshead's formula had a positive, but statistically nonsignificant correlation with the return of follow-up questionaires. The correlation of .033 suggested that individuals of higher social class were no more likely to return questionaires than individuals of lower social classes (Table l). The one-tailed ”t" statistic was calculated to be 0.24 (p>0.25, df=54). Hypothesis.; The second hypothesis proposed that: Those individuals that have a significant decrease in symptom severity or frequency as measured by the SCL-90-R pre- and post- therapy will be more likely to return follow- up questionaires. Global Symptom Indices were calculated for each of the SCL-90-R questionaires completed. A negative correlation was calculated between change in symptom severity and the return of follow-up questionaires (Table 1). Those clients reporting a greater number of symptoms after therapy than before were more likely to return follow-up questionaires. The correlation of -0.206 is opposite to the direction 29 30 Table 1 Pearson Correlations of Predictors with Return Variable Correlation "t” Social Class 0.033 0.2426 Perceived Symptom Change 0.019 0.1396 Satisfaction -0.111 -0.8207 Interest in Client 0.067 0.4934 Just Another Client 0.101 0.7460 Global Symptom Index Change -0.206 -1.547 * * 0.10>p>0.05 Table 2 Correlations between Predictor variables SocClass PercdSy Satisf Interest JustCl GSI Social Class 1.000 Percd Symtm Chnge -0.062 1.000 Satisfaction -0.088 0.715* 1.000 Interest in Client 0.017 0.652* 0.623* 1.000 Just Another Clt 0.070 0.662* 0.651* 0.861* 1.000 Global Symptom 0.154 -0.328 -0.270 -0.162 -0.121 1.000 Index Change (GSI) * p< 0.01 for first test of significance 31 predicted, but not statistically significant (0.10>p>0.05, df=54) for a one tailed ”t" test of -1.55. Hyppthesis 3 The third hypothesis proposed that: Those individuals who have a statistically greater perception of positive symptom change will be more likely to return follow-up questionaires. Perceived symptom change was assessed using five questions from the Post Therapy Client Ouestionaire. An individual with a high score on the variable indicated they have improved more than individuals who scored low on this variable score. The result suggested that no inference could be made on the basis of perceived symptom change (Table l). A correlation of 0.019 was calculated with a resulting one tailed "t" test of 0.14 that was not statistically significant (p>.25, df=54). Hypothesis 4_ The fourth hypothesis proposed that: Those clients that indicate a high level of satisfaction as compared with low satisfaction on the Post Therapy Client Questionaire will be more likely to return follow-up questionaires. A question on the PTCQ specifically asks clients about how satisfied they are with the psychotherapy they have just completed. While the correlation was in the expected direction, it was not statistically significant (Table 1). A correlation coefficient of -0.111 was found. A one tailed ”t" statistic of -0.82 was found (0.25>p>0.10, df=54). 32 H othesis g The fifth hypothesis proposed: Those clients who experience the therapist as being interested in them as a person would be more likely to return follow-up questionaires. Four questions from the PTCO were combined. Greater scores indicated increasing disagreement with each of the presented statements. A positive correlation was predicted. While a correlation was found in the expected direction, it was not statistically significant (Table l). A correlation of 0.067 was calculated using a one tailed "t" test (p>.25, df=54). Hypothesislg The sixth hypothesis proposed that: Those individuals who experienced the therapist as being interested in them as ”just another patient" would be significantly less likely to return follow-up questionaires. Six questions from the PTCQ were averaged and prorated on a scale from one to ten. Low scores indicate a positive experience with their therapist and a negative correlation is expected. A correlation of 0.101 was calculated and not found to be statistically significant using a one tailed ”t" test (Table 1, 0.25>p>0.10, df=54). Combining Correlated variables A correlation matrix was created to examine the strengths of correlations between the six variables examined (Table 2). Four variables were found to have intercorrelations between 0.651 and 0.861. They include 33 client's ratings on the following scales: feeling like "just another client"; believing that the therapist was ”interested in them as a person”; the amount of symptom change perceived; and the level of satisfaction experienced from therapy. The other two variables, social status and symptom change over time, were not significantly correlated to each other or to any of the previously mentioned variables. The Component Loadings of each variable in Principal Component Analysis suggested the creation of three ”clusters" out of the original six variables (Table 3). The four intercorrelated variables had a significantly high loading on the first "cluster." They were broadly considered to be the "Client's Perceptions" of the therapeutic experience. Factor loadings ranged from 0.851 to 0.887 for each of the four variables and the amount of variance explained by the first factor was 31.74%. The second factor constructed contained a high loading for Social Class. and had a component loading of 0.792 and accounted for 11.7% of the variance. The third factor created contained a high loading on symptom change over time and had a loading of 0.653. This factor accounted for eight percent of the variance. Together the factors accounted for 52.44% of the variance in the data. Standard scores were created from the raw data averaging the variables that made up each ”Cluster." Cluster 1 became the average of the four standard scores from each of the four variables that comprise it. Cluster 2 34 Table 3 Component Loadings of Predictor variables Using varimax Rotation Correlation with Componentl Component2 Component3 Return Just Another Client Interest In Client Percd Symptom Change Satisfaction Social Class 0.887* 0.882* 0.869* 0.851* -0.049 Global Symptom Index -0.355 0.260 0.196 -0.083 -0.074 0.796* 0.644 0.116 0.122 -0.050 0.042 -0.597 0.653* Change (GSI) * Indicates the variables in each cluster. Table 4 0.101 0.067 0.019 -0.111 0.033 -0.206 Discriminant Function Analysis Beta Weights for Clusters variable Beta Coefficient Constant 0.482 Clusterl -0.019 Cluster2 0.034 Cluster3 -0.113 35 and Cluster 3 became the standard scores of the variables Social Class and Symptom Change Over Time respectively. A discriminant function analysis provided Beta weights for each of the three clusters. The weights and the constant provided a Multiple R value of .219, indicating that 4.8% of the variance can be explained by this model (Table 4). None of the three clusters reached statistical significance. Cluster 3, or Symptom Change Over Time, was calculated closest to significance (p= 0.118). Predicting Return The model from the discriminant function analysis and the standardized cluster scores were combined to calculate return behavior (Table 5). There were 27 questionaires returned and 25 predicted returns of the 56 cases examined. The model correctly predicted 15 actual returns (true positives) and 19 actual non-returns (true negatives). The model correctly identified the outcome of 34 of the 56 cases. A sixty percent "hit” rate was calculated to be within two standard deviations of a probability of 0.5, or chance occurrence (2 = 1.71, 0.10>p>0.05). Ten clients were incorrectly predicted to have returned the questionaires when they did not (false positives), and twelve clients were incorrectly identified to have not returned their questionaires when in fact they did (false negatives). The Chi—square statistic resulting from the 2 x 2 table was calculated to be 1.81 (0.10>p>0.05). The validity of the model can be further assessed in two ways. The sensitivity of the model describes the 36 Table 5 Predicted Return vs. Actual Return Actual Return Estimated Return Returned Not Returned Return P U! 0'3 "U H M an 2 I Not Returned I 10(FP) l 19(TN) l l TP=True Positive, FN=False Negative, FP=False Positive, TN=True Negative. Chi-squared=l.808, 0.25>p>0.10 TP + TN = 34 or 60%, z=1.71, 0.10>p>0.05 SensitivitysTP/(TP+FN)=55.5% Specificity=TNl(TN+FP)=65.5% 37 ability of the model to correctly identify returned questionaires. Of those questionaires that were returned, 55.5% of them were correctly identified by the model, or conversely, the model had a false negative rate of 44.5%. The specificity of the model is the ability of the model to correctly identify non-return. Of those questionaires that were not returned, the model correctly identifies 65.5% of them, or conversely, has a false positive rate of 34.5%. The change that might be observed in the multiple correlation with subequent sampling was estimated by calculating the squared population cross validity (Schmitt, Coyle, and Rauschenberger, 1977). Three formulas to calculate this fluctuation are presented, the Wherry statistic was chosen. A fluctuation of 0.0788 is possible in the square multiple R of 0.048 with these variables, 56 subjects, and three cluster predictors. The results suggest that subsequent sampling would probably not identify this set of variables as accounting for a major proportion of the variance to explain the return of follow-up questionaires by former psychotherapy clients six months after the end of treatment. DISCUSSION The first hypothesis was not statistically supported though the direction was correctly identified. Individuals rated as being of higher social class were no more likely to return questionaires than were individuals judged to be from a lower social class. On examination of the sample consistency, this study did not suggest that the results were clouded by social class variables. Self—addressed stamped envelopes were included in all of the follow-up packets mailed at six months post treatment. The range of social class rankings spanned the full range as outlined by Hollingshead's two factor index. The mean ranking was toward the upper end of the third of five categories suggesting a normal distribution of individuals from the spectrum of social classes. These findings add to the conflicting results previously described concerning the influence of social class on return rates in the psychotherapy outcome literature. Positive results in other studies suggest that social class may be an important contributor in some cases. In this study, an interaction may have occurred with other variables which have either not been identified or whose influence has not been adequately explored, but social class, by itself, was not a contributing variable. 38 39 The second hypothesis predicted that individuals who indicated improvement from pre-test to post-test would be more likely to return follow-up questionaires. A statistically nonsignificant trend in the opposite direction was found indicating that individuals who had experienced more symptoms were more likely to return questionaires than those who had improved. One could speculate that those individuals who had experienced less symptoms at the end of therapy than at the beginning were not as invested in providing feedback. The ”out of sight, out of mind” proverb suggests itself as these clients had received some relief from their symptoms and may have changed their focus away from their therapy by the time follow-up questionaires arrived six months after therapy's end. Clients who had indicated more symptoms may have been just as aware of their symptoms six months after therapy as at the time of its end encouraging more responses from this group than the former. But all this is clearly speculation given the absence of significant findings. I The third hypothesis found no significant correlation between a client's perception of symptom change at the end of therapy and the return of questionaires. A bimodal distribution of the raw data would account for the results as individuals indicating either extreme in symptom change would have sufficient motive to return questionaires. Their combined effect could be sufficient to offset one another. An examination of the raw data found a distribution skewed slightly towards those reponses which indicate a positive 40 change in their symptoms and appears to reject this alternative explanation. A more likely explanation is that clients' perception of symptom change is not sufficient to account for the return of client's follow-up questionaires. The second and third hypotheses both attempted to correlate symptom change with the return of follow-up questionaires. Neither clients' perception of change over the course of therapy nor differences between data gathered pre- and post-therapy were able to account for the data received to an accepted level of statistical significance. In contrast to the results that clients' perceptions of symptom change did not aid in predicting return, change in pre- and post- measures did provide some aid in identifying an increase in the rate of return of questionaires. The fourth hypothesis was not statistically supported by the data, though the trend was in the expected direction; clients who indicated they were satisfied with the results of their therapy were more likely to return the questionaires. These results appear to conflict with those of the second hypothesis. Were one to assume these to be ”true" results, it would seem contradictory that individuals can be both satisfied with their therapy and indicate that symptoms remain or have increased. One explanation is that the course of therapy encourages awareness before symptoms can abate, and that therapy may intensify symptoms as a natural course to their relief. A second explanation suggests that the client may have experienced sufficiently severe disturbance pre-therapy that 41 some symptomatic relief during therapy may have been enough to satisfy the client. The follow-up questionaires may have served as a trigger to both remind the client how painful things used to be and encourage them to return the survey . materials. A third explanation to these apparently contradictory results is that the two variables differ in strength. It is just possible that client satisfaction is a more potent variable than is symptom relief. ”Chairside Manner” may be worth exploring further. A fourth explanation suggests that client's may have had something to be satisfied about and that therapeutic work may not have been completed. The client may not have directly focused on issues that were connected with the symptoms. The client might have been involved in satisfying therapeutic work and remained focused on peripheral issues. This scenario would allow for the clients to have successfully avoided their central complaints and be satified about their therapy experience while not having reduced their symptoms. The fifth hypothesis was not statistically supported as those clients who experienced their therapist as being "interested in them as people" were not significantly more likely to return follow-up questionaires. A trend was found that supported the hypothesis and suggests that a client's perception of their therapist as having a positive opinion of them may have some influence in client's decision to return follow-up questionaires. 42 The sixth hypothesis was not statistically supported by the data in this study as clients who described themselves as "just another patient" were not significantly more likely to return questionaires. A trend in the predicted direction was found that supported the hypothesis that client's perceptions of how they were being treated by their therapist may have some influence in predicting their behaviors. Hypotheses five and six test the influence of positive and negative perceptions the client had of their therapist's opinion of them. In comparing the impact of each, it appears that a stronger, but still statistically nonsignificant, correlation was obtained when client's perceived their therapist as treating them as "just another patient.” It is these clients that were more likely to return questionaires. As suggested above as an alternative explanation for contradictory results, the behavior of the clinician may have some impact in determining the return of questionaires. While not statistically supporting the hypotheses, the trends suggest that clients who return follow-up questionaires tend to experience similar or increased symptoms when compared to the start of therapy, and indicate some satisfaction with their therapeutic experience. In addition, the results of hypothesis six suggests that clients' responding may also be enhanced should the clients perceive themselves as having been treated as "just another patient.” 43 Intercorrelations between the variables suggested that combining them into clusters may have created more robust variables. The three clusters created were Client's Perceptions, Social Class, and Symptom Change Over Time and were found to account for a small portion (4.8%) of the variance in this model. This suggests that though the variables in each cluster are significantly correlated, they do not sufficiently account for the return of follow-up questionaires. In examining the model further, sensitivity and specificity statistics indicate that the model is better at identifying which questionaires are least likely to be returned. The model is fifteen percent better than chance would predict at using information available at post—therapy to identify a client who will not return their followbup questionaire. These results become more significant as biases which inhibit return are examined. Clients may never receive the packet as they may get lost in the mail, or they may not be delivered because of the lack of a forwarding address. Once received, they may be lost or inadvertently destroyed. If the pattern of these events can be assumed to be random, then the model's ability to predict non-return may be more significant than the results initially suggest. This study attempted to identify those variables that may influence the return of follow-up questionaires in therapy clients. The argument made by Nicholson and Berman (1983) suggests that the results which follow-up studies 44 provide do not justify their use. It was initially observed that few mailed questionaire studies achieve one hundred percent return and therefore allow the possibility of methodolgical bias to skew their results. The results of this study does not provide statistically significant evidence of an identifiable bias occurring in psychotherapy outcome studies using mailed questionaires. However, the absence of statistically significant evidence to contradict Nicholson and Berman (1983) does not provide evidence that bias is not present in mailed questionaire studies. This study supports the findings that bias in the return of mailed questionaires has not yet been identified. Of major concern is the possibility that follow-up studies will be discontinued without a definite understanding of what major variables contribute to the behavior of returning a follow-up questionaire. The field will then be left with a potentially false belief based on the response of a specific, and as yet, unidentified population. On the other hand, for those who insist that no intervention study is complete unless there is follow-up, the current study does not provide support for that acceptance. The follow-up phase of a study can be a sizable portion of a studies allotted time, money, and manpower. It may be necessary to examine the cost-benefit of investing those resources in the follow-up of subjects against the value of the information that might be obtained. Attempting to obtain information that is not routinely collected about 45 subjects years after their involvement can lead to serious difficulties. Subjects may have long disappeared or may not wish to participate to name two obstacles. One would have to consider seriously the costs involved before committing the admittedly scarce resouces for what may turn out to be limited benefits. At this point in time such an effort can be convincingly argued to be not cost-effective. However, if the information can be collected with relatively little expense or is routinely collected by someone else, it would be reasonable to examine the data further. BIBLIOGRAPHY BIBLIOGRAPHY Bebbington, A.C. (1970). The effect of non-response in the sample survey with an example. Human Relations, 21(23), 169-180. Bennett, C.M., & Hill, R.E.Jr. (1964). A comparison of selected personality characteristics of responders and non-responders to a mailed questionaire study. Journal 2; Educational Research, 58, 178-180. Blumberg, H.H., Fuller, C., & Hare, A.P. (1975). Response rates in postal surveys. Public Opinion Quarterly, 38, 113-123. Brown, C.E. (1984). Own group bias in survey return rate: An unobtusive attitude measure. Psychological Reports, Campbell, D.T., a Stanley, J.L. (1963). Eypgpippppgl,gpg_ guasi-experimental design for rgsearch. Boston: Houghton Mifflin. Clark, A., & Friedman, M.J. (1983). Factor structure and discriminant validity of the SCL-90 in a veteran Psychiatric Population- 193ml 21 W Assessment, 11(4), 396-404. Cooley, E.J., s Lajoy, R. (1980). Therapeutic relationship and improvement as perceived by clients and therapists. m; of C_lini_e_c 1 amnion. 1M2). 562-570- Crutchfield, R.S. (1955). Conformity and character. App;iggp,Psychologigt, 19, 191-198. Derogatis, L.R. (1977). pp; SCL-90 appual L; Scoping, administration, ppg procedures fig; pp; SOL-90. Baltimore, MD: Johns Hopkins University School of Medicine, Clinical Psychometrics Unit. Derogatis, L.R. (1977). Confirmation of the dimensional structure of the SCL-90: A study of construct validity. gourppl g;_Qlinical Ps chol , 3;, 981-989. Derogatis, L.R., Lipman, R.S., & Covi, L. (1973). SOL-90: An outpatient psychiatric rating scale preliminary report. Psychgpharmagglogy Bulletin, 2, 13-28. 47 48 Derogatis, L.R., Lipman, R.S., Covi, L., & Rickels, K. (1972). Factor invariance of symptom dimensions in anxious and depressive neuroses. Archives gg_General Psychiatry, 21, 659-665. Derogatis, L.R., Lipman, R.S., Covi, L., & Rickels, K. (1971). Neurotic symptom dimensions as perceived by psychiatrists and patients of various social classes. Archives 2; Geperal Psychiatry, 21, 454-464. Derogatis, L.R., Lipman, R.S., Covi, L., Rickels, K., & Uhlenhuth, E.H. (1970). Dimensions of outpatient neurotic pathology: Comparison of a clinical versus an empirical assessment. Journal 2; Cppsulping gpg_ Qiinisal.2:xshelesxp.1i. 164-171- Derogatis, L.R., Lipman, R.S., Rickels, D., Uhlenhuth, E.H., & Covi, L. (1974). The Hopkins Symptom Checklist: A self-report inventory. Behaviprgl fig;gpgg, 12, 1-15. Evenson, R.C., Holland, R.A., Metha, 8., 5 Yasif, F. (1980). Factor analysis of the Symptom Checklist 90. Epygpological R ts, 4g, 695-699. Fuller, C.H. (1974). Weighting to adjust for survey nonresponse. Publig Qpipiog Qpapperly, 1;, 239-246. Gannon, M., Nothern, J., & Carrol, 3. Jr. (1971). Characteristics of non-response among workers. Journal 2;. 392141.41 who 0 : 5.5.: 535-5”- Garfield, S.L. (1983). Effectiveness of psychotherapy: The perennial. controversy- Ersfesaienel.2axsbelesx, 11(1). 35-43. Gough, H.G., & Hall, W.B. (1977). A comparison of physicians who did or did not respond to a postal questionaire. gppgpgl_g§_App;;gg_£sychg;ogy, 61(6), 777-780. Heberlein, T.A., a Baumgartner, R. (1978). Factors affecting response rates to mailed questionaires: A questionaire analysis of the published literature. Appricgn 222121291211.Rexie2, 11. 447-462- Hinrichs, J.R. (1975). Factors related to survey response rates: Effects of sampling follow-up letters and commitment to participation in mail attitude survey responses. Journal 2; Applied ngchology, pp, 249-251. Hoffman, N.G., 5 Overall, P.B. (1978). Factor structure of the SOL-90 in a psychiatric population. Journal 2;, Qgpgultipg gpg_glipiggl_2sychglggy, 16, 1187-1191. 49 Holcomb, W.R., Adams, N.A., a Ponder, H.M. (1983). Factor structure of the Symptom Checklist-90 with acute psychiatric inpatients. qurngl 91 92% m Clinical Ps cholo , §l(4), 535-538. Hollingshead, A.B. (1957). Two fpcto; lpggy,pi_pocial position. Unpublished manuscipt. Yale University. Hollingshead, A.B., and Redlich, F.c. (1958). Social class and mental illness: A community study. New York:J. Wiley. Howard, G.S., Millham, J., Slaten, 8., & O'Donnell, L. (1981). Influence of subject response style effects on retrospective measures. Appllgg,gpygpplpglg§l, Measurement, 5(1), 89-100. Jacobs, J.A., Aronson, H., & Nystrom, B. (1983). Recidivism as a follow-up methodology. Journal 2; Community P cholo , ll(3): 224-227. Kanuk, L., a Berenson, C. (1975). Surveys and response rateS- M3 na 2;. WM. 1.2. 440-453- Kirchner, J.R., & Hogan, R.A. (1982). Patient feedback at a community mental health center: Year three. W 21191121931, 11(3). 431-438- Kivlin, J. (1965). Contributions to the study of mail-back bias. Rurgl Sgciology, 39, 322-326. Kleinbaum, D.G., & Kupper, L.L. (1978). Appllpg,rggression gpalygls ppg_p;pgp,ppltivlgtg pptpgds. North Scituate Massachusettes:Duxbury Press. Konske, P., Staple, 8., & Graf, R.G. (1979). Compliant reactions to guilt: self-esteem or self-punishment. 19.13921 9!. 39.2111 Swagger. 11.8(2116 half). 207-211- Larson, R.F., & Catton, W.R. Jr. (1959). Can the mail-back bias contribute to a study's validity? American W Series. 2.4.. 243-245- Lehman, E.c. Jr. (1963). Tests of significance and partial returns to mailed questionaires. Rural agglplpgy, 22, 284-289. Linsky, A.S. (1965). A factorial experiment in inducing responses to a mailed questionaire. Sociology and figslgl,3egggrch, 12, 183-189. Lubin, 8., Eugene, E.L., a Zuckerman, M. (1962). Some personality differences between respondents and non- respondents to a survey questionaire. gourpgl 2; Consulting gpg Clinical Ps cholo , 2;, 192. 50 Macek, A.J., a MIles, G.H. (1975). I.O. score and mailed questionaire response. Journal 2; Applied Psychology, 62, 258-259. Mandell, L. (1974). When to weight: Determining nonresponse bias in survey data. prllc Qpipion ngygpgly, 18(2), 247-252. Marziali, E. (1984). Three viewpoints on the therapeutic alliance: Similarities, differences, and associations with therpaeutic outcome. Journal of Nervous and Mental legpgg, lvzt7), 417-423. ‘- Mausner, J.S., a Bahn, A.K. (1974). Epidgplglogy: 5p lntrpgpgtory Igy_. Philadelphia:W.B. Sanders. Nelson, R.O., (1981). Realistic dependent measures for clinical use. Journal p;_Consulting and Clinical Psygholggy, 12(2), 168-182. Nicholson, R.A., s Berman, J.S. (1983). Is follow-up necessary in evaluating psychotherapy? Egychology Eplletin, 21(2), 261-278. Ognibene, D. (1970). Traits affecting questionaire response. Journgl pg Advgrtising Resear h, lg(3), 18- 20. Pilkonis, P.A., Imber, 8.0., & Rubinsky, P. (1984). Influence of life events on outcome in psychotherapy. Journal g; Nervous gnd Mental Qiegsg, 172(8), 468-474. Reuss, C.F. (1943). Differences between persons responding and not responding to mail questionaires. American Sogiological Rgview, 8, 433-438. Robins, L.N. (1963). The reluctant respondent. zpblic Opinion Quarterly, 21, 276-286. Rosen, C.E. (1978). The compliant client. Man and Medicine, 3(1), 1-7. Sargent, H. (1960). Methodological problems of follow-up studies in psychotherapy research. App;lggp,§pg;pgl g; Orthgpsychiagry, 39, 495-506. Schmitt, N., Coyle, B.W., a Rauschenberger. (1977). A Monte Carlo evaluation of three formula estimates of cross- validated multiple correlation. Egygpplpglgpl_anllg§lp; §1(4), 751- 758. Schramski, T.G., Beutler, L.E., Lauver, P.J., Armizmeldi, T.A., & Shanfield, 8.8. (1984). Factors that contribute to posttherapy persistence of therapeutic change. gggppgl_pglglipigal Psychology, 10(1), 78-85. 51 Schwirian, K.P., & Blaine, H.R. (1966-67). Ouestionaire return bias in the study of blue collar workers. ggblic Opinlon Quarterly, 55, 656-663. Sirles, R.A. (1984). Who responds to follow-up studies? Spgial Casework, 55(6): 354-356." benefits g£_Psychothgrapy. Baltimore:Johns Hopkins University Press. Vincent, C.E. (1964). Socioeconomic status and familiar variables in mail questionaire responses. American Wallace, D. (1954). A case for and against mail questionaires. Public Opinion Quarterly, l5, 40-52. Winokur, A., Guthurie, M.B., Rickels, H., & Nael, S. (1982). Extent of agreement between patient and physician ratings of emotional distress. Psychosomatics, 55, 1135-1146.