INFORMATION TO USERS Hits was produced from a copy of a document sent to us for microfilming. While the most advanced technological means to photograph and reproduce this document have been used, the quality is heavily dependent upon the quality of the material submitted. The following explanation of techniques is provided to help you understand markings or notations which may appear on this reproduction. 1.The sign or "target'' for pages apparently lacking from the document photographed is "Missing Page(s)". If it was possible to obtain the missing page(s) or section, they are spliced into the film along with adjacent pages. This may have necessitated cutting through an image and duplicating adjacent pages to assure you of complete continuity. 2. When an image on the film is obliterated with a round black mark it is an indication that the film Inspector noticed either blurred copy because of movement during exposure, or duplicate copy. Unless we meant to delete copyrighted materials that should not have been filmed, you will find a good image of the page in the adjacent frame. 3. When a map, drawing or chart, etc., is part of the material being photo* graphed the photographer has followed a definite method in "sectioning” the material. It is customary to begin filming at the upper left hand comer of a large sheet and to continue from left to right in equal sections with small overlaps. If necessary, sectioning is continued again^beginning below the first row and continuing on until complete. 4. For any illustrations that cannot be reproduced satisfactorily by xerography, photographic prints can be purchased at additional cost and tipped into your xerographic copy. Requests can be made to our Dissertations Customer Services Department. 5. Some pages in any document may have indistinct print. In all cases we have filmed the best available copy. University Microfilms International 300 N. 2EE8 ROAD. ANN ARBOR. Ml 48106 IB BEDFORD ROW. LONDON WC1R 4EJ. ENGLAND 79^7^25 I WE I N S T E I N r H A R O L D P H I L I P THE IMPACT OF F A C I L I T Y - B A S E D V O C A T I O N A L EVALUATION SERVICES ON REFERRING MICHIGAN B U R E A U OF R E H A B I L I T A T I O N C O U N S E L O R S . M I C H I G A N S T A T E U N I V E R S I T Y * P H . D . , 197(1 Untarcftv , M tadflm s nerrutkxvtl x» n z t t a n o A o , a n n A n s o n , mi w k k © Copyright by HAROLD PHILIP WEINSTEIN 1978 THE IMPACT OF FACILITY-BASED VOCATIONAL EVALUATION SERVICES ON REFERRING MICHIGAN BUREAU OF REHABILITATION COUNSELORS By Harold Philip Weinstein A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Counseling, Personnel Services and Educational Psychology 1978 ABSTRACT THE IMPACT OF FACILITY-BASED VOCATIONAL EVALUATION SERVICES ON REFERRING MICHIGAN BUREAU OF REHABILITATION COUNSELORS By Harold Philip Weinstein The purpose of this study wa 3 to explore the impact of information generated by facility-based work evaluation services on referring state vocational rehabilitation counselors. fined Impact, for purposes of measurement, was de­ as information the counselor gained as a result of evaluation, increases in certainty regarding information counselors possessed as a result of work evaluation, and the degree to which information gained was considered useful in planning for client services. A second focus of the study was concerned with the construction of an instrument designed to measure information counselors possess with regard to their clients' vocational assets and liabilities. Thirty rehabilitation counselors employed by the Michigan Bureau of Rehabilitation served as subjects for this study. Eligible counselors were those having one client on their active caseload who was participating in a three-or four-week facility-based work evaluation (experimental) and a similar client, in terras of age, gender, Harold Philip Weinstein and primary disability, who had not participated, or would not be participating in work evaluation until sometime after the first client completed evaluation services (control). The amount of information counselors possessed concern* ' j ing each client's vocational assets and liabilities was assessed during the first five days of evaluation services for the experimental ellent. • * During the period of time ! that the experimental client was in an evaluation program, counselors maintained a log of contacts made with or on behalf of both the experimental and the control client. Information counselors possessed concerning each client's vocational assets and liabilities was again assessed following the experimental client's final staffing at the facility. Measures used in this study included a demographic data | survey used to define both counselor and client character­ istics, a pre/post administration of the Client Assessment | Survey used to measure the type and amount of information i counselors possessed regarding their olients' assets and liabilities, a posttest data survey, and finally, the contact logs described earlier. The Client' Assessment Survey consisted of 75 items grouped into nine a priori scales. Demographic comparisons between experimental and control client groups revealed two significant differences of 18 analyzed. Multivariate analysis of variance revealed that counselors gained significantly more information and Harold Philip Weinstein certainty (P less than .05) regarding information they possessed for clients who attended evaluation than for a matched group of clients who did not attend evaluation. Significant differences occurred between experimental and control client groups on six of nine a. priori scales. Only the physical capacities and Job seeking skills scales failed to show significance at the .05 level. Analysis of certainty gained revealed significant difference on all of the a priori scales. An estimate of inter-rater reliability revealed three independent samples of individuals completing a degree of usefulness sorting task to be homogeneous. However, ratings of item usefulness were not significantly related to item information gain. Exploratory examinations of various demographic variables on the amount of Information gained revealed no significant relationships between information gained and the number and type of counselor oontacts made with or on behalf of their clients, the number of days clients attended evaluation services, primary disability type, secondary disability, or counselor caseload type. A slight relation­ ship was found between the number of years of experience the counselor possessed and information gained on the Job Retention Skills scale. An inverse relationship was found between the amount of information gained on a number of a priori scales and the number of reports avail­ able for the counselor at the pretest interview. Finally, Harold Philip Weinstein analysis of variance revealed information gained on the Client Aptitude scale was highest for counselors who referred clients primarily to aid in determining feasibility or to aid in developing a tentative vocational objective. DEDICATED To Debbi ACKNOWLEDGEMENTS The completion of thi3 dissertation has involved the support of many different people. 1 would first like to express my deepest appreciation to my wife, Debbi, for her love, understanding, encourage­ ment and total support throughout my doctoral program. I would also like to express my appreciation to Dr. Anne L. Chandler whose friendship, guidance, and support was instrumental in my successfully completing my doctoral studies. To Dr. F, Robert Wilson I extend my gratitude and respect for his friendship and contributions toward developing a workable research methodology. I would also like to express my appreciation to Dr. Gregory Miller, my advisor and dissertation chairman for his honesty, support, and guidance throughout my doctoral studies. I wish to thank the other members of my committee as follows: to Dr. James R. Engelkes for his support and excellent guidance throughout my doctoral program as well as his interest in helping me to develop, carry-out, and write my dissertation; to Dr. Charles C. Coker for his encouragement and support of the research project; iii to Dr. Herbert H. Burks, Jr. for his excellent editorial skills; and to Dr. Bertram Karon for his willingness to serve. Also, I would like to thank the many people in the field who contributed their time and resources to the successful completion of this research project. In particular, I would like to thank the Michigan Bureau of Rehabilitation for its support in conducting this research $ through the efforts of Mr. Pete Griswold and Mr. Ray Hinz as well as the many district office supervisors and counselors who actually participated in the study. In addition, I would like to extend my appreciation to Mr. Charles Foster of the Michigan Association of Rehabilitation Facilities and its various member agencies for their support and cooperation. My sincerest appreciation is extended to those individuals who volunteered to be research assistants and worked many long hours during the period of time in which data collection took place. Specifically, I would like to thank David Brown, William Crimando, John Grants, Jill Laufer, Michael Lieberroan, John Moore, Cathy O'Neill, Diane Sweeney, Larry Urban, Terry Vander-Molen, and Margaret Van Gelder. Finally, I wish to gratefully acknowledge the partial support for this research provided by the Research and Training Center at the University of Wisconsin-Stout iv through Grant no. 16-P-56821/5, from the Rehabilitation Services Administration, Department of Health, Education, and Welfare, Washington, D. C. v TABLE OF CONTENTS Page LIST OF T A B L E S .................................... viii LIST OF F I G U R E S ................................. xi LIST OF APPENDICES................................. xii Chapter I. II. THE P R O B L E M ............................. 1 N e e d .................................. Purpose .............................. Hypotheses ............................ Definition of Terms .............. Overview . . . . ■...................... 1 5 5 7 REVIEW OF THE LITERATURE................. S u m m a r y .............................. III. 6 9 22 M E T H O D O L O G Y ............................. 26 Population Defined .................... Sample Selection ...................... Identification ofFacilities . . . . Identification ofClients ......... Identification ofEligible Subjects. Characteristics of Counselor Participants ........................ Instrumentation ...................... Client AssessmentSurvey (CAS) . . . Method of Administration .......... Validity.......................... R e l i a b i l i t y ...................... Scale H o m o g e n e i t y ................ •Client AssessmentSurvey Gain Scores Shift S c o r e s ...................... Certainty Scores .............. Summary Scores .................... CAS Scale Homogeneity Using Gain Scores I n d i c e s............................... Other Data Collected.................. Demographic Data Survey (DDS) ... Posttest Data Survey.............. 26 26 27 vi 28 29 32 32 33 3^ 37 39 48 51 51 53 55 55 56 56 63 Page Chapter Contact L o g ....................... Sorting T a s k ....................... Pilot S t u d y ........................... Interviewer Training .................. Data Collection Procedures ............ Pretest Interview ................ Posttest Interview .............. Hypotheses...................... . . . . Research Design and Statistical Analysis S u m m a r y ............................... IV. R E S U L T S ........................ Client Characteristics ................ Hypotheses Tested ..................... Trends in the D a t a ..................... Relationship between Information Gained and Counselor Contacts .............. Relationship between Information Gained and Counselor Experience ............ Relationship between Information Gained and the Number of Reports in the Clients' Files at the Pretest In t e r v i e w .................... .. Relationship between Information Gained and the Number of Days the Client Attended Work EvaluationServices ... Effect of Disability Type onInformation G a i n e d .......................... 98 Effect of Counselor Caseload Type on Information Gained ................... Summary of R e s u l t s ................. 103 V. 63 64 64 65 66 66 68 71 71 75 78 78 84 90 92 93 94 96 98 SUMMARY AND C O N C L U S I O N S ............ 107 Discussion......................... 111 Limitations of the S t u d y ........... 115 Implications for FurtherResearch ... C o n c l u s i o n s ....................... 120 118 APPENDICES................................... 122 BIBLIOGRAPHY ...................................... vii 156 LIST OF TABLES Table 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 Page Arrangement of Items Into Scales on the Client Assessment Survey .............. 36 Test-Retest Reliabilities for Independent Reliability Sample (N=12) .............. 40 Test-Retest Correlations for the Degree of Certainty Task Broken Down into the A Priori S c a l e s .................... 7 .. 43 Average Item Test-Retest Correlations for the Problem Identification and Combined Problem Identification and Degree of Certainty Tasks Grouped According to A Priori Scales ........................ 44 Test-Retest Correlations of the A Priori Scales on the Degree of Certainty Task for the Work Evaluation, No Work Evaluation, and Combined Work Evaluation/ No Work Evaluation G r o u p s .............. 46 Test-Retest Correlations of the A Priori Scales on the Combined Problem” Identification and Degree of Certainty Tasks for Work Evaluation, No Work Evaluation, and Combined Work Evaluation/ No Work Evaluation G r o u p .............. 47 Pre- and Posttest Coefficient Alpha Estimates of Scale Homogeneity on the Degree of Certainty Task for the Combined Work Evaluation/No Work Evaluation Groups (N=60)................ 49 Pre- and Posttest Coefficient Alpha Estimates of Scale Homogeneity on the Combined Problem Identification and Degree of Certainty Tasks for the Work Evaluation, No Work Evaluation, and Combined Work Evaluation/No Work Evaluation Groups ...................... 50 viii Table 3.9 3.10 Page Coefficient Alpha Homogeneity Estimates for the £ Priori Scales of the Work Evaluation and No Work Evaluation Groups Using Item Shift S c o r e s .......................... 57 Coefficient Alpha Homogeneity Scales for the Work Evaluation and No Work Evaluation Groups Using Item Certainty S c o r e s ................................ 58 3.11 Coefficient Alpha Homogeneity Estimates of the A Priori Scales for the Work Evaluation and No Work Evaluation Groups Using Item Summary Scores . . . . 3.12 3.13 3.14 3.15 4.1 4.2 4.3 4.4 59 Intra-Scale Correlations for the A Priori Scales of the Work Evaluation Group Using Item Summary S c o r e s ........ 60 Intra-Scale Correlations for the A Priori Scales of the No Work Evaluation Group Using Item Summary S c o r e s ........ 6l Intra-Scale Correlations for the A Priori Scales of the Combined Work Evaluation/ No Work Evaluation Groups Using Item Summary S c o r e s .......... . ............ 62 Repeated Measures Design . . . . . . . . . 73 Chi Square Comparisons of Experimental and Control Clients on Demographic Characteristics ........................ 79 Chi Square Comparisons of Number and Types of Contacts with or on Behalf of Experimental and Control Clients .... 83 Summary of Mean Differences and Multivariate Analysis of Variance for Repeated Measures Designs Associated with Information Gained ................ 86 Summary of Mean Differences and Multivariate Analysis of Variance for Repeated Measures Designs Associated with Certainty Gained .................. 88 ix Table 4.5 Page Correlation Matrix of Mean Item Usefulness Ratings Across Three Independent Samples and with Mean Item Information Scores ...................... 4.6 4.7 4.8 4.9 4.10 4.11 4.12 91 Pearson Product-Moment Correlations between Amount of Information Gained and Counselor Experience .................. 94 Pearson Product-Moment Correlations between Amount of Information Gained and the Number of Reports Available to the C o u n s e l o r s ............................ 95 Pearson Product-Moment Correlations between Amount of Information Gained and the Number of Days Clients Attended Work E v a l u a t i o n ............................ 97 One-Way Analysis of Variance for Information Gained as a Function of Primary Disability Type and the Presence, Absence, or Unknown Status of a Secondary D i s a b i l i t y ............................ 99 One-Way Analysis of Variance for Information Gained as a Function of Counselor Caseload Type ................ 100 Multivariate Analysis of Variance for Repeated Measures Designs Associated with the Effect of Primary Referral Question on Information Gained .................. 101 Mean Gain and Standard Deviation for Each of the Five Referral Reasons Associated with the Client Attitude S c a l e ...........102 x LIST OF FIGURES Figure 3.1 3.2 3.3 Page Schematic for Item Scoring on the Client Assessment Survey ...................... 38 Range of Computational Values for the Combined Problem Identification and Degree of Certainty T a s k .............. 41 Diagram of Scoring System to Summarize Movement on the CAS across All Tasks Between the First and Second Adminis­ trations .............................. 52 xi LIST OF APPENDICES Appendix A Page Standardized Presentation Format for Identifying and Selecting Eligible Counselor Participants .................. 123 B The Client Assessment Survey ............... 127 C The Demographic Data S u r v e y ................. 137 D The E The Contact L o g ........................... 143 F Directions for Completing the Sorting Task and the Sorting Record F o r m ............ 145 Instructions for Conducting the Pretest I n t e r v i e w .............................. 148 G Posttest Data S u r v e y .................. 141 H The Model Contact L o g ....................... 151 I Instructions for Conducting the Posttest I n t e r v i e w .............................. xii 153 CHAPTER I THE PROBLEM Need Traditionally, vocational rehabilitation has been defined as the process of restoring an individual to the fullest physical, social, mental, vocational, and economic usefulness of which he is capable. Its specific goals have focused upon the "cultivation, restoration, and conservation of human resources" (Whitehouse, 1955). Inherent in the rehabilitative process and prerequisite to the provision of restorative services, the rehabilitation potential of the individual must be assessed and evaluated. As mandated by the Rehabilitation Act of 1973 (PL 93-112), this assessment includes: (a) an initial evaluation to determine that the individual has a substantial handicap to employment, and that vocational rehabilitation services are needed; (b) a thorough and comprehensive review of pertinent medical, cultural,. social, and environmental factors relevant to the individual's handicap to employ­ ment and rehabilitation potential, as well as any other pertinent data which might be helpful in determining the nature and scope of rehabilitation services; (c) an evaluation of the individual's work behaviors including potential for acquiring occupational skills and developing appropriate work habits, tolerances, and behavior patterns for successful job performance. This may include the utilization of real or simulated work to assess and develop the individual's work capacities; (d) any other goods or services which may aid in ascertaining the nature of the handicap and whether the individual can be expected to benefit from additional vocational rehabil­ itation services (PL 93-112, 1973). In the early days of vocational rehabilitation, it was the state vocational rehabilitation counselors who assumed primary responsibility for determination of their clients' rehabilitation potential. Through a process of collecting and synthesizing data received from physicians, social workers, psychologists, former employers, the clients' fam­ ilies, and the clients themselves, the counselors were able to determine the clients' eligibility and feasibility for services, as well as plan a meaningful sequence of services directed toward maximizing the clients' ability to secure and maintain employment commensurate with their abilities. However, in the first half of the 1950's, vocational reha­ bilitation entered into a phase of accelerated growth, expanding its programs and services while at the same time extending them to more difficult cases. Indeed this trend seems to have continued through the present as a mandate of the Rehabilitation Act of 1973 (PL 93-112, 1973). With the extension of services to more difficult cases, it was necessary for the rehabilitation counselor to deal with case complexities never before encountered. As stated in the final report of the Vocational Evaluation Project, sponsored by the Vocational Evaluation and Work Adjustment Association with the assistance of the National Rehabil­ itation Association and the University of Arizona Rehabil­ itation Center and supported in part by the Rehabilitation Services Administration, Many of these clients had less vocational experience and less obvious assets. A large number were victims of their physical, mental, or emotional disability - unable to see their own potential. Unlike those who had gone before them, many of these people had little idea as to what they wanted to do, or were unrealistic about either their abilities or the availability of their ideal job (Evaluation Project Final Report part 3, 1975 p. 1 0 0 ). As a result, it became necessary to establish an alternative method of assessing the clients' vocational potential. The counselor was no longer able to determine accurately the eligibility or plan services based upon previously established procedures. Nor was there sufficient time, in an already busy schedule, to undertake the kind of in-depth, intensive study necessary to obtain the needed information. Many of these counselors subsequently turned toward an already existing rehabilitation facility to provide the needed input (Evaluation Project Final Report part 3, 1975; Rice & Thornton, 1972; see also Couch & McDaniel, 1978; Busse, 1 9 7 8 ). Rehabilitation facilities have, over the years, developed techniques and programs designed to assess voca­ tional potential through the process of vocational or work evaluation. The specific goals of this process are consistent with those outlined in the Rehabilitation Act of 1973 and have been defined as being "prognostic, answering questions such as whether a client will be able to work, what kinds of work he will be able to do, and what types of training are necessary to enable him to work"(Gellman, 1968, p. 99). While Gellman (1 9 6 8 ) further suggests that "vocational eval­ uation is coterminous with the rehabilitation process" (p. 1 0 1 ), it would appear, from the standpoint of the voca­ tional rehabilitation counselor, to be most critical at the points of determining client eligibility and planning a suitable sequence of services to be provided. Opinions concerning the goals, processes, and outcomes of the facility-based work evaluation process differ depend­ ing upon one's classification as a referring counselor, client, or vocational evaluator. These divergent opinions have resulted in considerable ambiguity concerning the relative efficacy of vocational evaluation (Sink, 1969; Gwilliam, 1970). Indeed there is very little empirically generated information concerning the utility of work eval­ uation within the context of the total rehabilitation process from the perspective of the counselor, client, or vocational evaluator. Each year state vocational rehabilitation counselors spend thousands of dollars on facility-based work evaluation services. Theoretically, the ensuing results enable the counselor to more effectively plan for, and provide, reha­ bilitation services consistent with each individual client's vocational potential and needs. It is logical and appealing to conclude that because rehabilitation counselors continue to refer clients to facilities for work evaluation, such services are valuable and provide significant information toward that goal. There is, however, a paucity of research concerning the validity of this conclusion, regardless of its logic and general appeal. Purpose It was the intent of this study to explore the impact of information generated by facility-based work evaluation services on state vocational rehabilitation counselors. Impact, for purposes of measurement, was operationally defined as (a) information the counselor gains as a result of evaluation and (b) the degree to which specific informa­ tion gained is considered necessary in planning for client services. A review of the literature concerning the impact of evaluation services on state rehabilitation counselors revealed little in terms of the issues of measurement and instrumentation. A second focus of this study was, there­ fore, concerned with the construction and validation of an instrument designed to measure information counselors possess with regard to their clients' vocational assets and liabilities. Hypotheses The following hypotheses, which are stated in general terms,were investigated by this study: H. Counselors will gain significantly greater amounts of information with respect to clients who complete work evaluation than for a matched group of clients who do not complete evaluation. H, Counselors will gain a significantly greater amount of certainty with regard to information possessed for those clients who complete work evaluation than for a matched group of clients who do not complete evaluation. H. ^ Item information gain (e.g., mean item information gain scores) will be directly related to ratings of item usefulness (e.g., mean item usefulness scores). In addition to the major hypotheses to be tested a substan­ tial amount of information will be presented that is de­ scriptive in nature. Though much of this information is not amenable to hypothesis-testing, it will serve to aid in better defining the outcomes of facility-based work evaluation. Definition of Terms 1* Client: A handicapped individual receiving vocational rehabilitation services in a rehabilitation setting. 2 * Facility: An agency providing direct client services in the form of evaluation, treatment and training. 3. IWRP (Individual Written Rehabilitation Program): A specification of the 1973 Rehabilitation Act (PL 93— 112> directing state rehabilitation counselors to state in writing each client's vocational goal and the services to be provided, including costs, time, expectations, probable outcome, and progress evaluation. 4. Rehabilitation (Referring) Counselor: That individual who assumes major responsibility for developing, supervising, and implementing a rehabilitation plan for clients who have been referred to a facility for vocational evaluation. 7 5. Rehabilitation Process: A planned, orderly sequence of services related to the total needs of the handicapped individual. It is a process built around the problems of a handicapped individual and attempts to resolve these problems and thus bring about vocational adjustment. The process begins with initial referral and ends with successful'placement on a job. (Allison, 1970, p. 4). 6. Vocational (Work) Evaluation: A comprehensive process that systematically utilizes work, real or simulated, as the focal point for assessment and vocational exploration, the purpose of which is to assist individuals in vocational development. Vocational (work) evaluation incorporates medical, psychological, social, vocational, educational, cultural, and economic data in the attainment of the goals of the evaluation process. (Tenth Institute on Rehabilitation Services, 1972, p. 2). 7. Work Adjustment: A treatment/training process utilizing Individual and group work, or work related activities, to assist Individuals in understanding the meaning, value, and demands of work; to modify or develop attitudes, personality characteristics, and work behavior; and to develop functional capacities, as required, in order to assist individuals towards their optimum level of vocational development. (Tenth Institute of Rehabilitation Services, 1972, p. 4). Overview In Chapter II, pertinent literature concerning work evaluation is reviewed. An attempt is made to point out the relative strengths and weaknesses of existing arguments in the work evaluation research and literature. an attempt In addition, is made to identify a plausible research methodology for conducting an exploration of the issues identified previously in the present chapter. Chapter III includes a careful and explicit definition of the research methodology including sample selection, instrumentation, and pilot studies as well as procedures for data collection and data analysis. Chapter IV reports, in-depth, the findings of the present research and includes statistical information regarding the major hypotheses tested as well as descriptive data reflective of facility-based work evalua­ tion outcomes. are defined. In Chapter V, major limitations of the study In addition, findings are discussed and conclusions drawn concerning the impact of the study. Recommendations for future research are also enumerated. CHAPTER II REVIEW OF THE LITERATURE It is clear, for a variety of reasons, that there is an increasing trend on the part of state-employed vocational rehabilitation counselors to seek alternative, and frequently external, methods of assessing their clients* vocational potential. It has been suggested that there exist at least two basic dimensions of utility for voca­ tional evaluation services. The first concerns diagnostic and prescriptive planning information the referring counselor obtains as a result of the evaluative process. The second addresses positive client change in terms of re­ ducing functional disability and/or greater self-understand­ ing (Vocational Evaluation Project Final Report part 1, 1975, p. 29; Baker & Loren2 , 1978: Jones, 1978; McAlees, 1978; Thomas, 1978). It is with the former, informational gains of the counselor, that this and subsequent chapters will be concerned. Research concerning the effectiveness of evaluation services has, for the most part, centered around develop­ ing and validating predictive devices including Job samples, psychological tests, and evaluator ratings of clients (Overs, 1970). While some of these studies have been well-designed and executed, many are poorly conceived 9 10 and contribute little to the state of the art. Spergel (1970) notes that those engaged in the process of evaluation tend to rationalize this state of affairs by suggesting that evaluation is "intuitively11 valid or emphasizing the diffi­ culty in doing controlled studies when primary responsibil­ ities are in the area of service delivery. However, if vocational evaluation is to be considered a truly valid technique of assessment, the "intuitive" elements of validity mu3t be replaced with empirical confirmation. A number of authors have suggested the importance of research concerning follow-up and feedback systems (Nadolsky, 1971; Brolin, 1969; Walker, 1970). However, few studies have addressed this issue, especially with regard to the relative usefulness of work evaluation in the total rehabilitation process. One of the primary,difficulties in developing a useful research methodology evolves from the difficulty in defining the outcomes of evaluation as well as criteria behaviors in terms of the client. The typical predictive study correlates the results of one or more evaluation devices (e.g., work samples, psychological tests, evaluator clinical ratings) with client outcomes following the entire rehabili­ tation process. For example, as reported by Overs (1970), the Jewish Employment and Vocational Service in Philadelphia ( 1 9 6 8 ) conducted a follow-up study of clients participating in work evaluation prior to employment counseling and place­ ment services and compared them with a sample of clients not enrolled in evaluation. Using the criteria of employment 11 and training, it was found that a significantly larger, percentage of those clients exposed to work evaluation were (a) employed or (b) in training and/or (c) had completed employment counseling than did members of a control group. Other criteria studied have included type of employment (Institute for Crippled and Disabled, 1967), length of employment (Rosenberg 4 Usdane, 1963), and employment success (Gellman, Stern, & Soloff, 1963), as well as measures of job satisfaction, movement off welfare rolls, and level of wages earned. The difficulty in using these criteria, however, is the many confounding variables that may possibly contribute to the client's final status. Overs (1970), for example, suggests that "a major weakness in the research has been the tendency to treat factors in the economic and work environment as contingent factors, when in fact they may be dominant" (p. 20). He goes on to say that "...the level of employment in a community may be more predictive than any characteristic of the individual client" (p. 2 0 ). Bolton (1972), in reviewing prediction studies concern­ ing rehabilitation outcome, concluded that no studies have predictions which account for as much as one-half of the criterion variance. One factor which may account for this apparent lack of success in predicting vocational outcome may be in defining the parameters of the criteria evaluated. Data concerning criteria variables such as employment status, job satisfaction, and the like can only be collected at the conclusion of the entire rehabilitation process. The space 12 of time intervening between the evaluation process and these criteria is often considerable, especially when considering the severely disabled client. Layton (1972) points out that "...tests have greatest validity for criterion behavior close in time to test behavior"(p. 404). Brolin (1973) agrees, and questions, "if vocational evaluators can really predict, with any degree of accuracy, most clients' future vocational potentials"(p. 4). He further implores that "we must come to the realization that there are so many influences on what one becomes, that our most important role is for shortrange planning and re-evaluation as the clients develop new skills and horizons"(p. 4). It would, therefore, appear that results of the evaluative process 3 hould be validated in terms of the short-range criteria of client planning and service outcomes, rather than long-range predictions concerning employment status and Job satisfaction. Indeed, H. V. Cobb (1967, in Tseng et al., 1976) reviewed a number of studies in vocational evaluation and concluded "that the measures yielded by vocational evaluation at the intake phase of vocational rehabilitation services tended to provide better prediction of the client's trainability than his/her employability or job success"(p. 154). It would, therefore, seem useful to assess the degree to which data generated through vocational evaluation has an impact on planning for client services at the immediate next phase of the rehabilitation process. In general, it is agreed that the evaluation report is the most clearly identified product of evaluation (Gwilliam, 13 1970). As stated by Sieri (1977), "one means by which we can begin to measure the effectiveness of vocational evaluations is to see if anyone follows the recommendations made after a client is evaluated"(p. 2). Pursuing this line of reasoning, the extent to which the recommendations made by evaluators are followed by the rehabilitation counselor would be indicative of the relative value of vocational evaluation to the rehabilitation process as a whole. Several studies have attempted to explore precisely this concept. Sindberg, Roberts, and Pfeifer (1968), in attempting to measure the extent to which rehabilitation counselors found recommendations resulting from psychological eval­ uations useful, reviewed a sample of 35 psychological eval- . uation reports and their corresponding vocational rehabil­ itation files. A team of three independent judges was then asked to rate recommendations as definitely followed, followed to a large extent, partially followed, not followed, or insufficient information. Results indicated that 51 per cent of the recommendations were definitely followed or followed to a large extent. While the authors conclude that psychological reports are generally useful to the rehabil­ itation counselor, it is interesting that almost half of the recommendations provided were followed only'partially or not at all. In another study, Brolin and Wright (1971) explored the extent to which social, medical, psychological, vocational, and educational recommendations from an eval- 14 uation center for the retarded were implemented as reported by client caseworkers. Data on 229 former outpatients of a diagnostic center were obtained using a follow-up ques­ tionnaire and evaluation center records. Caseworkers reported the extent to which recommendations were definitely followed, partially followed, or not followed at all. In addition, data were collected on a number of variables thought to influence the implementation of the various types of recommendations. Results of their study indicated that of all recommendations made, only 55 per cent for men and 54 per cent for women were definitely followed. Recommendations classified as social tended to be followed less frequently than those in other categories. However, the extent to which the various recommendations were followed was not, in general, a function of their type. The influence of different variables on the implementation of recommendations tended to vary according to the type of recommendation. The only clear-cut effects seem to result from the client's sex and the influence of the client's family. One of the obvious difficulties of this particular study was the use of the questionnaire and the need to rely on the recollec­ tions of caseworkers. It is possible that some biasing may have resulted as a function of the interval of time between the evaluation and implementation of services, and the collection of data from caseworkers. Bieri (1977) seems to have overcome the difficulty of relying on the recollections of caseworkers by bringing the study of the extent to which recommendations are followed 15 back to the actual planning stage of the rehabilitation process. Using a random sample of 27 case files from the North Dakota Division of Vocational Rehabilitation, comparisons were made between recommendations made following a two-week vocational evaluation and services outlined on the client's Individual Written Rehabilitation Program. Recommendations were classified into types corresponding to the service categories into which they fell. These included (a) primarily counseling and guidance, (b) physical or mental restoration, (c) training, (d) further evaluation, and (e) other. The extent to which recommendations were utilized was rated by the experimenter in the categories of (1) definitely followed, (2) followed to a large extent, (3) partially followed, and (4) not followed. The results of this study indicated that only 37 per cent of the total number of recommendations made were def­ initely followed. large extent. Another 11 per cent were followed to a However, more than half of the recommenda­ tions made (52 per cent) were only partially followed or not followed at all. The author points out that the category of recommendations for physical and mental res­ toration was followed more closely than any other category (100 per cent followed definitely or to a large extent). This finding seems consistent with that of Brolin and Wright (1971) in which social recommendations tended to be followed less frequently than those in other categories. Bieri suggests that one possible explanation for this may be that recommendations concerning physical or mental restoration "tended to be much more specific, obvious, and easily implemented than others"(p. 21). It would further seem evident in reviewing Bieri's recommendation categories that counselors are more likely to implement recommendations concerning restorative services and less likely to implement those concerned with vocational planning and/or training. Bieri failed to find any specific variables which cause recommendations to be poorly followed. However, it might be suggested that this was primarily a descriptive study not designed to explore the differential utilization of recommendations based upon the effect of specific counselor, client, or evaluator variables. In accounting for the finding that over half of the recommendations were followed poorly or not at all, Bieri suggests that either evaluators are making poor recommendations or that counselors are disregarding recommendations which should be followed in meeting their clients' needs. In another descriptive follow-up study, Adamczak (1977) found that of a total sample of 17 educable mentally retard­ ed clients, 10 (59 per cent) were pursuing a course of action consistent with recommendations made following in­ volvement in work evaluation. In looking at specific client variables, her data further suggest that there is a significant difference in the functional ability between a group of clients recommended for competitive employment and a similar group of clients recommended for work adjustment services. She notes that those recommended 17 for competitive employment were viewed by their counselors as being more similar to the normal work force despite their similarities to the group recommended for work adjustment services in terms of age level and level of retardation. The research results, though limited, tend to be compa­ rable, suggesting that recommendations following evaluation services are frequently not implemented. This, however, 4 may not be as surprising as the researchers would have us believe, and to some extent adds support to several implications drawn by Gwilliam (1974). Using a case comparison approach, Gwilliam undertook an exploratory investigation designed to study both the pro­ cesses and outcomes of a particular work evaluation service within the context of the total rehabilitation process. As Gwilliam states: While no particular hypothesis was to be tested and little experimental efficacy has been claimed for the statistical analyses performed, the results of the study have been highly suggestive of need for critical attention by both students and practitioners of vocational rehabilitation and assessment, (p. ix) A study group comprised of 60 cases referred to the Rehabilitation Work Evaluation Adjustment Service located in Salt Lake City, Utah, was compared with 29 cases who were not referred during the period of time ranging from 18 to 24 months after intake. The comparison cases were selected by the same counselors who made referrals to the evaluation unit "on the basis of judged similarity of 18 non-referred to referred client^' (p. ix). Matches based upon age, sex’, ethnic and family background, and disability were provided for approximately 50 per cent of the cases. Questionnaires were administered to referring counselors, clients In both groups, and to training or job supervisors when it was found that subjects were in training or employed. Structured interviews were also conducted with as many selected clients as could be located. In concurrence with those studies previously cited, Gwilliam found that "at least some of the recommended treatment in most of the modes indicated had been provided — but with considerable less frequency and consistency than might be considered ideal or desirable"(p. 13*0. He suggests that one possible reason for this may be as a result of "some inability on the part of the predominate using agency to accommodate such information"(p. 203). A3 stated by Gwilliam: Potentially useful diagnostic data are limited in their usefulness by an extremely narrow array of available treatment modalities and a dearth of appropriate program resources.(p. 203) A major focus of Gwilliam's study as well as the studies previously cited was to assess the degree to which recommendations resulting from evaluation services were implemented in terms of client services. Sindberg, Roberts and Pfeifer (1968), Brolin and Wright (1971), and Adamczak (1977) sought evidence in terms of actual and subsequent client involvement. Gwilliam (197*0 and Bieri (1977), on the other hand, focused on the extent to which recommenda- 19 tions were incorporated into each individual client’s employability plan (Gwilliam, 1974) or Individual Written Rehabilitation Program (Bieri, 1977). In all studies, whether data were generated in terms of actual client out­ come or from case planning documents, the results seem compatible. Approximately 48-59 per cent of the recommenda­ tions resulting from evalution services were implemented. However, several limitations concerning the underlying assumptions of these studies restrict our ability to draw any firm conclusions concerning the efficacy of evaluation services with regard to their relative value within the context of the total rehabilitation process. As was suggested earlier, those studies which sought evidence in terms of subsequent client status and involvement are subject to a variety of intervening variables including not only the availability of appropriate program resources and training modalities, but also changes in client motivation, familial influences, and various restrictions posed by labor market conditions and the economy in general. Additionally, the amount of time intervening between evaluation, the provision of services, and the collection of data introduces questions of response reliability and possible bias due to forgetting. Those studies which sought evidence regarding the implementation of evaluation recommendations as reflected within the context of employability plans or Individual Written Rehabilitation Programs are also limited, but in a somewhat different sense. Gwilliam (1974) found while 20 counselors tend to attach high priority to work evaluation as a basis "on which to develop a vocational (employability) plan that ...employability plans were getting completed at a significantly higher rate in the comparison group than in the study group...in spite of what would seem to be a much higher total of evaluation service provided for the subjects in the study group. (P. 95) He concluded that the availability of evaluation information did not have any significant effect upon expediting the completion of employability plans. Gwilliam further sug­ gests that the "functions of the employabilityplan in the total rehabilitation process are not consistent,either among agencies or among counselors within the same agency" (p. 97). Bieri (1977), seeking to reotify this situation to some extent, employed Individual Written Rehabilitation Programs (PL 93-112) as a standardized format to measure the extent to which recommendations resulting from evaluation services were incorporated into client planning. It should be noted, however, that this format suffers from a limitation similar to that affecting the employability plan. Specifically, while the federal law has outlined the basic components of the IWRP, there appears to be considerable variance among agencies in different states as well as among counselors within the same state agency in terms of the quantity and quality of information required to complete it. Additionally, the IWRP is primarily a document designed to 21 outline each client's goals and objectives, and to define a procedure for attaining them. As such, it may not be solely reflective of data resulting from vocational evaluation, but rather a synthesis of data and judgments accumulated by the ** counselor and client from a variety of sources. As a result, specific recommendations coming from evaluation may be hidden or subsumed under the more general components of the plan. A third limitation of the studies described concerns the utilization of recommendations as the primary outcome of the evaluation process. While the evaluation report is the most clearly identified product of evaluation (Gwilliam, 1970), the decision to employ recommendations as the sole outcome variable may be somewhat short-sighted. Bieri (1977) acknowledges this problem, noting that "there may well be actions or recommendations alluded to by an evaluator in the body of a report..." (p. 18). However, Bieri, chose not to incorporate these as "...it was felt that only those which were important and necessary would be included in the recommendation section" (p. 18). Addition­ ally both Bieri (1977) and Gwilliam (1974) indicate the necessity of lengthy and in-depth reviews of both evaluation reports and completed case files in order to determine, code, and classify various bits of information for purposes of analysis an admittedly complicated and time-consuming task. In order to more thoroughly evaluate the impact of evaluation services within the context of the total rehabil­ itation process, we must take into account the total product 22 of evaluation, including not only specific recommendations, but also the various test data, conclusions, and implications alluded to within the context of the entire evaluation experience. Regardless of the extension of rehabilitation services to more difficult cases and the ever-increasing use of external evaluation and diagnostic services, it is still the primary responsibility of the state rehabilitation counselor to determine the client's eligibility and plan a meaningful sequence of services. As stated by Reagles (1978^ see also Bozarth, 1978): The evaluator recommends — the counselor implements. Thus, an evaluator and a rehabilitation counselor are a team, in one respect, but the counselor ha3 ultimate responsibility for the client and must have control over the rehabilitation process, (p. U5) The counselor thus incorporates data from a variety of sources in developing and planning for these services and may be somewhat selective in synthesizing data emanating from work evaluation into a final course of action. Clearly, a different methodology must be developed with which to assess the impact of work evaluation on both the referring counselor and the rehabilitation process as a whole. Summary There is an increasing trend on the part of vocational rehabilitation counselors to use external vocational evalution services as an alternative method of assessing their clients' vocational assets, liabilities, and potential. 23 This trend, while intuitively valid, does, to some extent, lack the empirical confirmation necessary to justify its continuation. Most of the research concerning the effec­ tiveness of evaluation has concentrated primarily around developing and validating long-term predictive devices including Job samples, psychological tests, and evaluation ratings of clients (Overs, 1970). However, in reviewing the results of these studies it has been found that none have predictions which account for as much as one-half of the criterion variance (Bolton, 1972). There is some evidence which suggests that measures yielded by vocational evaluation provide better prediction of the client's trainability than his/her employability or job success (Cobb, 1967 in Tseng et al., 1976). Therefore, it seemes useful to assess the degree to which data generated through voca­ tional evaluation have an impact on planning for client services at the immediate next phase of the rehabilitation process, Given that the evaluation report is the most clearly identified product of evaluation (Gwilliam, 1970), several studies attempted to assess the degree to which recommenda­ tions made to rehabilitation counselors are followed in planning for client services. Sindberg, Roberts and Pfeifer (1968), Brolin and Wright (1971), and Adamczak (1977) sought evidence in terms of actual and subsequent client involve­ ment. Gwilliam (197*0 and Bieri (1977), on the other hand, focused on the extent to which recommendations were incorpo­ rated into each individual client's employability plan 24 (Gwilliam, 1974) or Individual Written Rehabilitation Program (Bieri, 1977). The results of these studies, re­ gardless of the method, are essentially comparable. Approximately 48 to 50 per cent of all recommendations resulting from evaluation services are implemented. Further, results of two of the studies (Brolin & Wright, 1971; Bieri, 1977) suggest that those recommendations which concern physical or mental restoration services tend to be more frequently and fully implemented than those concerning social, counseling, or training objectives. Several limitations concerning the underlying assumptions of these studies restrict our ability to draw any firm conclusions concerning the efficacy of evaluation services with regard to their relative value within the context of the total 1. rehabilitation process. Those studies which sought evidence in terms of subsequent client status and involvement are subject to a variety of intervening variables including (a) the availability of appropriate program resources and training modalities; (b) changes in client motivation; (c) familial influences; (d) general labor market and economic restrictions in the environment; and (e) the amount of time intervening between evaluation, the provision of services, and the collection of data. 2. Those studies which sought evidence as reflected within the context of employability plans or 25 Individual Written Rehabilitation Programs are limited to the degree that the documents are inconsistently used across agencies and/or among counselors within the same agency. It should be noted that both the employability plan and the Individual Written Rehabilitation Program are designed to outline each client's goals and objectives and to define a procedure for attaining them. As such they may not be solely reflective of data resulting from Vocational evaluation, but rather a synthesis of data and judgment accumulated by the counselor and client from a variety of sources. 3. All of the studies are limited in the sense that they employ only the recommendations resulting from evaluation as the sole criterion in assessing utility. It is possible, however, that the counselor may partially or entirely ignore an evaluator's recommendations and still derive considerable benefit from the diagnostic data generated by the evaluation process. In order to more thoroughly evaluate the impact of vocational evaluation services within the context of the total rehabilitation process, we must take into account the total product of evaluation, not just specific recommenda­ tions. Clearly, a different methodology must be developed. CHAPTER III METHODOLOGY Population Defined Rehabilitation counselors employed by the Michigan Bureau of Rehabilitation were chosen as the general popula­ tion to be studied. Eligible counselors were those having at least one client on their active caseload who would be participating in a three- or four-week work evaluation program at a Michigan Association of Rehabilitation Facil­ ities member agency beginning on a specific date selected for the start of data collection. Eligibility was further determined by the counselor's ability to identify at least one other client on his or her active caseload, similar to the client scheduled to begin evaluation in terms of age, sex, and primary disability. It was also required that this second client be one who had not already entered, or would not be entering into a work evaluation program until some­ time after the first client completed evaluation. It was highly desirable for the counselor to choose a second client who would likely be referred to an evaluation program at some point in time after the conclusion of the study. Sample Selection The first step in identifying a pool of prospective counselors was to generate a list of clients who had been 26 27 referred to the various evaluation facilities across the state. An alternative approach might have been to contact each counselor working in each district office individually. However, aside from the time considerations of such an undertaking, a large number of these contacts would be with Individuals who, because of the nature of their caseload, rarely referred clients to evaluation. In addition, because of scheduling difficulties in the facilities, many coun­ selors were frequently not aware of the actual date on which their clients would begin evaluation services. Identification of Facilities After obtaining the approval of the executive director of the Michigan Association of Rehabilitation Facilities and the director of field operations for the Michigan Bureau of Rehabilitation Services, facilities were surveyed to identify those having work evaluation programs of three- to four-weeks duration. Initial contacts were made by phone and letter with the executive directors of 33 facilities across the entire State of Mich­ igan. The objectives, scope, and general procedures for conducting the research were discussed with each facility at this time and several other times prior to the actual point of data collection. Of the facilities contacted, 29 had work evaluation programs within the specified limits of duration. A decision was made to use only those facilities whose referrals came from district offices located within one day's travel time, by car, from Michigan State University. 28 Twenty-three of the 29 facilities met this requirement. Of the remaining 23 facilities, one chose not to participate and was therefore not included in the study. It should be noted that this particular facility was under considerable pressure to meet the requirements of the Commission for the Accreditation of Rehabilitation Facilities and felt that they would not have sufficient time to participate. Identification of Clients Verbal agreements were negotiated with each of the 22 facilities to provide a list of Bureau of Rehabilitation Counselors who had clients that would be beginning work evaluation services sometime during the weeks of March 6, 1978 through April 10, 1978. Facilities were additionally requested to provide the name of the district office with which each counselor was associated and the first name and last initial of the referred client. This additional information was deemed necessary to facilitate communication with referring counselors. It was agreed between all parties concerned that obtaining only the clients' first name and last initial would insure the maintenance of clients' rights, specifically with regard to those issues concerning confidentiality. Because of the facility scheduling difficulties previously indicated, information concerning referring counselors and their clients were in most cases not available until the Wednesday, Thursday, or Friday of the week prior to the clients' commencement of work evaluation 29 services. It was, therefore, necessary to contact each facility on a weekly basis to generate a weekly pool of potentially eligible counselor participants. Facilities were informed that weekly contacts would be maintained until such a time as a sufficient number of eligible counselors had been identified. Identification of Ellbible Subjects There are 43 district offices which comprise the Michigan Bureau of Rehabilitation Services. Thirty-nine (90.7%) of these provided referrals to the 22 facilities previously identified. At a point in time concurrent with the initial survey of Michigan Association of Rehabilitation Facilities having work evaluation programs of three to four weeks duration, all district office supervisors were contacted by letter to inform them of the objectives, scope, and general procedures for conducting the research. An additional letter authorizing the research was disseminated by the director of field operations for the Michigan Bureau of Rehabilitation Services. In addition, prior to contacting individual counselors to ascertain their eligibility and/or willingness to participate, their respective district office supervisors were contacted by phone to answer any questions they might have concerning the project and to secure their permission to contact the individual counselors. After obtaining the weekly list of potential subjects from the 22 facilities, counselors having clients scheduled to begin work evaluation during the following week were 30 contacted individually by phone. Using a standardized presentation format (see Appendix A) the primary researcher or one of two research assistants familiar with the basic components and specific procedures of the study presented a brief explanation of the study's sponsors and objectives, along with a thorough specification of participant tasks and necessary time commitments. The counselors were also asked if they could identify at least one additional client on their active caseload, similar to the client scheduled to begin evaluation in terms of age, gender, and primary disability. It was also required that the second client be one who had not already been through a work evaluation pro­ gram and would not be entering into one prior to the first client's completion of evaluation services. Further, it was highly desirable that the second client be one whom the counselor planned to refer to an evaluation program at some point in time after the conclusion of the study. Some flexibility was exercised in the extent to which the severity of the primary disability, as perceived by the counselor, was identical for both clients. In one case, the primary disability of the client scheduled to begin evaluation was matched with the secondary disability of the matched client. In addition, clients with differences in age of up to 12 years were considered eligible if the primary disability category and gender were identical. While the 12-year age range was considerably larger than might be statistically desirable, results of a pilot study of the experimental procedures and instrumentation indicated that 31 any narrowing of the range would prove extremely restrictive in light of the other matching criteria. During the six weeks between March 6, 1978 and April 10, 1978 a total of 125 potentially eligible counselor participants were identified. Two of the 22 previously identified facilities had no evaluation referrals of three to four weeks' duration during that period of time. Subsequently, counselors who typically referred clients for evaluation at these facilities were not included in the final sample. The 125 potentially eligilbe counselors represented 30 (77%) of the 39 district offices which provided referrals to the 20 participating facilities and 69.7 per cent of all district offices in the state. Contact was made with 118 (95.2%) of the 125 potentially eligible counselors. Forty-two or 35.6 per cent of these were found eligible to participate in the study. Thirty (71.4%) of .the 42 eligible counselor participants representing 19 (44.2%) of all district offices, completed all of the required research tasks. Thus 24 per cent of the 125 potentially eligible counselor participants served as subjects in the study. The remaining 12 eligible counselors did not complete all of the required research tasks. The reason for this, in most cases, was that the client identified as being scheduled for evaluation did not in actuality attend. In one case, a client was transferred from one counselor's caseload into another's, thus providing a discontinuity of services. 32 Characteristics of Counselor Participants Eleven male and 19 female counselors employed by the Michigan Bureau of Rehabilitation Services and working in 19 different district offices served as participants in this study. The combined sample had been employed by the Bureau for an average of 3.9 years with a standard deviation of 2.564 years. The range of years employed was between 3 months and 10 years. Educationally, 19 (6 3 .3 %) had completed the Masters of Arts or Bachelor of Science degree. Nine (30%) held Bachelor of Arts or Bachelor of Science degrees, and two (6.7%) had completed the requirements for Doctor of Philosophy or Educational Specialist. Forty- three per cent of the participant counselors worked primarily with a general caseload. The remaining 57 per cent had specialized caseloads which included social security, psychiatric, or workman's compensation cases. Instrumentation A review of the literature concerning the impact of evaluation services on state-employed rehabilitation counselors revealed little with regard to the issues of measurement and instrumentation. A secondary focus of this study was, therefore, concerned with the construction and validation of an instrument designed to measure information counselors possess with regard to their clients' vocational capacities. 33 Client Assessment Survey (CAS) An extensive review of client rating instruments currently in use revealed little with regard to the comprehensive measurement of information counselors possess concerning their client's vocational capacities. Many of the scales reviewed had, as their primary focus, attempted to rate the client along one or more dimensions of adjustment and/or employability, and further to provide summary data and scores to aid the counselor in planning for case services. They tended, by and large, to be inferential in nature and were frequently not designed to assess the counselor's awareness of specific data concerning the client's aptitudes, temperaments, and physical capacities, as well as job seeking and retention skills, stability characteristics, and general educational development. Scoring was in many cases tedious, and difficult to incorpo­ rate into a research methodology. In addition, there existed an alarming paucity of research data concerning both the reliability and validity almost uniformly across all instruments. A decision was therefore made to construct a measurement device that would be comprehensive in nature, easily scored by subjects participating in the study, and amenable to analysis under the current research design. The literature revealed several categories of infor­ mation rehabilitation counselors deemed necessary for client planning (see Gwilliam, 1970; Menz & Dunn, 1977; Sturm, Otto, & Bakeman, 1972). These include information concerning job goals, general educational development, aptitudes, interests, temperaments, physical capacities, work attitudes, attributes, and interests, job retention skills, job seeking skills, and client stability character­ istics. A number of items were assembled and keyed to these categories on a logical basis (see Mehrens & Lehmann, 1975). As many of the proposed items had a strong relation­ ship to worker traits associated with job analysis, the format developed by the Dictionary of Occupational Titles. Third Edition (Volume II, Appendix B, pp. 651-656) was employed where possible. Other items were adaptations of those suggested within the context of the Functional Capacities Inventory (Menz & Dunn, 1977) and the Vocational Diagnostic Interview (Strum, Otto, & Bakeman, 1972). From a total pool of items generated, 75 were selected as being most appropriate and combined to form the Client Assessment Survey (see Appendix B). The items were arranged into nine scales on an a priori basis (see Table 3.1). Method of Administration Although an interviewer was responsible for adminis­ tering the CAS in this study, a system was developed by which the instrument could also be self-administered (see directions for CAS in Appendix B). Each item, with the exception of items 1 and 2 on the Job Goal scale, were responded to in identical fashion. For items 1 and 2 the respondent indicated Y (Yes) if job goals had been identified for the client, N (No) if job goals had not been identified, and DK (Don't Know) if the respondent was unsure as to 35 Table 3.1 Arrangement of Items into Scales on the Client Assessment Survey Item Numbers Scale Marne Job Goal 1, 2 General Educational Development (not labeled on CAS) 3, 4, 5 » Client Aptitudes 6, 7, 8 , 9, 10, 11, 12, 13, 14, 15, 16. Client Temperaments 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, Physical Capacities 29, 30, 31, 3? ’ 33, 34, 35, 36, 37, 38 Work Attitudes, Attributes, and Interests (not labeled on CAS) 39, UO, 41, 42, 43, 44, 45, 46, 47 Job Retention Skills 48, 49, 50, 51, 5f ’ 53, 56, 57, 58, 59, f4 ' f5 61 ’ 60, Job Seeking Skills 62, 63, 64, 65, 66, 67, 68 Stability Characteristics 69, 70, 71, 72, 73, 74, 75 whether or not goals had been Identified. The remainder of the items (3-75) were responded to in a step-wise manner beginning with problem identification. An item was marked Y (Yes) if it had been identified as being a problem for a client, N (No) if it had been identified as not being a problem for the client, and DK (Don't Know) if the respondent was currently unaware of the client's status concerning an item. If the respondent identicated a DK for a particular item he/she went right on to the next item. If, however, the respondent indicated a X or N for a particular item, the degree of certainty concerning the accuracy of the problem identification response needed to be specified. A 1 was circled to correspond to complete certainty that the item posed or did not pose a problem for the client. A 6 was circled to correspond to a minimal amount of certainty that the problem identification response was accurate. integers 2, 3, The and 5 were to be considered at equal intervals between certainties 1 and 6. In the event that an item had been identified as being a problem for the client (Y) and a 1 had been circled indicating the degree of cer­ tainty concerning the response, respondents were also asked to indicate Y (Yes) if they were currently aware of the extent of the problem or N (No) if they were currently unaware of the extent of the problem. In other words, the respondents did not have to knowledge concerning indicate the extent of the problem unless they were completely cer­ tain that the item posed a problem for the client. The method of responding to items (3-75) on the CAS appears in 37 diagrammatic form in Figure 3.1. Validity Prior to piloting the CAS in the field, a panel of five experts in rehabilitation and/or psychometrics, each having a Ph.D., and employed in a university setting, was consulted to, in part, establish the instrument's face validity and review its psychometric characteristics. Their suggestions concerning modifications of item formats and scoring procedures were incorporated into the current revision of the survey. Comments by those who participated in the pilot study were also solicited and incorporated, where possible. It was generally agreed by both experts and pilot study participants that the instrument adequately sampled the content area related to work evaluation and vocational diagnostics. Reliability The topic of reliability was approached in a number of different ways based primarily on the manner in which the CAS would be employed in exploring the research hypothses. Since the first two research hypotheses concerned a pre/post measurement of information using only one form of the instrument, a measure of test-retest reliability seemed most appropriate. It was felt that this estimate would provide some indication of the instrument's stability over time. An independent sample of 12 vocational rehabilitation counselors employed at two district offices of the Michigan 38 Figure 3.1 GO ON TO NEXT ITEM 2. INDICATE DEGREE CERTAINTY 1. PROBLEM-----IDENTIFICATION DK GO ON TO NEXT ITEM 2. INDICATE DEGREE CERTAINTY 3. INDICATE KNOWLEDGE CONCERNING EXTENT OF PROGRAM GO ON TO NEXT ITEM Figure 3.1 Schematic for Item Scoring on the Client Assessment Survey. 39 Bureau of Rehabilitation Se'rvices served as participants in the test-retest reliability study. Prior to administering the CAS, counselors were each requested to identify one re­ cently closed case, of any type, and to review the contents of that case file. The CAS was administered orally by the primary researcher or one research assistant. Counselors responded basing answers to specific items on the recently reviewed case files. The counselors were instructed to answer questions using the best and most current information available to them at that time. They were also encouraged to refer to the case file whenever they were unsure of a particular response. After pre-testing each counselor individually, appoint­ ments were set up for the second administration of the CAS. It was decided that a five-day period of time would sufficiently reduce the chance of counselors remembering the specific pattern of their responses across the 75 items. In addition it was deemed highly unlikely that they would come into contact with their clients during such a short period of time. The primary researcher and research assistant each administered the CAS on the second occasion to the same counselors they had interviewed at the pretest administra­ tion. Counselors were again instructed to answer questions using the best and most current information they had available to them. Test-retest reliability for the problem identification task alone (e.g., Y, N, DK) and for the problem identifica­ tion task and degree of certainty task combined were com­ 40 puted using the Pearson product-moment correlation technique. Results of these calculations along with the estimates of average pre- to post-item correlations are summarized and Table 3.2. In calculating estimates based on the degree of certainty task, all items which were scored as DK (Don't Know) on the problem Identification task received a value of zero on the degree of certainty task. In addition the values of 1 through 6 were inverted so that a 6 corresponded Table 3.2 Test-Retest Reliabilities for Independent Reliability Sample (N=12) Task Definition Problem Identification Problem Identification and Certainty Combined Reliability Estimate Signif­ icance Test-Retest r = .95 .001 Average Item Cor­ relation r = .867 Test-Retest r = .95 Average Item Cor­ relation r = .861 Estimate Definition to complete certainty and a 1 corresponded to a minimum of certainty. In this way, estimates were reflective of a continuum between not knowing if an item was or was not a problem for a client (scored zero) and .001 U1 extreme certainty that an item was or was not a problem for the client (scored 61. Computational estimates based on the combined problem identification and degree of certainty tasks were recoded in a similar manner. In addition, how­ ever, if an item on the problem identification task was scored N (No), the value on the degree of certainty task was multiplied by minus one. Thus, values for the combined tasks ranged from a minus 6 to a plus 6 (see Figure 3.2). Figure 3.2 Possible values for items in which an N (No) was indicated on the Problem Identification Task -6, -5, -4, -3, -2. -1 Figure 3.2 Possible values for items in which a DK (Don't Know) was indicated on the Problem Iden­ tification Task 0 Possible values for items in which a Y (Yes) was indicated on the Problem Identification Task ♦1, +2, +3» +4, +5, +6 Range of Computational Values for the Combined Problem Identification and Degree of Certainty Task. While the continuum of interval scores for the combined problem identification and degree of certainty tasks was arbitrarily defined, it seemed intuitively logical to assign higher values to those items identified as being problems for clients than for those items identified as not being problems for clients. It is with the items identified as being problems that the counselor must be concerned when 42 planning for a meaningful sequence of rehabilitation services. In a similar sense counselors must also be concerned, to a greater extent, with items scored "don't know" than for items which pose no problem for the client as these may possibly effect the eventual rehabilitation out­ come. In addition to the traditional test-retest estimates of reliability, the percentage of exact agreement across all tasks (e.g., problem identification, degree of certainty, and extent of problem) was calculated between the two administrations of the CAS. At the 95 per cent confidence level it was found that, across all subjects, an average of 94.8 per cent of all items were responded to identically on both administrations. Test-retest correlations were also calculated for eight of the nine a priori scales based on the degree of certainty task. Estimates for the job goal scale could not be calcu­ lated as the degree of certainty was not indicated for items 1 and 2. Results of these computations are presented in Table 3.3. The average item test-retest correlations were calculated for the eight a priori scales using the problem identification task, and the combined problem identification and degree of certainty task. 3.4. These are presented in Table 43 Table 3-3 Test-Retest Correlations for the Degree of Certainty Task Broken Down into the A Priori Scales Scales Test-Retest Correlations Job Goal - — General Educational Development r s .81 Client Aptitudes r = .76 Client Temperaments r = .89 Physical Capacities r = .48 Work Attitudes, Attributes, Interests r = .68 Job Retention Skills r = .46 Job Seeking Skills r = .76 Stability Characteristics r s .88 •Certainty was not scored for the Job Goal Scale. 44 Table 3.4 Average Itein Test-Retest Correlations for the Problem I Identification and Combined Problem Identification and;Degree of Certainty Tasks Grouped iAccording to A Priori Scales Correlations for PROBLEM ID+ Scale Correlations for PROBLEM ID/ DEGREE CERT++ Job Goal General Educational Development .916 .90 .906 .863 .736 .759 .857 .871 .793 .854 Job Retention Skills .879 .841 Job Seeking Skills .943 .929 Stability Characteristics .908 .889 Client Aptitudes i Client Temperaments | Physical Capacities i i | Work AttitudesL Attributes Interests i[ i •Certainty was not scored for the Job Goal Scale ♦Problem Identification Task ♦■►Degree of Certainty Task 45 In addition to the reliability estimates calculated for the independent reliability sample, estimates of scale stability, scale homogeneity, and the percent of exact agreement between the pre- and posttest administrations of the CAS were calculated for the sample of 30 counselors who actually participated in the study. Many of the computa­ tions, in addition to being broken down to the scales, were also estimated separately for those of the counselor's clients who attended work evaluation (N=30) and those who did not attend work evaluation (N=30), as well as the combined work evaluation/no work evaluation group (N=60). The percentage of exact agreements across all tasks (e.g., problem identification, degree of certainty, and extent of problem) calculated between the two administrations of the CAS at the 95 per cent confidence level was 64.5 per cent for the evaluation group and 86.8 per cent for the no work evaluation group. Test-retest correlations for the eight scales using the degree of certainty test were computed separately for the work evaluation group, no work evaluation group, and the combined work evaluation/no work evaluation groups. These calculations appear in Table 3*5. Test-retest correlations for the eight scales were also calculated using the combined problem identification and degree of certainty tasks. These again were computed separately for the work evaluation group, no work evaluation group, and the combined work evaluation/no work evaluation groups and appear in Table 3-6. 146 Table 3 * 5 Test-Retest Correlations of the A Priori Scales on the Degree of Certainty Task for the Work Evaluation, No Work Evaluation, and Combined Work Evaluation/ No Work Evaluation Groups. Scale Work Evaluation Group (N=30) No Work Evaluation Group (N=30) Combined Work Evaluation/No Work Evaluation (Ns60) General Educational Development r .19 r Client Aptitudes r = .114 r Client Temperaments r = .22 r = .93 r = .61 Physical Capacities r = .62 r = .82 r = .68 Work Attitudes, Attributes, Interests r — .33 r — .65 r = CO • Job Retention Skills r = .08 r = .68 r = .21 Job Seeking Skills r - .52 r ** .73 r r .61 Stability Character­ istics r r .63 r ** .38 .09 — .67 r .66 r — .144 • to CD •* 47 Table 3.6 Test-Retest Correlations of the A^ Priori Scales on the Combined Problem Identification and Degree of Certainty Tasks for the Work Evaluation, No Work Evaluation, and Combined Work Evaluation/No Work Evaluation Groups. Work Evaluation (N=30) No Work Evaluation (N=30) Work Evaluation/ No Work Evaluation Scale (N=60) General Educational Development r = .20 r - Client Aptitudes r ** .43 r Client Temperaments r = in • r Physical Capacities - .41 .59 r mm .80 r 3 .61 mm .76 r r .54 r = .85 r = .87 r mm .86 Work Attitudes, Attributes, Interests r .32 r z .71 r - .52 Job Retention Skills r = .32 r = .69 r r .44 Job Seeking Skills r = .58 r a* .73 r = .66 Stability Character­ istics r r _ .63 r s .09 mm .38 48 Scale Homogeneity While the estimates of reliability thus far presented provide a picture of the Client Assessment Survey*s testretest stability over time, it was felt that additional information concerning scale homogeneity, or the degree to which item responses for a particular scale correlated with the total scale score, would be useful. Estimates were cal­ culated using the coefficient Alpha (* ) technique developed by Cronbach (1951), which is a generalization of the KuderRiohardson 20 formula to be used when items are not scored dichotomously. Computations were performed separately for tasks, groups, and pretest or posttest administration. Homogeneity estimates for the degree of certainty task alone were calculated for both the pretest and the post­ test administrations only on the combined work evaluation/ no work evaluation groups. These are presented in Table 3*7. Scale homogeneities were also calculated using the combined problem identification and degree of certainty tasks. These are presented in Table 3.8. It should be noted, however, that only pretest estimates were computed for the combined work evaluation/no work evaluation groups. It was felt that calculating scale homogeneity for the posttest administration of the combined sample would produce misleading data in as much as only one subgroup of the sample was expected to have changed as a result of work evaluation. 49 Table 3-7 Pre- and Posttest Coefficient Alpha Estimates of Scale Homogeneity on the Degree of Certainty Task for the Combined Work Evaluation/No Work Evaluation Groups. (Ns60) Scale Pretest Coeffi­ cient Alpha Signifiance Posttest Coeffi­ cient Alpha Signif iance General Educational Development .57 .0001 .63 .0001 Client Aptitudes .86 .0001 .88 .0001 Client Temperaments .89 .0001 .92 .0001 Physical Capacities .87 .0001 .87 .0001 Work Attitudes, Attributes, Interests ,78 .0001 .84 .0001 Job Retention Skills .89 .0001 .95 .0001 Job Seeking Skills .81* .0001 .84 .0001 Stability Character­ istics .71 .0001 .71 .0001 Table 3.8 Pre- and Posttest Coefficient Alpha Estimates of Scale Homoeneity on the Combined Problem Identification and Degree of Certainty Tasks for the Work Evaluation, No Work Evaluation, and Combined Work Evaluation/No Work Evaluation Groups. Combined Work Evaluation/No Work Evaluation Work Evaluation Group No Work Evaluation Group Pre­ test Sig Pre­ test General Educational Development .27 .0017 .38 .0057 .42 .0395 .*15 .0030 •36 .0001 Client Aptitudes .72 .0008 .75 .0003 .87 .0001 .87 .0159 .82 .0001 Client Temperaments .77 .0001 .85 .0001 .79 .0001 .87 .0007 .78 .0001 Physical Capacities .83 .0001 .81 .0001 .83 .0001 .81 .0001 .84 .0001 Work Attitudes, Attributes, Interests .68 .0001 .62 .0001 .7*1 .0382 .8H .0003 .71 .0001 Job Retention Skills .76 .0001 .87 . 0 0 1 4 .73 .02*11 .7*1 . 1 0 6 8 .75 .0001 Job Seeking Skills .70 .0001 .67 .0001 .77 .0039 .77 .0001 .74 .0001 Stability Character­ istics .60 .0014 .79 .0003 .55 .0001 .47 .0001 .58 .0001 Scale Post­ test Sig Sig Post­ test Sig Pre­ test Sig 51 Client Assessment Survey Gain Scores In order to assess the extent to which counselors gained information and certainty as a result of work evaluation services, a scoring system which summarized movement on the Client Assessment Survey across all tasks (e.g., problem identification, degree of certainty, and extent of problem) between the first and second administrations was devised (see Figure 3*3)* Individual item scores were assigned based on (a) the degree to which an item shifted on the problem identification and/or extent of the problem tasks and (b) the direction and/or extent (e.g., high or low) to which an item showed movement on the degree of certainty task. Shift Scores The extent to which an item response shifted on the problem identification task was categorized in terms of major shift, moderate shift, minor shift and no shift. A major shift was defined as movement from Y (Yes), the item posed a problem for the client to N(No), the item did not pose a problem for the client or from N (No), the item did not pose a problem for the client to pose a problem for the client. (Yes), the item did Items showing a major shift between the first and second administrations of the CAS received a shift score of 4. A moderate shif.t was defined as movement from DK (Don't Know), unsure if the item did or did not pose a problem for the client to (Yes) or N (No), the item did or did not pose a problem for the client Figure 3.3 MAJOR SHIFT I *1 Points I Pre to W H 4-> 2c < w ■H W2< o K H B a, o cc 2 Wm o cn Post Pre Post MINOR SHIFT I 1 Point~| N ■* I DK Y(high)** Pre N Y N DK N(high)** Y Post DK DK +++ ++♦ Y(low)+++ N(low)+++ N(low)-VDK Y N »I ■> N +Y-1-yf->rY-1-n DK ♦ DK Y(low)-^DK Y-1-n w cc *Y-1-y to o W >* tr> P 2 G td M ■H « Y-1-y OfH CO < Pre El ■» Y Y NO SHIFT I0 Points | >1 12 NJH £-i J2 M 2< MODERATE SHIFT I 2 Points | Y-1-n Y-1-y 8 N ■> Y ++ Y ■* N ++ N(high>>DK N *N Y(high)*DK Y * Y El shirt and certainty scores, ♦Indicates a Y(Yes) on the problem identification task, a one on the degree of certainty task, and a y(yes) or ri(No) on the extent of problem task. ♦♦Indicates movement to or from a high degree of certainty (*1, 5, 6) +♦+ Indicates movement to or from a low degree of certainty (1, 2 , 3) Figure 3.3 Diagram of Scoring System to Summarize Movement on the CAS across All Tasks Between the First and Second Administrations. 53 or OK (Don’t Know), unsure if the item posed or did not pose a problem for the client. Items showing a moderate shift between the first and second administrations were assigned a shift score of 2. A minor shift wa3 defined as an increase or decrease in certainty that an item did or did not pose a problem for the client. Items for which a subject responded Y (Yes), the item posed a problem for the client, and a 1 (absolute certainty) was indicated concerning the accuracy of the response on both the first and second administrations, were considered minor shifts if knowledge concerning the extent of the problem showed movement from Y (Yes) or N (No) or from N (No) to Y (Yes). Items showing a minor shift between the first and second administration were assigned a shift score of 1. A no shift situation occurred when an item response on the first administration of CAS was identical across all tasks (e.g., problem identification, degree of certainty, and extent of problem) to the response for the same item on the second administra­ tion. Items showing no shift between the first and second administrations were assigned a shift score of zero. Certainty Scores The direction and/or extent (e.g., high or low) to which an item showed movement on the degree of certainty task was categorized in terms of increases in certainty, equivalence of certainty, and decreases in certainty concerning the accuracy of the problem identification response. An increase in certainty was defined as movement 5H from a lesser to a greater amount of certainty that the problem identification response, for that item, was accurate. In addition, items which shifted from DK (Don't Know) to j Y (Yes) or fl (No) on the problem identification task were scored as increases ih certainty only if the corresponding j certainty value for the Y (Yes) or N (No) response was high I (e.g., 1, 2, or 3)’ Items showing an increase in certainty between the first and the second administrations of the CAS were assigned a certainty score of 3. An equivalence in certainty occurred when a value designated on the degree of certainty tabks during the first administration of the CAS was identical to the value designated for the same item on the second administration* In addition items which shifted from DK (Don't Know) to Y (Yes) or N (No) or from Y (Yes} or N (NO) to DK (Don't Know)on the problem identification task were scored as equal certainty if the certainty valuh for the corresponding Y (Yes) or N (No) response was low (e.g., 5, or 6). Items showing an equivalence in certainty between the first and second administrations of the CAS were assigned a certainty score of 2. A decrease in certainty was defined as movement from a greater to a lesser amount of certainty that the problem identification response, for that item, was accurate. In additior , items which shifted from Y (Yes) or N (No) or DK (Don't Know) on the problem identification task were scored as decreases in certainty if the corre­ sponding certainty val ue for the Y^ (Yes) or N (No) response 55 was high (e.g., 1, 2, or 3)* Items showing a decrease in certainty were assigned a score of 1. Summary Score While the assigned shift and certainty scores previously defined provided the opportunity to evaluate preto posttest movement along each of these dimensions separately, it was felt that a combined index of movement across all task (e.g., problem identification, degree of certainty, and extent of problem) would add considerably to the evaluation of information counselors gain as a result of work evaluation services. A summary score was therefore derived by calculating the cross products of the shift and certainty scores (see Figure 3.3). CAS Scale Homogeneity Using Gain Score Indices In a previous section homogeneity estimates for the scales were reported separately for the pre- and posttest administrations of the CAS using the raw data responses generated by the sample of 30 counselors who actually participated in the study. In the present section homogeneity estimates and intrascale correlations are reported for the same sample (N=30) using the gain score indices discussed in the preceding section. Estimates of homogeneity were again calculated using the Coefficient ' Alpha ( * ) technique developed by Cronbaoh (1951). Intra­ scale correlations were reported as Pearson product-moment correlation coefficients. 56 Homogeneity estimates for the scales were computed separately for the work evaluation and no work evaluation groups using item shift scores, item certainty scores, and item summary scores. These are presented in Tables 3.9, 3.10 and 3.11, respectively. Intrascale correlations were also computed separately for the work evaluation and no work evaluation groups. Calculations in this case were carried out U3ing only the item summary scores and are presented in Tables 3.12 and 3.13. Intra­ scale correlations computed for the combined work evaluation/no work evaluation groups are presented in Table 3.14. Other Data Collected Demographic Data Survey (DPS) The literature revealed a number of factors which might differentially affect the amount of information counselors gain as a result of work evaluation services. Struthers (1971) in a study designed to explore factors related to the vocational stability among people rehabilitated in Michigan, found several client characteristics significantly related to successful employment. These include major disability type, employment status at acceptance for services, education, presence and absence of secondary disability, and age. Other factors which seemed likely to affect counselor information gained included the educational background and experience of the counselor, the facility at which evaluation services were completed, and the counselor's 57 Table 3.9 Coefficient Alpha Homogeneity Estimates for the A Priori Scales of the Work Evaluation and No Work Evaluation Groups Using Item Shift Scores Scale Work Evaluation Group Reliability SignifEstimate icance Job Goal No Work Evaluation Group Reliability Signif­ Estimate icance ____ # ___ _ * General Educational Development .09 .8991 .26 .7317 Client Aptitudes .04 .0136 .55 .5913 Client Temperaments .57 .1257 .70 .0161 Physical Capacities .65 .0416 .69 .9156 -.05 .0104 .47 .009 Job Retention Skills .62 .1488 .77 .2922 Job Seeking Skills .65 .0637 .60 .4603 Stability Character­ i s e cs .54 .2648 •39 .1457 Work Attitudes, Attributes, Interests •Item shift scores were not calculated for this analysis 58 Table 3.10 Coefficient Alpha Homogeneity Estimates of the A_ Priori . Scales for the Work Evaluation and No Work Evaluation Groups Using Item Certainty Scores Scale General Educational Development No Work Work Evaluation Evaluation Group Group Reliability Signif­ Reliability Significance Estimate icance Estimate -.03 .443 -.05 .0567 Client Aptitudes .73 .7995 .65 .9720 Client Temperaments .61 .9460 .55 .7242 Physical Capacities .65 .7394 .34 .6892 Work Attitudes, Attributes, Interests .58 .3499 .48 .5296 Job Retention Skills .76 .0015 .53 .4643 Job Seeking Skills .67 .9008 .57 .5332 Stability Character­ istics .49 .0207 .18 .4142 59 Table 3.11 Coefficient Alpha Homogeneity Estimates of the A Priori Scales for the Work Evaluation and No Work Evaluation Groups Using Item Summary Scores Scale Work Evaluation Group No Work Evaluation Group Reliability Signif­ Reliability Signif' Estimate icance Estimate icanc General Educational Development .03 .99 .09 .1105 Client Aptitudes .58 .0061 .64 .7561 Client Temperaments .65 . 1874 .66 .0193 Physical Capacities .62 .1144 .60 .9559 Work Attitudes, Attributes, Interests .05 .0064 .42 .0044 Job Retention Skills .74 .0008 .79 .738 Job Seeking Skills .71 .1156 .61 .1885 Stability Character­ istics .34 .1194 .22 .22 60 Table 3.12 Intrascale Correlations for the £ Priori Scales of the Work Evaluation Group Using Item Summary Scores 1 1 d 2 <5 3 0 2 0 1.0 3 0 .19 1.0 4 0 .02 .00 5 0 .07 .07 .00 6 0 .13 .21 .52 . 16 7 0 .29 -.11 .39 .07 .21 8 0 .34 10 .07 .00 -.11 .27 9 0 .27 -.13 .47 -.19 .16 .47 .19 4 0 5 6 7 . 3. 4. 5. 6. 7. 8. 9. 9 0 0 0 0 ..02 .07 .13 .29 .34 .27 .00 .07 .21 -. 11 -.10 •p.13 .00 .52 .39 .07 .47 .16 .07 -.00 -. 19 .21 -. 11 . 16 .27 .47 1.0 1.0 1.0 KEY FOR SCALES 1 2. 8 0 Job Goal (Zero variance) General Educational Development Client Aptitudes Client Temperaments Physical Capacities Work Attitudes, Attributes, Interests Job Retention Skills Job Seeking Skills Stability Characteristics 1.0 1.0 .40 .40 1.0 61 Table 3.13 Intrascale Correlations for the £ Priori Scales of the No Work Evaluation Croup Using Item Summary Scores 1 2 3 *4 5 0 0 0 .04 -.10 .22 -.20 0 2 0 3 0 .36 1.0 4 0 .04 5 0 -.10 -.20 .05 6 0 -.02 .26 7 0 .28 8 9 , 6 7 , 8 , 9 0 0 0 -.02 .28 .46 .16 .26 .33 -.06 .36 .05 . -.01 .71 .00 .35 1.0 -.10 -.13 -.17 -.09 *.01 -.10 1.0 .02 .27 .27 .33 .71 -.13 .02 .30 .55 0 ,46 -.06 -.17 .27 .30 0 .16 .35 -.09 .27 .55 1.0 .35 .22 .36 1.0 KEY FOR SCALES 1. 2. 3* 4. 5. 6. 7. 8. 9. , o o• 0 “ 5------ 1 Job Goal (Zero Variance) General Educational Development Client Aptitudes Client Temperaments Physical Capacities Work Attitudes, Attributes, Interests Job Retention Skills Job Seeking Skills Stability Characteristics 1.0 1.0 .40 .40 1.0 62 Table 3.11* Intrascale Correlations for the Priori Scales of the Combined Work Evaluation/No Work Evaluation Groups Using Item Summary Scores 1 2 3 4 0 5 0 6 6 7 0 8 0 9 0 .41 .28 .05 .25 .45 .42 .34 .44 .03 .46 .43 .04 .27 .07 .52 .74 .16 .54 .09 .05 -.04 -. 10 .42 . 16 .37 .33 .60 1 6 0 2 0 1.0 3 0 .41 1.0 4 0 .28 .44 5 0 .05 .03 .07 6 0 .25 .46 .52 .09 7 0 .45 .43 .74 .05 .42 8 0 .42 .04 .16 -.04 .16 .33 9 0 .34 .27 .54 -.10 .37 .60 1.0 1.0 1.0 1.0 KEY FOR SCALES 1. Job Goal (Zero Variance) 2. General Educational Development 3. Client Aptitudes 4. Client Temperaments 5. Physical Capacities 6. Work Attitudes, Attributes, Interests 7. Job Retention Skills 8. Job Seeking Skills 9. Stability Characteristics 1.0 .44 .44 1.0 63 initial reasons for referring the client to evaluation. Those factors, along with items concerning the client's gender and mean3 of primary support, were combined in the Demographic Data Survey (see Appendix C) Posttest Data Survey In addition to the information to be collected as defined by the Demographic Data Survey a number of additional items thought to affect the amount of information counselors gained as a result of work evaluation services were generated (see Appendix D). These included the number of days the client scheduled for evaluation actually attended evaluation, the length of time between commencement of evaluation services and the final staffing during which the counselor received a summary of the diagnostic findings, and the nature of the counselor's caseload in terms of the predominant population the counselor served. Counselors were also asked to specify the number of days, if any, that the matched client attended vocational evaluation. While this was not a likely occurrence in light of the subject identification process, it was possible. Additional comments the counselors had concerning any aspect of the study were also collected. Contact Log It was possible that counselors could gain information from sources other than evaluation during the one client's 614 period of involvement in the evaluation process. Indeed, in­ formational gains might also have occurred from these other sources for the matched client not scheduled to attend work evaluation. Therefore, in addition to the Client Assessment Survey, Demographic Data Survey, and Posttest Data Survey, a standard contact log (see Appendix E) was developed to assess the quantity and nature of contacts counselors made with or on behalf of their clients during the period of time betweenthe first and second administrations of the Client Assessment Survey. Sorting Task The degree to which items appearing on the Client Assessment Survey were considered essential, useful but not essential, or not useful in developing future services for rehabilitation clients was to be establishedthrough independent sorting tasks. Items on the Client Assessment Survey were individually typed on 3 x 5 cards. A set of directions describing the task along with a sort­ ing record form on which to record the responses was constructed (see Appendix F). Pilot Study Prior to collecting data on the 30 subjects identified as eligible to participate, a pilot study was conducted over a four-week period which included approximately 10 Bureau of Rehabilitation counselors employed at two separate district offices. All counselors participating 65 in the pilot study had referred clients for a two-week work evaluation at one centrally located facility. The pilot study was designed as a preliminary test of all materials, instruments, and procedures which would later be employed in the actual research study. An additional goal was to define the approximate amount of time necessary for counselors to complete all research tasks 3 0 that a fairly accurate estimate could be provided to those counselors who would later be participating in the actual study. Results of the pilot study were used to refine both the instrumentation in terms of item formats and the scoring procedures and methods of data collection. In addition, statistical analysis of the pilot data showed results to be in the anticipated direction. Interviewer Training A group of 10 volunteer doctoral and master's level students enrolled in the Michigan State University rehabilitation and counseling programs served as research assistants in this study. A meeting of all assistants was undertaken to explain the general purposes of the study and to acquaint them with the form, content, and method of administration of all research instruments. A taped example of a role-played pre- and post-administration interview was presented to aid in explaining experimental rapport and the degree to which items could be interpreted and clarified. All interviewers were reimbursed for travel and meal expenses incurred while participating in the study. . 66 Data Collection Procedures After identifying an eligible counselor participant, a 1j hour interview-appointment was arranged at a mutually acceptable time to take place at the counselor's district office. Results from the pilot study had indicated that it was extremely unlikely that the work evaluation facility and the referring counselor would have any contact concern­ ing a client's performance during the client's first five days of participation in work evaluation services. Appoint­ ments were therefore arranged to take place at a time no greater than five days following the client's commencement of evaluation. Ten interviewers serving as research assistants for the study along with the primary researcher were assigned appointments based primarily upon their availability and ability to travel independently to a counselor's district office. an In all cases the primary researcher was assigned interview only when and if one of the research assistants was unavailable. Pretest Interview For each appointment, interviewers were provided with a research packet containing all necessary research mate­ rials, a detailed set of instructions for conducting the interview (see Appendix G), and directions for driving to and from the interview site. In addition, the name of the counselor to be interviewed, the first names and last initials of the clients around whom the interviews would 67 focus, and the order in which the clients would be discussed were provided. Interviewers were not aware of which client was scheduled for work evaluation and which was the matched client not scheduled for evaluation. They were further instructed to inform the counselor of this fact and to request that the counselor not reveal that information be­ fore going on to complete any of the research tasks. The order in which the two clients would be discussed was assigned alternately to the first or second position at the time the interview was arranged. Interviewers were instructed to have counselors base their responses on the most recent information they had available to them at that time. In addition, subjects were encouraged to refer back to the client's case file whenever they were unsure of a particular response. Starting with the client identified as number 1 in the research packet, interviewers orally administered the Demographic Data Survey and the Client Assessment Survey. After completing this for the first client the process was repeated for the client identified as number 2 in the research•packet, After completing a Demographic Data Survey and Client Assessment Survey for each client the counselor.was instruct ed in both the purpose and method of completing the contact log. A model log was presented as an example (see Appendix H), and four incomplete contact logs were provided for each of the two clients. Counselors were requested to place the 68 contact logs Into each of the clients* files and to record any contact made with or on behalf of the client during the period of time between pretest and posttest interviews. Before leaving the pretest interview, research assistants were requested to ask the counselor for the approximate date on which the final report from evaluation was expected. In addition, the counselor was informed that he or she would be contacted for the purpose of setting up a second interview. During the period of time intervening between the first and second interviews, each participating counselor was contacted once or twice by phone to ascertain the client's continued involvement in the evaluation process. Posttest Interview Prior to setting up the posttest interview - appoint­ ment, a comparison was made between the date that a particular client was scheduled to complete evaluation services and the approximate date on which the counselor expected to receive the client's final evaluation report. In most cases a discrepancy of between two and six weeks was discovered. It was felt that in order to avoid a potentially serious contamination of the impact of evaluation services on the referring counselor, the effects of any information the counselor received during the period of time between the cessation of evaluation services and receipt of the final report should be eliminated. A decision 69 was therefore made to conduct the posttest interview at a time no greater than five days following the client's final evaluation staffing at the facility. During the final staffing, the participating counselor received a comprehen­ sive verbal report of the evaluation findings. This was, in most cases, accompanied by a brief written summary of the major findings and recommendations. Given the delay in time between the final staffing and the counselor's receipt of the final report, it was felt that the final staffing would provide sufficient information to assess the major hypotheses of this research. A 1J hour posttest interview-appointment was arranged with each counselor at a mutually acceptable time. These appointments again took place at the counselor's district office at a point in time no greater than five days follow­ ing the client's final staffing at the evaluation facility. Interviewers were assigned appointments based primarily upon their availability and ability to travel independently to a counselor's district office. The primary researcher was assigned interviews only when, and if, one of the research assistants was unavailable. For each appointment, interviewers were provided a posttest research packet containing all necessary research materials, a set of instructions for conducting the interview (see Appendix I) and driving directions. The name of the counselor, first names and last initials of the clients, and the order in which the clients would be discussed were also included. Interviewers were again 70 instructed to inform counselors that the interviewer was not aware of which client had completed evaluation and which was the matched client who had not attended evaluation services. As in the pretest interview, counselors were instructed to base their responses on the mo3t recent information available to them at that time and to refer back to the clients' case files whenever they were unsure of a particular response. Interviewers orally administered the Client Assessment Survey for each of the two clients In the order specified in the research packet. After completing this for both clients, counselors were asked to complete the sorting task in which the items appearing on the Client Assessment 'Survey were sorted according to whether they were considered essential, useful but not essential, or not useful for developing future services for rehabilitation clients in general. The counselor's responses were recorded on the Sorting Record Form. After completing the sorting task, the interviewer and counselor completed the Posttest Data Survey and collected completed Contact Logs for each of the clients. Before the interviewer left the counselor's district office, additional sorting tasks were obtained from the district office supervisor, assistant district office supervi­ sor, and several additional counselors who had not already participated in the study. A sample of 30 sorting tasks completed by the supervisors and assistant supervisors, and a sample of 30 sorting tasks completed by the additional counselors were randomly selected for later use in analyzing the research data concerning the usefulness of information 71 generated through work evaluation. Hypotheses The following hypotheses were tested: H. Counselors will gain significantly greater amounts of information with respect to clients who complete work evaluation than for a matched group of clients who do not complete evaluation. Hp Counselors will gain a significantly greater amount of certainty with regard to information possessed for those clients who complete work evaluation than for those clients who do not complete work evaluation. Ho Item information gain (i.e., mean item information gain scores) will be directly related to ratings for item usefulness (i.e., mean item usefulness scores). 5 Research Design and Statistical Analysis This study is quasi-experiraental in nature in that it employs neither random selection from a large population nor random assignment to experimental and control groups. Campbell and Stanley (1963) encourage the U3e of such designs "where better designs are not feasible." (p. 34) This was particularly the case here in that an attempt was made to evaluate an activity taking place in the field rather than in the somewhat artificial, but more easily manipulated environment of the laboratory. Data were collected using a repeated measures format in which counselors responded to dependent measures on a pretest posttest basis for clients attending work evaluation 72 (experimental) and matched clients not attending work evaluation (control). In addition, throughout the period of time intervening between the pretest and the posttest, counselors were required to maintain a log of contacts made with or on behalf of both the experimental and the control client. The basic design of the study is shown in Table 3.15. A second part of the study was designed to assess the relationship between information gained through the process of work evaluation and the degree to which that information was rated useful for developing future services for rehabil­ itation clients in general. This phase of the research was exploratory in nature and employed correlational techniques designed to assess the interrelationships between two or more variables simultaneously. Independent sorting tasks were completed by counselors who participated in the study as well as a sample of counselors who did not participate in other phases of the study. Additional independent sorting tasks were completed by a sample of Bureau of Rehabilitation District Office Supervisors and/or Assistant Supervisors. While the client matching criteria of gender and primary disability were closely followed, the criterion concerning the similarity in age between the experimental and the control client had a range of between zero and 12 years. A t-test for means was therefore used to assess whether or not the mean age for each of these groups differed from each other. Descriptive statistics were employed to describe the demo- 73 Table 3.15 Repeated Measures Design T M Ml T1 M2 T2 M3 Ml M2 M3 S1 S2 S3 SU • • • • • • S29 S30 , T = Treatment T1 = Experimental (clients who attended work evaluation)' T2 = Control (clients who did not attend work evaluation) M = Measures M1 s Pretest (Client Assessment Survey) M2 = During treatment period (Contact Log) M3 = Posttest (Client Assessment Survey) S = Subjects (counselor participants) 74 graphic variables between the two groups of clients. Hypothesis 1 was assessed using multivariate analysis of variance for repeated measures. It 3hould be noted that this technique is a variation of a t-test for correlated means appropriate when there are two or more dependent variables. The dependent variables for this hypothesis were scores for the nine scales and a total score for the entire Client Assessment Survey derived from individual item summary scores. An alpha level of .05 was chosen to indicate significant differences. Differences in certainty between experimental and control clients (Hypothesis 2) were also assessed using multivariate analysis of variance for repeated measures. The dependent variables for this hypothesis were scores for the nine scales and a total score for the entire Client Assessment Survey derived from individual item certainty scores. An alpha level of .05 was again chosen to indicate significant differences. To assess Hypothesis 3, the extent to which item information gain was strongly associated with ratings of item usefulness, it was first necessary to compute mean item information gain scores across the 30 counselors who participated in the study. In addition, mean item usefulness scores were calculated for counselors who participated in the study, the sample of supervisors and assistant supervisors, and the sample of counselors who not participate in other phases of the study. did The degree to which the three samples were similar in their ratings 75 of item usefulness was assessed using a coefficient alpha (Cronbach, 1951 ). This technique is frequently employed in situations in which inter-rater reliability is to be calculated. The extent to which item information gain was directly related to ratings of item usefulness was assessed by computing Pearson product-movement correlation coeffi­ cients. Strong relationships (i.e., those differing significantly from zero) were assessed at the .05 level of significance. Summary A sample of rehabilitation counselors employed by the Michigan Bureau of Rehabilitation was used to assess the impact of facility-based work evaluation services on referr­ ing counselors. Eligible counselor participants were those having at least one client on their active caseload who would be participating in a three- or four-week facility-based work evaluation (experimental) and a similar client in terms of age, gender, and primary disability who had not partici­ pated, or would not be participating, in work evaluation until sometime after the first client completed evaluation services (control). The amount of information counselors possessed concern­ ing each client's vocational assets and liabilities was assessed within five days of the commencement of evaluation services for the experimental client. During the period of time that the experimental client was in a work evaluation program, counselors maintained a log of contacts made with 76 or on behalf of both the experimental and the control client. Information counselors possessed concerning each client's vocational assets and liabilities was again assessed follow* ing the experimental client's final staffing at the conclusion of the evaluation services. The measures used in this study included (a) a demo­ graphic data survey used to define both counselor and client characteristics; (b) a pre-post administration of the Client Assessment Survey used to measure the type and amount of information counselors possessed regarding their clients' vocational assets and liabilities; (c) a posttest data survey to obtain additional information concerning counselor case­ load characteristics, the number of days the client actually attended vocational evaluation, and any additional comments counselors had concerning the study; (d) and finally a contact log used to examine the number of type of contacts counselors made with or on behalf of their clients during the period of time intervening between the pre- and post­ test administrations of the Client Assessment Survey. Hypothesis 1 concerned differences in the amount and type of information counselors gained as a result of work evaluation. Differences between experimental and control clients was assessed using a multivariate analysis of variance for repeated measures designs. Hypothesis 2 concerned the degree to which counselors became more certain of information they possessed as a result of work evaluation. Multivariate analysis of variance for repeated measures was again employed to assess differences between 77 experimental and control clients. Hypothesis 3 concerned the degree to which item information gain was directly related to independent ratings of item usefulness. Items were rated by counselors who participated in the study, a sample of office supervisors and assistant supervisors, and a sample of counselors who did not participate in other phases of the study. Correlational techniques were employed to assess the degree of relationship between samples of raters and between item information gain and ratings of item usefulness. CHAPTER IV RESULTS This chapter is divided into four major sections. first deals with client demographic characteristics. The The second section provides a restatement of the major hypotheses and presents the statistical findings associated with each. The third section is exploratory in nature and concerns trends in the data. A summary of the results is presented in the fourth section. Client Characteristics The client matching criteria of gender and primary disability were closely followed. However, the criterion concerning the similarity in age between the experimental and control clients had a range of between zero and 12 years. A t-test for means between two populations was therefore employed to assess whether or not the two groups differed from each other. A two-tailed test at the .05 level failed to reveal any significant difference be­ tween the mean ages of the experimental and oontrol client groups (t = -.209). Table 4.1 presents chi square comparisons between the experimental and control clients on a number of demographic characteristics. 78 79 Table 4. 1 Chi Square Comparisons of Experimental and Control Clients on Demographic Characteristics Variable Experimental (N) % Control (N) % Test of Significance P X 2 Client Education 12 or More Grades 9-11 Grades 0-3 Grades 36.7 46.7 16.7 (11) (14) ( 5) 43.3 (13) 36.7 (11) 20.0 ( 6) .616 .70 Secondary Disability Absence Presence Unknown 50.0 43.3 6.7 (15) (13) ( 2) 23.3 ( 7) 5.002 60.0 (18) 16.7 ( 5) .10 Employment Status at Application Student Employed Unemployed 3.3 6.7 90.0 ( 1) ( 2) (27) 6.7 ( 2) 6.7 ( 2) 86.7 (26) .3514 .50 40.4 ’ (12) 10.0 ( 3) 36.7 (11) 36.7 (11) 10.0 ( 3) 36.7 (11) .156 13.3 ( 4) 16.7 ( 5) 10.0 13.3 70.0 0.0 6.7 ( 3) ( 4) (21) ( 0) ( 2) 63.3 3.3 30.0 3.3 0.0 Client Residence Independently/Owns Independently/Rents Lives with Parents Lives in Supervised Setting B. R. Status at Pretest Status 02 Status 06 Status 10 Status 16 Status 24 Sources of Client Support Self-Support 10.0 Yes 90.0 No Parental Support Yes No 36.7 63.3 (19)15.75 ( 1) ( 9) C D ( 0) ( 3) (27) 6.7 ( 2) 93.3 (28) (11) (19) 33.3 (10) 66.7 (20) .80 .01 .218 .50 .074 .70 Table 4.1 Con't Variable Experimental % (N) Control % (N) Test of Significance x2 P Source of Client Support Public Assistance Yes No 30.0 70.0 ( 9) (21) 36.7 (11) 63.3 (19) .3 .50 Workman's Compensation Yes No 10.0 90.0 ( 3) (27) 3.3 ( 1) 96.7 (29) 1.077 .30 SSDI Yes No 26.7 73.3 ( 8) (22) 20.0 ( 6) 80.0 (24) .373 .50 SSI Yes No 23.3 76.7 ( 7) (23) 16.7 ( 5) 83.3 (25) .418 .40 0.0 100 ( 0) (30) 0.0 ( 0) 100 (30) .00a .00a Other Sources Yes No 13.3 86.7 (’4) (2 6 ) 16.7 ( 5) 83.3 (25) .132 .70 Reports in Client Files At Pretest General Medical Yes No 90.0 10.0 (27) ( 3) 63.3 (19) 36.7 (11) 5.912 .02 60.0 40.0 (18) (12) 56.7 (17) 43.3 (13) .068 .80 Other Medical Yes 56.7 No 43.3 Statistics not Appropriate (17) (13) 50.0 (15) 50.0 (15) .268 .50 Unemployment Benefits Yes No Psychological Yes No 81 Table 4.1 Con’t. Variable Reports in Client Files at Pretest Psychiatric Yes No Other Reports Yes No Experimental % (N) Control % (N) Test of Significance 40.0 60.0 (12) (18) 30.0 ( 9) 70.0 (21) .658 .40 43.3 56.7 (13) (17) 43.3 (13) 56.7 (17) .00a .00a aStatistics not Appropriate An Inspection of Table 4.1 shows only two demographic variables to be significant. Clients in the experimental group significantly differed from clients in the control group with regard to rehabilitation status codes.1 The majority of experimental clients were in status 10 (eligible 1 Rehabilitation status codes are a nationally standardized, administrative record keeping format, designed to track clients through the rehabilitation process. Status 02 signifies that the client*s application has been submitted. Clients in status 06 are being evaluated in terms of eligibility for rehabilitation services. In status 10 a client is declared eligible for rehabilitation services. Status 16 indicates that the client is receiving restorative services, and status 24 indicates that the client’s rehabilitation services have been interrupted. 82 for rehabilitation services). Control clients, however, tended to be newer in the rehabilitation process (status 02, application for services submitted). The second demo­ graphic variable showing significance concerned reports in the clients' files at the pretest interview. Specifically, the general medical report was more frequently available to counselors for experimental clients than for control clients. In addition to informational gains resulting specifically from work evaluation services, counselors could obtain information from other sources during the experimental client's involvement in the evaluation process. The method chosen for "neutralizing" the possible effect of this "other" information was to have each counselor keep track of the number and type of contacts they made with or on behalf of both the experimental and control clients. If it could be demonstrated that the number and types of counselor contacts were equivalent for both the experimental and control client groups, the effect of this "other" information would also be equivalent. In other words, differences between experimental and control group informa­ tion gain would be primarily reflective of informational inputs resulting from experimental clients' involvement in the evaluation process. Table 4.2 summarizes the chi square comparisons of the number and type of contacts counselors made with or on behalf of the experimental and control clients. It will be noted that significant differences occurred only with regard to contacts made with the evaluation facility. 83 Counselors were in contact with the evaluation facility more frequently for the experimental group than for the control group. Differences between experimental and control group information gain was therefore considered to be primarily reflective of the experimental clients' involvement in facility-based work evaluation services. Table 4.2 Chi Square Comparisons of Number and Type of Contacts with or on Behalf of Experimental and Control Clients Experimental % (N) Variable Control % (N) Contacts with Client 0 1-2 3-4 5 or more 18.3 21.6 3.3 6.7 (11) (13) ( 2) ( 4) 23.3 22.3 3.3 0.0 Contacts with Client's Family 0 1-2 3 or more 41.7 3.3 5.0 (25) ( 2) ( 3) 46.7 (28) 3.3 ( 2) 0.0 C 0) Contacts with Evaluation Facility 0 0.0 1-2 26.7 3-4 21.7 5 or more 11.7 ( 0) (16) (13) ( 7) 41.7 5.0 3.3 0.0 Contacts with Physicians 0 48.3 1 41.7 (29) ( 2) 46.3 (25) 0.0 ( 0) (14) (14) ( 2) ( 0) Test of Significance *2 P 8.419 .297 3.17 .205 • •Yates' corrected chi square for 2 x 2 21 cases. (25) ( 3) ( 2) ( 0) 43.96 .00001 1.667* .197 tables with more than 34 Table 4.2 Con* t Variable Experimental % (N) Control % CN) Test of Significance x2 p Contacts with Employers 0 46.7 1 3.3 (28) ( 2) 50.0 (30) 0.0 ( 0) Contacts with Psychologists 50.0 0 1 48.3 (30) (29) 0.0 ( 0) 1 .7 ( 1) .0* Contacts with Medical Specialist 0 1 2 46.7 3.3 0.0 (28) ( 2) ( 0) 46.7 (28) 1.7 ( 1) 1.7 ( 1) 1.33 Contacts with Client's School 0 1 or more 50.0 0.0 (30) ( 0) 48.3 (29) 1.7 ( 1) .0* Contacts with Others 0 1-2 3-4 40.0 6.6 3.4 (24) ( 4) ( 2) 41.7 (25) 6.7 ( 4) 1.7 ( 1) 1.55 *Yatesf corrected chi square for 2 x 2 21 cases. .517* .472 1.00 .513 1.00 .817 tables with more than Hypotheses Tested Hypothesis 1 Hypothesis 1 is restated followed by a presentation of the associated statistical findings. Hypothesis 1: Counselors will gain significantly greater amounts of information with respect to clients who complete work evaluation than for a matched group of clients who do not complete evaluation. ~ Multivariate analysis of variance for repeated measures designs was selected to test the null hypothesis that there was no difference between the amounts of information counselors gained for clients who completed work evaluation and clients who did not complete work evaluation. Dependent measures were the mean differences of the summary score totals between the experimental and control client groups as measured by the nine scales. An overall probability of less than .0001 was found, thus permitting a rejection of the null hypothesis. This suggested that the dependent variables are significantly affected by the absence or presence of work evaluation services (see Table 4.3). An inspection of Table 4.3 indicates the direction of differences to be higher for experimental clients on all dependent measures. To evaluate which dependent measures were significant, each univariate F-ratio (one-way analysis of variance) and its probability were examined. Only two of the nine scales (physical capacities and job seeking skills) failed to show significance at the .05 level. In addition to the multivariate and univariate analysis of the nine scales, a univariate one-way analysis or variance was used to evaluate the mean differ­ ences of the summary score totals between the experimental 86 Table 4.3 Summary of Mean Differences and Multivariate Analysis of Variance for Repeated Measures Designs Associated With Information Gained F-ratio for Multivariate Test of Equality of Mean Vectors 9.2493 Degrees of Freedom 9, 21 Probability less than .0001 Variables Scales Mean Difference (ExperimentalControl) Univariate F-ratio — P Job Goal 3.7664 17.8651 .0003* General Educational Development 3.2522 11.2719 .0023* Client Aptitudes 13.6922 27.7734 .0.001 * Client Temperaments 19.4425 53.6501 .0001* Physical Capacities 1.6968 .5875 Work Attitudes, Attributes and Interests 9.3560 16.8323 .0004* 24.5093 44.4625 .0001* Job Seeking Skills 3-3700 2.6317 Stability Characteristics 5.5382 12.0610 Job Retention Skills ^Probability of less than .05 .4496 .1156 .0017* 87 and control client groups as measured by the entire Client Assessment Survey (items 1 through 75). Results of this analysis were significant at the .0001 level (F=84.7823 with 1 and 29 degrees of freedom). Hypothesis 2 Hypothesis 2 of is restated followed by a presentation the associated statistical findings. Hypothesis 2: Multivariate designs was again Counselors will gain a significantly greater amount of certainty with regard to information possessed for those clients who complete work evaluation than for those clients who do not complete work evaluation. analysis ofvariance for repeated measures selected totest the null hypothesis that there was no difference between the amount of certainty counselors gained with regard to information possessed for clients who completed work evaluation and clients who did not complete work evaluation. Dependent measures were the mean differences of the certainty score totals between the experimental and control clients as measured by the nine a priori scales. An overall probability of less than .0001 was found, thus permitting a rejection of the null hypothesis. As with Hypothesis 1f this suggested that the dependent variables are significantly affected by the absence or presence of work evaluation services (see Table 4.11) 88 Table 4.4 Summary of Mean Differences and Multivariate Analysis of Variance for Repeated Measures Designs Associated with Certainty Gained F-ratio for Multivariate Test of Equality of Mean Vectors 397.143 Degrees of Freedom 8, 22 Probability less than .0001 Variables Scales Mean Difference fExperimentalControl) Job Goal General Educational Development 00.00 Univariate F-ratio P, 00.00 .0000a 4.2467 573.1480 .0001* Client Aptitudes 16.7025 884.6335 .0001* Client Temperaments 20.5599 1812.7870 .0001* Physical Capacities 14.6473 1211.7507 .0001* Work Attitudes, Attributes 1 and Interests 13.9344 1169.8052 .0001* Job Retention Skills 24.3346 1448.5006 .0001* Job Seeking Skills 11.0453 993.9347 .0001* Stability Characteristic 10.9347 1151.0991 .0001* aCertainty was not scored for items in the Job Goal scale. •Probability of less than .05 89 An inspection of Table 4.4 indicates that the direction of differences for the nine scales is in favor of the experimental client group. In addition, univariate F-ratios (one-way analysis of variance) for each of the dependent measures are all significant beyond the .05 level. A univariate one-way analysis of variance was used to evaluate the mean differences of the certainty score totals between the experimental and control client groups as measured across all 76 items of the Client Assessmement Survey. Results of this analysis were significant at the .0001 level (F=3669.7099 with 1 and 29 degrees of freedom). Hypothesis 3 Hypothesis 3 is restated followed by a presentation of the associated statistical findings. Hypothesis 3: Item information gain (i.e., mean item information gain scores) will be directly related to ratings for item usefulness (i.e., mean item usefulness scores). Independent sorting tasks were completed by counselors who participated in the study, a sample of counselors who did not participate in other phases of the study, and a sample of district office supervisors and assistant office supervisors. The task required each counselor or supervisor to sort each of the items appearing on the Client Assessment Survey into the categories of: (a) essential for develop­ ing future services for clients; (b) useful, but not essential, for developing future services for clients; or 90 (c) not useful for developing future services for clients. Mean item usefulness scores were aomputed from these ratings for each independent sample. The degree to which the three samples were similar in their ratings of item usefulness was assessed using a coefficient alpha (Cronbach, 1951). Results of this analysis revealed ratings between the three samples to be highly homogeneous (alpha s ,965, p < .0001). Mean item information gain scores were computed for each item, using item survey scores across the 30 counselors who participated in the study. The extent to which item information gain was directly related to ratings of item usefulness was assessed using Pearson product-moment correlation. analysis. Table 4.5 summarizes the results of this It will be noted that mean item usefulness scores tended to correlate highly across the three independent samples. However, correlations between mean item information scores and ratings of item usefulness were extremely low. The null hypothesis that item information gain will not be directly related to ratings for item usefulness could not be rejected at the .05 level of significance. Trends in the Data The main thrusts of this study were to assess the amount of information counselors gained and the extent to which their certainty concerning information they possessed increased as a result of work evaluation services. 91 Table 4.5 Correlation Matrix of Mean Item Usefulness Ratings Across Three Independent Samples and with Mean Item Information Scores .15 .103 CO CD ii *1 CD • ii -j .77 .001 r 5 -.03 s s .414 r = s 3 r = 1.0 3 = . 0 0 1 r = . .05 ss .347 r = s= s = .77 .001 u> r = s = r = s = o . 0 0 1 r = 1.0 s = .001 i .88 .001 • CD ■ u r = 3 = s = .414 .05 .347 s s o . D .001 .15 .103 .001 3 = ti C 3 = r s .35 n £. B r = 1.0 s s .001 -J A D‘ C B • A . 0 0 1 Mean item usefulness ratings for Counselors who participated in all phases of the study (N=3 0 ) Mean item usefulness ratings for Counselors who only participated in the sorting task (N=30) Mean item usefulness ratings for Supervisors and Assistant Supervisors (N=30) Mean item information gain socres Pearson product-moment correlation coefficient Significance level 92 However, a number of issues concerning the impact of various counselor and client characteristics on the amount of information counselors gained were explored as a method of identifying possible trends in the data. While the size of the research sample and methods of subject selection and assignment prohibit any firm statistical conclusions regarding these analyses, results presented might provide some direction in developing future research studies. Relationship between Information Gained and Counselor Contacts While the research design "neutralized" the effects of "other" information the counselors received during the period of time intervening between the pre- and posttest interviews, the specific impact of the number and types of contacts the counselor made with or on behalf of both the •experimental and control client groups on the amount of in­ formation counselors gained was a source of considerable interest. Information derived from the contact logs indicated that for both experimental and control client groups, counselors were in contact with the clients, the clients' families, the facility in which the experimental client completed work evaluation services, physicians, employers, psychologists, medical specialist, and in some cases the clients* schools. Pearson product-moment correlations coefficients were computed to assess the degree of relationship between the specific types and total number 93 of counselor contacts and the amount of information counselors gained between the pre- and posttest interviews. Dependent measures were totals of the summary scores for each of the nine scales and the total of all summary scores across the entire Client Assessment Survey. Coefficients were computed separately for the experimental and control client groups. Results of this analysis failed to show any significant relationship which could account for more than 22 per cent of the 3hared variance. It there­ fore seems unlikely that there is any relationship between the number and type of contacts counselors made with or on behalf of their clients and the amount of information they gained. It should be noted that this trend is consistent across all types of counselor contacts including those with the facility in which the experimental client completed work evaluation services. Relationship between Information Gained and Counselor Experi­ ence Relationships between the amount of information counselors gained and the number of years experience counselors possess was assessed using Pearson productmoment correlation coefficients. Dependent measures were totals of the summary scores for each of the nine scales and the total of all summary scores across the entire Client Assessment Survey. Coefficients were computed only for those clients who had completed work evaluation services. Results of this analysis revealed only one coefficient statistically different from zero (see Table M.6). 94 Table 4.6 Pearson Product-Moment Correlations between Amount of Information Gained and Counselor Experience Variables Scales Correlations With Experience Cr) Significance -.14 .223 -.15 .19 a Client Aptitudes .04 .420 Client Temperaments .12 .255 Physical Capacities .22 .116 Work Attitudes, Attributes, and Interests .1 1 .285 Job Retention Skills .31 .046 Job Seeking Skills .08 .340 Stability Characteristics .04 .421 .22 .118 Job Goal General Educational Development Total Client Assessment Survey Specifically, the amount of information gained on the Job Retention Skills scale appeared to be related to the number of years of experience counselors possess. Reltaionship between Information Gained and the Number of Reports in the Clients1 Files at the Pretest Interview. It seemed likely that the number of reports in each client's file at the pretest interview could affect the 95 amount of information counselors gained following the client's involvement in work evaluation services. Pearson product-moment correlation coefficients were therefore computed between the total number of reports available to the counselor prior to evaluation and the amount of infor­ mation the counselor gained following the client's involve­ ment in work evaluation. Dependent measures were again totals of the summary scores for each of the nine scales and the total of all summary scores across the entire Client Assessment Survey for those clients who had completed work evaluation services. Results of this analysis appear in Table 4.7. Table 4.7 Pearson Product-Moment Correlations between Amount of Information Gained and the Number of Reports Available to the Counselor Variable Scale Correlations with # of Reports (r) Job Goal General Educational Development Client Aptitudes Client Temperaments Physical Capacities Work Attitudes, Attributes, and Interests Job Retention Skills Job Seeking Skills Stability Characteristics Total Client Assessment Survey Signif­ icance .08 .330 -.34 .034 .324 .428 -.09 -.03 -.23 .113 .05 -.36 -.40 -.22 .392 .026 .014 -.37 .022 .117 96 An inspection of Table 4.7 reveals information gained on the General Educational Development Scale, the Job Retention Skills scale, the Job Seeking Skills scale and the total Client Assessment Survey to be negatively correlated with the number of reports available to counselors at the pretest interview. While the magnitude of these relationships is minimal, they are signficantly different from zero and suggest that the more reports the counselor has prior to evaluation the less information will be gained as a result of work evaluation services. Relationship between Information Gained and the Number of Days the Client Attended Work Evaluation Services One of the critera for selecting counselors to participate in study was that they have one client scheduled to complete a three- or four-week work evaluation program. However, a large number of these clients (50%) failed to attend evaluation services for the scheduled number of days. It therefore seemed useful to assess the degree of relationship between information counselors gained and the number of days clients actually attended a work evaluation program. Pearson product-moment correlation coefficients were computed between the number of days clients attended work evaluation and the amount of information counselors gained. Dependent measures were the totals of the summary scores for each of the nine scales and the total of all summary scores across 97 the entire Client Assessment Survey. Results of this analysis failed to show any relationship significantly different from zero (see Table 4.8). Based on these findings it seems unlikely that there i3 any relationship between the amount of information counselors gain and the number of days clients attend work evaluation. Table 4.8 Pearson Product-Moment Correlations between Amount of Information Gained and the Number of Days Clients Attended Work Evaluation Variable Scale Job Goal Correlation With # of Days (r) Signif­ icance -.002 .495 .09 .312 -.10 .298 Client Temperaments .13 .246 Physical Capacities -.004 .491 .16 .197 -.13 .251 Job Seeking Skills .11 .280 Stability Characteristics .15 .210 Total Client Assessment Survey .06 .373 General Educational Development Client Aptitudes Work Attitudes, Attributes, and Interests Job Retention Skills 98 Effect of Disability Type on Information Gained Clients were categorized according to their primary disability type* Specific categories included orthopedic disabilities, other physical disabilities, mental retardation, and psychiatric disabilities. Fifty per cent of all counselors who participated in the study had clients who were classified as having psychiatric disabilities. A one-way analysis of variance was employed to assess the effect of primary disability type on the amount of Information counselors gained following work evaluation services. Dependent measures were the totals of all summary scores across the entire Client* Assessment Survey. Results of this analysis failed to show any significant differences in the amount of information counselors gained as a function of client disability types (see Table 4.9). In addition to primary disability type, clients were categorized in terms of the presence, absence, or unknown status of a secondary disability. A one-way analysis of variance again failed to show any significant difference in the amount of information counselors gained as a function of this categorization (see Table 4.9). Effect of Counselor Caseload Type on Information Gained Counselors who participated in the study were categorized according to the nature of their caseloads. Thirteen (4336) of the counselors worked with a general rehabilitation caseload. Ten (3336) provided services to social security related cases. Five counselors ( 16 ,736) 99 Table 4.9 One-Way Analysis of Variance for Information Gained in a Function of Primary Disability Type and the Presence, Absence, or Unknown Status of a Secondary Disability Independent Variable Dependent Variable Degrees of Freedom F-Ratio Significance Primary Dis­ ability Type Total of CAS Summary Scores 3, 26 .495 .689 Presence of Secondary Disability Total of CAS Summary Scores 2, 27 .660 .525 had caseloads dealing primarily with psychiatric cases and two (6 .756 ) provided services to clients receiving workman's compensation. A one-way analysis of variance was used to assess the affect of counselor caseload type on the amount of information counselors gained following work evaluation serivces. The totals of all summary scores across the entire Client Assessment Survey were again employed as the dependent variable. Results of this analysis failed to show any significant differences in the amount of information counselors gained as a function of counselor caseload type (see Table 4.10). 100 Table 4.10 One-Way Analysis of Variance for Information Gained as a Function of Counselor Caseload Type Independent Variable Counselor Caseload Type Dependent Variable Degrees of Freedom Total of 3, 26 CAS ' Summary Sores F-Ratio Significance .306 .821 Effect of Primary Referral Question on Information Gained Counselors participating in the study were asked to prioritize their reasons for referring their clients to a facility for work evaluation services. Five counselors (16.7?) indicated that their primary reason for referral was to aid in determining their clients* feasibility for achieving success as a result of rehabilitation services. Six counselors (20?) referred primarily to test their clients' physical capacities and tolerance for industrial type pressure. Two (6.7?) indicated that their primary referral reason was to aid in developing a tentative vo­ cational objective for their clients. Fourteen (46.6?) of the participating counselors indicated that their primary reason for referral was to assess their clients' vocational skills, assets, arid liabilities, and three counselors (10?) referred clients to aid in developing a rationale for providing additional case services. A multivariate analysis 101 of variance was employed to assess the effect of primary referral question on the amount of information counselors gained. Dependent measures were the totals of the summary socres for each of the nine scales. Results of this analysis indicated an overall probability of less than .0711 that the dependent variables are significantly affected by the type of primary referral question (see Table 4.11). Table 4.11 Multivariate Analysis of Variance for Repeated Measures Designs Associated with the Effect of Primary .Referral Question on Information Gained F-ratio for Multivariate Test of Equality of Mean Vectors 1.5181 Degrees of Freedom 36, 65.4 Probability less than .0711 Variables Scales Job Goal General Educational Development Client Aptitudes Client Temperaments Physical Capacities Work Attitudes, Attributes, and Interests Job Retention Skills Job Seeking Skills Stability Characteristics "Probability less than .05 Univariate F-ratio .5974 .5982 3.2061 1.8279 .9111 .9367 .7568 1.1715 .7877 P .6803 .6674 .0296" .1550 .4728 .4589 .3631 .3471 .5441 102 An inspection of Table *1.11 reveals only one variable F-ratio (one-way analysis of variance) to be significant beyond the .05 level. Specifically, information gained on the Client Aptitude scale might have been affected by the type of primary referral question counselors indicated. Table 4.12 presents the mean information gain and standard deviation for each of the five referral reasons associated with the Client Aptitude scale. It should be noted that the greatest amounts of information gained occurred when counselors referred clients primarily to aid in determing feasibility or to aid In developing a tentative vocational objective. Table. 4.12 Mean Gain and Standard Deviation for Each of the Five Referral Reasons Associated with the Client Attitude Scale Referral Reason To Aid in Determing the Client's Feasibility To Test the Client's Physical Capacities and Tolerance for Industrial Type Pressure To Develop a Tentative Vocational Objective To Assess the Client's Vocational Skills, Assets, and Liabilities To Provide a Rational for Providing Case Services Mean Information Gain Standard Deviation 52.00 11.38 38.00 12.17 49.5 „ 5.95 32.6 14.38 33.7 17.01 103 Summary of Results 1. There were only two significant differences between experimental and control client groups on all 18 demographic variables analyzed. Specifically, clients in each group differed with regard to rehabilitation status codes and the availability of the general medical examination report at the time of the 2. pretest interview. Multivariate analysis of variance revealed that counselors gained significantly more information (p. less than .0001) for clients who attended work evaluation than for a matched group of clients who did not attend work evaluation. An examination of the associated univariate F-ratios and probabilities revealed significant differences between the two groups on six of the nine a priori scales. Only the Physical Capacities and Job Seeking Skills scales failed to show significance at the .05 level. The direction of differences between the experimental and control client groups on all nine scales were in favor of the experimental group. A univariate one-way analysis of varience revealed significant differences (p less than .05) between the experimental and control client groups across all 75 items of the Client Assessment Survey. Multivariate analysis of variance revealed that counselors gained a significantly greater amount of of certainty (p less than .0001) with regard to information possessed for those clients who completed work evaluation than for those clients who did not complete work evaluation. An inspection of the associated univariate F-ratios and probabilities revealed significant differences between the two groups on all of the a priori scales. A univariate one-way analysis of variance revealed significant differences (p less than .05) between the experimental and control client groups across all 75 items of the Client Assessment Survey. A coefficient alpha estimate of inter-rater reliability revealed three independent samples of Individuals completing the degree of usefulness sorting task to be homogeneous ( * = .965). However, ratings of item usefulness (i.e., mean item usefulness scores) were not significantly related to item information gain (i.e., mean item information gain scores). An exploratory examination of various trends in data revealed the following: a) There were no significant relationships between the number and type of contacts counselors made with or on behalf of their clients and the amount of information they gained. 105 b) There was only one significant relationship between the amount of information counselors gained and the number of years experience the counselor possessed. Specifically,, the amount of information gained on the Job Retention Skills scale was slightly related to counselor experience. c) Information gained on the General Educational Development scale, the Job Retention Skill scale, the Job Seeking Skills scale, and the total gain score for the entire Client Assessment Survey were negatively correlated with the number of reports available to counselors at the d) pretest interview. There were no significant relationships between the amount of information counselors gained and the number of days clients attended work evaluation. e) Information counselors gained was not affected by client primary disability type, secondary disability, or counselor caseload type. f) A multivariate analysis of variance revealed an overall probability of less than .07 that information gained is significantly affected by the type of primary referral question the counselor indicated. Only one univariate F-ratio was significant beyond the .05 level. 106 Specifically, information gained on the Client Aptitude scale was highest for counselors who referred clients primarily to aid in determining feasibility or to aid in developing a tentative vocational objective. CHAPTER V SUMMARY AND CONCLUSIONS It was the intent of this study to explore the impact of information generated by facility-based work evaluation services on referring state vocational rehabilitation counselors. Impact, for purposes of measurement, was operationally defined as (a) information the counselor gained as a result of evaluation, (b) Increases in certainty regarding Information counselors possessed as a result of work evaluation and (c) the degree to which information gained was considered useful in planning for client services. A second focus of the study was concerned with the construction of an instrument designed to measure information counselors possess with regard to their clients' vocational assets and liabilities. A sample of 30 rehabilitation counselors employed by the Michigan Bureau of Rehabilitation served as subjects for this study. Eligible counselor participants were those having at least one client on their active caseload who would be participating in a three- or four-week facilitybased work evaluation (experimental) and a similar client in terms of age, gender, and primary disability who had not participated, or would not be participating in work evaluation until sometime after the first client completed evaluation services (control). 107 108 The amount of information counselors possessed concern­ ing each client's vocational assets and liabilities was assessed within five days of the commencement of evaluation services for the experimental client. During the period of time that the experimental client was in a work evaluation program, counselors maintained a log of contacts made with or on behalf of both the experimental and the control client. The information counselors possessed concerning such clients' vocational assets and liabilities was again assessed following the experimental client's final staffing at the conclusion of evaluation services. The measures used in this study included (a) a demo­ graphic data survey used to define both counselor and client characteristics; (b) a pre/post administration of the Client Assessment Survey used to measure the type and amount of information counselors possessed regarding their clients' assets and liabilities; (c) a posttest data survey to obtain additional information concerning counselor caseload characteristics, the number of days clients actually attended vocational evaluation, and any additional comments counselors had concerning the study; (d) and finally, a contact log to examine the number and type of contacts counselors made with or on behalf of their clients during the period of time intervening be­ tween the pre- and posttest administrations of the Client Assessment Survey. The Client Assessment Survey was developed specifically for the study as a method of measuring the type and amount 109 of information counselors possessed concerning their clients' vocational assets and liabilities. The instrument consists of 75 items groups into nine a priori scales. It was generally agreed by both a panel of experts in rehabilitation and/or psychometrics and participants of a pilot study that the instrument adequately sampled the content area related to work evaluation and vocational diagnostics. An estimate of test-retest reliability indicated the instrument's over­ all stability to be .95. Test-retest correlations for each of the nine scales, across tasks, ranged from .76 to .93. Demographic comparisons between the experimental and control client groups revealed only two significant differences of 18 analyzed. Specifically, clients in each group differed with regard to rehabilitation status codes and the availability of the general medical examination report at the time of the pretest interview. Multivariate analysis of variance revealed that counselors gained significantly more information (P less than .0001) and certainty regarding information they possessed (P less than ,0001) for clients who attended work evaluation than for a matched group of clients who did not attend work evaluation. An examination of the univariate F-ratios and probabilities associated with the analysis of information gained revealed significant differences between the experimental and control client groups on six of the nine scales. Only the Physical Capacities and Job Seeking Skills scales failed to 110 show significance at the .05 level. An examination of the univariate F-ratios and probabilities associated with the analysis of certainty gained revealed significant differ­ ences on all of the scales. A coefficient alpha estimate of inter-rater reliability revealed three independent samples of individuals completing the degree of usefulness sorting task to be homogeneous ( « = .965). However, ratings of item usefulness (i.e., mean item usefulness scores) were not significantly related to item information gain (i.e., mean item information gain scores). An exploratory examination of various demographic variables on the amount of information counselors gained revealed no significant relationships between the number and type of counselor contacts made with or on behalf of their clients, the number of days clients attended work evaluation services, primary disability type, secondary disability, or counselor caseload type and the amount of information counselors gained following work evaluation. A slight relationship was found between the number of years of experience the counselor possessed and informa­ tion gained on the Job Retention Skills scale. An inverse relationship was found between the amount of information gained on a number of the scales and the number of reports available to the counselor at the pretest interview. Finally, a multivariate analysis of variance revealed an overall probability of less than .07 that in­ formation gained is significantly affected by the type of 111 referral question the referring counselor asked. Only one univariate F-ratio was significant beyond the .05 level. Specifically, information gained on the Client Aptitude scale was highest for counselors who referred clients primarily to aid in determining feasibility or to aid in developing a tentative vocational objective. Discussion It would appear, based upon statistical analysis of * the research data, that counselors gain a significant amount of information following a client's involvement in work evaluation services. The results further indicate that evaluation provides counselors with the ability to effec­ tively define their client's vocational assets and liabilities with regard to job goals, general educational development, aptitudes, temperaments, work attitudes, attributes and interests, Job retention skills, and stability character­ istics. The results also indicate that evaluation tends not to provide the counselor with any additional information concerning the client's physical capacities and Job seeking skills. While additional- information concerning physical capac­ ities and job seeking skills may not be produced following a client's involvement in work evaluation, counselors did seem to become more certain of information they already possessed. Increases in certainty also occurred for information counselors possessed on the other six scales. This suggests that work evaluation services may serve the 112 dual function of both providing the counselor with new information and confirming or disconfirming information the counselor already possesses. An exploratory examination of various demographic charac­ teristics on the amount of information counselors gained, re­ vealed no significant relationships between the number and type of counselor contacts made with or on behalf of their clients, the number of days clients attended work evaluation services, primary disability type, the presence or absence of a secondary disability, or counselor caseload type and the amount of information counselors gained following work eval­ uation. Because the size of the research sample in this study was small, little experimental efficacy is claimed with re­ gard to these findings. It is entirely possible that future research studies employing a large sample designed specifi­ cally to measure these variables may produce significant results. A slight, positive relationship was found between the number of years counselors had been employed in the Michigan Bureau of Rehabilitation and the amount of information gained on the job retention scale. Specifically, counselors having more experience tended to gain more information on this scale than counselors having less experience. One possible explanation for this finding is that facility evaluation re­ ports may fail to specify client characteristics along this dimension. Thus, referring counselors may have to infer client characteristics concerning job retention skills. This being the case, it would seem logical to ex­ pect counselors with more experience to gain more 113 information. However, a far more likely explanation for this finding is that the relationship is a spurious artifact resulting from the specific characteristics of the sample studied. A slight, inverse relationship was found between the amount of information gained concerning general educational development, job retention skills, and Job seeking skills and the number of reports available to the counselor at the pretest interview. It is therefore possible that some of the information resulting from work evaluation along these dimensions is also produced through other resources. For example, a comprehensive psychological report would likely provide the counselor with information concerning the client's general educational development. Thus, the counselor might be expected to gain less information along this dimension than a counselor who did not have a comprehensive psychological report. In the same sense, a counselor who had obtained a report from a client's previous employer might be expected to gain less information following work evaluation concerning the client's job retention skills and job seeking skills than a counselor who had not obtained a report from the client's previous employer. While the magnitude of the relationship concern­ ing this finding is somewhat greater than the relationship between counselor experience and information gained, the reader is again cautioned that it may be an artifact resulting from the specific characteristics of the research sample. m While the type of referral question counselors asked did not generally affect the amount of Information gained, counselors who referred clients to work evaluation to aid in determining feasibility or to aid in developing tenta­ tive vocational objectives tended to gain more information concerning client aptitudes than did counselors who referred clients to evaluation for other reasons. These results suggest the possibility that information concerning client aptitudes is to some extent related to the determination of client feasibility and the development of tentative vocational objectives. this finding However, it is again possible that was an artifact attributable to the specific characteristics of the research sample. Counselors who participated in all phases of the study along with a sample of counselors who did not participate in all phases of the study and a sample of Bureau of Rehabilitation district office supervisors and assistant supervisors were asked to independently sort the 75 items appearing on the Client Assessment Survey into three categories concerning the extent to which each item was useful in developing future services for'rehabilitation clients in general. A coefficient alpha estimate of inter-rater reliability revealed the three samples to be homogeneous in their ratings. However, ratings of item usefulness were not significantly related to item information gain. That is, items of information receiving the highest average gain scores across the 30 counselors who participated in all phases of the research tended not 115 to be related to those Items deemed most essential for developing future services for rehabilitation clients in general. The most clearly identified explanation for this finding concerns the fact that all raters tended to rate all 75 items as being essential or useful. The resulting average item ratings across the three independent samples tended to be fairly consistent, thus producing relatively small variance terms. Since correlation coefficients tend to in­ crease as variance increases it is not surprising that correlations concerning item information gain and item usefulness ratings were not significant. Limitations of the Study While statistical analysis tended to confirm the hypotheses that counselors gain significant amounts of information, as well as certainty regarding the information they possess, a number of methodological implications come to bear on the extent to which conclusions can be firmly drawn. Possibly the greatest single limitation in inter­ preting the results of this study centers around the quasiexperimental nature of the research design. Field studies, while frequently providing a more accurate picture of the real world, are subject to a variety of uncontrolled in­ fluences. In the present research, for example, counselor participants were neither randomly selected from the popu­ lation of all Michigan rehabilitation counselors nor were their clients randomly assigned to work evaluation and no work evaluation groups. 116 Restrictions imposed by the client matching criteria may have resulted in the selection of a sample of counselor participants who were not fully representative of the total population of rehabilitation counselors employed by the Michigan Bureau of Rehabilitation. As a result, the generalizability of the research findings may be limited only to those counselors who actually participated in the study. Cornfield and Tukey (1956) suggest if a non-random sampl? is employed, that careful descriptions of subject characteristics, dependent and independent variables, and identification of other variables likely to be reactive, allows the researcher to generalize to similar populations. In this case, it would seem that results could be general­ ized to counselors employed by state divisions of vocational rehabilitation who refer clients to facilities for threeor four-week evaluation. However, prior to drawing any conclusions concerning the relevance of the research find­ ings, the reader is cautioned to carefully review the similarities between the characteristics of the sample de­ fined in the present study and his or her own population of referring counselors. While the matching criteria attempted to control for some of the major differences between the experimental and control client groups it was possible that undetected, systematic differences affected the research outcomes. A number of demographic comparisons between the client groups revealed only two significant differences. First, clients in the no work evaluation group tended to differ from clients 117 in the work evaluation group with regard to rehabilitation status codes. Specifically, clients participating in evaluation tended to be further along in the rehabilitation process than clients not participating in work evaluation. This was also reflected by the second significant difference between the two groups. Specifically, general medical examination reports were less frequently available to the counselor at the time of the pretest interview for control clients than for experimental clients. While these differ­ ences may have affected the differential amount of inform­ ation and certainty counselors gained, it seem3 likely that they provided a more rigorous te3t of the impact of work evaluation services. Despite the fact that counselors continued to collect information on the control group clients throughout the time that the experimental clients partici­ pated in work evaluation, the impact of information result­ ing from work evaluation significantly surpassed the impact of information the counselor had collected concerning control group clients. In addition to the demographic characteristics explored, it is possible that other undetected and systematic differ­ ences existed between the experimental and control client groups. The reader is therefore advised to interpret the results of this study with caution. A second major limitation in interpreting the results of the study concerns the possibility that counselors responded to items on the Client Assessment Survey differ­ ently for clients participating in work evaluation and 118 clients who did not participate in work evaluation. In selecting counselors to participate in the research it was necessary for them to identify not only a client who would be participating in evaluation, but also a matched client who would not be participating in evaluation. In addition, it was necessary to apprise potentially eligible counselors of the objectives of the research as well as the methods of data collection. Counselor participants, therefore, may have biased their responses in favor of the experimental group. An attempt to control for this biasing affect centered around encouraging counselors to refer to client case-files whenever they were unsure of how to respond to a particular item. It remains uncertain, however, whether or not this strategy was effective in eliminating counselor biasing in favor of the experimental client group. A third limitation of this research concerns the utilization of the Client Assessment Survey as the sole out­ come measure of Information and certainty gain. While estimates of reliability and validity associated with this study were well within acceptable limits, some caution must be exercised in the interpretation of research results until such time as the instruments track record can be proven through replication in future research studies employing similar or related methodologies. Implications for Further Researoh One of the primary implications from the research centers around developing a better strategy for assessing 119 the extent to which information gained is useful in devel­ oping future services for rehabilitation clients. The sort­ ing task, employed in the present study, produced average ratings of item usefulness so similar that an attempt to correlate them with average item information gain produced coefficients which were not significantly different from zero. While it was possible that no relationship exists be­ tween information gained and the extent to which that infor­ mation is useful in developing future services for clients, it is more likely that the present research methodology was simply not powerful enough to produce interpretable results. A number of exploratory analyses revealed the variables of counselor experience, the number of reports available to the counselor prior to work evaluation, and the type of referral question the counselor asked, to be related to the amount of information counselors gain following a client's involvement in work evaluation services. Though little experimental efficacy is claimed for these findings in the present research, future studies designed specifically to measure the impact of these and other variables may produce significantly valid results. It has been suggested that the two basic dimensions of utility for vocational evaluation servioes are diagnostic and perceptive planning information the referring counselor obtains and positive client change in terms of reducing functional disability and/or greater self understanding (Vocational Evaluation Project Final Report part 1, 1975 p. 29; Baker & Lorenz, 1978; Jones, 1978; McAlees, 1978; 120 Thomas, 1978). The present study has confirmed, to some extent, the utility of vocational evaluation services with regard to the referring counselor. However, a recent study by Chandler (1978) failed to find any significant differences between clients who participated in work evaluation and clients who did not participate in work evaluation along the dimensions of increased self-concept, vocational maturity, . and general vocational self-awareness. If a research methodology could be effectively developed to assess the impact of vocational evaluation services on the client, a combined study incorporating both the dimensions of utility for the referring counselor and the client could be under­ taken. Thus, questions concerning differential impact along a number of demographic characteristics could be addressed. Additional implications for future research center around the development and refinement of the Client Assess­ ment Survey. While estimates of reliability and validity associated with the present research were well within acceptable limits, the instrument has yet to be proven through replication in future research studies employing similar or related methodologies. Conclusions It was the intent of this study to explore the impact of information generated by facility-based work evaluation services in referring Michigan Bureau of Rehabilitation Counselors. A review of the research findings seem to 121 support the general conclusion that counselors gain signif­ icant amounts of information following a client's involvement in work evaluation. Specifically, evaluation provides coun­ selors with the ability to effectively define their clients' vocational assets and liabilities with regard to (a) defining Job goals; (b) general educational development; (c) vocational aptitudes; (d) vocational temperaments; (e) work attitudes, attributes, and interests; (f) Job retention skills; and (g) stability characteristics. Evaluation tends not to provide counselors with any additional information concerning the client's physical capacities or Job seeking skills. In addition to gaining new information regarding their client's vocational a3set3 and liabilities, it appears that work evaluation serves the secondary function of confirming or disconfirming information the counselor already possessed prior to the client's involvement in a work evaluation program. Conclusions regarding the extent to which information gained following a client *3 involvement in work evaluation is deemed useful in developing future client services can not at the present time be supported. While it is possible that no relationship exists between information gained and information usefulness it is more likely that the present research methodology was simply not powerful enough produce interpretable results. to APPENDICES APPENDIX A STANDARDIZED PRESENTATION FORMAT FOR IDENTIFYING AND SELECTING ELIGIBLE COUNSELOR PARTICIPANTS 123 1. I understand that you have a client, fatata client'a first nama and laat initial) beginning work evaluation at (facility name) on the (date client ia achadulad to start) 2. Aftar getting confirmation from the counselor say, "I'd like to discuss a research project we've undertaken. The project concerns the iapact of facility-based work evaluation services on referring Michigan Bureau of Rehabilitation Counselors. We are interested in finding out how auch inforaation counselors gain following a client's involvement in evaluation. The project has been cleared through the state office of the Bureau of Rehabilitation, the Michigan Association of Rehabilitation Facilities, and has been funded by an RSA grant through the University of Wisconsin-Stout. Your district office supervisor is aware of the project and has given us pernlsslon to contact you. The tins requirements for counselors participating in the study will be a 1-1/2 hour interview prior to the client's completion of the first five days of work evaluation services and a second 1-1/2 hour Interview to take place following the client's final staffing at the evaluation facility." 3. After getting some feedback concerning the counselor's interest in participating, ask the counselor if they have another client on their active caseload who is similar to the first client with regard to age (plus or minus 10 years), gender, and primary disability who has not already participated in work evaluation but will likely be starting evaluation sometime after the first client completes evaluation. If the counselor can identify a second client go on to 4. If the counselor cannot Identify a second client thank them and suggest that we may be getting back to them during the next few weeks if ve find another of their clients scheduled to begin evaluation. 4. These are the specifics of the study: (a) We have trained a number of Interviewers to help us out with data collection and one will be assigned to meet with you at your earliest possible convenience. (b) The Interviewer will not be aware of which client is beginning evaluated and which client is matched. And we ask that you please try not to reveal which is which. Pre-Test Interview (c) The interviewer will first briefly explain the study once again. 125 (d) A Demographic Data Survey will than ha completed for tha flrat ellant. Items on tha survay concern tha client'a age, education, aourcea of support and raaaona for referral to evaluation (remember not to reveal which client la which). (a) Following the completion of the DDS the interviewer will aak you a series of questions about the client using a Client Assessment Survey. This is an Instrument that wa developed and concerns various elements of tha client's assets and liabilities. Interviewers will explain the manner of responding when they meet with you. You are encouraged to make responses based on the best information available to you at that time. Feel free to refer to your case file at any time. This la not a teat of your knowledge, but rather an assessment of how much information is currently available to you concerning the client. (f) Following completion of the CAS for the first client, you will be askdd to go through a similar procedure for the second client.. (g) After both clients have been completed, the interviewer will explain tha format, purpose, and use of tha contact log. This ia primarily a tool for controlling for additional data you might receive concerning the dlenta during the time that the one client is in evaluation and until the second set of Interviews following your receipt of the final evaluation report. It is not a vary time consuming task and will add considerably to the study. It asks briefly that you record eny contacts you make with or on behalf of tha client and in just a couple of words tha nature of the contact. You will be given more than sufficient numbers of copies to place in tha clients* files. (h) We won't be bothering you again until just before you have the final staffing for the client who attended work evaluation. At that time wa will contact you and aet up another Interview (1-1/2 hours) usually less. Post-Test Interview (a) An Interviewer will again be assigned to come meet with you. Try again not to reveal which client ia which — don't confirm if asked. (b) A CAS will again ba done for both clients in tha same order aa tha first interview — Interviewers will be Informed as to the correct order to follow. 126 (c) Following completion of the GAS for both client*» you will be asked to complete e short sorting task In which Items ere sorted Into three categories of essentiality for planning for client services* Interviewers will provide you with specific Instructions. 5* Can we set up an appointment now? When would be convenient for you? We do however,need to conduct Interview before you've received any substantive feedback from evaluation (within five days of the first clients Involvement In evaluation). 6. Thanks for your help — we'll bein touch again 7. Record the names of the first and second client andthe date and time of the Interview. APPENDIX B THE CLIENT ASSESSMENT SURVEY 127 CLIENT ASSESSMENT SURVEY (Experiment*! Version) by Harold Weinstein, C.R.C 129 Please detach this page and refer to it when completing this survey. Directions: For items 1 and 2 please indicate your response by marking Y (yes) if job goals have been identified, N (no) if job goals have"~not been Identified, and DR (don't know*) if you are unsure whether or not goals have been identified. For the remainder of the items (3-75) mark Y (yes) if the item has been Identified as a problem for the client. Mark N (no) if the item has been identified as not being a problem for the client, and mark DR (don't know) if you are currently unaware of the client's status concerning an item. In addition, please indicate by circling, your degree of certainty concerning the accuracy of each response. A one would correspond to complete certainty while a six would indicate a complete lack of certainty that the response is accurate. When an item is scored DR (don't know), you need not indicate the degree of certainty. When an item has been Identified as being a problem for the client and a one has been circled indicating the degree of certainty con­ cerning that response, please indicate by circling Y (yes) if you are currently aware of the extent of the problem and N (no) if you are currently unaware of the extent of the problem. If an item has been identified as not being a problem for the client, or if it has been responded to with DR (don* t know), than knowledge concerning the extent of the problem need not be specified. Know­ ledge concerning the extent of the problem also need not be spec­ ified when the degree of certainty Indicated for a particular item is anything other than one. Y 130 N DK X. The client's job goal has been identified. □ □ □ 2. An alternative job goal has been identified. □ □ □ Extent Problem Degree Certain 3. 4. 5. Is the client's general ability to apply counon sense and to solve practical problems a problem? □ Is the client's general ability to understand and perform basic arithmetic operations a problem? □ □ Is the client's general ability to communicate and understand lan­ guage a problem? □ □ □ 1 2 3 4 5 6 Y N P 1 2 3 4 5 6 Y N □ □ 1 2 3 4 5 6 Y N □ □ □ 1 2 3 4 5 6 Y N Is the client's verbal aptitude: abil­ ity to understand meanings of words and ideas associated with them, and to use them effectively a problem? □ □ □ 1 2 3 4 5 6 Y N Is the client's numerical aptitude: ability to perform arithmetic opera­ tions quickly and accurately a problem? □ □ □ 1 2 3 4 5 6 Y N Is the client's spatial aptitude: ability to comprehend forms in space and understand relationships of plane and solid objects a problem? □ 1 2 3 4 5 6 Y N Is the client's form perception: abil­ ity to perceive pertinent detail in objects or in plctoral or graphic form a problem? □ 1 2 3 4 5 6 Y N Client Aptitudes 6. 7. 8. 9. L0. Is the client's intelligence: general learning ability. The ability to "Catch on" or understand instructions and underlying principles. Ability to reason and make judgements a problem? □ □ □ □ 11. 12. 13. 14. L5. L6. Is the client's clerical perception; ability to perceive pertinent detail in verbal or tabular material a problem? □ Is the client's motor coordination; ability to coordinate eyes and hands or figures rapidly and accurately in making precise movements with speed a problem? □ Is the client's finger dexterity; eblllty to move the fingers and manipulate smell objects with the fingers rapidly or accurately a problem? 131 Y N DK Degree Certain □ □ 1 2 3 4 5 6 □ 1 □ Extent Problei Y N 2 3 4 5 □ □ □ 1 2 3 4 5 6 Is the client's manual dexterity: ability to move hands easily and skillfully a problem? □ □ 1 2 3 4 5 6 Is the client's eye-hand-foot coor­ dination; ability to move the hand and foot coordlnately with each other In accordance with visual stimuli a problem? D D 1 2 3 4 5 6 Y N Is the client's color discrimination; ability to perceive or recognize sim­ ilarities or differences In colors, or in shades or other values of the same color a problem? O D 1 2 3 4 5 6 Y N □ O 6 Y N N . D N !lient Temperaments L7. Does the client have a problem adjusting to occupational situations characterized by frequent change? DO O 1 2 3 4 5 6 Y N L8, Does the client have a problem adjusting to situations characterized by repeti­ tive or short cycle operations carried out according to set procedures or sequences? DO O 1 2 3 4 5 6 Y N ,9. Does the client have a problem adjusting to situations characterized by doing things only under specific Instruction, allowing little or no room for inde­ pendent action or judgment in working out problems? D Q O 1 2 3 4 5 6 Y N 132 N DK Degree Certain Extent of Problem O O 1 2 3 4 5 6 Y D 1 2 3 4 5 6 Y N □ □ 1 2 3 4 5 6 Y H □ □ 1 2 3 4 5 6 Y N □ 1 2 3 4 5 6 Y N DD D 1 2 3 4 5 6 Y N DD D 1 2 3 4 5 6 Y N □ 1 2 3 4 Y D 1 2 3 4 5 6 Y 20. Does the client have a problem adjusting to situations characterized by the direc­ tion, control, and planning of an entire activity or the activities of others? (Leadership activities?) D 21. Does the client have a problem adjusting to situations Involving the necessity of dealing with people In actual Job duties beyond giving and receiving instruc­ tions? DD 22. Does the client have a problem adjusting to situations characterized by working alone and apart In physical isolation from others, although the activity may be Integrated with that of others? □ 23. Does the client have a problem adjusting to situations involving Influencing peo­ ple In their opinions, attitudes, or judgments about ideas or things? □ 24. Does the client have a problem adjusting to situations Involving performing ade­ quately under stress when confronted with the critical or unexpected or when taking risks? □ 25. Does the client have a problem adjusting to situations Involving the evaluation of Information against sensory or judgmental criteria? 26. Does the client have a problem adjusting to situations Involving the evaluation of Information against measurable or vevlflcable criteria? 27. Does the client have a problem adjusting to situations involving the interprets* tlons of feelings, ideas or facts in teems of personal viewpoint? □ 28. Does the client have a problem adjusting to situations involving the precise attainment of set limits, tolerances, or standards? □ □ DD 5 6 Y N N | Y 133 H DK Degree Certain Extent of Problem Physical Caoacltles (Functional Ability) 29. Does the client have a problem sitting? 30. Does the client have a problem standing? 31. Does the client have a problem lifting, carrying, pushing or pulling? □□ □ □□ □ □□ □ □□ 2 3 4 5 6 Y R 2 3 4 5 6 Y R 2 3 4 5 6 Y R 2 3 4 5 6 Y N 2 3 4 5 6 Y R 32. Does the client have a problem bending? □ 33. Does the client have a problem with respiratory functions? □□ □ Does the client have a problem speaking? □□ □ □□ □ □□ □ 2 3 4 5 6 Y R 2 3 4 5 6 Y N 2 3 4 5 6 Y N 34. 35. Does the client have a problem seeing? 36. Does the client have a problem hearing? 37. Does the client have a problem climb­ ing or balancing? □□ □ 2 3 4 5 6 Y R Does the client have a problem reach­ ing, handling, fingering, or feeling? b□ □ 2 3 4 5 6 Y N □□ □ 2 3 4 5 6 Y N □□ □ 2 3 4 5 6 Y N □□ d 2 3 4 5 6 Y N Is the client's attitude toward working a problem? □□ □ 1 2 3 4 5 6 Y N Is the client's self image as a worker a problem? on □ Y N 38. 39. 40. 41. 42. 43. Is the client's past work experience a problem with regard to possible vocational objectives? Is the client's present level of skill training a problem with regard to possible vocational objectives? Is the client's present level of formal education a problem with regard to pos^. sible vocational objectives? 1 2 3 4 5 6 Y 134 N DK Degree Certain Extent of Problem 44. Are the client's job expectations with regard to salary a problem? 45. Are the client's Job expectations with regard to Job duties a problem? 46. P P| 2 3 4 5 6 Y N □ □ □ 2 3 4 5 6 Y N □ □ □ 2 3 4 5 6 Y N 2 3 4 5 6 Y N 2 3 4 5 6 Y N Is there a conflict between the client's stated Interests and the client's abil- lei..? 47. PI . Is there a conflict between the client's stated Interests and the client's lnventorled interests? [*11*111 Job Retention Skills (Please refer to instructions for Items 3-7 48. Is absenteeism a problem? P 49. Is tardiness a problem? □ da 2 3 4 5 6 Y N 50. Is getting along with co-workers a problem? □ □ □ 2 3 4 5 6 Y N Is working well independently a problem? □ □ □ 2 3 4 5 6 Y N Is working well in groups a problem? □ □ □ 2 3 4 5 6 Y N 53. Are the client's supervision require­ ments a problem? □ □ □ 2 3 4 5 6 Y N 54, Does the client have a problem accept­ ing supervision? P PI PI 2 3 4 5 6 Y N □ O □ 2 3 4 5 6 Y N □ O □ 2 3 4 5 6 Y N I1 P |""1 2 3 4 5 6 Y N 2 3 4 5 6 Y N 51. 52. 55. 56. 57. Is the client's quantity of work a problem? Is the client's quality of work a problem? P Is the client's grooming and physical appearance a problem? 58. P Is following written directions a problem? □ □ □ Y 135 N 59. Is following oral directions a problem? Q Cl 60. Is following multiple step directions a problem? □ 61. □ Does the client have a problem per­ sisting In assignments until completion? Q DK Degree Certain Extant of Problem Q 1 2 3 4 5 6 Y 1 2 3 4 5 6 Y N □ Q [3 1 2 3 4 5 6 Y N C3 D 1 2 3 4 5 6 Y N 1 2 3 4 5 6 Y N □ 1 2 3 4 Y Cl 1 2 3 4 5 6 Y N N Job Seeking Skills 62. 63. 64. 65. 66. 67. 68. Is the client's work history a problem with regard to future employment? Does the client have a problem identi­ fying Job leads? □ □ Does the client have a problem seeking employment with sufficient frequency? □ □ Does the client have a problem ade­ quately explaining skills in Inter­ viewing situations? D O □ 5 6 Does the client have a problem ade­ quately answering problem questions in Interview situations? Q Q Q 1 2 3 4 5 6 Y N Does the client have a problem dres­ sing appropriately for Interviews? D D □ 1 2 3 4 5 6 Y N Does the client have a problem main­ taining enthusiasm about seeking employment? □ □ □ 1 2 3 4 Y O 1 2 3 4 5 6 Y N □ 1 2 Y 5 6 K N Stability Characteristics 69. Does the client have a problem main­ taining a predictable and stable pat­ tern of behavior in coonon situations? C 70. Does the client have a problem travel­ ing independently withoutsupervision? □ O □ 3 4 5 6 » 136 N DK Degree Certain Does the client have a problem getting along with others in social situations? Q Q Q 1 2 3 4 5 6 Y N Does the client have a problem making appropriate use of leisure time? Q O D 1 2 3 4 5 6 Y N Does the client have a problem seeking help when problems cannot be solved Independently? □ □ □ 1 2 3 4 5 6 Y N Does the client have a problem accept­ ing help given by others? □ □ □ 1 2 3 4 5 6 Y N Does the client have a problem taking medication without supervision? Q Q 1 2 3 4 5 6 Y N Y 71. 72. 73. 74. 75. Q Extent of Problem APPENDIX C THE DEMOGRAPHIC DATA SURVEY 137 Demographic Data Survey Counselor Information 1, Counselor's Name 2. Highest Degree Earned 3. Number of years vorking for Vocational Rehabilitation Services Client Information 1. Client's Name 2. Age 3. Sex 4. Vocational Rehabilitation Services status ^ 5. Disability Type (circle one) 6. 7. 8. a) Vision - hearing - speech (specify: b) Orthopedic (specify: cl Other physical (specify: ) d) Mental retardation (specify: ) a) Mentally 111 (specify: ) Employment status at application (circle one) a) student b) employed c) unemployed Secondary disability (circle one) a) absence bl presence (specify: c) unknown Client has chemical dependency ) yes 138 no 139 9. Types of reports currently In clients file (check all those appropriate) General Medical Other (specify; ) Psychological Other Medical (specify; ) Psychiatric 10. Sources of Client Support (circle all those appropriate) a) self b) parents c) public assistance d) workmans compensation e) Social Security Disability Insurance f) Supplemental Security Income g) unemployment benefits h) other (specify; * 11. 12. 13. Client resides: a) Independently: rents b) with parents c) In supervised setting: owns group home institution Facility to which the client was referred for evaluation a) Name of facility b) Address of facility Education completed by client (circle one) a) 12 or more grades b) 9-11 grades c) 0-8 grades 1i»0 14. Reason(s) client was referred to facility for evaluation services. (Please number each of the following In order of their importance for this particular case. The number one should correspond to the most Important reason for referral; the number two to the next most Important reason, etc. If a particular reason Is entirely Inappropriate, please indicate this by placing an "X" in the appropriate space). a) to aid in determining the client's feasibility (e.g. to determine whether or not the client will benefit from V.R.S. services) b) to test the client's physical capacities and tolerance for Industrial type pressure c) to develop a tentative vocational objective d) to assess the client's vocational skills, assets, and liabilities e) to provide a rationale for providing case services (e.g. to aid in developing a plan for additional case services) f) for documentation purposes (e.g. to justify a particular course of action) g) for purposes h) other: of client growth please specify APPENDIX D THE POSTTEST DATA SURVEY 141 Foittwt Data Survey 1. How many days waa the client In evaluation? a) Client # 1 b) Client # 2_____________ 2. What waa the date o£ the Client's staffing at the facility? a) Client #1 b) Client #2 3. Do you have a general caseload? 4. Do you have a specialised caseload? a) 5. yea vea If yes please specify: Do you have any consents about the study? no — no APPENDIX E THE CONTACT LOG 143 Contact Log Counselor's Name Client's Name Fora of Contact Date Phone Correspondence Interview__ other Phone Correspondence Interview other Phone Correspondence Interview other Phone. Correspondence Interview other Phone Correspondence Interview other__ Name and title of person contacted Name of agency with which contact person Is affiliated State briefly the nature of the contact APPENDIX F DIRECTIONS FOR COMPLETING SORTING TASK AND THE SORTING RECORD FORM 146 DIRECTION FOR COMPLETING THE SORTING TASK We ere interested in finding out whether you think that information regarding each of the following items would be Essential. Useful but not Essential, or Not Useful for developing future vocational rehabilitation services for B.R. clients in general. We are Interested in your honest opinion. If you feel that information concerning particular items is not really useful than please do not hesitate to so indicate. By better defining the kinds of information we require concerning our clients we will be in a much better position to seek specific answers to our questions. 147 Sorting Record Mane of Feraon Sorting! Title of Peraon Sorting; Study Participant? yea no Interviewer Kane______________________________ Eaaentlal for Developing Future Client Servicea Uaeful but not Eaaentlal for Developing Future Client Servicea Date: Not Uaeful for Developing Future Client Servicea APPENDIX G INSTRUCTIONS FOR CONDUCTING THE PRETEST INTERVIEW 148 149 DirectIona for conducting the Interview 1. Introduce yourself. 2. "Thank you for taking the time to see me and for helping us out with the project." 3. "Very briefly the project has been designed to assess the impact of facility based work evaluation services on you, the referring rehabilitation counselor. Impact, for pur purposes has been defined as the amount of information you pick up as a result of work evaluation and the degree to which that information is deemed essential by you in developing a plan for future client services." 4. "I'd like to start with (give name of #1) first if that's alright with you. I do not know which of the clients is beginning evaluation and Which is the matched client and it is important that you try not to reveal that information." 5. "The first thing we'll do is complete a Demographic Data Survey on (give name of fll) , Before we start on the client I need to ask just a couple of questions concerning you. (After this is completed go right on to the client portion of the survey. Encourage the counselor to refer back to the client's case file whenever this seems appropriate.) 6. "The next thing we'll do is complete a Client Assessment Survey on (give name of fll) . This is an instrument that has been developed specifically for this study and explores the information that you current­ ly posses concerning the client's assets and liabilities. You are encour­ aged to make your responses based on the best information available to you at this time. Feel free to refer to your case file at any time. This is not a test, but rather an assessment of how much information is current­ ly available to you concerning the client. Lets go over the directions for completing the CAS together. (Refer to the directions at the begin­ ning of the CAS.) I'll read each item aloud while you read along with me." (Check for counselor understanding at this point, and occasionally review the directions with the counselor during the process of completing the CAS.) 7. After completing the CAS for the first client complete a Demographic Data Survey and CAS for the second client in thesame manner as for the first. 8. After completing both clients pull out the model contact log and four incomplete contact logs for each of the clients. 9. "This contact log Is primarily a tool for controlling for any additional data you might receive concerning either of the clients during the time that one of the clients is in evaluation. When the study is completed we would like to be able to attribute any informational gains on your part to the evaluation process. However, It Is possible that you might learn things about the client from other sources. It is not a very time consuming task as you can see from the model and it will add considerably to the validity of the study. The log requires that you briefly record 150 any contacts made with, or on behalf of, the client during the period of time between now and the second Interview following your receipt of the evaluation repcrt from the facility. I have four contact logs for you to place Into each of the client's files. These will probably be more than enough to cover any contacts you might make. The logs will be picked up during the second Interview." 10. "Do you, at this point, have any Idea as to when you might be receiving the final report from evaluation?" (If yes record this on the Inside cover of the file folder In which you brought the materials.) 11. Ask the counselor If they have any questions. Answer them If you can. If you cannot answer the question record It and tell the counselor that we will get In touch with them during the next few days. 12. Thank the counselor for participating In the study and tell them that we will be contacting them to set up another Interview following their receipt of the flnel report. (A different Interviewer will probably be conducting the second Interview.) 13. Be sure not to leave any materials with the counselor other than the contact logs. Bring all completedmaterials back to M.S.U. APPENDIX H THE MODEL CONTACT LOG 151 Contact Log HM Counselor's Kara, Client's RftWyj h L __ Fora of Contact Date g * fa/n Kane and title of person contacted Phone Correspondence Interview / other H u e of agency with which contact person la affiliated State briefly the nature of the contact asrow-k. Ro\aeA^* eu^l Phone / Correspondence Interview other lU .w s F*C* (U l< iv^ Cc7w |>/*t Phone t/~ Correspondence__ Interview ✓ other % Phone Corresphndence Interview y other nU--* c(*uh 1i c j i w h l^ c lv 1 . lEfer* Cf, ^03-^07. McAlees, D. C. Convergence and divergence in rehabilitation counseling and vocational evaluation: A review, The Journal of Applied Rehabilitation Counseling. 197b, £ (1), 49-54. Mehrens, W. A,, & Lehmann, I. J. Standardized tests in education. (2nd ed.) New York: Holt, Rinehart & Winston, 1975* Menz, F., & Dunn, D. A Functional Capacities Inventory. Unpublished document, Menomonie, Wisconsin, 1977. Nadolsky, J. Patterns of consistency among vocational evaluators. Vocational Evaluation and Work Adjustment Bulletin. 1971, ** (4), 13-2$. Overs, R. Vocational evaluation: Research and implication. Journal of Rehabilitation. 1970, 36 (1), 18-21 Reagles, K. W. Reaction to convergence and divergence in rehabilitation counseling and vocational evaluation. The Journal of Applied Rehabilitation Counseling. 15F7B7 ( ( 1) , ^ 4 Rehabilitation Act of 1973, P. L. 93-112, H. R. 8070. Washington, D.C.: Superintendent of Documents, September 26, 1973. Rice, 3. D., & Thornton, C. L. Utilization of prescriptive vocational evaluation to increase counselor effectiveness. Rehabilitation Research and Practice Review, 1972, 4 (U). pp. 59-63. Rosenberg, B., & Usdane, W, The TOWER system: Vocational evaluation of the severely handicapped for train­ ing and placement. Personnel and Guidance Journal. 1963, J*2, ll|$-1$2. Sindberg, R., Roberts, A., & Pfeifer, E. The usefulness of psychological evaluations to vocational rehabilitation counselors. Rehabilitation Literature. 1968, 29, 290-29^ Sink, J. M. Evaluation - a reason for concern. In W. A. Pruitt & R. N. Pacinelii (Eds.), Work Evaluation in Rehabilitation. Materials Development Center, Menomonie, WI, 1969. / 159 Spergel, P. Vocational evaluation: Research and implications for maximizing human potential. Journal of Rehabilitation. 1970, 36 (1), 21-24. Struthers, R. D. Factors related to vocational stability among persons rehabilitated by the Michigan Division of Vocational Rehabilitation. Unpublished doctoral dissertation, Michigan State University, 1971. Strum, T., Otto, N., & Bakeman, M. Vocational Diagnostic Interviewing. Minneapolis Rehabilitation Center, Inc., Minneapolis, Minn., 1972. Tatsuoka, M, M. Siginificance tests: Univariate and multivariate. Selected topics in advanced statis­ tics: An elemeniTary approach. Number 4. Champaign, 111.: Institute for Personality and Ability Testing, 1971. Tenth Institute on Rehabilitation Services. Vocational evaluation and work adjustment services in voca- . tional rehabilitation. Washington, D.C.: Rehabilitation Services Administration, 1972. Thomas, S. W. Rehabilitation counseling and vocational evaluation from a vocational evaluation service perspective, The Journal of Applied Rehabilitation Counseling. 197b, 9 C1), 71-78. U. S. Department of Labor. Dictionary of Occupational Titles. Vol. II, 3rd ed. Washington, D.C.: Government Printing Office, 1965. Vocational evaluation final project report. Part I.. Vocational Evaluation and Worl< "A"d'justment~^ulletin1 B_ (Special edition), July, 1975. Vocational evaluation final project report. Part III.. Vocational Evaluation and Work Adjustment Bulletin, ti (Special Edition;, July, 1975. Walker. R. A future for vocational evaluation? of Rehabilitation. 1970, 36 (1), 38-39. Journal Whitehouse, F. A. Rehabilitation as a Concept in the Utilization of Human Resources. Rehabilitation Consultant, American Heart Association of America, Association of Medical Social Workers Training Course for Leaders of Institutes, Pittsburgh, Pennsylvania, June 20, 1955.