REHABILITATION COUNSELOR EDUCATORS’ PERCEPTIONS OF IMPORTANCE, STUDENT PREPAREDNESS, AND TEACHING PROFICIENCY IN CLINICAL JUDGMENT SKILL DOMAINS By Bryan Scott Austin A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Rehabilitation Counselor Education 2012 ABSTRACT REHABILITATION COUNSELOR EDUCATORS’ PERCEPTIONS OF IMPORTANCE, STUDENT PREPAREDNESS, AND TEACHING PROFICIENCY IN CLINICAL JUDGMENT SKILL DOMAINS By Bryan Scott Austin Given the importance of clinical judgment in rehabilitation counseling (Strohmer & Leierer, 2000), prevalence and consequences of rehabilitation counselor biases (Berven & Rosenthal, 1999), and the emerging trend to educate rehabilitation counselors in evidence-based practice (EBP) (Leahy & Arokiasamy, 2010), the explicit teaching of clinical judgment skills that can effectively bridge science and practice to improve clinical judgment accuracy appears critically needed (Spengler et al., 1995). Despite such noted importance of clinical judgment and research evidence of counselor biases, there were no clinical judgment training studies found that provide data useful for better understanding rehabilitation counselor educators’ perceptions of teaching clinical judgment skills. To address this gap in the literature and add to the knowledge-base of clinical judgment training research in rehabilitation counselor education, the current exploratory study was conducted to gather data to better understand rehabilitation counselor educators’ perceptions of teaching clinical judgment skills for effective rehabilitation counseling practice. A sample of 126 rehabilitation counselor educators who were employed in master’s degree rehabilitation counseling programs was obtained for this study from an estimated population that represent the United States, Puerto Rico, Australia, and Canada. Results of this study indicate that rehabilitation counselor educators’ perceive clinical judgment skills to be highly important for effective rehabilitation counseling practice. New knowledge of empirically-validated clinical judgment skill domains was identified that warrant consideration for future Council on Rehabilitation Education (CORE) standard revisions. In addition, significant differences were found between rehabilitation counselor educators’ reported importance of clinical judgment skill domains relative to their gender, race, clinical experience (i.e., setting), and program emphasis of clinical judgment training. Significant differences were also found between rehabilitation counselor educators’ reported level of student preparedness relative to their race and program emphasis of clinical judgment training, as well as between educators’ teaching proficiency and program emphasis. Research findings of the present study also showed significant relationships between rehabilitation counselor educators’ (a) reported importance and teaching proficiency of clinical judgment skills, (b) perceived teaching proficiency and how well they perceived students’ preparedness in using clinical judgment skills, and (c) reported importance of clinical judgment skills and how well they perceived students were prepared in using such skills upon graduation. Data generated from this study can be used to inform and enhance master’s and doctorallevel rehabilitation counseling clinical training curricula and to address the challenges that come with deciding how to effectively educate rehabilitation counselors in EBP. Specific implications for clinical training and future research are provided. ACKNOWLEDGEMENTS I want to thank the many important people in my life who made the process and completion of this dissertation project possible. I want to sincerely thank my beautiful and loving wife, Lindsay. Without her encouragement and her ability to hold down the fort at home, this accomplishment would not be possible. I also want to thank my beautiful daughters, Logan, Blake, and Lane. The joy in their eyes and fun-loving spirits every evening when I would come home kept me going. I want to thank my parents, Paula and David, who encouraged me to embark upon this journey. Further, I want to thank my mother-in-law, Kathleen, for braving East Lansing in the dead of winter and helping to take care of our Michigan babies, Blake and Lane. I want to thank my father-in-law, Milt, for also supporting our family to make this endeavor a success. I am also very appreciative of my dissertation committee who provided guidance when needed, but also for allowing me to explore and develop my project independently. I want to thank Dr. Michael Leahy for his wholehearted belief in me that I could do this. I am also appreciative of Dr. Leahy’s sharing of stories that helped me learn a lot along the way. I want to thank Dr. John Kosciulek for his investment and interest in my ideas and writing. I want to thank Dr. Matthew Diemer, especially for his help with dealing with missing data and factor analysis and Dr. Cary Roseth for helping to strengthen the logic and final instrument used for this study. I want to thank Dr. Virginia Thielsen for her guidance and opportunities she provided that helped me learn research skills needed to complete this project. Also, her understanding of what it is like to be a student and family man, I believe helped me maintain perspective in completing this project. iv I want to thank the panelists, Drs. Douglas Strohmer, Vilia Tarvydas, Matthew Diemer, Michael Leahy, and John Kosciulek, who served in an expert capacity and provided important feedback to help strengthen the item content of the instrument used for this study. Without them, my instrument would not have turned out the way it did. I also want to thank the eight participants who took the time to provide extra feedback as part of my pilot study. This information helped to improve my final electronic survey used for this study. I want to thank my friend and colleague, Mr. Chun-Lung Lee, who was extremely helpful when I was going through the complex process of selecting appropriate statistical tests to use for this study. His opinions and knowledge from both a statistical and measurement perspective helped me build confidence in my data analyses. Finally, I also want to thank Ms. Satinder Atwal for all her help with the invitation letter used for this study’s launch and the NCRE research committee for approving the use of their email listserve necessary to recruit participants and conduct this study. v TABLE OF CONTENTS LIST OF TABLES…………………………………………………………………………….....ix LIST OF FIGURES……………………………………………………………………………....xi LIST OF ABBREVIATIONS……………………………………………………………...……xii CHAPTER 1 INTRODUCTION………………………………………………………………………………...1 Statement and Significance of the Problem.…………………………………….………...4 Purpose of Study……………………………………………………………….……..…...5 Definition of Terms…………………………………………………………….……….…8 Assumptions and Limitations……………………………………....................................10 Overview of Study Methodology…………….……………………………………...…...10 CHAPTER 2 LITERATURE REVIEW………………………………………………………………………..12 Clinical Judgment………...………………………………...............................................12 Clinical Judgment in Rehabilitation Counseling………………………………...14 Clinical Judgment Research in Rehabilitation Counseling……………………....16 Theoretical Framework………………………………………………………………..…19 Judgment Heuristics, Cognitive Biases, and Knowledge Structures…………….20 A Scientist-Practitioner Model…….…………………………………………….23 Graduate Education, Experience, and Judgment Accuracy…………………………...…26 Sources of Judgment Error…………………………………………….………....28 Counselor Confirmatory Bias…………………………………………....29 Counselor Cognitive Complexity………………………………………..30 Client Characteristics…………………………………………………….32 Additional Cognitive and Social Factors………………………………...34 The Use of Debiasing Techniques…………….…………………........................35 Evidence-Based Practice………………...……………………………………………….36 Evidence-Based Practice in Rehabilitation Counseling.…………………………37 Evidence-Based Clinical Decision-Making……………………...........................38 Clinical Judgment, Decision-Making Skills, and the CORE Standards…………39 Barriers and Challenges for the Training and Clinical Practice of Evidence-Based Practice in Rehabilitation Counseling……………………………………………40 Summary and Implications……………………………………………………….…...…41 CHAPTER 3 METHOD………………………………………………………………………………………..46 Research Questions……………………………………………………………………....46 Participants….....................................................................................................................48 Instrumentation…………………………………………………………………………..48 Variables………………………………………………………………………………....52 vi Demographic Questionnaire…………………………………………………….……….53 Procedure………………………………………………………………………………...54 Research Design and Data Analysis……………………………………………………..55 Limitations of Study………………………………………………………………….….58 CHAPTER 4 RESULTS……………………………………………………………………………………..…61 Participant Characteristics……………………………………………………………….61 Response Rate……………………………………………………………………62 Demographic and Programmatic Characteristics……………………………...…62 Handling of Missing Data………………………………………………………..65 Exploratory Factor Analysis…………………………………………………………..…66 Importance For Practice and Student Preparedness Scales……..…...…………...66 Teaching Proficiency Scale………………………...…………………………….74 Research Questions………………………………………………………………………76 Research Question 1: How important are clinical judgment skills for effective rehabilitation counseling practice?........................................................................76 Research Question 1a: Does the level of importance differ across clinical judgment skill domains?........................................................................................78 Research Question 1b: Are there differences in the level of importance across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics?.........................78 Research Question 2: What are rehabilitation counselor educators’ perceptions of master’s students’ level of preparedness in clinical judgment skill domains upon graduation to effectively address clinical judgment tasks in rehabilitation counseling practice?...............................................................................................82 Research Question 2a: Does the level of perceived student preparedness differ across clinical judgment skill domains?................................................................83 A Clustering Technique……………………………………………….…83 Research Question 2b: Are there differences in the level of student preparedness across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics?..86 Research Question 3: How do rehabilitation counselor educators perceive their own proficiency in teaching clinical judgment skills?...........................................89 Research Question 3a: Does the level of teaching proficiency differ across clinical judgment skill domains?........................................................................................90 Research Question 3b: Are there differences in the level of teaching proficiency across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics?..90 vii Research Question 4: What are the relationships among rehabilitation counselor educators’ perceived importance, student preparedness, and teaching proficiency of clinical judgment skills?....................................................................................93 CHAPTER 5 DISCUSSION………………………………………………………………………………...….98 Relationships with Previous Clinical Judgment and Training Research……………...…99 High Priority Clinical Judgment Training Needs………………………………………104 Demographic and Programmatic Characteristics……………………………………….105 Program Emphasis of Clinical Judgment Training……………………………..106 Clinical Setting………………………………………………………………….106 Race and Gender………………………………………………………………..107 Relationships Between Importance, Student Preparedness, and Teaching Proficiency..109 Importance and Teaching Proficiency………………………………………….109 Teaching Proficiency and Student Preparedness……………………………….110 Importance and Student Preparedness………………………………………….110 Limitations……………………………………………………………………………...111 Implications……………………………………………………………………………..113 Implications for Clinical Training……………………………………………...113 Future Research………………………………………………………………...118 Conclusion……………………………………………………………………………...120 APPENDICES………………………………………………………………………………….123 Appendix A…………………………………………………………………………..…124 Appendix B…………………………………………………………………………..…128 Appendix C……………………………………………………………………………..132 REFERENCES………………………………………………………………………………....134 viii LIST OF TABLES Table 1. Professional Characteristics………………………………………………………….…63 Table 2. Academic Characteristics………………………………………………………………64 Table 3. Rehabilitation Counseling Program Characteristics…………………………………....65 Table 4. Factor Loadings of Exploratory Factor Analysis with Varimax Rotation of Importance Scale…………………………………………………………………………………………..….69 Table 5. Important Clinical Judgment Skills for Effective Rehabilitation Counseling Practice...72 Table 6. Factor Loadings of Exploratory Factor Analysis with Varimax Rotation of Proficiency Scale…………………………………………………………………………………………...…75 Table 7. Proficiency of Teaching Clinical Judgment Skills…………………………………..…76 Table 8. Highest and Lowest Rated Items of Importance……………………………………..…77 Table 9. Tests of Between-Subjects Effects: Importance of Clinical Judgment Skills……….…79 Table 10. Estimates and Pairwise Comparisons for Gender, Race, and Program Emphasis….…81 Table 11. Individual Clinical Judgment Skills Where Students Are Most and Least Prepared....82 Table 12. High Priority Clinical Judgment Training Needs…………………………………..…85 Table 13. Tests of Between-Subjects Effects: Student Preparedness in Clinical Judgment Skill Domains………………………………………………………………………………………….88 Table 14. Estimates and Pairwise Comparisons for Race and Program Emphasis…………...…89 Table 15. Tests of Between-Subjects Effects: Teaching Proficiency of Clinical Judgment Skill Domains…………………………………………………………………………………….……92 Table 16. Estimates and Pairwise Comparisons for Program Emphasis……………………...…93 Table 17. Summary of Intercorrelations for Overall Mean Scores on the Importance for Practice, Student Prepardness, and Teaching Proficiency Scales…………………………………….……94 Table 18. Relationships Between Importance and Teaching Proficiency of Clinical Judgment Skills…………………………………………………………………………………………..…95 Table 19. Relationships Between Student Preparedness and Teaching Proficiency of Clinical Judgment Skills………………………………………………………………………………..…96 ix Table 20. Relationships Between Importance and Student Preparedness of Clinical Judgment Skills…………………………………………………………………………………………..…97 x LIST OF FIGURES Figure 1. Judgment Heuristics, Cognitive Biases, and Knowledge Structures………………..…20 Figure 2. A Scientist-Practitioner Model…………………………………..…………………….25 Figure 3. Important Clinical Judgment Skills (Debiasing Techniques) For Effective Rehabilitation Counseling Practice………………………………………………………………68 xi LIST OF ABBREVIATIONS CORE Council on Rehabilitation Education CRCC Commission on Rehabilitation Counselor Certification CRC Certified Rehabilitation Counselor NCRE National Council on Rehabilitation Education CACREP Council on Counseling and Related Educational Programs NCC National Certified Counselor CJSI Clinical Judgment Skill Inventory EBP Evidence-Based Practice xii Chapter 1 Introduction Clinical judgment is defined as a clinician’s observations and inferences about uncertain client characteristics or events (Hastie & Dawes, 2001; Pepinsky & Pepinsky, 1954) that are systematic, deliberate, and explicit (Schalock & Luckasson, 2005). Expertise in clinical judgment results from education, clinical experience, and knowledge of individuals and their environment (Schalock & Luckasson, 2005). Within the professional discipline of rehabilitation counseling, clinical judgment is considered an essential part of counseling practice (Strohmer & Leierer, 2000) and has been regarded as an essential skill throughout the clinical interviewing (Berven, 2004, 2008), vocational assessment (Lustig, 1996), and case management processes (Rubin & Roessler, 1980). Clinical judgment and the cognitive processes that can explain rehabilitation counselor judgments can be complex in nature, vary from counselor to counselor, and occur throughout the client rehabilitation process. Rehabilitation counselors have a multifaceted array of job functions and skills required to deliver services in a variety of practice settings (Leahy, Chan, & Saunders, 2003). For example, some of the job duties rehabilitation counselors perform include career counseling, vocational planning, counseling interventions, case management, and assessment services. Within these practice domains, rehabilitation counselors need to (a) assist clients with selecting realistic rehabilitation goals, (b) make behavioral observations and inferences, (c) integrate assessment data for planning, (d) describe clients’ strengths, limitations, and interests, (e) develop positive working relationships, (f) interpret diagnostic information (i.e., medical/psychological or vocational reports), (g) maintain current case records, and (h) select evaluation tools useful for 1 clients (Leahy et al., 2003). These job responsibilities are nowhere near exhaustive of what rehabilitation counselors do on a daily basis, but provide a snapshot of when clinical judgments are necessary. Rehabilitation counselors must also have requisite skills to make clinical judgments during a client interview that involve a chain of decisions that continuously adapt to meet the unique needs of each individual client (Ivey & Ivey, 2008). In particular, rehabilitation counselors need to be well versed in their skill and ability to accurately interpret the meaning of client verbal and non-verbal behavior during the clinical interview process (Berven, 2008). They must also continue to build and facilitate a trusting relationship to draw out important client information that might otherwise be missed (Berven, 2008). Accordingly, accurate clinical judgments are particularly important to ensure clients are given valid information about themselves to make informed choices about their own rehabilitation services and planning (Kosciulek, 1999). In addition, rehabilitation counselors are often required to make accurate clinical judgments related to client eligibility for services, the selection of vocational objectives, counseling interventions, and the development of rehabilitation plans (Berven, 2004). Despite the importance for rehabilitation counselors to be unbiased and accurate in their clinical judgments to accomplish such clinical judgment tasks, they are prone to bias (Strohmer & Leierer, 2000; Athanasou & Kaufmann, 2010; Strohmer, Pellerin, & Davidson, 1995; Rosenthal, 2004; Rosenthal & Berven, 1999; Spengler, Strohmer, & Prout, 1990). The research suggests that biased rehabilitation counselor clinical judgments can translate to (a) whether or not a client qualifies for or receives rehabilitation services (Berven & Rosenthal, 1999; Rosenthal, 2004), (b) a focus on more negative client information (Strohmer, Pellerin, & Davidson,1995), (c) inaccurate predictions about client potential (Strohmer & Leierer, 2000), (d) 2 decisions that are based on perceived client satisfaction (Athanasou & Kaufmann, 2010) and, (e) a minimization of behavior symptoms and needs of clients with intellectual disabilities (Spengler et al., 1990). However, in spite of such rehabilitation counselor clinical judgment and bias research and the importance of making accurate clinical judgments in rehabilitation counseling (Strohmer & Leierer, 2000; Berven, 2008), it is unknown to what extent clinical judgment skills are being explicitly taught in graduate programs that prepare master’s level rehabilitation counselors. In an analysis of the current Council on Rehabilitation Education (CORE) accreditation standards (2011) that form the basis for graduate curricula in rehabilitation counseling, clinical judgment and related terms such as “clinical reasoning” or “clinical decision making” were not found. It may be that some rehabilitation counselor educators assume as possibly in other related disciplines such as nursing (Facione & Facione, 2008) and clinical psychology (Harding, 2007) that clinical judgment knowledge and skills will be developed implicitly as part of the graduate education experience. The new emphasis and stated importance of evidence-based practice (EBP) in rehabilitation counselor education (Leahy & Arokiasamy, 2010; Shultz, Koch, & Kontosh, 2007; Shaw, Leahy, Chan, & Catalano, 2006; Chan, Bezyak, Ramirez, Chiu, Sun, & Fujikawa, 2010; Kosciulek, 2010; Koch, Kiener, & Gitchel, 2009) provides additional support for the need for rehabilitation counselors to develop a high level of clinical judgment skills. For example, to effectively deliver EBP, it is essential that counselors are accurate in their client conceptualizations (Meier, 2003). They also must learn to (a) develop answerable questions to clinical issues, (b) know how and where to locate the best available evidence, (c) critically appraise the evidence, (d) make meaningful translations to effectively apply the evidence to a 3 diverse clientele, and (e) evaluate outcomes of interventions applied (Straus, Glasziou, Richardson, & Haynes, 2011; Kosciulek, 2010; Chan et al., 2010). Statement and Significance of the Problem Given the importance of clinical judgment in rehabilitation counseling (Strohmer & Leierer, 2000; Berven, 2004, 2008; Lustig, 1996; Rubin & Roessler, 1980; Bellini & Rumrill, 1999; Rosenthal & Kosciulek, 1996), prevalence and consequences of rehabilitation counselor biases (Strohmer & Leierer, 2000; Strohmer et al., 1995; Berven & Rosenthal, 1999; Rosenthal, 2004; Spengler et al., 1990), and the emerging trend within the rehabilitation counseling discipline to educate rehabilitation counselors in EBP (Leahy & Arokiasamy, 2010), the explicit teaching of clinical judgment skills that can effectively bridge science and practice to improve rehabilitation counseling clinical judgment accuracy appears critically needed. Such explicit training of clinical judgment in rehabilitation counselor education can provide master’s rehabilitation counseling students the knowledge and skills necessary to recognize and overcome their own biases and improve their own clinical judgment accuracy and decision making when providing rehabilitation services to clients (Berven, 2004; Strohmer & Leierer, 2000). To improve clinical judgment accuracy, counselors must have a high level of selfawareness, clinical reasoning skills, and openness to integrating science with practice (Spengler, Strohmer, Dixon, & Shivy, 1995). Further, it is essential that counselors are knowledgeable of their own cognitive processes (i.e., judgment heuristics, knowledge structures, and cognitive biases) and of the relevant research on how they and other clinicians make clinical judgments (Garb, 1998; Strohmer & Leirer, 2000; Spengler et al., 1995; Meier, 2003). It is also important for counselors to understand how these processes can be biased or even limited (Garb, 1998). For example, counselors may have a limited memory capacity (Garb, 1989), may not be consciously 4 aware of how they make judgments (Garb, 2010), hold onto their initial client impressions and disregard other pertinent information (Strohmer, Shivy, & Chiodo, 1990; Haverkamp, 1993), and become overconfident (Oskamp, 1965). In addition, considerable research has shown counselor clinical judgments to be influenced by social factors such as client characteristics (i.e., gender, race, social class, disability, and age) (Broverman, Broverman, Clarkson, Rosenkrantz, and Vogel, 1970; Lopez, 1989; Rybarczyk, Haut, Lacey, Fogg, & Nicholas, 2001) as well as the context of their clinical setting (Keller et al., 1986; Rosenhan, 1973). Counselors also have a range of conceptual levels that can affect their clinical judgment, particularly under greater uncertainty (Spengler & Strohmer, 1994; Holloway & Wolleat, 1980; Falvey, Bray, & Hebert, 2005). Accordingly, rehabilitation counselors must be skilled at using debiasing techniques effectively to help them overcome their inherent cognitive biases or limitations (Morrow & Deidan, 1992; Spengler et al., 1995; Garb, 1998, 2005; Berven, 2008). Debiasing techniques are defined as the use of internal dialogues that serve as reminders to counselors to question their biases and use of other strategies or tools including scientific-based methods to improve clinical judgment accuracy (Spengler et al., 1995). Moreover, the use of such debiasing techniques has the potential to improve rehabilitation counselor clinical judgment accuracy and quality of direct client services to individuals with disabilities throughout the counseling process (Berven, 2008; Spengler et al., 1995). Purpose of the Study Given the importance of clinical judgment in the literature (Strohmer & Leierer, 2000; Berven, 2004; Garb, 2010) and its importance for major accrediting bodies of counselor training programs, such as CORE (CORE, 2011), it is important to understand both rehabilitation 5 counselor educators’ perceptions of clinical judgment skills and their perceived ability to prepare students in using such skills. Despite few studies that measured the effectiveness of computerbased case simulations for master’s level rehabilitation counseling students learning of clinical problem solving (Berven & Scofield, 1980; Peterson, 2000) and behavior prediction skills (Janikowski, Berven, Meixelsperger, & Roedl, 1989), there were no studies found that provide data useful for better understanding rehabilitation counselor educators’ perceptions of teaching clinical judgment skills for effective rehabilitation counseling practice. Therefore, it is unknown the extent rehabilitation counselor educators perceive clinical judgment skills as being important in the training of master’s level rehabilitation counseling students. It is also unknown how well rehabilitation counselor educators perceive they are proficient in the teaching of clinical judgment skill domains or how well they perceive students are prepared in such areas upon graduation. Consequently, this exploratory study will investigate rehabilitation counselor educators’ (a) perceived importance of teaching clinical judgment skills to master’s students in rehabilitation counseling, (b) perceived level of student preparedness in clinical judgment skills to effectively address clinical judgment tasks upon graduation from master’s rehabilitation counseling programs across the country, and (c) perceived proficiency of their own teaching of clinical judgment skills. The research questions of interest in this study are as follows: 1. How important are clinical judgment skills for effective rehabilitation counseling practice? a. Does the level of importance differ across clinical judgment skill domains? b. Are there differences in the level of importance across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, 6 race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? 2. What are rehabilitation counselor educators’ perceptions of master’s students’ level of preparedness in clinical judgment skill domains upon graduation to effectively address clinical judgment tasks in rehabilitation counseling practice? a. Does the level of perceived student preparedness differ across clinical judgment skill domains? b. Are there differences in the level of student preparedness across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? 3. How do rehabilitation counselor educators perceive their own proficiency in teaching clinical judgment skills? a. Does the level of teaching proficiency differ across clinical judgment skill domains? b. Are there differences in the level of teaching proficiency across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? 7 4. What are the relationships among rehabilitation counselor educators’ perceived importance, student preparedness, and teaching proficiency of clinical judgment skills? Definition of Terms Rehabilitation counselor. A rehabilitation counselor assists people with physical, cognitive, and psychiatric disabilities achieve their psychological, vocational, and independent living goals through the counseling process (Commission on Rehabilitation Counselor Certification (CRCC), 2011). A rehabilitation counselor can be considered a clinician or counselor and for the purposes of the current study such terms are used synonymously. Clinician. A professional practitioner that provides clinical counseling or direct client or patient services within the context of a helping relationship. Counselor. A counselor is defined as a professional that empowers diverse individuals and groups through a counseling relationship to achieve their psychological wellness, educational, and career goals (American Counseling Association, 2011). Clinical judgment. Clinical judgment is defined as a clinician’s observations and inferences about uncertain client characteristics or events (Hastie & Dawes, 2001; Pepinsky & Pepinsky, 1954) that are systematic, deliberate, and explicit (Schalock & Luckasson, 2005). 8 Inferences. Explanations and/or conclusions that are made based on observations of client or patient data that involve making predictions or hypotheses about future client behavior (Pepinsky & Pepinsky, 1954). Hypothesis testing. Hypothesis testing is defined as testing whether or not clinical inferences are accurate through means of client data collection, use of interventions, and debiasing techniques (Spengler et al., 1995). Debiasing techniques. Debiasing techniques are defined as the use of internal dialogues that serve as reminders to counselors to question their biases and use of other strategies or tools including scientificbased methods to improve clinical judgment accuracy (Spengler et al., 1995). Judgment heuristics. Judgment heuristics are defined as cognitive strategies used to simplify more complex cognitive tasks that involve probability and making predictions (Tversky & Kahneman, 1982). Cognitive bias. Cognitive bias is defined as simply an error in judgment (Garb, 1998). Knowledge structures. Knowledge structures are defined as cognitive schemas which are mental representations clinicians make about their clients that can serve as potential contributors to bias (Garb, 1998). Intellectual disability. Intellectual disability is defined as a “…disability characterized by significant limitations both in intellectual functioning and in adaptive behavior, which covers many everyday social and 9 practical skills. This disability originates before the age of 18.” (American Association of Intellectual and Developmental Disabilities, 2011). Assumptions and Limitations The primary assumption of this study relied on the validity of rehabilitation counselor educators’ self-report and knowledge of clinical judgment tasks that relate to their own perceptions of importance, their own teaching proficiency, and level of master’s student preparedness in clinical judgment skill domains. It was assumed that rehabilitation counselor educators who participated in this study would make every attempt to be accurate in their own self-perceptions and possess the requisite knowledge of the clinical judgment tasks necessary for effective rehabilitation counseling practice. Another assumption was that the Clinical Judgment Skill Inventory (CJSI) (See Appendix A) constructed specifically for this study accurately depicted the skill domains of the clinical judgment tasks for effective rehabilitation counseling practice. However, prior to the current study, no reliability or validity studies had been conducted to validate the psychometric properties or construct validity of the CJSI. A primary limitation acknowledged going into this study was the use of self-report questionnaires and use of Internet-based surveys as they run the risk of low response rates (Wright, 2005). Further, self-report surveys are susceptible to participant volunteer bias and response bias (i.e., participants may attempt to guess and confirm researcher hypotheses or answer survey questions in socially desirable ways) (Heppner et al., 1999). Overview of Study Methodology To address the research questions and generate data useful for better understanding rehabilitation counselor educators’ perceptions of teaching clinical judgment skills for effective 10 rehabilitation counseling practice, a quantitative Internet-based ex post facto survey design was used. The population of interest in this study was rehabilitation counselor educators who were employed in master’s level rehabilitation counseling programs. The sampling frame for this study was drawn using a non-probability, convenience sample (Trochim, 2006) from a list generated from the National Council on Rehabilitation Education (NCRE) 2010-2011 membership directory of rehabilitation counseling programs located in the United States (U.S.) (including the District of Columbia), Puerto Rico, Australia, and Canada. 11 Chapter 2 Literature Review The purpose of this exploratory study was to investigate rehabilitation counselor educators’ (a) perceived importance of teaching clinical judgment skills to master’s students in rehabilitation counseling, (b) perceived level of student preparedness in clinical judgment skills to effectively address clinical judgment tasks upon graduation, and (c) perceived proficiency of their own teaching of clinical judgment skills. To provide a context for this investigation, a literature review was needed. The following areas are addressed: (a) clinical judgment, (b) theoretical framework, (c) the relationship between graduate education, clinical experience, and judgment accuracy, (d) evidence-based practice, and (e) summary and implications. Clinical Judgment Clinical judgment is defined as a clinician’s observations and inferences about uncertain client characteristics or events (Hastie & Dawes, 2001; Pepinsky & Pepinsky, 1954) that are systematic, deliberate, and explicit (Schalock & Luckasson, 2005). Expertise in clinical judgment results from education, clinical experience, and knowledge of individuals and their environment (Schalock & Luckasson, 2005). Clinical judgment tasks occur within micro and macro analytic encounters throughout the counseling process (Spengler, et al., 1995). In particular, these tasks include: …all counselor judgment activities regardless of whether these decisions occur at micro or macro levels of analysis. Micro levels of analysis occur during moment-to-moment interactions between the counselor and the client and include counselor impression formation, hypothesis testing, evaluation of in-session change following a therapeutic technique…[and] macroscopic levels of analysis include evaluation of session outcomes, 12 diagnostic and treatment judgments, data from tests and inventories, and, ultimately, final treatment and follow-up outcomes. (Spengler, et al., 1995, p. 507) Throughout this process, the use of mneumonics or self-statements as a form of self-checking to account for personal biases is employed (Spengler et al., 1995) while a clinician actively gathers and synthesizes pertinent client data together to more accurately depict presenting client issues (Groth-Marnat, 2009). Consequently, the accuracy of clinicians’ clinical judgments to address such clinical judgment tasks is of the utmost importance, although counselor judgments are prone to error (Garb, 1998; Spengler et al., 2009; Berven, 2004). To improve judgment accuracy, clinicians’ must have a high level of self-awareness, reasoning skills, and openness to integrating science with practice (Spengler et al., 1995). Further, it is essential that clinicians’ are knowledgeable of their own cognitive processes (i.e., judgment heuristics, knowledge structures, and cognitive biases) of how they and other clinicians’ make judgments (Garb, 1998; Strohmer & Leierer, 2000). It is also important for clinicians to understand how these processes can be biased or even limited (Garb, 1998). For example, clinicians may not be consciously aware of how they make judgments (Garb, 1998). In addition, considerable research has shown clinicians’ judgments to be influenced by social factors such as client characteristics (i.e., gender, race, social class, disability) (Lopez, 1989; Berven & Rosenthal, 1999; Garb, 2010) as well as the context of their clinical setting (Garb, 2010). Clinicians also have a range of cognitive complexity that can affect their judgment (Spengler & Strohmer, 1994). Accordingly, clinicians’ must be skilled at using debiasing strategies effectively to help them overcome these cognitive biases or limitations (Morrow & Deidan, 1992). Debiasing techniques are defined as the use of internal dialogues that serve as 13 reminders to counselors to question their biases and use of other strategies or tools including scientific-based methods to improve clinical judgment accuracy (Spengler et al., 1995). Moreover, the use of debiasing strategies can improve clinicians’ clinical judgment accuracy and quality of direct client services throughout the counseling process (Strohmer & Leirer, 2000; Berven, 2004; Garb, 2010; Spengler et al., 1995; Meier, 2003). Clinical Judgment in Rehabilitation Counseling Within rehabilitation counseling, clinical judgment is considered an essential part of counseling practice, regardless of work setting (Strohmer & Leierer, 2000). It can also be argued that clinical judgment and the decision-making processes that make up counselor judgments can be complex in nature, vary from counselor to counselor, and occur throughout the rehabilitation process. Rehabilitation counselors have a multifaceted array of job functions and skills required to deliver services in a variety of practice settings (Leahy, Chan, & Saunders, 2003). For example, some of the job duties rehabilitation counselors provide include career counseling, vocational planning, counseling interventions, case management, and assessment services. Within these practice domains, counselors need to (a) assist clients in selecting realistic employment goals, (b) make behavioral observations and inferences, (c) integrate assessment data for planning, (d) describe clients’ strengths, limitations, and interests, (e) develop positive working relationships, (f) interpret diagnostic information (i.e., medical/psychological or vocational reports), (g) maintain current case records, and (h) select evaluation tools useful for clients (Leahy et al., 2003). Given such tasks require rehabilitation counselor clinical judgment, the accuracy of counselor judgments are particularly important to ensure clients are given valid information about themselves to make informed choices regarding their rehabilitation services and planning 14 (Kosciulek, 1999). For example, an accurate or valid client conceptualization used to predict client potential is critical in identifying appropriate rehabilitation intervention services and goals, as well as, in developing comprehensive rehabilitation plans to assist clients in overcoming barriers and attaining their personal goals (i.e., vocational, psychological, independent living). Rehabilitation counselors must also have clinical judgment skills during a client interview that involve a chain of decisions that continuously adapt to meet the unique needs of each individual client (Ivey & Ivey, 2008). In particular, rehabilitation counselors need to be well versed in their skill and ability to accurately interpret meaning of client verbal and non-verbal behavior during the clinical interview process (Berven, 2008). They must also continue to build and facilitate a trusting relationship to draw out important client information that might otherwise be missed (Berven, 2008). Berven (2004) also notes that rehabilitation counselors are often required to make clinical decisions and judgments related to client eligibility, selection of vocational objectives, client interventions, the development of rehabilitation plans, and in making disability determinations. Further, clinical judgment in rehabilitation counseling is essential throughout the clinical interviewing (Berven, 2008), vocational assessment (Lustig, 1996), and case management processes (Rubin & Roessler, 1980). Given the potential consequences of counselor judgments within key decision points and throughout the rehabilitation process to clients they serve, having clinical expertise and the skills to make sound clinical judgments and decisions could not be more critical to effectively assist clients in achieving positive outcomes. As such, counselors need to pay close attention to how they make judgments and evaluate their own accuracy of their decisions with each and every client (Koch, Arher, & Wells, 2000). 15 With the complexity of human nature and the need for rehabilitation counselors to digest and make sense of a multitude of client-related information needed to make clinical judgments that are helpful to clients, Berven (2004) suggested that effective rehabilitation counselors use a systematic approach to their work with clients and develop a client model while going through a continual cognitive process of hypothesis testing that incorporates both inductive and deductive reasoning processes. A procedure of continuously testing new information with the old guides counselor decision-making and helps form accurate conceptualizations of clients in order to make more valid predictions. This counselor-as-scientist model (Pepinsky & Pepinsky, 1954) has been a guiding method in the clinical work of many counselors and researchers for decades and has been especially useful for learning about clinical judgment and the thought processes rehabilitation counselors go through to make decisions and judgments about clients (Strohmer & Leierer, 2000). Clinical Judgment Research in Rehabilitation Counseling Despite the importance of clinical judgment in rehabilitation counseling, there is only limited knowledge in the rehabilitation literature that address the underlying cognitive processes (i.e., judgment heuristics, cognitive biases, and knowledge structures) rehabilitation counselors’ go through to make clinical judgments in their work with clients (Strohmer & Leierer, 2000; Athanasou & Kaufmann, 2010; Strohmer, et al., 1995; Rosenthal, 2004; Rosenthal & Berven, 1999; Spengler et al., 1990). In particular, models of rehabilitation counselor clinical judgment (Strohmer & Leierer, 2000; Athanasou & Kaufmann, 2010), racial bias in clinical judgment (Rosenthal, 2004; Rosenthal & Berven, 1999), disability bias (Spengler et al., 1990), negative bias (Strohmer et al., 1995), and computer-based case simulation training (Berven & Scofield, 1980; Peterson, 2000; Janikowski et al., 1989) were the only studies found that addressed either 16 the rehabilitation counselor clinical judgment process or training methods to enhance clinical judgment skills. Strohmer and Leierer (2000) studied a model of rehabilitation counselor clinical judgment and found that rehabilitation counselors’ may make predictions about client potential based on their observations and inferences from their perception of client functioning level and underlying cause. In a case study, Athanasou and Kaufmann (2010) used an analogue to study influential factors as well as cognitive strategies that affected judgment accuracy of an “expert” rehabilitation counselor (defined as having 30 years of counseling experience). The researchers found that a complex problem-solving style was used by the expert when only a simple one was needed. Further, perceived client satisfaction was a mechanism that drove the rehabilitation counselor’s decision process that involved an individualized approach in attempt to balance various client data. In other studies, racial bias was found in rehabilitation counselors’ clinical judgments. In an experimental study by Rosenthal and Berven (1999), rehabilitation counseling master’s students were randomly assigned to judge case materials of either African American or White/European American client groups. They found that African American clients were judged significantly more negatively and to have less potential than White clients. In a related study of White/European American rehabilitation counselors, Rosenthal (2004) found similar results in that White rehabilitation counselors judged African American clients more negatively in relation to academic and employment potential than their White clients. Consequently, counselor judgments can result in whether or not a client qualifies or receives rehabilitative services (Berven & Rosenthal, 1999; Rosenthal, 2004). For example, rehabilitation counselors are charged with making disability determinations, eligibility decisions, and in assessing client needs 17 to help develop beneficial goals and interventions (Berven, 2004). As can be seen here, errors in counselor judgments when accomplishing such tasks can be detrimental and result in clients’ not receiving the best available interventions or services or not qualifying for services at all. Strohmer et al. (1995) studied rehabilitation counselor hypothesis testing as a function of how counselors make clinical judgments about clients and found that counselors’ selectively attended to more negative client information when presented with equal amounts of both positive and negative client information. Further, more experienced counselors attended to more negative client information than less experienced counselors and type of information (positive or negative) attended to was positively related to rehabilitation counselor clinical judgments (favorable or less favorable). In a study of the overshadowing bias among rehabilitation counselors, Spengler et al. (1990) found that rehabilitation counselors, like other professionals in similar helping disciplines, have a tendency to minimize mental health symptoms of clients with intellectual disabilities. In particular, researchers found that recommendations for psychological counseling or psychotropic medication treatment occurred significantly less for lower intellectual client profiles (i.e., Intelligence Quotient of 58). Such recommendations were also negatively correlated with amount of time spent providing direct client services to individuals with intellectual disabilities. Three studies were conducted to better understand how computer-based instructional methods can be used to enhance the training of master’s level rehabilitation counseling students’ in developing clinical problem solving (Berven & Scofield, 1980; Peterson, 2000) and behavioral prediction skills (Janikowski et al., 1989). 18 In studying a computerized case management simulation used to train clinical problemsolving skills, Berven and Scofield (1980) compared master’s level rehabilitation counseling students with experienced vocational rehabilitation counselors who were used as a criterion for skill competencies. The researchers suggested multiple potential uses of such a program as a training and evaluative tool for developing rehabilitation counseling students’ clinical problemsolving and decision-making skills. In a related study, Peterson (2000) found that a computerized case simulation identified unique problem-solving approaches of individual rehabilitation counseling students. Based on the findings of this study, Peterson suggested that a similar computerized case study program be used to complement student learning of both theoretical and applied skills. Further, computer simulation could also be used to pinpoint deficiencies to promote more individualized training and growth in learning more effective problem-solving techniques. In addition, Janikowski et al. (1989) found that a computer-based case simulation training helped rehabilitation counseling students better predict client behavior. Further, the researchers indicated that such an instructional strategy demonstrated potential for evaluating student skills in rehabilitation counseling competency areas. Also, directing students to talk out loud in working through case simulations was considered an additional benefit to using this type of computer simulation training method. Theoretical Framework Two conceptual frameworks provide the basis for this study. An overview of (a) judgment heuristics, biases, and knowledge structures, and (b) a scientist-practitioner model will be provided. 19 Judgment Heuristics, Cognitive Biases, and Knowledge Structures Extensive research has shown that multiple cognitive processes are at play at any given time that can influence how counselors make judgments (Garb, 1998). Judgment heuristics, cognitive biases, and knowledge structures have been used extensively as a framework for the study of how clinicians make clinical judgments (Garb, 1998) (See Figure 1 below). Judgment Heuristics, Cognitive Biases, and Knowledge Structures Figure 1. Cognitive processes that influence clinical judgments. For interpretation of the references of color in this and all other figures, the reader is referred to the electronic version of this dissertation. Judgment heuristics are defined as cognitive strategies used to simplify more complex cognitive tasks that involve probability and making predictions (Tversky & Kahneman, 1982). Tversky and Kahneman (1982) describe three types of heuristics: representativeness, availability, and adjustment and anchoring. 20 Representativeness is activated when comparisons based on one’s familiarity or stereotype is assigned to a particular circumstance. For example, a rehabilitation counselor might have a specialized caseload working only with clients with traumatic brain injury and therefore may make inferences regarding vocational potential about a particular client based on experiences with other clients deemed similar. Availability is utilized when inferences are made from information that easily surface to the forefront of one’s consciousness. For example, a rehabilitation counselor may have recently received a phone call from a client who reported getting a job in customer service at a call center and then use this information to tell another client that the job market is promising for call center positions. Adjustment and anchoring occur when individuals hold on to their initial impression about something or someone else where comparisons are made or altered based on one’s initial impression. For example, a rehabilitation counselor might review case records prior to an initial interview and after meeting with a client, proceed to form an opinion based on what was initially read in the case record. The past-behavior heuristic occurs when predictions of future behavior is based from one’s past (Garb, 1998). For example, a rehabilitation counselor might expect a client to be more likely to achieve an employment goal if a client has a long and stable work history. The affect heuristic occurs when an individual’s emotional reactions or “gut feelings” in response to an external stimulus are used either consciously or unconsciously to guide one’s judgment or decision making (Slovic, Finucane, Peters, & MacGregor, 2002). For example, a rehabilitation counselor who has a positive emotional response to a client who appears motivated 21 for change might judge this client as having high potential to benefit from services and may then more readily invest in the helping relationship. Cognitive bias is defined as simply an error in judgment (Garb, 1998). Examples of cognitive bias include (a) confirmatory bias: when clinicians seek or attend to only information that confirms their initial hypothesis, (b) hindsight bias: when clinicians learn of an outcome and then perceive the outcome as being more likely to have occurred after the fact, (c) misestimation of covariance: when clinicians do not accurately recall how pertinent client information relates to specific client outcomes or make cause and effect correlations that appear related but are not (Garb, 1998), (d) fundamental attribution error: when clinicians make causal inferences about clients that consider only their traits and not environmental factors, and (e) diagnostic overshadowing: when clinicians attend too much to what appears to be the presenting client issue and minimizes other essential client factors (Spengler, et al., 1995). Knowledge structures are defined as cognitive schemas which are mental representations clinicians make about their clients that can serve as potential contributors to bias (Garb, 1998). Such cognitive schemas are used to assist counselors to make sense of information by means of comparing a client’s characteristics or behaviors with others who are considered similar (Garb, 1998). Knowledge structures (exemplars, stereotypes, and prototypes) are activated when clinicians utilize the representative heuristic. Exemplars are used when clinicians make judgments about their clients based on direct experience with other clients who may present the same way. Stereotypes are used when clinicians make judgments about their clients that are based on general information about a client population, and prototypes are used when clinicians judge their clients based on their theoretical understanding of a particular client issue. 22 Knowledge structures may also be developed from a clinician’s counseling theory orientation, use and knowledge of decision aids (e.g., Diagnostic and Statistical Manual of Mental Disorders), knowledge of the relevant research, or through clinical experience (Falvey, Bray, & Hebert, 2005). Having knowledge of such mental processes (i.e., judgment heuristics, knowledge structures, cognitive bias) and the current counselor bias and judgment research can inform rehabilitation counselors of how they make clinical judgments and decisions (Strohmer & Leierer, 2000). Further, if counselors have knowledge of their own thought processes and cognitive limitations in relation to making clinical judgments they will likely be better positioned to account for bias and minimize judgment error in counseling practice (Garb, 2005; Meier, 2003). A Scientist-Practitioner Model A second theoretical framework used to guide the current study is a scientist-practitioner model (Spengler et al., 1995) based on human inference theory and research (Nesbitt & Ross, 1980) grounded in Pepinsky and Pepinsky’s (1954) original “counselor-as-scientist” model. The model incorporates two categories of prerequisite attitudes that are considered necessary to improve counselor judgment accuracy; Counselor self-awareness and Counselor Openness/Curiosity. Counselor self-awareness considers the importance of counselor’s being aware of their values, biases, theoretical assumptions, and how they process information (Spengler et al., 1995). Counselor Openness/Curiosity includes the need for counselors to be open to new and alternative explanations, making tentative decisions and formulations in work with clients, having a tolerance for ambiguity, and utilizing and integrating knowledge that is based in theory and research into their practice (Spengler et al., 1995). 23 This model focuses on clinical judgment tasks that occur within micro and macro analytic encounters throughout the counseling process (Spengler et al., 1995). In particular, these tasks include: …all counselor judgment activities regardless of whether these decisions occur at micro or macro levels of analysis. Micro levels of analysis occur during moment-to-moment interactions between the counselor and the client and include counselor impression formation, hypothesis testing, evaluation of in-session change following a therapeutic technique…[and] macroscopic levels of analysis include evaluation of session outcomes, diagnostic and treatment judgments, data from tests and inventories, and, ultimately, final treatment and follow-up outcomes. (p. 507, Spengler et al., 1995) Throughout this process, the use of mneumonics or self-statements as a form of self-checking to account for personal biases is employed (Spengler et al., 1995) while a clinician actively gathers and synthesizes pertinent client data together to more accurately depict presenting client issues (Groth-Marnat, 2009). As shown in Figure 2 below, this client model is considered cyclical and non-prescriptive and used to guide counselors through the interactive and on-going client assessment and decision making process (Spengler et al., 1995). This model views counseling research and practice as being intertwined with an underlying attitude of appreciation of science when engaging in the counseling process. Further, the cyclical process this framework suggests starts with the counselor making client observations that lead to inferences and working hypotheses that are considered tentative in nature and continuously revised to accommodate new client information as a way to make predictions (i.e., vocational goals, rehabilitation services) regarding future client behavior (Spengler et al., 1995). 24 A Scientist-Practitioner Model (Spengler et al., 1995) Counselor Self-Awareness Values; Biases; Preferences; Theoretical Assumptions; Information Processing Styles Counselor Openness/Curiosity New Explanations; Tentativeness; Ambiguity; Theory; Practice Observation* Inference* Disconfirm. Hypothesis* Alternative Hypothesis* Test Disconfirm. Additional Data Collection Confirm. Hypothesis* Test Confirm. Hypothesis Test Alt. Hypothesis Tentative Rejection(s) No* Yes* *Debiasing Techniques Tentative Working Client Model(s) Figure 2. A number of debiasing techniques or internal dialogues that serve as reminders to counselors to question their biases and use of other strategies or tools including scientific-based methods, must be considered when implementing this model, especially when asterisks (*) are noted as important counselor decision-making or judgment points within the model. Since there are several research studies that show confirmatory bias in counselor judgment (Strohmer et al., 1990; Strohmer & Shivy, 1994), that is, the tendency for a counselor to make initial client inferences and then proceed to seek and attend to only information that confirms this initial inference, both disconfirmatory and alternative hypothesis testing are added to the model to facilitate a counselor’s conscious and active hypothesis testing process that can 25 lead to more accurate tentative hypotheses (Spengler et al., 1995). Further, a number of debiasing techniques are used to improve counselor clinical judgment accuracy at judgment and/or decision-making points within the model. Debiasing techniques are defined as the use of internal dialogues that serve as reminders to counselors to question their biases and use of other strategies or tools including scientific-based methods to improve clinical judgment accuracy (Spengler et al., 1995). Debiasing techniques can include, but are not limited to: considering probability and base rate data, combining actuarial and clinical prediction techniques, delaying judgments, and reducing overconfidence in client hypotheses generated. The implementation of this model requires high-level critical thinking skills, but needed skills for the competent scientist-practitioner to engage in this cyclical scientific process that is continuous throughout the course of the counselor-client relationship and serves as a guide for making more valid counselor judgments (Spengler et al., 1995). Graduate Education, Experience, and Judgment Accuracy Beginning in the 1950’s, scholars in counseling psychology started a movement for the personal accountability of clinicians in their work with clients as the profession knew very little about how clinicians went about making diagnostic decisions in regard to their clients (McArthur, 1954). Early studies showed a lack of relationship between the professional training of graduate students and their clinical judgments related to client diagnosis, behavior predictions, and interpersonal skill development (Carkhuff, 1968; Goldberg, 1968). For example, Carkhuff (1968) provided a review of the early research literature and concluded that there was no difference in terms of therapeutic client change outcomes between professional and lay helpers. Also, Goldberg (1968) reviewed early studies of accuracy in clinical judgment and also found 26 similar results of which seemed to suggest that professional training did not matter in relation to clinicians making accurate judgments. A study by Oskamp (1965) indicated that as judges at various levels (i.e., undergraduate students, graduate students, and experienced post-graduate clinicians) received more information about a case, the confidence of their judgment accuracy increased. However, these participants were no more accurate and tended not to change their responses. These findings suggested that there may be a “ceiling” effect or a point where predictive accuracy does not improve, regardless of amount of clinical training or experience. Since these early years, an extensive body of knowledge about clinical judgment has developed, but not without debate in regard to the findings. To what extent counselor education and clinical experience can improve clinical judgment skills appears to be at the crux of the current debate. In a comprehensive review of the research, Garb (1998) found significant differences in clinical judgment skills between novice and advanced graduate students. Specifically, Garb concluded that graduate students’ clinical judgment accuracy appears to improve from the beginning to advanced clinical phases of the graduate training process, but not after. In a recent meta-analysis, Spengler et al. (2009), who used rigorous coding methods to eventually deduce 35,000 clinical judgment studies (from 1970 to 1996) to 75, found that clinicians were more accurate at developing valid treatment recommendations based on having more training and clinical experience than less experienced clinicians. More specifically, expert clinicians were approximately 13% more accurate in their judgments than novice clinicians and as clinical experience increased, decision making abilities in relation to judgment accuracy also appeared to improve. Another notable finding was that experienced clinicians were better able to 27 make accurate predictions under greater uncertainty. In a similar meta-analysis of clinical judgment accuracy studies (from 1997 to 2010) of mental health professionals, Pilipis (2011) also found a significant positive relationship between clinical judgment accuracy and clinical experience, although the effect size was small. These two meta-analysis findings are significant given that a number of studies have countered these conclusions by suggesting that counselors are typically no more accurate in their clinical judgments than advanced graduate students regardless of experience level (Garb, 1998). Such findings also contrast with Lichtenberg’s (1997) review of the research who concluded that there is lack of evidence to support the notion that education or clinical experience effect clinical judgment accuracy. It is important to clarify that the research reviewed includes the literature that measured relationships between education, clinical experience, and judgment accuracy, and not client outcomes, although this can be inferred. Counselor clinical experience and its effect on client outcomes were considered to be a separate body of research and outside the scope of this literature review. Sources of Judgment Error There have been numerous papers and studies that shed light as to why graduate education and clinical experience may not produce more desired effects or why counselors do not develop more refined and advanced clinical judgment skills. The primary factors commonly found in the research relate to counselor confirmatory biases, counselor cognitive complexity, client characteristics, and additional cognitive and social factors (Garb, 1998; Garb, 1989; Garb, 2010; Spengler, et al., 2009). 28 Counselor confirmatory biases. There are multiple studies in the counseling literature that have shown counselors to hold a confirmatory bias, or more likely to remember and attend to information that confirms their initial clinical impression of a client (Strohmer et al., 1990; Strohmer & Shivy, 1994; Owen, 2008). For example, in an analogue study of master’s level counselors, Strohmer et al. (1990) assigned participants to one of three groups (high, medium, and low-confirmatory report conditions). Each participant was asked to review and study one of three case narratives relative to their assigned report condition. One week later participants were asked to review additional materials that assigned a clinical hypothesis (i.e., loss of self-control) to the initial case narrative reviewed. Participants were asked to recall evidence that either supported or disconfirmed the clinical hypothesis. Findings showed counselors to recall significantly more confirmatory evidence to support the self-control hypothesis across the three groups even when the initial narrative provided substantially more disconfirmatory evidence. In another study by Strohmer et al. (1990), counselors-in-training in their graduate program participated in essentially the same study, but were asked to respond to all the case materials instead of waiting a week. As in the prior study, counseling graduate students provided significantly more confirmatory responses than disconfirmatory responses across the three groups. It is also important to note, that for both studies, participants’ level of certainty increased about the clinical hypothesis (i.e., loss of self-control) the more confirmatory evidence they were able to provide. Strohmer & Shivy (1994) conducted two similar studies with doctoral counseling and school psychology students, and master’s level counselors, and found participants again to recall significantly more confirmatory than disconfirmatory information when given equal amounts of 29 each in relation to the provided clinical hypothesis (i.e., loss of self-control). This same finding was also found when the experimental situation created a sense of increased accountability to better simulate a real case. In addition, Owen (2008) conducted a study of doctoral and master’s students in mental health counseling that measured their initial judgments of factitious case vignettes to further test the confirmatory bias hypothesis. Similar to previous studies, Owen found mental health counselors to provide significantly more confirmatory information about a clinical hypothesis (i.e., anxiety or personal growth) than disconfirmatory evidence. Further, it was also found that questions asked to confirm clinical hypotheses were more often prompted by initial client information or related to clarifying a diagnosis. Questions were also more concise than questions seeking disconfirmatory evidence. Counselor cognitive complexity. There appears to be a range of complexity in terms of reasoning processes that counselors possess that are hierarchical in nature that have been shown to be linked to clinical judgment accuracy (Falvey et al., 2005; Spengler & Strohmer, 1994; Holloway & Wolleat, 1980). Owen and Lindley (2010) define cognitive complexity as the ability to (a) selectively attend to and differentiate between pertinent clinical details, (b) have awareness of one’s own cognitive processes, and (c) reason through uncertain clinical situations. In a study that measured conceptual level (CL), Holloway & Wolleat (1980) found that students who were more cognitively complex thinkers, or high CL, were better able to find new sources of information and more skilled in articulating their hypotheses than cognitively simple thinkers, or low CL (Holloway & Wolleat, 1980). In a related study, Spengler and Strohmer (1994) found that counselors with lower cognitive complexity were more likely to 30 underdiagnosis clients. Further, it was also inferred from this same study that counselors with higher cognitive complexity may be (a) less likely to use the stereotyping bias, (b) better able to combine conflicting client information, (c) potentially more accurate in their judgments, and (d) better equipped to avoid other biases. In a unique qualitative study using a “think out loud”, process-tracing methodology, Falvey et al. (2005) studied the problem-solving methods and clinical judgment strategies for case conceptualization and treatment planning. The researchers of this study also were particularly interested to see if there were differences between professional disciplines (i.e., psychologists, mental health counselors, and social workers). Their findings were notable and revealed further support for previous research of cognitive complexity as clinicians who engaged in more complex judgment strategies were found to be more able to integrate information into an explainable schema and provide more accurate judgments. In contrast, clinicians who did not use a more complex cognitive strategy and who also lacked clinical experience did much worse on the clinical judgment tasks. The researchers also note that no quantifiable differences between mental health disciplines were observed which supports the notion that the approach to clinical judgment tasks (i.e., case conceptualization and treatment planning) is unique to each individual practitioner and not a function of discipline. These findings suggest that counselors with a more complex reasoning style or greater cognitive complexity may be better suited to synthesize and integrate pertinent and complex client information under greater uncertainty, be less prone to bias, and may then be better positioned to make more accurate clinical judgments in their work with clients (Falvey et al., 2005; Spengler & Strohmer, 1994; Holloway & Wolleat, 1980). 31 Client characteristics. Counselors can be vulnerable to racial bias (Rosenthal, 2004; Rosenthal & Berven, 1999), and may hold biases related to client disabilities, social class (Lopez, 1989), gender (Broverman et al., 1970), and age (Rybarczyk et al., 2001). In an extensive research review of client variables that can affect counselor clinical judgment, Lopez (1989) noted a consistent counselor bias toward individuals with intellectual disabilities (i.e., under diagnosing or minimizing symptoms), and low socio-economic status (i.e., over diagnosing). Lopez also found the results of age and gender bias to be inconsistent, with racial bias showing more consistent empirical support when diagnostic judgments were being made. Age bias. In a further review of studies that measured the affect of age bias on clinical judgments, findings were mixed, but leaned toward a lower perception of potential and functional capabilities toward older adults. For example, Rybarczyk et al. (2001) sampled clinicians across the medical rehabilitation discipline (e.g., occupational therapists, physiatrists, vocational counselors, speech pathologists, etc.) and found that older patients were perceived to be less worthy and to have less potential for benefiting from rehabilitation services. However, Saxon and Spitznagel (1992) did not find significant age bias in decision-making by vocational rehabilitation counselors when client feasibility ratings for achieving job outcomes were compared across client age groups. Danzinger and Welfel (2000) found a sample of licensed clinical social workers and clinical counselors, ages 28 to 75 with one to 45 years of experience, to judge older clients (72 vs. 52 years old) as less competent and to have a worse prognosis than younger clients. Further, the more experienced practitioner, the more negatively biased they were in relation to rating 32 client competency. These findings were also supported by Meeks (1990), who studied the diagnostic decisions of clinical psychologists and graduate students-in-training and found that participants expected “elderly” clients to benefit significantly less from psychological treatment than younger clients. Additionally, another similar finding was found as a result of James and Haley’s (1995) study. They randomly selected experienced clinical psychologists, who were middle-aged on average, and who served clients 65 years of age and older. Potential age bias was measured related to clinical decisions made about clients aged 35 or 70. Consistent with previous studies (Danzinger and Welfel, 2000; Meeks, 1990; Rybarczyk et al., 2001), James and Haley found that older clients were viewed as being less able to benefit from treatment interventions and had more negative prognosis’ than younger clients. Gender bias. The research of gender effects on clinical judgments have also been mixed over time (Garb, 2010). In a study of gender effects on judgments made by psychologists, psychiatrists, and social workers, Broverman et al. (1970) found that females were differentially judged compared to males or adults in relation to what is perceived to be psychologically healthy behavior. For example, stereotypic feminine traits (i.e., being passive, dependent, overly emotional, etc.) were deemed psychologically healthy behavior for females, but not for males or adults. Whereas, more masculine traits (e.g., being competitive, aggressive, etc.) were viewed as psychologically healthy behavior for males. Such findings provide empirical support for stereoptypic attitudes held by clinicians (both men and women) based on gender. However, in a relatively recent critique of Boverman et al.’s (1970) study, Kelley & Blashfield (2009) cite several pieces of literature that point out methodological limitations of this research and suggest no such bias. 33 Additional cognitive and social factors. Additional cognitive and social factors that can bias counselor judgments have been described in the clinical judgment literature. For example, Garb (1989) suggested that clinicians may not receive adequate or even accurate feedback. Counselors may not be consciously aware of how they make decisions (Garb, 1998). They also have a limited ability in what they can take in and process at any given time (Garb, 1998). Garb (1989) also suggested that counselors may not be using an organized client hypothesis testing strategy (e.g., counselor-as-scientist) as a reason why clinicians may not learn from their clinical experiences. Counselors have a limited memory capacity (Garb, 1989) and may be biased in how they reconstruct memories of their clients (Loftus, 1993). There are also varying cognitive styles that counselors use to make decisions that can influence judgments (Gambrill, 2005). Such decision making styles include rational, analytical styles versus intuitive ones, or a mixture of both (Gambrill, 2005). Counselors may also be significantly influenced by contextual issues within their practice setting (Keller et al., 1986; Rosenhan, 1973; Garb, 2010). For example, in a study by Keller et al. (1986), similar patient variables (i.e., major depressive disorder, mania, or schizophrenia) and recommended treatments (e.g., psychotherapy, drug therapy, etc.) of a sample of 338 patients were studied across five medical (inpatient and outpatient) centers. The researchers found that patient characteristics were not reliable predictors of treatments prescribed. Further, there was significant variability of treatments delivered between the five medical centers which were interpreted to be a function of clinician decision making differences between university centers. Similarly, Rosenhaun (1973) conducted a study of eight participants who posed as pseudopatients within 12 psychiatric hospitals. The hospital floor staff were unaware of this 34 study. Rosenhaun found that psychiatric diagnoses (e.g., schizophrenia) were easily made and all pseudopatients were admitted. After admission, pseudopatients then acted as they normally would. Participants were hospitalized on average of 19 days with a range of 7 to 52 days despite not actually experiencing severe behavioral symptoms. Further, it was concluded that the context of working in such hospital settings appeared to cloud the judgment of professional staff and their ability to differentiate between patients who had legitimate mental illnesses and those who did not. The Use of Debiasing Techniques Given the extensive clinician bias research, counselors must accept the fact that how they process client information used to make clinical judgments and decisions can produce inaccurate judgments (Faust, 1986; Nisbett & Ross, 1980; Strohmer & Leierer, 2000). Gambrill (2005) suggests clinicians can learn to overcome their limitations in thinking errors by developing metacognitive strategies that involves instituting a self-reflective, self-corrective stance that continuously questions one’s own thoughts and actions. Consequently, all counselors should have awareness of their information processing errors and strategies that can help trigger use of debiasing techniques to make more accurate clinical decisions (Arkes, 1981; Garb & Grove, 2005). Awareness of how individual clinician’s make decisions can also help inform counselors about the judgment heuristics they use under clinical conditions (Hoshmand, 1991). Further, a number of debiasing strategies can and should be used by clinician’s to enhance the accuracy of their judgments and decisions (Hoshmand, 1991; Faust, 1986; Garb, 1998; Spengler et al., 1995; Meier, 2003; Gambrill, 2005; Goodheart, 2006; Moran & Tai, 2001; Arkes, 1981; Bell & Mellor, 2009; Morrow & Deidan, 1992; Arnoult & Anderson, 1988; Egisdottir et al., 2006; Haverkamp, 35 1994). Debiasing techniques often cited from this literature include (a) remaining open to alternative explanations and considering alternative hypotheses, (b) applying statistical and clinical prediction techniques, (c) considering probability and base rate data, (d) using multiple sources of assessment, (e) seeking out accurate feedback, (f) focusing on client strengths, (g) using clinical practice guidelines, and (g) delaying judgments. In addition, extensive judgment and clinical training literature and research has also proposed theoretical models (Spengler et al., 1995; Goodheart, 2006; Pepinsky & Pepinsky, 1954; Bellini & Rumrill, 1999; Stricker & Trierwiler, 1995; Chwalisz, 2003; Chan et al., 2010; Kosciulek, 2010; Facione & Facione, 2008; Morran et al., 1995; Owen & Lindley, 2010; Schon, 1983; Koch, Kiener, & Gitchel, 2009; Stoltenberg & McNeill, 2010; Granello, 2000; Sexton, 2000) and empirical evidence (Falvey et al., 2005; Spengler & Strohmer, 1994; Holloway & Wolleat, 1980; Fong, Borders, Ethington, & Pitts, 1997; Duys & Hedstrom, 2000; Little, Packman, Smaby, & Maddux, 2005; Whiston & Coker, 2000; Buser, 2008) that supports the need for counselors to develop more advanced cognitive skills. Such skills are needed to implement more increasingly cognitively complex models of clinical practice which in and of itself can be considered a debiasing technique (Spengler et al., 1995). Evidence-Based Practice EBP was originated from medicine and has been around since the mid-19th century (Sackett, 1997; Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996). However, it only started to gain traction in the United States within the mental health field in 1993 when psychology adopted empirically-supported treatments (EST) (Wampold & Bhati, 2004; Tarvydas, Addy, & Fleming, 2010). EBP has become popular across helping disciplines and is now emphasized within the rehabilitation counseling discipline as a newly integrated model of 36 practice (Leahy & Arokiasamy, 2010). Medicine (Straus et al., 2011), psychology (Goodheart, Kazdin, & Sternberg, 2006), nursing (Tanner, 2008), social work (Howard, Allen-Meares, & Ruffolo, 2007), counseling (Sexton, 1999), and rehabilitation counseling (Chan et al., 2009) have all offered perspectives of how to best integrate EBP given its benefits and challenges in providing direct services to clients. EBP in Rehabilitation Counseling In the discipline of rehabilitation counseling, the underlying philosophy of EBP is for clinicians to use the best available evidence in relation to interventions that lead to the best possible client outcomes (Chan et al., 2009). Chan et al. (2009) suggests that the rehabilitation counseling discipline can no longer make clinical decisions based on logic and intuition because managed care and accountability for outcomes of government programs and providers will continue to increase (Chan, Rosenthal, & Pruett, 2008). It is this bridge between research and practice that EBP can be considered a promising practice in rehabilitation counseling (Johnston, Vanderheiden, Farkas, Rogers, Summers, & Westbrook, 2009). Johnston et al. (2009) suggests EBP be defined as a combination of the best available interventions (i.e., counseling strategies, environmental changes, policy changes, transportation, etc.), but also infused with the expertise and clinical judgment of the rehabilitation counseling professional. However, to effectively identify and integrate the best available interventions, rehabilitation counselors will need to be skilled in making accurate client observations and inferences (Pepinsky & Pepinsky, 1954) and able to identify, appraise and apply the best available research into their practice (Kosciulek, 2010; Chan et al., 2010). 37 Evidence-Based Clinical Decision-Making Given the need to demonstrate service effectiveness and accountability (Chan et al., 2008), making clinical decisions that are evidence-based is critically important and top priority in rehabilitation counselor education (Leahy & Arokiasamy, 2010; Shultz et. al., 2007; Shaw et al., 2006). This clinical decision making process involves: (a) articulating a question about a clinical problem, (b) locating the best research available to address the problem, (c) appraising the validity and clinical applicability of evidence, (d) implementing the most useful findings that address the clinical questions with clients in practice, and (e) evaluating outcomes of interventions applied (Straus et al., 2011; Kosciulek, 2010; Chan et al., 2010; Chronister et al., 2008). Evidence-based clinical decision-making is a shared and informed process within a counselor-client helping relationship and considered essential for ethical practice (Gambrill, 2005). Further, clinical decision making directs the clinician to use the best available evidence in relation to interventions that lead to the best possible client outcomes (Chan et al., 2009). However, teaching an EBP model of practice used to inform clinical decisions with clients will pose significant challenges for the rehabilitation counseling discipline. Bridging the long-standing gap between research and practice in rehabilitation counseling (Chan et al., 2008; Chan, Miller, Lee, Pruett, & Chou, 2004; Graves, 1992; Johnston et al., 2009; Tarvydas et al., 2010) will not be easy. This research-practice divide is not new for other professional disciplines as well. For example, the scientific-practitioner model or “boulder model” which took form in the 1950’s and 60’s in professional psychology as their signature model for training clinical and counseling psychologists was intended to effectively integrate research and practice (Chwalisz, 2003). However, scholars in the professional psychology 38 literature have voiced discontent regarding the achieved effectiveness of this model integrating the two (Trierweiler, 2006; Stricker & Trierweiler, 1995; Chwalisz, 2003). In 2006, EvidenceBased Psychotherapy (Goodheart, Kazdin, & Sternberg, 2006) was published and offers arguments and suggestions that can be translated across helping disciplines to improve the utility and integration of research and practice within the helping disciplines (i.e., counseling, rehabilitation counseling, social work, etc.). Clinical Judgment, Decision-Making Skills, and the CORE Standards Clinical judgment and decision-making skills appear to be mostly absent within the newly revised CORE standards (CORE, 2011). However, the need for the explicit training of clinical judgment (Facione & Facione, 2008) and clinical decision-making skills (Harding, 2007) has been recently advocated for in both the nursing and clinical psychology clinical training literature, respectively. Further, there appears to be a limited emphasis in education on the skill competencies necessary for effective decision-making, including how to overcome judgment errors (Gambrill, 2005). Consequently, clinical judgment skills training needed to make evidence-based clinical decisions may need to come from educators who are visionaries who also share in the belief that it is critically important to explicitly train clinical judgment and decision making skills to students. Given the recent urgency within rehabilitation counselor education and newly adopted Code of Professional Ethics for Rehabilitation Counselors (CRCC, 2009) to move toward EBP (Johnston et 2009; Leahy & Arokiasamy, 2010; Burker & Kazukauskas, 2010) and the importance of clinical judgment in rehabilitation counseling practice (Strohmer & Leierer, 2000; Berven, 2004, 2008; Lustig, 1996; Rubin & Roessler, 1980), the timing and need for the explicit training of clinical judgment and decision-making skills is now. 39 To address such challenges, the rehabilitation counseling discipline can consider adopting an EBP or scientist-practitioner model for the teaching and practice of rehabilitation counseling. This idea is not new and has been advocated for many years (e.g., Bellini & Rumrill, 1999; Leung, 1987; Chan et al, 2009; Dellario, 1996; Koch et al., 2009). An EBP model can be infused throughout the master’s program curriculum. A new approach to teaching research skills can be taught in a way that favors its applicability to delivering counseling services (Heppner et al., 1999). Further, master’s students can be informed from the get go of how to use research to inform practice as an ethical duty (CRCC, 2009) and be taught that using research to inform practice is what rehabilitation counselors are expected to do. Spengler et al. (1995) provide an example of an EBP model that can be infused within the master’s rehabilitation counseling curriculum. This model prioritizes counselor clinical judgment and debiasing techniques to improve judgment accuracy (Spengler et al., 1995) and decision making throughout the counseling process. Barriers and Challenges for the Training and Clinical Practice of EBP in Rehabilitation Counseling Research skills may be taught more to understand research and not necessarily for its use and application purposes in actual counseling practice (Heppner, 1999). Chan et al. (2004) also suggest that rehabilitation counseling master’s students often view research as their least favorite class. Role and function and knowledge and skill studies overtime have demonstrated that Certified Rehabilitation Counselors (CRC’s) view research as the least important domain related to practice compared to other knowledge and skill domains (Leahy, Szymanski, & Linkowski, 1993; Leahy Muenzen, Saunders, & Strauser, 2009). This may be a result of curricula within master’s programs that do not emphasize the importance of research in relation to practice. 40 Another barrier to EBP is that people with disabilities and rehabilitation counseling practitioners may not consider research conducted to be relevant or applicable to their needs (Johnston et al., 2009). Further, rehabilitation counselors may have limited or no evidence (Gambrill, 2005; Kosciulek, 2010; Chan et al., 2009) or the process of locating and appraising research may be too time consuming (Meier, 2003). Johnston et al. (2009) made the suggestion that more research involving individuals with disabilities and practitioners in the field are needed to increase the relevancy of research in relation to practice. Another way to make the connection between research and practice is to train future rehabilitation counselors to be action researchers to improve their own counseling practice (Kiener & Koch, 2009; Koch et al., 2009). Summary and Implications The importance of clinical judgment in providing counseling services to clients has been stressed for decades (Pepinski & Pepinski, 1954; Garb, 1998; Spengler, et al., 2009; Spengler et al., 1995; Strohmer & Leirer, 2000; Harding, 2007; Garb, 2010). Research has shown that counselors vary in cognitive capacity (Spengler & Strohmer, 1994 ; Holloway & Wolleat, 1980; Falvey et al., 2005), take shortcuts in making decisions (Garb, 1998; Tversky & Kahneman, 1982; Gilovitch et al., 2002), lack awareness of how they make decisions (Garb, 1998), hold numerous cognitive biases and limitations (Garb, 1998), become overconfident (Oskamp, 1965), and may not be improving their judgment accuracy over time (Garb, 1998). Counselors may also be significantly influenced by contextual issues within their practice setting (Garb, 2010). There remains continued debate as to whether training and clinical experience have an effect on the accuracy of counselor clinical judgment post-graduation (Spengler et al. 2009; Garb, 1998). However, the empirical evidence seems well established to support the claim that 41 counselors-in-training become more accurate, although only modestly so, in making clinical judgments in their counseling work with clients during the process of their graduate training (Garb, 1998). Further, Spengler et al. (2009) provide recent evidence in a meta-analysis that shows that clinical experience does, in fact, increase judgment accuracy, although again, only modestly. It was also found that experienced counselors, who have to deal with greater uncertainty, seem better able to make clinical judgments with greater accuracy than novice clinicians (Spengler et al., 2009). In contrast, a substantial collection of alternative evidence suggests that post-graduate training and clinical experience do not affect counselor judgment accuracy (Garb, 1998) which may be attributable to counselor bias (Garb, 1989). In fact, counselors may become more biased and less accurate in their clinical judgments while working with clients over time (Strohmer & Leierer, 2000). Such mixed findings on the effects of graduate education and clinical experience on counselor clinical judgment accuracy (Garb, 1998; Spengler, 2009; Lichtenberg, 1997) leave a lot to be desired from an educator’s perspective. It is concerning to think that graduate education and clinical experience do not have more of an effect on counselor clinical judgment given the vital importance of judgment accuracy in providing effective and holistic services to individuals with disabilities. It is even more concerning to think that counselors may not be learning from their clinical experience or becoming even more biased in their thinking about clients as their clinical work experiences continue (Strohmer & Leierer, 2000). The evidence on counselor bias of which was reviewed suggests that counselors may have a limited cognitive capacity in memory (Garb, 1989), are susceptible to confirmatory (Strohmer et al., 1990; Strohmer & Shivy, 1994) and racial bias (Rosenthal, 2004; Rosenthal & 42 Berven, 1999), and may hold biases related to particular client characterstics such as gender, disability, social class, and age (Lopez, 1989; Broverman et al., 1970; Rybarczyk et al., 2001). There also appears to be a range of complexity in terms of clinical reasoning processes that counselors possess that are hierarchical in nature that have been shown to be linked to clinical judgment accuracy (Falvey et al., 2005; Spengler & Strohmer, 1994; Holloway & Wolleat, 1980). More specifically, counselors with a more complex reasoning style or greater cognitive complexity may be better suited to synthesize and integrate pertinent and complex client information under greater uncertainty, be less prone to bias, and better positioned to make more accurate clinical decisions in their work with clients (Falvey et al., 2005; Spengler & Strohmer, 1994). In addition, it may also be that counselors lack the type of feedback that is conducive to their learning of clinical judgment skills (Garb, 1989). Given the evidence presented here, the need for greater sophistication in terms of structured clinical rehabilitation counselor training, one that emphasizes an organized and individualized approach to teaching clinical judgment skills (Garb, 1989), and that recognizes the need to teach complex levels of clinical problem-solving strategies, such as the use of schemas (Falvey et al., 2005) and hypothesis testing (i.e., scientist-practitioner model) (Strohmer, Shivy, & Chodo, 1990; Strohmer & Shivy, 1994), appears warranted. In a practice environment where managed care costs are rising, and where government funding is limited, the need for service efficiency and for rehabilitation counselors to implement a model of EBP to deliver counseling services that can demonstrate their service effectiveness appears eminent (Chronister et al., 2008). The implementation of EBP is now an ethical duty (CRCC, 2009). Teaching a model of EBP in master’s level rehabilitation counseling programs provides a positive outlook for future change within the discipline. Making EBP central to 43 rehabilitation counseling has potential to enhance the image of the discipline and provides an opportunity to demonstrate the value of the rehabilitative services counselors provide (Chan et al., 2009). Further, this new paradigm shift provides a fresh opportunity to thinking about how to best integrate rehabilitation counseling research and practice and how to best train master’s students to become competent evidence-based practitioners. Despite this multitude of clinical judgment and decision making research accumulated over decades’ time which has shown significant errors in how counselors and other clinicians make judgments and decisions in their work with clients (Garb, 2010; Garb, 1998; Spengler, et al., 2009), it is unknown to what extent clinical judgment skills are explicitly taught in rehabilitation counselor education programs. The dearth of clinical judgment research in rehabilitation counselor education despite the evidence of counselor bias and need for counselor judgment accuracy is of great concern given its central function and importance in rehabilitation counseling practice. Given the importance of clinical judgment in rehabilitation counseling (Strohmer & Leierer, 2000; Berven, 2004, 2008; Lustig, 1996; Rubin & Roessler, 1980; Bellini & Rumrill, 1999; Rosenthal & Kosciulek, 1996), need for clinical judgment accuracy given the prevalence and consequences of rehabilitation counselor biases (Strohmer & Leierer, 2000; Strohmer et al., 1995; Berven & Rosenthal, 1999; Rosenthal, 2004; Spengler et al., 1990), and the emerging trend within the rehabilitation counseling discipline to educate rehabilitation counselors in EBP (Leahy & Arokiasamy, 2010), the explicit teaching of clinical judgment skills that can effectively bridge science and practice to improve rehabilitation counselor judgment accuracy appears critically needed. Further, by establishing a baseline of the most important clinical judgment skills for effective rehabilitation counseling practice, individual skills can then be targeted and used to develop evidenced-based 44 instructional methods to more effectively educate master’s students into becoming competent clinicians who apply a model of EBP to ensure that the most effective services are delivered to clients with disabilities (Kosciulek, 2010). 45 Chapter 3 Method To address the research questions and generate data useful for better understanding rehabilitation counselor educators’ perceptions of teaching clinical judgment skills for effective rehabilitation counseling practice, a quantitative Internet-based ex post facto survey design was used. It was anticipated that findings from this study would inform and enhance both master’s and doctoral-level rehabilitation counseling clinical training curricula. This chapter addresses the research questions, participants, instrumentation, variables, demographic questionnaire, procedure, research design and data analysis, and limitations of the study. Research Questions This study will investigate rehabilitation counselor educators’ (a) perceived importance of teaching clinical judgment skills to master’s students in rehabilitation counseling, (b) perceived level of student preparedness in clinical judgment skills to effectively address clinical judgment tasks upon graduation of master’s rehabilitation counseling programs across the country, and (c) perceived proficiency of their own teaching of clinical judgment skills. The research questions of interest in this study are as follows: 1. How important are clinical judgment skills for effective rehabilitation counseling practice? a. Does the level of importance differ across clinical judgment skill domains? b. Are there differences in the level of importance across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of 46 experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? 2. What are rehabilitation counselor educators’ perceptions of master’s students’ level of preparedness in clinical judgment skill domains upon graduation to effectively address clinical judgment tasks in rehabilitation counseling practice? a. Does the level of perceived student preparedness differ across clinical judgment skill domains? b. Are there differences in the level of student preparedness across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? 3. How do rehabilitation counselor educators perceive their own proficiency in teaching clinical judgment skills? a. Does the level of teaching proficiency differ across clinical judgment skill domains? b. Are there differences in the level of teaching proficiency across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? 4. What are the relationships among rehabilitation counselor educators’ perceived importance, student preparedness, and teaching proficiency of clinical judgment skills? 47 Participants The population of interest in this study was rehabilitation counselor educators who were employed in master’s level rehabilitation counseling programs. The sampling frame for this study was drawn using a non-probability, convenience sample (Trochim, 2006) from a list generated from the NCRE 2010-2011 membership directory of rehabilitation counseling programs located in the U.S. (including the District of Columbia), Puerto Rico, Australia, and Canada. The directory list was updated from cross-checking the updated faculty lists on the NCRE 2010-2011 membership directory with master’s rehabilitation counseling program websites to reflect a current list of an estimated population of rehabilitation counselor educators. Currently, there are 97 accredited master’s programs in rehabilitation counseling and several institutional candidates for accreditation and/or under review (CORE, 2011). Only current master’s level rehabilitation counseling faculty (e.g., assistant professor, associate professor, full professor, adjunct professor, clinical instructor, instructor, etc.) were eligible to participate in this study. Instrumentation In review of the clinical judgment literature, no instruments were located that directly measured rehabilitation counselor educators’ perceptions of clinical judgment skills or adequately measured the construct of interest (based on the conceptual definition) for this study. Accordingly, a new measure of clinical judgment skills (i.e., the CJSI) was developed. The development of clinical judgment skill items were established based on two theoretical frameworks used to guide the current study. Items were developed based on a scientist-practitioner model (Spengler et al., 1995) informed by human inference theory and research (Nesbitt & Ross, 1980) and Pepinsky and Pepinsky’s (1954) original counselor-as- 48 scientist model of clinical practice. This model was selected because of its intention to encourage more cognitively complex counselor thinking which has been linked to greater judgment accuracy among practitioners within the counseling research (Spengler & Strohmer, 1994; Falvey et al., 2005; Holloway & Wolleat, 1980). The model incorporates three constructs of interest or sub-domains used for this study considered necessary to improve counselor judgment accuracy; counselor self-awareness, counselor openness/curiosity, and debiasing techniques. The conceptual definitions of these three constructs from the model (Spengler et al., 1995) were used to draw initial items for each sub-domain. Judgment heuristics, cognitive biases, and knowledge structures are additional constructs that have been used repeatedly in the research to study and explain the cognitive processes counselors go through to make clinical judgments (Garb, 1998). It has been noted in the rehabilitation counseling literature that master’s students should at minimum be made aware of the counselor bias and judgment research (Strohmer & Leierer, 2000). Strohmer and Leierer (2000) propose that knowledge of such mental processes (i.e., judgment heuristics, knowledge structures, and cognitive biases) and the current counselor bias and judgment research can inform rehabilitation counselors of how they make clinical judgments and decisions. Further, it is theorized that if counselors have knowledge of their own thought processes and cognitive limitations in relation to making clinical judgments they will likely be better positioned to account for bias and minimize judgment error in counseling practice (Garb, 2005; Meier, 2003). For such reasons, the majority of items were informed by a review and synthesis of findings of the clinical judgment and bias research to date across professional disciplines (e.g., counseling psychology, rehabilitation counseling, clinical psychology, etc.) (Strohmer & Leierer, 49 2000; Athanasou & Kaufman, 2010; Strohmer, et al., 1995; Rosenthal, 2004; Rosenthal & Berven, 1999; Spengler et al., 1990; Garb, 1989, 1996, 1998, 2010; Spengler et al., 2009; Spengler & Strohmer, 1994; Falvey et al., 2005; Holloway & Wolleat, 1980; Haverkamp, 1993; Oskamp, 1965; Lopez, 1989; Faust, 1986; Turk & Salovey, 1988; Strohmer & Shivy, 1994; Strohmer et al., 1990; Tversky & Kahneman, 1982; Nisbett & Ross, 1980; Slovic et al., 2002; Arkes, 1981). Sources considered to be seminal pieces or often cited in the clinical judgment literature were selected. Items were drawn from the most common or notable bias research findings and implications and then reframed to fit each of the initial three sub-domains and structure of the CJSI. Additional items were sought and developed from sources within the clinical training literature also across professional disciplines (e.g., counseling, rehabilitation counseling, counseling psychology, etc.) (Buser, 2008; Lundervold & Belwood, 2000; Morran et al., 1995; Schon, 1983; Heppner et al., 1999; Granello & Granello, 1998; Stricker & Trierweiler, 1995; Hoshmand, 1991; Facione & Facione, 2008; Berven, 2004, 2008; Leahy & Arokiasamy, 2010; Owen & Lindley, 2010; Borders et al. 1994; Sexton, 2000; Whiston & Coker, 2000; Haverkamp, 1994; Dellario, 1996; Chwalisz, 2003; Kiener & Koch, 2009; Koch et al., 2009; Chan, Shaw, McMahon, Koch, Strauser, 1997; Power, 2006; Leung, 1987; Bellini & Rumrill, 1999; Meier, 2003; Gambrill, 2005; Ivey & Ivey, 2008; Duys & Headstrom, 2000; Morrow & Deidan, 1992; Koch, et al., 2000; Sexton, Whiston, Bleuer, & Walz, 1997; Fong et al., 1997; Urbani et al, 2002; Little et al., 2005; Richardson & Molinaro, 1996; Chan et al., 2010; Goodheart, 2006; Kazdin, 2006; Kosciulek, 2010; Trierweiler, 2006; Garb & Grove, 2005; Harding, 2007). In addition, the CORE (2011) accreditation standards and knowledge and function studies (Leahy et al., 2003; Leahy, et al., 2009) for rehabilitation counselors were analyzed to ensure 50 skills already established as curriculum standards or deemed important or performed regularly by CRC’s were included or modified as skill items within each respective sub-domain if they appeared to fit. Initially, a list of 181 items was generated that was subsequently reduced to 35 items after repetitive reviews and confusing, irrelevant or redundant items were eliminated. Given the CJSI’s layout and order of items, sub-domains (counselor self-awareness and counselor openness/curiosity) were also deleted from the instrument to lower the likelihood of a possible priming effect (Dillman, Smyth, & Christian, 2009) of participant responses. Items from these two sub-domains were then analyzed further. Selected items were reframed and positioned within the debiasing techniques domain if they seemed to reflect its conceptual definition; the use of internal dialogues that serve as reminders to counselors to question their biases and use of other strategies or tools including scientific-based methods to improve clinical judgment accuracy (Spengler et al., 1995). In constructing the items of the CJSI, careful attention was paid toward statements that were similar in structure. By doing this, it was intended that items would be clear and easy to understand for participants. After this initial draft was developed and reviewed by another rehabilitation counseling faculty member, an expert panel of six faculty members considered knowledge experts of clinical judgment tasks for effective rehabilitation counseling practice were contacted and asked to serve as expert reviewers of the instrument. Each potential reviewer was sent (a) an invitation letter with specific questions to respond to, (b) research questions, (c) draft instrument, and (e) narrative of how items were developed. Expert reviewers were instructed to provide feedback in regard to the constructs, content, item consistency, and clarity (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999). Such 51 feedback was provided by five expert reviewers (four faculty members from rehabilitation counseling and one from educational psychology) and used to make needed revisions to upgrade item content of the survey. Variables There was one construct (i.e., domain) of interest to be measured in this study. Clinical Judgment Skills. Clinical judgment skills were defined as debiasing techniques; the use of internal dialogues that serve as reminders to counselors to question their biases and use of other strategies or tools including scientific-based methods to improve clinical judgment accuracy (Spengler et al., 1995). In this study, clinical judgment skills was operationally defined by a mean scale score of one domain, clinical judgment skills (debiasing techniques), as measured by the CJSI developed specifically for this study. There were 35 items on the CJSI with two 1 to 7 seven point Likert scales for each item that measured rehabilitation counselor educators’ perceived importance (1=not at all important, 2=slightly important, 3=somewhat important, 4=important, 5=quite important, 6=very important, 7=extremely important) and perceived level of student preparedness (1=not at all prepared, 2=slightly prepared, 3=somewhat prepared, 4=prepared, 5=quite prepared, 6=very prepared, 7=extremely prepared) of clinical judgment skills upon graduation. Eight remaining items measured rehabilitation counselor educators’ own proficiency of teaching (1=not at all proficient, 2=slightly proficient, 3=somewhat proficient, 4=proficient, 5=quite proficient, 6=very proficient, 7=extremely proficient) selected skills within the overall clinical judgment skill domain. The use of these three scales for items is based on a similar study that measured rehabilitation counselor educators’ perceptions of disability management training 52 (Coduti, 2009) in a similar manner. Further, Coduti’s (2009) study was used as a guide for the scaling of items and structure of the CJSI. A forced-choice scale of measurement without a neutral mid-point item was selected as research suggested a forced choice format without a mid-point or neutral item (i.e., neutral or don’t know) could decrease the likelihood of participants responding in a socially desirable manner or minimize participant responses that were positively or negatively skewed (Garland, 1991). One example item that measured the overall clinical judgment skill domain of debiasing techniques stated, “Implements conscious and continuous self-evaluation of own thinking about clients”. Each participant used the two scales (i.e., importance for practice and student preparedness) to rate this example item and 34 subsequent items. For the eight items measured with only one scale (i.e., teaching proficiency), an example item, “Formulates questions or strategies to test and evaluate multiple hypotheses”, measured the overall clinical judgment skill domain along with seven subsequent items. Demographic Questionnaire A demographic questionnaire was used to obtain participant demographic and programmatic characteristics. Ten demographic questions were asked to inquire about general participant demographic information, professional characteristics, and program information (See Appendix B). Such demographic information was sought to help better describe participants as it related to developing population parameters of the CJSI (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999), be used to answer specific research questions of the current study, and provide context for data analysis and interpretation of results. 53 Procedure Approval for the use of human subjects by the Michigan State University (MSU) Institutional Review Board (IRB) was sought. Following IRB approval, a submittal letter and the current study’s research proposal was sent to the NCRE research committee who provided approval and granted access to the NCRE memberships used to conduct the study. A convenience, non-probability sample (Trochim, 2006) was drawn from an updated list of current email addresses of rehabilitation counselor educators taken from NCRE’s 2010-2011 membership directory of rehabilitation counseling programs in the U.S. (including the District of Columbia), Puerto Rico, Australia, and Canada. This directory list was further updated by crosschecking faculty lists on rehabilitation counseling program websites to reflect the most up to date and accurate list of rehabilitation counseling faculty across the country. This process resulted in a total population of rehabilitation counselor educators and considered an accurate estimate and representation of the population of rehabilitation counselor educators to date. In addition, the current NCRE email listserve that includes all NCRE members was also requested and granted by the NCRE research committee for use to recruit participants in attempt to include additional rehabilitation counselor educators not included or missing on the email address list drawn from NCRE’s 2010-2011 membership directory or program websites. Prior to launch of the study, the CJSI was developed on Survey Monkey, a web-survey service program. Further, an Internet-based pilot study of rehabilitation counseling faculty from the population was conducted. Eight rehabilitation counseling faculty members participated in the pilot study. Reactions and comments from pilot study participants were sought to inform further revision of the survey as needed. In particular, pilot study participants were asked to make comments about time needed to complete survey, length and clarity of items, instructions, 54 adequacy of scales, and ease of taking the survey electronically. Such a pilot study was intended to further improve the CJSI’s content, items, survey structure, and scales (Creswell, 2009). A data check was also performed to ensure successful importing of data collected for this study. When launching the pilot and actual study, an email announcement was used to invite rehabilitation counselor educators to participate in this study. A meta-analysis suggested that topic saliency, making pre-contacts and follow up contacts with participants that are personalized can increase response rates of Internet-based survey research (Cook, Heath, & Thompson, 2000). As such, an email announcement provided an introduction of the researcher, description and importance of the current study, link to the survey, and approximate time needed to take the survey (e.g., 15-20 minutes). Further, the study announcement provided NCRE’s approval of the study to further the mission and goals of the professional organization. The human subjects’ informed consent form was created and provided to all participants (See Appendix C). For the actual study, the survey’s web link was sent electronically to all potential participants’ email addresses derived from NCRE’s 2010-2011 national membership directory list and NCRE listserve. Email announcements of the survey were sent in blocks using the hidden email feature to give participants a sense of receiving a personalized invitation to participate in the current study. After the initial launch of this study, a total of four follow-up reminder emails (two from the membership directory list and two from the NCRE listserve) were sent over about a month’s period of data collection. Such personalized email delivery and follow up contacts were intended to increase the response rate of surveys completed (Cook et al., 2000). Research Design and Data Analysis The quantitative research design of this study was an Internet-based ex post facto survey design where an exploratory factor analysis (EFA), univariate and correlational statistical 55 analyses were used to analyze and interpret the data. After first performing a data cleansing procedure (i.e., importing data to SPSS, recoding variables, and checking for accuracy), an EFA was conducted. An EFA was performed to determine if items on the CJSI measured theoretical constructs of interest and if the items could be better explained by fewer components or factors that held common item interrelationships and significant loadings (Raykov & Marcoulides, 2008). Subject to item ratios considered acceptable varies (Costello & Osborn, 2005), however, in a review of 303 factor analysis studies, researchers found that the majority used 10:1 or less subject to item ratios (Costello & Osborne, 2005). Given the current study’s sample size was too low to meet specifications for satisfactory subject to item ratio (i.e., 10:1) for factor analysis of the total CJSI measure (Costello & Osborne, 2005), the Kaiser-Meyer-Olkin (KMO) and Bartlett’s test of sphericity were computed to test adequacy of sample size and inter-item correlations for factor analysis (Raykov & Marcoulides, 2008). Subsequent statistical analyses proceeded based on two new factor structures (i.e., importance for practice and teaching proficiency scales) that were retained. To establish reliability of the CJSI, the Cronbach alpha statistic was used to estimate the internal consistency of items (Thorndike, 2005) relative to the clinical judgment skill domains that were measured. To more clearly define the sample of the population of rehabilitation counselor educators, descriptive statistics, frequencies, and percentages were computed for general participant demographic information (i.e., gender and race/ethnicity), professional characteristics (i.e., faculty rank, clinical experience/setting, degree major, and years of experience as rehabilitation counselor educator, professional credentials), and program information (i.e., emphasis of clinical judgment training). Such data was to be used to make comparisons among the sample obtained and population from which the sample was drawn to provide an estimate of the current study’s 56 representativeness of the population (i.e., rehabilitation counselor educators across the country). However, comparisons to the population could not be made as such population demographic information could not be found. To answer research questions 1, 2, and 3, descriptive statistics, percentages, and frequency tables were used to summarize demographic characteristics of the sample and to derive mean and standard deviation scores of rehabilitation counselor educators’ perceived importance, student preparedness, and proficiency of their own teaching in individual clinical judgment skills. A 7-point Likert scale was used to measure clinical judgment skills (debiasing techniques). To answer research questions 1a, 2a, and 3a, descriptive statistics, percentages, and frequency tables were used to summarize demographic characteristics of the sample and to derive mean and standard deviation scores of rehabilitation counselor educators’ perceived importance, student preparedness, and proficiency of their own teaching in clinical judgment skill domains. In addition, a clustering technique (Berven, 1979; Chan et al., 2003) was administered to determine differences of importance and preparedness (1a and 2a) across clinical judgment skill domains, as well as, to specify high priority clinical judgment training needs by prioritization and pairing of individual skills identified as “high importance”/“limited student preparation”. To answer research questions 1b, 2b, and 3b, a series of factorial independent groups univariate analyses of variance (ANOVA) tests (Taylor, 2011) and post-hoc pairwise comparisons of marginal means were used to compare the group mean differences of importance, student preparedness, and teaching proficiency of the clinical judgment skill domains (debiasing techniques) relative to the demographic (i.e., gender, race/ethnicity, faculty rank, clinical 57 experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics of interest. Based on an a priori power analysis pre-determined to detect a medium (.15) to large (.35) effect size at power =.80, with an alpha level of .05 (Cohen, 1992), a minimum sample size of 107 was needed to test if significant differences existed in level (i.e., importance, student preparedness, and teaching proficiency) of clinical judgment skill domains between rehabilitation counselor educators’ relative to their demographic and programmatic characteristics (i.e., eight independent variables). To answer research question 4, a correlation analysis (Field, 2009) was conducted to measure the relationships among rehabilitation counselor educators’ perceived importance, student preparedness, and teaching proficiency of clinical judgment skills (debiasing techniques). A t-test of significance for the product-moment correlation coefficient r was used. A minimum sample size of 85 to 125 was needed to test the relationships among variables based on an a priori power analysis pre-determined to detect a medium (.30) to large (.50) effect size at power =.80, with an alpha level of .05 or .01, respectively (Cohen, 1992). Correlation matrices were also used to analyze relationships among clinical judgment skill domains (debiasing techniques). In addition, effect sizes specified were based on Cohen’s effect size indexes (Cohen, 1992), commonly used in the counseling research (Trusty, Thompson, & Petrocelli, 2004). Further, it has been suggested that researchers report effect sizes to help counselors interpret significance of findings for counseling practice (Trusty et al., 2004). Limitations of the Study Anticipated limitations of this study included the use of self-report questionnaires and potential for lower than desirable response rate. Self-report questionnaires are susceptible to participant response bias (i.e., participants may attempt to guess and confirm researcher 58 hypotheses or answer survey questions in socially desirable ways) (Heppner et al., 1999). The use of self-report questionnaires and use of internet-based surveys run the risk of low response rates (Wright, 2005). For example, a meta-analysis of Internet-based survey research showed an average response rate of about 35%-40% (Cook et al., 2000). In rehabilitation counselor education, similar research that surveyed rehabilitation counselor educators had response rates of 12% (Coduti, 2009) and 29.2% (Zanskas, 2007), respectively. Such low response rates may limit the representativeness of the sample to the population, thereby limiting the generalizability of the results to the population of rehabilitation counselor educators across the country. Another anticipated limitation was that the primary assumption of this study relied on the validity of rehabilitation counselor educators’ self-report and knowledge of clinical judgment tasks that relate to their own perceptions of importance, their own teaching proficiency, and level of master’s student preparedness of clinical judgment skill domains. It was unknown the degree to which rehabilitation counselor educators’ were going to be accurate in their perceptions. An additional expected limitation was that the CJSI had been originally constructed specifically for this study and no reliability or validity studies had been conducted to validate the psychometric properties or construct validity of the CJSI. Small sample size was also likely to be a limiting factor for having a desirable subject to item ratio for computing a simultaneous factor analysis of the full CJSI measure. However, it should be noted that even with large sample sizes, exploratory factor solutions are prone to error above the standard alpha level of p<05 (Costello & Osborne, 2005). Consequently, additional follow-up studies were anticipated to be needed to further test the validity of the CJSI. In addition, the inability to compare the CJSI measure with other related or dissimilar instruments made it impossible to test the CJSI’s concurrent or discriminant validity, respectively. However, knowing that the quality of the instrument was 59 most important in a survey design (Heppner et al., 1999) as employed in the current study, several steps were taken to address such limitations and enhance the CJSI’s reliability and construct validity for this study. 60 Chapter 4 Results The purpose of this exploratory study was to investigate rehabilitation counselor educators’ (a) perceived importance of teaching clinical judgment skills to master’s students in rehabilitation counseling, (b) perceived level of student preparedness in clinical judgment skills to effectively address clinical judgment tasks upon graduation from master’s rehabilitation counseling programs, and (c) perceived proficiency of their own teaching of clinical judgment skills. There was one variable of interest measured in this study; clinical judgment skills (debiasing techniques). Clinical judgment skills were measured by three scales (i.e., importance for practice, student preparedness, and teaching proficiency) and were defined as debiasing techniques which are the use of internal dialogues that serve as reminders to counselors to question their biases and use of other strategies or tools including scientific-based methods to improve clinical judgment accuracy (Spengler et al., 1995). This chapter will address the characteristics of participants, how missing data was handled, results of an EFA, and analyses to address each research question. The IBM Statistical Package for the Social Sciences 19 (SPSS, 2010) and Microsoft Excel 2007 were used to conduct all data analyses for this study. Participant Characteristics As stated in chapter 3, the population of interest in this study is rehabilitation counselor educators who were employed in master’s level rehabilitation counseling programs. The sampling frame for this study was drawn using a non-probability, convenience sample (Trochim, 2006) from a list generated from the NCRE 2010-2011 membership directory of rehabilitation counseling programs located in the U.S. (including the District of Columbia), Puerto Rico, 61 Australia, and Canada. The directory list was updated from cross-checking the updated faculty lists on the NCRE 2010-2011 membership directory with master’s rehabilitation counseling program websites to reflect a current list of an estimated population of rehabilitation counselor educators. Only current master’s level rehabilitation counseling faculty (e.g., assistant professor, associate professor, full professor, adjunct professor, clinical instructor, instructor, etc.) were eligible to participate in this study. Response Rate From the initial 647 participants who were sent an electronic survey, 81 failed to deliver, which then constituted an initial population estimate of 566. There were a total of 126 survey responses which generated an initial sample response rate of 22.3%. In addition, there were five respondents that did not qualify, one participant who was not responding to emails due to taking a leave of absence, and five participants who had retired. These additional 11 surveys were subtracted from the initial population estimate which left 555 as the final number for the estimated population of rehabilitation counselor educators. The adjusted sample response rate used for this study was 22.7% (126/555). Demographic and Programmatic Characteristics Of the 111 rehabilitation counselor educators who provided demographic characteristics (i.e., gender, race/ethnicity), 54.1% (n=60) were female and 45.9% (n=51) male. For race/ethnicity, 13.5% (n=15) were African American/Black, 2.7% (n=3) Asian/Pacific Islander, 3.6% (n=4) Latino, 2.7% (n=3) Multiracial, 0.9% (n=1) Native American, and 76.6% (n=85) Caucasian/White. As indicated in Table 1, the professional CRC credential was held by the majority of rehabilitation counselor educators (90.1%; n=100) compared to the proportion of other certifications and licenses. Years of clinical experience (n=110) had a varied distribution 62 across categories of 2 years or less (5.5%; n=6), 2-5 (23.6%; n=26), 6-10 (20.0%; n=22), 11-15 (17.3%; n=19), 16-20 (11.8%; n=13), and 20 years or more (21.8%; n=24). Most educators have 2-5 years of clinical experience providing direct client services (23.6%; n=26). Further, study participants (n=109) reported to receive the majority of their clinical experience working for a state-federal vocational rehabilitation agency (24.8%; n=27), followed by private-not-for-profit (21.1%; n=23), college or university (14.7%; n=16), mental health center (12.8%; n=14), privatefor-profit (9.2%; n=10), private practice (7.3%, n=8), hospital (5.5%, n=6), and other (4.6%; n=4). Table 1 Professional Characteristics Variables Professional n % Credentials CRC 100 90.1 NCC 17 15.3 CVE 5 4.5 LPC 31 27.9 Other 21 18.9 Total 111 Note: Professional credentials can have multiple answers and missing data which are reasons why total n varies. CRC=Certified Rehabilitation Counselor; NCC=National Certified Counselor; CVE=Certified Vocational Evaluator; LPC=License Professional Counselor. As shown in Table 2, the majority of participants earned their highest degree with a major in rehabilitation counseling (37.3%; n=41). Rehabilitation counselor education (21.8%; n=24) was the second most frequent degree earned, followed by rehabilitation psychology (14.5%; n=16), and counselor education (11.8%; n=13). There was relatively even dispersion of faculty rank between assistant, associate, and full professors among participants, and large range of years of experience (i.e., from 1 to 40) as a rehabilitation counselor educator. 63 When asked how much clinical judgment training was emphasized within the overall master’s level curriculum in rehabilitation counseling, Table 3 illustrates that most participants reported clinical judgment being emphasized (38%; n=41), followed by being very emphasized (32.4%; n=35). In addition, clinical judgment skill training is mostly provided in practicum and internship courses (80.0%; n=88) and about half provide clinical judgment training across all courses (50.9%; n=56). Whereas, clinical judgment training is provided less often within the multicultural counseling course (40.9%; n=45) when compared to practicum or internship, counseling skills (i.e., individual and/or group counseling), or assessment courses. Table 2 Academic Characteristics Variables Major Degree Earned N % Clinical Psychology 1 0.9 Counseling Psychology 3 2.7 Rehabilitation Science 1 0.9 Rehab. Psych. 16 14.5 Rehab. Counseling 41 37.3 Rehab. Counselor Ed. 24 21.8 Counselor Education 13 11.8 Other 11 10.0 Total 110 100 Faculty Rank N % Asst. Prof. 31 28.2 Assoc. Prof. 28 25.5 Prof. 30 27.3 Adjunct Prof. 13 11.8 Clinical Instr. 1 0.9 Other 7 6.4 Total 110 100 Years as RCE N % 1-10 yrs. 49 44.5 11-20 yrs. 37 33.6 21-40 yrs. 24 21.8 Total 110 100 Note: Major degree earned, faculty rank, and years as RCE have missing data. 64 Table 3 Rehabilitation Counseling Program Characteristics Variables CJ Emphasis N % Not emphasized 3 2.8 Somewhat emphasized 16 14.8 Emphasized 41 38.0 Very emphasized 35 32.4 Extremely emphasized 13 12.0 Total 108 100 Note: Different n’s are a result of missing data. Handling of Missing Data To address missing data for a relatively small sample size and to maximize statistical power for factor analysis and subsequent statistical tests (i.e., factorial independent groups ANOVA, post-hoc pairwise comparisons, Pearson-moment correlation) for this study, single imputation was used to estimate 78 continuous variables intended to measure clinical judgment skills in three ways (i.e., with scales of importance for practice, student preparedness, and teaching proficiency). In particular, expectation maximization (EM) was used to impute all missing data for such variables (SPSS, 2010). EM is considered a valid way to estimate missing values (McKnight, McKnight, Sidani, & Figueredo, 2008) particularly when the mechanism for missing data is Missing at Random (MAR) and missing values constitute a small percentage (Scheffer, 2002). For this study, missing data was approximately 8% and the data was considered to have a pattern of missing data that is MAR given that Little’s Missing Completely at Random (MCAR) test was not significant, Chi-Square=2243.28, DF=2142, Sig.=.063 (Kline, 2011). Further, the amount of missing data imputations were not rounded and kept “as is” to maintain the integrity of getting the most precise parameter estimates. Categorical variables (i.e., 65 participant demographic and programmatic characteristics) were not imputed and had missing values. Exploratory Factor Analysis An EFA was performed to determine if items on the CJSI measured theoretical constructs of interest and if the items could be better explained by fewer components or factors that held common item interrelationships and significant loadings for such respective scales; importance and teaching proficiency. Factor analysis on the preparedness scale was not performed since analyses of the student preparedness scale were to be based on its relationships between the new factor structure of importance and teaching proficiency. It was anticipated that subsequent statistical analyses would take place based on a newly retained factor structure if one could be meaningfully generated and interpreted. First, the importance scale was analyzed. Importance For Practice and Student Preparedness Scales The principle component method was used for factor analysis to extract components that could explain the item interrelationships among the 35 importance items constructed for the CJSI (Raykov & Marcoulides, 2008). The Kaiser-Meyer-Olkin (KMO) measure substantiated the adequacy of sample size for the factor analysis, KMO=.88 (very good according to Field, 2009), and all KMO values for individual items were > .78, which is above the adequacy minimum of .50 (Field, 2009), which indicated that factor analysis could be carried out and achieve a reliable solution (Raykov & Marcoulides, 2008). Bartlett’s test of sphericity χ2 (595) =2776.43, p < .001, indicated that inter-item correlations were substantial and adequate (Field, 2009). Using Kaiser’s eigenvalue criterion >1 (Raykov & Marcoulides, 2008), seven factors were initially extracted for analysis. An analysis of the scree plot was also used to determine 66 number of factors to retain (Raykov & Marcoulides, 2008). However, in analyzing the scree plot, it was challenging to interpret given the subtleties of the inflexions; it appeared that either three or seven factors could be retained. Orthogonal rotation (varimax) method was then used to assist with making meaningful interpretations of both the initial seven factor solution and factor extraction set at three. Based on ease and practical nature of interpretability, the initial seven factor solution that met the Kaiser eigenvalue criterion greater than one (Raykov & Marcoulides, 2008) with an explained variance of 66.23% was retained as the final component solution (see figure 3). Further, coefficient values for all items were set at .40 or higher, since this value has been deemed as having a loading of interpretive value (Field, 2009). For example, Stevens (2002; as cited in Field, 2009) suggested that by squaring r (e.g., .40), approximately 16% of the variance of a given factor can be explained by such an item factor loading. Table 4 shows a summary of factor loadings for the importance scale after varimax rotation. Individual factor mean scores were then generated for educators’ perceived importance and student preparedness scales based on this new seven factor structure. The individual and overall means of the seven factors and corresponding skill items are presented in Table 5. Factor 1: Importance for practice and student preparedness in using a scientific attitude. This debiasing technique consists of six skill items that characterize clinical judgment skills related to a counselor’s appreciation of the use of science throughout the counseling process and prevailing view that research and practice are intertwined and inform each other (Spengler et al., 1995). The mean inter-item correlation of .43 and computed cronbach alpha α=.82 suggested very good internal consistency reliability (DeVellis, 2003) among the items that comprised this factor. The mean importance and student preparedness of this factor were 4.76 (SD=0.98) and 3.58 (SD=1.02), respectively. 67 Important Clinical Judgment Skills (Debiasing Techniques) for Effective Rehabilitation Counseling Practice Core Phenomenon Factor 1: scientific attitude Factor 2: cultural bias Clinical Judgment Skills (Debiasing Techniques) Factor 7: evidencebased practice Factor 6: negative bias Factor 3: cognitive complexity Factor 4: memory bias Factor 5: confirmatory bias Figure 3. Seven factors retained from an exploratory factor analysis which defines new factor structure for subsequent data analyses of educators’ perceived importance and student preparedness for effective rehabilitation counseling practice. 68 Table 4 Factor Loadings of Exploratory Factor Analysis With Varimax Rotation of Importance Scale Rotated Factor Loadings 1 2 3 4 5 6 7 Importance Items Builds in a baseline period of clie -.06 .13 .15 .14 .15 .08 .81 Considers probability and base rat .15 .06 -.07 .11 .30 .14 .71 Discards or tentatively accepts hy -.02 .15 .21 .22 .17 -.17 .70 Weighs empirical results more he .07 .05 .20 -.08 -.10 .26 .67 Combines statistical and clinical .17 .22 .38 .11 -.02 .30 .49 Uses reliable and valid instrument .24 .39 .35 .01 .18 .08 .42 Evaluates own cultural biases bas -.00 .12 .20 .29 .26 .34 .71 Adapts use of counseling style to .03 .33 .24 .06 .24 .24 .70 Openly discusses diversity and m .16 .47 .14 -.04 .14 .16 .67 Seeks out supervisory feedback .32 .17 .12 .35 .04 -.23 .62 Implements conscious self-evalua -.05 .17 .14 .45 .20 .06 .57 Does not become attached to one .12 .13 .16 .29 .23 .07 .72 Assess trait and situational factors .06 .34 .07 .26 .16 .02 .61 Uses multiple sources of assessme .20 .19 .35 .06 .42 .06 .56 Attends to details of sessions that .48 .18 .16 .22 -.09 .05 .55 Focuses interview on multiple di .04 .51 .24 .05 .09 .25 .55 Actively considers and/or discuss .17 .12 .31 .40 .30 .09 .50 Delays final judgments about clie .25 .31 .19 .37 .14 -.09 .48 Uses systematic and comprehensi .25 .20 -.00 .05 .10 -.01 .73 Carefully documents all salient ob .03 .07 .35 .14 .05 .17 .71 Utilizes knowledge base of client .03 .07 .25 .15 .25 .28 .67 Periodically reviews case notes an .29 .29 .15 .24 -.02 .17 .64 Examines biases associated with .43 .42 .10 .30 .05 .15 .49 Makes rationale for decision-maki .31 .36 .20 .14 .04 .32 .42 Re-checks initial case formulation .29 .11 .33 .19 .09 .18 .68 Formulates questions or strategies .33 .08 .21 .08 .00 .27 .66 Takes into account own emotional .01 .44 .07 .27 .42 .10 .56 Actively questions the quality of -.08 .24 .25 .09 .31 .03 .54 Becomes informed about problem .12 .27 .17 .06 .04 .00 .77 Focuses on client strengths .10 .09 .16 .15 .32 .14 .67 Attends to positive client emotion .19 .05 .46 -.02 .11 .23 .57 Uses decision aids when making .31 .42 .01 .44 .04 .03 .49 Critically appraises and applies re .09 .20 -.01 .18 .41 .04 .71 Uses clinical practice guidelines .41 .21 .07 .23 -.10 .22 .57 Separates from the counseling pro .15 .05 .25 .27 .19 .18 .46 Eigenvalues 13.61 2.61 1.81 1.45 1.35 1.26 1.09 % of variance 38.90 7.46 5.16 4.14 3.86 3.60 3.12 Note: Factor loadings for each factor appear in boldface. N=126. 1=Scientific Attitude; 2=Cultural Bias; 3=Cognitive Complexity; 4=Memory Bias; 5=Confirmatory Bias; 6=Negative Bias; 7=Evidence-Based Practice. 69 Factor 2: Importance for practice and student preparedness in addressing cultural bias. This debiasing technique consists of five skill items that are representative of clinical judgment skills used to overcome judgment errors related to culture (e.g., race/ethnicity, socioeconomic status, disability, gender, age, etc.). The mean inter-item correlation of .55 and computed cronbach alpha α=.86 suggested very good internal consistency reliability (DeVellis, 2003) among the items that comprised this factor. The mean importance and student preparedness of this factor were 5.94 (SD=0.91) and 4.54 (SD=1.20), respectively. Factor 3: Importance for practice and student preparedness in cognitive complexity. This debiasing technique consists of seven skill items that characterize clinical judgment skills intended to improve judgment accuracy that relate to thinking about and approaching work with clients in a multidimensional way and making multiple client observations that lead to inferences and working hypotheses that are considered tentative in nature (Bieri et al., 1966; Pepinsky & Pepinsky, 1954). Within this construct, counselor judgments about clients are delayed and selfreflective questioning is also performed to reduce overconfidence in decisions while engaging in the counseling process (Spengler et. al., 1995; Bieri et al., 1966). The mean inter-item correlation of .50 and computed cronbach alpha α=.87 indicated very good internal consistency reliability (DeVellis, 2003) among the items that comprised this factor. The mean importance and student preparedness of this factor were 5.77 (SD=0.80) and 4.39 (SD=1.07), respectively. Factor 4: Importance for practice and student preparedness in addressing memory bias. This debiasing technique consists of six skill items that characterize clinical judgment skills intended to address and overcome the limitations in how counselors process and remember client information (Garb, 1998: Garb, 2010). The mean inter-item correlation of .51 and computed cronbach alpha α=.86 suggested very good internal consistency reliability (DeVellis, 2003) for 70 the items that comprised this factor. The mean importance and student preparedness of this factor were 5.49 (SD=0.95) and 4.31 (SD=1.10), respectively. Factor 5: Importance for practice and student preparedness in addressing confirmatory bias. This debiasing technique consists of four skill items that characterize clinical judgment skills intended to address and overcome counselors’ tendency to hold onto their initial client impressions and disregard other pertinent information (Strohmer et al., 1990; Haverkamp, 1993). The mean inter-item correlation of .48 and computed cronbach alpha α=.78 indicated respectable internal consistency reliability (DeVellis, 2003) among the items that comprised this factor. The mean importance and student preparedness of this factor were 5.63 (SD=0.86) and 4.02 (SD=1.11), respectively. Factor 6: Importance for practice and student preparedness in addressing negative bias. This debiasing technique consists of four skill items that characterize clinical judgment skills intended to counteract counselors’ tendency to focus on more negative client information (Strohmer et al.,1995), as well as, judgment errors related to the possible underestimation of client potential or whether or not a client qualifies for or receives appropriate counseling services (Rosenthal & Berven, 1999; Rosenthal, 2004). The mean inter-item correlations of .45 and computed cronbach alpha α=.76 suggested respectable internal consistency reliability (DeVellis, 2003) among the items that comprised this factor. The mean importance and student preparedness of this factor were 5.78 (SD=0.86) and 4.61 (SD=1.11), respectively. Factor 7: Importance for practice and student preparedness in evidence-based practice. This debiasing technique consists of three skill items that characterize clinical judgment skills intended to bridge the gap between research and practice in a relevant and contextually specific manner to improve accuracy of judgments made within the counseling 71 process (Stricker & Trierweiler, 1995). The mean inter-item correlation of .40 and computed cronbach alpha α=.66 indicated minimally acceptable internal consistency reliability (DeVellis, 2003) among the items that comprise this factor. The mean importance and student preparedness of this factor were 5.27 (SD=0.99) and 3.72 (SD=1.12), respectively. Table 5 Important Clinical Judgment Skills for Effective Rehabilitation Counseling Practice Clinical Judgment Skills (debiasing techniques) Importance M SD 5.52 0.72 Preparedness M SD 4.17 0.99 Factor 1: Scientific Attitude Builds in a baseline period of client data collection Considers probability and base rate data Discards or tentatively accepts hypotheses based on counseling and/or rehabilitation outcome data (i.e., intervention effects, client progress) Weighs empirical results more heavily than own clinical experiences Combines statistical (e.g., vocational, psychological tests) and clinical prediction techniques (e.g., behavioral assessment methods) Uses reliable and valid instruments in data collection 4.76 4.48 4.01 0.98 1.40 1.44 3.58 3.38 2.72 1.02 1.41 1.26 4.73 1.36 3.65 1.37 4.14 1.43 3.36 1.15 5.25 5.95 1.30 1.09 3.81 4.58 1.34 1.36 Factor 2: Cultural Bias Evaluates own cultural biases based on such categories as gender, ethnic minority status, disability, and social class Adapts use of counseling style (i.e., qualities and skills) to suit different individuals and cultures Openly discusses diversity and multicultural differences Seeks out supervisory feedback Implements conscious and continuous self-evaluation of own thinking about clients 5.94 0.91 4.54 1.20 6.07 1.09 4.45 1.48 5.82 5.82 5.93 1.13 1.29 1.08 4.26 4.48 5.09 1.47 1.50 1.30 6.04 1.08 4.39 1.37 Factor 3: Cognitive Complexity 5.77 Does not become attached to one interpretation of the client data 5.86 0.80 4.39 1.07 1.03 4.03 1.40 72 Table 5 (cont’d) Clinical Judgment Skills (debiasing techniques) Assesses trait and situational factors (i.e., social, economic, and environmental) Uses multiple sources of assessment (e.g., client interview, reports from significant other and/or family, situational assessment, interest inventory, etc.) and avoids use of single method Attends to details of sessions that initially seem less significant Focuses interview on multiple dimensions (i.e., client, problem/concern, significant others, culture/environment) Actively considers and/or discusses both client’s and own expectations for services Delays final judgments about clients by treating them as tentative decisions and formulations open to revision Importance M SD Preparedness M SD 5.79 1.14 4.72 1.24 6.24 0.85 4.89 1.33 4.76 1.32 3.70 1.32 6.09 0.90 4.74 1.23 5.89 1.08 4.64 1.39 5.79 1.11 4.02 1.39 Factor 4: Memory Bias Uses systematic and comprehensive interviews (i.e., use of semi-structured and/or structured interviews) Carefully documents all salient observations immediately after meeting with clients Utilizes knowledge base of client issues and how they are remedied Periodically reviews case notes and specifies important themes Examines biases associated with counseling theory orientations Makes rationale for decision-making explicit (i.e., writes summary, explains it to supervisor or colleague) 5.49 0.95 4.31 1.10 5.25 1.38 4.43 1.40 5.81 1.11 4.40 1.45 6.01 1.02 4.67 1.23 5.17 1.35 4.08 1.42 5.10 1.38 4.00 1.46 5.63 1.15 4.26 1.36 Factor 5: Confirmatory Bias Re-checks initial case formulations after several sessions Formulates questions or strategies to test and evaluate multiple hypotheses Takes into account own emotional reactions to clients Actively questions the quality of counseling relationship and applies strategies to build and/or maintain it 5.63 5.42 0.86 1.06 4.02 3.95 1.11 1.23 5.24 5.93 1.25 1.11 3.50 4.26 1.40 1.36 5.91 0.97 4.35 1.30 Factor 6: Negative Bias Becomes informed about problems treated by other mental health or health professionals 5.78 0.86 4.61 1.11 5.39 1.23 4.12 1.54 73 Table 5 (cont’d) Clinical Judgment Skills (debiasing techniques) Focuses on client strengths Attends to positive client emotions Uses decision aids (i.e., DSM-IV, State-federal vocational rehabilitation eligibility criteria) when making mental health diagnoses or eligibility determinations M 6.32 5.66 SD 0.89 1.11 M 5.13 4.69 SD 1.23 1.29 5.74 1.25 4.49 1.46 Factor 7: Evidence-Based Practice Critically appraises and applies research literature to practice (e.g., to choose appropriate interventions, to plan assessments) based on client needs and specific context of the case Uses clinical practice guidelines to degree they are supported by empirical research Separates from the counseling process to make more objective third-person observations 5.27 0.99 3.72 1.12 5.61 1.21 3.81 1.44 5.13 1.30 3.67 1.32 5.08 1.34 3.69 1.28 Teaching Proficiency Scale The same factor analytic procedure for the importance scale was used to extract and retain a factor solution of the eight-item teaching proficiency scale. KMO=.78 (good, as noted by Field, 2009), and all KMO values for individual items were > .70, which is well above the adequate limit of .5 (Field, 2009). Bartlett’s test of sphericity χ2 (28) =300.28, p < .001, indicated again, that inter-item correlations were adequate (Field, 2009). The scree plot was also analyzed, but this time provided a clear inflexion that further supported Kaiser’s eigenvalue criterion greater than one (Raykov & Marcoulides, 2008) for retaining two factors with 58.80% of total variance explained. Consequently, two factors of teaching proficiency were retained for subsequent statistical analyses. As shown, Table 6 illustrates a summary of factor loadings for the teaching proficiency scale after rotation. Individual factor mean scores were then generated for the teaching proficiency scale based on this new two factor structure. Table 7 presents the means of the two factors and skill items that constitute each factor. 74 Table 6 Factor Loadings of Exploratory Factor Analysis With Varimax Rotation of Proficiency Scale Teaching Proficiency Items Delays final judgments about clie Rotated Factor Loadings 1 2 .09 .84 Does not become attached to one .09 .80 Discards or tentatively accepts hy .31 .69 Implements conscious and contin .35 .59 Combines statistical and clinical .03 .88 Considers probability and base rat .21 .71 Formulates questions or strategies .38 .65 Critically appraises and applies re .18 .64 Eigenvalues 3.46 1.25 % of variance 43.21 15.59 Note: Factor loadings for each factor appear in boldface. N=126. Cognitively Complex Approach=1; Evidence-Based Practice Process=2. Factor 1: Proficiency in teaching a cognitively complex approach. This debiasing technique consists of four skill items that focus on educators’ proficiency in teaching master’s students to think and practice counseling in a multidimensional manner, using data to make more informed judgments and decisions (Owen & Lindley, 2010), as well as teaching students to withhold judgments and engaging in a self-reflective practice to lower confidence in decisions throughout the counseling process (Koch et al., 2000). The mean inter-time correlation of .44 and computed cronbach alpha α=.76 suggested respectable internal consistency reliability (DeVellis, 2003) among the items that comprised this factor. The mean teaching proficiency of this factor was 5.62 (SD=0.78). Factor 2: Proficiency in teaching an evidence-based practice process. This debiasing technique consists of four skill items that address educators’ proficiency in teaching master’s students how to implement an evidence-based practice process in an effective and realistic manner into their own counseling practice (Leahy & Arokiasamy, 2010; Kosiciulek, 2010; Chan et al., 2010). The mean inter-item correlation of .41 and computed cronbach alpha α=.74 75 indicated respectable internal consistency reliability (DeVellis, 2003) among the items that comprise this factor. The mean teaching proficiency of this factor was 5.01 (SD=1.00). Table 7 Proficiency of Teaching Clinical Judgment Skills Clinical Judgment Skills (debiasing techniques) Teaching Proficiency M SD 5.31 0.77 Factor 1: Cognitively Complex Approach Delays final judgments about clients by treating them as tentative decisions and formulations open to revision Does not become attached to one interpretation of the client data Discards or tentatively accepts hypotheses based on counseling and/or rehabilitation outcome data (i.e., intervention effects, client progress) Implements conscious and continuous self-evaluation of own thinking about clients 5.62 0.78 5.95 0.97 5.63 0.95 5.32 1.18 5.58 0.97 Factor 2: Evidence-Based Practice Process Combines statistical (e.g., vocational, psychological tests) and clinical prediction techniques (e.g., behavioral assessment methods) Considers probability and base rate data Formulates questions or strategies to test and evaluate multiple hypotheses Critically appraises and applies research literature to practice (e.g., to choose appropriate interventions, to plan assessments) based on client needs and specific context of the case 5.01 1.00 5.05 4.32 1.39 1.54 5.40 1.16 5.27 1.24 Research Questions 1. How important are clinical judgment skills for effective rehabilitation counseling practice? As shown in Table 5, participants rated clinical judgment skills (debiasing techniques) as being quite important with a mean of 5.52 (SD=0.72). A mean rating of 6.32 for the highest rated skill item (i.e., Focusing on client strengths) to a mean of 4.01 for the lowest (i.e., Considers 76 probability and base rate data) constituted the range of importance of clinical judgment skills. Of the 35 items that comprise the CJSI, all individual skill items were rated as important (i.e., a mean rating of 4.0 or better) for effective rehabilitation counseling practice. There were six skill items with a mean importance rating of 6.0 or higher and five with an importance rating of 4.0 or higher, but less than 5.0. Table 8 lists individual skill items deemed most important, as well as of least relative importance. Table 8 Highest and Lowest Rated Items of Importance Highest Rated Individual Items (6.0 or higher) Focuses on client strengths Uses multiple sources of assessment (e.g., client interview, reports from significant other and/or family, situational assessment, interest inventory, etc.) and avoids use of single method Focuses interview on multiple dimensions (i.e., client, problem/concern, significant others, culture/environment) Evaluates own cultural biases based on such categories as gender, ethnic minority status, disability, and social class Implements conscious and continuous self-evaluation of own thinking about clients Utilizes knowledge base of client issues and how they are remedied Lowest Rated Items (Less than 5.0) Attends to details of sessions that initially seem less significant Discards or tentatively accepts hypotheses based on counseling and/or rehabilitation outcome data (i.e., intervention effects, client progress Builds in a baseline period of client data collection Weighs empirical results more heavily than own clinical experiences Considers probability and base rate data M 6.32 SD 0.89 6.24 0.85 6.09 0.90 6.07 1.09 6.04 1.08 6.01 1.02 4.76 1.32 4.73 4.48 1.36 1.40 4.14 4.01 1.43 1.44 Note: Importance rating scale; 1.0=Not at all important; 2.0=Slightly important; 3.0=Somewhat important; 4.0=Important; 5.0=Quite important; 6.0=Very important; 7.0=Extremely important. 77 The lower relative importance items show larger standard deviations, thereby greater variation and indicate a wider array of rehabilitation counselor educator opinions about such skills and their level of importance. a. Does the level of importance differ across clinical judgment skill domains? As Table 5 shows, clinical judgment skills that represent the debiasing technique of cultural bias had the highest mean of 5.94 and was considered most important by rehabilitation counselor educators. Clinical judgment skills that represent debiasing techniques that address negative bias was second most important with a mean of 5.78, followed by cognitive complexity (M=5.77), confirmatory bias (M=5.63), memory bias (M=5.49), evidence-based practice (M=5.27), and scientific attitude (M=4.76). Consequently, only scientific attitude was rated as being less than “quite important”. b. Are there differences in the level of importance across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? A factorial independent groups ANOVA using General Linear Modeling (GLM) in SPSS was used to test a main effects model to determine if differences existed in level of clinical judgment skill importance between rehabilitation counselor educator study participants relative to their demographic and programmatic characteristics. Computing the mean of the seven individual factor scores, an overall mean factor score of importance was generated and entered into the model at the interval level as the dependent variable. The simultaneous entry method (Kline, 2011; Mundry, Roger, & Nunn, 2009) was used to enter all eight categorical independent 78 variables of interest for this study into the model as fixed factors; seven demographic variables (i.e., rehabilitation counselor educator demographic variables-gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and one programmatic variable (i.e., clinical judgment emphasis). As Table 9 shows, gender, F(1, 84)=8.69, p<.01, η²=.09, race, F(1,84)=4.33, p<.05, η²=.05, clinical setting F(5, 84)=2.43, p<.05, η²=.13, and perceived program emphasis of clinical judgment training F(3, 84)=3.97, p<.05, η²=.12 were significantly related to study participants perceptions of importance of clinical judgment skills for effective rehabilitation counseling practice. Table 9 Tests of Between-Subjects Effects: Importance of Clinical Judgment Skills Source Type III SS 21.152a df Mean Square 1.058 F PValue .002 Corrected 20 2.536 Model Intercept 1213.428 1 1213.428 2909.911 .000 Gender 3.624 1 3.624 8.692 .004 Race 1.805 1 1.805 4.328 .041 Faculty 1.310 3 .437 1.047 .376 Rank CJ 4.965 3 1.655 3.969 .011 Emphasis Clinical Exp 1.737 4 .434 1.041 .391 Clinical 5.067 5 1.013 2.430 .042 Setting .749 1 .749 1.796 .184 Degree YrsRCE 2.003 2 1.002 2.402 .097 Error 35.028 84 .417 Total 3263.392 105 Corrected 56.180 104 Total Note: a. R Squared=.377 (Adjusted R Squared=.228). N=105 due to missing data. 79 Significant main effects (i.e., gender, race, clinical setting, and program emphasis of clinical judgment training) were further examined using pairwise comparison of marginal means. This post hoc testing method was selected because marginal means are adjusted to estimate the population to make comparisons taking into account all other variable contributions within the main effects model (Taylor, 2011). Consequently, the use of marginal means can be considered a more realistic estimate of the population for this study. Bonferroni adjustments were also used to control for type I error (Field, 2009). Pairwise comparisons computed for gender showed that female participants (M=5.87) had significantly higher clinical judgment skill importance mean scores than males (M=5.44); with a significant mean difference of .43, p<.01. Pairwise comparisons computed for race showed that study participants who were minorities had significantly higher importance ratings of clinical judgment skills (M=5.83) than White educators (M=5.48); with a significant mean difference of .35, p<.05. For clinical setting, no significant differences between settings after the Bonferroni adjustment at the p<.05 level were found. Pairwise comparisons for program emphasis of clinical judgment training showed that study participants working within a rehabilitation counseling master’s program where clinical judgment skills were extremely emphasized had significantly higher mean scores of importance (M=6.15) compared to a program where clinical judgment skill emphasis was either somewhat or not emphasized (M=5.26), with significant mean difference of .89, p<.01, and emphasized (M=5.54), with a significant mean difference of .62, p<.05. There were no significant differences between extremely and very emphasized, very emphasized and emphasized, or between somewhat or not emphasized and emphasized. To illustrate marginal mean estimates and mean 80 differences, Table 10 illustrates pairwise comparisons for both gender and clinical judgment program emphasis. Table 10 Estimates and Pairwise Comparisons for Gender, Race, and Program Emphasis Est. M SE 95% CI LB Male Female 5.44 5.87 .13 .13 5.18 5.62 5.69 6.12 Minority White 5.83 5.48 .16 .10 5.50 5.28 6.15 5.67 CJ Program Emphasis Est. M SE Pairwise Comparison Gender and Race LB UB UB Mean Diff. -.43 .43 Groups .35 -.35 Mean Diff. SE Sig. 95% CI LB UB .15 .15 .004 .004 -.72 .14 -.14 .72 .17 .041 .16 .041 Pairwise Comparison .02 -.68 .68 -.02 SE Sig. 95% CI LB UB Extrem 6.15 .21 5.74 6.56 Extrem emphszd emphszd Very 5.66 .14 5.39 5.94 Very .49 .24 .269 -.16 1.14 emphszd emphszd Emphszd 5.54 .14 5.26 5.81 Emphszd .62 .23 .00 1.23 .049 Somewht 5.26 .19 4.89 5.63 Somewht .89 .27 .16 1.63 .009 or Not or Not Emphszd Emphszd Note: Bonferroni adjustment was used for multiple comparisons. N=105. Group means shown in the table are estimated marginal means. .62 and .89 indicates that educators for this study who perceived their rehabilitation counseling master’s program to extremely emphasize clinical judgment skills had significantly higher overall factor mean scores of clinical judgment skill importance compared to study participants from programs where clinical judgment skills were either emphasized or somewhat or not emphasized. 81 2. What are rehabilitation counselor educators’ perceptions of master’s students’ level of preparedness in clinical judgment skill domains upon graduation to effectively address clinical judgment tasks in rehabilitation counseling practice? Participants rated students as being prepared in the use of clinical judgment skills (debiasing techniques) upon graduation of master’s degree rehabilitation counseling program with a mean of 4.17 (SD=0.99). A mean rating of 5.13 for the highest rated skill item (i.e., Focusing on client strengths) to a mean of 2.72 for the lowest (i.e., Considers probability and base rate data) constituted the range of student preparedness of clinical judgment skills. Of the 35 items that comprise the CJSI, 23 constituted mean ratings of students’ being prepared (i.e., 4.0 or better) with two skill items students were rated with a 5.0 or higher; considered to be “quite prepared”. Whereas, 12 individual skill items had mean ratings of less than prepared (i.e., less than 4.0); “somewhat prepared”. Only one item had a mean rating of below 3.0; “slightly prepared”. There were six skill items with a mean importance rating of 6.0 or higher and five with an importance rating of 4.0 or higher, but less than 5.0. Table 11 lists five individual skill items where master’s students in rehabilitation counseling programs were perceived as being most and least prepared upon graduation, respectively. The preparedness skill items show noticeable variation for both “most prepared” and “least prepared” groups. Table 11 Individual Clinical Judgment Skills Where Students Are Most and Least Prepared Most Prepared Upon Graduation Focuses on client strengths Seeks out supervisory feedback Uses multiple sources of assessment (e.g., client interview, reports from significant other and/or family, situational assessment, interest inventory, etc.) and avoids use of single method 82 M 5.13 5.09 SD 1.23 1.30 4.89 1.33 Table 11 (cont’d) Most Prepared Upon Graduation M SD Focuses interview on multiple dimensions (i.e., client, problem/concern, significant others, culture/environment) 4.74 1.23 Assesses trait and situational factors (i.e., social, economic, and environmental) 4.72 1.24 Least Prepared Upon Graduation Considers probability and base rate data 2.72 1.26 Weighs empirical results more heavily than own clinical experiences 3.36 1.15 Builds in a baseline period of client data collection 3.38 1.41 Formulates questions or strategies to test and evaluate multiple hypotheses 3.50 1.40 Discards or tentatively accepts hypotheses based on counseling and/or rehabilitation outcome data (i.e., intervention effects, client progress) 3.65 1.37 Note: Student preparedness rating scale; 1.0=Not at all prepared; 2.0=Slightly prepared; 3.0=Somewhat prepared; 4.0=Prepared; 5.0=Quite prepared; 6.0=Very prepared; 7.0=Extremely prepared. a. Does the level of perceived student preparedness differ across clinical judgment skill domains? Rehabilitation counselor educators rated students as being prepared (i.e., a rating of 4.0 or higher) in the clinical judgment skill areas overall with a mean of 4.17 (SD=0.99). Clinical judgment skills that represent the debiasing technique of negative bias (M=4.61) had the highest perceived student preparedness rating, followed by cultural bias (M=4.54), cognitive complexity (M=4.39), memory bias (N=4.31), and confirmatory bias (M=4.02). Evidence-based practice (M=3.72) and scientific attitude (M=3.58) were the only two clinical judgment skill areas where students were considered to be less than prepared (i.e., “somewhat prepared”) with a mean rating of less than 4.0. A Clustering Technique In addition, a clustering technique (Berven, 1979; Chan et al., 2003) was administered to determine differences of importance and preparedness across clinical judgment skill areas, as 83 well as, to specify high priority clinical judgment training needs by prioritization and pairing of individual skills identified as “high importance”/“limited student preparation”. First, all 35 clinical judgment skill items from the importance scale were rank ordered based on participant frequencies which were then clustered together based on “high importance” (i.e., 5.0=Quite important; 6.0=Very important; 7.0=Extremely important). For example, skill item, “Uses multiple sources of assessment (e.g., client interview, reports from significant other and/or family, situational assessment, interest inventor, etc.) and avoids of single method” was rank ordered first as 95.2% of participants responded to this item with a 5.0 or higher. The next highest rated skill item (94.4%) based on frequency count of 5.0 or higher, “Focuses interview on multiple dimensions (i.e., client, problem/concern, significant others, culture/environment)” was ordered second, and so on, until all 35 importance were placed in rank order based on this process. This same procedure was used for all 35 items measured on the student preparedness scale. However, clinical judgment skill items from the preparedness scale were rank ordered based on “limited student preparation” (i.e., accumulated frequency count of items rated as 1.0=Not at all prepared, 2.0=slightly prepared, and/or 3.0=somewhat prepared). For example, 73.8% of participants rated skill item, “Considers probability and base rate data”, with a 3.0 or lower and was rank ordered first. Skill item, “Weighs empirical results more heavily then own clinical experiences”, had 54.0% of participants’ rate student preparedness with a 3.0 or less. Again, this process continued in this manner until all 35 preparedness items were positioned in rank order. Second, clinical judgment skill items for both importance and preparedness at or above the median were designated as having “high importance” and “limited student preparation”. To 84 enhance the breath of findings since items had about the same frequency percentage, the median for both scales was arbitrarily lowered; 4% for importance and 2% for student preparedness scales, respectively. High priority clinical judgment training needs was then defined as having at least 80% (or more) of rehabilitation counselor educators rate the skill as having “high importance” (i.e., a rating of 5.0 or higher) and 28% or more rated the same skill as having “limited student preparation” (i.e., a rating of 3.0 or lower). Table 12 illustrates 10 individual skills of high importance-limited preparation across clinical judgment skill domains identified as high priority clinical judgment training needs based on the clustering technique (Berven, 1979; Chan et al., 2003) explained. Table 12 High Priority Clinical Judgment Training Needs High priority training need areas High Importance (%) Limited Preparation (%) 90.5 28.6 87.3 27.8 85.7 29.4 Factor 1: Scientific Attitude No items identified Factor 2: Cultural Bias Evaluates own cultural biases based on such categories as gender, ethnic minority status, disability, and social class Implements conscious and continuous self-evaluation of own thinking about clients Adapts use of counseling style (i.e., qualities and skills) to suit different individuals and cultures Openly discusses diversity and multicultural differences Total Number of Items Factor 3: Cognitive Complexity Does not become attached to one interpretation of the client data Delays final judgments about clients by treating them as tentative decisions and formulations open to revision Total Number of Items 85 84.9 27.8 4 90.5 36.5 87.3 33.3 2 Table 12 (cont’d) High priority training need areas High Importance (%) Limited Preparation (%) 89.7 28.6 87.3 81.7 28.6 31.0 Factor 4: Memory Bias No items identified Factor 5: Confirmatory Bias Actively questions the quality of counseling relationship and applies strategies to build and/or maintain it Takes into account own emotional reactions to clients Re-checks initial case formulations after several sessions Total Number of Items 3 Factor 6: Negative Bias No items identified Factor 7: Evidence-Based Practice Critically appraises and applies research literature to practice (e.g., to choose appropriate interventions, to plan assessments) based on client needs and specific context of the case Total Number of Items 80.2 40.5 1 b. Are there differences in the level of student preparedness across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? A factorial independent groups ANOVA using GLM in SPSS was again used to test a main effects model to determine if there were differences in level of perceived student preparedness across clinical judgment skill areas relative to study participants’ demographic and programmatic characteristics. Computing the mean of the seven individual factor scores, an 86 overall mean score of importance was generated and entered into the model at the interval level as the dependent variable. The simultaneous entry method (Kline, 2011; Mundry et al., 2009) was again used to enter all eight categorical independent variables of interest for this study into the model as fixed factors; seven demographic variables (i.e., rehabilitation counselor educator demographic variables-gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and one programmatic variable (i.e., clinical judgment emphasis). As Table 13 shows, race, F(1, 84)=8.39, p<.01, η²=.09 and perceived program emphasis of clinical judgment training F(3, 84)=8.83, p<.001, η²=.24 were significantly related to study participants perceptions of students’ preparedness of clinical judgment skills for effective rehabilitation counseling practice upon graduation. Significant main effects (i.e., race and program emphasis of clinical judgment training) were further explored using pairwise comparison of marginal means and Bonferroni adjustments to control for type I error. Pairwise comparisons computed for race showed that rehabilitation counselor educator minorities perceived master’s students to be significantly better prepared (M=4.66) than White educators (M=4.00) to effectively use clinical judgment skills upon graduation; with a significant mean difference of .67, p<.01. Pairwise comparisons for program emphasis of clinical judgment training showed that study participants working within a rehabilitation counseling master’s program where clinical judgment skills were extremely emphasized had significantly higher mean scores of students being prepared (M=4.83) compared to a program where clinical judgment skill emphasis was either somewhat or not emphasized (M=3.62); significant mean difference of 1.22, p<.01. 87 Table 13 Tests of Between-Subjects Effects: Student Preparedness in Clinical Judgment Skill Domains Source Type III Sum of Squares 47.828a Df Mean Square 2.391 F PValue .000 Corrected 20 3.053 Model Intercept 711.588 1 711.588 908.326 .000 Gender .108 1 .108 .137 .712 Race 6.576 1 6.576 8.394 .005 Faculty 1.968 3 .656 .837 .477 Rank CJ 20.742 3 6.914 8.826 .000 Emphasis Clinical 2.460 4 .615 .785 .538 Exp Clinical 6.057 5 1.211 1.546 .184 Setting Degree .462 1 .462 .590 .445 YrsRCE .804 2 .402 .513 .600 Error 65.806 84 .783 Total 1958.360 105 Corrected 113.634 104 Total Note: a. R Squared=.421 (Adjusted R Squared =.283). N=105 due to missing data. Further, there were also significant differences between very emphasized (M=4.81) and emphasized (M=4.05); significant mean difference of .76, p<.01; and between very emphasized and somewhat or not emphasized (M=3.62); significant mean differences of 1.19, p<.001. To illustrate marginal mean estimates and mean differences, Table 14 illustrates pairwise comparisons for both race and clinical judgment program emphasis. 88 Table 14 Estimates and Pairwise Comparisons for Race and Program Emphasis Est. M Minority White 4.66 4.00 SE Pairwise Comparison Race 95% CI LB UB .22 4.22 .13 3.73 Est. M SE LB Somewht or Not Emphszd Emphszd 3.62 .26 3.11 4.05 .19 3.67 Very emphszd Extrem emphszd 4.81 .19 4.43 4.83 .28 4.27 Mean Diff. 5.11 Minority .67 4.62 White -.67 CJ Program Emphasis UB Groups 4.13 Somewht or Not Emphszd 4.42 Very emphszd 5.19 Extrem emphszd 5.40 Emphszd Mean Diff. SE Sig. .23 .005 .23 .005 Pairwise Comparison SE Sig. 95% CI LB UB .21 -1.12 1.12 -.21 95% CI LB UB -1.19 .27 .000 -1.92 -.47 -1.22 .37 .010 -2.22 -.21 Very -.76 .23 .007 -1.38 -.15 emphszd Note: Bonferroni adjustment was used for multiple comparisons. N=105. Group means shown in the table are estimated marginal means. Somewhat or Not Emphasized had significantly lower mean score of student preparedness than Very emphasized (-1.19) and Extremely emphasized (1.22). Emphasized also had significantly lower mean score of preparedness than Very emphasized. 3. How do rehabilitation counselor educators perceive their own proficiency in teaching clinical judgment skills? Rehabilitation counselor educators rated themselves as being quite proficient (i.e., a rating of 5.0 or higher) in teaching clinical judgment skills overall with a mean of 5.31 (SD=0.77). A mean rating of 5.95 for the highest rated skill item (i.e., “Delays final judgments about clients by treating them as tentative decisions and formulations open to revision”) to a 89 mean of 4.32 for the lowest (i.e., “Considers probability and base rate data”) constituted the range of mean proficiency ratings of teaching clinical judgment skills. Of the eight skill items that measured teaching proficiency, all individual items had mean ratings of educators being quite proficient (i.e., 5.0 or better) except for one item that fell below 5.0; “Considers probability and base rate data” (M=4.32). An analysis of frequencies for this clinical judgment skill item revealed that 33.3% provided a rating “high importance” (i.e., a rating of 5.0 or higher), 65.9% reported this skill as being important (i.e., a rating of 4.0), and 73.8% reported students having “limited preparation” (i.e., a rating of 3.0 or less). Further, 40.5% reported high proficiency (i.e., a rating of 5.0 or higher) and 23.8% reported limited proficiency (i.e., a rating of 3.0 or less) in teaching this skill. Also, 12.7% of participants perceived themselves to be less than proficient (i.e., a rating of 3.0 or lower) in teaching the clinical judgment skill, “Combines statistical (e.g., vocational, psychological tests) and clinical prediction techniques (e.g., behavioral assessment methods)”. a. Does the level of teaching proficiency differ across clinical judgment skill domains? As shown previously in Table 7, proficiency in teaching the clinical judgment skills that represent the debiasing technique of a cognitively complex approach (M=5.62) had the highest teaching proficiency rating. Proficiency in teaching the clinical judgment skill area of evidencebased practice process was second with a mean of 5.01. b. Are there differences in the level of teaching proficiency across clinical judgment skill domains relative to rehabilitation counselor educators’ demographic (i.e., gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation 90 counselor educator) and programmatic (i.e., clinical judgment emphasis) characteristics? As used to answer research questions, 1b and 2b, a factorial independent groups ANOVA using GLM in SPSS was again used to test a main effects model to determine if there were differences in level of perceived proficiency in teaching across clinical judgment skill areas relative to study participants’ demographic and programmatic characteristics. Computing the mean of the two individual factor scores, an overall mean factor score of teaching proficiency was generated and entered into the model at the interval level as the dependent variable. The simultaneous entry method (Kline, 2011; Mundry et al., 2009) was again used to enter all eight categorical independent variables of interest for this study into the model as fixed factors; seven demographic variables (i.e., rehabilitation counselor educator demographic variables-gender, race/ethnicity, faculty rank, clinical experience/setting, degree major, years of experience as rehabilitation counselor educator) and one programmatic variable (i.e., clinical judgment emphasis). As Table 15 shows, only perceived program emphasis of clinical judgment training F(3, 84)=4.37, p<.05, η²=.14 was significantly related to study participants perceptions of their own teaching proficiency of clinical judgment skill domains. Significant main effects (i.e., program emphasis of clinical judgment training) were further explored using pairwise comparison of marginal means and Bonferroni adjustments to control for type I error. Pairwise comparisons for program emphasis of clinical judgment training showed that study participants working within a rehabilitation counseling master’s program where clinical judgment skills were extremely emphasized had significantly higher mean scores (M=5.88) of perceived proficiency of teaching such skills compared to a program where clinical judgment skill emphasis were emphasized (M=5.07); with a significant mean difference of .81, 91 p<.05. There were no other significant differences between clinical judgment program emphasis categories and perceived teaching proficiency. To illustrate marginal mean estimates and mean differences of teaching proficiency, Table 16 illustrates pairwise comparisons for clinical judgment program emphasis. Table 15 Tests of Between-Subjects Effects: Teaching Proficiency of Clinical Judgment Skill Domains Source Type III Sum of Squares 18.229a Df Mean Square .911 F PValue .112 Corrected 20 1.477 Model Intercept 1120.150 1 1120.150 1814.849 .000 Gender .018 1 .018 .029 .866 Race .890 1 .890 1.442 .233 Faculty 1.348 3 .449 .728 .538 Rank CJ 8.096 3 2.699 4.373 .007 Emphasis Yrs 1.991 4 .498 .806 .525 Clinical Exp Clinical 1.335 5 .267 .433 .825 Setting Degree .010 1 .010 .016 .901 YrsRCE 1.163 2 .582 .942 .394 Error 51.846 84 .617 Total 3038.874 105 Corrected 70.075 104 Total Note: a. R Squared=.288 (Adjusted R Squared=.118). N=105 due to missing data. 92 Table 16 Estimates and Pairwise Comparisons for Program Emphasis CJ Program Emphasis Est. M SE LB UB Groups Pairwise Comparison Mean Diff. SE Sig. 95% CI LB UB Extrem 5.88 .25 5.38 6.38 Extrem emphszd emphszd Very 5.60 .17 5.26 5.93 Very .28 .29 1.00 -.51 1.07 emphszd emphszd Emphszd 5.07 .17 4.74 5.41 Emphszd .81 .28 .06 1.56 .026 Somewht 5.17 .23 4.72 5.62 Somewht .71 .33 .208 -.18 1.61 or Not or Not Emphszd Emphszd Note: Bonferroni adjustment was used for multiple comparisons. N=105. Group means shown in the table are estimated marginal means. Extremely emphasized had significantly higher mean score (.81) of proficiency than Emphasized. 4. What are the relationships among rehabilitation counselor educators’ perceived importance, student preparedness, and teaching proficiency of clinical judgment skills? A correlation analysis (Field, 2009) was conducted to measure the relationships among rehabilitation counselor educators’ perceived importance, student preparedness, and teaching proficiency of clinical judgment skills (debiasing techniques). A t-test of significance for the product-moment correlation coefficient r was used. First, the overall factor means for each scale was analyzed. Table 17 shows a significant relationship between importance and teaching proficiency of clinical judgment skills, r=.43, p (two-tailed) < .01. Teaching proficiency was also significantly related to how well rehabilitation counselor educators perceived students preparedness in using clinical judgment skills upon graduation, r=.39, p (two-tailed) <.01. Further, the importance of clinical judgment skills for practice as perceived by rehabilitation 93 counselor educator participants was significantly related to how they perceived students were prepared in using such skills upon graduation, r=.34, p (two-tailed) <.01. Table 17 Summary of Intercorrelations for Overall Mean Scores on the Importance for Practice, Student Prepardness, and Teaching Proficiency Scales Measure 1 2 3 M SD 1. Importance 1.00 5.52 0.72 2. Preparedness .34** 1.00 4.17 0.99 3. Proficiency .43** .39** 1.00 5.31 0.77 Note: Overall mean scores of seven factors of importance and preparedness and overall mean scores of two factors of teaching proficiency were used. **Correlation is significant at the .01 level (2-tailed). The relationships between individual clinical judgment skill domains relative to each respective scale was analyzed next. The level of importance of having a scientific attitude for effective rehabilitation counseling practice was significantly related to rehabilitation counselor educators’ perceptions of their own proficiency in teaching clinical judgment skills (debiasing techniques) within the evidence-based process domain, r=.49, p (two tailed) < .01. The importance of cognitive complexity was also significantly related to rehabilitation counselor educator proficiency in teaching a cognitively complex approach to rehabilitation counselor practice, r=.43, p (two tailed) < .01. Interestingly, the importance of evidence-based practice was the only clinical judgment skill domain that did not have a statistically significant relationship with proficiency in teaching a cognitively complex approach (See Table 18). 94 Table 18 Relationships Between Importance and Teaching Proficiency of Clinical Judgment Skills 2 2. ebpprcss (T) 1 1 .51** 3. scientfc (I) 4. cultr (I) 5. cgcmplx (I) 6. memry (I) 7. cnfrmtry (I) .28** .35** .43** .30** .25** 8. negatv (I) .31** 1. cgcplxaprch (T) 3 4 5 6 7 .49** .21* .30** .39** .19* 1 .41** .58** .63** .47** 1 .72** .64** .64** 1 .65** .71** 1 .59** 1 .32** .51** .61** .63** .52** .57** 8 9 1 1 9. ebp (I) .075 .22* .52** .51** .50** .63** .54** .47** 1 Note: Data was imputed for missing values; there is no missing data. N=126. (T)=Teaching proficiency scale; (I)=Importance scale. 1=Cognitively Complex Approach; 2=Evidence-Based Practice Process; 3=Scientific Attitude; 4=Cultural Bias; 5=Cognitive Complexity; 6=Memory Bias; 7=Confirmatory Bias; 8=Negative Bias; 9=Evidence-Based Practice. **. Correlation is significant at the .01 level (2-tailed). *.Correlation is significant at the .05 level (2-tailed). The relationships between student preparedness in clinical judgment skill domains and teaching proficiency were also analyzed. As shown in Table 19, proficiency in teaching a cognitively complex approach was significantly related to perceived student preparedness in a number of clinical judgment skill domains such as, scientific attitude, r=.41, p (two tailed) < .01; memory bias, r=40, p (two tailed) < .01; cognitive complexity=.38, p (two tailed) < .01; and cultural bias, r=35, p (two tailed) < .01. Rehabilitation counselor educator proficiency in teaching an evidence-based practice process was also significantly related to student preparedness in memory bias, r=.35, p (two tailed) < .01; cognitively complexity, r=.34, p (two tailed) < .01; and scientific attitude, r=.34, p (two tailed) < .01. 95 Table 19 Relationships Between Student Preparedness and Teaching Proficiency of Clinical Judgment Skills 2 2. ebpprcss (T) 1 1 .51** 3. scientfc (P) 4. cultr (P) 5. cgcmplx (P) 6. memry (P) 7. cnfrmtry (P) .41** .35** .38** .40** .30** 8. negatv (P) .34** 1. cgcplxaprch (T) 3 4 5 6 7 .34** .25** .34** .35** .22* 1 .74** .79** .82** .72** 1 .87** .81** .85** 1 .88** .82** 1 .79** 1 .30** .74** .84** .81** .75** .75** 8 9 1 1 9. ebp (P) .24** .14 .79** .72** .71** .74** .74** .63** 1 Note: Data was imputed for missing values; there is no missing data. N=126. (T)=Teaching proficiency Scale; (P)=Student Preparedness Scale. Factors; 1=Cognitively Complex Approach; 2=Evidence-Based Practice Process; 3=Scientific Attitude; 4=Cultural Bias; 5=Cognitive Complexity; 6=Memory Bias; 7=Confirmatory Bias; 8=Negative Bias; 9=Evidence-Based Practice. **. Correlation is significant at the .01 level (2-tailed). *.Correlation is significant at the .05 level (2-tailed). Finally, an analysis of the relationships between study participants perceived importance of clinical judgment skill domains and perceived student preparedness in such areas was conducted. Table 20 illustrates numerous significant interrelationships between importance and student preparedness. The clinical judgment skill domain of memory bias was significantly related to perceived student preparedness in this same clinical judgment skill domain, r=.41, p (two tailed) < .01; importance of clinical judgment skill domain, negative bias, was also significantly related to student preparedness in the same skill area, r=.40, p (two tailed) < .01; importance of cognitive complexity was significantly related to preparedness of students in the negative bias domain, r=.40, p (two tailed) < .01. Whereas, evidence-based practice and confirmatory bias had the least significant relationships between study participants’ reported importance and student preparedness across clinical judgment skill areas. 96 Table 20 Relationships Between Importance and Student Preparedness of Clinical Judgment Skills 1. (I) 2. (I) 3. (I) 4. (I) 5. (I) 6. (I) 7. (I) 8. (P) 9. (P) 10. (P) 11. (P) 12. (P) 13. (P) 14. (P) 1 1 .41 ** .58 ** .63 ** .48 ** .51 ** .52 ** .39 ** .21 * .26 ** .30 ** .18 .28 ** .19 * 2 3 4 5 6 7 8 9 10 11 12 13 14 1 .72 ** .64 ** .64 ** .61 ** .51 ** 1 .65 ** .71 ** .63 ** .50 ** .59 ** .52 ** .63 ** .21 * .37 ** .33 ** .27 ** .28 ** .35 ** .21 * .26 ** .30 ** .34 ** .32 ** .28 ** .28 ** .20 * .28 ** .30 ** .40 ** .41 ** .31 ** .33 ** .20 * 1 1 .57 ** .54 ** .11 .21 * .18 * .16 .32 ** .28 ** .14 1 .48 ** 1 .25 ** .28 ** .26 ** .21 * .22 * .40 ** .15 .07 1 .12 .75 ** .79 ** .82 ** .72 ** .74 ** .79 ** .11 .09 .18 * .13 .18 * 1 .87 ** .81 ** .85 ** .84 ** .72 ** 1 .88 ** .82 ** .81 ** .71 ** 1 .79 ** .75 ** .74 ** 1 .75 ** .74 ** 1 .63 ** 1 Note: Data was imputed for missing values; there is no missing data. N=126. (I)=Importance for Practice Scale; (P)=Student Preparedness Scale. Clinical Judgment Skill Domains; 1 and 8=Scientific Attitude; 2 and 9=Cultural Bias; 3 and 10=Cognitive Complexity; 4 and 11=Memory Bias; 5 and 12=Confirmatory Bias; 6 and 13=Negative Bias; 7 and 14=EvidenceBased Practice. **. Correlation is significant at the .01 level (2-tailed). *.Correlation is significant at the .05 level (2-tailed). 97 Chapter 5 Discussion To address the research questions and generate data useful for better understanding rehabilitation counselor educators’ perceptions of teaching clinical judgment skills for effective rehabilitation counseling practice, a quantitative Internet-based survey design was used. It was anticipated that findings from this study would inform and enhance both master’s and doctorallevel rehabilitation counseling clinical training curricula. The results of this study indicate that clinical judgment skills are highly important for effective rehabilitation counseling practice. This finding is consistent with previous training research in clinical judgment and decision-making (Harding, 2007). Further, the current study provides new knowledge of empirically-supported clinical judgment skill domains. Seven factors (scientific attitude, cultural bias, cognitive complexity, memory bias, confirmatory bias, negative bias, and EBP) were identified from an EFA that constitute debiasing techniques that characterize clinical judgment skills for effective rehabilitation counseling practice. Two additional factors (i.e., a cognitively complex approach and EBP process) were also identified that address educators’ proficiency of teaching these two clinical judgment skill domains. The results of such EFAs performed validate the considerable steps taken to develop a reliable and valid instrument for this study. For example, the underlying clinical judgment skills that comprise each debiasing technique were developed based on two theoretical frameworks used for this study (i.e. judgment heuristics, biases, and knowledge structures, and a scientistpractitioner model) (Garb, 1998; Tversky & Kahneman, 1982; Slovic et al., 2002; Spengler et al., 1995). The majority of items were generated from a review and synthesis of the most notable findings of the clinical judgment and bias research (Lopez, 1989; Garb, 2010; Garb, 1998; 98 Strohmer & Shivy, 1994; Rosenthal & Berven, 1999) and from sources within the clinical training literature (Ivey & Ivey, 2008; Morrow & Deidan, 1992; Buser, 2008; Meier, 2003; Chan et al., 2010; Chwalisz, 2003; Kazdin, 2006; Stricker & Trierweiler, 1995). CORE standards (CORE, 2011) and knowledge and skill studies (Leahy et al., 2003; Leahy et al., 2009) were also analyzed and used to develop items. Further, feedback provided by five expert panelists was used to upgrade item content. Consequently, the current study’s findings from EFAs conducted provide significant statistical evidence for the construct validity and reliability of the CJSI. Relationships with Previous Clinical Judgment and Training Research Rehabilitation counselor educators reported that all clinical judgment skills (debiasing techniques) were important. Such findings are consistent with the rehabilitation literature that suggest that clinical judgment is important (Strohmer & Leierer, 2000; Berven, 2004, 2008; Lustig, 1996; Rubin & Roessler, 1980; Bellini & Rumrill, 1999; Rosenthal & Kosciulek, 1996) and an essential part of counseling practice (Strohmer & Leierer, 2000), clinical interviewing (Berven, 2004, 2008), vocational assessment (Lustig, 1996), and case management processes (Rubin & Roessler, 1980). Further, such findings supports the research literature that indicates that the use of debiasing techniques has potential to improve rehabilitation counselor clinical judgment accuracy and quality of direct client services to individuals with disabilities throughout the counseling process (Berven, 2008; Spengler et al., 1995). The finding that the clinical judgment skill domain of cultural bias was considered most important by educators is consistent with the rehabilitation counseling literature that has advocated for the integration of multicultural competency training (i.e., knowledge, awareness, and skills) into the CORE curriculum (Middleton, et al., 2000). Further, considerable research has shown counselor clinical judgments to be influenced by social factors such as client 99 characteristics (i.e., gender, race, social class, disability, age) (Broverman, Broverman, Clarkson, Rosenkrantz, and Vogel, 1970; Lopez, 1989; Rybarczyk, Haut, Lacey, Fogg, & Nicholas, 2001). Such research further supports the importance of training debiasing techniques or clinical judgment skills that effectively address cultural bias. In light of these findings, it is also important to consider the possible relative importance of clinical judgment skills that address cultural bias as well as relative importance of all other clinical judgment skill domains. Although the data show different levels of importance when comparing educators’ mean and standard deviation scores (See Table 5), it may be that the rated importance of clinical judgment skill domains is relative. For example, the clinical judgment skills that characterize the debiasing technique of cultural bias may be more important for certain client populations and less important for others. Consequently, all clinical judgment skill domains are considered important, and special consideration of relative importance for each respective skill domain should be given. Another important result of the current study showed that rehabilitation counselor educators rated students as being generally prepared in the use of clinical judgment skills upon graduation. Rehabilitation counselor educators reported students to be most prepared upon graduation in using clinical judgment skills that represent the debiasing technique of negative bias. This debiasing technique consists of clinical judgment skills intended to counteract counselors’ tendency to focus on more negative client information (Strohmer et al., 1995), as well as, judgment errors related to the possible underestimation of client potential (Rosenthal & Berven, 1999; Rosenthal, 2004). Given that a strengths-based, consumer-driven (Kosciulek, 1999, 2004) approach toward working with clients underlies prevailing rehabilitation philosophy, it is not surprising that master’s students were reported to be most prepared in using skills within 100 this domain. In addition, rehabilitation counselors are often required to make accurate clinical judgments related to client eligibility for services, the selection of vocational objectives, counseling interventions, and the development of rehabilitation plans (Berven, 2004). The use of decision-aids (i.e., DSM-IV, State-federal vocational rehabilitation eligibility criteria) and knowledge of problems treated by other mental health or health professionals (Garb, 1998) may be important to maintaining a positive client orientation (Kosciulek, 2004) when accomplishing such tasks. Results from this study also showed that espousing a scientific attitude for effective rehabilitation counseling practice was the least of relative importance (although important). In addition, master’s students in rehabilitation counseling degree programs were also reported to be the least prepared upon graduation in using such “scientifically-oriented” clinical judgment skill domains in the areas of scientific attitude and EBP. To explain such findings, EBP has only recently become more explicitly emphasized within the rehabilitation counseling discipline as a newly integrated model of practice (Chan et al., 2008; Leahy & Arokiasamy, 2010). However, this research-practice dichotomy is not new in rehabilitation counseling (Chan et al., 2008; Chan et al., 2004; Graves, 1992; Johnston et al., 2009; Tarvydas et al., 2010). For example, research skills may be taught more to understand research and not necessarily for its use and application purposes in actual counseling practice (Heppner, 1999). Chan et al. (2004) also suggest that rehabilitation counseling master’s students often view research as their least favorite class. Research from the role and function and knowledge and skills studies overtime have demonstrated that CRC’s view research as the least important domain related to practice compared to other knowledge and skill domains (Leahy et al., 1993; Leahy et al. 2003; Leahy et 101 al., 2009). This may be a result of how curricula are delivered within rehabilitation counseling master’s programs which may not emphasize the importance of research in relation to practice. In addition, the underlying philosophy of EBP in rehabilitation counseling is for counselors to use the best available evidence in relation to interventions that lead to the best possible client outcomes (Chan et al., 2009). A clinical decision-making process is employed that involves: (a) articulating a question about a clinical problem, (b) locating the best research available to address the problem, (c) appraising the validity and clinical applicability of evidence, (d) implementing the most useful findings that address the clinical questions with clients in practice, and (e) evaluating outcomes of interventions applied (Straus et al., 2011; Kosciulek, 2010; Chan et al., 2010; Chronister et al., 2008). This paradigm shift with its roots in medicine (Sackett et al., 1996) may seem counter to the long-standing, interpersonally and intuitively driven model of counseling practice (Sexton et al., 1997; Chan et al., 2009) and rehabilitation philosophy (Tarvydas et al., 2010). Further, the foundations of rehabilitation philosophy of focusing on client strengths (Ivey & Ivey, 2008), client empowerment (Kosciulek, 1999), and shared decision-making within the context of the helping relationship (Gambrill, 2005), may appear minimized using a more “scientifically-oriented” approach and, thus, provide a false perception of adopting a counseling orientation that lacks empathy (Spengler, et al., 1995). Additional results from the current study showed that rehabilitation counselor educators rated themselves as being generally quite proficient in teaching clinical judgment skills. In particular, educators reported that they were most proficient in teaching clinical judgment skills that represent the debiasing technique of a cognitively complex approach, followed by the EBP process. The finding within the clinical judgment skill domain of a cognitively complex 102 approach can be explained by the rehabilitation education literature, as well as the clinical training literature across helping disciplines (i.e., rehabilitation counseling, counseling, counseling psychology, etc.). For example, rehabilitation counselor educators likely appreciate the complexities involved to being effective within the counseling process based on their clinical experience (Berven, 2004). Further, educators may be knowledgeable of the cognitive complexity research which has been linked to greater judgment accuracy among counselors-intraining and practitioners within the counseling research (Falvey et al., 2005; Spengler & Strohmer, 1994; Holloway & Wolleat, 1980). Rehabilitation counselor educators may also value and educate students in a self-reflective counseling approach in working with clients (Koch et al., 2000) and consequently, teach master’s students to be cognitively complex thinkers in their work with clients. In addition, it may be that many counselor educators use a developmental model (e.g., Stoltenberg & McNeill, 2010) of clinical supervision which also uses a framework that views counselor development through a lens of developing a more cognitively complex or higher order thinking about clients. It is also not surprising that educators report to be proficient in teaching an EBP process. Most rehabilitation counselor educators have attained research degrees (Ph.D.’s), have research skills, and extensive training in measurement and evaluation, statistics, and have spent considerable time critiquing the quality of research methodology in applied journal articles. However, it may also be that a more in-depth critical analysis has not yet been given to the complexities involved within counselor clinical decision-making processes of deciding what evidence to use, how to apply it, within the context of what clinical setting, and particular individual client case (Trierweiler, 2006). 103 High Priority Clinical Judgment Training Needs Results of this study also identified 10 individual skills across four (i.e., cultural bias, cognitive complexity, confirmatory bias, and EBP) clinical judgment skill domains as high priority clinical judgment training needs. High priority clinical judgment training needs were defined as clinical judgment skills rated as being highly important, but limited in student preparation. The debiasing technique to address cultural bias (n=4) had the largest number of individual skills considered to be high priority clinical judgment training needs, followed by confirmatory bias (n=3), cognitive complexity (n=2), and EBP (n=1). As noted in previous rehabilitation counselor training research (Havranek & Brodwin, 1994; as cited in Chan et al., 2003), it may be that the reason for such identified training needs is that CORE accreditation standards do not adequately address such debiasing techniques, or due to training dollars provided by the Rehabilitation Services Administration and its influence on shaping curriculum, as well as state licensure or CRCC requirements for counselors. The finding of cultural bias having the most training needs is consistent with Chan et al’s (2003) study that also identified multicultural counseling and gender issues as critical training needs for rehabilitation counselors. It may be that more clinical training than two years is needed to develop minimum competencies in such clinical judgment skill domains than what is realistic for a master’s degree rehabilitation counseling program. Such rationale is consistent with related rehabilitation counseling training research (Chan et al., 2003) that suggested a potential need to institute specialty tracks (i.e., mental health, non-profit rehabilitation, private rehabilitation) within master’s degree rehabilitation counseling programs. Further, it is possible that some master’s degree programs in rehabilitation counseling do not offer a multicultural counseling course or it is unclear how such training is offered or if it is delivered effectively (Donnell, 104 Robertson, & Shannon, 2009). This possible explanation can also be explained by recently revised code of professional ethics for rehabilitation counselors which added additional culturally sensitivity and diversity content (Cartwright & Fleming, 2010). Further, in the current study, only 40.9% (n=45) of rehabilitation counselor educators reported that clinical judgment skills were taught in the multicultural counseling course; the least among other clinical coursework offered. The high priority clinical judgment training need in the area of cognitive complexity can be attributable to the continuum of clinical judgment skill abilities across students in master’s degree rehabilitation counseling programs. For example, research suggests a range of complexity in terms of reasoning processes that counselors possess (Falvey et al., 2005; Spengler & Strohmer, 1994; Holloway & Wolleat, 1980). There are also multiple confirmatory bias studies in the counseling literature that suggest counselors are more likely to remember and attend to information that confirms their initial client impressions (Strohmer et al., 1990; Strohmer & Shivy, 1994; Owen, 2008). Educators’ awareness of such research may also help explain the high priority training needs identified within these clinical judgment skill domains. The high priority training needs identified in the clinical judgment skill domain of EBP can be explained by the relatively new emphasis and stated importance of EBP in rehabilitation counselor education (Leahy & Arokiasamy, 2010; Shultz et al., 2007; Shaw et al., 2006; Chan et al., 2010; Kosciulek, 2010; Koch et al., 2009) and the need for rehabilitation counselors to develop a high level of clinical judgment skills in this area. Demographic and Programmatic Characteristics The results of this study will be discussed that relate to the differences in the level of importance, student preparedness, and teaching proficiency across clinical judgment skill 105 domains relative to rehabilitation counselor educators’ demographic and programmatic characteristics. Program Emphasis of Clinical Judgment Training The results of the current study found programmatic emphasis of clinical judgment training to be significantly related to importance, student preparedness, and teaching proficiency in clinical judgment skill domains (debiasing techniques). In particular, the results of the current study suggest that students, who receive training from master’s degree programs where clinical judgment training is highly emphasized, are better prepared in using clinical judgment skills upon graduation. Also, such programs that highly emphasize clinical judgment skills appear to have educators who are more proficient and assign greater importance in teaching such skills than educators from programs where clinical judgment skills are only emphasized, somewhat or not emphasized. In addition, program emphasis of clinical judgment training had small (η²=.12 for importance and proficiency) to medium (η²=.24 for preparedness) effect sizes (ES) (Cohen, 1992). Such ESs suggest that program emphasis of clinical judgment training is a practically meaningful factor that explains differences in relation to level of student preparedness, importance, and educator proficiency in teaching clinical judgment skill domains. For example, η²=.24 suggests that 24% of the differences in the overall mean score of student preparedness across clinical judgment skill domains were explained by programmatic emphasis of clinical judgment training. Clinical Setting The results from this study suggest that the clinical setting of where educators’ had the majority of their experience providing direct client services is likely to influence what clinical 106 judgment skills they think is important. This finding can be explained by the rehabilitation counselor knowledge and skill studies that have shown over time that there are differences between knowledge and skill areas deemed important by CRC’s across rehabilitation settings (Leahy et al., 2003; 2009). Further, the ES of Clinical setting (η²=.13) is considered to be a small effect (Cohen, 1992), but practically meaningful and suggests that 13% of the differences in the overall importance of clinical judgment skill domains were explained by clinical setting of where educators’ had the majority of their experience providing direct client services. Race and Gender The findings from this study suggest that female and minority educators attribute clinical judgment skills as being more important than male or White educators, respectively. Further, master’s students of minority educators are reported to be better prepared in clinical judgment skills than students of White educators. It is also important to note that small ESs (Cohen, 1992) for both gender (η²=.12 for importance) and race (η²=.05 for importance and η²=.09 for preparedness) were detected. Although ESs are small, such findings suggest practically meaningful factors that explain the differences attributed to importance and student preparedness in using clinical judgment skills upon graduation. To interpret the finding that females rated clinical judgment skills to be significantly more important than males, it is possible that gender differences can be better understood within the context of previous gender bias research. For example, a study by Broverman et al. (1970) found that females were differentially judged compared to males and provided empirical support for stereotypic attitudes held by clinicians (both by men and women) based on gender. In light of this research, it may be that female educators have had more negative experiences by being treated differently than males due to a gender stereotyping bias. It may also be that females are 107 on the receiving end of experiencing micro aggressions or subtle disparaging comments by others that relate to being female (Sue & Sue, 2008) more so than males. For these reasons, it appears plausible that female educators could hold greater importance of such clinical judgment skill domains intended to address inherent counselor biases than males. The significantly higher importance and student preparation ratings in clinical judgment skill domains for minority than White educators may be interpreted in similar fashion as gender. For example, the rehabilitation counseling clinical judgment literature provides evidence of racial bias among practitioners, particularly, White counselors judging African-American clients more negatively as well as underestimating their educational and vocational potential (Rosenthal & Berven, 1999; Rosenthal, 2004). In light of this research, it may be that minority educators have experienced similar negative consequences as a result of racial bias more often or in a more salient fashion than White educators, and therefore place higher importance on teaching skills and preparing counselors in debiasing techniques to address counselor biases. Another possible explanation for such differences in importance and student preparation between White and minority educators may be that White counselor educators’ use or teach a “universalistic” approach to counseling when working with different cultural groups based on the premise that all individuals are part of the same human race (Patterson, 2004; Sue & Sue, 2008). Such a monocultural view may not adequately take into account racial attitudes, stereotypes, prejudices, biases, beliefs, and values and their effect within the counseling relationship. In addition, results of a study by Bellini (2002) found that minority counselors rated higher in multicultural counseling competencies than White counselors, with multi-racial counselors rating the highest. Taken within the context of this finding, it appears plausible that minority educators may have more advanced multicultural counseling competencies than White educators, and 108 therefore might also explain the significantly higher ratings of importance and student preparedness in clinical judgment skill domains for minority educators. Relationships Between Importance, Student Preparedness, and Teaching Proficiency Results that showed significant relationships with medium ESs (Cohen, 1992) were found across multiple relationship combinations between importance, student preparedness, and teaching proficiency in clinical judgment skills. Proficiency and importance held the strongest relationship (r=.43), followed by proficiency and student preparedness (r=.39), and importance and preparedness (r=.34). Consequently, such results are practically meaningful and suggest that student preparation in using clinical judgment skills upon graduation is highly related to how important educators’ rate clinical judgment skills to be, along with how proficient educators’ are in teaching such skills. Importance and Teaching Proficiency Additional results from this study showed a strong positive relationship, very close to a large ES of .50 (Cohen, 1992), between importance of having a scientific attitude for effective rehabilitation counseling practice and educators’ proficiency in teaching clinical judgment skills within the EBP Process domain (r=.49). Further, the importance of cognitive complexity was found to be highly related to educators’ proficiency in teaching a cognitively complex approach of clinical practice (r=.43). Such findings seem to explain the similarities between the debiasing technique of a having a scientific attitude and taking an EBP process approach to practice, as well as, between cognitive complexity and a cognitively complex approach. In light of these results, it appears that the more important educators’ believe having a scientific attitude or thinking in a cognitively complex manner is for effective rehabilitation counseling practice, the more proficient they may be in teaching an EBP process or cognitively complex approach. On 109 the other hand, it could also be that the more proficient educators’ are in teaching an EBP process or cognitively complex approach, the greater importance they place on having a scientific attitude or cognitively complexity. Teaching Proficiency and Student Preparedness The results of the current study show a number of strong positive correlations with medium ESs (Cohen, 1992). There were significant relationships between teaching proficiency of a cognitively complex approach and student preparedness in having a scientific attitude (r=.41), memory bias (r=.40), cognitive complexity (r=.38), and cultural bias (r=.35). There were also similar positive relationships found between teaching proficiency of an EBP process and student preparedness in memory bias (r=.35), cognitive complexity (r=.34), and scientific attitude (r=.34). These findings that show proficiency in teaching a cognitively complex approach and EBP process as being highly correlated with perceived student preparedness across a number of clinical judgment skill domains is noteworthy. Essentially, such findings suggest that if educators’ have a greater degree of proficiency in teaching both a cognitively complex approach and EBP process, it appears that students will be significantly better prepared across such important clinical judgment skill domains (i.e., scientific attitude, memory bias, cognitive complexity, and cultural bias) than students who receive the same training from educators’ who are less proficient in such skill areas. Importance and Student Preparedness This study’s results that indicate significant positive correlations between the importance and student preparedness of memory bias (r=.41), and importance of negative bias (r=.40) and cognitive complexity (r=.40) with student preparedness in the negative bias domain, are considerable findings. Such findings show medium ESs (Cohen, 1992) and are practically 110 meaningful. Further, it appears that the greater level of importance educators’ place on the clinical judgment skill domains of memory bias, the better students are prepared upon graduation in such skill area. Likewise, the higher level of importance educators’ place on the clinical judgment skill domains of negative bias and cognitive complexity, the better students are prepared in the negative bias skill domain upon graduation. The finding that EBP and confirmatory bias had the least significant relationships between importance and student preparedness across clinical judgment skill areas may suggest that such clinical judgment domains are more discrete or less interrelated between the other five skill domains. Limitations Limitations of this study included the use of a self-report questionnaire. Self-report questionnaires are susceptible to participant response bias (i.e., participants may attempt to guess and confirm researcher hypotheses or answer survey questions in socially desirable ways) (Heppner et al., 1999). Another limitation is the low response rate for this study (22.7%; N=126). The low response rate for the current study falls in-between response rates of previous research that surveyed rehabilitation counselor educators; 12% (Coduti, 2009) and 29.2% (Zanskas, 2007). The low response rate may limit the representativeness of the sample to the population; thereby limiting the generalizability of the results to the population of rehabilitation counselor educators across the country. Further, given the low response rate and lack of demographic data available for rehabilitation counselor educators, it is impossible to make comparisons between this study’s sample and the population of educators. If additional participants responded to the present study’s survey, they may have answered differently and different results may have been found. 111 Another limitation is that the primary assumption of this study relies on the validity of rehabilitation counselor educators’ self-report and knowledge of clinical judgment tasks that relate to their own perceptions of importance, their own teaching proficiency, and level of master’s student preparedness of clinical judgment skill domains. It is unknown the degree to which rehabilitation counselor educators’ are accurate in their perceptions. Also, it may be that rehabilitation counseling practitioners have different perceptions of what clinical judgment skills are needed for effective practice. Further, practitioners may also have different opinions about their level of preparation or competencies in using such skills upon graduation. An additional limitation is that the CJSI was originally constructed specifically for this study and no reliability or validity studies were conducted prior to this study to validate the psychometric properties or construct validity of the CJSI. Consequently, additional follow-up studies are needed to further test the construct validity of the CJSI. In addition, the inability to compare the CJSI measure with other related or dissimilar instruments make it impossible to test the CJSI’s concurrent or discriminant validity, respectively. Small sample size may also be a limiting factor for having a desirable subject to item ratio for computing a simultaneous factor analysis of the full CJSI measure (Costello & Osborne, 2005). However, it should be noted that even with large sample sizes, exploratory factor solutions are prone to error above the standard alpha level of p<05 (Costello & Osborne, 2005). Knowing that the quality of the instrument is most important in a survey design (Heppner et al., 1999), several steps were taken to address such limitations to enhance the CJSI’s reliability and validity for this study. Further, this study’s EFA performed for the importance scale of clinical judgment skills, had a KMO=.88 (very good according to Field, 2009) which provided evidence to substantiate the adequacy of sample size for factor analysis. Also, Bartlett’s test of 112 sphericity χ2 (595) =2776.43, p < .001, indicated that inter-item correlations were substantial and adequate (Field, 2009). Also, the EFA conducted for the eight-item teaching proficiency scale had a similar result; a KMO=.78 (good, as noted by Field, 2009) and Bartlett’s test of sphericity χ2 (28) =300.28, p < .001, indicated again, that inter-item correlations were adequate (Field, 2009). Implications The analyses of data have led to a number of implications for both master’s and doctorallevel rehabilitation counseling clinical training curricula and future research. Implications for Clinical Training Clinical judgment skill domains should be considered for inclusion in the CORE standards and used for clinical curriculum development. Rehabilitation counselor educators provided clear evidence that clinical judgment skills (debiasing techniques) are highly important for effective rehabilitation counseling practice. Of the 35 items that comprise the CJSI, all individual skill items were rated as important. Further, the results of an EFA validated the constructs of the CJSI and provided new knowledge of empirically-supported clinical judgment skill domains reported by rehabilitation counselor educators to be important for effective rehabilitation counseling practice. The results of this study also suggest variation in student preparation in using clinical judgment skills for effective rehabilitation counseling practice upon graduation. For example, of the 35 items that comprise the CJSI, students were reported to be prepared in 23 clinical judgment skills, whereas, there were 12 clinical judgment skills where students were reported by educators to have limited preparation. These findings can be used to inform and guide program development of clinical curricula, across coursework, but may be particularly useful in designing clinical skill 113 competencies for the clinical practicum and internship experiences since CORE skill standards are not clearly stated in these areas. Rehabilitation counseling programs should highly emphasize the training of clinical judgment skills. The current study’s results that indicate that students, who receive training from master’s degree programs where clinical judgment training is highly emphasized, are better prepared in using clinical judgment skills upon graduation. Further, programs that highly emphasize clinical judgment skills appear to have educators who view themselves as being more proficient and assign greater importance in teaching such skills than educators from programs where clinical judgment skills are only emphasized to somewhat or not emphasized. These findings suggest that programmatic emphasis plays an influential role on educators’ opinions about what clinical judgment skill areas they should teach, how important they are, how well prepared students should be, as well as how proficient educators’ should be in teaching such skills. Given these findings coupled with this study’s results that shows educators’ reported importance of clinical judgment skills across domains, it appears that increased program emphasis of clinical judgment training will likely not only benefit students and enhance their clinical skills, but also benefit clients with disabilities who will be receiving counseling services from students. Focus clinical training of master’s students on high priority clinical judgment skills. The results of this study identified 10 individual skills across four (i.e., cultural bias, cognitive complexity, confirmatory bias, and EBP) clinical judgment skill domains as high priority clinical judgment training needs (See Table 12). High priority clinical judgment training needs were defined as a clinical judgment skill rated as being highly important, but limited in student preparation. The debiasing technique to address cultural bias had the largest number of 114 individual skills considered to be high priority clinical judgment training needs (n=4), followed by confirmatory bias (n=3), cognitive complexity (n=2), and EBP (n=1). The prioritized need for clinical training in these clinical judgment skill areas are well supported in the professional literature (Garb, 1998; Lopez, 1989; Rosenthal, 2004; Ivey & Ivey, 2008; Falvey et al., 2005; Spengler & Strohmer, 1994; Strohmer et al., 1990; Chan et al., 2010). Consequently, the 10 clinical judgment skills identified as high priority can be used to inform educator decisionmaking in establishing clinical skill competencies or skills that are deserving of more attention. Teach clinical judgment skill domains within the multicultural counseling course and as part of multicultural competency and diversity education across the curriculum. The results of this study that suggest clinical judgment skills are being taught the least in the multicultural counseling course relative to other clinical coursework offered (i.e., individual and group counseling, assessment, internship and practicum) is noteworthy given that the clinical judgment skills empirically-validated in this study address cultural diversity. Also, the present study’s findings show significantly higher importance ratings for female and minority educators, as well as higher student preparedness ratings for minority educators in using clinical judgment skills. In light of these findings, integrating clinical judgment skill training within the multicultural counseling course and as a component of multicultural competency and diversity education across the CORE curriculum appears justified. Further, selected clinical judgment skill domains (e.g., cultural bias) taught in this manner is likely to enhance educators’ perceived importance and student preparation in using such skills for effective rehabilitation counseling practice. 115 Emphasize the importance of clinical judgment skill domains of counselor memory bias, cognitive complexity, and negative bias. Results from this study suggest that the greater level of importance educators’ place on the clinical judgment skill domain of memory bias, the better students are prepared upon graduation in such skill area. Likewise, the higher level of importance educators’ place on the clinical judgment skill domains of negative bias and cognitive complexity, the better students were found to be prepared in the negative bias skill domain. In light of these findings, educators should enhance their knowledge-base in the clinical judgment skill domains of memory bias, cognitive complexity, and negative bias. Further, educators’ should be able to clearly articulate why such skill areas are important for rehabilitation counseling practice and provide instruction of these important skill areas to their students. Doctoral and continuing education programs should provide clinical pedagogical training of a cognitively complex approach and EBP process. Findings of the present study that show proficiency in teaching a cognitively complex approach and EBP process as being highly correlated with perceived student preparedness across a number of clinical judgment skill domains was noteworthy. In light of such findings, there appears to be considerable potential in providing clinical pedagogical training for rehabilitation counselor educators in teaching a cognitively complex approach and EBP process (See Table 7). This type of clinical pedagogical training for educators may contribute to their continued learning and development and has potential to translate into greater student preparedness across a number of clinical judgment skill domains (i.e., scientific attitude, memory bias, cognitive complexity, and cultural bias). 116 In addition, the clinical training research has shown positive student learning outcomes using instructional methods that focus on increasing counseling students’ cognitive complexity (Fong et al., 1997; Duys & Hedstrom, 2000; Little et al., 2005; Meier, 1999). The importance of implementing cognitively complex counselor training models to enhance counseling students’ clinical judgment skills is well supported in the counseling literature across professional disciplines (Spengler et al., 1995; Owen & Lindley, 2010; Morran et al., 1995; Meier, 2003; Bellini & Rumrill, 1999). Further, such clinical education of educators may also effectively address the emerging trend in rehabilitation counselor education to educate rehabilitation counselors in EBP (Leahy & Arokiasamy, 2010). Rehabilitation counseling programs should provide advanced clinical judgment training and supervision post-graduation. It may well be that some of the clinical judgment skill domains in this study may take longer to master than what can be realistically expected in a two or three year 48 to 60 course credit curriculum (Chan et al., 2003). Offering continuing education of clinical judgment skills and supervision post-graduation may have a number of benefits. It can provide an easy avenue for counselors to receive regular and potentially more accurate clinical supervision feedback (Garb, 1989) from educators who have knowledge of the clinical judgment and counselor bias research. Such continuing education may also assist counselors with learning from their clinical experiences and increase their judgment accuracy over time (Spengler et al., 2009). Having clinical judgment training and supervision available may encourage more master’s graduates to pursue counselor licensure following graduation. Counselors working in the field will also have the opportunity to develop more advanced clinical judgment skills that can be tailored to their setting of practice. 117 Teach master’s students the clinical judgment skill domains of a scientific attitude and EBP. The results of current study showed that a scientific attitude and EBP were the only two clinical judgment skill domains where students were reported by educators to have limited preparation. Given the need to demonstrate service effectiveness and accountability in rehabilitation settings (Chan et al., 2008), making clinical decisions that are evidence-based is critically important and top priority in rehabilitation counselor education (Leahy & Arokiasamy, 2010; Shultz et. al., 2007; Shaw et al., 2006). Further, current practice environments reflect rising managed care costs, limited government funding, and need for counselors to demonstrate service effectiveness (Chronister et al., 2008). Consequently, the empirically-validated clinical judgment skill domains of scientific attitude and EBP (See Table 5) of this study can and should be used to address such contemporary challenges within the discipline. Further, such skill domains can assist rehabilitation counselor educators with the design of courses, syllabi and development of lesson plans to effectively prepare students in using an EBP model of practice. Future Research In light of current study’s findings as well its limitations, future research should study CRC’s to learn about their perceptions of level of importance and competencies in clinical judgment skill domains for effective rehabilitation counseling practice. Also, given the potential for a future CORE-CACREP merger (Evenson, Lane, & Nunez, 2012, April), along with the trend within the discipline of moving toward a more mental health counseling focus (Tarvydas, Leahy, & Zanskas, 2009), it may be beneficial to survey counselor educators from CACREP accredited programs. This may enlarge the scope and impact of this research line, and may be used to find differences between educators from CORE and CACREP programs in relation to 118 their perceptions of training clinical judgment skills. Further, counseling practitioners who hold the credential of NCC, in addition to CRC’s, could also be surveyed in a similar manner to generate a broader baseline of both educator and practitioner perceptions of clinical judgment skills for effective practice. Such research will also be useful for continuing to validate and revise the CJSI as necessary and to develop a solid empirical-base of important clinical judgment skill domains that can be used to inform future accreditation standard revisions and clinical curricula to meet the contemporary needs of counselors across specialty areas of practice. The significant findings of programmatic emphasis of clinical judgment training are considerable and suggest a need for further investigation in this area. Identifying specific factors within a rehabilitation counseling program that influence clinical judgment training emphasis may provide data useful for better understanding how programs can be developed to best support the training of clinical judgment skill domains. The significant finding of clinical setting in the current study also poses additional questions for future research, such as: What are the differences between rehabilitation settings in regard to importance of clinical judgment skills? How often do practitioners use clinical judgment skills across settings? The significant finding of gender and race are interesting and also warrant further investigation. Similar to what has been specified in the rehabilitation education literature (Donnell et al., 2009), future research should investigate the cultural competencies of rehabilitation counselor educators in relation to their proficiency in using and teaching clinical judgment skills for effective rehabilitation counseling practice. Such findings may be used to inform clinical pedagogical curricula for doctoral education programs and continuing education course offerings to enhance educators’ professional development in clinical judgment education. 119 In addition, future research should also consider using another method of EFA to further validate the current study’s factor solutions since subsequent analyses were based on the new factor structure of importance for practice and teaching proficiency scales. Given that CJSI items show considerable cross-loadings across individual factors and that items developed were expected to be related, an alternative EFA procedure other than principle components analysis (e.g., principal axis factoring, maximum likelihood, parallel analysis, etc.) that also uses a rotation method that allows factors to correlate (e.g., oblique rotation) (Raykov & Marcoulides, 2008) appears warranted. Using a different factor analysis method may provide a more valid methodological approach and have greater potential impact to the field. This study’s discussion also poses important questions about the relative importance of clinical judgment skills, as well as, the need for future research that provides a more in-depth analysis of the relationships between teaching proficiency, student preparedness, and importance. Such research can provide rehabilitation counselor education programs and CORE with useful information to further enhance both student learning and clinical training outcomes. Research questions that address this area of inquiry can include: What are the benefits of rank ordering clinical judgment skill domains? Are there differences between clinical judgment skill domains level of importance relative to different populations being served? How does teaching proficiency lead to enhanced student learning outcomes in clinical judgment domains? Conclusion Given the importance of clinical judgment in rehabilitation counseling (Strohmer & Leierer, 2000), prevalence and consequences of rehabilitation counselor biases (Berven & Rosenthal, 1999), and the emerging trend to educate rehabilitation counselors in EBP (Leahy & Arokiasamy, 2010), the explicit teaching of clinical judgment skills that can effectively bridge 120 science and practice to improve clinical judgment accuracy appears critically needed (Spengler et al., 1995). The dearth of clinical judgment training research in rehabilitation counselor education despite the evidence of counselor bias and need for counselor judgment accuracy is of great concern given its central function and importance in rehabilitation counseling practice. To address this gap in the literature and add to the knowledge-base of clinical judgment training research in rehabilitation counselor education, the current exploratory study was conducted to gather data useful for better understanding rehabilitation counselor educators’ perceptions of teaching clinical judgment skills. Results indicated that clinical judgment skills are highly important for effective rehabilitation counseling practice and provide new knowledge of empirically-validated clinical judgment skill domains that warrant consideration for future CORE standard revisions. The significant differences in level of importance, student preparedness, and teaching proficiency in clinical judgment domains relative to rehabilitation counselor educator demographic (i.e., gender, race, and clinical setting) and programmatic characteristics (i.e., program emphasis of clinical judgment training) shed new light on notable factors that merit consideration. Although the significant relationships between teaching proficiency, student preparedness, and importance can be expected, such evidence can be used to instigate new thinking among educators’ in relation to how they select clinical judgment skill domains to teach in the most efficient and effective manner. In conclusion, the findings from this study provides further empirical evidence that supports the importance and need for students to develop skills necessary to overcome their biases to improve their own clinical judgment accuracy and decision making when providing rehabilitation services to clients (Berven, 2004; Strohmer & Leierer, 2000). Further, the data 121 generated from the present study can be used to inform and enhance master’s and doctoral-level rehabilitation counseling clinical training curricula and to address the challenges that come with deciding how to effectively educate rehabilitation counselors in EBP. 122 APPENDICES 123 APPENDIX A: CLINICAL JUDGMENT SKILL INVENTORY (CJSI) PART 1 INSTRUCTIONS FOR COMPLETING THIS INVENTORY This inventory presents a list of clinical judgment skills that can be used to address clinical judgment tasks in rehabilitation counseling practice. For each item, you will be asked to rate its importance for practice and how well you perceive students are prepared upon graduation in the clinical judgment skill area. IMPORTANCE FOR PRACTICE: Importance of this skill for effective rehabilitation counseling practice. SCALE FOR IMPORTANCE FOR PRACTICE 1=not at all important 2=slightly important 3=somewhat important 4=important 5=quite important 6=very important 7=extremely important STUDENT PREPAREDNESS: Overall student preparedness of this skill upon graduation from a rehabilitation counseling master’s degree program. SCALE FOR STUDENT PREPAREDNESS 1=not at all prepared 2=slightly prepared 3=somewhat prepared 4=prepared 5=quite prepared 6=very prepared 7=extremely prepared 124 Clinical Judgment Skills This skill area is associated with counselors’ use of internal dialogues that serve as reminders to question their biases and use of other strategies such as scientific-based methods to improve judgment accuracy. For each item, rate its importance for effective rehabilitation counseling practice and how well you perceive students are prepared in the skill area upon graduation from a rehabilitation counseling master’s degree program. IMPORTANCE FOR PRACTICE STUDENT PREPAREDNESS 1=not at all important 2=slightly important 3=somewhat important 4=important 5=quite important 6=very important 7=extremely important 1=not at all prepared 2=slightly prepared 3=somewhat prepared 4=prepared 5=quite prepared 6=very prepared 7=extremely prepared 1. 2. 3. 4. Utilizes knowledge base of client issues and how they are remedied Carefully documents all salient observations immediately after meeting with clients Implements conscious and continuous self-evaluation of own thinking about clients Focuses interview on multiple dimensions (i.e., client, problem/concern, significant others, culture/environment) 5. Actively questions the quality of counseling relationship and applies strategies to build and/or maintain it 6. Separates from the counseling process to make more objective third-person observations 7. Attends to positive client emotions 8. Critically appraises and applies research literature to practice (e.g., to choose appropriate interventions, to plan assessments) based on client needs and specific context of the case 9. Formulates questions or strategies to test and evaluate multiple hypotheses 10. Combines statistical (e.g., vocational, psychological tests) and clinical prediction techniques (e.g., behavioral assessment methods) 11. Delays final judgments about clients by treating them as tentative decisions and formulations open to revision 12. Openly discusses diversity and multicultural differences 13. Does not become attached to one interpretation of the client data 14. Uses clinical practice guidelines to degree they are supported by empirical research 15. Makes rationale for decision-making explicit (i.e., writes summary, explains it to supervisor or colleague) 16. Examines biases associated with counseling theory orientations 125 Clinical Judgment Skills For each item, rate its importance for effective rehabilitation counseling practice and how well you perceive students are prepared in the skill area upon graduation from a rehabilitation counseling master’s degree program. IMPORTANCE FOR PRACTICE STUDENT PREPAREDNESS 1=not at all important 2=slightly important 3=somewhat important 4=important 5=quite important 6=very important 7=extremely important 1=not at all prepared 2=slightly prepared 3=somewhat prepared 4=prepared 5=quite prepared 6=very prepared 7=extremely prepared 17. Assesses trait and situational factors (i.e., social, economic, and environmental) 18. Periodically reviews case notes and specifies important themes 19. Focuses on client strengths 20. Evaluates own cultural biases based on such categories as gender, ethnic minority status, disability, and social class 21. Takes into account own emotional reactions to clients 22. Re-checks initial case formulation after several sessions 23. Adapts use of counseling style (i.e., qualities and skills) to suit different individuals and cultures 24. Becomes informed about problems treated by other mental health or health professionals 25. Considers probability and base rate data 26. Uses decision aids (i.e., DSM-IV, state-federal vocational rehabilitation eligibility criteria) when making mental health diagnoses or eligibility determinations 27. Weighs empirical results more heavily than own clinical experiences 28. Discards or tentatively accepts hypotheses based on counseling and/or rehabilitation outcome data (i.e., intervention effects, client progress) 29. Builds in a baseline period of client data collection 30. Seeks out supervisory feedback 31. Uses multiple sources of assessment (e.g., client interview, reports from significant other and/or family, situational assessment, interest inventory, etc.) and avoids use of single method 32. Uses reliable and valid instruments in data collection 33. Attends to the details of sessions that initially seem less significant 34. Uses systematic and comprehensive interviews (i.e., use of semi-structured and/or structured interviews) 35. Actively considers and/or discusses both client’s and own expectations for services 126 PART 2 INSTRUCTIONS FOR COMPLETING THIS INVENTORY There are eight (8) additional teaching proficiency questions. You will be asked to rate your own perceived proficiency in teaching selected skills to master’s rehabilitation counseling students within the overall clinical judgment skill domain. YOUR PROFICIENCY: Educators own proficiency in teaching this skill. SCALE FOR YOUR PROFICIENCY 1=not at all proficient 2=slightly proficient 3=somewhat proficient 4=proficient 5=quite proficient 6=very proficient 7=extremely proficient 36. Implements conscious and continuous self-evaluation of own thinking about clients 37. Formulates questions or strategies to test and evaluate multiple hypotheses 38. Critically appraises and applies research literature to practice (e.g., to choose appropriate interventions, to plan assessments) based on client needs and specific context of the case 39. Does not become attached to one interpretation of the client data 40. Considers probability and base rate data 41. Combines statistical (e.g., vocational, psychological tests) and clinical prediction techniques (e.g., behavioral assessment methods) 42. Discards or tentatively accepts hypotheses based on counseling and/or rehabilitation outcome data (i.e., intervention effects, client progress) 43. Delays final judgments about clients by treating them as tentative decisions and formulations open to revision 127 APPENDIX B: DEMOGRAPHIC QUESTIONNAIRE PART 3 Demographic Questionnaire (10 Questions) 44. What is your gender?  Female  Male 45. What is your ethnic/racial background?  African American/Black  Caucasian/White  Asian/Pacific Islander  Native American  Latino  Multiracial  Other (please specify) _______________ 46. What are your current professional certifications/licenses (select all that apply)?  CRC  NCC  CVE  LPC  Other (please specify) _______________ 47. How many years have you been employed as a Rehabilitation Counselor Educator (please specify)? ____ 128 48. What was the major of your highest degree earned?  Clinical Psychology  Counseling Psychology  Rehabilitation Science  Rehabilitation Psychology  Rehabilitation Counseling  Rehabilitation Counselor Education  Counselor Education  Social Work  Other (please specify) _______________ 49. What is your current faculty rank?  Assistant Professor  Associate Professor  Professor  Professor Emeritus  Adjunct Professor  Clinical Instructor  Other (please specify) _______________ 129 50. How many years of clinical experience do you have providing direct client services?  Less than 2 years  2 - 5 years  6 - 10 years  11 - 15 years  16 - 20 years  More than 20 years 51. What setting have you gained the majority of your clinical experience providing direct client services?  State/Federal vocational rehabilitation agency  Private, for profit agency (i.e., workers compensation/insurance agency)  Private, not-for-profit agency  Mental health center  Private practice  Medical center or general hospital  College or university  Other (please specify) _______________ 52. Considering the overall Master’s level Rehabilitation Counseling curriculum at your institution, how much is clinical judgment emphasized?  Not emphasized  Somewhat emphasized  Emphasized  Very emphasized  Extremely emphasized 130 53. How does your current Rehabilitation Counselor Education (RCE) curriculum offer students clinical judgment skill preparation (please select all that apply)?  Provided across all courses  Provided in practicum or internship courses  Provided in assessment course  Provided in counseling skills courses (i.e., individual, group counseling)  Provided in multicultural counseling course  Other (please specify) __________________ 131 APPENDIX C: INFORMED CONSENT INFORMED CONSENT 1. PURPOSE OF RESEARCH: You are being asked to participate as a research participant in this internet-based survey study of rehabilitation counselor educators’ perceptions of teaching clinical judgment skills. You have been selected as a participant in this study because you have been identified as a rehabilitation counselor educator who is currently employed in a master’s level rehabilitation counseling program. Your participation in this study will take about 15-20 minutes of your time. 2. WHAT YOU WILL DO: All that will be required of you is that you take the time to complete this internet-based survey. There are three parts and a total of 53 questions to answer. You can save your selected answers by pushing the next button. In addition, you have the option to save your responses and log out and return to the survey where you left off. However, you will be unable to go back and change your answers once you have submitted them since no identifying information will be included with your responses. 3. POTENTIAL BENEFITS: Your participation in this study may generate data useful for better understanding rehabilitation counselor educators’ perceptions of teaching clinical judgment skills for effective rehabilitation counseling practice. Further, it is anticipated that the findings from this study have the potential to both inform and enhance the clinical training curricula of master’s and doctoral-level rehabilitation counseling programs. 4. POTENTIAL RISKS: There are no foreseeable risks associated by participating in this study. 5. PRIVACY AND CONFIDENTIALITY: The data for this project will be kept confidential. All data will be collected on the web using the web-survey service program, Survey Monkey. Access to Survey Monkey will be password protected. Only the researchers will have access to the password to Survey Monkey needed to access the data.The researchers will maintain your privacy throughout the research process by ensuring you are automatically assigned a Case ID number that remains blind to the researchers. No identifying information will be stored with the data. The only indentifying information will be your email address that is linked to your survey on Survey Monkey and will be used only for sending email reminders to you to complete this survey. All the data will be imported and stored on one of the researcher’s personal computer and software programs where data will be entered and stored for data analysis. The personal computer used and data files created will be password protected to ensure protection of all participant data. Only the researchers and Michigan State University Institutional Review Board will have access to the data. The results of this study may 132 be published or presented at professional meetings, but the identities of all research participants will remain anonymous. 6. YOUR RIGHTS TO PARTICIPATE, SAY NO, OR WITHDRAWAL: Participation in this research project is completely voluntary. You have the right to say no. You may change your mind at any time and withdraw. There are no consequences of withdrawal or incomplete participation. You may choose not to answer specific questions or to stop participating at any time. 7. COSTS AND COMPENSATION FOR BEING IN THE STUDY: There are not costs to you to participate in this study other than the value you place on your time. You will not receive money or any other form of compensation for participating in this study. If you have any questions about this study, such as scientific issues, how to do any part of it, or prefer an alternative method for taking this survey (e.g., by phone or hard copy), please contact the researcher, Bryan S. Austin, Michigan State University, 455 Erickson Hall, East Lansing, MI 48824, baustin156@gmail.com, 503-816-9250. If you have any questions about your role and rights as a research participant, you can also contact responsible project investigator, Dr. Michael Leahy, at 517-432-0605, or e-mail leahym@msu.edu, or if would like to register a complaint about this study, you may contact, anonymously if you wish, the Director of MSU’s Human Research Protection Programs, Judy McMillan, at 517-355-2180, FAX 517-432-4503, or e-mail irb@msu.edu, or regular mail at: 207 Olds Hall, MSU, East Lansing, MI 48824-1047. By clicking the “NEXT” button below, it means that you voluntarily agree to participate in this research study. *Please print a copy of this form to keep before proceeding.* THANK YOU 133 REFERENCES 134 REFERENCES American Counseling Association. (2011). 20/20: A vision for the future of counseling. Retrieved on December 7, 2011 from http://www.counseling.org/20-20/definition.aspx Arnoult, L. H., & Anderson, C. A. (1988). Identifying and reducing causal reasoning biases in clinical practice. Reasoning, inferences, and judgment (pp. 209-232). New York, NY: The Free Press. Athanasou, J. A., & Kaufmann, E. (2010). Analysing the expert judgment of a rehabilitation counsellor: A case study. Australian Journal of Rehabilitation Counseling, 16(2), 73-84. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Arkes, H. R. (1981). Impediments to accurate clinical judgment and possible ways to minimize their impact. Journal of Consulting and Clinical Psychology, 49(3), 323-330. Bell, I., & Mellor, D. (2009). Clinical judgements: Research and practice. Australian Psychologist, 44(2), 112-121. Bellini, J. (2002). Correlates of multicultural counseling competencies of vocational rehabilitation counselors. Rehabilitation Counseling Bulletin, 45(2), 66-75. Bellini, J. L., & Rumrill, P. D. (1999). Implementing a scientist-practitioner model of graduatlevel rehabilitation counselor education: Guidelines for enhancing curricular coherence. Rehabilitation Education, 13(3), 261-275. Berven, N. L. (2004). Assessment. In T.F. Riggar, & D. R. Maki (Eds.), Handbook of Rehabilitation counseling (pp. 199-217). New York, NY: Springer Publishing Company, Inc. Berven, N. L. (2008). Assessment interviewing. In B. Bolton, & R. Parker (Eds.), Handbook of measurement and evaluation in rehabilitation (pp. 243-261). Austin, TX: Pro-Ed. Berven, N. L. (1979). Training needs of state agency rehabilitation counselors. Rehabilitation Counseling Bulletin, April, 320-329. Berven, N. L., & Scofield, M. E. (1980). Evaluation of clinical problem-solving skills through standardized case-management simulations. Journal of Counseling Psychology, 27(2), 199-208. Borders, L. D., Bloss, K. K., Cashwell, & Rainey, L. M. (1994). Helping students apply the scientist-practitioner model: A teaching approach. Counselor Education and Supervision, 34, 172-178. 135 Broverman, I. K., Broverman, D. M., Clarkson, F. E. (1970). Sex-role stereotypes and clinical judgments of mental health, Journal of Consulting and Clinical Psychology, 34(1), 1-7. Burker, E. J., & Kazukauskas, K. A. (2010). Code of ethics for rehabilitation educators and counselors: A call for evidence-based practice. Rehabilitation Education, 24 (3-4), 101112. Buser, T. J. (2008). Counselor training: Empirical findings and current approaches. Counselor Education and Supervision, 48(2), 86-100. Carkhuff, R. R. (1968). Differential functioning of lay and professional helpers. Journal of Counseling Psychology, 15(2), 117-126. Cartwright, B. Y., & Fleming, C. L. (2010). Multicultural and diversity considerations in the new code of professional ethics for rehabilitation counselors. Journal of Applied Rehabilitation Counseling, 41(2), 20-24. Chwalisz, K. (2003). Evidence-based practice: A framework for twenty-first-century scientistpractitioner training. The Counseling Psychologist, 31(5), 497-528. Chan, F., Bezyak, J., Ramirez, M. R., Chiu, C., Sung, C., & Fujikawa, M. (2010). Concepts, challenges, barriers, and opportunities related to evidence-based practice in rehabilitation counseling. Rehabilitation Education, 24(3-4), 179-190. Chan, F., Leahy, M. J., Saunders, J. L., Tarvydas, V. M., Ferrin, J. M., & Lee, G. (2003). Training needs of certified rehabilitation counselors for contemporary practice. Rehabilitation Counseling Bulletin, 46(2), 82-91. Chan, F., Miller, S. M., Lee, G., Pruett, S. R., & Chou, C. C. (2004). Research. In T. F. Riggar & D. R. Maki (pp. 159-170) Handbook of Rehabilitation Counseling. New York: Springer. Chan, F., Rosenthal, D. A., & Pruett, S. R. (2008). Evidence-based practice in the provision of rehabilitation services. Journal of Rehabilitation, 74(2), 3-5. Chan, F., Shaw, L. R., McMahon, B. T., Koch, L., & Strauser, D. (1997). A model for enhancing rehabilitation counselor-consumer working relationships. Rehabilitation Counseling Bulletin, 41(2), 122-137. Chan, F., Tarvydas, V., Blalock, K., Strauser, D., & Atkins, B. J. (2009). Unifying and elevating rehabilitation counseling through model-driven, diversity-sensitive evidenced-based practice. Rehabilitation Counseling Bulletin, 52(2), 114-119. Chronister, J. A., Lynch, R. T., Chan, F., Rosenthal, D. A., & Cardoso, E. d. S. (2008). The evidence-based practice movement in healthcare: Implications for rehabilitation. Jounal of Rehabilitation, 74(2), 6-15. Cohen, J. (1992). Quantitative methods in psychology: A power primer. Psychological Bulletin, 112(1), 155-159. 136 Commission on Rehabilitation Counselor Certification. (2011). Crc/crcc scope of practice. Retrieved from http://www.crccertification.com/pages/crc_ccrc_scope_of_practice/56.php Commission on Rehabilitation Counselor Certification. (2009). Code of professional ethics for rehabilitation counselors. Retrieved from http://www.crccertification.com/filebin/pdf/CRCCodeOfEthics.pdf Council on Rehabilitation Education. (2011). Standards for Rehabilitation Counselor Education Programs. Retrieved from http://www.corerehab.org/PDF%20Documents/rcestand.05192011.pdf Coduti, W. (2009). Perceived importance, educator proficiency and student preparedness in disability management knowledge domains (Doctoral dissertation). Available from ProQuest Disserations and Theses database. (UMI No. 3381129) Cook, C., Heath, F. , & Thompson, R. L. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60(6), 821-836. Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation, 10(7), 1-9. Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks, CA: SAGE Publications, Inc. Danzinger, P., & Welfel, E. R. (2000). Age, gender and health bias in counselors: An empirical analysis. Journal of Mental Health Counseling, 22(2), 139-149. Dellario, D. J. (1996). In defense of teaching master’s-level rehabilitation counselors to be scientist-practitioners. Rehabilitation Education, 10(2-3), 229-232. DeVellis, R. F. (2003). Scale development: Theory and applications (2nd Ed.). Thousand Oaks, CA: Sage Publications, Inc. Dillman, D. A., Smyth, D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys: The tailored design method (3rd Ed.). Hoboken, NJ: John Wiley and Sons, Inc. Donnell, C. M., Robertson, S. L., & Shannon, C. D. (2009). Multicultural education and training in rehabilitation counseling education programs. Rehabilitation Education, 23(2), 193202. Duys, D. K., & Hedstrom, S. M. (2000). Basic counselor skills training and counselor cognitive complexity. Counselor Education and Supervision, 40, 8-18. 137 Egisdottir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., & Rush, J. D. (2006). The meta-analysis of clinical judgment project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34(3), 341-382. Evenson, T., Lane, F., & Nunez, P. (2012, April). Recent decisions by the aca governing council’s 20/20 workgroup and the impact on graduates from core accredited programs. Paper presentation at the National Rehabilitation Educators Conference, San Francisco, CA. Facione, N. C., & Facione, P. A. (2008). Critical thinking and clinical judgment. Critical thinking and clinical reasoning in the health sciences: A teaching anthology (pp. 1-13). Milbrae, CA: The California Academic Press. Falvey, J. E., Bray, T. E., Hebert, D. J. (2005). Case conceptualization and treatment planning: Investigation of problem-solving and clinical judgment. Journal of Mental Health Counseling, 27(4), 348-372. Faust, D. (1986). Research on human judgment and its application to clinical practice. Professional Psychology: Research and Practice, 17(5), 420-430. Field, A. (2009). Discovering statistics using spss (3rd ed.). Thousand Oaks, CA: SAGE Publications Ltd. Fong, M. L., Borders, L. D., Ethington, C. A., Pitts, J. H. (1997). Becoming a counselor: A longitudinal study of student cognitive development. Counselor Education and Supervision, 37, 100-113. Gambrill, E. (2005). Critical thinking in clinical practice: Improving the quality of judgments and decisions (2nd Ed.). Hoboken, NJ: Wiley & Sons, Inc. Garb, H. N. (2005). Clinical judgment and decision making. Annual Review of Clinical Psychology, 1, 67-89. Garb, H. N. (1989). Clinical judgment, clinical training, and professional experience. Psychological Bulletin, 105(3), 387-396. Garb, H. N. (1998). Studying the clinician: judgment research and psychological assessment. Washington, DC: American Psychological Association. Garb, H. N. (1996). The representativeness and past-behavior heuristics in clinical judgment. Professional psychology: Research and practice, 27(3), 272-277. Garb, H. N. (2010). In J. E. Maddux & J. P. Tangey (Eds.). The social psychology of clinical judgment. Social foundations of clinical psychology (pp. 297-311). New York, NY: Guilford Publications, Inc. 138 Garb. H. N., & Grove, W. M. (2005). On the merits of clinical judgment. American Psychologist, September, 658-659. Garland, R. (1991). The mid-point on a rating scale: Is it desirable? Marketing Bulletin, 2(3), 6670. Goldberg, L. R. (1968). Simple models or simple processes? Some research on clinical judgments. American Psychologist, 23, 483-496. Goodheart, C. D. (2006). Evidence, endeavor, and expertise in psychology practice. Evidencebased psychotherapy: Where practice and research meet (pp. 37-61). Washington, DC: American Counseling Association. Goodheart, C. D., Kazdin, A. E., & Sternberg. (2006). Evidenced-based psychotherapy: Where practice and research meet. Washington, DC: American Psychological Association. Granello, D. H. (2000). Encouraging the cognitive development of supervisees: Using bloom’s taxonomy in supervision. Counselor Education and Supervision, 40, 31-46. Granello, P. F., & Granello, D. H. (1998). Training counseling students to use outcome research. Counselor Education and Supervision, 37, 224-237. Graves, W. H. (1992). Participatory research: A partnership among individuals with disabilities, rehabilitation professionals, and rehabilitation researchers. Rehabilitation Education, 6, 221-224. Groth-Marnat, G. (2009). Handbook of psychological assessment (5th Ed). Hoboken, NJ: Wiley & Sons, Inc. Harding, T. P. (2007). Clinical decision-making: How prepared are we? Training and Education in Professional Psychology, 1(2), 95-104. Hastie, R., & Dawes, R. M. (2001). Rational choice in an uncertain world: The psychology of judgment and decision making. Thousand Oaks, CA: Sage Publications, Inc. Haverkamp, B. E. (1994). Cognitive bias in the assessment phase of the counseling process. Journal of Employment Counseling, 31, 155-167. Haverkamp, B. E. (1993). Confirmatory bias in hypothesis testing for client-identified and counselor self-generated hypotheses. Journal of Counseling Psychology. 40(3), 303-315. Heppner, P. P., Kivlighan, D. M., & Wampold, B. E. (1999). Research design in counseling (2nd Ed.). Belmont, CA: Wadsworth Publishing Company. Holloway, E. L., & Wolleat, P. L. (1980). Relationship of counselor conceptual level to clinical hypothesis formation. Journal of Counseling Psychology, 27(6), 539-545. 139 Hoshmand, L. L. (1991). Clinical inquiry as scientific training. The Counseling Psychologist, 19(3), 431-453. Howard, M. O., Allen-Meares, P., & Ruffolo, M. C. (2007). Teaching evidence-based practice: Strategic and pedagogical recommendations for schools of social work. Research on Social Work Practice, 17(5), 561-568. Ivey, A.E., & Ivey, M.B. (2008). Essentials of Intentional Interviewing: Counseling in a Multicultural World (1st Ed.) Belmont, CA: Thomson Brooks/Cole. James, J. W., & Haley, W. E. (1995). Age and health bias in practicing clinical psychologists. Psychology and aging, 10(4), 610-616. Janikowski, T. P., Berven, N. L., Meixelsperger, M. K., & Roedl, K. E. (1989). A computerbased case simulation to assess skill in predicting client behavior. Rehabilitation Counseling Bulletin, 33(2), 127-139. Johnston, M. V., Vanderheiden, G. C., Farkas, M. D., Rogers, E. S., Summers, J. A., & Westbrook, J. D. (2009). The challenge of evidence in disability and rehabilitation research and practice. A position paper. National Center for the Dissemination of Disability Research. Kazdin, A. E. (2006). Assessment and evaluation in clinical practice. Evidence-based psychotherapy: Where practice and research meet (pp. 153-177). Washington, DC: American Counseling Association. Keller, M. B., Lavori, P. W., Klerman, G. L., Andreasen, N. C., Endicott, J., Coryell, W., Fawcett, J., Rice, J. P., & Hirschfeld, R. M. (1986). Low levels and lack of predictors of somatotherapy received by depressed patients. Archives of General Psychiatry, 43, 458466. Kelley, L. P., Blashfield, R. K. (2009). An example of psychological science’s failure to selfcorrect. Review of General Psychology, 13(2), 122-129. Kiener, M., & Koch, L. (2009). Action research in rehabilitation counseling. Journal of Applied Rehabilitation Counseling, 40(3), 19-26. Kline, R. B. (2011). Fundamental concepts. Principles and practice of structural equation modeling (3rd Ed.) (pp. 19-45). New York, NY: The Guilford Press. Koch, L. C., Arhar, J. M., & Wells, L. M. (2000). Educating rehabilitation counseling students in reflective practice. Rehabilitation Education, 14(3), 255-268. Koch, L., Kiener, M. S., Gitchel, D. (2009). Action research in rehabilitation education: Curricular applications. Rehabilitation Education, 23(2), 205-214. Kosciulek, J. F. (2004). Empowering people with disabilities through vocational rehabilitation counseling. American Rehabilitation, 28(1), 40-47. 140 Kosciulek, J. F. (2010). Evidence-based rehabilitation counseling practice: A pedagogical imperative. Rehabilitation Education, 24(3-4), 205-212. Kosciulek, J. F. (1999). The consumer-directed theory of empowerment. Rehabilitation Counseling Bulletin, 42(3), 196-213. Leahy, M. J., & Arokiasamy C. V. (2010). Prologue: Evidence-based practice research and knowledge translation in rehabilitation counseling. Rehabilitation Education, 24(3-4), 173-176. Leahy, M. J., Chan, F., & Saunders, J. L. (2003). Job functions and knowledge requirements of certified rehabilitation counselors in the 21st century. Rehabilitation Counseling Bulletin, 46(2), 66-81. Leahy, M. J., Muenzen, P., Saunders, J. L., & Strauser, D. (2009). Essential knowledge domains underlying effective rehabilitation counseling practice. Rehabilitation Counseling Bulletin, 52(2), 95-106. Leahy, M. J., Szymanski, E. M., & Linkowski, D. C. (1993). Knowledge importance in rehabilitation counseling. Journal of Applied Rehabilitation Counseling, 24(4), 36-45. Leung, P. (1987). Rehabilitation counselor education or rehabilitation technical training? Rehabilitation Education, 1, 29-33. Lichtenberg, J. W. (1997). Expertise in counseling psychology: A concept in search of support. Educational Psychology Review, 9(3), 221-238. Little, C., Packman, J., Smaby, M. H., Maddux, C. D. (2005). The skilled counselor training model: Skills acquisition, self-assessment, and cognitive complexity. Counselor Education and Supervision, 44, 189-200. Loftus, E. F. (1993). The reality of repressed memories. American Psychologist, 48(5), 518-537. Lopez, S. R. (1989). Patient variable biases in clinical judgment: Conceptual overview and methodological considerations. Psychological Bulletin, 106(2), 184-203. Lustig, D. C. (1996). Clinical judgment in vocational assessment. Vocational Evaluation and Work Adjustment Bulletin, 29(3), 59-63. Lundervold, D. A., & Belwood, M. F. (2000). The best kept secret in counseling: Single-case (n=1) experimental design. Journal of Counseling and Development, 78(1), 92-102. McArthur, C. (1954). Analyzing the clinical process. Journal of Counseling Psychology, 1(4), 203-208. McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. New York, NY: The Guilford Press. 141 Meeks, S. (1990). Age bias in the diagnostic decision-making behavior of clinicians. Professional Psychology: Research and Practice, 21(4), 279-284. Meier, S. T. (2003). Bridging case conceptualization, assessment, and intervention. Thousand Oaks, CA: Sage Publications, Inc. Middleton, R. A., Rollins, C. W., Sanderson, P. L., Leung, P., Harley, D. A., Ebener, D., & LealIdrogo, A. (2000). Endorsement of professional multicultural rehabilitation competencies and standards: A call to action. Rehabilitation Counseling Bulletin, 43(4), 219-245. Moran, D. J., & Tai, W. (2001). Reducing biases in clinical judgment with single subject treatment design. The Behavior Analyst Today, 2(3), 196-203. Morran, D. K., Kurpius, D. J., Brack, C. J., & Brack, G. (1995). A cognitive-skills model for counselor training and supervision. Journal of Counseling and Development, 73, 384389. Morrow, K. A., & Deidan, C. T. (1992). Bias in the counseling process: How to recognize and avoid it. Journal of Counseling and Development, 70(5), 571-577. Mundry, R., & Nunn, C. (2009). Stepwise model fitting and statistical inference: Turning noise into signal pollution. American Naturalist, 173(1), 119-123. Nisbett, R., & Ross, L. (1980). Human inference: Strategies and shortcomings of human judgment. Englewood Cliffs, NJ: Prentice-Hall. Oskamp, S. (1965). Overconfidence in case-study judgments. Journal of Counseling Psychology, 29(3), 261-265. Owen, J. (2008). The nature of confirmatory strategies in the initial assessment process. Journal of Mental Health Counseling, 30(4), 362-374. Owen, J., & Lindley, L. D. (2010). Therapists’ cognitive complexity: Review of theoretical models and development of an integrated approach for training. Training and Education in Professional Psychology, 4(2), 128-137. Patterson, C. H. (2004). Do we need multicultural competencies? Journal of Mental Health Counseling, 26(1), 67-73. Pepinsky, H. B., & Pepinsky, P. N. Counseling theory and practice. New York, NY: The Ronald Press Company. Peterson, D. B. (2000). Clinical problem solving in micro-case management: Computer-assisted instruction for information-gathering strategies in rehabilitation counseling. Rehabilitation Counseling Bulletin, 43(2), 84-96. Pilipis, L. A. (2011). Meta-analysis of the relation between mental health professionals’ clinical and educational experience and judgment accuracy: Review of clinical judgment 142 research from 1997 to 2010 (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3438776). Power, P. (2006). A guide to vocational assessment (4th Ed.). Austin, TX: Pro-ed, Inc. Raykov, T., & Marcoulides, G. A. (2008). An introduction to applied multivariate analysis. New York, NY: Routledge. Richardson, T. Q., & Molinaro, K. L. (1996). White counselor self-awareness: A prerequisite for developing multicultural competence. Journal of Counseling and Development, 74, 238242. Rosenhan, D. L. (1973). On being sane in insane places. Science, 179, 250-258. Rosenthal, D. A. (2004). Effects of client race on clinical judgment of practicing european american vocational rehabilitation counselors. Rehabilitation Counseling Bulletin, 47(3), 131-141. Rosenthal, D. A., & Berven, N. L. (1999). Effects of client race on clinical judgment. Rehabilitation Counseling Bulletin, 42(3), 243-264. Rosenthal, D. A., & Kosciulek, J. F. (1996). Clinical judgment and bias due to client race or ethnicity: An overview with implications for rehabilitation counselors. Journal of Applied Rehabilitation Counseling, 27(3), 30-36. Rubin, S. E., & Roessler, R. T. (1980). Diagnostic and planning guides for the vocational rehabilitation process. In B. Bolton, & D. W. Cook (Eds.), Rehabilitation client assessment (pp. 257-265). Baltimore, MD: University Park Press. Rybarczyk, B., Haut, A., Lacey, R. F., Foff, L. F., & Nicholas, J. J. (2001). A multifactorial study of age bias among rehabilitation professionals. Archives of Physical Medicine Rehabilitation, 82, 625-632. Sackett, D. L. (1997). Evidence-based medicine. Seminars in Perinatology, 21(1), 3-5. Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: what it is and what it isn’t. British Medical Journal, 312, 7172. Saxon, J. P., & Spitznagel, R. J. (1992). Age bias: An intervening variable in vocational rehabilitation feasibility decision-making? Journal of Applied Rehabilitation Counseling, 23(1), 22-26. Schalock, R. L., & Luckasson, R. (2005). Clinical Judgment. Washington, DC: American Association on Mental Retardation. Scheffer, J. (2002). Dealing with missing data, Res. Lett. Inf. Math. Sci., 3, 153-160. 143 Schon, D. A. (1983). The reflective practitioner. USA: Basic Books, Inc. Schultz, J. C., Koch, L. C., & Kontosh, L. G. (2007). Establishing rehabilitation research priorities for the national council on rehabilitation education. Rehabilitation Education, 21(3), 149-158. Sexton, T. L. (1999). Evidence-based counseling: Implications for counseling practice, preparation, and professionalism. ERIC Digest. Retrieved from http://www.eric.ed.gov/PDFS/ED435948.pdf Sexton, T. L. (2000). Reconstructing clinical training: In pursuit of evidence-based clinical training. Counselor Education and Supervision, 39(4), 218-227. Sexton T. L., Whiston S. C., Bleuer J. C., & Walz, G. R. (1997). Integrating outcome research into counseling practice and training. Alexandria, VA: American Counseling Association. Shaw, L. R., Leahy, M. J., Chan, F., & Catalano, D. (2006). Contemporary issues facing rehabilitation counseling: A delphi study of the perspectives of leaders of the discipline. Rehabilitation Education, 20(3), 163-178. Slovic, P., Finucane, M., Peters, E., & MacGregor, D. G. (2002). The affect heuristic. Heuristics and biases (pp. 397-420). Cambridge, UK: Cambridge University Press. Spengler, P. M., & Strohmer, D. C. (1994). Clinical judgmental biases: The moderating roles of counselor cognitive complexity and counselor client preferences. Journal of Counseling Psychology, 41(1), 8-17. Spengler, P. M., Strohmer, D. C., Dixon, D. N., & Shivy, V. A. (1995). A scientist-practitioner model of psychological assessment: Implications for training, practice and research. The Counseling Psychologist, 23(3), 506-534. Spengler, P. M., Strohmer, D. C., & Prout, H. T. (1990). Testing the robustness of the diagnostic overshadowing bias. American Journal on Mental Retardation, 95(2), 204-214. Spengler, P. M., White, M. J., Egisdottir, S., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G. R., & Rush, J. D. (2009). The meta-analysis of clinical judgment project: Effects of experience on judgment accuracy. The Counseling Psychologist, 37(3), 350-399. Straus, S. E., Glasziou, P., Richardson, W. S., & Haynes, R. B. (2011). Evidence-based medicine: How to practice and teach it (4th Ed.). Edinburgh, UK: Churchhill Livingstone. Stricker, G., & Trierweiler, S. J. (1995). The local clinical scientist: A bridge between science and practice. American Psychologist, 50(12), 995-1002. 144 Strohmer, D. C., Pellerin, M. F., & Davidson, K. J. (1995). Rehabilitation counselor hypothesis testing: The role of negative information, client disability, and counselor experience. Rehabilitation Counseling Bulletin, 39(2), 82-93. Strohmer, D. C., & Shivy, V. A. (1994). Bias in counselor hypothesis testing: Testing the robustness of counselor confirmatory bias. Journal of Counseling and Development, 73, 191-197. Strohmer, D. C., Shivy, V. A., & Chiodo, A. L. (1990). Information processing strategies in counselor hypothesis testing: The role of selective memory and expectancy. Journal of counseling psychology, 37(4), 465-472. Strohmer, D. C., & Leierer, S. J. (2000). Modeling rehabilitation counselor clinical judgment. Rehabilitation Counseling Bulletin, 44(1), 3-9. Stoltenberg, C. D., & McNeill, B. W. (2010). IDM Supervision: An integrative developmental model for supervising counselors and therapists (3rd Ed.). New York, NY: Routledge. Sue, D. W., & Sue, D. (2008). Counseling the culturally diverse: Theory and practice (5th Ed.). Hoboken, N. J: John Wiley and Sons, Inc. Tanner, C. A. (2008). Clinical judgment and evidence-based practice: Toward pedagogies of integration. Journal of Nursing Education, 47(8), 335-336. Tarvydas, V. M., Leahy, M. J., & Zanskas, S. A. (2009). Judgment deferred: reappraisal of rehabilitation counseling movement toward licensure parity. Rehabilitation Counseling Bulletin, 52(2), 85-94. Tarvydas, V., Addy, A., & Fleming, A. (2010). Reconciling evidenced-based research practice with rehabilitation philosophy, ethics and practice: From dichotomy to dialectic. Rehabilitation Education, 24(3-4), 191-204. Taylor, A. (2011). Using the GLM procedure in spss. Department of Psychology, Macquarie University. Retrieved from http://www.psy.mq.edu.au/psystat/documents/GLMSPSS.pdf Thorndike, R. M. (2005). Measurement and evaluation in psychology and education (7th Ed.). Upper Saddle River, NJ: Pearson Education, Inc. Trierweiler, S. J. (2006). Training the next generation of psychologist clinicians: Good judgment and methodological realism at the interface between science and practice. Evidencebased psychotherapy: Where practice and research meet (pp. 211-238). Washington, DC: American Counseling Association. Trochim, M. K. (2006). Research methods knowledge base. Retrieved on December 7, 2011 from http://www.socialresearchmethods.net/kb/sampnon.php 145 Trusty, J., Thompson, B., & Petrocelli, J. V. (2004). Practical guide for reporting effect size in quantitative research in the journal of counseling and development, Journal of Counseling and Development, 82, 107-110. Turk, D. C., & Salovey, P. (1988). Reasoning, inference, and judgment, in clinical psychology. New York, NY: The Free Press. Tversky, A., & Kahneman, D. (1982). Judgment under uncertainty: Heuristics and biases. In. D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 3-20). Cambridge, LN: Cambridge University Press. Wampold, B. E., & Bhati, K. S. (2004). Attending to the omissions: A historical examination of evidence-based practice movements. Professional Psychology: Research and Practice, 35(6), 563-570. Whiston, S. C., & Coker, J. K. (2000). Reconstructing clinical training: Implications for research. Counselor Education and Supervision, 39(4), 228-253. Wright, K. B. (2005). Researching internet-based populations: Advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. Journal of Computer-Medicated Communication, 10(3), article 11. http://jcmc.indiana.edu/vol/issue3/wright.html Zanskas, S. A. (2007). Consultation competencies in rehabilitation counselor education: A mixed methods investigation (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3282236) 146