TEACHING THE WORKING ALLIANCE THROUGH SYSTEMATIC EVALUATION OF 

PROFESSIONAL DISPOSITIONS IN COUNSELOR EDUCATION 

ASSESSMENT AS GROWTH: 

 

By 

 

Allison Levine 

 

 

 

 

 

 

 

 

 

A DISSERTATION 

Submitted to 

Michigan State University 

in partial fulfillment of the requirements 

for the degree of 

 

Rehabilitation Counselor Education -- Doctor of Philosophy 

 

2018 

 

 

 
 
 

 

 

 

 

 

 

 

ABSTRACT 

ASSESSMENT AS GROWTH: 

TEACHING THE WORKING ALLIANCE THROUGH SYSTEMATIC EVALUATION OF 

PROFESSIONAL DISPOSITIONS IN COUNSELOR EDUCATION 

 

By 

 

Allison Levine 

 

The 2016 CACREP Standards call for the formal assessment of student professional dispositions 

throughout a counselor education program. This is a complex and often unwelcome task for 

faculty given the subjective nature of assessing non-academic competencies. Furthermore, 

considering the ethical and legal implications of this task, an empirically-based instrument would 

benefit faculty in counselor education programs in a variety of ways including ensuring due 

process during evaluations. Using a two-phase methodology, the current study began with a 

Delphi panel to establish items to be included in an instrument. The second phase involved 

validation of the instrument items using a survey distributed to Certified Rehabilitation 

Counselors (n = 148). Instrument items were rated on Relevance and Utility to assess their 

appropriateness for being included in the instrument. The Assessment as Growth Inventory 

(AGI), was developed, being framed using the working alliance in order to encourage the 

development of student professional dispositions in line with this evidence-based practice. AGI 

comprises four sections, with 48 items (Professional Dispositions – 13, Tasks – 10, Goals – 8, 

and Bonds – 17). Implications for research, counselor education, and clinical practice are 

discussed at length. 

 

 

 

 

 
 
 

 

 

 

 

 

 

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Copyright by 
ALLISON LEVINE 
2018 

 

 

 

 
 
 

 

ACKNOWLEDGEMENTS 

 

The journey I have taken over the last four years has been the most impactful of my life. I 

learned not only professional and practical skills, but also learned a lot about myself in the 

process. The self-discipline, drive, and intellectual skill necessary to reach this milestone seemed 

out of reach and uncertain four years ago, at the outset of my doctoral work. Learning to take 

broad, conceptual ideas and being able to design and conduct a study in pursuit of such answers 

has been one of the most empowering things that I have learned to do in my life. 

 

 

Reaching this point, and achieving these goals did not come without help from many 

people that I am very lucky to be surrounded by. First and foremost, I most likely never would 

have even gone through with getting my PhD without the support, encouragement, and direction 

from Dr. Nerlich. From introducing me to rehab counseling at Hofstra up through applying to 

MSU and now at the completion of my program, she has been a guiding light, an amazing friend, 

and a supportive colleague. Attending MSU introduced me to even more support, in Drs. Leahy 

and Kosciulek. After my initial shock of having the privilege to work with such impactful 

researchers in the field wore off, I found two people who have always had the time to vet any 

idea, review any paper, or just to catch up. Finally, this would not have been possible without Dr. 

Sung, who has become the voice in my head, helping me to cultivate new ideas, challenge 

myself to be the best, to seek out as many opportunities as possible, and to remain grounded in 

being a lifelong learner. Never have I met someone who is able to wear so many impressive hats 

and yet remains so humble, kind, and truly invested in all of the work her students are doing. 

 

 

In addition to my academic colleagues, my partner Greg has been invaluable throughout 

this dissertation. Thank you for your unwavering support, for bringing me 

snacks/coffee/whatever I need whenever things were getting tough, and accepting the long 

working weekends. I will never be able to thank you enough. 

 

 
 

 

iv 

 
 
 

 

 

 

 

 
 

TABLE OF CONTENTS 

 

LIST OF TABLES ............................................................................................................ vii 

LIST OF FIGURES ......................................................................................................... viii 

CHAPTER ONE ..................................................................................................................1 
Introduction ..........................................................................................................................1 
Background ..........................................................................................................................1 
Struggles for Gatekeepers ....................................................................................................5 
Lack of Formalized Procedures & Assessment Tools .............................................8 
Statement and Significance of the Problem .........................................................................9 

CHAPTER TWO ...............................................................................................................12 
Literature Review ..............................................................................................................12 
Background ........................................................................................................................12 
Defining Gatekeeping ........................................................................................................13 
Components of Gatekeeping ..............................................................................................15 
Terminology ..........................................................................................................15 
Due Process ...........................................................................................................20 
Legal Issues ...........................................................................................................21 
Ethical Issues .........................................................................................................24 
Barriers to Gatekeeping .....................................................................................................26 
Difficulties in Operationalizing: Subjectivity ........................................................26 
Reactive Gatekeeping ............................................................................................27 
Contextual, Professional, and Legal Issues ...........................................................28 
Professional Disposition Development and the Working Alliance ...................................30 
Working Alliance: An Overview ...........................................................................30 
WA and Counselor Education ...............................................................................32 
Using Performance Appraisals in Counselor Education....................................................36 
Theoretical Framework for PDD .......................................................................................39 
Summary ............................................................................................................................42 

CHAPTER THREE ...........................................................................................................43 
Method ...............................................................................................................................43 
Phase 1 ...............................................................................................................................43 
Study Design: Phase 1 ...........................................................................................44 
Procedures: Phase 1 ...............................................................................................45 
Measures: Phase 1 ..................................................................................................51 
Data Analysis: Phase 1 ..........................................................................................54 
Phase 2 ...............................................................................................................................55 
Study Design: Phase 2 ...........................................................................................56 
Sampling: Phase 2 ..................................................................................................56 
Procedures: Phase 2 ...............................................................................................59 

v 

 
 
 

 

 

 

 
 

Measures: Phase 2 ..................................................................................................60 
Data Analysis: Phase 2 ..........................................................................................61 
Summary ............................................................................................................................62 

CHAPTER FOUR .............................................................................................................63 
Results ................................................................................................................................63 
Phase 1: Round 1 ...................................................................................................63 
Phase 1: Round 2 ...................................................................................................67 
Phase 1: Round 3 ...................................................................................................74 
Phase 2 ...............................................................................................................................81 
Qualitative Comments ...........................................................................................88 
Summary ............................................................................................................................89 

CHAPTER FIVE ...............................................................................................................91 
Discussion ..........................................................................................................................91 
Summary of Findings ............................................................................................91 
Discussion of the Findings .....................................................................................94 
Limitations of the Study ......................................................................................101 
Implications for Research ....................................................................................102 
Implications for Counselor Education .................................................................105 
Implications for Clinical Practice ........................................................................106 
Conclusion .......................................................................................................................107 

APPENDICES .................................................................................................................108 
APPENDIX A: INTERNAL REVIEWBOARD (IRB) APPROVAL .................109 
APPENDIX B: EMAIL INVITATION TO DELPHI PANEL ...........................110 
APPENDIX C: EXPERT PANEL DEMOGRAPHICS SURVEY .....................112 
APPENDIX D: DELPHI ROUND 1: EMAIL ....................................................114 
APPENDIX E: DELPHI ROUND 1: INSTRUCTIONS .....................................116 
APPENDIX F: DELPHI ROUND 2 INSTRUCTIONS ......................................117 
APPENDIX G: DELPHI ROUND 3 INSTRUCTIONS .....................................118 
APPENDIX H: INFORMED CONSENT/INVITE FOR PILOT STUDY .........119 
APPENDIX I: ASSESSMENT AS GROWTH INVENTORY ...........................121 
APPENDIX J: CRCC CEU COMPLETION FORM ..........................................125 
 
REFERENCES ................................................................................................................127 
 

 

 

vi 

 
 
 

 
 

 

 
 

LIST OF TABLES 

Table 3.1: Summary Table of the Steps, Phases, and Activities Involved in the Execution 
of a Delphi Method with Three Rounds (Vázquez-Ramos et al., 2007)............................... 
 
Table 3.2: Participant Demographic Characteristics.............................................................  58 

 
46 

Table 4.1: Round 2: Professional Dispositions (Descending by mean Relevance)..............  68 

 

Table 4.2: Round 2: Tasks (Descending by mean Relevance)..............................................  70 

 

 

Table 4.3: Round 2: Goals (Descending by mean Relevance)..............................................  71 

Table 4.4: Round 2: Bonds (Descending by mean Relevance).............................................  73 

 

Table 4.5: Round 3: Professional Dispositions (Descending by mean Relevance)..............  75 

Table 4.6: Round 3: Tasks (Descending by mean Relevance)..............................................  77 

Table 4.7: Round 3: Goals (Descending by mean Relevance)..............................................  78 

 

Table 4.8: Round 3: Bonds (Descending by mean Relevance).............................................  79 

 

 

 

 

 

vii 

 
 
 

 
 
 

 

 

 
 

LIST OF FIGURES 

Figure 3.1: Round 2 Survey..................................................................................................  53 

Figure 3.2: Round 3 Survey..................................................................................................  54 

Figure 4.1: Professional Disposition Relevance Score Frequency Histogram......................  
 
Figure 4.2: Task Relevance Score Frequency Histogram.....................................................  82 

81 

Figure 4.3: Goal Relevance Score Frequency Histogram.....................................................  83 

Figure 4.4: Bond Relevance Score Frequency Histogram.................................................... 
 
Figure 4.5: Professional Disposition Utility Score Frequency Histogram............................ 
 
Figure 4.6: Task Utility Score Frequency Histogram........................................................... 
 
Figure 4.7: Goal Utility Score Frequency Histogram...........................................................  86 

85 

84 

85 

Figure 4.8: Bond Utility Score Frequency Histogram.......................................................... 
 

87 

 

 

 

 

 

 

 

 

viii 

 
 
 

 

CHAPTER ONE 

Introduction 

Counselor education programs strive to produce graduates who will engage in sound 

ethical practice, and who will facilitate positive outcomes in their future clients. The working 

alliance has been demonstrated as an important factor in the ability to facilitate such outcomes, 

even regardless of a counselor’s chosen theoretical orientation (Wampold, 2011). The 

accrediting body of counselor education program, the Council for Accreditation of Counseling & 

Related Programs (CACREP) provides standards that guide counselor education programs in 

filling the aforementioned goal. This chapter outlines caveats in accreditation standards and 

current practices in counselor education that have contributed to a lack of resources and tools 

which allow faculty to evaluate students’ professional dispositions in a systematic, legally 

defensible manner. 

Background 

 

In counselor education, master’s-level counseling programs have been challenged with 

meeting the standards of the Council on Rehabilitation Education (CORE) and/or the Council for 

Accreditation of Counseling & Related Programs (CACREP), while keeping up with ongoing 

changes in legislation, service delivery systems, and practice settings (Kelly, 2011; Leahy, 

Muenzen, Saunders, & Strauser, 2009). Beyond that, counselor education has a unique challenge 

in both preparing students with academic knowledge about counseling, and aiding them in 

switching gears from acquisition to knowledge translation and action: putting the content that has 

been learned into practice— building therapeutic relationships in dynamic and diverse practice 

settings (Connor & Leahy, 2016). Specifically, the working alliance (WA) is an evidence-based 

practice that facilitates client change through the relationship between the client and counselor. 

 
 

1 

 
 
 

WA has been demonstrated to be more predictive of positive treatment outcomes, regardless of 

theoretical orientation or length of treatment, which underscores the significance of this practice 

within counseling (Horvath & Symonds, 1991; Lambert, 1992; Lustig, Strauser, Rice, & Rucker, 

2002; Wampold, 2001). A meta-analysis of WA involving over 200 studies found the aggregate 

correlation between alliance and outcome was approximately 0.27, which supports the earlier 

work of Wampold (2001), identifying that at least 70% of psychotherapeutic effects are due to 

common factors such as WA, empathetic listening, and goal setting (Horvath, Del Re, Flückiger, 

& Symonds, 2011; Wampold, 2015).  

A particularly difficult task in counselor education is that counselor educators are 

expected to “…move beyond assessment of specific counseling skills and content knowledge, 

and to consider how to appropriately monitor and evaluate behaviors and attributes that are 

clinical and interpersonal in nature” (Kelly, 2011, p. 112), meaning that in addition to supporting 

students through standards-driven knowledge acquisition, skill building, and knowledge 

translation, educators must also be assessing students at another level: identifying if they have 

the professional dispositions required to become a counselor, and especially, if they are capable 

of developing WA with clients. The overall process of evaluating student suitability for 

professional practice including counseling skills, content knowledge and professional 

dispositions, is called gatekeeping (Brear, Dorian, & Luscri, 2008).  

Gatekeeping is a complex concept, but it is specifically comprised of three components: 

(1) it involves a gatekeeper (i.e., educators or supervisors) and a student/trainee, (2) the 

evaluative process involved in gatekeeping should be based on specific criteria within a defined 

framework, and (3) the main purpose is protection for the field in which it occurs (Brear et al., 

2008). The specific criteria referred to by Brear et al. (2008) include both academic 

 
 

2 

 
 
 

competencies (content knowledge and clinical skills), as well as professional dispositions, which 

are stipulated primarily via CACREP Standards. Accreditation standards are a primary way in 

which there has been an attempt to systematically assess counselor education students across 

various educational institutions. While gatekeeping and evaluation are typically unwelcome tasks 

in counselor education and supervision, they are absolutely necessary functions for ethical and 

legal practice (Bernard & Goodyear, 2014). Relating to ethical supervision practice: “If 

supervisors come to a conclusion that a supervisee does not meet criteria for endorsement … this 

should never come as a surprise to the supervisee in question” (Glosoff & Matrone, 2010, p. 

252). This applies in the counselor education realm as well, and speaks to a need for proactive, 

transparent student evaluation procedures and instruments that allow educators to facilitate such 

mechanisms. 

Gatekeeping is complex and multifaceted, which contributes to its lack of favor amongst 

those who must do the evaluating. One way that programs have addressed gatekeeping gaps is by 

implementing specific admissions criterion, which typically comprise of academic abilities 

gleaned from submitted documentation, and interpersonal interactions during the interview prior 

to admittance (Ziomek-Daigle & Christensen, 2010). The current study and its ultimate product 

are not intended to be used at the admissions stage; rather it is intended to address the ongoing 

process of gatekeeping within the context of a counselor education program (i.e., throughout 

multiple points in the program, following admission). 

A problem that has arisen in gatekeeping is the evaluation of professional dispositions, 

the nature of which are elusive and subjective. This impedes counselor educators’ abilities and 

likelihoods of evaluating professional dispositions in a systematic way (see Kelly, 2011). When 

factoring in the CACREP Standard requirement to provide data at a minimum of two points 

 
 

3 

 
 
 

throughout a student's program on Key Performance Indicators (KPIs) of professional 

dispositions, this is especially problematic (2016 CACREP Standards, Section 4.G). CACREP 

seeks to assist programs by providing the definition of professional dispositions as “the 

commitments, characteristics, values, beliefs, interpersonal functioning and behaviors that 

influence the counselor’s professional growth and interactions with clients and colleagues”. This 

definition, however, is still lacking in measurable or observable components, or “clear, shared 

and consistent language to represent the different types of problem behaviors” (Forrest, Elman, 

Gizara, & Vacha-Haase, 1999, p. 629).  

 

Many studies have now been conducted in an attempt to remedy this issue via checklists 

or evaluations (see Kerl, Garcia, McCullough, & Maxwell, 2002; Moorhouse, 2008), although 

few have been replicated (Brear et al, 2008; Forrest et al., 1999) and none have been endorsed by 

CACREP. Student evaluation is complex, ongoing, and often unpleasant for faculty (Foster & 

McAdams, 2009). Faculty are evaluating students even prior to admission, via screening 

processes and interviews, however specific student issues such as lack of appropriate 

professional dispositions may not become evident until later in their program, and may not 

necessarily impede a student’s academic abilities. Certainly during the clinical portion of a 

program do these problematic behaviors come to light, however professional dispositions speak 

to more than just clinical skills and academic competence (Kelly, 2011; Lumadue & Duffey, 

1999; Ziomek-Daigle & Christensen, 2010). Although it is clearly so crucial to continually 

identify and evaluate student competence academically, clinically, and in the realm of 

professional dispositions, it has been reported that only 40% of programs used a systematic, 

global assessment of student progress throughout a program (Brear & Dorrian, 2010).  

 
 

 

4 

 
 
 

 

 

 

 

Struggles for Gatekeepers 

Brear & Dorrian (2010) identified three general areas where gatekeeping issues may fall:  

(a) difficulties in delineating and operationalizing the core competencies necessary for  

professional practice and the development of systematic assessment methods; (b) the  

 

degree to which the mechanisms implemented by training programs to facilitate  

gatekeeping incorporate a proactive approach as distinct from a reactive one; and (c)  

diverse contextual, professional, and legal issues. (p. 264) 

Studies that have attempted to address the delineation and operationalization of core 

competencies (including academic, clinical skills, and behaviors) necessary for professional 

practice are typically found in clinical supervision literature (i.e., Fouad et al., 2009; Moorhouse, 

2008; Thielsen & Leahy, 2001). Bernard and Goodyear (2014) indicate that while these lists may 

be somewhat helpful, they are essentially useless unless they are embedded in a well established 

supervisory relationship. Additionally, as outlined above, counselor educators have limited 

guidance from accrediting bodies as to how to systematically assess such competencies. 

Furthermore, there is no current evaluation that directly ties student evaluation to WA. 

 

Accreditation standards indicate that evaluation should be ongoing, and held at multiple 

times throughout a student’s time in a program (CACREP, 2016). What has been largely 

indicated by the literature, however, is that evaluation of professional dispositions has been 

coupled with clinical skills, and are evaluated simultaneously during the internship or practicum 

phase of a student’s program (Moorhouse, 2008). This contributes to the reactive nature of 

gatekeeping processes; students who are in practicum or internship are receiving supervision and 

thus most issues are addressed at that point, after they have come to light in a work-based setting. 

Ideally, in order to alleviate the issue of reactivity, and to be proactive in student evaluation, 

 
 

5 

 
 
 

particularly of professional dispositions, it has been suggested that programs adopt a more 

transparent method for evaluation which begins at admissions and continues throughout a 

student’s time within a program (Foster & McAdams, 2009).  

Another issue related to the reactive nature of gatekeeping is that programs also are less 

likely to focus on a formal or systematic way to track student evaluation until after there has 

been legal action taken as a result of a dismissal or intervention in a student’s program of study 

(Foster & McAdams, 2009). Although there is no doubt that all counselor education programs 

experience problematic students who demonstrate inadequate professional dispositions, or any 

number of other inadequacies, it appears that the majority use ad hoc evaluations, which are 

inconsistently done on an as needed basis. These ad hoc and inconsistent procedures (which may 

also be referred to as ‘non-formalized procedures’) leave programs vulnerable to legal action by 

disgruntled students (Foster & McAdams, 2009; Frame & Stevens-Smith, 1995; Kelly, 2011; 

Kerl et al., 2002; McAdams & Foster, 2007).  

 

While legal issues may seem like low incidence, and low risk, they still serve as a 

deterrent to gatekeeping for faculty, and may not actually be very low incidence, especially as 

our society becomes increasingly litigious (Bernard & Goodyear, 2014; Brear & Dorrian, 2010). 

The history of lawsuits from disgruntled students dismissed from academic programs on the 

basis of “non-academic competencies” stems back to Greenhill v. Bailey (1975), which is the 

first case in which a student’s academic performance was defined by the court as “a student’s 

demonstrated knowledge, technical and interpersonal skills, attitudes, and professional character” 

(Knoff & Prout, 1985, p. 792). This is notable given the inclusion of interpersonal skills and 

professional character, which would align now with our term of professional dispositions, or 

non-academic competencies. Board of Curators of the University of Missouri v. Horowitz 

 
 

6 

 
 
 

(1978), just three years later, is considered the landmark case in the area of gatekeeping, as the 

United States Supreme Court ruled in favor of the inclusion of interpersonal factors as an 

academic factor. The ruling also underscored an issue that is central to the current study: the 

procedural requirements necessary to implement an academic dismissal are much less rigorous in 

comparison to dismissals that are due to factors outside of student academic achievements, such 

as professional dispositions (Frame & Stevens-Smith, 1995). In other words, formalized 

procedures in the evaluation of professional dispositions, or non-academic competencies, are 

crucial to effective, legally defensible, gatekeeping practices. 

Fear of judgement or disagreement among other faculty also plays a role in gatekeeping, 

which can compound the concerns faculty already have as a result of legal vulnerability. Faculty 

may feel that implementing a remedial protocol may cause other faculty members to view them 

as overreacting to a particular situation, which can be especially concerning to new and non-

tenured faculty (Foster & McAdams, 2009). Additionally, disagreement among faculty has been 

identified as the one of the biggest barriers to dismissing a student, second only to fear of 

litigation (Vacha-Haase, Davenport, & Kerewsky, 2004). A common feeling in many programs 

is that problems will self-correct over time, or within the supervisory relationship that the student 

will have with his/her boss once employed (Foster & McAdams, 2009; McAdams, Foster, & 

Ward, 2007). Contextual issues such as these can also contribute to social loafing among faculty, 

or the assumption that someone else will eventually intervene in a particular situation (Brear & 

Dorrian, 2010).  

 

It has been suggested that a way to overcome these issues, and many issues related to 

gatekeeping, is through the development and regular use of formalized procedures (Bhat, 2005; 

Brear & Dorrian, 2010; Kelly, 2011; Kerl, et al. 2002; Lumadue & Duffey, 1999), and to 

 
 

7 

 
 
 

implement a culture of transparency around student evaluations which extends from pre-

admission, through graduation (Foster, Leppma, & Hutchinson, 2014; Foster & McAdams, 2009; 

Ziomek-Daigle & Christensen, 2010). 

Lack of Formalized Procedures & Assessment Tools 

 

The lack of formalized gatekeeping procedures available is the most notable challenge 

cited in gatekeeping literature (Foster, Leppma, & Hutchinson, 2014; Foster & McAdams, 2009; 

Gaubatz & Vera, 2002, 2006; Kelly, 2011). It has also been determined that faculty have, 

historically, preferred to address students’ professional performance issues in an informal 

manner, without documenting what actions were taken to resolve such an issue (Foster & 

McAdams, 2009). Furthermore, many educational programs do not even provide written 

remediation and termination policies for students regarding non-academic reasons other than 

plagiarism, cheating, or sexual/racial harassment; only 29% of programs studied by Brear & 

Dorrian (2010) indicated having such policies in place. Gaubatz and Vera (2002) provide 

additional insight into this difficulty for counselor educators:  

 

 

 

 

 

 

...counselor educators must balance the need to protect clients from impaired practitioners 

against legal and institutional pressures to cautiously identify deficiencies among their  

trainees...however this tension is amplified by the unique nature of the training itself,  

which aims not only to increase the trainee’s skill in conducting therapeutic work but also 

to modify his/her personhood. (p. 295) 

Counselor educators are therefore in a difficult position of being expected to train 

students and assess for academic competencies, but also for “personhood” or “professional 

disposition” or “deficits” not otherwise accounted for or specified in academic standards or 

accreditation requirements. Without appropriate documentation, procedures and interventions, 

 
 

8 

 
 
 

the process of intervening with a problematic student becomes not only troublesome at the 

faculty level, but can leave the faculty, program or university vulnerable to the legal 

consequences outlined above. 

 

A contributing factor here is that faculty have been under the impression that they are 

already adequately intervening with “deficient trainees”, even though 98% of faculty in 

counselor education programs identified that there were currently deficient trainees in their 

programs (Gaubatz & Vera, 2006). This, in combination with resistance to evaluation both 

formal and informal, leads to inconsistencies in student evaluation, and most likely “gate-

slippage”, or the graduation of deficient students who do not receive remediation (Gaubatz & 

Vera, 2002). 

 

Gaubatz & Vera (2002) studied whether formalized gatekeeping procedures impacted the 

likelihood of gate-slippage and found that using more formalized procedures lead to significantly 

lower reports of gate-slipping. The researchers also found that less than 20% of counselor 

training programs actively conducted routine formal reviews of their students, and Bhat (2005) 

reported that 54% of programs in a survey by the American Psychological Society did not have 

formally written guidelines for intervention with problematic students. More recently, Brear and 

Dorrian (2010) found that more than half of counselor educators surveyed reported having 

passed a student in one of their courses who they considered to be unsuitable to practice in the 

field of counseling.  

Statement and Significance of the Problem 

 

Counselor educators are expected to prepare future counselors who will work with 

individuals at extremely vulnerable places in their lives. As such, graduate level programs should 

be equipped to identify students with academic deficiencies, and non-academic, professional 

 
 

9 

 
 
 

disposition issues alike. Evaluation and remediation of such students is imperative in order to 

protect the profession, as well as any future clients of those students. Formalized procedures for 

collecting data and evaluating students’ non-academic performance, or professional dispositions, 

are necessary in order to assist faculty in systematically, objectively, and fairly evaluating 

students in counselor education programs.  

 

An instrument for the purpose of specifically meeting the need of evaluating student 

professional dispositions will serve multiple purposes in counselor education. These include: 

protection against legal backlash, improvement of conversations around student professional 

development (among faculty and with students), as well as adhering to requirements for 

accreditation. The current study sought to create such an instrument which can be embedded into 

counselor education practice, for use with counseling students from admission through 

graduation, remediation, and/or dismissal. 

 

A key way that this instrument differentiates itself from previously established 

competency lists is its inclusion of criteria necessary for the development of the WA. The Delphi 

method was used to develop consensus among rehabilitation and counselor educators, and those 

who have published gatekeeping literature. The experts involved developed and refined items 

regarding professional dispositions as well as items specifically related to the WA, honing in on 

the essence of Connor and Leahy (2016): “the challenge in counselor education is to effectively 

teach the cognitive and affective complexity of WA so that graduates possess entry-level 

competence in establishing therapeutic relationships” (p. 375). 

 

The purpose of this study is to fill two gaps in counselor education via the creation of an 

instrument for faculty use in student evaluation: (1) meeting the CACREP professional 

disposition KPI requirements, as well as (2) providing counselor educators with a tool that 

 
 

10 

 
 
 

fosters transparency in evaluation of professional dispositions and students’ ability to develop 

WA in practice. The resolution of such gaps was driven by the following research questions, to 

be addressed by a two-phase research study:  

1.  What items would be appropriate for faculty to use in the evaluation of 

professional dispositions in master’s level counseling students?  

2.  How much do those items relate to the concepts of WA (i.e., goals, tasks and 

bonds)?  

3.  How useful are those items for counselor educators? 

 

 
 

11 

 
 
 

 

CHAPTER TWO 

Literature Review 

To address the previously stated research questions, it is first necessary to provide a 

comprehensive literature review that synthesizes the concepts and constructs involved in the 

current study. This literature review begins by introducing the concept of gatekeeping in 

counselor education. This includes an in-depth discussion of the components of gatekeeping, 

such as terminology and current issues. This sets the stage for a discussion of barriers to 

gatekeeping, and specifically to the assessment of professional dispositions of counselor 

education students. Finally, with the inclusion of a discussion of how WA relates to professional 

dispositions, it is possible to discuss mechanisms of evaluation, including performance 

appraisals. Using social exchange and goal setting theories, as guided by performance appraisal 

literature, the theoretical framework for the current study is established. 

Background 

Accrediting bodies such as the Council on Rehabilitation Education (CORE) and the 

Council for Accreditation of Counseling and Related Educational Programs (CACREP) exist for 

the purpose of guiding educational programs in preparing future professionals who will be able 

to provide the best services possible in their respective disciplines (CACREP, 2016; CORE, 

2016). Following the merger of CORE and CACREP in 2015, it was determined that CACREP 

would begin to carry out the mission of both organizations as of July 2017, and CORE ceased to 

exist (CORE- CACREP Merger Agreement, 2015). As such, the current study is primarily 

concerned with the current CACREP standards regarding gatekeeping in counselor education, 

and will honor CORE’s history of guiding standards with empirical support. 

 
 

12 

 
 
 

The evaluative roles of counselor educators, and rehabilitation counselor educators, 

serves a protective function for the particular discipline in which a student is being trained for 

(Brear, Dorian, & Luscri, 2008). Knowing that graduates will be working directly with 

vulnerable populations is a specific consideration, and adds a level of urgency to the need for 

gatekeeping in all types of counselor education. Faculty of counselor education programs are one 

of the last lines of defense between a potentially deficient counselor and future clients seeking 

services. In the many roles that faculty occupy, they are required to adhere to accreditation 

standards, university policies, program policies, the Commission on Rehabilitation Counseling 

Certification (CRCC) Code of Ethics, as well as the American Counseling Association (ACA) 

Code of Ethics, making this task complex and multifaceted. 

Defining Gatekeeping 

Given the complexity of gatekeeping, and the numerous stakeholders in the process, it is 

extremely important that all groups are adhering to the same, or at least similar, definitions of 

what gatekeeping means. Several studies have already discussed the issue of a lack of adherence 

to one definition of gatekeeping (Brear et al., 2008; Elman & Forrest, 2007; Forrest et al., 1999; 

Glance, Fanning, Schoepke, Soto, & Williams, 2012). The term gatekeeping, in itself, has a 

troubled history within many groups, and especially within the disability community, from the 

times when “gatekeepers” were those who were perceived as keeping persons with disabilities 

(PWD) away from the services they needed and wanted. This causes the term to carry a negative 

connotation with it, which underscores the importance of providing clear, well-thought and 

operational definitions to be used in gatekeeping literature going forward. Therefore, an essential 

focus of the current study is to alleviate some of the negative associations with gatekeeping, and 

to assist counselor educators in understanding gatekeeping as a mechanism for remediation and 

 
 

13 

 
 
 

support, rather than for dismissal or punishment. It is also important to recognize that, given a 

distaste for the term “gatekeeping”, some papers have chosen to forego the term gatekeeping for 

“evaluation” (i.e., Kerl, Garcia, McCullough, & Maxwell, 2002), which makes synthesis and 

systematic use of one definition even more challenging. Systematic, as defined in the CACREP 

Standards, is “a regular, planned, and comprehensive manner”. 

First and foremost, counselor educators should be familiar with is the definition of 

gatekeeping as provided by CACREP (2015): “the ethical responsibility of counselor educators 

and supervisors to monitor and evaluate an individual’s knowledge, skills, and professional 

dispositions required by competent professional counselors and to remediate or prevent those 

that are lacking in professional competence from becoming counselors”. The importance of 

being aware of the CACREP definition cannot be denied, however, there are certain areas in 

which the definition is lacking for use and application to counselor education practice. Above all, 

it is most important to remain consistent as a “lack of definition and criteria makes it even more 

difficult for counselor educators to address the issue of impairment and stay atop of best 

practices in the field” (Glance, Fanning, Schoepke, Soto, & Williams, 2012, p. 2). 

Brear et al. (2008) provide the following definition of gatekeeping, which goes beyond 

the CACREP definition to include accountability on the part of the gatekeepers (i.e., supervisors 

and counselor educators) to both monitor suitability as well as to identify the evaluative criteria 

to be used in evaluation:  

...the evaluation of student suitability for professional practice. It is a mechanism that 

aims to ensure the health of the profession by controlling access to it. It involves the 

identification of evaluative criteria and process, and the accountability of the gatekeeper 

to apply the criteria and take responsibility for the evaluative decisions (pp. 93-94). 

 
 

14 

 
 
 

The definition provided by CACREP, while ensures alignment with an accrediting body, 

leaves much ambiguity about some of the more specific requirements of gatekeepers, such as 

identifying and applying evaluative criteria. Finally, the CACREP definition does not 

specifically include what many would consider the most important concept of gatekeeping; the 

implication that gatekeeping serves to protect the profession in which the gatekeeper currently 

serves and a trainee is attempting to enter.  

As a whole, gatekeeping involves the evaluation of both the academic knowledge and 

professional dispositions necessary for students to become counselors. The current study aligns 

best with the Brear et al. (2008) definition of gatekeeping. It is also important to note here that 

the current study diverges from this whole concept of gatekeeping, given the focus on 

identification and remediation of professional dispositions.  

Components of Gatekeeping 

Terminology 

 

As discussed above, it is necessary to set the stage for systematic evaluation by providing 

definitions that are operationalized to be used in the current study. In addition to defining the 

actual concept of gatekeeping, there are several other terms that arise in the literature which may 

not be widely utilized in counselor education. These are almost as important as the definition of 

gatekeeping. Operationalizing concepts such as “gate-slipping”, “professional dispositions”, 

“impairment”, and/or “competence” are invaluable for use in gatekeeping literature and research.  

 

Gate-slipping. Gatekeeping is predicated on the protection of a field via intervention 

with inappropriate students, or trainees, prior to their entrance into that field (by graduating). 

This can result in remediation of a particular problem, or the dismissal of that individual from the 

field, or training program, if necessary. In a 2006 study, Gaubatz and Vera (2006) uncovered that 

 
 

15 

 
 
 

98% of faculty surveyed were aware of potentially deficient students in their programs, and in a 

2010 study of Australian counseling and counseling psychology programs, 58% of respondents 

indicated that they had passed a student that they had considered inappropriate for the counseling 

field (Brear & Dorrian, 2010). Most recently, in 2016, more than 60% of rehabilitation counselor 

educators and program directors who responded to the survey indicated that they were aware of 

currently deficient students in their master’s programs (Levine, 2016). 

Gate-slipping is essentially the opposite of gatekeeping, whereby “...potentially deficient 

students may advance through their training without remediation” (Gaubatz & Vera, 2002, p. 

36). The definition of this term also serves to provide some clarity about a main function of 

gatekeeping, which is to identify and remediate students whenever possible, not necessarily to 

barr entry into a field unless absolutely necessary. Emphasis on remediation over dismissal is 

absolutely necessary to change some of the negative connotations associated with gatekeeping, 

and therefore to increase faculty comfort with the concept and implementation of necessary 

procedures. 

Professional Dispositions. Section 4 of the CACREP Standards stipulates the 

development of key performance indicators (KPIs) that “represent a mix of important knowledge 

and skill measures at multiple points throughout the program” (CACREP, 2016, p. 2) in order to 

evaluate students. CACREP requires programs to develop KPIs per each core content area, as 

well as for professional dispositions, which are to be assessed using various measures at a 

minimum of two time points throughout the student's program. The professional dispositions 

KPIs are expected to be measured using the same criterion as the content KPIs (i.e., 

systematically, objectively). A key issue that needs to be addressed is the ambiguity of the term 

“professional disposition”, and what that means.  

 
 

16 

 
 
 

The 2016 CACREP Standards define professional dispositions as “the commitments, 

characteristics, values, beliefs, interpersonal functioning, and behaviors that influence the 

counselor’s professional growth and interactions with clients and colleagues” (p. 43). This breaks 

from some previous legal precedent and literature where interpersonal functioning had been 

included as part of a student’s academic performance (Frame & Stevens-Smith, 1995). The 

ruling of Greenhill v. Bailey (1975) defined academic performance as “a student’s demonstrated 

knowledge, technical and interpersonal skills, attitudes, and professional character” (Knoff & 

Prout, 1985, p. 792). It can be assumed that CACREP has intentionally separated academic 

knowledge and skills from professional dispositions, as the standards require two distinct 

avenues of KPI data (knowledge and skills, and professional dispositions).  

CACREP’s current definition of professional dispositions is problematic in two particular 

ways: (1) faculty attempting to measure a student’s commitments, beliefs, and values may 

struggle significantly with operationalizing such vague and subjective concepts, and (2) 

researchers may find that such subjective concepts may not be able to be measured 

systematically. Lumadue and Duffey (1999) emphasize the importance of performance standards 

using behaviorally specific student assessment categories, and to be clarified in program policies. 

It is quite difficult, however, to provide a comprehensive and behavioral assessment category of 

a student’s commitments or values, and furthermore, how to identify ways in which faculty will 

assess those and how they affect a student’s professional growth or interactions with clients and 

colleagues. Use of vague, subjective, and non-operationalized language can make students 

uneasy, and even more so if they are a part of an underrepresented group such as: students of 

color, students with disabilities, students learning English as a second language, students who 

identify as lesbian, gay, bisexual, transgender, or questioning (LGBTQ), and other minority 

 
 

17 

 
 
 

groups (Foster & McAdams, 2009). It is imperative to create and provide clarity regarding this 

term for future use and collection of data on student professional dispositions.  

The current study seeks to obtain and disseminate clarity around the term professional 

dispositions and its operationalization. It is hypothesized that a Delphi study will reveal 

professional dispositions to be made up components such as: integrity, concern for the wellbeing 

of others, helping disposition, self-awareness, and interpersonal skills (Fouad et al., 2009).  

 

Naming Impairment. There has been a considerable amount of discussion around the 

most appropriate way to discuss a student whose professional dispositions are deemed 

inadequate or inappropriate for the counseling field. Terminology problems likely contribute to 

issues with consistently identifying and acting on issues with student competence (Elman & 

Forrest, 2007). Historically, many different terms have been used to label student issues such as 

‘unsuitable’, ‘incompetent’, ‘inadequate’ and others (Elman & Forrest, 2007; Forrest et al., 

1999). In the early 1980’s, use of the term ‘impairment’ had begun and was used throughout the 

field of psychology to describe deficiencies in professional performance (Forrest et al., 1999). 

The most commonly used definition of impairment at that time was Laliotis and Grayson (1985): 

“interference in professional functioning due to chemical dependency, mental illness, or personal 

conflict” (p. 84).  

The Americans with Disabilities Act (ADA, 2008) uses the term impairment in 

describing disability, and thus, use of the term impairment creates legal risk if used to describe a 

student who does not have a disability (Brown-Rice & Furr, 2015; Elman & Forrest, 2007). 

Elman & Forrest (2007) recommend to retire the term impairment. In addition to the potential for 

legal issues, ‘impairment’ does not provide insight into the nature of the issues of a particular 

student; whether they are impaired as a result of lack of knowledge, experience, or in 

 
 

18 

 
 
 

professional dispositions or skills (Elman & Forrest, 2007). The authors suggest use of the term 

“problems of professional competency”, or PPC, given that the term impairment provides no 

precision in its use, and PPC indicates clearly a problem with professional competence. 

However, Brown-Rice and Furr (2015) warn that changing terminology may not be so simple. 

Impairment is already included in both the ACA Code of Ethics (ACA, 2014) and it can also be 

found in the Commission on Rehabilitation Counselor Certification Code of Ethics (CRCC, 

2016).  

A continuum of impairment. An additional issue related to this terminology is the 

vocabulary used by CACREP. While valid arguments have been made for use of the term PPC, 

CACREP standards specifically use the term professional dispositions (CACREP, 2016). 

Deviations from the language used in accreditation standards would perpetuate the present issues 

with vocabulary in student evaluations. Systematic evaluation begins with the consistent use of 

cohesive, operational definitions (Bhat, 2005). As such, for the purpose of the current study, 

issues related to student competence, specifically regarding the development of necessary 

professional dispositions will referred to as ‘inadequate professional disposition development’ 

(IPDD). Although this is a different term than that what has been proposed previously, it still 

adheres to the three critical components of new terminology identified by Elman and Forrest 

(2007): “(a) the inclusion of the concept that there is a problem with performance, (b) a 

professional standard, and (c) a focus on competence” (p. 505), and aligns with the vocabulary of 

choice by CACREP. The term “inadequate” lends itself to a spectrum ranging from unacceptable 

to acceptable and therefore indicates that students are not permanently problematic, rather that 

their current status as inadequate and may be improved in the future. IPDD is a term that can be 

used to describe students at a particular level, therefore it would also be possible to identify 

 
 

19 

 
 
 

someone as adequately developing professional dispositions as APDD, and another student who 

is above adequate development as exceptional professional dispositions development, or EPDD. 

Maintaining the term “professional dispositions” in the terminology keeps consistency with the 

CACREP standards, which will contribute to clarity in discussing these concepts. Finally, the 

entire phrase indicates a focus on a particular competency that is required of students in 

CACREP accredited programs. 

Due Process 

The Fifth and Fourteenth Amendments of the Constitution stipulates that no one shall be 

“deprived of life, liberty or property without due process of law” by either federal or state 

governments (U.S. Const. amend. V & XIV). As a fundamental value of rights in America, due 

process is one of the most important pieces to fair and effective student evaluation procedures, or 

gatekeeping. Essentially, due process exists to protect an individual’s rights throughout an 

accusatory proceeding, and to ensure fairness in order to yield accurate and truthful results 

(Strauss, n.d.). It has long been acknowledged that while evaluating students, it is very important 

to maintain documentation in order to ensure that due process rights have been observed, in the 

event of requiring remediation or dismissal (Bernard, 1975). While it is common practice to 

regularly document and monitor student academic progress, much less information has been 

found which provides guidance as to how to protect a student’s due process rights in the event of 

a dismissal due to non-academic characteristics (Knoff & Prout, 1985).  

Two types of due process exist: substantive or procedural. Substantive due process cases 

would involve faculty mistreatment of the student that is prejudicial, inconsistent, or erratic in 

nature (Knoff & Prout, 1985). A case concerning procedural due process would be related to a 

lack of timely notice of dismissal, or without the student being given an opportunity for a hearing 

 
 

20 

 
 
 

or to formally defend his or herself (Knoff & Prout, 1985). There are legal precedents of both 

types of due process involved in the dismissal of students from academic training programs, as 

due process is the most common basis for legal action by disgruntled students (Frame & Stevens-

Smith, 1995). Specifically, there is legal precedent that courts will consistently uphold the rights 

of institutions to dismiss or terminate students based on evaluation by qualified faculty (i.e., 

Goss v. Lopez, 1975; Greenhill v. Bailey, 1975) so long as due process has been observed 

(Lumadue & Duffey, 1999). Greenhill v. Bailey (1975) also provides clarity about what exactly 

would satisfy due process requirements: any notification of a student’s academic shortcomings 

prior to termination (Lumadue & Duffey, 1999).  

In recommending a procedure for teacher evaluation, Matula (2011) succinctly states: 

“Perhaps the most effective way to provide substantive due process measures is to ensure that the 

components of a sound evaluation system are present and implemented properly” (p. 101). By 

extension, use of a sound evaluation system would also most likely satisfy procedural due 

process, assuming that a sound evaluation system would involve transparency between faculty 

and students (timely notice of any issues), and steps for remediation prior to dismissal 

(opportunity for self defense). Thus, due process as a central goal for any evaluation of student 

professional dispositions would not only satisfy requirements related to accreditation, but also to 

the law and the U.S. Constitution. 

Legal Issues 

 

With vague guidance and high stakes, faculty often are uncertain how to handle this type 

of situation and may feel alone (Foster & McAdams, 2009). Legal issues regarding gatekeeping 

can impact multiple stakeholders, and it may seem more troubling to intervene rather than to 

 
 

21 

 
 
 

disrupt status quo (Foster & McAdams, 2009). Some types of legal issues that have been 

documented include:  

●  disgruntled students who have been dismissed from a program (i.e., Harris v. Blake and 

the Board of Trustees of the University of Northern Colorado, 1986); 

●  clients suing educational institutions for the work of an inappropriate counselor (i.e., a 

lawsuit directed at Louisiana Tech University because “...a university has an obligation 

not only to the degree participants, but also to the public, [to ensure] that a person who 

graduates from its program is competent in the area in which the degree is bestowed” 

(Custer, 1994, p. 7); 

● 

issues relating to First Amendment rights (i.e., Keeton v. Anderson-Wiley, 2011: a student 

sued her institution after being required to complete a remediation plan due to repeated 

statements of a desire to convert, and an inability to work, with a homosexual client in 

practicum) 

 

Each of these cases provide a unique insight into the issues that counselor educators must 

take into account when considering gatekeeping interventions. Further investigation of court 

rulings and statements also sheds light on needed components of gatekeeping policies and 

procedures. For instance, in Harris v. Blake, the student in question was dismissed after 

evaluation from just one faculty member, and although the judge ruled in favor of the faculty, 

current literature, and the ruling of that case, underscore the need for multiple faculty 

perspectives in a gatekeeping scenario (Bhat, 2005; Foster & McAdams, 2009; Gaubatz & Vera, 

2002; Kelly, 2011).  

 

The case at Louisiana Tech University involved a client suing a clinic where the 

counselor worked, which then evolved into a case against the institute where the counselor was 

 
 

22 

 
 
 

trained (Custer, 1994; Frame & Stevens-Smith, 1995). This lends itself to the weight of student 

evaluation; a graduate from a counseling program essentially has been endorsed by that 

program’s faculty as fit and competent to practice counseling. In the event of graduating a 

student with questionable motives or abilities, faculty run the risk of legal action by any client 

who may be harmed by that student. These legal precedents provide extremely valuable insight 

into the components that are necessary to effectively evaluate and remediate students when 

necessary in a way that also protects students’ rights.  

Keeton v. Anderson-Wiley (2011), highlights an issue that is current and pressing in our 

society. A student (Keeton) was required to complete a remediation program before she was 

allowed to move onto the program’s clinical practicum. This decision by faculty was a reaction 

to Keeton’s expression that, based in her religious beliefs, lesbian, gay, bisexual, transgender or 

questioning (LGBTQ) individuals have identity confusion and that she would intend to convert 

any such client she encountered. The federal court found that the faculty upheld due process and 

offered remediation to Keeton, therefore her efforts were denied.  

While there are certainly more examples of legal cases regarding student evaluations, the 

three outlined above provide sufficient information to accentuate the importance of systematic 

evaluation in all student competencies, and lend insight into specific components that should be 

incorporated in such evaluations. Empirical evidence reveals that programs which have 

formalized procedures report significantly lower rates of gate-slipping (Gaubatz & Vera, 2002). 

As such, the development and implementation of an instrument for use in systematic evaluation 

of professional dispositions development (PDD) satisfies legal and pragmatic needs of university 

programs. Additional needs that would be met by the use of a systematic evaluation of PDDs 

would be those of the accreditation bodies and the ACA, and CRCC Codes of Ethics. 

 
 

23 

 
 
 

Ethical Issues 

 

Ethically, it is expected that educators are active and proactively monitoring the students 

in counselor education programs so as to protect any potential future clients from the harm that 

can come from an inappropriate or inadequate counselor. This particular task, however, can be 

an ethical dilemma in and of itself. Gatekeeping requirements and ethical codes, although 

understood as required functions of faculty, are vague and lack specific descriptors to guide 

faculty through the process of gatekeeping, making the task unwelcome and daunting (Bhat, 

2005; Gaubatz & Vera, 2002; Kelly, 2011). Furthermore, evaluation of professional dispositions, 

which are not as objectively measured as academic competencies, can add an additional level of 

ethical consideration and discomfort for faculty. 

 

Until the most recent iteration of the CRCC Code of Ethics (2016), gatekeeping was not 

mentioned. Previously, section H.5: Rehabilitation Counselor Supervisor Evaluation, 

Remediation and Endorsement, included the statement, “…rehabilitation counselor educators do 

not endorse students whom they believe to be impaired in any way…” (CRCC, 2009, p. 22). This 

has changed with the newest revision of CRCC Code of Ethics (2016), in Section H.8.: 

Education Evaluation, Remediation, and Endorsement; specifically, H.8.b: Gatekeeping and 

Remediation for Students. This new section indicates three key requirements of rehabilitation 

counselor educators with regard to “…the inability of some students to achieve required 

competencies, which may be due to academic performance or personal concerns”: 

(1) assist students in securing remedial assistance, including counseling, when needed;  

 

(2) seek professional consultation and document the decision to recommend dismissal or   

refer students for assistance; and  

24 

 

 

 

 
 

 
 
 

(3) make reasonable efforts to ensure that students have recourse in a timely manner to 

address decisions requiring them to seek assistance, or to dismiss them and provide 

students with due process, according to institutional policies and procedures (CRCC, 

2016, p. 27). 

 

Similarly, Section F.9 of the ACA Code of Ethics (2014), Evaluation and Remediation, 

applies to counselor educators’ roles as evaluators of students. In addition to three points that 

align with those found in CRCC Code of Ethics, the ACA Code of Ethics also states that 

“Counselor educators clearly state to students, prior to and throughout the training program, the 

levels of competency expected, appraisal methods, and timing of evaluations…” (p. 15). It goes 

on to indicate that communication of evaluations and their results should be ongoing during a 

student’s time in an educational program. The use of an instrument that would support the 

systematic and objective measurement of particular competencies such as professional 

dispositions would satisfy the codes of ethics and would alleviate some of the daunting nature of 

evaluations. 

In addition to the specific guidelines in the codes of ethics, there are six ethical principles 

that counselors in both disciplines honor and adhere to which are also relevant to student 

evaluation in counselor education. Specifically, nonmaleficence is of concern in gatekeeping and 

student evaluation. 

Nonmaleficence. As defined in the ACA Code of Ethics, nonmaleficence is “avoiding 

actions that cause harm” (ACA, 2014). Counselor educators acting to intervene with students 

demonstrating IDDP are adhering to the ethical principle of nonmaleficence in regards to any 

potential clients that a student may work with in the future. Simultaneously, this principle serves 

to protect the welfare of the students in counselor education programs. Transparency about 

 
 

25 

 
 
 

student evaluations in counselor education programs is a method that faculty can use to improve 

student investment in evaluations, which can also serve to alleviate student anxieties about being 

evaluated and improve any feelings related to experiences with perceived deficient peers (Foster 

& McAdams, 2009). That would also align with the aforementioned stipulations in the ACA 

Code of Ethics, with regard to clearly and regularly disclosing information related to student 

evaluations. 

There is no question that counselor educators are legally and ethically required to 

evaluate students in academic and non-academic competencies. However, the mechanisms for 

how they are to accomplish this, and what those non-academic competencies are remains vague. 

These are just some of the many barriers to accomplishing gatekeeping that is systematic and 

consistent. 

Barriers to Gatekeeping 

 

Brear and Dorrian (2010) postulate three key areas where barriers to gatekeeping fall: (1) 

difficulties in operationalizing core competencies necessary for professional practice, (2) reactive 

rather than proactive approaches to gatekeeping, and (3) diverse contextual, professional, and 

legal issues. The current study seeks to address each of these three areas for the benefit of 

counselor educators, counseling students, and the clients that will be served in the future. 

Difficulties in Operationalizing: Subjectivity 

 

The earlier sections in this chapter provide an overview of some of the challenges caused 

by the lack of operationalized definitions in gatekeeping. The current study seeks to alleviate this 

problem by remaining aligned with the terminology provided by CACREP, and by introducing 

terms such as IPDD. Another issue not previously discussed in relation to operationalizing 

definitions is that of subjectivity in evaluation. Supervisors of graduate student counselors 

 
 

26 

 
 
 

reported that the most common feedback that is withheld from supervisees is negative feedback, 

and that the rationale for this is that the feedback is based on the supervisor’s personal concerns, 

and the anticipation of the supervisee’s negative reaction (Ladany & Melincoff, 1999). Fear of 

negative reactions to feedback that may be perceived as subjective leads to withholding 

potentially valuable feedback, and may also lead to a lack of intervention with problematic 

students (Hoffman, Hill, Holmes, & Freitas, 2005). Furthermore, Hoffman et al. (2005) found 

that “nonclinical issues, such as personality and professional issues, were even more likely to be 

viewed as subjective and difficult than were clinical issues” (p. 10). Operationalized definitions 

of the concepts deemed necessary for professional success are requisite to the successful 

evaluation of non-academic competencies such as professional dispositions (Lumadue & Duffey, 

1999; Ziomek-Daigle & Christensen, 2010). 

Reactive Gatekeeping 

Current practices in gatekeeping are such that programs do not act until after there has 

been a particularly problematic student, situation, or lawsuit (Brear & Dorrian, 2010; Foster & 

McAdams, 2009; Kerl et al., 2002; McAdams et al., 2007; Ziomek-Daigle & Christensen, 2010). 

Program faculty need to determine the appropriate criteria and instruments that they will use for 

student evaluations which satisfies CACREP Standards, students’ rights, and due process (Brear 

& Dorrian, 2010; Knoff & Prout, 1985). The current lack of available formalized procedures for 

evaluating professional dispositions in a systematic way hinders this first step, and therefore 

makes reactivity common among programs (Kelly, 2011). Additionally, in regard to gatekeeping 

as a whole (not just evaluation of professional dispositions), educators face poorly defined 

protocols, inconsistent procedures, and an overall lack of policy to guide them (Brear & Dorrian, 

2010; Forrest et al., 1999; Gaubatz & Vera, 2002; Olkin & Gaughen, 1991). Gaubatz and Vera 

 
 

27 

 
 
 

(2006) found that “Deficient students exist in counseling training programs, but well-designed 

gatekeeping procedures appear to improve the effectiveness with which they are identified and 

prevented from progressing unremediated into the counseling field” (p. 41).  

It has been found that more students are being dismissed from counseling programs for 

personal reasons rather than those of an academic nature (Brear et al., 2008). Given this trend, it 

is imperative that counselor educators have a systematic way to identify, evaluate, and remediate 

PDD at multiple stages throughout a student’s time in a program. The model of gatekeeping 

proposed by Ziomek-Daigle and Christensen (2010) includes four phases, and recommends 

student evaluation at each phase, from pre-admission through remediation outcomes. This is also 

supported by the work of Gaubatz and Vera (2002) whereby the researchers found that faculty 

were more likely to follow up with a problematic student when there were formal gatekeeping 

procedures in place. It is imperative that faculty feel that they are within a program that would 

support and back them if they were to feel that it was necessary to intervene with a problematic 

student. 

Contextual, Professional, and Legal Issues 

 

The above portion of this chapter outlines the legal issues that are involved in 

gatekeeping. Beyond legal concerns, there is a clear rationale as to why evaluation of 

professional dispositions are considerably more challenging than academic evaluation (i.e. Kelly, 

2011). While the creation of instruments and suggestions of policies for remediation are useful, if 

there is not an environment that supports the regular use of these instruments and policies, they 

hold no weight (Brear & Dorrian, 2010). Specifically, institutional pressures and social loafing 

contribute to these contextual issues in programs. 

 
 

28 

 
 
 

 

Institutional pressures. Enrollment is an ever-present factor in the lives of university 

faculty. Unfortunately, the perceived pressure to maintain high enrollment can hinder 

gatekeeping decisions at admissions and throughout a student’s time in the program (Brear & 

Dorrian, 2010). Counselor educators are already uncomfortable with the evaluative component 

of their jobs (Bhat, 2005). Real, or perceived, pressures around enrollment lends some insight 

into why gatekeeping can fall by the wayside in counselor education programs, when combined 

with a lack of appropriate tools or resources, it is very clear how this problem has evolved 

(Forrest et al., 1999).  

 

Social loafing. Social loafing has been described as “the diminished effort by those 

submerged in a group” (Elman, Forrest, Vacha-Haase & Gizara, 1999, p. 716). People in groups 

can feel less accountable, and as thought their actions are not important. As such, in an academic 

institution, faculty may assume that someone else (i.e., another faculty member or future 

supervisor) will take the responsibility for a student with IPDD (Elman et al., 1999). This 

particular problem, and most contextual issues discussed by Elman et al. (1999) would be best 

addressed at the systemic level: “Trainee impairment or incompetence is not a single, static event 

or outcome but a state that emerges over an extended period of time while embedded in 

contextual elements of the system” (p. 718). Immersing formalized evaluations, and a culture of 

transparency into the system of counselor education at a particular university program would 

alleviate the issue of social loafing-- faculty would have clear guidelines for addressing students 

demonstrating IPDD and would be capable of discussing these issues openly with students and 

faculty alike (Foster & McAdams, 2009; Gaubatz & Vera, 2002). 

 

 

29 

 
 

 
 
 

 

Professional Disposition Development and the Working Alliance 

As discussed throughout this chapter, evaluating students based on professional 

dispositions, or non-academic criteria, is more challenging, less desirable, and requires more 

stringent procedures than the traditional method of evaluating students based on academic merits 

(Brear & Dorrian, 2010; Forrest & McAdams, 2009; Gaubatz & Vera, 2002; Hoffman et al., 

2005; Ziomek-Daigle & Christensen, 2010). Briefly, this paper has mentioned the importance of 

the WA with regard to client outcomes. Within the context of the aforementioned barriers to 

gatekeeping, the current study seeks to incorporate WA into evaluation of professional 

dispositions as a means to anchor the vague concept of professional dispositions to an evidence-

based practice, such as WA. WA already has operational definitions, empirical evidence in 

support of its utility and underlying components, and a tool for assessing the strength of WA in 

various types of relationships (Bordin, 1979; Efstation, Patton & Kardash, 1990; Horvath & 

Greenberg, 1989; Horvath & Symonds, 1991; Lustig, Strauser, Rice & Rucker, 2002; Wampold, 

2001). These established components will assist in the development of an instrument which 

evaluates PDD, specifically intending to aid students in the development of dispositions which 

would enhance their ability to develop WA with clients across cultures, diagnoses and various 

demographic factors. 

Working Alliance: An Overview 

 

The beginning of the development of WA started as a reaction to the rigidity of 

behaviorism. Carl Rogers posited that humans were inherently good, and should have the ability 

to direct their own lives via the person-centered approach (PCA) in counseling (Rogers, 1951). 

Rogers theorized that growth and therapeutic change were facilitated by counselors, and key 

counselor qualities were the “central conditions” that elicit positive client change. The goal of 

 
 

30 

 
 
 

PCA is to assist a client in achieving self-actualization, and Rogers (1951) proposed that this is 

facilitated by counselors who are supportive and collaborative rather than directive and 

authoritative. 

 

The term working alliance was originated by Greenson (1965), after identifying that “The 

key to understanding the essential pathology as well as the therapeutic stalemate was in the 

failure of the patient to develop a reliable working relation with the analyst” (p. 77). Greenson 

(1965) stipulated that WA was essential for psychoanalytic therapy. Bordin (1979) takes the 

PCA and WA and provides a framework for the application of PCA’s core conditions into a 

counseling relationship. Furthermore, Bordin (1979) proposed that WA “...is one of the keys, if 

not the key, to the change process” (p. 252). This has been empirically supported over time, and 

specifically discuss Wampold (2001): “The alliance appears to be a necessary aspect of therapy, 

regardless of the nature of the therapy” (p. 137). It is in light of the significance of WA that the 

current study was developed: clearly, the importance of WA cannot be understated, and therefore 

it is crucial that counselor educators are both aiding students in learning how to develop WA, but 

also in identifying professional dispositions, or non-academic qualities that may hinder one’s 

ability to form WA.  

 

Three features of WA. Bordin (1979) specifically identified three features of WA with 

the intent to allow researchers to test WA. These three features are: agreement on goals, an 

assignment of task(s), and the development of bonds (Bordin, 1979). Bordin’s theory is based 

philosophically in the concept that “in any helping relationship the task of participating 

collaboratively in the counseling process taps directly into aspects of the client’s self-defeating 

behavior” (Horvath & Greenberg, 1989, p. 224).  

 
 

31 

 
 
 

Goals, or outcomes, of the relationship should be agreed upon at the outset of therapy, 

primarily having been determined by the client, but in collaboration between both parties 

(Bordin, 1979). The goals can be considered the target of the intervention (Horvath & 

Greenberg, 1989). There must be a mutuality in the agreement of the change goals set, which 

more specifically can be defined as the change in “...thought, feeling, and action, or some 

combination” which is necessary for the desired change to occur (Bordin, 1983, p. 35).  

Tasks will vary based on the theoretical orientation of the counselor and involve the in-

counseling behaviors that make up the counseling relationship. Furthermore, both parties must 

accept responsibility for the tasks, and perceive the tasks as pertinent and effectual (Bordin, 

1979; Horvath & Greenberg, 1989). Tasks can be described as the activities or exercises that are 

identified by the therapist, or counselor, as necessary steps for the client to meet the identified, 

desired change goal(s). Bordin (1983) has gone so far as to emphasize the importance of the 

tasks that he identified that the strength of the WA is predicated on how well client understand 

the relationship between a chosen task and their desired outcome or goals.  

Finally, the development of bonds speaks to personal attachments which are based in 

mutual trust, acceptance, and confidence (Bordin, 1979). More succinctly, “Bordin’s concepts of 

bond, goal, and task involve collaboration and hinge on the degree of concordance and joint 

purpose between the counselor and client” (Horvath & Greenberg, 1989, p. 224). Bonds can be 

defined as shared feelings of liking, trusting, and caring within the relationship (Bordin, 1983).  

WA and Counselor Education 

 

WA has been demonstrated as crucial to client outcomes in both rehabilitation 

counseling, psychotherapy, and supervisory relationships (Bordin, 1983; Horvath & Symonds, 

1991; Lustig, Strauser, Rice & Rucker, 2002; Martin, Garske & Davis, 2000). As there is much 

 
 

32 

 
 
 

empirical evidence in support of WA, it follows that “If counselor education programs are to 

make any level of evidence-based practice, then they should be addressing WA before all else” 

(Connor & Leahy, 2016, p. 375). Tying the evaluation of professional dispositions to a student’s 

ability to develop WA satisfies both CRCC and ACA Codes of Ethics given its place as an 

established evidence-based practice. The CRCC Code of Ethics is the statement that 

rehabilitation counselors have an ethical responsibility to use evidence-based practices (EBP), 

that are based on accepted research practices (CRCC, 2016). Similarly, the ACA Code of Ethics 

states that counselors must engage in counseling practices that are based on rigorous research 

methodologies (ACA, 2014). EBP has been described as “...a clinical decision-making process 

beginning with formulating clinical questions to ask, determining the best practice, and critically 

appraising the evidence for validity and applicability to the particular situation” (Chan, et al., 

2010, p. 180). Said another way, EBP provides a solution to the timeless question of Paul (1967): 

“What treatment, by whom, is most effective for this individual with that specific problem, and 

under which set of circumstances?” (p. 111). WA even goes a step farther, as a common factor, 

demonstrating that regardless of what theoretical orientation or counseling technique is used, 

WA accounts for a large amount of outcome variance (Wampold, 2001).  

It has been discussed, in this paper and others, that the development of instruments for 

use in gatekeeping have struggled with empirical evidence or replication, and furthermore, that 

establishment of an operational and systemic method for assessing professional dispositions 

continues to elude counselor educators (Bhat, 2005; Kelly, 2011; Kerl et al., 2002). Using WA as 

a means to ground evaluation of student professional dispositions aligns with ethical practice per 

ACA and CRCC, and meets the accreditation standards of CACREP (2016), which specifically 

 
 

33 

 
 
 

cites the need for counselor educators to address a student’s ability to develop therapeutic 

relationships using evidence-based strategies. 

Moving past microskills. Currently, counselor education is based in a microskills 

approach. The concept of microskills to address inadequacies in counselor training programs 

were introduced by Truax and Carkhuff (1967). The authors proposed a curriculum that was 

intended to teach discrete therapeutic skills which would increase students’ autonomy, decrease 

defensiveness, and increase ability to establish warmth and empathy. Microskills training, and 

the cornerstone works of Truax and Carkhuff (1967), and Ivey (1971), are “based on the 

assumption that the instructors can lessen therapeutic complexity for training purposes by 

focusing on single skills and allowing students to practice and master them individually” (Ridley, 

Kelly & Mollen, 2011, p. 803).  

Recently, there has been a call for reform in counselor education-- a need to move from a 

microskills approach to “a more comprehensive model of counseling competence” (Ridley, et al. 

2011, p. 820). Connor and Leahy (2016) outline a logical and pragmatic conceptualization for 

teaching WA as a means to take this step in moving towards educational practices which 

encourage the development of “higher order, cognitively complex skills such as attachment 

formation and case conceptualization” (p. 4). In alignment with this movement, it is not only 

practical but timely to provide counselor educators with a means to assess counseling students at 

a similarly higher level, and to encourage the focus on development of such higher order skills. 

Personality and WA. In some of the empirical evidence available on WA, there has been 

an indication that there are particular characteristics of both clients and therapists which can 

affect the alliance (Castonguay, Constantino, & Holtforth, 2006). Therapist characteristics or 

behaviors that have been found to positively relate to the quality of an alliance are warmth, 

 
 

34 

 
 
 

flexibility, and accurate interpretation (Ackerman & Hilsenroth, 2003). Characteristics and 

behaviors which negatively affect a quality alliance are rigidity, criticalness, and inappropriate 

self-disclosure (Ackerman & Hilsenroth, 2001). As stated by Wampold (2001): “It seems 

intuitive that some characteristics would be more desirable than others…” (p. 170). This 

uncovers the question of the relationship between WA and personality, and furthermore, can, and 

should, counselor educators evaluate student personalities as a function of gatekeeping? 

Chapman, Talbot, Tatman, & Britton (2009) align “enduring dispositions” with 

“personality characteristics”. Coming back to CACREP vocabulary, and the term “professional 

dispositions”, it can become troubling and confusing for counselor educators to feel that they are 

being required to evaluate students based on personality type. Chapman et al. (2009) explore the 

Five Factor Model in relation to WA and specifically indicate that information about this 

relationship would be beneficial to educators and supervisors alike. In alignment with the 

position of evaluation as a catalyst for remediation, the authors indicate that once educators are 

aware of a student who demonstrates troubling dispositions, or IPDD, they can then focus on that 

particular skill in order to aid the student in developing better alliances. 

Chapman et al. (2009) found that three domains from the Five Factor Model appeared to 

relate to the alliance include Neuroticism, Openness, and Agreeableness. A key implication of 

this study is the suggestion that educators or supervisors should be aware of personality traits, 

and levels of such traits, which can help or hinder the ability to develop WA with clients. 

Specifically, “Such therapy trainees may be at risk for engaging in tactics which inhibit or 

impede the development of a therapeutic bond and/or shared goals and tasks with their clients” 

(Chapman et al., 2009, p. 593). Furthermore, the researchers indicate the need for supervision to 

include increasing self-awareness and self-monitoring, which is a particular consideration for the 

 
 

35 

 
 
 

development of an instrument which evaluates professional dispositions: in implementing a tool 

for an ongoing evaluation and discussion around these particular qualities, or dispositions, 

counselor educators can aid students in their own self-awareness and development. Providing 

theoretical groundwork and background information can serve to aid educators in increasing 

their comfort with the evaluative roles they must fill. Moreover, providing an accurate theoretical 

framework for situating professional disposition evaluation in counselor education and student 

development may also serve to alleviate these negative feelings or avoidance. 

Using Performance Appraisals in Counselor Education 

Bhat (2005) puts forth the concept of bridging the disciplines of personnel management 

and counseling gatekeeping via the adoption of performance appraisals (PAs) into counselor 

education. Use of PAs would go a step beyond the existing literature on evaluations in counselor 

education by providing centuries of theories, data, and resources. PAs have also been specifically 

designed to minimize legal risk and have empirical basis for fostering employee investment 

(Bhat, 2005; Grote, 2002). The incorporation of PA theory into student evaluation provides a 

unique lens through which to theorize, and assess, student engagement in various aspects of their 

education. 

Early research on PA did not so much determine if PA worked, but rather explored its 

utility in the context of a company. Latham & Locke (1984) identify some key characteristics of, 

and arguments for PA, which are largely applicable to the study of professional dispositions in 

counselor education. Of particular relevance are the points: “Feedback on goal-directed 

performance motivates higher performance only when it leads to the setting of higher goals”, and 

“Feedback alone won’t improve performance, but if it’s missing from a performance system, 

performance can’t improve” (As cited in Grote, 1996, p. 5). Translating to counselor education 

 
 

36 

 
 
 

and student evaluation: students are interested in feedback that allows them to improve and set 

higher goals in the future, and feedback about professional dispositions is absolutely necessary 

for the improvement of student professional disposition development. 

 

These points are directly applicable to the need for an instrument which evaluates 

professional dispositions in counselor education programs, but also alludes to the need for a 

culture of transparency in which discussion of performance is ongoing. It is interesting to note 

Latham and Locke’s (1979) early awareness of issues with evaluation, stating that one main 

problem with motivating employees can be that motivation is internal, cannot be directly 

observed, and that “most managers are not in a position to change an employee’s basic 

personality structure” (p. 68). This further substantiates the rationale for using PA in counselor 

education and gatekeeping. Bhat (2005) takes six good management practices espoused by Grote 

(2002) for the development of sound appraisal systems and provides an adapted version for their 

use in counseling: 

1. Base evaluations on an analysis of the job. Evaluation instruments should focus on 

standards set forth by professional bodies. Trait-based rating systems that do not directly relate to 

specific job responsibilities are ill advised (Grote, 1996). The current study seeks to use experts 

in the fields of rehabilitation counseling and counseling in order to specifically align an 

instrument with accreditation standards, codes of ethics, and the required roles of counselors 

(specifically, the development of WA).  

2. Define performance dimensions in behavioral terms and use observable, objective 

evidence. This has been the catalyst for the development of the current study; abstract and ill-

defined concepts have been identified as requiring observation. It is imperative to include 

specific behaviors which indicate such desired attributes. Grote (2002) suggests that 

 
 

37 

 
 
 

understanding how a “master performer” would perform or exemplify a particular skill or 

characteristic can be used as the baseline for assessing that particular skill or characteristic. A 

Delphi panel of experts in collaboration will be able to identify what a “master performer” would 

look like in counseling. 

3. Keep things simple. Provide legitimate rationale for the inclusion of a particular 

assessment category; only include assessments that are critical and relevant. The current study 

has been designed specifically to address CACREP accreditation requirements, and thus has 

been determined as being critical, relevant, and mandatory in counselor education.  

4. Check yourselves regularly. Monitoring evaluation instruments and procedures 

regularly to check for discrimination is imperative to ensure equality across all groups of people 

in training programs. Regular review of student progress in PDD can encourage useful dialogue 

among faculty, and provides the opportunity to evaluate if there is any discrimination. The 

current study intends to embed evaluation into counselor education programs, and to propose the 

importance of a culture of transparency about evaluation in all areas of counselor education. 

Faculty should feel that they are able to check on another in the event of any suspected 

discrimination. 

5. Train evaluators adequately. Counselor education literature on gatekeeping policies 

has indicated that counselor education faculty have not often received formal training in 

gatekeeping policies or procedures (Gaubatz & Vera, 2002). Bhat (2005) suggests that a detailed 

and comprehensive training should be given to supervisors or educators. This would also serve to 

increase inter-rater reliability, and would decrease the likelihood of discrimination as discussed 

in the previous point. Grote (2002) states that not only should the evaluators be trained, but that 

the trainee should also be well informed of the process and should have the opportunity to 

 
 

38 

 
 
 

discuss results or feedback. This also aligns with the transparent design discussed throughout this 

paper, and originally suggested by Foster and McAdams (2009). An additional recommendation 

by Bhat (2005) is that, in addition to adequate training, inexperienced supervisors or educators 

should have their evaluations reviewed by those experienced in doing so, such that they feel 

supported and protected during the process. 

6. Ensure formal appeal mechanisms are built in. In keeping with the needs of due 

process outlined earlier in this chapter, all evaluation procedures should have appeal mechanisms 

available for students to defend themselves. These should be easily available to the students, and 

reiterated each time an evaluation takes place. 

Theoretical Framework for PDD 

An added benefit to considering PA as valuable for counselor educators is the application 

of the theories that have been used in PA to evaluation of professional dispositions in counselor 

education. There are many factors which contribute to the effectiveness of PA, most significant 

of which is “ratee reactions”, or when key stakeholders consider the PA to be useful (Iqbal, 

Akbar, & Budhwar, 2015). Primary theories discussed in PA literature include expectancy 

theory, goal setting theory, and social exchange theory (SET). These theories all provide 

different explanations as to the ratee-reactions to PA. 

Reduced to simplistic terms, expectancy theory states that the way to raise an employee’s 

interest in an organization is to reward them in correspondence to their performance (Harder, 

1992; Iqbal et al., 2015). This postulates that the better the employee’s performance, the greater 

the reward. A great drawback to expectancy theory is that if an employee perceives that they are 

not being rewarded in a way that is congruent with their performance, that employee will 

underperform as a way to achieve perceived equity with the organization (Harder, 1992). 

 
 

39 

 
 
 

Expectancy theory would not be an effective lens through which to view the evaluation of 

professional dispositions in counselor education. Students in counselor education programs are 

typically not in receipt of rewards, especially not those that would commiserate with their 

performance levels. A rewards system in counselor education could be viewed as discriminatory 

and unethical. 

On the other hand, goal setting theory and social exchange theory are both viable options 

for the theoretical framework of the current study. Goal setting theory was developed along with 

Latham & Locke’s development of PA (1979). This theory stipulates that: 

specific, high (hard) goals lead to a higher level of task performance than do easy goals 

or vague, abstract goals… So long as a person is committed to the goal, has the requisite 

ability to attain it, and does not have conflicting goals, there is a positive, linear 

relationship between goal difficulty and task performance (Locke & Latham, 2006, p. 

265).  

The open nature of goal setting theory lends itself to multiple applications, and it 

certainly is applicable to evaluation of professional dispositions in counselor education. 

Theoretically, students may set goals to improve PDD within the broader goal of the completion 

of a master’s degree. A significant factor of goal setting theory, identified by London et al. 

(2004), is that ‘setting goals’ is better than ‘assigning goals’, especially in the context of PA. 

Given the nature of an educational program, and the accreditation requirements faculty abide by, 

it is possible that students can perceive PDD as an assigned goal, rather than a chosen, or ‘set’ 

goal, and may view such evaluations as a nuisance or hindrance to their overall goals of 

graduation and employment, which would be their chosen goals. 

 
 

40 

 
 
 

 

However, incorporating goal setting theory into the context of social exchange theory 

produces an ideal theoretical framework for the current study. Emerson (1976) remarks that 

social exchange theory is not a theory at all, rather a frame of reference for looking at various 

theories whereby “Implied is a two-sided, mutually contingent, and mutually rewarding process 

involving “transactions” or simply “exchange” (p. 336). This is a reciprocal process which 

includes obligations that are contingent upon the actions of another person (Blau, 1964). 

Cropanzano and Mitchell (2005) discuss the foundational ideas of SET’s explanatory power: (a) 

rules and norms of exchange, (b) resources exchanged, and (c) relationships that emerge. 

Underlying these foundational ideas is the core concept that both the giver and receiver must feel 

appropriately compensated in order for some sort of change to occur (Blau, 1964; Cropanzano & 

Mitchell, 2005). Furthermore, “actors” (invested parties as individuals or a group) are self-

interested, and their behavior is a result of a need to secure a particular benefit. SET most 

adequately fits with counselor education and PDD evaluation, when considering the explanation 

that “when individuals feel that the organization is keen for their long-term development, they 

try to reciprocate” (Iqbal et al., 2015, p. 516).  

According to SET, students who believe that they are being evaluated for the purpose of 

their long-term, professional development, will be more committed to the program, and to the 

process of evaluation as a whole. We can then theorize that students who have set a goal to 

obtain a master’s degree in counseling will not be troubled with smaller, more specific or 

challenging goals en route to the overall achievement of that larger goal (i.e., completion of a 

remedial program, or a specific PDD), so long as they believe that those are central to their long-

term development and success. Using SET as the groundwork, if faculty and students have 

developed a reciprocal, working relationship, and faculty have demonstrated transparency about 

 
 

41 

 
 
 

evaluation procedures with an emphasis on improving student outcomes, we can then assume 

that students believe that the program (or organization) is committed to their long-term 

development, would be more committed to the evaluation process, to their role in the program as 

a whole, and these goals would no longer be ‘assigned’ but could now be considered as ‘set’. 

Summary 

Goal setting theory based in the framework of social exchange theory allows the 

conceptual basis for the current study, which seeks to create an instrument to systematically 

assesses professional dispositions in master’s level counselor education students using 

transparency and reciprocal relationships with faculty as a means to foster student involvement 

and investment in the evaluation process. The current study seeks to (a) bridge gaps in current 

gatekeeping literature, (b) meet the needs of counselor education programs which adhere to 

CACREP standards, and (c) provide an empirically based instrument for systematically 

evaluating student professional dispositions which may influence a student’s ability to develop 

WA. This was completed using two phases: Phase 1 involved the completion of a Delphi study 

to develop the items needed for an instrument, and Phase 2 involved a validation survey to 

establish initial psychometric properties. 

 

 

 
 

 

42 

 
 
 

CHAPTER THREE 

Method 

Systematic evaluation of students’ professional dispositions in counselor education is a 

necessary, yet complex task to accomplish for faculty. Currently, there is no available instrument 

to guide them in doing so. As such, the purpose of this study is to create an instrument for 

evaluation of professional dispositions using consensus from experts, and to then provide initial 

psychometric properties of the instrument items. The current study took place over two phases. 

Phase 1 includes the completion of a Delphi study involving experts in the fields of counselor 

education and rehabilitation counselor education, as well as those with experience in gatekeeping 

and clinical supervision research. The Delphi study provided the items necessary for the 

development of the instrument to be used in evaluating professional dispositions of master’s 

level counseling students. Phase 2 involved the completion of a validation survey in order to 

provide initial psychometric properties of the instrument items.  

The following chapter describes, in detail, the methodological choices made for the 

completion of the study. An in-depth description of the Delphi method and its utility in this study 

is provided, including the establishment of an expert panel, and the facilitation of such a panel by 

the researcher. This is followed by an overview of the validation survey, which involved a 

sample of rehabilitation counselor educators. The survey allows for the description of certain 

psychometric properties of the initial instrument items. 

Phase 1 

The specific research question for Phase 1 is: What items would be appropriate for 

faculty to use in the evaluation of professional dispositions in master’s level counseling students?  

 

 
 

43 

 
 
 

Study Design: Phase 1 

 

Research Question 1 has been addressed by way of a three-round Delphi study, in order 

to identify which items belong in an instrument that intends to evaluate professional dispositions 

of master’s level counseling students. Literature review and collaboration between experts in the 

fields of counselor education, rehabilitation counselor education, and in the concept of 

gatekeeping guided the development of items intended to measure student professional 

dispositions. The Delphi method, originally developed by the RAND Corporation, was created 

for the purpose of obtaining a reliable, expert consensus on a variety of complex issues in a 

systematic way (Okolai & Pawlowski, 2004; Vázquez-Ramos, Leahy, & Hernández, 2007).  

Delphi methodology has been defined as a “systematic solicitation and collation of 

judgements on a particular topic through a set of carefully defined sequential questionnaires 

interspersed with summarized information and feedback of opinions derived from earlier 

responses” (Delbecq, Van de Ven & Gustafson, 1975, p. 10). The goal of using a Delphi method 

is to aggregate judgement from a number of experts on a particular topic or problem which 

would serve to improve decision making about that particular topic or problem (Delbecq, Van de 

Ven & Gustafson, 1975; Dawson & Brucker, 2001). Dawson and Brucker (2001) have also 

stated that use of the Delphi method “allows for grouping and analyzing the speculations of 

many experts on a topic to move closer to knowledge on that topic” (p. 126), so long as the 

researcher remains aware and warned that truth is relative and may change over time. Perhaps 

the most succinct and relevant description of the Delphi method for the purpose of the current 

study is that a Delphi method is best used for a situation in which there is some evidence, but not 

yet knowledge about a particular topic, phenomena or problem (Ziglio, 1996). Professional 

dispositions are a concept that have various levels of evidence in counselor education literature, 

 
 

44 

 
 
 

but lack consistency and mechanisms for measurement (Fouad et al., 2009; Glance et al., 2012). 

As a result, currently there is not an instrument available which is theoretically based, 

psychometrically sound, or associated with outcome data from clients. 

Procedures: Phase 1 

The current study adhered to Table 3.1, adapted from Vázquez-Ramos et al., 2007 (p. 

113), which clearly delineates the steps, phases, and activities required in order to complete a 

successful Delphi study. The remainder of this section will discuss the accomplishment of the 

activities completed at each step. Heeding warning from Linstone and Turoff (2011), the number 

of rounds ultimately used were based on when stability was obtained; not only consensus. This 

happened quickly within this study, and thus only three rounds were used. Finally, the use of the 

Delphi method, in addition to the aforementioned benefits, adheres to what is defined as “good 

scale construction” by Clark and Watson (1995), in which an iterative process is undertaken to 

continually assess, and identify deficiencies in the initial item pool. The collaborative nature of 

the Delphi process as well as the access to a panel of experts uniquely suit Research Question 1. 

Prior to any research methods taking place, the researcher applied for approval by the 

Michigan State University Human Research Protection Program and the Institutional Review 

Board (IRB) in order to assure all proposed procedures would be ethical and protect the 

participants involved in the study. The researcher applied for an Exempt IRB review, given that 

the involvement of human subjects requires only survey procedures, and is therefore exempt 

from federal regulations under 45 CFR 46.101(b)(2). The request was granted for exemption and 

IRB approval, which allowed for the recruitment procedures to begin (IRB approval can be 

found in Appendix A). 

 

 
 

45 

 
 
 

 
 

Table 3.1  

Summary Table of the Steps, Phases, and Activities Involved in the Execution of a Delphi 
Method with Three Rounds (Vázquez-Ramos et al., 2007) 

Steps 

Phases 

Step 1  Selection 

Activities 

a.  Identification of potential experts 
b.  Invitation to participate 
c.  Recruitment of panelists 
d.  Constitution of the panel of experts 

Step 2  Exploration 

a.  Distribution of Delphi Round 1 (questionnaire usually involves an 

(Round 1) 

open-ended approach to elicit items or themes from panelists) 

Step 3  Evaluation 
(Round 2) 

Step 4  Re-

evaluation 
(Round 3) 

b.  Follow-up of Delphi Questionnaire 1 
c.  Completion of data collection for Delphi Questionnaire 1 
d.  Collation and categorization of results (content analysis) 
e.  Construction of Delphi Questionnaire 2 (first generation of potential 

items 

a.  Distribution of Delphi Round 2 
b.  Follow-up of Delphi Questionnaire 2 
c.  Completion of data collection for Delphi Questionnaire 2 
d.  Collation and categorization of results (Assessment of these items 

uses a Likert-type scale. Results are provided in terms of central 
tendency and measures of dispersion of participants’ responses. 
Those items that fall under the required central tendency and 
dispersion measures are included in the next round.) 

e.  Construction of Delphi Questionnaire 3 

a.  Distribution of Delphi Round 3 (Participants are provided with 
summary statistics for the previous round and are encouraged to 
reevaluate their answers based on their individual answers and 
group responses.) 

b.  Follow-up of Delphi Questionnaire 3 
c.  Completion of data collection for Delphi Questionnaire 3 
d.  Re-collation and categorization of results (Assessment of these 
items uses a Likert-type scale. Results are provided in terms of 
central tendency and measures of dispersion of participants’ 
responses.) 

e.  Calculation of summary statistics 

Step 5  Final 

a.  Identification of potential items on which consensus was obtained 

consensus 

(Those items that fall under the required central tendency and 
dispersion measures are included in the prototype item list.) 

b.  Summary of final results 
c.  Development of instrument prototype based on experts’ consensus 

46 

 
 
 

Participants: Expert Panel Selection (Step 1). The successful Delphi method includes 

the use of an expert panel, who are tasked with the development and ratings of particular items, 

concepts or predictions relevant to the topic being explored (Ziglio, 1996). The validity of a 

Delphi study is directly related to the expert panel; the knowledge of the panelists must be 

relevant to the topic being explored (Dawson & Brucker, 2010). Randomization does not 

improve this process, as it would in a traditional research approach, and there is not a statistical 

formula that lends itself to the correct number of panelists for any given Delphi study (Ziglio, 

1996). It has been warned that the size of the panel remain manageable; as communication across 

too many individuals will be problematic for lack of response time and volume of information to 

synthesize, although there is very little literature about the ideal size of a panel (Dawson & 

Brucker, 2010; Ziglio, 1996).  

 

The current study intended to use ten experts from the fields of rehabilitation counselor 

education, counselor education, and in the topics of supervision and/or gatekeeping. The term 

“expert” is also one that has remained elusive both in the literature about the Delphi method, as 

well as in rehabilitation and counselor education literature (Moorhouse, 2008). For the purpose 

of this study, the selection criteria of the “expert” panelists is as follows (experts meet at least 

one criterion): 

●  Has been a tenure-track faculty member (or core faculty member) at a CACREP 

accredited counselor education program; 

●  Has published on the topics of supervision, gatekeeping or professional dispositions in 

literature of counselor education or rehabilitation counselor education 

Using electronic survey software (e.g., Qualtrics), an email invitation was distributed to 

all potential participants (N = 41), including information about the study, informed consent, 

 
 

47 

 
 
 

contact information for the researcher, and a link to participate in a demographic survey. 

Following participation in the demographic survey, participants received a link to Round 1. 

There was attrition of participants throughout the three rounds of Phase 1, which is an inherent 

limitation of the Delphi methodology (Vázquez-Ramos et al., 2007; Ziglio, 1996). Each round 

had a different number of participants: Round 1 (n = 8), Round 2 (n = 5), Round 3 (n = 5). 

Demographics were only collected prior to the first round, in an attempt to decrease the length of 

time necessary to complete the surveys. 

Demographics of the sample of the expert panel was primarily female (n = 8), between 

the ages of 45 and 54 years of age (n = 6), and was entirely White (100%). The majority of 

participants reported over 10 years of experience as a counselor educator (n = 10), and identified 

as the program director (n = 9).  All participants held a CRC, and about half also identified as a 

Licensed Professional Counselor (LPC), or its equivalent (n = 6). Two individuals additionally 

noted licensure as vocational rehabilitation counselors (LVRC), and one individual additionally 

noted National Counselor Certification (NCC) as well as the Registered Play Therapist 

Supervisor (NCC, RPT-S) credential. All participants identified as working in a CORE-

accredited program (n = 12), and just over half identified CORE and CACREP-accreditation (n = 

7). One participant indicated only CACREP accreditation.  

 

Recruitment. In order to recruit panelists, following IRB approval, the researcher sent 

personal communication to identified experts via email (N = 41). Experts were primarily 

identified via publications about gatekeeping and/or professional dispositions in counselor 

education and/or rehabilitation counselor education journals (i.e., Counselor Education & 

Supervision, Rehabilitation, Research, Policy, and Education, Counseling and Psychotherapy 

Research). The identification of experts was expanded to include conference presentations on the 

 
 

48 

 
 
 

same topics. The recruitment email contained a summary of the study, including IRB approval, 

and time expectations per month of participation on the panel (see Appendix B). Once agreement 

was obtained, panelists received an informed consent document including the purpose of the 

study, as well as any anticipated risks or benefits. A brief demographic survey was included to 

collect basic information about the panelists (i.e., age, gender, years of teaching, credentials, etc.) 

The participants were not compensated for their participation. Following completion of the 

demographic survey, participants were provided with a link to Round 1 of the Delphi. 

Exploration (Step 2). Round 1 was distributed using electronic survey software (i.e., 

Qualtrics). Reminder emails were sent at two-week intervals. The survey remained open for four 

weeks in order to accommodate the length of the responses being requested. At the conclusion of 

the collection period, all responses were downloaded. A content analysis was completed 

following the open-ended questions in the initial questionnaire. Following the analysis of 

responses for themes and concepts, the questionnaire for Round 2 was created, including the first 

iteration of items for the instrument. Items were reviewed by the faculty mentor on this project, 

and approximately four items (e.g., Understands the tendency and the problem of racial 

stereotyping; Knowing how to consult or refer to resources available in ethnocultural 

communities) were added from the unpublished Multicultural Supervision Competencies 

Questionnaire (Wong & Wong, 2003) as multicultural competency was under-addressed in the 

responses to Round 1. 

Evaluation, Reevaluation & Final Consensus (Steps 3-5). Steps 3 through 5 in a 

Delphi study involve evaluating panelist responses for measures of central tendencies, 

reorganizing the items to reflect the most highly rated items, and returning the list to panelists for 

further refinement. In these steps, there is the potential for a regression to the mean, whereby 

 
 

49 

 
 
 

panelists see how others have rated the items, and they modify his/her own opinion based on 

seeing that information (Ziglio, 1996). The researcher took this under advisement throughout the 

monitoring of responses. 

Evaluation (Step 3). Once the instrument items evolved out of the responses from Round 

1, a survey was created and distributed for Round 2. Round 2 was distributed to the expert 

panelists using electronic survey software. Reminder emails were sent at two-week intervals.  

Responses from Round 2 were analyzed for measures of central tendency and dispersion, 

and qualitative responses were reviewed. It was established that any items falling below 3 on 

both Relevance and Utility would be removed from the item pool. No items fell below the 

threshold, and thus the survey for Round 3 was created. Items were organized in each section by 

mean (high to low). A survey was created for each participant who completed Round 2, and 

included their scores in addition to the group means and standard deviation. Using survey 

software, each survey was simultaneously distributed with the same instructions, and the follow-

ups simultaneously at the same intervals (two weeks).  

Re-Evaluation (Step 4). Re-evaluation is where panelists were able to see their scores in 

relation to the scores of other participants, and to re-rate items given this information. Round 2 

revealed that there was little dispersion amongst the scores of items, and Round 3 afforded the 

opportunity to reconsider previous ratings-- certainly a strength of the Delphi methodology. 

Round 3 was collected, and responses were again analyzed for measures of central tendency and 

dispersion. The same threshold was considered for items during this round, and again, no items 

fell below the threshold of 3 in Relevance and Utility. In fact, no item fell below a rating of 3 in 

either category, which will be discussed at length in Chapter 4. Given the limited dispersion 

among scores, it was determined that there was no need to pursue another distribution of the 

 
 

50 

 
 
 

survey items, and thus, the initial instrument prototype was created, and named the Assessment 

as Growth Inventory (AGI). 

Measures: Phase 1 

 

Phase 1 of this study did not involve the use of established instrumentation. In the process 

of this Delphi study, instrumentation evolved from an open-ended prompt to a Likert-scale rating 

relevance and utility of items. 

Round 1. Round 1 of a Delphi typically includes a broad, open-ended questionnaire 

designed to ascertain preliminary opinions and thoughts of the expert panel (Vázquez-Ramos et 

al., 2007; Ziglio, 1996). In the current study, these questions revolved around the three core 

components of WA: tasks, goals and bonds, (Bordin, 1979), as well as general thoughts about 

professional dispositions in counselor education. In order to maintain the reliability of responses, 

all panelists received the same electronic instructions, as identified as necessary in the literature 

(see Delbecq et al., 1975; Ziglio, 1996). Additionally, panelists received instructions to preserve 

homogeneity of language, as described by Ziglio (1996), which included specific definitions of 

the concepts they were asked about: 

●  CACREP defines Professional Dispositions as: “the commitments, characteristics, 

values, beliefs, interpersonal functioning, and behaviors that influence the counselor’s 

professional growth and interactions with clients and colleagues” This concept/definition 

has been deemed problematic given its ambiguous nature and subjective phrasing. 

Gatekeeping literature indicates the need for behaviorally specific assessment categories, 

and clear, operationalized language (Foster & McAdams, 2009; Lumade & Duffey, 

1999). 

Please adhere to the following definition of WA while responding to the below prompts: 

 
 

51 

 
 
 

●  WA is composed of three features: agreement on goals, an assignment of task(s), and the 

development of bonds (Bordin, 1979). Goals can be considered outcomes or the target of 

an intervention. Tasks are the steps necessary to achieve the agreed upon goals. Bonds 

are the personal attachments necessary for the relationship to work, and are based in 

mutual trust, acceptance, and confidence. 

Provision of these definitions allows for all participants to respond to the prompts with 

the same understanding of the definitions and intentions of the study. The prompts provided 

were: 

1.  Please identify behavioral and/or identifiable (i.e., measurable, observable) characteristics 

which embody professional disposition to you. 

2.  Please identify items for a potential instrument which collect observations around a 

student’s ability to develop WA with potential clients. Specifically, within each of the 

three components of WA: tasks, goals, and bonds. Please identify a minimum of 5 items 

per component. 

 

Round 2. Instrumentation for Round 2 used Likert-scales for ratings of relevance and 

utility of instrument items. Instructions in Round 2 indicated that there were four sections of the 

survey: Professional Dispositions, Tasks, Goals, and Bonds. Each section included the items 

which were developed out of Round 1, with two, four-point Likert-scales: Relevance and Utility 

(Figure 3.1). Participants were asked “Please rate the following items on their relevance & utility 

in the evaluation of student competency in [particular section] (range from 1 = Not at all 

useful/relevant to 4 = Extremely useful/relevant)”. This rating scheme is intended to achieve a 

ranking of the items, allowing for the removal of items below a particular threshold (Vázquez-

 
 

52 

 
 
 

Ramos et al., 2007). Participants were also provided the opportunity to include qualitative 

feedback on items via a comments area at the end of each section of items.  

 

 

Figure 3.1:  Round 2 Survey 

 

Round 3. Instrumentation in Round 3 was almost identical to that of Round 2. There 

were two main differences in this round: (1) definitions of relevance and utility were provided 

following feedback from participants in Round 2, (2) items were presented in rank order (highest 

mean scores to lowest), and included participants’ previous scores alongside the group means 

and standard deviations from Round 2 (Figure 3.2). The definitions of relevance and utility 

provided were: 

●  Utility: the item will be useful for faculty in their evaluation of pre-service counseling 

students and their ability to develop the working alliance with future clients. 

 
 

53 

 
 
 

●  Relevance: the item is connected to, or appropriate for, the evaluation of pre-service 

counseling students and their ability to develop the working alliance with future clients. 

 

Figure 3.2: Round 3 Survey 

 

Data Analysis: Phase 1 

 

In order to answer Research Question 1: “What items would be appropriate for faculty to 

use in the evaluation of professional dispositions in master’s level counseling students?”, it was 

necessary to begin the Delphi with a qualitative exploration of the types of items an expert panel 

would see as relevant and useful to evaluation of professional dispositions in master’s level 

counseling students, and which relate to WA. As such, Round 1 of the Delphi differed in data 

analysis given the qualitative nature of the data collected. Round 1 involved the content analysis 

of responses to the prompts. Content analysis has been described “...as a flexible method for 

analyzing text data”, which has also been considered problematic when not specifically 

differentiated from conventional content analysis, due to a lack of concrete definitions or 

 
 

54 

 
 
 

procedural methods often associated with content analysis (Cavanagh, 1997; Hsieh & Shannon, 

2005). In light of these issues, the current study used a directed content analysis, which is more 

structured than a conventional content analysis, using existing theory to determine a coding 

scheme (Hickey & Kipping, 1996; Hsieh & Shannon, 2005). The items provided through the 

open-ended questions were organized into four categories: Professional Dispositions, Tasks, 

Goals, and Bonds. Coding adhered to the Bordin (1979) definitions of tasks, goals, and bonds, 

and the CACREP (2016) definition of professional dispositions. Items were coded into the 

section which definition it best matched. Some participants provided which sections they 

believed matched their items, and these were verified or re-categorized by the researcher. Items 

and their respective categorization were reviewed by the research mentor before distribution in 

Round 2. 

In Rounds 2 and 3, the Statistical Package for the Social Sciences ® Version 24 was used 

to calculate descriptive statistics, measures of central tendency and dispersion for all items on 

both relevance to their category as well as utility in the evaluation of pre-service counselor 

education students. Items were organized into categories by mean utility, then relevance to the 

category. Items with both relevance and utility means below 3.0 (Moderately Relevant) will be 

removed from the instrument.  

Phase 2 

 

The specific research questions for Phase 2 include: How much do those items relate to 

the concepts of WA (i.e., goals, tasks and bonds)? How useful are those items for counselor 

educators? 

 

 

 
 

55 

 
 
 

Study Design: Phase 2 

 

In order to establish confidence in the instrument that was developed through Phase 1, 

additional statistical procedures were necessary. Additional validity is especially important for 

future implementation and research using this instrument. A goal of this assessment measure is 

that it aligns with the well-established concept of the WA, and aids faculty in monitoring 

students’ professional dispositions which may impede their ability to develop a WA with future 

clients. 

Phase 2 seeks to further identify the psychometric properties of the proposed instrument, 

including internal consistency/reliability, and content/construct validity. Psychometric properties 

were established by having counselors and counselor educators rate the items developed through 

the Delphi process on their relevance to the particular category to which they have been assigned 

(e.g., Professional Dispositions, Tasks, Goals, or Bonds). Phase 2 affords the opportunity to 

evaluate measures of internal consistency in each section, using Cronbach’s alpha. Furthermore, 

Phase 2 seeks to provide information of the perceived utility of the instrument in counselor 

education, therefore this sample was also asked to rate items on their usefulness in evaluating 

pre-service counseling students and student ability to develop WA. 

Sampling: Phase 2 

In order to achieve the appropriate validation of the instrument, the first population to be 

sampled in its development is that of Certified Rehabilitation Counselors (CRCs). The 

population will be homogeneous in that way, allowing the researcher to focus on the instrument 

items and its development, rather than an exploration or comparison between groups of 

counselor educators. Although the instrument created intends to meet a need in CACREP 

accredited programs, evaluation of professional dispositions of counselors-in-training would 

 
 

56 

 
 
 

benefit counselors or counselor educators of any specialty. Having a sample inclusive of 

rehabilitation counselor educators and rehabilitation counselors in practice sheds light on 

considerations of site supervisors, educators of varying experience, and both new and established 

counselors in the field.  

To maximize participant response, the researcher received approval from the 

Commission on Rehabilitation Counseling Certification (CRCC) to provide one (1) continuing 

education credit (CEU) for completion of the survey (see Appendix J). Along with this approval, 

CRCC provided a randomly selected sample of CRCCs (N=1,025) from their database, to whom 

the survey was sent. Of this population, 29 emails bounced, leaving a population of N = 996. An 

ultimate sample of 148 participants was achieved (14.9% response rate), following the removal 

of any participants who had completed less than 55% of the survey (only the demographics 

section; 20.1%, n = 200). The sample was primarily female (n = 113), with one individual 

responding that they identified as “transman”. The sample was also overwhelmingly white (n = 

114). While more diversity would certainly be ideal, the sample is indeed representative of the 

current makeup of the national population of rehabilitation counselors: in 2015, White females 

accounted for 43% of all individuals who earned master’s degrees in Rehabilitation Counseling, 

and by extension, women account for 78.5% of all master’s degrees awarded in Rehabilitation 

Counseling (National Center for Educational Statistics, 2015). A full representation of the 

sample characteristics can be found below, in Table 3.2. 

 

 

 

 

 
 

57 

 
 
 

 

 

Table 3.2 
 
Participant Demographic Characteristics 

 

Gender 
           Male 
           Female 
           Other 

Ethnicity 

White 
Black/African American 
Asian 
Hispanic or LatinX 
Other 
Prefer not to respond 

Years of experience 
in counselor education 
           0-4 
           5-10 
           10+ 

Credential 

CRC 
LPC or equivalent 
LLPC 
Other 

Counselor Educator 

Yes 
No 

Frequency (%) 

 

34 (22.5%) 
113 (74.8%) 

1 (0.7%) 

 

114 (75.5%) 
18 (11.9%) 

3 (2.0%) 
6 (4.0%) 
4 (2.6%) 
3 (2.0%) 

 
 

20 (13.2%) 
11 (7.3%) 
17 (11.3%) 

 

147 (97.4%) 
29 (19.2%) 

2 (1.3%) 

40 (26.5%) 

 

48 (32.4%) 
100 (67.6%) 

Most of the respondents who did not identify as counselor educators also indicated that 

their current position does not involve the clinical supervision of Master’s level rehabilitation 

counseling students (n = 76). 40 participants identified having an “other” credential. The most 

common of these included Certified Case Manager (CCM), and Licensed Clinical Addiction 

Specialist (LCAS). It should be noted that upon further review, a number of participants who 

 
 

58 

 
 
 

responded “other” did indeed hold LPC or LLPC equivalent credentials, such as Licensed Mental 

Health Counselor (LMHC), Licensed Professional Clinical Counselor (LPCC), or LPC-Intern. 

Procedures: Phase 2 

Following IRB approval (obtained prior to Phase 1), and the completion of Phase 1, the 

researcher completed the Application for Use of the CRCC Database for a Research Project, and 

submitted it along with a copy of the proposed survey, to CRCC for approval. Once approved, 

the researcher was required to complete the CRCC Mailing List Rental Agreement, stating that 

the project will not use the mailing list in excess of three times, over three months, and that the 

mailing list will remain confidential, and will be returned upon completion of its use (via email). 

In addition, the list was leased for the amount $250, to be paid to the CRCC. This approval 

allowed the researcher not only to obtain contact information of CRCs, but to provide one 

continuing-education credit to any participants who completed the study. 

Following attainment of approval from CRCC, an email was sent out to the list that 

contained (1) a summary of the purpose of the survey, (2) the purpose of the larger study, and (3) 

IRB information and contact information for the researcher (Appendix B). A link then took 

participants to both the informed consent and subsequent measure, using an electronic survey 

platform (e.g., Qualtrics). The introduction to the survey included a reiteration of informed 

consent, the study purpose, and operational definitions of “relevance” and “utility”. The survey 

includes a brief collection of demographics related to the participant’s work experience, as well 

as a description of his/her job title, or educational program (e.g., accreditation status). Following 

the initial distribution, three follow-up emails were sent at one-week intervals. In order to obtain 

their CRCC CEU, at the completion of the survey, participants were taken to a separate site (e.g., 

Google Forms), where they provided their email address, and the CEU instructions were sent by 

 
 

59 

 
 
 

the researcher at the end of each week the survey is open (see Appendix J). Confidentiality of 

participants was protected by ensuring that personal information was not linked to survey 

responses, which is provided via the use of the survey software redirecting participants to the 

Google Form upon survey submission. 

Measures: Phase 2 

 

The instrument that was generated in Phase 1 of this study via the Delphi method was 

used to complete this phase. A goal of this study is to create an instrument that aligns with the 

Working Alliance Inventory (WAI). WAI was created to be based on Bordin’s conceptualization 

of WA, specifically as “a measure that not only captures outcome variance but has a clearly 

articulated relation with a specified body of theory, which in turn clearly explicates the relation 

of the theoretical constructs to the counseling process” (Horvath & Greenberg, 1989, p. 225). 

The WAI is composed of three, 12-item subscales, and there was an attempt in the current study 

to create an instrument that is similarly constructed. Due to the results of Phase 1, the Assessment 

as Growth Inventory, will differ from the WAI in subscales and number of items. There was high 

agreement in the relevance and utility of the items throughout the Delphi, leaving Phase 2 with 

an instrument of 4 sections and varying lengths (Professional Dispositions: 13 items; Tasks: 10 

items; Goals: 8 items; Bonds: 17 items) for a total of 48 items. 

In order to assess the relationship of the instrument items to their category (e.g., 

Professional Dispositions, Tasks, Goals, Bonds), the survey in Phase 2 asks participants to rate 

item relevance to their category: “Please rate the following items on relevance in the evaluation 

of student competency in (category)”, on a four-point Likert scale (1 = Not at all relevant; 4 = 

Extremely relevant). Participants were also asked to rate the utility of the items for evaluation of 

 
 

60 

 
 
 

pre-service counseling students: “Please rate the following items on utility in the evaluation of 

student competency in (category)”; (1 = Not at all useful; 4 = Extremely useful).  

Data Analysis: Phase 2 

 

In order to answer the research questions: “How much do those items relate to the 

concepts of WA (i.e., goals, tasks and bonds)?” and “How useful are those items for counselor 

educators?”, Phase 2 used the Statistical Package for the Social Sciences ® Version 24, to 

calculate descriptive statistics, measures of central tendency and variation for all items on both 

relevance to their category (e.g., goals, tasks, bonds), as well as utility in the evaluation of pre-

service counseling students. Items were again organized by mean utility, then relevance to the 

category, and explored for any with both relevance and utility means below 3.0 (Moderately 

Relevant) to be removed from the instrument.  

 

Furthermore, a measure of internal consistency (i.e., Cronbach’s alpha) are necessary to 

establish the relation of items within each section of Assessment as Growth Inventory. 

Cronbach’s alpha lends itself to the credibility of the sections of the instrument, as it is not 

intended to be unidimensional, and this provides justification for the grouping of items in their 

respective sections. Following recommendations in the literature (e.g., Bland & Altman, 1997; 

Goforth, 2015), the goal of this measure will be to identify internal consistency reliability 

coefficients of 0.65 or higher. Weaker coefficients would indicate that the items in each section 

are not correlated, and therefore would require reorganization or removal. Finally, a between-

groups analysis (i.e., an independent samples t-test) will be conducted to identify any statistically 

significant differences in the mean scores between educators and non-educators (e.g., 

participants who selected ‘no’ to the demographic question “Are you a counselor educator?”). 

 
 

 

61 

 
 
 

 

In sum, for completion of Phase 1, an expert panel was invited to participate in a three-

Summary 

round Delphi study with the goal of identifying the appropriate items to include in an instrument 

for measuring the professional dispositions of master’s level counseling students. Items are 

related to the concept of WA, which is a known evidence-based practice that positively impacts 

client outcomes. The initial iteration of items were compiled and distributed for validation in 

Phase 2. 

Phase 2 included a survey for the purpose of establishing reliability and validity measures 

for the Assessment as Growth Inventory. The instrument was comprised of 4 sections and 

varying lengths (Professional Dispositions: 13 items; Tasks: 10 items; Goals: 8 items; Bonds: 17 

items) for a total of 48 items. Content validity is being assessed, asking CRCs if the items 

developed in the Delphi align with the three core constructs of WA: tasks, goals, and bonds 

(Bordin, 1979). Internal consistency of each section is also to be assessed, using Cronbach’s 

alpha to identify reliability of each section of the instrument. Completion of Phases 1 and 2 

provide sufficient empirical evidence in support of an instrument for faculty use in the systematic 

evaluation of master’s level counseling student’s professional dispositions. 

 

 

 

62 

 
 

 
 
 

 

CHAPTER FOUR 

Results 

Results of the study will be presented by phase, and the Delphi in Phase 1 will be 

discussed by round, to ensure clarity.  

Phase 1: Round 1 

Round 1 of the Delphi study involved the accumulation of potential instrument items in 

the form of qualitative, open-ended prompts. Eleven expert panelists responded, providing a total 

of 136 potential items for the Assessment as Growth Inventory. Using a directed content analysis, 

items were sorted into four sections: Professional Dispositions, Tasks, Goals, and Bonds. Items 

were sorted based on the section that the participants identified in their responses, however, upon 

analysis, items did not always remain in their originally identified category.  

 

Professional Dispositions. A specific prompt asked for participants to identify items 

related to professional dispositions, using the following definition provided by CACREP (2015): 

“the commitments, characteristics, values, beliefs, interpersonal functioning, and behaviors that 

influence the counselor’s professional growth and interactions with clients and colleagues.” 

Participants were asked specifically to identify behavioral and/or identifiable (i.e., measurable, 

observable) characteristics which embody professional dispositions of counselors. Sixty (n = 60) 

total items were provided. Items were coded into broad categories which exemplified the nature 

of the item, in order to narrow down the volume while honoring the essence of what participants 

identified as important. Twelve categories emerged for initial coding:  

1.  Professional skills (administrative and personal) 

2.  Lifelong learning and commitment to counseling profession 

3.  Core conditions of counseling 

 
 

63 

 
 
 

4.  Conflict resolution 

5.  Ethics 

6.  Counseling identity/professional identity 

7.  Maturity 

8.  Microskills 

9.  Multicultural competence 

10. Self-awareness  

11. Tasks 

12. Integrity 

 

In adherence to a directed content analysis, the definition of professional dispositions 

guided this coding process, leading to moving items to other sections (e.g., tasks into Tasks 

section and core conditions and microskills into Bonds section), as well as consulting literature 

on professional dispositions and WA when necessary to remedy a concern. An example of this 

came up several times in the professional dispositions section, such as “Awareness of how self is 

impacting the counseling relationship”, or “put needs of client before self (especially the need to 

fix things for the client- client needs to be supported to fix things for themselves and the student 

counselors own needs to talk and be heard)” while submitted as professional dispositions, these 

items were more clearly aligned with the Bonds section, which falls under the definition of 

“liking, trusting, or a feeling of common purpose and understanding between counselor and 

client” (Bordin, 1994; Horvath & Greenberg, 1989). Ultimately, after categorization, 

organization, and rewording, 14 items emerged from the responses of participants in the area of 

Professional Dispositions. Examples of such items include: “Engaging in difficult conversations 

with clients, coworkers, and/or supervisors in an appropriate manner”, and “A dedication to, 

 
 

64 

 
 
 

and embodiment of, the ethical values (autonomy, justice, veracity, beneficence, nonmaleficence, 

fidelity”.  

 

Tasks. Tasks, according to the WA, are the activities that occur within the counseling 

relationship (by both client and counselor) in order to bring the client closer to their identified 

change goal (Bordin, 1979; Lustig et al., 2002). Originally, 25 items were submitted for the 

Tasks section of the instrument (n = 25). There was much overlap within the items, which 

allowed the analysis to consist of the identification of items that were similar in nature, in order 

to create one item that would represent multiple submissions. An example is the item: “Ability to 

collaborate with clients to design tasks that are congruent with the mutually agreed upon goal”, 

which encompassed five other submissions:  

○  I feel that the counselor’s assignments will help me achieve my desired change,  

○  I feel that the way we are working towards my desired change will be effective,  

○  Tasks will be mutually agreed upon,  

○  Consistently involves the client in the design of tasks to accomplish, and  

○  Ability to identify tasks that will lead to goal attainment 

 

Ultimately, the original 25 items were represented by 12 items to be included in the 

instrument. Examples of such items include: “Application of the use of Evidence-Based Practice 

(EBP),” “Being able to show support when holding clients accountable for task completion, or 

non-completion,” “Knowing how to consult or refer to resources available in ethnocultural 

communities,” and “Adjusting with the client as his/her needs and problems evolve during the 

counseling process.” 

 

Goals. Goals are what is viewed as the outcome of the counseling relationship, and in the 

WA, they are to be mutually agreed upon by both counselor and client (Bordin, 1979). 21 items 

 
 

65 

 
 
 

were submitted under the heading of Goals in Round 1, and were condensed into 9 items for the 

instrument. There was again a redundancy in the submitted items, most commonly around the 

concept of mutually agreed upon goals, which yielded the instrument item “Collaborating with 

clients in the development of mutually agreed upon goals.” Items also reflected collaboration in 

the implementation of goal plans, including “Developing SMART (Specific, Measurable, 

Achievable, Relevant/Realistic, Timely) goals that are mutually agreed upon,” and “Mutual 

cooperation with the client when establishing and implementing the treatment plan or IPE.” 

 

Bonds. Of the three components of the WA, bonds had the most submitted items (n = 

29), and also was the section where most misidentified items fit into. As such, it was more 

challenging to narrow this section down, even while adhering to the WA concept of bonds, 

including “liking, trusting, or a feeling of common purpose and understanding between counselor 

and client” (Bordin, 1994; Horvath & Greenberg, 1989). Maintaining confidentiality and a non-

judgemental disposition were commonly identified as important for this section. This led to items 

such as “Maintaining a nonjudgmental disposition regarding client values,” “Demonstrating 

unconditional positive regard for clients,” and “Communicating in a confidential, responsive, 

and empathic manner to establish rapport in a way that promotes openness and sensitivity to 

potential cultural differences.” Additionally, items about cultural sensitivity and addressing 

multicultural considerations came out of this section, including items such as “Actively avoiding 

cultural biases and discriminatory practices in working with clients of minority backgrounds,” 

“Addressing multicultural issues when presented in the session that may affect the counseling 

relationship or the client’s ability to pursue a goal,” “Willingness to advocate for clients of 

minority backgrounds who experience institutional discrimination.” The multifaceted nature of 

client-counselor bonds lead to more items in this section; 17 items.  

 
 

66 

 
 
 

Phase 1: Round 2 

 

Participants in Round 2 were asked to rate items in each section on a 4-point likert scale 

in Relevance and Utility (1 = Not at all useful/relevant; 4 = Extremely useful/relevant). For the 

purpose of the Delphi study, items were then assessed for mean utility and relevance, and 

standard deviation. It was predetermined that any items which fell below a mean of 3 in both 

relevance and utility would be eliminated from the subsequent round.  

 

Professional Dispositions. Items in the Professional Dispositions section were 

consistently highly rated, with relatively small variation (Table 4.1). The mean Relevance of all 

items in Professional Dispositions for Round 2 was 3.62 (SD = 0.415), which was just slightly 

higher than the mean Utility of all items in Round 2, which was 3.59 (SD = 0.509). One item, A 

dedication to, and embodiment of, the ethical values (autonomy, justice, veracity, beneficence, 

nonmaleficence, fidelity), was rated Extremely Relevant and Useful by all participants in this 

round (M = 4.00; SD = 0.000). Showing unconditional acceptance of all clients, peers, or 

coworkers regardless of their demographics (i.e., age, gender, race, ethnicity, sexual 

identity/orientation, culture) was rated as Extremely Useful by all participants (M = 4.00; SD = 

0.000), with a mean Relevance of 3.83 (SD = 0.408). Half of the items had the same ratings for 

both Relevance and Utility (n = 6), while just under half of the items were rated more highly on 

Relevance than Utility (n = 5). No item was rated below 3 in either category.  

 

 

 

 

 

 
 

67 

 
 
 

 
 

Table 4.1 
 
Round 2: Professional Dispositions (Descending by mean Relevance) 

 

Item 

A dedication to, and embodiment of, the 
ethical values (autonomy, justice, veracity, 
beneficence, nonmaleficence, fidelity) 

Showing unconditional acceptance of all 
clients, peers, or coworkers regardless of their 
demographics (i.e., age, gender, race, 
ethnicity, sexual identity/orientation, culture) 

Understanding the tendency and the problem 
of racial stereotyping 

Demonstrates professional and personal 
maturity such as accepting feedback, 
following through on commitments, 
commitment to professional growth 

The development of a counselor identity (e.g., 
theoretical orientation, helping disposition, 
professional advocacy) 

Sufficient professional administrative skills 
(i.e., punctuality, organizational skills, 
preparation, professional written 
communication/documentation, awareness of 
policy/procedures) 

Maintains a respectful countenance in all 
interactions: perceives and honors diversity, 
boundaries, and appropriate communication 
style 

Sufficient professional interpersonal skills 
(i.e., developing personal rapport with 
coworkers, appropriate tone of voice, 
language use, use of humor, appropriate dress, 
etc.) 

Use of problem-solving skills in a timely and 
professional fashion 

Relevance 

Utility 

Mean 

SD 

Mean 

SD 

4.00 

0.00 

4.00 

0.00 

3.83 

0.408 

4.00 

0.00 

3.83 

0.408 

3.60 

0.548 

3.80 

0.447 

3.60 

0.548 

3.80 

0.447 

3.60 

0.548 

3.67 

0.516 

3.40 

0.894 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

68 

 
 
 

 

 

Table 4.1 (cont’d) 
 
Engaging in difficult conversations with 
clients, coworkers, and/or supervisors in an 
appropriate manner 

Commitment to a career in the counseling 
field via indications of a desire to be a 
lifelong learner (e.g., always seeking new 
information and resources, participation in 
professional organizations) 

Demonstrates professionalism and 
professional behavior in interactions with 
peers, supervisors, clients, and as a 
representative of their educational program 

Awareness of his/her role as a counselor, 
including self-awareness, humility, and 
integrity 

 

3.50 

0.548 

3.60 

0.548 

3.50 

0.837 

3.40 

0.894 

3.40 

0.894 

3.40 

0.894 

3.40 

0.894 

3.40 

0.894 

Tasks. Task items were also highly rated, with limited variability (Table 4.2). The overall 

mean Relevance for items in the Task section for Round 2 was 3.68 (SD = 0.356), with the 

overall mean Utility for Task items of 3.62 (SD = 0.356); each higher than the overall means for 

all items in Professional Dispositions. The item with the highest mean Relevance was Ability to 

collaborate with clients to design tasks that are congruent with the mutually agreed upon goal 

(M = 4.00; SD = 0.00). This item was rated as Extremely Relevant by all participants, with a 

mean Relevance of 3.80 (SD = 0.447). Five items were rated with higher mean Relevance than 

Utility (n = 5), with four items showing higher mean Utility than Relevance (n = 4). No item was 

rated below 3 on either Relevance or Utility, and the lowest rating overall was the Utility of 

Being able to apply theory to justify rationale behind assigned tasks (M = 3.20; SD = 0.837). 

 

 

 
 

69 

 
 
 

 

 
 

Table 4.2 
 
Round 2: Tasks (Descending by mean Relevance) 

 

Item 

Ability to collaborate with clients to design 
tasks that are congruent with the mutually 
agreed upon goal 

Developing options to address clients’ needs 
and problem with identified 

Ability to identify tasks that will lead to goal 
attainment 

Adjusting with the client as his/her needs and 
problems evolve during the counseling 
process 

Ability to do needs assessments with client in 
order to identify needs that a client may have 
but may not be aware of 

Ability to communicate the link between the 
chosen task and the agreed upon goal 

Knowing how to consult or refer to resources 
available in ethnocultural communities 

Being able to show support when holding 
clients accountable for task completion, or 
non-completion 

Application of Evidence-Based Practices 
(EBP) 

Ability to design tasks to meet the unique 
developmental and individual needs of the 
client 

Being capable of helping clients make non-
successes into successful learning experiences 

Being able to apply theory to justify rationale 
behind assigned tasks 

Relevance 

Utility 

Mean 

SD 

Mean 

SD 

4.00 

0.00 

3.80 

0.447 

3.80 

0.447 

3.80 

0.447 

3.80 

0.447 

3.80 

0.447 

3.80 

0.447 

3.60 

0.548 

3.67 

0.516 

3.60 

0.548 

3.67 

0.516 

3.80 

0.447 

3.60 

0.548 

3.80 

0.447 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

3.40 

0.548 

3.50 

0.548 

3.60 

0.548 

3.40 

0.894 

3.40 

0.894 

3.40 

0.894 

3.20 

0.837 

70 

 
 
 

 

Goals. The mean Relevance rating of all items in the Goals section for Round 2 was 3.71 

(SD = 0.329), and the mean Utility rating of all items in the Goals section for Round 2 was 3.68 

(SD = 0.372). The highest rated Goal item was Incorporating client voice into the goal setting 

process, which was rated as Extremely Relevant and Useful by all participants in this round (M = 

4.00; SD = 0.00). There were equal amount of items rated more highly on Relevance and Utility 

(n = 3), with the item Allowing clients to take the lead in identification of potential goals being 

rated Extremely Useful by all participants (M = 4.00; SD = 0.00), but representing some 

variation in Relevance (M = 3.80; SD = 0.447). The lowest rating was the Utility of the item 

Developing SMART (Specific, Measurable, Achievable, Relevant/Results-Based, Time-sensitive) 

goals that are mutually agreed upon (M = 3.20; SD = 0.837). 

 

 
 

Table 4.3 
 
Round 2: Goals (Descending by mean Relevance) 

 

Item 

Incorporating the client voice into the goal 
setting process 

Demonstrating empathy in understanding the 
various influences which have impacted the 
client's goals and experiences (i.e., cultural 
background, socioeconomic status, etc.) 

Mutual cooperation with the client when 
establishing and implementing the treatment 
plan 

Allowing clients to take the lead in the 
identification of potential goals 

Relevance 

Utility 

Mean 

SD 

Mean 

SD 

4.00 

0.00 

4.00 

0.00 

3.83 

0.408 

3.80 

0.447 

3.80 

0.447 

3.80 

0.447 

3.80 

0.447 

4.00 

0.00 

Collaborating with clients in the development 
of mutually agreed upon goals 

3.80 

0.447 

3.80 

0.447 

71 

 
 
 

 

 

Table 4.3 (cont’d) 
 
Applying theory to a situation in order to 
provide guidance in the goal setting process 

Developing goals that follow a strengths 
based approach, and identifies potential in 
clients (as opposed to focusing solely on 
dysfunction) 

Being able to facilitate goal development in 
areas that may be in conflict with the 
counselor’s personal values 

Developing SMART (Specific, Measurable, 
Achievable, Relevant/Results-Based, Time-
sensitive) goals that are mutually agreed upon 

 

3.60 

 

0.894 

 

3.40 

 

0.894 

3.50 

0.837 

3.60 

0.894 

3.40 

0.548 

3.60 

0.548 

3.40 

0.894 

3.20 

0.837 

Bonds. Mean overall scores of Relevance and Utility in the Bonds section were higher 

when compared with the other sections of the instrument (M = 3.79, SD = 0.245; M = 3.74, SD = 

0.305, respectively). There were seven items that were rated as Extremely Relevant by all 

participants in Round 2 (n = 7), and three of those were also rated as Extremely Useful by all 

participants (M = 4.00; SD = 0.00). The lowest rating was the Utility of the item Willingness to 

advocate for clients of minority backgrounds who experience institutional discrimination (M = 

3.20; SD = 0.447). Twelve items were rated the same on both Utility and Relevance, while five 

were rated more highly on Relevance than Utility. 

 

 

 

 

 

 

 
 

72 

 
 
 

 
 

Table 4.4 
 
Round 2: Bonds (Descending by mean Relevance) 

 

Item 

Adherence to ethical practice for respecting 
client’s informed consent and other client 
rights in order to develop trust, boundaries, 
and transparency in the counseling 
relationship 

Communicating in a confidential, responsive, 
and empathic manner to establish rapport in a 
way that promotes openness and sensitivity to 
potential cultural differences 

Demonstrating unconditional positive regard 
for clients 

Using active and reflective listening to ensure 
effective collaboration, problem-solving, and 
decision-making 

Demonstrating ethical behavior  in the 
development of bonds with clients (i.e., 
appropriate boundaries, etc.) 

Addressing multicultural issues when 
presented in the session that may affect the 
counseling relationship or the client's ability 
to pursue a goal 

Actively avoiding cultural biases and 
discriminatory practices in working with  
clients of minority backgrounds 

Maintaining a nonjudgmental disposition 
regarding client values 

Establishing trust with the client as evidence 
by the communication that occurs between 
the counselor and client 

Capacity to appropriately communicate 
acceptance to the client 

Relevance 

Utility 

Mean 

SD 

Mean 

SD 

4.00 

0.00 

4.00 

0.00 

4.00 

0.00 

3.80 

0.447 

4.00 

0.00 

4.00 

0.00 

4.00 

0.00 

4.00 

0.00 

4.00 

0.00 

3.80 

0.477 

4.00 

0.00 

3.80 

0.477 

4.00 

0.00 

3.80 

0.477 

3.80 

0.477 

3.80 

0.477 

3.80 

0.477 

3.80 

0.477 

3.80 

0.477 

3.80 

0.477 

73 

 
 
 

 

Table 4.4 (cont’d) 
 
Ensuring client autonomy 

 
 

 
 

 
 

 
 

3.80 

0.477 

3.80 

0.477 

To demonstrate openness and flexibility when 
addressing the clients issues and problems 

3.60 

0.548 

3.60 

0.548 

Being capable of adjusting interactions with 
clients to meet his/her individual needs and 
communication style 

Identification of client’s needs and problems 
in congruence with his/her priorities 

Managing the power differential between 
counselor and client 

Sustaining the effort to help a client whether 
or not he/she makes progress 

Willingness to advocate for clients of 
minority backgrounds who experience 
institutional discrimination 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

3.60 

0.548 

3.80 

0.477 

3.40 

0.548 

3.40 

0.894 

3.40 

0.894 

3.20 

0.447 

Phase 1: Round 3 

Participants in Round 3 were again asked to rate items in each section on a 4-point likert 

scale in Relevance and Utility (1 = Not at all useful/relevant; 4 = Extremely useful/relevant), 

after having the opportunity to review their previous scores, as well as the group means and 

standard deviations. Responses were then assessed for mean utility and relevance, and standard 

deviations. It was predetermined that any items which fell below a mean of 3 in both relevance 

and utility would be eliminated from the subsequent round.  

Professional Dispositions. The overall mean scores of Relevance and Utility for 

Professional Dispositions in Round 3 were 3.63 (SD = 0.383), and 3.52 (SD = 0.482), 

respectively. Relevance and Utility ratings did not change much from Round 2 to Round 3, but 

the Utility ratings that changed, most often decreased by a tenth or twentieth of a point. On the 

 
 

74 

 
 
 

other hand, most of the Relevance ratings that changed from round to round increased by a tenth 

or twentieth of a point (Table 4.5). The lowest rated item in both Relevance and Utility was 

Awareness of his/her role as a counselor, including self-awareness, humility, and integrity 

(Relevance: M = 3.40, SD = 0.894; Utility: M = 3.40, SD = 0.894). Overall, the changes from 

Round 2 to Round 3 were not substantial, and no items fell below the threshold of 3 in both 

Relevance and Utility in this round. 

Table 4.5 
 
Round 3: Professional Dispositions (Descending by mean Relevance) 

 

Item 

A dedication to, and embodiment of, the 
ethical values (autonomy, justice, veracity, 
beneficence, nonmaleficence, fidelity) 

Showing unconditional acceptance of all 
clients, peers, or coworkers regardless of their 
demographics (i.e., age, gender, race, ethnicity, 
sexual identity/orientation, culture) 

Relevance 

Utility 

Round 2 

Round 3 

Round 2 

Round 3 

Mean 
(SD) 

Mean 

SD 

Mean 
(SD) 

Mean 
(SD) 

4.00 

4.00 

4.00 

3.60 

(0.000) 

(0.000) 

(0.000) 

(0.894) 

3.83 

4.00 

4.00 

3.80 

(0.408) 

(0.000) 

(0.000) 

(0.447) 

Understanding the tendency and the problem 
of racial stereotyping 

3.83 

3.80 

3.60 

3.60 

(0.408) 

(0.447) 

(0.548) 

(0.548) 

The development of a counselor identity (e.g., 
theoretical orientation, helping disposition, 
professional advocacy) 

Demonstrates professional and personal 
maturity such as accepting feedback, following 
through on commitments, commitment to 
professional growth 

Sufficient professional interpersonal skills (i.e., 
developing personal rapport with coworkers, 
appropriate tone of voice, language use, use of 
humor, appropriate dress, etc.) 

3.80 

3.80 

3.60 

3.60 

(0.447) 

(0.447) 

(0.548) 

(0.548) 

3.80 

3.80 

3.60 

3.50 

(0.447) 

(0.447) 

(0.548) 

(0.577) 

3.60 

3.60 

3.60 

3.60 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

 
 

75 

 
 
 

 

Table 4.5 (cont’d) 
 
Maintains a respectful countenance in all 
interactions: perceives and honors diversity, 
boundaries, and appropriate communication 
style 

Engaging in difficult conversations with 
clients, coworkers, and/or supervisors in an 
appropriate manner 

Commitment to a career in the counseling field 
via indications of a desire to be a lifelong 
learner (e.g., always seeking new information 
and resources, participation in professional 
organizations) 

Demonstrates professionalism and professional 
behavior in interactions with peers, 
supervisors, clients, and as a representative of 
their educational program 

Sufficient professional administrative skills 
(i.e., punctuality, organizational skills, 
preparation, professional written 
communication/documentation, awareness of 
policy/procedures) 

 

3.60 

 

3.60 

 

3.60 

 

3.60 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

3.50 

3.60 

3.60 

3.40 

(0.548) 

(0.894) 

(0.548) 

(0.548) 

3.50 

3.60 

3.40 

3.20 

(0.837) 

(0.548) 

(0.894) 

(0.837) 

3.40 

3.60 

3.40 

3.60 

(0.894) 

(0.548) 

(0.894) 

(0.548) 

3.67 

3.40 

3.40 

3.40 

(0.516) 

(0.548) 

(0.894) 

(0.548) 

Use of problem-solving skills in a timely and 
professional fashion 

3.60 

3.40 

3.60 

3.40 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

Awareness of his/her role as a counselor, 
including self-awareness, humility, and 
integrity 

3.40 

3.40 

3.40 

3.40 

(0.894) 

(0.894) 

(0.894) 

(0.894) 

Tasks. The overall means for Relevance and Utility of items in the Tasks section for 

Round 3 were 3.57 (SD = 0.345) and 3.47 (SD = 0.321), respectively. These scores are slightly 

lower than the overall means in Round 2 (M = 3.68, SD = 0.356; M = 3.62, SD = 0.356). There 

were several items that decreased on both Relevance and Utility (Table 4.6). The lowest rated 

item in this round was the Utility rating of Being able to apply theory to justify rationale behind 

assigned tasks (M = 3.00,  SD = 1.000). 

 
 

76 

 
 
 

 
 

Table 4.6 
 
Round 3: Tasks (Descending by mean Relevance) 

 

Item 

Relevance 

Utility 

Round 2 

Round 3 

Round 2 

Round 3 

Mean 
(SD) 

Mean 
(SD) 

Mean 
(SD) 

Mean 
(SD) 

Ability to collaborate with clients to design 
tasks that are congruent with the mutually 
agreed upon goal 

4.00 

4.00 

3.80 

3.80 

(0.000) 

(0.000) 

(0.447) 

(0.447) 

Ability to identify tasks that will lead to goal 
attainment 

3.80 

3.80 

3.80 

3.80 

(0.447) 

(0.447) 

(0.447) 

(0.447) 

Developing options to address clients’ needs 
and problem with identified 

3.80 

3.80 

3.80 

3.80 

(0.447) 

(0.447) 

(0.447) 

(0.447) 

Adjusting with the client as his/her needs and 
problems evolve during the counseling process 

3.80 

3.80 

3.60 

3.40 

(0.447) 

(0.447) 

(0.548) 

(0.548) 

Ability to communicate the link between the 
chosen task and the agreed upon goal 

3.67 

3.60 

3.80 

3.60 

(0.516) 

(0.548) 

(0.447) 

(0.548) 

Being able to show support when holding 
clients accountable for task completion, or 
non-completion 

3.60 

3.60 

3.60 

3.60 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

Application of Evidence-Based Practices 
(EBP) 

3.60 

3.60 

3.40 

3.40 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

Knowing how to consult or refer to resources 
available in ethnocultural communities 

3.60 

3.40 

3.80 

3.40 

(0.548) 

(0.548) 

(0.447) 

(0.548) 

Ability to design tasks to meet the unique 
developmental and individual needs of the 
client 

3.50 

3.40 

3.60 

3.40 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

Being able to apply theory to justify rationale 
behind assigned tasks 

3.40 

3.40 

3.20 

3.00 

(0.894) 

(0.548) 

(0.837) 

(1.000) 

Ability to do needs assessments with client in 
order to identify needs that a client may have 
but may not be aware of 

3.67 

3.20 

3.60 

3.20 

(0.516) 

(0.447) 

(0.548) 

(0.447) 

Being capable of helping clients make non-
successes into successful learning experiences 

3.40 

3.20 

3.40 

3.20 

(0.894) 

(0.837) 

(0.894) 

(0.837) 

77 

 
 
 

 

Goals. The overall mean Relevance and Utility ratings for Goals in Round 3 were 3.73 

(SD = 0.181) and 3.64 (SD = 0.130), respectively. This represents an increase in Relevance (up 

from 3.64), but a very slight decrease in Utility from Round 2 (by 1/100th) (Table 4.7). No items 

fell below the threshold of 3 in both Relevance and Utility. 

Table 4.7 
 
Round 3: Goals (Descending by mean Relevance) 

 

Item 

Relevance 

Utility 

Round 2 

Round 3 

Round 2 

Round 3 

Mean 
(SD) 

Mean 
(SD) 

Mean 
(SD) 

Mean 
(SD) 

Incorporating the client voice into the goal 
setting process 

4.00 

4.00 

4.00 

4.00 

(0.000) 

(0.000) 

(0.000) 

(0.000) 

Demonstrating empathy in understanding the 
various influences which have impacted the 
client's goals and experiences (i.e., cultural 
background, socioeconomic status, etc.) 

3.83 

4.00 

3.80 

4.00 

(0.408) 

(0.000) 

(0.447) 

(0.000) 

Collaborating with clients in the development 
of mutually agreed upon goals 

3.80 

4.00 

3.80 

3.80 

(0.447) 

(0.000) 

(0.447) 

(0.447) 

Allowing clients to take the lead in the 
identification of potential goals 

3.80 

3.80 

4.00 

3.80 

(0.447) 

(0.447) 

(0.000) 

(0.447) 

Mutual cooperation with the client when 
establishing and implementing the treatment 
plan 

Developing goals that follow a strengths 
based approach, and identifies potential in 
clients (as opposed to focusing solely on 
dysfunction) 

Being able to facilitate goal development in 
areas that may be in conflict with the 
counselor’s personal values 

3.80 

3.80 

3.80 

3.80 

(0.447) 

(0.447) 

(0.447) 

(0.447) 

3.50 

3.80 

3.60 

3.60 

(0.837) 

(0.447) 

(0.894) 

(0.548) 

3.40 

3.60 

3.60 

3.60 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

Applying theory to a situation in order to 
provide guidance in the goal setting process 

3.60 

3.40 

3.40 

3.20 

(0.894) 

(0.894) 

(0.894) 

(0.837) 

 

 
 

78 

 
 
 

 

Bonds. The overall mean scores for the Bonds section in Round 3 were 3.79 (SD = 

0.206) (Relevance) and 3.74 (SD = 0.153) (Utility). This did not change from Round 2 to Round 

3. There was an increase in the unanimous ratings of Extremely Relevant and Useful; in Round 

3, eight items were rated both Extremely Relevant and Useful by all participants (M = 4.00, SD = 

0.000). Several item ratings changed from Round 2 to Round 3 (Table 4.8). There was a decrease 

in both Relevance and Utility on the item Identification of client’s needs and problems in 

congruence with his/her priorities, making it the lowest rated item during this round (Relevance: 

M = 3.40, SD = 0.548; Utility: M = 3.40, SD = 0.548). No items fell below the threshold of 3 on 

both Relevance and Utility. 

 

 
 

Table 4.8 
 
Round 3: Bonds (Descending by mean Relevance) 

 

Item 

Relevance 

Utility 

Round 2 

Round 3 

Round 2 

Round 3 

Mean 
(SD) 

Mean 
(SD) 

Mean 
(SD) 

Mean 
(SD) 

Using active and reflective listening to ensure 
effective collaboration, problem-solving, and 
decision-making 

4.00 

4.00 

4.00 

4.00 

(0.000) 

(0.000) 

(0.000) 

(0.000) 

Demonstrating unconditional positive regard 
for clients 

4.00 

4.00 

4.00 

4.00 

(0.000) 

(0.000) 

(0.000) 

(0.000) 

Adherence to ethical practice for respecting 
client’s informed consent and other client 
rights in order to develop trust, boundaries, and 
transparency in the counseling relationship 

 
Actively avoiding cultural biases and 
discriminatory practices in working with  
clients of minority backgrounds 

4.00 

4.00 

4.00 

4.00 

(0.000) 

(0.000) 

(0.000) 

(0.000) 

 

4.00 

 

4.00 

 

3.80 

 

4.00 

(0.000) 

(0.000) 

(0.477) 

(0.000) 

79 

 
 
 

 

 
 

Table 4.8 (cont’d) 
 
Addressing multicultural issues when 
presented in the session that may affect the 
counseling relationship or the client's ability to 
pursue a goal 

Demonstrating ethical behavior in the 
development of bonds with clients (i.e., 
appropriate boundaries, etc.) 

Ensuring client autonomy 

4.00 

4.00 

3.80 

4.00 

(0.000) 

(0.477) 

(0.000) 

4.00 

4.00 

3.80 

4.00 

(0.000) 

(0.000) 

(0.477) 

(0.000) 

3.80 

4.00 

3.80 

4.00 

(0.477) 

(0.000) 

(0.477) 

(0.000) 

Maintaining a nonjudgmental disposition 
regarding client values 

3.80 

4.00 

3.80 

4.00 

(0.477) 

(0.000) 

(0.477) 

(0.000) 

Establishing trust with the client as evidence 
by the communication that occurs between the 
counselor and client 

Communicating in a confidential, responsive, 
and empathic manner to establish rapport in a 
way that promotes openness and sensitivity to 
potential cultural differences 

3.80 

3.80 

3.80 

3.80 

(0.477) 

(0.447) 

(0.477) 

(0.447) 

4.00 

3.60 

3.80 

3.60 

(0.000) 

(0.548) 

(0.477) 

(0.548) 

Capacity to appropriately communicate 
acceptance to the client 

3.80 

3.60 

3.80 

3.60 

(0.477) 

(0.548) 

(0.477) 

(0.548) 

Managing the power differential between 
counselor and client 

3.60 

3.60 

3.80 

3.60 

(0.548) 

(0.548) 

(0.477) 

(0.548) 

Being capable of adjusting interactions with 
clients to meet his/her individual needs and 
communication style 

3.60 

3.60 

3.60 

3.40 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

To demonstrate openness and flexibility when 
addressing the clients issues and problems 

3.60 

3.60 

3.60 

3.60 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

Willingness to advocate for clients of minority 
backgrounds who experience institutional 
discrimination 

3.40 

3.60 

3.20 

3.40 

(0.894) 

(0.548) 

(0.447) 

(0.548) 

Sustaining the effort to help a client whether or 
not he/she makes progress 

3.40 

3.60 

3.40 

3.20 

(0.548) 

(0.548) 

(0.894) 

(0.447) 

Identification of client’s needs and problems in 
congruence with his/her priorities 

3.60 

3.40 

3.60 

3.40 

(0.548) 

(0.548) 

(0.548) 

(0.548) 

80 

 
 
 

 

Results from Phase 2 intend to provide insight into the questions: (1) How much do those 

Phase 2 

items relate to the concepts of WA (i.e., goals, tasks and bonds)? (2) How useful are those items 

for counselor educators? 

 

Relevance. Participants rated items on their relevance using a 4-point Likert scale (1 = 

Not at all relevant; 4 = Extremely relevant). Each subscale was scored and then assessed for 

internal consistency using Cronbach’s alpha. 

 

Professional dispositions. The maximum possible score for this section was 52 (13 items, 

maximum score of 4 per item). The mean score was 47.84 (SD = 4.48), the median was 49.00, 

the mode was 52.00, and the range 25.00 (N = 148). 26.4% of participants rated all items 

Extremely Relevant (n = 39). The scores for Professional Disposition Relevance are represented 

in Figure 4.1. Cronbach’s alpha (ɑ = 0.877) for Professional Disposition Relevance represents 

strong internal consistency among items in this subscale; removal of any item from this section 

would decrease the alpha level.  

Figure 4. 1: Professional Disposition Relevance Score Frequency Histogram 

 
 

81 

 
 
 

Between Groups. The Bonds Utility sub-scale scores were not statistically different when 

explored comparing educators to non-educators, t(146) =- 0.634, p = 0.527. 

Tasks. The maximum possible score for the Task section was 40 (10 items, maximum 

score of 4 per item). The mean score was 36.35 (SD = 3.35), the median was 37.00, the mode 

was 40.00, and the range was 13.00 (N = 147). 20.4% of participants rated all items Extremely 

Figure 4. 2: Task Relevance Score Frequency Histogram 

Relevant (n = 30). The scores for Task Relevance are represented in Figure 4.2.  Cronbach’s 

alpha for the Task Relevance subscale was 0.809, which provides evidence of a strong internal 

consistency. The removal of any item would not increase the alpha coefficient. 

Between Groups. The Task Relevance sub-scale scores were not statistically different 

when explored comparing educators to non-educators, t(145) = 0.646, p = 0.519. 

 
 

82 

 
 
 

Figure 4. 3: Goal Relevance Score Frequency Histogram 

 

 

Goals. The maximum possible score in the Goals section was 32 (8 items, maximum 

score of 4 per item). The mean score was 29.21 (SD = 2.49), the median was 30, the mode was 

31.00, and the range was 12.00 (N = 147). The most common score in this section was 31, with 

21.1% of participants (n = 31). Scores for Goal Relevance are represented in Figure 4.3. 

Cronbach’s alpha for the Goal Relevance subscale represented satisfactory internal consistency 

(ɑ = 0.718). Removal of the item Applying theory to a situation in order to provide guidance in 

the goal setting process would lead to an increase in Cronbach’s alpha to 0.754. 

Between Groups. The Goals Relevance sub-scale scores were not statistically different 

when explored comparing educators to non-educators, t(145) = -0.925, p = 0.357. 

Bonds. The maximum possible score in the Bonds section was 68 (17 items, maximum 

score of 4 per item). The mean score was 63.46 (SD = 5.51), the median was 65.00, the mode 

was 68.00, and the range was 34 (N = 147). 29.3% of participants scored all items Extremely 

 
 

83 

 
 
 

Relevant (n = 43). Scores are represented in Figure 4.4. Cronbach’s alpha for Bond Relevance 

was very strong, at 0.908. The removal of any items would decrease the alpha level. 

Between Groups. The Bonds Relevance sub-scale scores were not statistically different 

when explored comparing educators to non-educators, t(145) = 0.121, p = 0.904. 

Figure 4. 4: Bond Relevance Score Frequency Histogram 

 

Utility. Participants rated items on their utility using a 4-point Likert scale (1 = Not at all 

useful; 4 = Extremely useful). Each subscale was scored and then assessed for internal 

consistency using Cronbach’s alpha. 

Professional Dispositions. The maximum possible score for Professional Disposition 

Utility was 52. Mean score was 47.42 (SD = 4.37), the median was 48.00, the mode was 52.00, 

and the range was 21.00 (N = 148). Cronbach’s alpha of the Professional Disposition Utility 

subscale indicates a strong internal consistency (ɑ = 0.850). The removal of any item would only 

decrease the alpha level.  

 
 

84 

 
 
 

 
 

Figure 4. 5: Professional Disposition Utility Score Frequency Histogram 

 

Between Groups. The Professional Dispositions Utility sub-scale scores were not 

statistically different when explored comparing educators to non-educators, t(146) = 0.236, p = 

0.814. 

 

Figure 4. 6: Task Utility Score Frequency Histogram 

 
 

85 

 
 
 

Tasks. The maximum possible score in this section was 40. The mean score was 36.11 

(SD = 3.72), the median score was 37.00, the mode was 40.00, and the range was 16 (N = 147). 

Scores are represented in Figure 4.6. 21.8% of the respondents indicated that all items were 

Extremely Useful (n = 32). Cronbach’s alpha revealed a strong internal consistency in the Task 

Utility section, ɑ = 0.836. This analysis also revealed that the alpha level may increase to 0.841 if 

the item Being able to apply theory to justify rationale behind assigned tasks is removed from 

the scale. 

Between Groups. The Task Utility sub-scale scores were not statistically different when 

explored comparing educators to non-educators, t(145) = 1.411, p = 0.161. 

 

 

 

Figure 4. 7: Goal Utility Score Frequency Histogram 

Goals. The maximum possible score in this section was 32. The mean score was 28.74 

(SD = 2.89), with a median of 29, a mode of 32.00, and a range of 12 (Figure 4.7). Cronbach’s 

alpha of the Goal Utility subscale was 0.769. Removal of the item Applying theory to a situation 

in order to provide guidance in the goal setting process would increase the alpha level to 0.803. 

 
 

86 

 
 
 

Between Groups. The Goals Utility sub-scale scores were not statistically different when 

explored comparing educators to non-educators, t(145) = 0.529, p = 0.598. 

Bonds. The maximum possible score in this section was 68. The mean score was 62.93 (SD = 

5.87), the median score was 65.00, the mode was 68, and the range was 34.00. Cronbach’s alpha 

of the Bond Utility subscale was 0.912. The removal of any item would not increase the alpha 

level. 

 

Between Groups. The Bonds Utility sub-scale scores were not statistically different when 

explored comparing educators to non-educators, t(145) = 0.749, p = 0.429. 

 

 

 

 

 
 

 
 

Figure 4. 8: Bond Utility Score Frequency Histogram 

87 

 
 
 

Qualitative Comments 

 

Participants were provided the opportunity to comment on instrument items at the end of 

each section and were asked to specifically indicate which item number they were referencing. 

The comment section was intended to capture concerns about particular items, comments on 

wording of items, and/or explanations for particular ratings. While the results were collected by 

section, they will be discussed here as aggregate data. 

There were 36 usable comments submitted from the 148 total participants. This is after 

the removal of 10 comments that were not usable, including comments of “none,” “NA,” or 

those that only submitted an item number with no other information. Many comments (n = 17) 

included general statements about the importance of the items or of the participant’s experience 

with that particular concept. Examples of these include: 

●  Rapport is a critical piece of counseling. All of these items are absolutely 

necessary; trust and lack of judgment cannot be underestimated in the counseling 

relationship. 

●  #2 [Showing unconditional acceptance of all clients, peers, or coworkers 

regardless of their demographics (i.e., age, gender, race, ethnicity, sexual 

identity/orientation, culture)] is very important that all counseling professionals 

should show unconditional acceptance to all people regardless of age, gender, 

race, ethnicity, and sexual orientation. 

●  Every quality listed above is extremely important. Without them, counselors, 

regardless of field, can cause potentially significant harm to a client, client's 

significant others, the agency's credibility, the credibility/relationship of the 

referral source, and the social image of counseling. 

 
 

88 

 
 
 

 

●  Collaboration in developing client goals is essential. Clients are more willing to 

participate in activities that lead to goal attainment when it is something they 

have agreed upon.  

An additional theme of the qualitative results include comments about the items which 

pertain to theory in practice (n = 6). These comments coincide with ratings of these items 

previously discussed in this chapter. Examples of these comments include: 

●  Theoretical orientation/counselor identity is fluid for most counselors and students I 

encounter - this is an ongoing development and self-discovery process. While theories 

and their application can be useful (high utility), I do not believe this to be crucial to a 

successful career in a rehabilitation field. 

●  #6 [Applying theory to a situation in order to provide guidance in the goal setting 

process], applying a theory, falls in line with the first comment; while theory can be 

useful in framing, use of a specific theory or theories is not necessarily requisite in 

service provision. 

●  Goal obtainment [sic] is extremely important, as all the theory in the world is worth 

nothing if you can obtain your goals. 

 

One comment was applicable to a specific item, in the Goals section: 8. And being able to 

refer out if this is not practical or violates personal values. The remaining comments could not 

be themed due to participants providing information that is excessive, unrelated to the research 

questions, or does not provide a means to change an item. 

Summary 

  

In general, scores across all items were rated very highly on both Relevance and Utility, 

throughout Phases 1 and 2. Most often, changes in average ratings of Relevance increased from 

 
 

89 

 
 
 

Round 2 to Round 3 (in Professional Dispositions and Goals), while average ratings of Utility 

decreased in the same categories from Round 2 to 3. In the Tasks category the ratings decreased 

in both Relevance and Utility from Round 2 to 3 and remained consistent in the Bonds category. 

Increases or decreases did not exceed a tenth or twentieth of a point, however, therefore overall, 

the items remained consistent and never were rated below the threshold of 3 on either Relevance 

or Utility. 

 

Similarly, in Phase 2, items were rated highly, and the subscale scores represented high 

ratings even throughout the larger sample. Furthermore, the modes, or most frequently occurring 

score in Professional Dispositions, Tasks, and Bonds were the maximum possible scores for that 

particular subsection (i.e., participants rated all items Extremely Relevant and Extremely 

Useful). Consistently, across both Phase 1 and 2, items regarding theory in practice were rated 

the lowest, and were indicated as increasing the Cronbach’s alpha if removed. These ratings will 

be discussed further in Chapter 5.  

 

 

 

90 

 
 

 
 
 

CHAPTER FIVE 

Discussion 

Phase 1 of the current study established the items necessary to begin an inquiry into what 

is appropriate for the evaluation of professional dispositions in pre-service counselor education 

students. The initial iteration of items were then tested in Phase 2 by a large sample of CRCs, 

both in education and practice. In general, items were rated very highly in Relevance and Utility, 

and those ratings remained consistent over time.  

The current study intended to answer three research questions: 

1.  What items would be appropriate for faculty to use in the evaluation of 

professional dispositions in master’s level counseling students?  

2.  How much do those items relate to the concepts of WA (i.e., goals, tasks and 

bonds)?  

3.  How useful are those items for counselor educators? 

The following chapter includes a discussion of the findings of the current study, including 

an overview of all findings, comparison with similar instruments, future directions for AGI, and 

implications for research, education, and clinical practice. 

Summary of Findings 

Phase 1. Results of Phase 1 of this study provided 48 items for the instrument (by 

section: Professional Dispositions: 13; Tasks: 10, Goals: 8, and Bonds: 17) named the 

Assessment as Growth Inventory (AGI), which provided the answer to Research Question 1. 

These items were developed using a Delphi methodology, and an expert panel. In addition to 

developing the items for the instrument, an iterative process allowed for there to be feedback on 

the item beyond the numeric ratings. At the end of both Rounds 2 and 3 of the Delphi, all items 

 
 

91 

 
 
 

remained highly rated, with minimal changes from round to round. There was a slight increase in 

relevance but a decrease in utility ratings from Round 2 to Round 3, which may be due to 

considerations of how the items would be used in practice. Many items could be considered 

relevant (hypothetically), but to really consider how they would be useful to a counselor educator 

may be more complicated or hard to assess, such as Applying theory to a situation in order to 

provide guidance in the goal setting process, or Identification of client’s needs and problems in 

congruence with their priorities. At any given point of the study, no item was rated below the 

cutoff threshold of 3 (on a 4-point Likert Scale) on both Relevance and Utility in assessment of 

pre-service counseling students. Given the consistently high ratings, three rounds were 

considered sufficient to move onto Phase 2, in order to collect further information about the 

instrument items and to be able to assess psychometric properties. 

The highest rated items in each section included items about ethical behavior, client-

centered, non-judgmental practice, and/or empathetic qualities. Lower rated items were more 

variable, but also included specific qualities such as: goal-setting skills, maintenance of support 

through client failure, and role and professional identity as a counselor. While there were 

‘highest’ and ‘lowest’ rated items, on average, all were rated nearest to Extremely Relevant 

and/or Useful. 

 

Phase 2. Phase 2 provided input from CRCs both in education and practice and revealed 

a similar agreement regarding Relevance and Utility of the items developed in Phase 1. The 

cutoff point was determined to be a rating of 3 on both Relevance and Utility. As with Phase 1, 

Phase 2 did not reveal any item to have a mean rating of below 3 on Relevance and Utility, as 

well as Relevance or Utility, therefore all 48 items developed through Phase 1 are to be retained 

for future use of the AGI.  

 
 

92 

 
 
 

While Phase 1 included several instrument items being rated Extremely Relevant and/or 

Useful by all participants, there was more variability in Phase 2. Ratings of Relevance were 

consistently higher than ratings of Utility per item in Phase 2, however these differences were 

approximately a tenth to a hundredth of a point, making the differences quite minimal. Items 

were rated on their relevance to the particular category in which they were distributed, lending 

validity to the organization of the instrument. In terms of how useful each item would be in the 

evaluation of students in master’s-level counseling programs, all items were rated highly useful, 

although ratings of utility were typically not higher than the ratings of relevance per item.  

Phase 2 included sub-scores of each section, in addition to examining each item 

individually, as was done in Phase 1. Upon an inquiry into the internal consistency of each 

subscale (Professional Dispositions, Tasks, Goals, Bonds) in regard to Relevance and Utility, 

Cronbach’s alpha provided evidence that each subscale demonstrated strong reliability, with each 

sections’ α ranging from 0.718 to 0.912 (Bland & Altman, 1997).   

 

The current format of gatekeeping in counselor education programs, especially in the 

realm of non-academic competencies, tends to be reactive and conducted in an ad-hoc format 

(Bhat, 2005; Brear & Dorrian, 2010). Additionally, gatekeeping and student evaluation has been 

historically unfavorable amongst faculty, and poorly explained to students throughout their time 

in a program (J. Foster et al., 2014; V. Foster & McAdams, 2009). In light of such a situation, the 

theoretical and philosophical design underlying the AGI is such that the process of evaluating 

students’ professional dispositions is systematic, transparent, and designed to ensure a 

collaboration between student, faculty, and supervisors. Furthermore, by aligning the evaluation 

with the WA, the AGI provides the opportunity for parallel process, and a better understanding 

of a significant evidence-based practice. 

 
 

93 

 
 
 

Discussion of the Findings 

Items within each section of the instrument covered a variety of areas within counselor 

education in which educators are expected to evaluate students but are “non-academic” in nature. 

Items also echo the CRCC Code of Ethics (2017) and the values and principles espoused 

including “ensuring the integrity of all professional relationships,” and “enhancing the quality of 

professional knowledge and its application to increase professional and personal effectiveness” 

(p. 2). Many of the items representing specific sentiments of the CRCC and ACA Codes of 

Ethics, such as maintenance of ethical practice, or appropriate boundaries with clients, were 

consistently some of the highest rated items in the study. Additionally, items referring to client-

centeredness, non-judgmental practice and multicultural competence were consistently highly 

rated, which would also align with both Codes of Ethics, as well as the values espoused by 

CRCC (e.g., A dedication to, and embodiment of, the ethical values (autonomy, justice, veracity, 

beneficence, nonmaleficence, fidelity), Adjusting with the client as his/her needs and problems 

evolve during the counseling process, Collaborating with clients in the development of mutually 

agreed upon goals, and Demonstrating ethical behavior in the development of bonds with clients 

(i.e., appropriate boundaries, etc.)). Item alignment with the Code of Ethics remained consistent 

from Phase 1, suggesting that these items are not only valuable to educators/experts, but also to 

counselors working in clinical practice settings. It is possible to consider, however, that these 

items were rated in this way as a result of response and/or sample bias. While the survey was 

anonymous, it is still possible that participants believe that pre-service counseling students 

should be being assessed on certain dispositions, rather than considering the practical use of such 

items. In the current study, the majority of participants indicated that they do not currently 

engage in the supervision of preservice counseling students. Additionally, it seems that the 

 
 

94 

 
 
 

supervision of pre-service counseling students is perceived as happening less frequently when 

compared with other job functions of rehabilitation counselors (Leahy, Chan, Sung, & Kim, 

2013). As such, the sample of respondents may simply have responded highly to the items that 

they consider idealistic, rather than a consideration of what items would be useful for such an 

instrument as the AGI. 

On the other hand, the lowest rated items through both Phase 1 and Phase 2 were 

consistently items pertaining to the use of theory in practice (e.g., Applying theory to a situation 

in order to provide guidance in the goal setting process, and Being able to apply theory to justify 

rationale behind assigned tasks). Furthermore, upon review of the Cronbach’s alpha test of 

internal consistency reliability, the removal of the item Applying theory to a situation in order to 

provide guidance in the goal setting process would increase the alpha level in both Relevance 

and Utility in the Goals section, and removal of the item Being able to apply theory to justify 

rationale behind assigned tasks would increase the alpha in Utility in the Tasks section. Again, 

when compared to the Leahy et al. (2013) study reporting the current perceived job functions and 

knowledge domains of rehabilitation counselors, the application of theory to rehabilitation falls 

below the average in perceived importance and perceived frequency of use. In light of this 

comparison, it is possible that, in general, CRCs feel removed from their educational preparation, 

and theory is not something they consult regularly in their day-to-day practice. These items may 

have been rated lower because participants feel less familiar with, or dedicated to, practical 

application of theory—they also may not feel that they would be able to assess a student on such 

competencies. As the current study is the initial instrument development process, these items will 

not be removed. These items will continually be tested in future studies of the instrument, 

including through factor analysis, to determine their appropriateness, or the necessity of their 

 
 

95 

 
 
 

removal. Future inquiries may also be necessary to explore perceptions of theory in practice in 

rehabilitation counseling. 

Items that, while rated as important, but fell to the middle of their category included 

components that participants may feel are difficult to assess, including maturity, engaging in 

difficult conversations, communication skills, use of evidence-based practices, facilitating goal 

development, and managing power differentials. It is not surprising that there would be 

participants who would rate items such as these slightly lower than average; the literature has 

demonstrated a general uneasiness in assessing such competencies, and even the uncertainty of 

being able to assess such competencies in the context of an academic program (Kerl, et al., 2002; 

Gaubatz & Vera, 2002; McAdams, Foster, V., & Ward, 2007). 

An additional consideration to be made regarding the items is that they may not all be 

applicable, all the time. For instance, items pertaining to working directly with a client, or 

knowledge of particular concepts, may not be applicable within the first semester of a student’s 

educational program. It is possible that faculty may choose to use particular items at an earlier 

stage, and other items at a later stage; this may be explored in future research. It should also be 

noted that the intention of the AGI was not to be used during the application process or 

admissions procedure. It would be unreasonable to expect faculty to be able to evaluate many of 

the components of the instrument during that process, and would be unfair to potential students. 

The AGI and other instruments. Tate, Bloom, Tassara, and Caperton (2014) conducted 

a review of 41 instruments that have been published for the use of measuring constructs related 

to counselor competence and/or counseling skills. Most of the instruments reviewed focused 

specifically on one competence domain (e.g., multicultural counseling), which underscores the 

issues highlighted by Ridley et al., 2011: there is a focus on specific microskills which can 

 
 

96 

 
 
 

hinder counselor development. Additionally, in the instruments reviewed by Tate et al. (2014), 

there was a significant emphasis on instruments using self-report, and those assessing self-

efficacy. 

Tate and colleagues (2014) note that there were fewer instruments based on expert-based 

evaluations, and that “Use of a psychometric measure of counselor competence would 

specifically benefit the quality of student evaluations by clinical supervisors who are supervising 

students’ practica and internships” (Tate et al., 2014, p. 302). The AGI, in its underlying 

theoretical framework, development, and intended use, is not necessarily intended to identify 

counseling competencies. In aligning with the CACREP definition of professional dispositions, 

AGI seeks to provide a mechanism for educators and supervisors to evaluate students beyond 

their counseling skills, in order to more comprehensively develop well-rounded future 

professionals as well as effective counselors. This bridges the shortcomings identified by Tate et 

al. (2014), and those left by instruments such as those described by Kerl et al. (2002), 

Moorehouse (2008), as well as the Counseling Competencies Scale developed by the University 

of Central Florida’s Counselor Education Faculty (unpublished instrument). 

 

The combination of counselors in practice and counselor educators is significant way that 

the AGI differentiates itself from previously published instruments. Feedback from CRCs in 

practice and their enthusiasm for such an instrument indicates that a need may be met via the 

publication and utilization of the AGI. The consistently high ratings of Relevance and Utility on 

all proposed items for the AGI in both Phases 1 and 2 potentially indicates the need for such a 

type of evaluation in counselor education, or more specifically, rehabilitation counselor 

education. This is certainly underscored by the qualitative comments provided at the end of each 

section, including “Every quality above is extremely important,” and “All of these items are 

 
 

97 

 
 
 

absolutely necessary; trust and lack of judgment cannot be underestimated in the counseling 

relationship.” Providing the opportunity to evaluate students on a continuum from Inadequate 

Professional Disposition Development (IPDD) to Exceptional Professional Disposition 

Development (EPDD) also includes a language and a framework for faculty, students, and 

supervisors to work within in order to best understand one another. 

 

While the current study had access to rehabilitation counselors and rehabilitation 

counselor educators, the concepts, items, and direction of the AGI meet needs expressed within 

counselor education. Aligning with the CACREP Standards, and not using language specific to 

any one counseling specialty were intentional elements built into the current study in order to 

encourage applicability across counseling specialties. The working alliance, or therapeutic 

alliance, has been discussed favorably and related to client change in many counseling specialties 

such as career counseling (Whiston, Rossier, & Hernandez Barón, 2015), substance abuse 

(Meier, Donmall, McElduff, Barrowclough, & Heller, 2006), school-based counseling 

(Zirkelback & Reese, 2010), and in mental health counseling (Baldwin, Wampold, & Imel, 2007; 

Wampold, 2001). As such, the AGI would be useful for all counselor educators attempting to aid 

preservice counselors in developing the WA with future clients. 

 

Professional dispositions and the working alliance. Currently, the instrument is broken 

into four distinct sections: Professional Dispositions, Tasks, Goals, and Bonds. For the purpose 

of being a cohesive, unified measure, it may be pragmatic to consider a better integration of the 

concepts of the WA into professional dispositions. Looking at the CACREP definition of 

professional dispositions which is inclusive of both interpersonal functioning and behavior, items 

within the Tasks, Goals, and Bonds sections should also be considered professional dispositions. 

This inclusion of both behavior and interpersonal functioning allows for many items in the 

 
 

98 

 
 
 

additional sections to be considered professional dispositions, as well as functions of the WA. 

Through ongoing studies of professional dispositions, it may become more clear as to how these 

items either are or are not professional dispositions.  

The current study considers all items to have the potential to be representative of the 

CACREP professional dispositions definition, and are organized for clarity with regard to the 

WA. Using the WA framework in the development of the AGI, and its eventual use will work on 

a number of levels. First, in developing an instrument specifically using an evidence-based 

practice at its center, evidence-based practice can be advanced upon, and infused into counselor 

education in a different way. By consistently presenting students with components and examples 

of the WA (via the instrument items), counselor educators are teaching the WA in a more 

dynamic way, and are underscoring its importance. Instrument items should be accessible by all 

students stemming from admittance into a program, allowing them to see the items and ask 

questions when necessary. Furthermore, by learning about and practicing the WA prior to 

reaching practicum and internship, students can have specific items that they are working to 

develop, or that are particularly challenging. This way of identifying specific items within 

components of the WA will help faculty in working with students to embody or enact a particular 

item.   

Towards student collaboration. Discussed in earlier chapters, this study seeks to 

provide faculty with an instrument that fosters student collaboration in the assessment process. 

This also allows the students to learn about the WA while engaging in professional development 

with their advisors, mentors, and supervisors. Research has certainly demonstrated that student 

involvement in gatekeeping is warranted, and desired by students (V. Foster & McAdams, 2009; 

 
 

99 

 
 
 

J. Foster et al., 2014). Furthermore, it has been made clear that faculty and students are not 

having the same experiences in terms of gatekeeping (Gaubatz & Vera, 2006).  

The AGI and its underlying theoretical underpinning of goal setting theory (GST) within 

the framework of social exchange theory (SET) intends to alleviate such issues within 

assessment in counselor education. To reiterate, SET has been described as a frame of reference 

through which other theories can be viewed (Emerson, 1976). Specifically, so long as there is a 

reciprocal process that involves potential for mutual reward and mutual contingencies, SET is 

applicable (Blau, 1964). In terms of the AGI, this might mean that students would feel evaluation 

is being conducted in a manner that encourages their professional development, which in turn 

would deepen their investment in a particular educational program and the process of evaluation. 

GST stipulates that high, hard goals lead to a higher level of performance when those goals are 

realistic, not conflicting, and an individual has the necessary ability to attain the goal (Locke & 

Latham, 2006). The AGI might provide realistic, tangible goals for all students in counselor 

education programs. If such goals (e.g., professional disposition development) are provided to 

students in a transparent way, within a program and a supervisory relationship that the student 

feels is committed to their development (per SET), then students would be more likely to 

consider AGI and its purpose as a ‘set’ goal, rather than an ‘assigned’ goal. While achieving 

“high scores” on the AGI may indicate that a student would be a suitable counselor, it is 

necessary to deter faculty from using evaluative results this way. The AGI intends to provide 

faculty with a means to collaborate with students in the development of particular dispositions 

which will not only help them to be adequate counselors (alongside their coursework, practicum, 

and internships), but also to help to develop them as professionals and future colleagues. 

 
 

100 

 
 
 

While the development of an instrument does not inherently fulfill these theories, the 

ways in which it is distributed and instructed for intended use can encourage use in such a way 

that clearly delineates its correct use. The philosophical beliefs of the current study are an 

implication that will be discussed later in this chapter regarding inter-rater reliability. It is a goal 

of this study to ensure that students are involved in these evaluations, and that they serve to 

foster professional disposition development as well as counseling skills. 

Limitations of the Study 

 

Limitations of the Delphi method include shortcomings in selection of the “expert panel”, 

the time it takes to complete a proper Delphi, and regression to the mean when panelists are able 

to see the responses of their peers (Vázquez-Ramos et al., 2007). Attempts to overcome such 

potential limitations were embedded in the study design outlined in previous chapters, and 

include: comprehensive criterion for selecting experts from various areas that relate to 

gatekeeping and counselor education, the use of technology which should have aided in the 

timeliness of responses, and sensitivity on the part of the researcher in order to effectively 

facilitate responses and maintain panel participation. A limitation inherent in the use of the 

Delphi methodology is attrition of the panel over time; due to the completion of the Delphi 

during the summer, this did occur in the current study. While there is no universally accepted 

size for a Delphi panel, the current study’s panel size was smaller than desired. In addition to the 

small size of the expert panel, the study design did not allow for demographics to be collected 

during each round of the Delphi, potentially causing a loss of useful insights. Future studies 

using this methodology will need to consider implications of timing and commitment of panelists 

(e.g., consider incentives).  

 
 

101 

 
 
 

 

The study design was intentional and intended to ameliorate as many potential limitations 

as possible, but there are limitations including the use of a panel and sample comprised of 

rehabilitation counselors, and rehabilitation counselor educators. Rehabilitation counselor 

educators were more easily accessed for two reasons: (1) the researcher’s position as a student of 

rehabilitation counselor education and ability to contact and/or meet panelists at rehabilitation 

counselor education conferences, and (2) the ability to purchase a large, random sample from 

CRCC. The sample does allow for in-depth testing in one population of counselors, but may not 

be generalizable to other counseling specialties. It is currently unclear, and unexplored, how 

different specialties of counseling view professional dispositions and/or student assessment. 

Additionally, targeting supervisors of pre-service students may also improve the quality of 

responses. Future studies will be needed to better understand these perceptions across various 

counseling programs/specialties. Future studies will also need to address the perception of the 

instrument and items in multicultural populations; the sample of the current study was 

overwhelmingly White and female, which limits generalizability. Additionally, given the 

intended use and philosophy of the instrument, it will not lend itself to traditional psychometric 

tests. The AGI will also require retest reliability and more rigorous research methodologies in 

order to confidently discuss the psychometrics. 

Implications for Research 

There are certain changes to be made to the AGI prior to additional studies using the 

instrument, including the removal of gendered pronouns in all instrument items. The AGI is a 

brand-new instrument that will require significant further exploration, refinement, and pilot 

testing. The current study provides justification for the establishment and use of the initial 

 
 

102 

 
 
 

iteration of items and their organization. Further testing will be needed in a variety of areas, 

including: 

1.  Testing instrument validity and reliability (e.g., construct validity, inter-rater 

reliability, and test-retest reliability) 

2.  Pilot testing and outcome studies with educators, students, and clients 

3.  Validation of items within additional counseling specialties (e.g., clinical mental 

health counseling, school counseling, etc.) 

 

Instrument validation. Future studies will be imperative to the efficacy and 

implementation of the AGI in counselor education. Validation of the content of the instrument 

within additional counseling specialties as well as within a more diverse sample will be 

necessary. Additional testing of the items may also allow for there to be a reduction of items in 

the instrument. Currently, 48 is higher than the intended goal and fewer items would make the 

process less cumbersome for faculty. Additional testing may also provide more clarity regarding 

items that were revealed to be near the cutoff threshold in the current study (e.g., those about 

theory), allowing those to be removed if necessary. Finally, with regard to the instrument’s 

content, future studies may provide insight into “essential dispositions”, or items that represent 

dispositional qualities determined to be absolutely necessary for a graduating counselor to 

embody.   

The current study utilized Cronbach’s alpha in order to assess the internal consistency of 

the items presented in the AGI, however the researcher recognizes that Cronbach’s alpha can be 

a flawed method for assessment of new instruments (Sijtsma, 2009). The current study’s goals 

were to assess the items necessary for assessment in professional disposition development, and to 

identify how well those items were believed to fit within the constructs of the WA. As such, 

 
 

103 

 
 
 

Cronbach’s alpha was the most appropriate statistical analysis for the study. Internal consistency 

is useful in the establishment of instrument items, but in future studies, the AGI will be used for 

assessment of students in counselor education programs, not in asking participants to rate 

instrument items. Going forward, the validation of the scale construction will be necessary to 

determine the underlying factors of the instrument, as well as the internal consistency when the 

instrument is applied in its intended fashion. Exploratory factor analysis will be conducted 

during such a study to see if the current underlying factors of the AGI do in fact align with the 

currently proposed factors of the instrument (professional dispositions, tasks, goals, and bonds). 

At that time, internal consistency will also be a useful measure for establishing reliability of the 

instrument. 

Inter-rater reliability is also significant future direction as it currently is a component that 

is not emphasized in many other published instruments (Tate et al., 2014), and it is the only way 

to ensure that evaluation is systematic both within and across counselor education programs. The 

focus of the AGI is to foster the growth of counselor education students through assessment. 

Therefore, outcome studies and cross-validation with alternate measures of competency will be 

useful in the validation of the AGI. Using established measures of the WA (e.g., WAI), these 

studies may be able to provide insight into criterion-related validity. Understanding the link 

between use of the AGI and a preservice student’s ability to develop the WA in practice would 

be immensely beneficial and insightful for faculty and clinical supervisors alike. Additionally, 

McCrae, Kurtz, Yamagata, & Terracciano (2011) provide evidence that retest reliability may be 

a more suitable long-term analysis than measures of internal consistency, which will also be 

taken into consideration for future study development. 

 

 
 

104 

 
 
 

Implications for Counselor Education 

The AGI fills a gap in counselor education literature by differentiating itself from other 

instruments via its theoretical underpinnings, item development, and intended use in practice. 

The AGI does not seek to provide thresholds of appropriateness, or minimum levels of 

competency that students should seek to reach. Rather, student are to be rated on a continuum of 

professional disposition development (PDD), from inadequate to exceptional. Faculty will be 

asked to rate students in the following manner: 

“The student in question demonstrates an understanding of, and/or capacity for…” 

a)  Inadequate demonstration of capacity; needs significant support 

b)  Occasional/Adequate demonstration of capacity; needs some support 

c)  Consistent/Exceptional demonstration of capacity; needs minimal support 

 

The AGI is intended to be accessible by all students at the outset of their educational 

program, and to be reviewed regularly by the student and their faculty advisor. Certain items 

pertain to a student’s ability to work with clients, therefore those items may be excluded from 

formal assessment while a student is in the pre-practica/internship stages of their program. 

However, such items should be reviewed and discussed between faculty and students, so that the 

expectation has been established, and the student has an opportunity to ask questions. Over time, 

a student should be able to achieve consistent or exceptional demonstration of the capacity 

described in the instrument items. Using the AGI on an electronic platform will also afford the 

opportunity for faculty to provide comments and examples for students to improve upon or as 

representative of an exceptional demonstration. Additionally, given the CACREP requirements 

for faculty to provide systematic data on student professional dispositions, an electronic platform 

significantly simplifies this process for programs, and could be expanded for use in program 

 
 

105 

 
 
 

evaluation measures. Within the context of our technology and app-driven society, moving the 

assessment to an online platform will streamline processes for faculty in the future, and will limit 

the amount of paperwork necessary to collect. 

Using the AGI in the above described transparent and collaborative manner may 

positively impact student experience in counselor education programs (e.g., J. Foster et al.,, 

2014; V. Foster & McAdams, 2009). Furthermore, use of the AGI will very likely simplify 

processes for faculty during their CACREP accreditation process. The AGI provides the 

potential for faculty to provide due process, parallel process, and to systematically document a 

facet of counselor education that is currently quite elusive and challenging. Ultimately, 

implementation of this instrument may improve student and faculty experiences and 

relationships, and quality of services provided to clients. 

The current study is the first step of many into the use of the AGI for counselor education 

programs. The AGI has been developed in order to meet a need of faculty, students, and 

accreditation bodies, and used intentional methodological choices to ensure its composition of 

items that are considered relevant and useful by both rehabilitation counselor educators and 

CRCs in practice. In order to ensure efficacy, and to provide faculty with adequate support, the 

AGI will obtain an online presence to include the instrument, its instructions, and resources for 

managing professional disposition development in counselor education programs.  

Implications for Clinical Practice 

 

Through the implementation of an instrument such as AGI, faculty and site supervisors 

may be able to not only effectively work together using the same measure for assessment, but 

would also be able to directly work with pre-service counseling students on the development of 

WA. This has been stated as a needed direction for counselor education: “...the challenge in 

 
 

106 

 
 
 

counselor education is to effectively teach the cognitive and affective complexity of WA so that 

graduates possess entry-level competence in establishing therapeutic relationships” (Connor & 

Leahy, 2016, p.375), and also aligns with the importance of embedding evidence-based practice 

into counselor education. Focusing on the development of the WA in counselor education 

students both through assessment and the use of parallel process in the supervisory relationship 

between faculty and students may also contribute to an improvement in the WA a student would 

be able to develop with their clients. As such, it is certainly feasible to consider the implication 

of improved client experiences and outcomes following the use of an instrument such as the 

AGI.   

Conclusion 

 

Results of the current study not only establish an instrument that fills a gap in counselor 

education, but it also provides insights into what rehabilitation counselor educators and 

practitioners consider to be relevant and useful in the evaluation of professional dispositions in 

pre-service counseling students. The AGI differentiates itself from other instruments that have 

been published in its development, theoretical framework, and philosophical underpinnings. The 

consistently high ratings on the proposed items for the instrument indicate that faculty and 

counselors in the field believe that professional dispositions need to be assessed in pre-service 

counseling students. The study results also revealed that specific components of counselor 

education such as ethics and client-centeredness are almost universally considered extremely 

relevant and useful, whereas applying theory to practice is less-so. There are many implications 

for the AGI, and the current study has allowed for an in-depth, and empirically-based starting 

point for this much needed, forward-looking instrument. 

 
 

 
 

107 

 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 

 
 
 
 
 

APPENDICES 

 

 

108 

 
 
 

 

 
 

APPENDIX A 

INTERNAL REVIEW BOARD (IRB) APPROVAL 

 

109 

March 9, 2017To:Ying Yuk Sung620 Farm Lane, Room 460Erickson HallRe:IRB# x17-334e Category:  Exempt 2Approval Date:March 9, 2017Title:Assessment as Growth: Teaching the working alliance through systematic evaluation ofprofessional dispositions in counselor educationThe Institutional Review Board has completed their review of your project.  I am pleased to adviseyou that your project has been deemed as exempt in accordance with federal regulations.The IRB has found that your research project meets the criteria for exempt status and the criteria forthe protection of human subjects in exempt research.  Under our exempt policy the PrincipalInvestigator assumes the responsibilities for the protection of human subjects in this project asoutlined in the assurance letter and exempt educational material. The IRB office has received yoursigned assurance for exempt research.  A copy of this signed agreement is appended for yourinformation and records.Renewals:  Exempt protocols do not need to be renewed.  If the project is completed, please submit anApplication for Permanent Closure.Revisions:  Exempt protocols do not require revisions.  However, if changes are made to a protocolthat may no longer meet the exempt criteria, a new initial application will be required.Problems:  If issues should arise during the conduct of the research, such as unanticipated problems,adverse events, or any problem that may increase the risk to the human subjects and change thecategory of review, notify the IRB office promptly.  Any complaints from participants regarding therisk and benefits of the project must be reported to the IRB.Follow-up:  If your exempt project is not completed and closed after three years, the IRB office willcontact you regarding the status of the project and to verify that no changes have occurred that mayaffect exempt status.Please use the IRB number listed above on any forms submitted which relate to this project, or on anycorrespondence with the IRB office.If we can be of further assistance, please contact us at 517-355-2180 or via email at IRB@msu.edu.Thank you for your cooperation.c: Allison LevineInitial IRBApplicationDetermination*Exempt*Office of Regulatory AffairsHuman ResearchProtection ProgramsBiomedical & HealthInstitutional Review Board(BIRB)Community ResearchInstitutional Review Board(CRIRB)Social ScienceBehavioral/EducationInstitutional Review Board(SIRB)Olds Hall408 West Circle Drive, #207East Lansing, MI 48824 (517) 355-2180Fax: (517) 432-4503Email: irb@msu.eduwww.hrpp.msu.eduMSU is an affirmative-action,equal-opportunity employer. 
 
 

APPENDIX B 

EMAIL INVITATION TO DELPHI PANEL 

 

Dear Dr. (last name), 
 
I am Allison Levine, a doctoral candidate at Michigan State University in the Rehabilitation 
Counselor Education program. I am emailing you regarding my dissertation study titled 
“Removing Uncertainty: Systematically Evaluating Professional Dispositions in Counselor 
Education”. As part of this study, we are soliciting the opinions of counseling educators who 
meet any of the following criteria:  

●  Has experience as a program director in a CACREP accredited counselor education 

program; 

●  Are a tenure-track faculty member (or core faculty member) at a CACREP accredited 

● 

counselor education program; or 
 Has published on the topics of supervision, gatekeeping, or professional dispositions in 
literature of counselor education or rehabilitation counselor education; 

More specifically, we are looking to identify and draw consensus on the development of items 
for an instrument which would evaluate student professional dispositions as they relate to 
master’s level counseling students and development of the working alliance. 
 
If you agree to take part in this study, you will be asked to participate in an anonymous, 4 round 
Delphi study. There are no anticipated risks or discomforts by participating on this study and 
participants may withdraw at any time without consequence. Panelists who participate in all 4 
rounds will receive a $30 gift card.  
 
The research project has been approved by The Michigan State University Institutional Review 
Board. Only authorized persons from Michigan State University involved in this research study 
have the legal rights to review the research records and will protect the confidentiality of those 
records to the extent by law or court order. If the results of the research are published or 
presented, all expert identities will remain anonymous. 
 
Your participation in this study is greatly appreciated. We understand your time limitations as an 
educator and seek to conduct this study in a thorough and efficient manner. It is anticipated that 
the total amount of time to complete all 3 rounds of the Delphi will not exceed 5 hours (Round 1 
- 1.5 hours; Round 2 - 1.5 hours; Round 3 - 1 hour; Round 4 - 1 hour). However, when 
considering the time associated with recruiting participants, analyzing the data, and allowing 
participants time to complete each round, the entire Delphi process may take up to 4 months. 
 
 
 
If you have any questions, please contact:  
 
Allison Levine, MS.Ed, CRC 
Doctoral Candidate, Rehabilitation Counselor Education 
Michigan State University 
levine31@msu.edu 

 
 

110 

 
 
 

OR 

 

 

Connie Sung, Ph.D., CRC, LLPC 
Assistant Professor, Rehabilitation Counselor Education 
Michigan State University 
csung@msu.edu 
 
If you have questions or concerns about your role and rights as a research participant, would like 
to obtain information or offer input, or would like to register a complaint about this study, you 
may contact, anonymously if you wish, the Michigan State University’s Human Research 
Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail 
at Olds Hall, 408 West Circle Dr Rm 207, East Lansing, MI 48824.  
 
You indicate your voluntary agreement to participate by completing and returning the 
demographics survey below. Further instructions will be sent upon the completion of the 
demographics survey. 
 
Please follow the link below to begin the survey. 
https://msucoe.az1.qualtrics.com/SE/?SID=SV_bx5vcBpWUwEZwTr 
 

 

 
 

111 

 
 
 

APPENDIX C 

EXPERT PANEL DEMOGRAPHICS SURVEY 

 
1 Gender 
❍  Male 
❍  Female 
❍  Other, please specify: ____________________ 
❍  Prefer not to respond 
 
2 Age 
❍  25 to 34 years old 
❍  35 to 44 years old 
❍  45 to 54 years old 
❍  55 to 64 years old 
❍  65 to 74 years old 
❍  75 years old or above 
❍  Prefer not to respond 
 
3 Ethnicity (check all that apply) 
❍  White 
❍  Black or African American 
❍  American Indian or Alaska Native 
❍  Asian 
❍  Native Hawaiian or Pacific Islander 
❍  Hispanic or Latino 
❍  Other, please specify: ____________________ 
❍  Prefer not to respond 
 
4 How many years have you been a counselor educator? 
______ 
 
5 What type of program do you currently work in? (i.e., Rehabilitation Counseling, Clinical 
Rehabilitation Counseling, Mental Health Counseling, etc.) _____________________ 
 
6 Are you the program director? 
❍  Yes 
❍  No 
 
If yes, how many years have you been the program director? ____ 
 

 
 

112 

 
 
 

7 What credentials do you currently hold? (Check all that apply) 
❑  CRC 
❑  LPC  
❑  LMHC 
❑  LCPC 
❑  LLPC 
❑  Other, please specify: ____________________ 
 
8 What accreditations does your program currently hold? (Check all that apply) 
❑  CACREP 
❑  CORE 
❑  Other, please specify: ____________________ 
 
9 Please indicate your preferred email address. This will be used only for the purpose of sending 
the instructions for the remainder of the Delphi process, and will not be stored with your 
responses to this survey or any additional data collected. 
 

 

 
 

113 

 
 
 

APPENDIX D 

DELPHI ROUND 1: EMAIL 

 

Dear Counselor Educators, 
This is a final follow up inviting you to join the expert panel for the Delphi study: Assessment as 
Growth: Teaching the Working Alliance through Systematic Evaluation of Professional 
Dispositions in Counselor Education. I know that you all are very busy, and I greatly appreciate 
your consideration of taking part in this study. I believe that your expertise is invaluable to the 
successful completion of this study, and have designed it to protect your time as best as possible. 
The goal of this research is to  identify and draw consensus on the development of items for an 
instrument which would evaluate student professional dispositions as they relate to master’s level 
counseling, and rehabilitation counseling, students and their development of the working 
alliance. 
 
I do not anticipate that 4 rounds of the Delphi will take more than 5 hours total over the span of 4 
months, at most. I have also revised the wording of the expert panel criteria, for clarity.  As part 
of this study, we are soliciting the opinions of counselor educators who meet any of the 
following criteria: 

●  Has published on the topics of supervision, gatekeeping, or professional dispositions in 

literature of counselor education or rehabilitation counselor education; 

●  Has experience as a program director in a CORE or CACREP accredited counselor 

education program; or 

●  Has been, or currently is a tenure-track faculty member at a rehabilitation, or counselor 

education program 

 
The research project has been approved by The Michigan State University Institutional Review 
Board (IRB Number:  x17-334e; i053564). Only authorized persons from Michigan State 
University involved in this research study have the legal rights to review the research records and 
will protect the confidentiality of those records to the extent by law or court order. If the results 
of the research are published or presented, all expert identities will remain anonymous. 
Please follow this link to participate in Round 1 of the Delphi: 
${l://SurveyLink?d=Take the Survey} 
Or copy and paste the URL below into your internet browser: 
${l://SurveyURL} 
 
If you have questions or concerns about your role and rights as a research participant, would like 
to obtain information or offer input, or would like to register a complaint about this study, you 
may contact, anonymously if you wish, the Michigan State University’s Human Research 
Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail 
at Olds Hall, 408 West Circle Dr Rm 207, East Lansing, MI 48824. 
 
You indicate your voluntary agreement to participate by completing and returning the survey 
above. I understand that you are all extremely busy, and that your time is valuable. I hope that 

 
 

114 

 
 
 

you are able to help in this research study, and appreciate your consideration of providing your 
time and expertise in doing so. 
 
With thanks, 
Allison Levine, MS. Ed., CRC 
Doctoral Candidate 
Graduate Teaching Assistantship Coordinator 
CEP 470 Instructor 
Research Assistant- Peckham Project 
 
Rehabilitation Counselor Education 
Office of Rehabilitation & Disability Studies 
Michigan State University 
Levine31@msu.edu 
 

 

 
 

115 

 
 
 

 

APPENDIX E 

DELPHI ROUND 1 INSTRUCTIONS 

●  Thank you and welcome to the Delphi Study for my dissertation, Assessment as Growth: 

Teaching the Working Alliance through Systematic Evaluation of Professional 
Dispositions in Counselor Education. This is Round 1 of the Delphi. 

●  CACREP Standards (Section 4) require program faculty to assess students Key 

Performance Indicators (KPIs) in content areas as well as on Professional Dispositions 
(click here for more information). CACREP defines Professional Dispositions as: “the 
commitments, characteristics, values, beliefs, interpersonal functioning, and behaviors 
that influence the counselor’s professional growth and interactions with clients and 
colleagues” This concept/definition has been deemed problematic given its ambiguous 
nature and subjective phrasing. Gatekeeping literature indicates the need for behaviorally 
specific assessment categories, and clear, operationalized language (Foster & McAdams, 
2009; Lumade & Duffey, 1999). 

●  The goal of the current study is to develop an instrument which allows counselor 

educators: 

○  (1) to evaluate master’s level counseling students as required by the KPI standards 

(i.e, systematically, at minimum of 2 points throughout program), 

○  (2) in a way that is less subjective, more legally defensible, and, 
○  (3) which aligns with evaluating a student’s ability to develop the working 

alliance (WA). 

There is no current mechanism which allows faculty to accomplish all three of these. 
 

●  Please adhere to the following definition of WA while responding to the below prompts: 
○  WA is composed of three features: agreement on goals, an assignment of task(s), 
and the development of bonds (Bordin, 1979). Goals can be considered outcomes 
or the target of an intervention. Tasks are the steps necessary to achieve the 
agreed upon goals. Bonds are the personal attachments necessary for the 
relationship to work, and are based in mutual trust, acceptance, and confidence. 

 
(1) Please identify behavioral and/or identifiable (i.e., measurable, observable) characteristics 
which embody professional dispositions for counselors to you. 
(2) Please identify items for a potential instrument which collect observations around a student’s 
ability to develop WA with potential clients. 
Specifically, please identify a minimum of 5 items per component: tasks, goals, and bonds. 
 
 
 

 
 

116 

 
 

 
 
 

APPENDIX F 

DELPHI ROUND 2 INSTRUCTIONS 

 
This is Round 2 of the Delphi for Assessment as Growth. This round involves the rating of the 
utility and relevance of items that were submitted in Round 1. You have the opportunity to add 
comments and feedback at the end of each section. There are 4 sections: Professional 
Dispositions, Tasks, Goals, and Bonds, there are 15-20 items in each section; this is because of 
many wonderful submissions in Round 1-- the goal is to get down to less than 10 per section. 
You do have the option to save this survey and return to it at a later time. You will have 2 weeks 
to complete this round. 
 
The items that are on the final instrument are to be rated by faculty as follows (measuring 
frequency and consistency of behaviors):  
 
“The student in question demonstrates an understanding of, and/or capacity for:”  
1 = never/no demonstration of capacity; needs significant support; 
3 = occasional demonstration of capacity, needs some support;  
5 = consistent demonstration of capacity, requires minimal support 
 
It is necessary to provide your email address at the end of this survey, as in Round 3, you will 
receive the rating you gave an item in relation to the group means and standard deviations. Your 
identity will remain anonymous to all other expert panelists and will stored in a password 
protected account on a password protected computer to protect confidentiality.  
 
Thank you for your time, expertise, and dedication to this research project. 
 
**If you are using a mobile device to complete this survey, please note that it will be easier to 
navigate if you turn your device to LANDSCAPE mode. 
 

 

 
 

117 

 
 
 

APPENDIX G 

DELPHI ROUND 3 INSTRUCTIONS 

 
This is Round 3 of the Delphi for Assessment as Growth. This round involves the final rating of 
the utility and relevance of items. Below the items are the group means and standard deviations 
from previous rounds. Items have been sorted by relevance, in descending order. 
  
You have the opportunity to add comments and feedback at the end of each section. There are 4 
sections: Professional Dispositions, Tasks, Goals, and Bonds, and there are 15-20 items in each 
section.  
You do have the option to save this survey and return to it at a later time. You will have 2 weeks 
to complete this round. 
  
Please use the following definitions of utility & relevance: 
 
Utility: the item will be useful for faculty in their evaluation of preservice counseling students 
and their ability to develop the working alliance with future clients. 
Relevance: the item is connected to, or appropriate for, the evaluation of preservice counseling 
students and their ability to develop the working alliance with future clients. 
 
The items that are on the final instrument are to be rated by faculty as follows (measuring 
frequency and consistency of behaviors):  
 
“The student in question demonstrates an understanding of, and/or capacity for:”  
1 = never/no demonstration of capacity; needs significant support; 
3 = occasional demonstration of capacity, needs some support;  
5 = consistent demonstration of capacity, requires minimal support 
  
Thank you for your time, expertise, and dedication to this research project.  
 
**If you are using a mobile device to complete this survey, please note that it will be easier to 
navigate if you turn your device to LANDSCAPE mode. 
 
 

 

 

 

 
 

 
 
 
 
 

118 

 
 
 

APPENDIX H 

INFORMED CONSENT/INVITE FOR PILOT STUDY 

 

Dear Rehabilitation Counselors, 
 
This is an invitation to participate in the validation of the Assessment as Growth Inventory, an 
instrument developed during the completion of a dissertation titled: Assessment as Growth: 
Teaching the Working Alliance through Systematic Evaluation of Professional Dispositions in 
Counselor Education. I believe that your expertise is invaluable to the successful completion of 
this study, and have designed it to protect your time as best as possible. The goal of this research 
is to  identify and validate items for an instrument which would evaluate professional 
dispositions as they relate to master’s level counseling and rehabilitation counseling students, 
and their development of the working alliance. 
 
I do not anticipate that this survey will take more than 20 minutes to complete, and at the 
completion, you are eligible to receive one (1) CRCC CEU. 
Please follow this link to participate in the Assessment as Growth Inventory: 
 
Follow this link to the Survey:  
${l://SurveyLink?d=Take the Survey} 
 
Or copy and paste the URL below into your internet browser: 
${l://SurveyURL} 
 
This research project has been approved by The Michigan State University Institutional Review 
Board (IRB Number:  x17-334e; i053564). Only authorized persons from Michigan State 
University involved in this research study have the legal rights to review the research records and 
will protect the confidentiality of those records to the extent by law or court order. If the results 
of the research are published or presented, all expert identities will remain anonymous. 
 
If you have questions or concerns about your role and rights as a research participant, would like 
to obtain information or offer input, or would like to register a complaint about this study, you 
may contact, anonymously if you wish, the Michigan State University’s Human Research 
Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail 
at Olds Hall, 408 West Circle Dr Rm 207, East Lansing, MI 48824.  
 
You indicate your voluntary agreement to participate by completing and returning the survey 
above. I understand that you are all extremely busy, and that your time is valuable. I hope that 
you are able to help in this research study, and appreciate your consideration of providing your 
time and expertise in doing so. 
 
With thanks, 
 
Allison Levine, MS. Ed., CRC 
 
Doctoral Candidate 

 
 

119 

 
 
 

Rehabilitation Counselor Education 
Office of Rehabilitation & Disability Studies 
Michigan State University 
 
Levine31@msu.edu 
 
 

 

 
 

120 

 
 
 

 

 
 
 
 

 
 

APPENDIX I 

ASSESSMENT AS GROWTH INVENTORY 

Professional Dispositions 

(1) A dedication to, and embodiment of, the ethical values (autonomy, justice, veracity, 
beneficence, nonmaleficence, fidelity)  

(2) Showing unconditional acceptance of all clients, peers, or coworkers regardless of their 
demographics (i.e., age, gender, race, ethnicity, sexual identity/orientation, culture)  

(3) Understanding the tendency and the problem of racial stereotyping 

(4) The development of a counselor identity (e.g., theoretical orientation, helping 
disposition, professional advocacy) 

(5) Demonstrates professional and personal maturity such as accepting feedback, following 
through on commitments, commitment to professional growth 

(6) Sufficient professional administrative skills (i.e., punctuality, organizational skills, 
preparation, professional written communication/documentation, awareness of 
policy/procedures)  

(7) Sufficient professional interpersonal skills (i.e., developing professional rapport with 
coworkers, appropriate tone of voice, language use, use of humor, appropriate dress, etc.) 

(8) Use of problem-solving skills in a timely and professional fashion 

(9) Maintains respectful countenance in all interactions: perceives and honors diversity, 
boundaries, and appropriate communication style 

(10) Engaging in difficult conversations with clients, coworkers, and/or supervisors in an 
appropriate manner 

(11) Commitment to a career in the counseling field via indications of a desire to be a 
lifelong learner (e.g., always seeking new information and resources, participation in 
professional organizations) 

(12) Awareness of his/her role as a counselor, including self-awareness, humility, and 
integrity  

(13) Demonstrates professionalism and professional behavior in interactions with peers, 
supervisors, clients, and as a representative of their educational program  

121 

 
 
 

 
 

 
 

Tasks 

(1) Ability to collaborate with clients to design tasks that are congruent with the mutually 
agreed upon goal 

(2) Ability to identify tasks that will lead to goal attainment 

(3) Developing options to address clients' needs and problems with identified priorities 

(4) Adjusting with the client as his/her needs and problems evolve during the counseling 
process 

(5) Ability to communicate the link between the chosen task and the agreed upon goal 

(6) Knowing how to consult or refer to resources available in ethnocultural communities 

(7) Being able to show support when holding clients accountable for task completion, or non-
completion 

(8) Application of Evidence-Based Practices (EBP) 

(9) Ability to design tasks to meet the unique developmental and individual needs of the client 

(10) Being able to apply theory to justify rationale behind assigned tasks 

(11) Ability to collaborate with clients to design tasks that are congruent with the mutually 
agreed upon goal 

 

122 

 
 
 

 

 
 

 
 

Goals 

(1) Incorporating the client voice into the goal setting process 

(2) Demonstrating empathy in understanding the various influences which have impacted the 
client's goals and experiences (i.e., cultural background, socioeconomic status, etc.) 

(3) Allowing clients to take the lead in the identification of potential goals 

(4) Collaborating with clients in the development of mutually agreed upon goals 

(5) Mutual cooperation with the client when establishing and implementing the treatment plan 

(6) Applying theory to a situation in order to provide guidance in the goal setting process 

(7) Developing goals that follow a strengths based approach, and identifies potential in clients 
(as opposed to focusing solely on dysfunction) 

(8) Being able to facilitate goal development in areas that may be in conflict with the 
counselor's personal values 

 

123 

 
 
 

 

 

 
 

Bonds 

(1) Using active and reflective listening to ensure effective collaboration, problem-solving, 
and decision-making 

(2) Demonstrating unconditional positive regard for clients 

(3) Adherence to ethical practice for respecting client’s informed consent and other client 
rights in order to develop trust, boundaries, and transparency in the counseling relationship 

(4) Actively avoiding cultural biases and discriminatory practices in working with clients of 
minority backgrounds 

(5) Addressing multicultural issues when presented in the session that may affect the 
counseling relationship or the client's ability to pursue a goal 

(6) Demonstrating ethical behavior in the development of bonds with clients (i.e., appropriate 
boundaries, etc.) 

(7) Communicating in a confidential, responsive, and empathic manner to establish rapport in 
a way that promotes openness and sensitivity to potential cultural differences 

(8) Ensuring client autonomy 

(9) Capacity to appropriately communicate acceptance to the client 

(10) Establishing trust with the client as evidenced by the communication that occurs between 
the counselor and client 

(11) Maintaining a nonjudgmental disposition regarding client values 

(12) Managing the power differential between counselor and client 

(13) Identification of client’s needs and problems in congruence with his/her priorities 

(14) Being capable of adjusting interactions with clients to meet his/her individual needs and 
communication style 

(15) To demonstrate openness and flexibility when addressing the client's issues and problems 

(16) Willingness to advocate for clients of minority backgrounds who experience institutional 
discrimination 

(17) Sustaining the effort to help a client whether or not he/she makes progress 

  

 

124 

 
 
 

 
 

APPENDIX J 

CRCC CEU COMPLETION FORM 

125 

 

  College of  Education  Department of  Counseling, Educational Psychology, and Special Education   620 Farm Lane, Rm. 447 East Lansing, MI  48824-1034  517-353-6417 Fax: 517-353-6393     CRC/CCRC VERIFICATION OF COMPLETION (Please Print or Type All Information)  SPONSOR INFORMATION (To be completed by program/activity sponsor) Michigan State University 00247563 Sponsoring Organization Sponsor Code 620 Farm Ln  Allison Levine Street address,  Contact Person East Lansing, MI, 48824-1600 516-356-7957 City/State/Zip Code Phone Number for Contact Person   PROGRAM/ACTIVITY INFORMATION (To be completed by program/activity sponsor.)        Assessment as Growth: Teaching the Working Alliance Through Systematic Evaluation of Professional Dispositions in Counselor Education Program/Activity Title 2017-10-05 - 2018-10-04 Program/Activity Valid Through Date  TRN2115233 1.0 Approval Number Clock Hours Attended/Completed   Signature of Individual in Charge of Verifying Completion Date of Signature  
 
 

 

 

 

 

 
 

 

 

 

 

126 

 
 
 

 
 

 

 

 

 

 

 

 

 

 

 

 

 

REFERENCES 

 

 

127 

 
 
 

REFERENCES 

 

Ackerman, S. J., & Hilsenroth, M. J. (2001). A review of therapist characteristics and techniques 

 negatively impacting the therapeutic alliance. Psychotherapy, 38, 171-184. 

 
Ackerman, S. J., & Hilsenroth, M. J. (2003). A review of therapist characteristics and techniques 

positively impacting the therapeutic alliance. Clinical Psychology Review, 23, 1-33.  

 
American Counseling Association. (2014). 2014 ACA Code of Ethics. Alexandria, VA: Author. 
 
Baldwin, S. A., Wampold, B.E., & Imel, Z.E. (2007). Untangling the alliance-- Outcome 

 correlation: Exploring the relative importance of therapist and patient variability in the 
 alliance. Journal of Counseling and Clinical Psychology, 75(6), 842-852. DOI: 
10.1037/0022-006X.75.6.842. 

 
Bernard, J. L. (1975). Due process in dropping the unsuitable clinical student. Professional 

 Psychology: Research & Practice, 6, 275-278. 

 
Bernard, J. M., & Goodyear, R. K. (2014). Fundamentals of Clinical Supervision. (5th ed.). 

 Upper Saddle River: NJ: Pearson. 

 
Bhat, C. S. (2005). Enhancing counseling gatekeeping with performance appraisal protocols. 

 International Journal for the Advancement of Counseling, 27(3), 399-411. 

 
Blau, P. M. (1964), Justice in social exchange. Sociological Inquiry, 34, 193-206.  

doi:10.1111/j.1475-682X.1964.tb00583.x. 

 
Bland, J. M., & Altman, D. G. (1997). Cronbach’s alpha. British Medical Journal, 314(7080), 

 572. 

 
Board of Curators of the University of Missouri v. Horowitz, 430 U.S. 964 (1978). 
 
Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. 

 Psychotherapy: Theory, research and practice, 16(3), 252-260. 

 
Bordin, E. S. (1983). A working alliance based model of supervision. The Counseling 

 Psychologist, 11(1), 35-42. 

 
Brear, P., & Dorrian, J. (2010). Gatekeeping or gate slippage? A national survey of counseling 

educators in Australian undergraduate and postgraduate academic training programs. 
Training and Education in Professional Psychology, 4(4), 264-273. DOI: 
10.1037/a0020714. 

 
 

 
 

128 

 
 
 

Brear, P., Dorrian, J., & Luscri, G. (2008). Preparing our future counseling professionals: 

Gatekeeping and the implications for research. Counseling and Psychotherapy Research, 
8(2), 93-101. 

 
Brown-Rice, K., & Furr, S. (2015). Gatekeeping ourselves: Counselor educators’ knowledge of 

 colleagues’ problematic behaviors. Counselor Education & Supervision, 54, 176-189. 
DOI: 10.1002/ceas.12012. 

 
Brusseri, M. A., & Tyler, J. D. (2003). Interchangeability of the Working Alliance Inventory and 

 Working Alliance Inventory, Short Form. Psychological Assessment, 15, 193-197. 

 
Castonguay, L. G., Constantino, M. J., & Holtforth, M. G. (2006). The working alliance: Where 
 are we and where should we go? Psychotherapy: Theory, Research, Practice, Training, 
43(3), 271-279. DOI: 10.1037/0033-3204.43.3.271. 

 
Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 

 1, 245-276. 

 
Cavanagh, S. (1997). Content analysis: Concepts, methods and applications. Nurse Researcher, 

 4(3), 5-16. 

 
Chan, F., Bezyak, J., Ramirez, M. R., Chiu, C. Y., Sung, C., & Fujikawa, M. (2010). Concepts, 

challenges, barriers, and opportunities related to evidence-based practice in rehabilitation 
counseling. Rehabilitation Education, 24(3 & 4), 179-190.  

 
Chapman, B. P., Talbot, N., Tatman, A. W., & Britton, P. C. (2009). Personality traits and the  

working alliance in psychotherapy trainees: an organizing role for the Five Factor Model? 
Journal of Social and Clinical Psychology, 28(5), 577-596. 

 
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale 

 development. Psychological Assessment, 7(3), 309-319. 

 
Commission on Rehabilitation Counselor Certification. (2017). Code of Professional Ethics for 

 Rehabilitation Counselors. Schaumburg, IL: Author. 

 
Commission on Rehabilitation Counselor Certification. (2009). Code of Professional Ethics for 

 Rehabilitation Counselors. Schaumburg, IL: Author. 

 
Connor, A., & Leahy, M. J. (2016). Teaching the working alliance: Bridging the gap between 

counseling microskills and establishing meaningful and productive relationships. 
Rehabilitation Research, Policy, and Education, 30(4), 371-388. DOI: 10.1891/2168-
6653.30.4.371. 

 
Council for Accreditation of Counseling and Related Education Programs. (2015). Gatekeeping. 

 Retrieved from: http://www.cacrep.org/glossary/gatekeeping/. 

 

 
 

129 

 
 
 

Council for Accreditation of Counseling and Related Education Programs. (2016). The 2016 

 CACREP standards. Alexandria, VA: Author. 

 
Cropanzano, R., & Mitchell, M. S. (2005). Social exchange theory: An interdisciplinary review. 

 Journal of Management, 31(6), 874-900. DOI: 10.1177/0149206305279602. 

 
Custer, G. (1994, November). Can universities be liable for incompetent grads? APA Monitor, 

 25(11), 7. 

 
Dawson, M. D., & Brucker, P. S. (2001). The utility of the Delphi method in MFT research. The 

American Journal of Family Therapy, 29(2), 125-140. DOI: 10.1080/01926180126229. 

 
Delbecq, A. L., Van de Ven, A., & Gustafson, D. H. (1975). Group techniques for program 

 planning: A guide to nominal group and delphi processes. Illinois: Scott, Foresman and 
Company. 

 
Efstation, J. E., Patton, M. J., & Kardash, C. M. (1990). Measuring the working alliance in 

 counselor supervision. Journal of Counseling Psychology, 37(3), 322-329. 

 
Elman, N. S., & Forrest, L. (2007). From trainee impairment to professional competence 
 problems: seeking new terminology that facilitates effective action. Professional 
Psychology, Research and Practice, 38(5), 501-509. 

 
Elman, N., Forrest, L., Vacha-Haase, T., & Gizara, S. (1999). A systems perspective on the  

trainee impairment: Continuing the dialogue. The Counseling Psychologist, 27(5), 712-
721. 

 
Emerson, R. M. (1976). Social exchange theory. Annual Review of Sociology, 2, 335-362. 
 
Forrest, L., Elman, N., Gizara, S., & Vacha-Haase. (1999). Trainee impairment: A review of 
 identification, remediation, dismissal, and legal issues. The Counseling Psychologist, 
27(5), 627-686. 

 
Foster, J. M., Leppma, M., & Hutchinson, T. S. (2014). Students' perspectives on gatekeeping in 

 counselor education: A case study. Counselor Education & Supervision, 53, 190-203. 

 
Foster, V. A., & McAdams, C. R. (2009). A framework for creating a climate of transparency for 

 professional performance assessment: Fostering student investment in gatekeeping. 
Counselor Education & Supervision, 48, 271-284. 

 
Fouad, N. A., Grus, C. L., Hatcher, R. L., Kaslow, N. J., Hutchings, P. S., Madson, M. B., … 

 Crossman, R. E. (2009). Competency benchmarks: A model for understanding and 
measuring competence in professional psychology across training levels. Training and 
Education in Professional Psychology, 3(4), S5-S26. DOI: 10.1037/a0015832. 

 
 

 
 

130 

 
 
 

Frame, M. W., & Stevens-Smith, P. (1995). Out of harm’s way: Enhancing monitoring and 

 dismissal processes in counselor education programs. Counselor Education &  
Supervision, 35, 118-129. 

 
Gaubatz, M. D., & Vera, E. M. (2002). Do formalized gatekeeping procedures increase  

programs' follow-up with deficient trainees? Counselor Education and Supervision,  
41(4), 294-305. 

 
Gaubatz, M. D., & Vera, E. M. (2006). Trainee competence in Master's-level counseling  

programs: A comparison of counselor educators' and students' view. Counselor 
Education & Supervision, 46, 32-43. 

 
Glance, D., Fanning, G., Schoepke, A., Soto, W., & Williams, M. A. (2012). Gatekeeping in 

 counselor education. VISTAS 2012, 1, 1-14. 

 
Glosoff, H. L., & Matrone, K. F. (2010). Ethical issues in rehabilitation counselor supervision 
 and the new 2010 code of ethics. Rehabilitation Counseling Bulletin, 53(4), 249-254. 

 
Goss v. Lopez, 419 U.S. 565 (1975). 
 
Goforth, C. (2015). Using and interpreting Cronbach’s alpha. Retrieved from: 

 http://data.library.virginia.edu/using-and-interpreting-cronbachs-alpha/ 

 
Greenhill v. Bailey. 519 F .2d 5 (8th Clr. 1975). 
 
Greenson, R. R. (1965). The working alliance and the transference neurosis. The Psychoanalytic 

 Quarterly, 34, 155-181.  

 
Grote, R. C. (1996). The Complete Guide to Performance Appraisal. New York, NY:  

AMACOM 

 
Grote, D. (2002). Performance appraisal. Executive Excellence, 19, 12. 
 
Hatcher, R. L., & Gillaspy, J. A. (2006). Development and validation of a revised short version 

 of the working alliance inventory. Psychotherapy Research, 16(1), 12-25. DOI: 
10.1080/10503300500352500. 

 
Harder, J. W. (1992). Play for pay: Effects of inequity in a pay-for-performance context. 

 Administrative Science Quarterly, 37(2), 321-335. 

 
Harris v. Blake and the Board of Trustees of the University of Northern Colorado, 798 F.2d 419 

 (CO10 1986). 

 
Hickey, G., & Kipping, C. (1996). Issues in research. A multi-stage approach to the coding of  

data from open-ended questions. Nurse Researcher, 4(1), 81-91. DOI:  
10.7748/nr.4.1.81.s9. 

 
 

131 

 
 
 

Hoffman, M. A., Hill, C. E., Holmes, S. E., & Freitas, G. F. (2005). Supervisor perspective on  

the process and outcome of giving easy, difficulty, or no feedback to supervisees. Journal 
of Counseling Psychology, 52(1), 3-13. DOI: 10.1037/0022-0167.52.1.3. 

 
Horvath, A. O., Del Re, A. C., Flückiger, C., & Symonds, D. (2011). Alliance in individual 

 psychotherapy. Psychotherapy, 48(1), 9-16. DOI: 10.1037/a0022186. 

 
Horvath, A. O., & Greenberg, L. S. (1989). Development and validation of the working alliance 

 inventory. Journal of Counseling Psychology, 36, 223-233. 

 
Horvath, A. O., & Symonds, B. D. (1991). Relation between working alliance and outcome in 

psychotherapy: A meta-analysis. Journal of Counseling Psychology, 38(2), 139-149. 

 
Hutchens, N., Block, J., & Young, M. (2012). Counselor educators’ gatekeeping responsibilities 
 and students’ First Amendment rights. Counselor Education and Supervision, 52(2), 82-
95. DOI: 10.1002/j.1556-6978.2013.0030x. 

 
Iqbal, M. Z., Akbar, S., & Budhwar, P. (2015). Effectiveness of performance appraisal: An 

 integrated framework. International Journal of Management Reviews, 17, 510-533. DOI: 
10.1111/ijmr.12050. 

 
Ivey, A. E. (1971). Microcounseling: Innovations in interviewing training. Springfield, IL: 

Charles C Thomas. 

 
Keeton v. Anderson-Wiley, 664 F.3d 865 (U.S. Ct. App., 11th Cir. 2011). 
 
Kelly, V. A. (2011). Assessing individual student progress: meeting multiple accreditation 
standards and professional gatekeeping responsibilities. The Journal of Counselor 
Preparation and Supervision, 2(2), 111- 123. 

 
Kerl, S. A., Garcia, J. L., McCullough, C. S., & Maxwell, M. E. (2002). Systematic evaluation of 

 professional performance: Legally supported procedure and process. Counselor 
Education and Supervision, 41(4), 321-334. 

 
Knoff, H. M., & Prout, H. T. (1985). Terminating students from professional psychology 

 programs: Criteria. procedures. and legal Issues. Professional Psychology: Research and 
Practice, 16(6), 789-797. 

 
Kosciulek, J. F. (2004). Empowering people with disabilities through vocational rehabilitation 

counseling. American Rehabilitation, 28(1), 40-47. 

 
Ladany, N., & Melincoff, D. S. (1999). The nature of counselor supervisor nondisclosure. 

 Counselor Education and Supervision, 38, 161-176. 

 

 
 

132 

 
 
 

Laliotis, D. A., Laliotis, D. A., & Grayson, J. H. (1985). Psychologist heal thyself: What is 

available for the impaired psychologist? American Psychological Association, 40(1), 84-
96. 

 
Lamb., D. H., Presser, N. R., Pfost, K. S., Baum, M. C., Jackson, V. R., & Jarvis, P. A. (1987). 
 Confronting professional impairment during the internship: Identification, due process, 
and remediation. Professional Psychology: Research and Practice, 18(6), 597-603. 

 
Lambert, M. J. (1992). Implications of outcome research for psychotherapy integration. In J. C. 
 Norcross & M. R. Goldfried (Eds.), Handbook of psychotherapy integration (94-129). 
New York: Basic. 

 
Latham, G. P., & Locke, E. A. (1979). Goal setting-- A motivational technique that works. 

 Organizational Dynamics, 8(2), 68-80. DOI: 10.1016/0090-2616(79)90032-9. 

 
Leahy, M. J., Chan, F., Sung, C., & Kim, M. (2013). Empirically derived test specifications for 
 the Certified Rehabilitation Counselor examination. Rehabilitation Counseling Bulletin, 
56(4), 199-214. DOI: 10.1177/0034355212469839. 

 
Leahy, M. J., Muenzen, P., Saunders, J. L., & Strauser, D. (2009). Essential knowledge domains 

underlying effective rehabilitation counseling practice. Rehabilitation Counseling 
Bulletin, 52(2), 95-106. DOI: 10.1177/0034355208323646 

 
Levine, A. (2016). Gatekeeping in master’s level rehabilitation counseling programs: A pilot 

study. Unpublished Manuscript. 

 
Linstone, H. A., & Turoff, M. (2011). Delphi: A brief look backward and forward. Technological 

Forecasting & Social Change, 78, 1712-1719. DOI:10.1016/j.techfore.2010.09.011 

 
Locke, E. A., & Latham, G. P. (2006). New directions in goal-setting theory. Current Directions 

 in Psychological Science, 15(5), 265-268. 

 
London, M., Mone, E. M., & Scott, J. C. (2004). Performance management and assessment: 

 Methods for improved rater accuracy and employee goal setting. Human Resource 
Management, 43(4), 319-336. 

 
Lumade, C. A., & Duffey, T. H. (1999). The role of graduate programs as gatekeepers: A model 

 for evaluating student counselor competence. Counselor Education and Supervision, 
39(2), 101-109. 

 
Lustig, D. C., Strauser, D. R., Rice, D., & Rucker, T. F. (2002). The relationship between 

 working alliance and rehabilitation outcomes. Rehabilitation Counseling Bulletin, 46(1), 
25-33. 

 
 
 

 
 

133 

 
 
 

Martin, D. J., Garske, J. P., & Davis, M. K. (2000). Relation of the therapeutic alliance with 

 outcome and other variables: A meta-analytic review. Journal of Consulting and Clinical 
Psychology, 68(3), 438-450. DOI: IO.I037//0022-006X.68.3.438. 

 
Matula, J. J. (2011). Embedding due process measures throughout the evaluation of teachers. 

NASSP Bulletin, 95(2), 99-121. DOI: 10.1177/0192636511409925. 

 
McAdams, C. R., & Foster, V. A. (2007). A guide to just and fair remediation of counseling 

 students with professional performance deficiencies. Counselor Education & 
Supervision, 47(1), 2-13.  

 
McAdams, C. R., Foster, V. A., & Ward, T. J. (2007). Remediation and dismissal policies in 

 counselor education: Lessons learned from a challenge in federal court. Counselor 
Education & Supervision, 46(3), 212-229. 

 
Meier, P. S., Donmall, M. C. McElduff, P., Barrowclough, C., & Heller, R. F. (2006). The role of 

the early therapeutic alliance in predicting drug treatment dropout. Drug and Alcohol 
Dependence, 83(1), 57-64. 

 
Moorhouse, M. D. (2008). Identifying benchmark competency criteria for a rehabilitation 

 counseling clinical supervision instrument: A delphi study. Retrieved from ProQuest 
Digital Dissertations. 

 
Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: an example, design 

 considerations and applications. Information & Management, 42, 15-29. 
DOI:10.1016/j.im.2003.11.002. 

 
Paul, L. (1967). Strategy of outcome research in psychotherapy. Journal of Counseling 

 Psychology, 31(2), 109-118.  

 
Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. New York, NY: 

 Routledge. 

 

 
Ridley, C. R., Kelly, S. M., & Mollen, D. (2011). Microskills training: Evolution, reexamination, 

 and call for reform. The Counseling Psychologist, 39(6), 800–824.  
http://dx.doi.org/10.1177/0011000010378438 

 
Rogers, C. R. (1951). Client-centered therapy, its current practice, implications, and theory. 

 Boston, MA: Houghton Mifflin. 

 
Rotondi, A., & Gustafson, D. (1996). Theoretical, methodological and practical issues arising 
 out of the Delphi method. In M. Adler & E. Ziglio (Eds.), Gazing into the Oracle: The 
Delphi Method and its Application to Social Policy and Public Health (34-55). 
Philadelphia, PA: Jessica Kingsley Publishers. 

 

 
 

134 

 
 
 

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. 

Psychometrika, 74(1), 107-120. 

 
Strauss, P. (n.d.). Due process. Retrieved from Cornell University Law School Legal Information 

 Institute: https://www.law.cornell.edu/wex/due_process 

 
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA: 

 Pearson. 

 
Tate, K. A., Bloom, M. L., Tassara, M. H., & Caperton, W. (2014). Counselor competence, 
 performance assessment, and program evaluation: Using psychometric instruments. 
 Measurement and Evaluation in Counseling and Development, 47(4), 291-306. 
DOI:10.1177/0748175614538063. 

 
Thielsen, V. A., & Leahy, M. J. (2001). Essential knowledge and skills for effective clinical 

 supervision in rehabilitation counseling. Rehabilitation Counseling Bulletin, 44(4), 196-
208. 

 
Truax, C. B., & Carkhuff, R. R. (1967). Toward effective counseling and psychotherapy:  

training and practice. Chicago: Aldine Pub. Co. 

 
Vacha-Haase, T., Davenport, D. S., & Kerewsky, S. D. (2004). Problematic students: 

 Gatekeeping practices of academic professional psychology programs. Professional 
Psychology: Research and Practice, 35(2), 115-122. DOI: 10.1037/0735-7028.35.2.115. 

 
Vázquez-Ramos, R., Leahy, M., & Hernandez, N. E. (2007). The Delphi method in rehabilitation 

 counseling research. Rehabilitation Counseling Bulletin, 50(2), 111-118. 

 
Wampold, B. E. (2015). How important are the common factors in psychotherapy? An update. 

 World Psychiatry, 14(3), 270–277. DOI: 10.1002/wps.20238. 

 
Wampold, B. E. (2001). The Great Psychotherapy Debate, Second Edition : Models, Methods, 

and Findings. Mahwah, NJ: Routledge. 

 
Whiston, S. C., Rossier, J., Hernandez Barón, P. M. (2015). The working alliance in career 
counseling: A systematic overview. Journal of Career Assessment, 24(4), 591-604.  

 
Ziglio, E. (1996). The Delphi method and its contribution to decision making. In M. Adler & E. 
 Ziglio (Eds.), Gazing into the Oracle: The Delphi Method and its Application to Social 
Policy and Public Health (3-33). Philadelphia, PA: Jessica Kingsley Publishers. 

 
Ziomek-Daigle, J., & Christensen, T. M. (2010). An emergent theory of gatekeeping practices in  

counselor education. Journal of Counseling and Development, 88(4), 407-415. 

 

 
 

135 

 
 
 

Zirkelback, E. A., & Reese, R. J. (2010). A review of psychotherapy outcome research: 

Considerations for school-based mental health providers, Psychology in the Schools, 
47(10), 1084-1100. DOI: 10.1002/pits.20526. 

 
 

136