EMPLOYMENT SUCCESS: FEASIBILITY, SOCIAL VALIDITY, & PRELIMINARY RELIABILITY OF A SOFT SKILLS ASSESSMENT FOR TRANSITION-AGE INDIVIDUALS WITH AUTISM By Alicia M. Strain A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Rehabilitation Counselor Education – Doctor of Philosophy 2023 ABSTRACT Autistic individuals have numerous strengths that are valuable in work settings (e.g., attention to detail, punctuality, low absenteeism, high work quality, strong work ethic, trustworthiness, loyalty). Yet, when compared to neurotypical peers, many autistic individuals continue to have difficulty securing, maintaining, and advancing in employment; this is especially salient for individuals transitioning from adolescence into adulthood (ages 14-30). In response to these challenges, evidence-based transition programming and employment-readiness interventions have progressively increased and have specifically focused on employment preparation and retention skills, occupation-specific skills, and soft skills (e.g., social- communication skills, responsibility, flexibility, teamwork, etc.); together, these skills are called employability skills. A key element to employment-focused evidence-based practices is socially and psychometrically validated employability instruments. Such tools can help transition-age individuals with autism (TAI-ASD) identify their employment skill strengths and skills that are in-development; these measurement tools can also be used to determine the efficacy and effectiveness of transition programming and employment-readiness interventions. However, there is a paucity of employability skills instruments that are theory-aligned, psychometrically sound, socially acceptable, and clinically applicable for autism communities. To help fill this research-practice gap, three instruments have been developed that measure soft skills, employment preparation skills, and employment retention skills. This dissertation specifically focuses on evaluating the social validity, feasibility, preliminary factor structure, and preliminary reliability of the soft skills assessment, titled Employment Success: Soft Skills (ESSS). Using a convergent parallel mixed-methods design, key stakeholders (20 TAI-ASD, 16 parent/caregivers, and 44 educators and service providers) living throughout the U.S. offered quantitative and qualitative feedback about the social validity and feasibility of the ESSS for individual, academic, and clinical purposes. Stakeholders first completed either the self-report or the informant-report of the ESSS; they then evaluated several domains of social validity (understandability, clarity, relevance, ease of use, utility) and feasibility (desirability, applicability, collaboration, understanding, system climate, system support). Descriptive statistics and qualitative content analysis were used to determine the overall social validity and feasibility of the ESSS and to understand similarities and differences in stakeholder perspectives. Multivariate analysis of variance (MANOVA) was used to determine the degree of differences across stakeholder perspectives. Merging the quantitative and qualitative responses identified which aspects of the ESSS may need improvement based on stakeholder perspectives. Exploratory factor analysis was used to examine the preliminary latent structure of the ESSS, and Cronbach’s alpha was used to determine the preliminary internal consistency (reliability). In general, stakeholders endorsed the social validity and the feasibility of the ESSS, suggesting they perceived it to be a useful instrument for measuring perceived soft skill strengths and skills that are continuing to develop. Educators and service providers further endorsed the relevancy, practicality, and utility of the ESSS for academic and clinical settings that serve TAI- ASD. However, results suggested that additional clarity in the introduction section, automating the scoring section, and expanding select items for more contextual clarity are needed prior to proceeding with more robust psychometric evaluations of the ESSS in the future. Preliminary results of the exploratory factor analysis suggest the ESSS may be a single-factor instrument measuring a global soft skills domain. Preliminary reliability suggests the ESSS has good internal consistency on the self-report (.88-.91) and the informant-report (.83-.92). Implications for research and practice are discussed along with suggestions for future research. Copyright by ALICIA M. STRAIN 2023 In dedication to the community contributors of this dissertation. Thank you for your guidance, feedback, and thoughtful suggestions. v ACKNOWLEDGEMENTS What a journey this has been. A long time ago, someone who I hold in high esteem had asked me to describe myself using only one word. I responded, “searching.” What I was searching for I could not say, I just knew it was the current running deep within myself. As I reflect on the journey that has brought me to this pivotal achievement, I am reminded of the many individuals who have been my lighthouses along the way. To my faculty chair, Dr. Connie Sung, thank you for the myriad opportunities to learn and grow new skillsets, for encouraging me to continuously step outside my comfort zone, and for believing in me, especially when imposter syndrome clouded my vision. Thank you, Dr. Martin Volker, for showing me how to be a grounded, compassionate, intentional and thoughtful scholar-practitioner, and especially for sharing your assessment and statistical expertise. Thank you, Dr. Michael Leahy, for teaching me the value of authenticity, warmth, collegiality, diplomacy, and what it means to teach with heart; I have a deeper understanding of how life really is “all about relationships.” Thank you, Dr. Gloria Lee, for demonstrating transformational leadership and for your sincerity, transparency, and openness. Thank you, Dr. Hung Jen Kuo, for encouraging me to view the world through different lenses and to agitate dominant narratives. Thank you, Nicole Jess and the Center for Statistical Training & Consulting, for generously sharing your time, expertise, and thoughtful guidance on statistical decisions, data analysis, and interpretation. To my colleagues – Dr. Brenna Breshears, Dr. Rosanne Renauer, Dr. Heekyung Lee, Dr. Jorem Awadu, Dr. Annemarie Connor, Dr. Eric Roberts, Kristin Houck, Trisha Easley, Eveliz Rosado-Martinez, Heerak Choi, Anni Wang, Anna Podlesna, and Shannon Aydukovic – you each have touched my life in so many different ways. Thank you for the vi walks, the impromptu hallway chats, the happy-hours, and between meetings catch-ups. I am deeply grateful for you and your unyielding encouragement, camaraderie, and cheerleading. To my husband and son, Mike and Bodhi, who are my biggest lighthouses. Thank you for realizing this dream with me. Your unlimited hugs, tenacious serenades, nimbleness, gentleness during moments of intensity, and bubbling laughter have been my guiding light. To my mom and sister, Carol Osepowicz and Aimée Lacouture, and my life-long friends, Peggy Lit, Mackenzie Albert, and Jessica Warren, you are my rocks and my sisterhood. Thank you for your gentle but firm reminders to keep showing up in life, to live joyfully, to be brave, and to stay true to myself every step of the way. To my dad, John Lacouture, whose life unexpectedly ended in 2004, the year before I graduated with my undergraduate degrees. I miss you. Although you are not here physically, I feel your love and I know you are proud of the life I have created. As I close this journey’s chapter and begin a new one, I am no longer searching. It has taken many years filtered by many experiences to discover that what I had been searching for was with me the whole time, I just needed more context to see it. While I thought my search existed outside myself, what this journey has taught me is that all along what I was searching for was me. The last person I would like to thank is myself – for not quitting when it seemed like that was the only way forward, for not compromising my integrity, for picking myself up over and over again, and for building an unbreakable trust that no matter what happens I will always have my back. vii TABLE OF CONTENTS CHAPTER 1: INTRODUCTION ................................................................................................... 1 Statement of the Problem ............................................................................................................ 4 Statement of Purpose .................................................................................................................. 6 Research Questions ..................................................................................................................... 9 Significance of the Study ............................................................................................................ 9 Positionality Statement ............................................................................................................. 10 CHAPTER 2: LITERATURE REVIEW ...................................................................................... 12 The Importance of Employability Skills ................................................................................... 12 Personal Factors That Impact Employment for TAI-ASD ....................................................... 16 Existing Employability Interventions ....................................................................................... 17 Existing Employability Assessments ........................................................................................ 21 Theoretical Framework: Social Cognitive Career Theory ........................................................ 25 Philosophical Framework: Participatory Action Research ....................................................... 28 The Role of Social Validity ...................................................................................................... 30 CHAPTER 3: METHODS ............................................................................................................ 32 Research Design........................................................................................................................ 32 Participants ................................................................................................................................ 33 Procedures ................................................................................................................................. 39 Measures ................................................................................................................................... 41 Data Analysis ............................................................................................................................ 49 CHAPTER 4: RESULTS .............................................................................................................. 57 Social Validity and Feasibility of the ESSS Assessment.......................................................... 57 Similarities and Differences in Stakeholder Perspectives on Social Validity and Feasibility .. 66 Preliminary Reliability and Factor Structure of the ESSS Assessment .................................... 74 CHAPTER 5: DISCUSSION........................................................................................................ 84 Social Validity and Feasibility of the ESSS Assessment.......................................................... 85 Preliminary Reliability and Factor Structure of the ESSS Assessment .................................... 89 Stakeholder-Informed Improvements to the ESSS Assessment ............................................... 90 Limitations ................................................................................................................................ 93 Implications for Research and Practice..................................................................................... 96 Closing Remarks ..................................................................................................................... 100 REFERENCES ........................................................................................................................... 101 APPENDIX A: EMPLOYMENT SUCCESS: SOFT SKILLS ASSESSMENT ........................ 113 APPENDIX B: SOCIAL VALIDITY & FEASIBILITY SURVEYS ........................................ 121 APPENDIX C: CORRELATION TABLES ............................................................................... 137 APPENDIX D: SOCIAL VALIDITY & FEASIBILITY MEANS TABLES ............................ 141 viii APPENDIX E: MANOVA MULTIVARIATE & UNIVARIATE TABLES ............................ 148 APPENDIX F: RELIABILITY TABLES .................................................................................. 153 APPENDIX G: EFA FACTOR LOADING COMPARISON TABLES .................................... 159 ix CHAPTER 1: INTRODUCTION The transition period from adolescence to adulthood can be a period characterized by a continuum of emotions, from elation about exploring future possibilities to fear about uncertainties and the unknown. For transition-age individuals between the ages of 14 and 30 with autism spectrum disorders (TAI-ASD), the backdrop to this period is frequently contextualized by experiences across numerous topics, including employment, postsecondary education, social relationships, and independent living (Bennett et al., 2018; Sosnowy et al., 2018). Among the variety of transition experiences, employment is a particularly salient topic. The benefits of employment for individuals with ASD have been well established in the literature. For example, Hendricks’ (2010) literature review demonstrated numerous benefits, including the promotion of personal dignity, improved quality of life, less reliance on government programming (e.g., SSI/SSDI), greater contributions to socioeconomic tax structures, the ability to support oneself financially and pursue personal interests, and reduced employer stigma and negative attitudes towards persons with ASD. Walsh and colleagues’ (2014) literature review revealed additional benefits including improved cognitive performance and peer relations, and reduced anxiety and depression. Roux’s team (2015) analyzed data from the National Longitudinal Transition Study–2 and found that 90% of TAI-ASD with higher functional abilities and stronger communication skills who had paid job experiences in high school also had paid employment during their early 20s. These beneficial outcomes underscore the importance of evidence-based transition programing that introduces transition-age individuals to the world of work while also promoting skill acquisition and proficiency that support successful transitions into adulthood and beyond. 1 Despite the well-established benefits of employment for individuals with ASD, the research and practice literature are saturated with concerns about transition outcomes for TAI- ASD. For example, when compared to neurotypical peers, there continues to be lower proportions of individuals with ASD who have secured and maintained paid employment (Bennett et al., 2018). Less than one third of individuals with ASD achieve paid employment, and of the individuals who are working, they tend to work part-time in low-wage jobs (Alverson & Yamamoto, 2017; Bennett et al., 2018). Furthermore, analyses from the National Longitudinal Transition Study-2 indicate that more than half of TAI-ASD are disengaged from employment and/or education several years after exiting high school (Shattuck et al., 2012). This accumulating evidence suggests that a disproportionate percentage of TAI-ASD are not well prepared to meet the varying demands of work environments (Lindsay et al., 2012). With an estimated 700,000 to 1.1 million youth with ASD entering adulthood over the next decade (Shattuck et al., 2020), a staggering number of TAI-ASD continue to be at-risk for experiencing transition challenges. However, disproportionate employment outcomes for TAI-ASD are not necessarily surprising when interpreted through the lens of Vygotsky’s sociocultural theory of development, which suggests that people may have difficulty reaching their full potential when they are not provided access to scaffolded learning opportunities within supportive social and cultural environments (Vygotsky, 1978). For example, chronic under- and unemployment for this population has been a function of several compounding issues, including difficulty accessing adequate and individualized employment preparation opportunities (Sosnowy et al., 2018), numerous challenging internal and external factors (Chen et al., 2015), underappreciated and 2 untapped individual potential (Wehman et al., 2014), and a paucity of integrative, strengths- based approaches to employment across the lifespan (Scott et al., 2019). In response to chronic under- and unemployment challenges, there have been progressive developments in transition programming and employment-readiness interventions that focus on TAI-ASD learning soft skills and skills necessary to prepare for, secure, and maintain employment (e.g., ASSET-EPASS and Project SEARCH+ASD; see Connor et al., 2019; Sung et al., 2019, 2021; Wehman et al., 2013, 2019). Preparing for and maintaining employment is an orchestrated process of career exploration and scaffolded skill acquisition that braids employment preparation and retention skills, occupation-specific skills, and soft skills; collectively, these skills are called employability skills. Entry-level occupation-specific skills are often learned on a job. In contrast, soft skills (e.g., social-communication skills, positive attitude, responsibility, flexibility, teamwork, work ethic, etc.; Robles, 2012) and how to obtain and maintain a job are typically learned prior to employment; this process is referred to as employment-readiness training. While ASSET-EPASS and Project SEARCH+ASD programming have made great strides in addressing the need for evidence-based employment-readiness training for TAI-ASD populations, these programs are not yet widely available so many ASD communities still struggle with accessing comprehensive transition programming in their localized areas. Furthermore, while other interventions have shown promising results, the general consensus among systematic literature review authors is that the existing body of research continues to be limited by scope and quality, including poor measurement of outcomes (see Hedley et al., 2016; Lee et al., 2018; Scott et al., 2019; Shattuck et al., 2020). In addition, few studies critically elevate the perspectives of TAI-ASD, their families, and the direct professionals who serve TAI- 3 ASD (also referred to as a participatory action research) in assessing the feasibility and social validity of transition programming and employment interventions (Anderson et al., 2017; Shattuck et al., 2018, 2020). Inclusion of these individuals’ voices is imperative across individual, community, and systems levels to ensure transition services and employment training are relevant and equitable (Bronfenbrenner, 1979; Mercer & Howe, 2012). Statement of the Problem A key element to evidence-based transition programming is socially and psychometrically validated evaluation tools; without these instruments, it would be difficult to produce valid claims of the efficacy and effectiveness of the transition program. Furthermore, stakeholders desire data-driven accountability for transition programming outcomes, which would also be difficult to produce without proper evaluation instrumentation. Opportunities for evaluation can occur at various systems levels with program evaluation and at individual levels with employability skills development. Regardless of the evaluation approach, incorporating multiple stakeholder perspectives—such as TAI-ASD, parents/caregivers, paraprofessionals, special educators, transition coordinators, vocational counselors—is necessary to obtain a comprehensive understanding of outcomes (Kohler et al., 2016; U.S. Government Accountability Office, 2016). Throughout transition programming, it is recommended that practitioners and educators use thoughtful assessment approaches to identify which employment skills are strengths and which are areas for improvement; this approach helps TAI-ASD and professionals who support them understand which employment skills to market to employers and which skills to improve via interventions that focus on employment skill development (Lee et al., 2018). Murry and colleagues (2016) additionally stressed that evaluation tools should demonstrate strong 4 psychometric properties and have clinical utility and social validity regarding the relevancy, feasibility, and practicality of the tool. However, given the foundational role that evaluation plays, there is a paucity of employability skills evaluation instruments that help TAI-ASD understand their employability strengths and areas for improvement. Literature reviews furthermore indicate that of the few employment instruments available for individuals with ASD, none of them appear to be informed by career theories; their clinical utility has yet to be explored; they have not been developed and clinically evaluated using participatory action research principles; and none have demonstrated strong psychometric properties for TAI-ASD (Hedley et al., 2016; Murray et al., 2016; Scott et al., 2019). Unpacking these statements further, Lent and colleagues (1999) discussed social cognitive career theory (SCCT) as a viable conceptual and developmental framework for transition populations. More specifically, Lent et al. (1999) suggested that career development interventions (and evaluation thereof) target six developmental themes throughout the transition process: formation of self-efficacy and outcome beliefs, interest development, interest-goal alignment, translation of goals into actions, performance skills, and negotiation of transition supports and barriers. SCCT suggests that across these career development themes self-efficacy and outcome beliefs are key drivers of employment pursuits and perseverance when faced with adversity. While these constructs are important to measure throughout the transition process, to date, there are no employment assessments designed and developed for TAI-ASD that align with key constructs of the SCCT framework. Secondly, Murray and colleagues (2016) emphasized that employment instruments demonstrate clinical utility in terms of their affordability, training for instrument fidelity, and 5 average time to administer, score, and interpret the results. Additionally, Shattuck and colleagues (2018, 2020) made clear the ethical priority to increasing involvement of individuals with ASD throughout the research process, including producing research findings and recommendations. Clinical utility of employment instruments is promoted when applied participatory action research approaches are blended with instrument affordability, streamlined fidelity training, and minimal time needed to administer and score. However, to date, commercially available employment instruments for TAI-ASD vary in their clinical utility with no instrument meeting all clinical utility components. Therefore, in sum, developing employment assessments that are theory-aligned, psychometrically sound, socially acceptable, and clinically applicable are imperative for accurately measuring and scaffolding employment skill development, transition programming accountability, and improving overall employment outcomes for TAI-ASD. Statement of Purpose Effectively addressing this critical gap in employment assessments for TAI-ASD requires a multiphase-multistep process (Boateng et al., 2018; see Figure 1). The first phase focuses on item development and includes two steps: (1) identifying the most important employability skills that promote successful employment outcomes for TAI-ASD, and (2) applying career theory to guide the translation of the employability skills into employment assessments for TAI-ASD. The second phase focuses on scale development and clinical utility and includes the following steps: (3 and 4) pretesting questions and administration with the projected users of the assessment, (5) modifying and/or reducing items according to pretesting results, and (6) extracting scale factors. The third phase involves psychometric evaluation of the assessment and includes the following steps: (7) testing scale dimensionality, (8) testing reliability, and (9) testing validity. Once the assessment has demonstrated adequate psychometric qualities, it can then be evaluated for 6 generalizability to other populations of transition-age individuals with and without neurodiverse abilities. Figure 1.1 Multiphase, Multistep Process to Scale Development Figure credit: Boateng et al. (2018 https://doi.org/10.3389/fpubh.2018.00149 7 This multiphase-multistep process began with my research apprenticeship where I accomplished the first phase of item development. More specifically, I conducted a Delphi study with 16 participants (researchers, employers, practitioners) who are subject matter experts in employment and ASD topics. Through an iterative process, the panel of experts anonymously identified and reached consensus on a total of 72 employability skills (23 soft skills, 25 employment preparation skills, and 24 employment retention skills) that are essential to employment success for TAI-ASD (see Strain & Sung, 2019). In the second step of this first phase, I applied social cognitive career theory’s (SCCT) performance model framework (Lent et al., 1994) to translate the employability skills into three assessments for TAI-ASD, titled Employment Success: Soft Skills (ESSS), Employment Success: Preparation Skills (ESPS), and Employment Success: Retention Skills (ESRS). To allow for comparison of multiple perspectives, each assessment includes self-report and informant-report versions. While each assessment focuses on respective employability skill domains, consistent across the assessments are measurement scales that align with SCCT’s performance model constructs of self-efficacy and performance actions; together, these constructs identify employability skills that are perceived strengths as well as skills that are developing and may be improved through various intervention strategies. Specifically, this dissertation focuses on the second phase (steps 3-6) of this multiphase- multistep process. Using a mixed-methods, participatory action research approach, the purpose of this study is to further develop and refine the Employment Success: Soft Skills (ESSS) assessment by pretesting the assessment questions and administration while also gathering feedback about the feasibility and social validity (i.e., clinical utility) of the assessment. This study includes the following stakeholders as they are the projected users of the Employment 8 Success assessments: transition-age individuals with ASD, parents and caregivers, paraprofessionals, special educators, vocational counselors, and other related professionals. Evaluating feasibility and social validity of the assessments across various stakeholder perspectives will determine the applicability of the assessments in academic and clinical settings that serve TAI-ASD. This study also incorporates stakeholder recommendations regarding assessment improvements; this is a necessary strategy to strengthen the clinical utility of the assessments prior to proceeding with steps 7-9 in the assessment development process. Research Questions 1. What is the social validity and feasibility of the Employment Success: Soft Skills assessment among projected users of the instrument? 1.1. Do stakeholder perspectives meet the benchmark for social validity of the Employment Success: Soft Skills assessment? (3 on a 4-point scale) 1.2. Do stakeholder perspectives meet the benchmark for feasibility of the Employment Success: Soft Skills assessment? (4 on a 6-point scale) 2. What are the similarities and differences in (a) social validity and (b) feasibility of the Employment Success: Soft Skills assessment across different types of stakeholders? 3. What is the preliminary reliability (internal consistency) and factor structure of the Employment Success: Soft Skills assessment? 4. What updates are necessary to improve the Employment Success: Soft Skills assessment? Significance of the Study Pragmatically, this dissertation seeks to fill a research-practice gap in employability assessments that promote multidisciplinary collaborations and data-driven decision making across systems that serve TAI-ASD (Chen et al., 2015). Aspirationally, this study seeks to offer 9 practical assessments for TAI-ASD that seamlessly translate across multiple settings and are effective and useful for multiple users. For example, the objective is to (a) develop assessments that may serve as counseling and guidance tools in clinical and academic settings (e.g., vocational rehabilitation, high school/postsecondary education) to identify perceived employment skill strengths and opportunities for further skill improvement, and (b) to develop measurement tools for intervention studies that focus on employability skills development. Theoretically, this study seeks to develop a quantitative tool for measuring key domains related to social cognitive career theory’s performance model. Positionality Statement Although positionality statements (also known as reflexivity statements) historically are linked to qualitative research, "a researcher's background and position will affect what they choose to investigate, the angle of investigation, the methods judged most adequate for this purpose, the findings considered most appropriate, and the framing and communication of conclusions" (Malterud, 2001, p. 483-484); therefore, a positionality statement is equally as relevant in quantitative and mixed-methods research. In promotion of positionality statements as a standard practice for all research, I am including my positionality statement in acknowledgement of this framework that implicitly and explicitly impacts this study. I am a white, middle class, agnostic, cisgender, straight female. I am a mother. I am a complex trauma survivor. I manage a neuromuscular disorder and nonvisible disability. I have several family members who live with chronic illness and disability. I grew up in a single-parent household. I individually financed my undergraduate and graduate pursuits. I have experienced different perspectives on diversity while living in rural, suburban, and urban communities across Colorado, Idaho, Michigan, Maine, and Alabama. I have experienced different global 10 perspectives on diversity while studying and traveling throughout urban and rural communities in India, France, Ireland, Mexico, and Canada. These intersecting identities and experiences have afforded me braids of privilege and power, exclusion and adversity; they also shape my perspectives on the value of conducting community-based research. In this participatory action research study, I occupy the space between being an insider and an outsider when interacting with various stakeholders (e.g., TAI-ASD, parents/caregivers, paraprofessionals, special educators, vocational counselors; Corbin Dwyer & Buckle, 2009). For example, I have lived with a neuromuscular disorder and nonvisible disability since my youth and I am considered neurotypical; therefore, I am an insider-outsider. I am a parent of a neurotypical child; therefore, I am an insider-outsider. Professionally, while I have clinical experience working in private community-based psychosocial rehabilitation and public vocational rehabilitation settings serving neurodiverse and neurotypical individuals across the lifespan, I have limited experience working as a transition educator in an educational setting; therefore, I am also an insider-outsider. Because I occupy insider and outsider spaces, I am purposefully aligning this study with feminist perspectives that advocate for participatory models that reinforce non-hierarchical, non-manipulative research relationships (Corbin Dwyer & Buckle, 2009). The aim is to minimize the separation between myself (as the researcher) and the diverse group of stakeholders while critically elevating both their separate and collective voices throughout the research study. 11 CHAPTER 2: LITERATURE REVIEW This second chapter provides an overview of employment topics that directly impact transition-age individuals with autism spectrum disorder (TAI-ASD). For example, this chapter further describes why employability skills are important, the differences between hard skills and soft skills, personal factors that impact employment for TAI-ASD, and existing employability interventions and employability assessments that are applicable to autistic communities. Closing this chapter is a discussion about how Social Cognitive Career Theory’s (SCCT) performance model was applied when developing the Employment Success assessments (soft skills, preparation skills, retention skills), the philosophical tenets of participatory action research that drive this study, and how social validity is applied to instrument development. The Importance of Employability Skills While employability skills have been investigated for decades, Sarfraz and colleagues (2018) found a lack of consensus in how to define these skills; after synthesizing the literature, they derived the following definition: Employability skills are personal skills and attributes demonstrated by an individual that distinguish one job seeker from another in their field of specialisation and help them to secure gainful employment, sustain them in that job and progress in their career to achieve their maximum potential and contribute towards their personal goals and that of their organisation. (p. 66) As employability skills change over time to reflect current workforce trends, recent research inquiries have started to reflect workforce expectations that align with 21 st century trends. Although further research is needed to determine which specific skills are necessary for successful employment, there is a general consensus across multiple stakeholder perspectives 12 (e.g., employers, educators, employment counselors) regarding employability skill domains that impact employment outcomes for both individuals with and without disabilities (see Table 2.1). Domains frequently cited include job search skills, impression management, self-efficacy, social supports, basic skills, higher order thinking skills (including executive functioning), disability- related skills (including self-regulation), global citizenship skills, and soft skills (including social-communication, flexibility, teamwork, work ethic, etc.; see Agran et al., 2016; Ju et al., 2012, 2014; Liu et al., 2014; Robles, 2012; Sarfraz et al., 2018). Table 2.1 Employability Domains that Impact Employment General Employability Skills Skill Domains Description Knowing where to find jobs, how to apply for jobs, using Job Search Skills social capital Impression Management Skills How a person presents themselves on paper and in person Using various strategies to strengthen a person’s beliefs Self-Efficacy Skills about their capabilities Using family and friends for emotional and tangible Social Support Skills supports (e.g., encouragement, arranging transportation) Basic skills Ability to read, listen, speak, communicate ideas Seek help when needed, follow schedules, cooperative, Basic work skills perseverance, work well with diverse groups, monitor quality of work Recognize/correct mistakes, critical thinking, solve Higher order thinking skills problems, negotiate/resolve conflict, computer/technology skills, goal-oriented, self-advocacy, creative thinking Requesting accommodations, knowing disability legislation, Disability-related skills managing disability symptoms, educating others about disabilities Appreciation of diversity and multiculturalism, awareness Global citizenship skills of global issues, multilingual skills 13 Table 2.1 (cont’d) Specific Soft Skills Honest, ethical, high morals, has personal values, does Integrity what’s right Communication Oral, speaking capability, written, presenting, listening Show respect, use socially acceptable language, accept Social skills authority, accept criticism, self-control Manners, etiquette, gracious, says please and thank you, Courtesy respectful Accountable, reliable, gets the job done, resourceful, self- Responsibility disciplined, wants to do well, conscientious, common sense Personable, sense of humor, friendly, empathetic, patient, Interpersonal Skills sociability, warmth, social skills Positive Attitude Optimistic, enthusiastic, encouraging, confident Professionalism Businesslike, dressed appropriately, poised Adaptability, willing to change, lifelong learner, accepts Flexibility new things, adjusts, teachable Cooperative, gets along with others, supportive, helpful, Teamwork collaborative Hard working, loyal, takes initiative, self-motivated, Work Ethic punctual, good attendance Note. domains were synthesized from Agran et al., 2016; Ju et al., 2012, 2014; Liu et al., 2014; Robles, 2012; Sarfraz et al., 2018. Employability skills can also be interpreted through the lens of hard skills and soft skills. Hard skills are defined as technical skills and abilities that allow an individual to perform a particular task or activity within a specific context. Hard skills can be further deconstructed into three specific employment skill domains: (a) preparation skills that help individuals prepare for and obtain employment (e.g., developing a resume, conducting an online job search, understanding how job interviews are organized, etc.); (b) occupation-specific skills that support individuals in their job (e.g., administrative, restaurant, plumbing, nursing, accounting, etc.); and (c) retention skills that promote an individual with maintaining and advancing in employment 14 (e.g., self-employment activities, managing work tasks, meeting work performance metrics and productivity, etc.). Soft skills are inter- and intrapersonal qualities and attributes that are transferable across hard skill contexts. In other words, while hard skills are specific to particular contexts, soft skills are generalized across contexts; both skills work in tandem and collectively promote employment success. For example, an individual primarily develops a resume (a hard skill) to obtain employment, and once employment has been secured, the resume gets filed away only to be used again when a new employment situation is pursued. On the other hand, professionalism (a soft skill) is communicated on a person’s resume, how they conduct themselves in an interview, throughout the daily interactions on their job, during job performance reviews, and while seeking promotion; this is a skill that is modeled across contexts and is fluid depending on the situation. In this scenario, an individual’s employability skills are a function of both having a resume and demonstrating different forms of professionalism across different contexts. Figure 2.1 depicts the relationship between hard skills and soft skills. Figure 2.1 Hard Skills and Soft Skills 15 Studies have determined that employability skills are necessary to obtain and maintain employment and also contribute to successful employment outcomes (Robles, 2012; Yang et al., 2015). In fact, global studies have demonstrated that lack of employability skills have greater impact on unemployment than a lack of job availability (Hasan et al., 2016; Rahmat et al., 2016). However, numerous studies indicate a growing gap between existing employability skills in job candidates and employability skills expected by employers (Sarfraz et al., 2018). As the employability skills gap has especially impacted employment outcomes for TAI-ASD, there is an emphasized need for improved implementation of interventions and assessments that target employability skill constructs (Lee & Carter, 2012; Snell-Rood et al., 2020). Personal Factors That Impact Employment for TAI-ASD Autism spectrum disorder (ASD) is a complex neurodevelopmental condition where expression of diagnostic characteristics (e.g., social-communication difficulties, intense focus and interests, environmental sensitivities, preference for routine and repetition) can vary in type and degree across persons (United States Government Accountability Office, 2016); in other words, ASD is a highly individualized condition. As such, various individual-level factors can challenge and promote TAI-ASD’s ability to obtain and maintain employment. Factors that can interrupt TAI-ASD’s employment outcomes may include social and communication challenges, such as difficulty understanding instructions and social colloquialisms, reading facial expressions and interpreting tone of voice, initiating and maintaining conversations, managing hygiene and grooming, and understanding others’ emotions (Chen et al., 2015). Other factors may include cognitive and behavioral challenges, such as inflexible routines, difficulty with transitions, emotion regulation and behavior management, and executive functioning skills including problem-solving, time management, 16 planning and organization, task execution, and working memory (Hendricks, 2010). On the other hand, strengths-based factors can promote successful employment outcomes for TAI-ASD, such as attention to detail, punctuality, low absenteeism, high quality of work, strong work ethic and sense of morality, trustworthiness and loyalty, comfortableness with repetitive or monotonous tasks, and specialized skills in technology, creative arts, mathematics, and other topical areas (Scott et al., 2019). Interpreted through the lens of social cognitive career theory, the degree of these personal factors coupled with contextual and environmental factors (e.g., quality of education and career development training, employment supports and barriers), impact the variability in employment outcomes for TAI-ASD as well as variability in the benefits acquired from employability skills training. For this reason, it is recommended that practitioners and educators regularly and thoughtfully collect relevant information throughout transition programming; this information can be used to determine whether an employability intervention is meeting the individual training needs of the TAI-ASD, if adjustments need to be made, or if a new approach should be implemented (Lee et al., 2018). While this type of data-driven decision making is necessary to address the employability skills gap in TAI-ASD populations, it hinges on the availability of feasible and effective employability skills interventions and assessments. Existing Employability Interventions Over the past two decades, there has been a progressive increase in the number of employability skills interventions as evidenced by several systematic literature reviews that focus on employment-related topics and individuals with ASD (see Anderson et al., 2017; Bennett & Dukes, 2013; Bennett & Goodall, 2021; Hedley et al., 2016; McDonald & Machalicek, 2013; Scott et al., 2019; Seaman & Cannella-Malone, 2016; Shattuck et al., 2020; Walsh et al., 2014). 17 Synthesizing these literature reviews indicates approximately 80 intervention studies that target specific employability skillsets and include TAI-ASD participants. Conducting a descriptive analysis to further understand how these studies are arranged across employability skill domains resulted in the following distributions: 11 studies targeted preparation skills, 32 studies targeted occupation-specific skills, 13 studies targeted retention skills, and 14 studies targeted soft skills (Table 2.2). Because some intervention studies included multiple skill domains, a fifth category was created to represent these comprehensive interventions (10 studies). Table 2.2 Intervention Studies from Systematic Reviews Organized by Employment-Related Domaina Preparation Occupation- Retention Multiple Soft Skills Skills Specific Skills Skills Skill Domains Bennett & Dukes (2013) - 5 - - - McDonald & Machalicek (2013) - 1 - 2 - Walsh et al. (2014) 1 2 - - - Hedley et al. (2016) 3 3 6 1 2 Seaman & Cannella- Malone (2016) 3 10 4 - - Anderson et al. (2017) 1 11 - 5 - Scott et al. (2019) 1 - 3 4 5 Shattuck et al. (2020) - - - 1 1 Bennett & Goodall (2021) 2 - - 1 2 Total 11 32 13 14 10b a Duplicate studies across literature reviews have been removed from this analysis. b Seven studies targeted all skill domains; one study targeted soft skills and retention skills; one study targeted preparation skills and soft skills; one study targeted soft skills and occupation-specific skills. Deconstructing these distributions highlights which skills were targeted across each skill domain. For example, the interventions in the preparation skills domain targeted cover letter 18 writing, interviewing, determining job task preferences, and using the internet. The interventions in the occupation-specific skills domain targeted clerical skills (e.g., sorting & stuffing envelopes, stocking, packaging); restaurant skills (e.g., cooking, rolling silverware, setting tables); custodial skills (e.g., cleaning & sanitizing); newspaper delivery; retail and merchandising skills in warehouses, schools, and department stores, including mascot costume entertainment; and human service skills (e.g., applied behavioral analysis technician for young children). The interventions in the retention skills domain targeted self-regulation skills, task management skills using audio-video prompting and personal digital assistants, work independence skills, executive functioning skills (e.g., time management, organization), and two interventions applied retention strategies from supported employment models. The interventions in the soft skills domain targeted skills related to teamwork, social greetings & interactions (including conversation skills, customer service skills, networking skills), asking for help, problem-solving, critical thinking, conflict resolution, enthusiasm and attitude, and professionalism. As indicated in Table 2.2, a more recent trend in employability interventions includes the delivery of more comprehensive programming that targets multiple employability skill domains. Considering supported employment models are evidence-based practices (Leahy et al., 2018), it is of no surprise that seven of the eight studies applied this strategy that engages, places, and trains individuals at a specific employment site and then supports them with maintaining and advancing in employment; thus, seven studies incorporated all the employability skills domains. More specifically, four of the seven studies that applied supported employment strategies were associated with the Project SEARCH+ASD intervention (Wehman et al., 2012, 2013, 2014, 2016), one study represented Project ABLE from Northern Ireland (Lynas, 2014), and one study 19 represented a 2-year vocational support program (Hillier et al., 2007). Given the expense of traditional supported employment models, a recent study piloted the individual placement and support (IPS) model with TAI-ASD (McLaren et al., 2017); as IPS is an evidence-based practice for people with mental health conditions, this study demonstrates an emerging strategy that may also generalize to neurodevelopmental conditions, such as ASD. While seven studies included all employability skills domains, one study targeted soft skills (social interactions) and retention skills (task management using audio prompting; Gilson & Carter, 2016); one study targeted preparation skills (job interviewing) and soft skills (hygiene, dress code, punctuality; Gorenstein et al., 2020); and one study targeted soft skills (teamwork and collaboration) and occupation- specific skills (sorting clothing in a retail setting and food preparation at a nonprofit agency; Nicholas et al., 2019). Although employability skills interventions have progressively increased over the past two decades, the general consensus among systematic literature review authors is that existing research is limited by scope and quality; this is evidenced by small sample sizes, an underreporting of and limited focus on social validity (feasibility), and a lack of rigorous intervention designs and quality measurement of short- and long-term outcomes across settings. Furthermore, based on the above synthesis, interventions appear to be disproportionately trending towards occupation-specific skillsets with interventions in the other three domains (preparation skills, retention skills, soft skills) receiving less emphasis. While supported employment models address the need for more comprehensive approaches to obtaining, maintaining, and advancing in employment, some of these programs are costly and are based on narrow eligibility criteria; inadvertently, this may exclude some TAI-ASD who could benefit from this type of comprehensive programming (McLaren et al., 2017). Overall, although there 20 are at least 80 intervention studies developed over two decades, which interventions are effective for which TAI-ASD and under which conditions continues to be undetermined (Lee et al., 2018). Existing Employability Assessments Several systematic literature review authors indicate a paucity of validated and socially accepted employability skills measures for ASD populations (Hedley et al., 2016; Iacomini et al., 2021; Murray et al., 2016; Scott et al., 2019; Strickland et al., 2013). Furthermore, these authors have noted that because of the lack of relevant instruments, numerous studies have designed measures for the specific study without conducting further psychometric testing. More recently, however, the research literature has started to reflect an increase in available measures. Globally, there currently are five employment-related assessments that specifically are applicable to ASD populations (discussed in detail below). The Autism Work Skills Questionnaire (AWSQ) was originally developed in Hebrew, and after initial validation, a standardized English version was created (Gal et al., 2013). The instrument incorporates vocationally related strengths and weaknesses to match individuals with high-functioning ASD to a compatible work setting (Gal et al., 2013). Individuals use a 5-point Likert scale to complete a 78-item, semi-structured questionnaire that produces an employment profile divided into six domains: work habits (12 items), working style (10 items), independence in work and study (10 items), routine daily activities (14 items), interpersonal skills (19 items), and sensory response and needs (13 items). Higher scores indicate better performance. Internal consistency reliability of the six domains ranges from moderate to high (Cronbach’s = .65-.90) and discriminant validity was established between individuals with and without ASD on four of the domains, excluding the sensory response needs and work habits domain (Gal et al., 2013, 2015). While Gal and colleagues (2015) indicate the AWSQ has clinical utility for documenting 21 individual strengths and challenges during employment training and job placement activities, the social validity of the instrument has yet to be described in the research literature. Additionally, other clinical utility domains (such as cost, training, and administrative time) are also unknown. The Work Performance Evaluation was originally developed for internal use by an organization in Israel that provides social services to individuals with disabilities (Katz et al., 2015). TAI-ASD and employment support staff members each use a 4-point Likert scale to complete parallel forms of a 31-item questionnaire that addresses various aspects of work (e.g., work quality, interpersonal relationships with employer/colleagues/clients, security, diligence, and efficiency); lower scores indicate better performance. Initial psychometric evaluation of the instrument indicates high internal consistency (Cronbach’s = .916), and a principal components analysis suggests the instrument measures three factors (interest and initiation in work, responsibility, and acceptance of rules and authority in the workplace). Social and psychometric validity and clinical utility have yet to be published. The Pre-work Multiple Stimulus Assessment was designed in the U.S. for individuals with ASD and co-occurring intellectual disability who have limited verbal communication skills and are participating in supported employment programs (Reid et al., 1998; Lattimore et al., 2002, 2003). The performance-based assessment determines individual preferences on cleaning tasks (e.g., dusting furniture, washing windows, cleaning sinks). For example, five different cleaning items that represent different cleaning tasks (polishing cloth, vacuum cleaner, duster, broom, mop) are presented to the participant who then selects their preferred task by touching the item. The participant performs the activity for three minutes and then the process is repeated several times to determine their preferred prework activity based on the frequency of their choices. Preferences are then evaluated and validated at a work site where the same assessment 22 procedures are conducted. Interobserver reliability is strong with 100% agreement on identifying work task preferences and 83-100% agreement on participants’ work engagement during both prework and on-the-job observations. Predictive validity is moderate to strong across three studies (.75 to .98) where participants selected previously assessed preferred work activities as their first choice when on-the-job (see Reid et al., 1998; Lattimore et al., 2002, 2003). At the time of publication, the assessment was still in preliminary stages of development; social and psychometric validity and clinical utility have yet to be published. The Basic Work Skills Assessment is a self-reference criterion measure originally developed in Thailand for adolescents with ASD (Suchart et al., 2015; Pongsaksri et al., 2017). The competency-based assessment aggregates two prompts (frequency and type) to measure work performance across two domains: work abilities (e.g., following instructions; sorting, filing, and copying with correct sequencing and organization; focused task completion) and work attitudes (social greetings; appropriate eye contact during social exchanges; listening while others are speaking; asking for help; problem-solving; taking initiative and showing responsibility). An examiner presents participants with three work-related tasks (each representing scaffolded levels of cognitive difficulty) while two clinicians observe and record the examiners prompts. Interrater and intrarater correlation coefficients on both work abilities and attitudes domains are high (interrater range: .93 to .98; intrarater range: .80 to .92; Pongsaksri et al., 2017). The authors indicate additional research is needed to determine the utility of the assessment for older adolescents as it may lack more advanced cognitive challenges to determine work-related competencies. Additionally, clinical utility and cross-cultural applications have yet to be determined. 23 Lerman, Grob, and colleagues (Lerman et al., 2017; Grob et al., 2019) developed a clinic- based assessment measuring work-related social skills in adolescents and adults with ASD in the U.S. Behavior analytic approaches (direct observation) are applied in a controlled, naturalistic setting where a set of common work conditions are presented to participants to evaluate how often they stay on-task and correctly perform work-related social skills. For example, participants are exposed to various work conditions (e.g., clear and vague instructions, insufficient materials to complete a task, problem solving when a task is outside their repertoire, multiple-step instructions) where they are assessed on the frequency of performing correct social responses, such as providing confirmation statements when given a task, asking for help, responding to corrective feedback, and notifying a supervisor of task completion. Across work- related social skill tasks that participants demonstrated, mean interobserver agreement ranged from 85% to 93% (Lerman et al., 2017). The authors note that social validity of the targeted social skills and clinical utility of the assessment both require further research inquiry. While these assessments represent burgeoning developments in addressing the critical gap in employment skills measures for individuals with ASD, they have various limitations. For example, there is no indication any measure was informed by career theories, psychometric properties were determined by small sample sizes (n < 50), clinical utility was rarely evaluated, and how the assessments impact employment outcomes (i.e., obtaining and maintaining employment) remains unknown. Clearly, there is a critical need for socially and psychometrically validated assessments for TAI-ASD that measure employment-related soft skills, preparation skills, and retention skills (Chen et al., 2015; Hedley et al., 2016; Murray et al., 2016; Scott et al., 2019). It is noted, however, that measures for occupation-specific skills is not included in this call to action; measuring these skills generally occurs in training programs 24 with summative evaluations for graduation, certification, and licenses and/or during periodic on- the-job performance reviews. Theoretical Framework: Social Cognitive Career Theory Initial development of the Employment Success assessments was guided by the tenets of Social Cognitive Career Theory (SCCT; Lent, Brown, & Hackett, 1994, 2000). Social cognitive career theory (Lent et al., 1994, 2000) is an integration of three well-established theories: social cognitive theory (Bandura, 1986a), self-efficacy theory (Hackett & Betz, 1981), and social learning theory (Krumboltz, 1979, 1994). SCCT is particularly applicable to transition-age individuals as it conceptualizes the adolescent-adulthood transition as a developmental process composed of “a lengthy preparation period followed by an extended period of adjustment to work and other life roles” (Lent et al., 1999, p. 299). The SCCT framework can be deconstructed into five models that share overlapping constructs but maintain conceptual distinctness (i.e., models of interest, choice, performance, satisfaction, and career self-management). Foundational to each model are three person-cognitive elements (self-efficacy, outcome expectations, and goals) that each operate in conjunction with personal, contextual, and environmental factors. Self-efficacy is how an individual perceives their capabilities when performing particular behaviors or courses of action (Bandura, 1986b). Choosing which challenges to engage in, how much effort to invest, and how long to persevere when faced with difficulties are all examples of self-efficacy (Bandura, 1999). Outcome expectations are beliefs about what is anticipated to occur as a result of engaging in particular behaviors or courses of action. Both self-efficacy and outcome expectation beliefs influence the choices and behaviors that individuals pursue (Lent et al., 1999). Goals are one’s intentions and determination to engage in a particular activity or to achieve a particular future outcome. Lent 25 (2012) suggests that ambitious goals are bolstered by strong self-efficacy and positive outcome expectations and that this relationship contributes to activating and sustaining performance efforts as well as persisting when faced with obstacles. Although all five models of the SCCT framework are complimentary across the adolescent-adulthood transition period, construction of the Employment Success assessments were specifically informed by and align with the performance model (see Figure 2.2). Figure 2.2 Social Cognitive Career Theory - Performance Model (adopted from Lent & Brown, 2013; Lent et al., 1994) Note. The highlighted boxes represent the three SCCT performance model domains that the Employment Success assessments target. The employability skills on the Employment Success assessments are represented by the Learning Experiences, Ability, Past Performances domain. The skills on each assessment are evaluated based on an individual’s self-efficacy and how often they perform the skills, represented by the Self-Efficacy and Performance Actions domains. The performance model interprets vocational performance based on two key components: the level of vocational success an individual achieves, and the degree to which an individual persists when encountering adversity and obstacles (Lent, 2012; Lent et al., 1994). In this model, vocational performance is an interaction between ability, self-efficacy, outcome expectations, and performance actions. Ability affects levels of performance both directly and indirectly. For example, the quality of TAI-ASD employability skills can directly affect their success in 26 securing and maintaining employment. Indirectly, self-efficacy and outcome expectations may be based on how they perceive their current employability skill capabilities, how well they have previously performed these skills, and the outcomes they have experienced in relation to past situations. Their level of self-efficacy and outcome expectations influence their vocational goals, which then influence the actions they engage in to achieve their goal of securing and maintaining employment (Lent et al., 1999). Several previous studies have examined the applicability of SCCT across diverse populations of transition-age individuals with disabilities, including youth with learning disabilities (Brown & Cinamon, 2016; Hampton & Mason, 2003; Ochs & Roessler, 2001), physical disabilities and developmental disabilities (Hutchinson et al., 2008), epilepsy (Sung & Connor, 2017), psychiatric disabilities (Milner et al., 2015; Willis, 2002), intellectual disabilities (Gibbons et al., 2016), and college students with disabilities from diverse racial and ethnic backgrounds (da Silva Cardoso et al., 2013). More recently, empirical studies have started evaluating the applicability of SCCT to transition-age individuals with ASD. For example, Wei and colleagues (2015) applied the SCCT framework in investigating the relationship between how STEM learning experiences and individual background characteristics in high school contribute to majoring in STEM disciplines at the postsecondary level for TAI-ASD. Farrow (2016) utilized the SCCT framework in exploring the relationships of work, career decision- making difficulties, and overall adjustment outcomes of college students with ASD. Connor (2017) applied the SCCT framework in examining the relationships between social functioning, self-efficacy, and psychological wellness of TAI-ASD. 27 Philosophical Framework: Participatory Action Research Foundational to this dissertation are the philosophical tenets of participatory action research (PAR). Just as the name implies, participation is the mindful and reflexive incorporation of multiple critical voices while promoting equitable cooperation; action is the process of change at sociocultural and individual levels; and research is the agent that stimulates the process (Kidd & Kral, 2005). PAR is a meta-methodology that promotes bridging the research-practice gap through action-oriented strategies rooted in social justice paradigms; this is achieved through a constructivist approach that incorporates multiple perspectives (Kiener & Koch, 2009). In other words, the intention of PAR is to develop conversations around frameworks of interpretation that reflect and respect multiple voices (Coghlan & Brydon-Miller, 2014). Foundational to PAR are a common set of values (e.g., empowerment, supportive relationships, social change, continuous improvement) that are expressed throughout a collaborative partnership with key stakeholders (Hergenrather et al., 2010). In the context of research, PAR produces greater meaning and relevancy, higher quality data, and more effective and applicable use of findings (Harley et al., 2007). This dissertation is guided by four PAR principles relevant to working with individuals with disabilities (Balcazar et al., 1998; Buettgen et al., 2012). The following describes how this study interprets and applies the principles while incorporating key stakeholders anticipated to be the direct users of the Employment Success: Soft Skills (ESSS) assessment, e.g., TAI-ASD, parents/caregivers, paraprofessionals, educators, rehabilitation professionals. 1. Key stakeholders participate directly in the process of defining, analyzing, and solving the problem. The problem has been well defined and articulated by autistic communities, communicated daily across academic and clinical practices, and documented thoroughly 28 in the research literature: (1) TAI-ASD experience disproportional challenges with securing, maintaining, and advancing in employment, and (2) there is a paucity of applicable academic and clinical assessment tools to help individuals and communities know which employment-readiness skills to target in transition programs. This dissertation specifically addresses the second problem, which requires a multiphase solution that incorporates multiple perspectives. As this study represents the second phase of scale development, PAR principles are applied by directly involving all key stakeholders in analyzing the feasibility and acceptability of the ESSS assessment across community, academic, and clinical settings. 2. Key stakeholders are directly involved in the research process, which facilitates a more accurate and authentic analysis of their social reality. This principle is associated with determining the social validity of the ESSS assessment. All key stakeholders will be asked to provide both quantitative and qualitative feedback about their experience taking the ESSS, suggestions for how to improve the assessment experience, the utility of the assessments both individually and across respective settings, and suggestions for how to improve utility. The ESSS will then be updated to reflect stakeholder perspectives; this step will maximize relevancy for multiple users across multiple settings and prepare the assessment for the next phase of validation. 3. The process of participatory research can increase awareness about strengths and resources. This principle is applied at both individual and systems levels. At the individual level, stakeholders—particularly TAI-ASD—will have an opportunity to explore perceived employability skill strengths and learn about strategies and resources that can help improve their confidence and skills. At the systems level, stakeholders— 29 particularly paraprofessionals, educators, and rehabilitation professionals—can evaluate whether the ESSS may help them make data-driven decisions about aligning transition programming with individual consumer needs, as well as consider whether existing programming address appropriate soft skill acquisition. 4. The goal of the research endeavor is to improve the quality of life for individuals with disabilities. This particular PAR principle is reflected in a number of ways. Employment is one of numerous quality-of-life domains that requires dynamic skillsets to secure, maintain, and advance in employment (Bishop et al., 2008). With the ESSS assessment identifying perceived skill strengths and skills that could be improved, TAI-ASD and the network of adults who support them may be able to clarify which soft skills to focus on improving, which skills to market to future employers as their perceived strengths, and which perceived strengths to utilize as a springboard towards employment advancement. Additionally, with feedback from TAI-ASD and other key stakeholders, the ESSS can be improved to have wider applicability across community, academic, and clinical settings; thus, having the potential to positively impact more individuals in their pursuit of employment and quality of life outcomes. The Role of Social Validity Social validity was first described by Wolf (1978) in the applied behavior analysis literature as a process of evaluating how acceptable and practical an intervention is for consumers who are participating in and/or impacted by the intervention. Examining potential facilitators and barriers of an intervention aids in understanding external validity factors that contribute to the implementation, adoption, and sustainability of the intervention. Wolf (1978, p. 207) suggests that social validation occur across three levels: 30 1. The social significance of the goals (i.e., are the specific behavioral goals really what society wants?) 2. The social appropriateness of the procedures (i.e., do participants, caregivers, and other consumers consider the treatment procedures acceptable?) 3. The social importance of the effects (i.e., are consumers satisfied with the results? All the results, including any unpredicted ones?) Thus, evaluating social validity is a process: first, feedback about the intervention is systematically gathered from consumers impacted by the intervention; then, their feedback is utilized to maintain and improve various elements of the intervention to increase its relevancy and sustainability in the community (Schwartz & Baer, 1991). While the concept of social validity has historically been connected to the field of applied behavior analysis, applications have expanded to a variety of social and health science disciplines (Carter & Wheeler, 2019). Furthermore, although social validity has more commonly been applied to intervention studies and program evaluation, this proposed study extends social validation concepts and processes to assessment development and implementation. In other words, while the concepts and processes are the same as those in intervention studies and program evaluation, they will be applied within an assessment context (Miller et al., 2013). For example, a question on a social validity survey for an intervention study may state, “the intervention procedures easily fit in with my current practices;” when adjusted to an assessment context the question may state, “the assessment procedures easily fit in with my current practices.” 31 CHAPTER 3: METHODS The overarching intention of this dissertation is to advance the previous development of the Employment Success: Soft Skills (ESSS) assessment. This advancement is guided by two objectives: (1) to determine the relevancy, practicality, and utility of the ESSS for academic and clinical settings that serve TAI-ASD; and (2) to determine the preliminary factor structure and psychometric reliability of the ESSS assessment. To achieve these aims, this study utilized (a) mixed-methods, participatory action research approach to evaluate the feasibility and social validity of the ESSS as articulated by multiple stakeholder perspectives, and (b) an exploratory factor analysis and Cronbach’s alpha to determine the preliminary factor structure and internal consistency (reliability) of the ESSS. The following key topics are discussed in this chapter: research design, participant demographics, recruitment and study procedures, measures utilized in the study, and quantitative and qualitative data analyses. Research Design This participatory action research (PAR) study is guided by a convergent parallel mixed- methods design (Creswell, 2014). In this type of research design, researchers collaborate with a variety of stakeholders in the community to incorporate their perspectives throughout the research process via the collection of quantitative and qualitative data that measure parallel concepts, such as feasibility and social validity (Balcazar et al., 2006). While the quantitative and qualitative data are analyzed separately, the results are then merged for a side-by-side comparison to determine how stakeholders’ perspectives converge or diverge. This type of study offers both generalized and in-depth understanding of the feasibility and social validity of the ESSS assessment. 32 Participants This study used a maximum variation strategy across purposive, convenience, and snowball sampling approaches to recruit participants living throughout the United States (Cohen et al., 2011; Etikan et al., 2016). The goal of these combined approaches was to maximize diverse and generalized perspectives across different types of academic, clinical, and community settings in order to evaluate and improve the relevancy, practicality, and utility of the ESSS assessment. To achieve this goal, outreach focused on various stakeholders who represent the potential end-users of the ESSS self-report and informant-report forms; for example, transition- age individuals with autism spectrum disorders (TAI-ASD) between 14-30 years old, parents/caregivers who are primary caregivers of a young adult with autism between 14-30 years old, and service providers who provide vocational training and supports to TAI-ASD (e.g., paraprofessionals, special educators, transition coordinators, vocational counselors, agency leadership, and academic professors). Figure 3.1 details inclusion criteria for stakeholders in this study. To note, autism diagnosis for TAI-ASD and parent/caregiver groups was self-reported as conducting formal diagnostics was outside the scope of this study. 33 Figure 3.1 Stakeholder Inclusion Criteria TAI-ASD Parents/Caregivers Service Providers • Are between 14-30 years • Are a parent/caregiver of • Are an educator, old and their own legal an autistic person clinician, or service guardian. between 14-30 years. provider who provides employment training or • Have been diagnosed • The autistic person they services to persons with with autism spectrum care for is learning about autism between 14-30 disorder (ASD), autistic the world of work, years old. disorder, Asperger’s looking for a job, or are syndrome, or pervasive currently working. • Can read at a 7th grade developmental disorder. level or higher. • The parent/caregiver can • Can read at a 7th grade read at a 7th grade level level or higher. or higher. • Are learning about the world of work, looking for a job, or are currently working. Across a six-month recruitment period, a total of 152 individuals expressed interest in participating in the study (see Table 3.1). (Note that it was difficult to calculate a response rate as the number of individuals who received the marketing materials is unknown.) Of the individuals who expressed interest, 26 did not meet the eligibility criteria, 46 were eligible but chose not to proceed with the study beyond the informed consent, and 80 were eligible and completed the study (these individuals are henceforth referred to as stakeholders). Of the 16 TAI-ASD that were ineligible to participate, approximately 81% had indicated they were both 18 years or older and not their own legal guardians. 34 Table 3.1 Responses by Stakeholder Group Eligible Eligible Stakeholder Ineligible (discontinued after (completed study) informed consent) TAI-ASD 16 9 20 Parent/Caregiver 3 8 16 Service Provider 7 29 44 The 80 stakeholders who were eligible to participate and completed the study were organized into two primary groups: consumers (n = 36) who included TAI-ASD and parents/caregivers, and service providers (n = 44) who provide vocational training and supports to TAI-ASD. The average age of TAI-ASD who participated in the study was 22.39 years (SD = 2.75, range = 17-27). Approximately two-thirds of the service provider group was represented by paraprofessionals and agency leadership, and the remaining one-third was represented by special educators, rehabilitation and mental health providers, and academic professors (see Table 3.2). Table 3.2 Distribution of Stakeholders Age Stakeholder Group n (%) Range (M/SD) Transition-age individuals with ASD 20 (55.6%) 22.39 (2.75) 17-27 Consumers Age 14-30 years (n = 36) Parents/Caregivers 16 (44.4%) Paraprofessionals 17 (38.6%) Paraeducators, Job Coaches Special Educators Special Ed Teachers, Transition 5 (11.4%) Service Coordinators Providers Counselors (n = 44) 8 (18.2%) Rehabilitation, Mental Health Agency Supervisors and Directors 11 (25.0%) Academic Professors 3 (6.8%) Note. M = mean; SD = standard deviation; ASD = autism spectrum disorders. 35 At the time of data collection, stakeholders were living in the following states: Alabama(1), Arizona(1), Arkansas(1), California(3), Colorado(6), Delaware(1), Florida(1), Georgia(1), Illinois(1), Kansas(2), Michigan(12), Minnesota(1), Montana(2), New York(1), Ohio(2), Oklahoma(19), Oregon(3), Pennsylvania(2), Tennessee(3), Texas(2), Utah(3), Virginia(2), Washington(2); seven stakeholders did not provide a response. The majority of stakeholders represented the following characteristics: TAI-ASD tended to be white men living in urban areas; parents/caregivers tended to be white women living in rural/suburban areas; and service providers tended to be white women living in rural/suburban areas (see Table 3.3). While some stakeholder perspectives from Black, Indigenous, People of Color (BIPOC) were represented in this study, representation was limited. For example, 11.2% of TAI-ASD, 46.2% of parent/caregivers, and 25.6% of service providers identified as BIPOC. Table 3.3 Characteristics Across All Stakeholder Groups Stakeholder Demographic n % Race/Ethnicity American Indian or Alaska Native 1 5.6 TAI-ASD Hispanic or Latinx 1 5.6 White or European American 16 88.8 Asian, Asian Indian, or Asian American 1 7.7 Black or African American 1 7.7 Parent/Caregiver Hispanic or Latinx 1 7.7 White or European American 7 53.8 Multiethnic 3 23.1 American Indian or Alaska Native 2 4.7 Black or African American 2 4.7 Service Provider Hispanic or Latinx 5 11.6 White or European American 32 74.3 Multiethnic 2 4.7 36 Table 3.3 (cont’d) Stakeholder Demographic n % Gender Identity Woman 3 17.6 TAI-ASD Man 13 76.5 Non-binary 1 5.9 Woman 8 61.5 Parent/Caregiver Man 3 23.1 Non-binary 2 15.4 Woman 38 88.4 Service Provider Man 4 9.3 Non-binary 1 2.3 Geographic Area Rural/Suburban 6 33.3 TAI-ASD Urban 12 66.7 Rural/Suburban 11 84.6 Parent/Caregiver Urban 2 15.4 Rural/Suburban 28 65.1 Service Provider Urban 15 34.9 TAI-ASD and parent/caregiver stakeholders offered additional characteristics related to their own or their child’s co-occurring conditions, work status, and vocational rehabilitation participation (VR; see Table 3.4). The three most common co-occurring conditions included ADD/ADHD, anxiety, and intellectual disability. To note, co-occurring conditions were self- identified as formal diagnostics were outside the scope of this study. While the majority of TAI- ASD were unemployed and looking for work, about one-third of each stakeholder group indicated they or their child were employed; a much smaller proportion of individuals were unemployed and not looking for work. For both stakeholder groups, the majority of individuals were not currently participating in their state’s VR programming. 37 Table 3.4 Consumer Characteristics Parent/Caregiver’s TAI-ASD Child with ASDa n % n % Co-Occurring Conditionsb ADD/ADHD 6 33.3 7 46.7 Anxiety 9 50.0 4 26.7 Bipolar 2 11.1 1 6.7 Depression 1 5.6 3 20.0 Dyscalculia 1 5.6 -- -- Dysgraphia 2 11.1 -- -- Dyslexia 1 5.6 -- -- Epilepsy -- -- 1 6.7 Gastrointestinal Disorder -- -- 3 20.0 Intellectual Disability 3 16.7 4 26.7 OCD 2 11.1 1 6.7 Personality Disorder 1 5.6 -- -- Work Status Employed 5 27.8 5 38.5 Unemployed, looking for a job 11 61.1 6 46.1 Unemployed, not looking for a job 2 11.1 2 15.4 Participation in VR Yes 7 35.0 5 31.2 No 11 55.0 7 43.8 Note. ADD/ADHD = attention deficit disorder/attention deficit hyperactive disorder; OCD = obsessive-compulsive disorder; VR = vocational rehabilitation. a Parents/caregivers provided responses in relation to their child’s characteristics. b Includes stakeholders who identified as having one or more co-occurring conditions. Service providers offered additional characteristics related to their highest level of education, how much experience they have serving TAI-ASD between 14-30 years old, their current work setting, and the size of their organization. The highest level of education for the majority of service providers was a master’s degree (39.5%), followed by a bachelor’s degree (27.9%) and some college (18.6%). Approximately half of all service providers had more than 10 years of experience serving TAI-ASD, and one-third of providers had 3-10 years of service. The work settings were fairly balanced across school districts (18.6%), colleges/universities (16.3%), and public agencies (13.9%), with a greater amount of service providers working in non-profit (27.9%) and private organizations (23.3%). Most service providers either worked in a small 38 organization with less than 100 employees (46.5%) or a very large organization with greater than 1,000 employees (34.9%); less than one-fifth of service providers worked in medium to large organizations (see Table 3.5). Table 3.5 Service Provider Characteristics n % Highest Level of Education High school 1 2.3 Some college 8 18.6 Associate’s degree 2 4.7 Bachelor’s degree 12 27.9 Master’s degree 17 39.5 Doctoral degree 3 7.0 Years of Experience Serving TAI-ASD < 3 years 6 14.0 3-5 years 9 20.9 6-10 years 6 14.0 11-15 years 13 30.2 16-20 years 4 9.3 20+ years 5 11.6 Work Setting College/university 7 16.3 Not-for-profit 12 27.9 Private organization 10 23.3 Public agency 6 13.9 School district 8 18.6 Size of Work Organization < 100 employees 20 46.5 100-999 employees 8 18.6 1000+ employees 15 34.9 Procedures After receiving Institutional Review Board approval (STUDY00006388), consumers and service providers were recruited to complete the study. Marketing materials were distributed throughout the U.S. to a variety of outlets (e.g., private, public, and non-profit programs of various sizes; advocacy groups; professional organizations; state agencies; academic institutions). Marketing materials included research study details, participant incentives, and a 39 link to the My Employment Success website (www.My-Employment-Success.weebly.com) where stakeholders could learn more about the study and access the Qualtrics experience management platform to participate in the study. For stakeholders who had limited access to technology, hard copies of all study materials were available as an alternate option to the digital platform. No hard copy requests were submitted by stakeholders. Stakeholders accessed the Qualtrics platform from either the marketing materials or the My Employment Success website. The welcome page in Qualtrics included (a) an outline of steps that stakeholders would complete in the study, and (b) eligibility questions. Stakeholders who met the eligibility criteria were then routed to the informed assent/consent. For TAI-ASD stakeholders under 18 years of age, they and their parent/guardian were asked to review an assent form to provide their permission to voluntarily participate; all other stakeholders who were 18 years or older reviewed a consent form. After completing the informed assent/consent form, stakeholders were then asked to complete and self-score the ESSS as a potential end-user of the assessment. TAI-ASD completed the self-report version, while parents/caregivers and service providers completed the informant- report version in relation to either their child or a specific TAI-ASD that they serve. The objectives of this first step were: (a) to engage stakeholders in experiencing the ESSS assessment as an end-user so they could then evaluate the social validity and feasibility of the assessment from a first-hand experience, and (b) to gather data to evaluate the preliminary reliability and factor structure of the self-report and informant-report versions of the assessment. After completing the ESSS assessment, stakeholders then completed social validity and feasibility surveys that included quantitative and qualitative feedback about their experiences completing and self-scoring the ESSS, and how well the assessment may translate into academic, 40 clinical, and community settings. The objectives of this second step were: (a) to use both quantitative and qualitative feedback to understand the relevancy, practicality, acceptability, and utility of the ESSS across different perspectives and settings; and (b) to implement stakeholder feedback to update and improve the ESSS so it critically reflects perspectives from projected assessment users. After completing social validity and feasibility surveys, stakeholders offered demographic information and selected their $25 gift card option. To protect stakeholder anonymity, a link was provided that routed stakeholders out of Qualtrics and into a Google form where they could then make their preferred gift card selection. Stakeholders selected their gift card from a variety of options, e.g., Panera Bread, Amazon, Petco, Home Depot, Target, Subway, and GameStop. All gift cards were distributed within two weeks after their completion of the study and were provided according to stakeholder preference (email or postal service). Measures Employment Success: Soft Skills Assessment The Employment Success assessments are a collection of assessments that focus on three primary components of work success: soft skills, preparation skills, and retention skills. The collection of assessments was originally developed using an iterative, Delphi method research design that incorporated subject matter expert perspectives to anonymously identify and consent on employment-related soft skills, preparation skills, and retention skills that were perceived to be essential to employment success for TAI-ASD (see Strain & Sung, 2019). Across three consensus-building iterations, a panel of 16 employers, practitioners, and researchers anonymously developed and consented on a total of 72 employability skills (23 soft skills, 25 employment preparation skills, and 24 employment retention skills). Guided by social cognitive 41 career theory’s (SCCT) performance model framework (Lent et al., 1994), the respective employability skills were then transformed into three Employment Success assessments that each include self-report and informant-report versions: Employment Success: Soft Skills (ESSS), Employment Success: Preparation Skills (ESPS), and Employment Success: Retention Skills (ESRS). Each assessment measures three key domains of the SCCT performance model framework: employability skills, self-efficacy, and performance actions. The intention of each assessment is to apply SCCT framework to identify employability skills that are perceived strengths, skills that are continuing to develop, and skills that may need improving based on a person’s level of confidence and how frequently they perform the skill on their own without help or reminders from others. Recommendations are also offered for how individuals can build upon their perceived skill strengths while also engaging in continuous skill improvement by engaging in confidence-building activities and/or direct practice. The scope of this study focuses specifically on the Employment Success: Soft Skills (ESSS) assessment (see Appendix A). The ESSS is appropriate for any TAI-ASD regardless of their employment status (unemployed or employed). Sample items on this assessment include “Have good hygiene (e.g., bathe, brush teeth); Show flexibility when things change; Use different ways to cope with stress (e.g., take a break, deep breathing); Show a positive attitude towards work; Plan my time to get tasks done.” The assessment takes about 20 minutes to complete, including self-administration and self-scoring. Based on a composite of seven readability formulas (https://readabilityformulas.com), the Employment Success: Soft Skills assessment reflects approximately a 6th-7th grade reading level. Completing items on the ESSS is a process of responding to two self-administered prompts that are each rated on 4-point Likert scales. The first prompt measures the self-efficacy 42 domain on the SCCT’s performance model; the prompt states, “How confident are you that you can do the skill at work on your own without help or reminders?” The Likert anchors on the confidence scale are 1 = I am not confident to 4 = I am totally confident. The second prompt measures the performance actions domain on the SCCT’s performance model; the prompt states, “How often do you perform the skill at work on your own without help or reminders?” The Likert anchors on the frequency scale are 1 = I never perform the skill on my own to 4 = I always perform the skill on my own. As soft skills are generalizable across settings, if the user has never had a job, they are then prompted to think about how they perform the skill in other settings (e.g., volunteering, home, school, or in the community). After users record their responses on the confidence and frequency scales, they then proceed to the Understanding Your Answers section where they self-score their responses. Using a crosstabulation table, users merge their responses on the confidence and frequency scales to generate a single response for each item that represents a continuum of skill development, from soft skills that may need improving to soft skills that are perceived strengths. Soft skills are organized across four developmental categories represented by the letters G-V-E-S. (To note, these letters were initially selected as placeholders in this study to test user responses to seemingly random letters versus other naming conventions. In hindsight, this decision would have benefited from more rigorous vetting before being implemented in this study as the G-V-E- S naming convention was a source of confusion for some stakeholders. More details about their confusion are discussed in chapters 4 and 5 of this document.) The following description outlines the process for organizing skills across each G-V-E-S category: • Skills organized in the G-category are highlighted in a peach color on the crosstabulation table on the ESSS. These are skills that were rated a 1 or 2 on both the confidence and 43 frequency scales. Skills that fall in this category are identified as areas for improvement that may benefit from both confidence-building and additional direct practice. • Skills organized in the V-category are highlighted in a light-yellow color on the crosstabulation table on the ESSS. These are skills that were rated a 1 or 2 on the confidence scale and a 3 or 4 on the frequency scale. Skills that fall in this category are identified as skills that are becoming strengths that may be improved through confidence- building activities. • Skills organized in the E-category are highlighted in a light-yellow color on the crosstabulation table on the ESSS. These are skills that were rated a 3 or 4 on the confidence scale and a 1 or 2 on the frequency scale. Skills that fall in this category are also identified as skills that are becoming strengths that may be improved from additional direct practice. • Skills organized in the S-category are highlighted in a light-green color on the crosstabulation table on the ESSS. These are skills that were rated a 3 or 4 on both the confidence and frequency scales. Skills that fall in this category are identified as perceived strengths that may further support an individual in obtaining, maintaining, and advancing in employment. Closing out the Understanding Your Answers section are several suggestions for how users can continue to improve their confidence and skill development. Some of the suggestions include: taking a class that focuses on learning and practicing soft skill development using different techniques, such as role playing; use work-based learning experiences, internships, or volunteering to practice soft skills; ask for honest feedback from people whom they trust to gauge how skills are improving; work with a career counselor at school, a Vocational 44 Rehabilitation agency, or a Department of Labor agency; seek supports from a mentor or job coach; read books, blogs, and articles or watch videos about soft skills; join a job club. Feasibility and Social Validity Surveys After completing the ESSS assessment, stakeholders are asked to complete two surveys that measure social validity and feasibility domains (see Appendix B). Items for the two surveys were developed under guidance from the academic literature published by Bowen et al. (2009), Fawcett (1991), Schwartz and Baer (1991), and multidisciplinary scholars affiliated with the Usage Rating Profile suite of instruments (see https://urp.uconn.edu/). Feasibility. The first survey for stakeholders focuses on measuring feasibility domains and includes two parallel versions: one version gathers feedback from consumers (TAI-ASD and parents/caregivers), while the other gathers feedback from service providers (see Table 3.6). Table 3.6 Feasibility Domains Measured by Stakeholder Group Affiliation Survey Items for Survey Items for Service Cronbach’s alpha Feasibility Domain (Consumer / Service Consumer Group Provider Group Provider) Desirability 7 9 .92 / .86 Applicability 4 6 .71 / .75 Collaboration 2 2 .81 / .77 Understanding 6 5 .72 / .65 System Climate -- 4 -- / .85 System Support -- 4 -- / .81 Qualitative Responses 1 1 -- The consumer feasibility survey includes 19 items rated on a 6-point Likert scale of agreement (1 = strongly disagree to 6 = strongly agree) that measures four domains related to desirability, applicability, collaboration, and understandability. Sample items include: “This assessment is a good way to help people” (desirability domain); “This assessment was too much 45 work for me” (applicability domain); “Collaboration with others is required in order to use this assessment, e.g., professionals, parents/caregivers” (collaboration domain); “I was able to do every step of the assessment with little or no help” (understanding domain). When determining overall feasibility of the ESSS assessment for consumers, items affiliated with the applicability and collaboration domains are reverse scored before aggregating totals; higher mean scores suggest stronger feasibility of the ESSS assessment. In this study, the Cronbach’s alpha for each of the consumer feasibility domains are as follows: desirability (.92), applicability (.71), collaboration (.81), understanding (.72). In addition, the consumer feasibility survey also includes one qualitative response that asks an open-ended question about how the ESSS assessment can be improved. The service provider feasibility survey includes 30 items rated on a 6-point Likert scale of agreement (1 = strongly disagree to 6 = strongly agree) that measures six domains related to desirability, applicability, collaboration, understandability, system climate, and system support. Sample items include: “This assessment is a good way to assess young adult's employment- related skills” (desirability domain); “The total time required to implement the assessment procedures would be manageable” (applicability domain); “Regular communication with others is needed to implement the assessment procedures, e.g., professionals, parents/caregivers” (collaboration domain); “I understand how to use this assessment” (understanding domain); “My administrator/supervisor would be supportive of my use of this assessment” (system climate domain); “I would need additional resources to carry out the assessment” (system support domain). When determining overall feasibility of the ESSS assessment for service providers, individual items that are negatively worded are reverse scored, along with all items affiliated with the collaboration and system support domains. Items are then aggregated; higher mean 46 scores suggest stronger feasibility of the ESSS assessment. In this study, the Cronbach’s alpha for each of the service provider feasibility domains are as follows: desirability (.86), applicability (.75), collaboration (.77), understanding (.65), system climate (.85), system support (.81). Similar to the consumer survey, the service provider feasibility survey also includes one qualitative response that asks an open-ended question about how the ESSS assessment can be improved. Social Validity. The second survey is for all stakeholders (consumers and service providers) and measures social validity domains related to stakeholder experiences completing the ESSS. The survey gathers quantitative and qualitative data that measures the understandability, clarity, relevance, ease of use, and utility pertaining to the structural components of the ESSS assessment (instructions, rating of items, scoring your answers, and understanding your answers; see Table 3.7). Table 3.7 Social Validity Domains Measured Across Structural Components of the ESSS Assessment Survey Items in Cronbach’s Section on ESSS Social Validity Domain Each Domain alpha Understandability 2 .95 Introduction Section Clarity 2 Qualitative Response 1 Understandability 23 .98 Clarity 23 Rating of Items Section Relevance 23 Qualitative Response 23 Understandability 1 .89 Scoring Your Answers Clarity 1 Section Ease of Use 4 Qualitative Response 1 Understandability 1 .93 Understanding Your Clarity 1 Answers Section Utility 2 Qualitative Response 1 Note. ESSS = Employment Success: Soft Skills. 47 Quantitatively, stakeholders use a 4-point Likert scale of agreement (1 = strongly disagree to 4 = strongly agree) to rate (a) the understandability and clarity of the introduction section, how items are written, and the scoring and understanding your answers sections; (b) whether each item is a relevant soft skill; (c) how easy it is to self-score their responses; and (d) the overall utility of the ESSS for identifying perceived soft skill strengths, skills that could be improved, and suggestions for how to improve skills. There are a total of 83 Likert-scale items on the social validity survey; higher mean scores suggest stronger social validity of the ESSS assessment. In this study, the Cronbach’s alpha for social validity across each of the ESSS sections are as follows: introduction section (.95), rating of items section (.98), scoring your answers section (.89), understanding your answers section (.93). Qualitatively, stakeholders offer additional feedback about their experiences completing the ESSS assessment and suggestions for how to improve the assessment. For example, in each of the respective ESSS sections, an open-ended question asks stakeholders to offer comments and suggestions for how to improve the introduction, scoring your answers, and understanding your answers sections. Additionally, for each of the 23 items on the ESSS, stakeholders can offer comments or suggestions for how to improve the respective item. Demographics Survey The demographics survey collects background information about stakeholders to further understand their identities and representations. Figure 3.2 describes the different demographic information collected for each stakeholder group. 48 Figure 3.2 Demographic Information Collected Across Stakeholder Groups TAI-ASD Parents/Caregivers Service Providers • Age • State they live in • State they live in • State they live in • Geographic area they • Geographic area they live in (urban, live in (urban, suburban, • Geographic area they suburban, rural) rural) live in (urban, suburban, rural) • Gender identity • Gender identity • Gender identity • Race/Ethnicity • Race/Ethnicity • Race/Ethnicity • Years of experience In relation to the person working with autistic • Co-occurring with autism they care for: individuals between conditions 14-30 years old • Autism diagnosis • School status • Highest level of • Co-occurring • Work status conditions education • Vocational • Type of setting they • School status Rehabilitation currently work in and participation • Work status their job title • Vocational • Size of organization they Rehabilitation work in participation Data Analysis All data were analyzed using IBM SPSS Statistics version 27. Prior to the main analyses, data were examined for outliers and missing data. Outliers were examined for possible data entry and measurement errors, none of which occurred. Upon further evaluation, it was determined these data points were genuinely unusual values when compared to other participant responses in the respective stakeholder groups. To critically account for all individual and collective stakeholder voices (a participatory action research approach; Corbin Dwyer & Buckle, 2009), all data outliers were included in the analyses. However, data that exceeded 25% missingness on either the ESSS, social validity, or feasibility surveys were eliminated from respective analyses. 49 All remaining missing data (.58% of social validity and .47% of service provider feasibility) were < 5% missing and were subsequently addressed using single imputation (mean). Normality was evaluated using histogram graphs, Q-Q plots, skewness, kurtosis, and Shapiro-Wilk tests. Any deviations from normality were minor, suggesting parametric analyses remained valid for the data set (Blanca et al., 2017). Respective correlations were conducted using: (a) Pearson’s r for interval variables, (b) Spearman’s rho for dichotomous variables, and (c) Eta for categorical x interval variables ( = √2 derived from a one-way ANOVA model). Descriptive statistics, inferential statistics, and qualitative content analyses (Cohen, 2011) were conducted to evaluate the social validity and the feasibility of the ESSS assessment, to identify where stakeholder perspectives converge and diverge, and to determine what adjustments are necessary to improve the ESSS assessment. Multivariate analysis of variance (MANOVA) was the primary statistical modeling approach for evaluating social validity and feasibility. Demographic variables that demonstrated significant correlations (p < .05) with the outcome variables were included as fixed factors in respective MANOVA models. For practicality, full factorial MANOVA models were not considered due to sample size limitations and interpretation complexities related to sample size in the context of potential interactions among independent variables. Therefore, for purposes of the present study, all MANOVA models were conducted as main effects models. If sample sizes were relatively equal and Box’s M test was non-significant (p > .001), Wilks’ Lambda () was used for the multivariate omnibus F-test; otherwise, if assumptions assessed by these parameters were violated, Pillai’s Trace was used instead. If multivariate omnibus F-tests alphas were significant (p < .05), univariate ANOVAs with Bonferroni corrections were then conducted to determine at the univariate level where stakeholder perspectives significantly differed on specific dependent variables. 50 To note, while nonparametric models may be a viable option for data that is considered non-normal, there is empirical evidence suggesting the univariate ANOVA F-test is robust to minor normality violations (Blanca et al., 2017). Thus, MANOVA was determined the most suited inferential statistic approach for this study. Specific data analyses that align with the study objectives are outlined in greater detail below. Social Validity Social validity was evaluated across the four structural components of the ESSS assessment, i.e., the introduction section, rating of items section, scoring your answers section, and understanding your answers section. To answer research question #1 (what is the social validity of the ESSS assessment among projected users of the instrument?), research question #1.1 (do stakeholder perspectives meet the benchmark for social validity of the ESSS?), and research question #3 (what updates are necessary to improve the ESSS assessment?), a series of descriptive statistics (mean, standard deviation, effect size) were conducted for a generalized understanding of stakeholder perspectives across various social validity domains (understandability, clarity, relevance, ease of use, utility). First, descriptive statistics for each stakeholder group (TAI-ASD, parent/caregiver, service provider) were conducted for the social validity grand means across each of the ESSS sections. For any grand mean that did not meet the social validity benchmark (3 on a 4-point scale), descriptive statistics were then conducted across each of the social validity domains being measured in the respective ESSS section. Last, for any social validity domain that did not meet the benchmark, descriptive statistics were then conducted at the item level. Any item, social validity domain, or ESSS section that did not meet the benchmark were flagged as critical components that may need to be improved. 51 To note, while higher mean scores can be interpreted as stronger perceived social validity, the social validity benchmark (3 on a 4-point scale) was established based on multiple considerations. First, a 3 represents the entry point of agreement on the 4-point Likert scale used on the social validity survey; thus, if a mean of 3 was achieved, it may suggest a minimal level of group agreement. Additionally, this study is exploratory in nature to further scaffold scale development and clinical utility; this generalized approach is necessary and will subsequently support future studies where more rigorous approaches will evaluate psychometric structure and stability of the assessment (Boateng et al., 2018). To answer research question #2 (what are the similarities and differences in social validity of the ESSS across different types of stakeholders?), inferential statistics were conducted for a more nuanced understanding of the degree to which group means (independent variable) differed on their social validity perspectives (dependent variables). First, correlation analyses were conducted between demographic variables of interest and social validity domains to determine whether any additional independent variables should be included in the main analysis. No demographic variables accounted for significant variance (p < .05) in the dependent variables, therefore fixed factors were not included in the main analysis. Second, a correlation analysis was conducted to determine whether multicollinearity existed between of the four structural components of the ESSS assessment (the social validity dependent variables). See Appendix C for the above correlation tables. While dependent variables were significantly correlated (p > .001), there was no evidence of multicollinearity (r > 0.9). Thus, the primary analysis consisted of a one-way MANOVA model. 52 Feasibility Feasibility was evaluated from two primary perspectives: consumers and service providers. The consumer group included perspectives from TAI-ASD and parents/caregivers. The service provider group included two subgroup perspectives based on their level of education (i.e., paraprofessionals with high school diplomas through bachelor’s degrees, and professionals with master’s degrees or higher). To answer research question #1 (what is the feasibility of the ESSS assessment among projected users of the instrument?), research question #1.2 (do stakeholder perspectives meet the benchmark for feasibility of the ESSS?), and research question #3 (what updates are necessary to improve the ESSS assessment?), descriptive statistics (mean, standard deviation, effect size) were conducted for the consumer group across four feasibility domains (desirability, applicability, collaboration, and understanding). Descriptive statistics were conducted for the service provider group across six feasibility domains (desirability, applicability, collaboration, understanding, system climate, and system support). To note, individual items that are negatively worded are reverse scored before aggregating means. For any mean that did not meet the benchmark (4 on a 6-point scale), descriptive statistics were then conducted at the item level for a deeper understanding of stakeholder perspectives. Any item or feasibility domain that did not meet the benchmark were flagged as critical components that may need to be improved. To further note, while higher mean scores generally suggest stronger perceived feasibility of the ESSS, the feasibility benchmark (4 on a 6-point scale) was established based on the same considerations discussed for the social validity benchmark (e.g., 4 is the entry point of agreement on the 6-point Likert scale, and the exploratory nature of the study). With that said, there are two domain exceptions where lower mean scores suggest stronger perceived feasibility: (a) lower 53 mean scores in the collaboration domain for both consumer and service provider groups suggest less collaborations are needed; and (b) lower mean scores in the system support domain for the service provider group suggests additional resources are not needed to effectively use the assessment. As feasibility interpretations for these two domains are reverse, the equivalent benchmark for these domains would be 3 or lower on the 6-point scale. To answer research question #2 (what are the similarities and differences in feasibility of the ESSS across different types of stakeholders?), inferential statistics were conducted for a more nuanced understanding of the degree to which respective group means (independent variable) differed on their feasibility perspectives (dependent variables). First, to ensure parallel comparisons across all feasibility domains, reverse-scored items and reverse-scored means on the above noted domains were generated (consumer and service provider collaboration, and service provider system support). Second, correlational analyses between demographic variables of interest and feasibility domains were each conducted for the consumer and service provider groups; demographic variables with significant correlations (p < .05) were included as covariates in the main analyses. Third, correlation analyses were conducted to determine whether multicollinearity existed between the respective consumer and service provider dependent variables (i.e., four consumer feasibility domains, and six service provider feasibility domains). See Appendix C for the above correlation tables. While most dependent variables were significantly correlated (p > .05), there was no evidence of multicollinearity (r > 0.9). Lastly, the main analyses included two multivariate analysis of variance models (MANOVA). The MANOVA main effects model for the consumer group included geographic area and school status as fixed factors. The MANOVA model for the service provider group did not include any fixed factors. 54 Qualitative Analysis To further answer research question #2, a qualitative content analysis (Cohen et al., 2011) was conducted to understand similarities and differences in stakeholder perspectives regarding the ESSS’s feasibility and social validity. Four open-ended questions asked stakeholders to (a) offer generalized feedback about how the ESSS assessment could be improved and (b) more specific comments pertaining to each of the structural components of the ESSS assessment (introduction section, rating of items section, and scoring and understanding your answers sections). To minimize bias, the content analysis results were triangulated with a doctoral-level colleague who is also a certified rehabilitation counselor. Triangulated content analysis results were then merged with the quantitative data results for a more in-depth and nuanced understanding of stakeholder perspectives. The ESSS assessment will be subsequently updated according to these merged results. Additionally, specific content and structural adjustments to the ESSS assessment will be implemented when three or more stakeholders (from any group) offered the same or similar recommendations for improvement. This comprehensive approach to updating the ESSS critically prioritizes voices and experiences of stakeholders and aligns with participatory action research approaches (Corbin Dwyer & Buckle, 2009). Preliminary Reliability and Factor Structure of the ESSS To evaluate the preliminary reliability of the self- and informant-report forms of the ESSS (research question #3), a Cronbach’s alpha was conducted to determine the internal consistency of the confidence scale and the frequency scale across each stakeholder group (TAI- ASD, parents/caregivers, service providers). To further evaluate the internal consistency of the confidence and frequency scales, classical test theory was applied to evaluate item means, 55 standard deviations, item-total correlations, and Cronbach alpha adjustments when the item is deleted were considered across each stakeholder group (Raykov & Marcoulides, 2011). To evaluate the preliminary factor structure of the confidence and frequency scales (research question #3), the correlation matrix was first examined to justify conducting an exploratory factor analysis. For both the confidence and frequency scales, the 2 for the Bartlett test of sphericity was significant (p < .001), and the Kaiser-Meyer-Olkin test was > 0.80, indicating the items on each scale are interrelated and the respective data are adequate for factor analysis. An exploratory factor analysis was then conducted for the confidence scale and the frequency scale using principal axis factoring with direct oblimin rotation. An oblique rotation was selected as factors are assumed to be correlated and presumably suggest a latent construct that represents soft skills (Howard, 2016). The number of factors retained was based on the following criteria: (a) the inflection point on the scree plot; (b) a parallel analysis; (c) the .40-.30- .20 rule; and (d) the interpretability of factors (Howard, 2016; Patil et al., 2007). 56 CHAPTER 4: RESULTS This chapter discusses the following data analyses results: social validity across all stakeholders, consumer feasibility, service provider feasibility, similarities and differences in stakeholder perspectives for social validity and feasibility, qualitative responses from stakeholders, and the preliminary reliability and factor structure of the ESSS assessment. Social Validity and Feasibility of the ESSS Assessment The following results align with research question #1 (what is the social validity and feasibility of the ESSS assessment among projected users of the instrument?), research question #1.1 (do stakeholder perspectives meet the benchmark for social validity of the ESSS?), and research question #3 (what updates are necessary to improve the ESSS assessment?). Descriptive statistics are first reviewed for social validity outcomes, followed by feasibility outcomes for the consumer group and then for the service provider group. Social Validity Three stakeholder groups (TAI-ASD, caregivers, service providers) offered their perspectives across five social validity domains (understandability, clarity, relevance, ease of use, and utility) in relation to four structural components of the ESSS assessment (introduction section, rating of items section, scoring your answers section, and understanding your answers section; see Appendix D for item-level results). Both grand means for the caregiver and service provider groups met the minimum benchmark (3 on a 4-point scale) on the introduction section of the ESSS (caregiver: M = 3.15, SD = 0.66; service provider: M = 3.14, SD = 0.46). These results suggest that on average caregivers and service providers agreed that the introductory instructions and how to rate items were understandable and clearly written. However, the benchmark for TAI-ASD was not achieved for the introduction section (M = 2.88, SD = 1.07), 57 therefore mean responses were then evaluated across the social validity domains (understandability and clarity) and then at the item-level. On average, the TAI-ASD group suggested the introductory instructions and how to rate items were not very clearly described (M = 2.95, SD = 1.10 and M = 2.84, SD = 1.09, respectively). This lack of narrative clarity presumably may have led to some difficulty with understanding the introductory instructions and how to rate items (M = 2.89, SD = 1.12 and M = 2.84, SD = 1.09, respectively). See Table 4.1. Table 4.1 Social Validity Results for the Introduction Section on the ESSS Social Validity Domain: Stakeholdera M (SD) f p Introduction Section TAI-ASD 2.88 (1.07) 0.16 .42 Grand Mean Caregiver 3.15 (0.66) Service Provider 3.14 (0.46) TAI-ASD 2.87 (1.09) 0.18 .33 Understanding (domain) Caregiver 3.17 (0.62) Service Provider 3.13 (0.45) TAI-ASD 2.89 (1.12) The instructions section is Caregiver 3.20 (0.77) easy to understand (item) Service Provider 3.18 (0.50) TAI-ASD 2.84 (1.09) How to rate items is Caregiver 3.13 (0.64) easy to understand (item) Service Provider 3.08 (0.57) TAI-ASD 2.89 (1.07) 0.16 .40 Clarity (domain) Caregiver 3.14 (0.74) Service Provider 3.15 (0.51) TAI-ASD 2.95 (1.10) The instructions section is Caregiver 3.13 (0.74) written clearly (item) Service Provider 3.18 (0.50) TAI-ASD 2.84 (1.09) How to rate items is Caregiver 3.15 (0.74) written clearly (item) Service Provider 3.13 (0.56) Note. Bolded M(SD) indicates the social validity domain/item did not achieve the benchmark (3 on a 4-point scale). aSample size for each stakeholder is as follows: TAI-ASD = 20, caregiver = 15, service provider = 40. The rating of items section on the ESSS is where users rate their confidence and frequency of performance across 23 soft skills. On the social validity survey, stakeholders were asked whether each of the 23 ESSS items were understandable, if each item was written clearly, 58 and if they considered the item a relevant soft skill. The grand means across all stakeholder groups met the minimum benchmark for overall social validity of the rating of items section on the ESSS (TAI-ASD: M = 3.61, SD = 0.49; caregiver: M = 3.63, SD = 0.43; service provider: M = 3.47, SD = 0.44). See Table 4.2. Table 4.2 Social Validity Results for the Rating of Items Section on the ESSS Social Validity Domain: Stakeholdera M (SD) f p Rating of Items Section TAI-ASD 3.61 (0.49) 0.20 .25 Grand Mean Caregiver 3.63 (0.43) Service Provider 3.47 (0.44) TAI-ASD 3.63 (0.56) 0.18 .29 Understanding (domain) Caregiver 3.65 (0.44) Service Provider 3.45 (0.46) TAI-ASD 3.65 (0.49) 0.21 .20 Clarity (domain) Caregiver 3.62 (0.45) Service Provider 3.44 (0.47) TAI-ASD 3.56 (0.47) 0.11 .67 Relevance (domain) Caregiver 3.62 (0.42) Service Provider 3.51 (0.43) aSample size for each stakeholder is as follows: TAI-ASD = 20, caregiver = 15, service provider = 40. The grand mean for the caregiver group met the minimum benchmark on the scoring section of the ESSS (M = 3.19, SD = 0.46). These results suggest that on average the caregiver group agreed the directions for how to score their answers were easy to understand and were written clearly, the overall ease of recording scores and using the scoring table was acceptable, and that it generally would be easy for others to score the ESSS on their own (e.g., parents, teachers, paraprofessionals, job coaches, etc.). Comparatively, the grand means for TAI-ASD and service provider groups did not meet the social validity benchmark (TAI-ASD: M = 2.95, SD = 0.83; service provider: M = 2.93, SD = 0.52), therefore mean responses were then evaluated 59 across the social validity domains (understandability, clarity, ease of use) and then at the item- level. Results suggest that while both stakeholder groups on average agreed the directions were easy to understand (TAI-ASD: M = 3.20, SD = 0.95; service provider: M = 3.17, SD = 0.59) and were written clearly (TAI-ASD: M = 3.26, SD = 0.96; service provider: M = 3.17, SD = 0.59), there was more variability in their perspectives on the overall ease of scoring the assessment. For example, while the service provider group on average agreed the scoring table was easy to use (M = 3.05, SD = 0.67), the TAI-ASD group suggested experiencing a little difficulty (M = 2.75, SD = 0.97). Furthermore, both groups on average expressed some challenges with recording their scores (TAI-ASD: M = 2.70, SD = 0.98; service provider: M = 2.92, SD = 0.72). With that said, all groups (TAI-ASD, caregivers, service providers) agreed that in general autistic individuals may have greater difficulty scoring the assessment on their own (TAI-ASD: M = 2.65, SD = 0.88; caregivers: M = 2.57, SD = 0.85; service provider: M = 2.23, SD = 0.96), whereas other people—such as parents, teachers, paraprofessionals, job coaches, etc.—may have less difficulty with scoring independently (TAI-ASD: M = 3.15, SD = 0.93; caregivers: M = 3.29, SD = 0.47; service provider: M = 3.05, SD = 0.50). See Table 4.3. Table 4.3 Social Validity Results for the Scoring Section on the ESSS Social Validity Domain: Stakeholdera M (SD) f p Scoring Section TAI-ASD 2.95 (0.83) 0.18 .34 Grand Mean Caregiver 3.19 (0.46) Service Provider 2.93 (0.52) The directions in this section are TAI-ASD 3.20 (0.95) 0.14 .50 easy to understand Caregiver 3.43 (0.65) (domain: understandability) Service Provider 3.17 (0.59) TAI-ASD 3.26 (0.96) 0.14 .51 The directions in this section are Caregiver 3.43 (0.65) written clearly (domain: clarity) Service Provider 3.17 (0.59) 60 Table 4.3 (cont’d) TAI-ASD 2.81 (0.82) 0.16 .40 Ease of Use (domain) Caregiver 3.07 (0.47) Service Provider 2.81 (0.59) TAI-ASD 2.75 (0.97) The scoring table is easy to use Caregiver 3.21 (0.58) (item) Service Provider 3.05 (0.67) TAI-ASD 2.70 (0.98) Recording scores is easy to do Caregiver 3.21 (0.58) (item) Service Provider 2.92 (0.72) In general, it would be easy for TAI-ASD 2.65 (0.88) autistic youth to score the Caregiver 2.57 (0.85) assessment on their own (item) Service Provider 2.23 (0.96) In general, it would be easy for TAI-ASD 3.15 (0.93) others to score the assessment on Caregiver 3.29 (0.47) their own e.g., parents, teachers, parapros, job coaches, etc. (item) Service Provider 3.05 (0.50) Note. Bolded M(SD) indicates the social validity domain/item did not achieve the benchmark (3 or higher on a 4-point scale). aSample size for each stakeholder is as follows: TAI-ASD = 20, caregiver = 14, service provider = 41. Further considering stakeholder perspectives on scoring the ESSS, accuracy of self- scoring the ESSS during the study was examined across participants to determine the amount of scoring errors within each stakeholder group and the percent of participants who consistently mis-scored their results (as defined by mis-scoring 3 or more items on the ESSS). Scoring errors across groups varied, with the caregiver group having the highest errors (17.12%), followed by the TAI-ASD group (15.43%) and the service provider group (11.96%). The differences between groups are statistically significant, Welch’s F(2, 792.96) = 3.49, p = .031. A Games-Howell post hoc analysis revealed the difference was only between the caregiver and service provider groups (.052, 95% CI (.002, .101), p = .037). The service provider group had fewer participants who consistently mis-scored their results (20.45%), whereas the caregiver and TAI-ASD groups each had 25% of participants with mis-scoring consistencies. Upon further evaluation, most scoring 61 errors appeared to be related to the participant transposing their answers from the confidence and frequency scales when using the scoring crosstabulation table to generate their results. The grand means across all stakeholder groups met the social validity benchmark for the understanding your answers section on the ESSS (TAI-ASD: M = 3.38, SD = 0.57; caregiver: M = 3.61, SD = 0.46; service provider: M = 3.28, SD = 0.54). Stakeholders on average endorsed the following: the section is easy to understand and is written clearly; the section is useful for understanding which soft skills are perceived strengths, which skills could be improved, and how they can improve these skills (see Table 4.4). Table 4.4 Social Validity Results for the Understanding Your Answers Section on the ESSS Social Validity Domain: Stakeholdera M (SD) f p Understanding Your Answers Section TAI-ASD 3.38 (0.57) 0.23 .18 Grand Mean Caregiver 3.61 (0.46) Service Provider 3.28 (0.54) TAI-ASD 3.47 (0.61) 0.17 .36 Understanding (domain) Caregiver 3.65 (0.47) Service Provider 3.25 (0.59) TAI-ASD 3.47 (0.61) 0.25 .13 Clarity (domain) Caregiver 3.62 (0.51) Service Provider 3.28 (0.55) TAI-ASD 3.29 (0.65) 0.20 .26 Utility (domain) Caregiver 3.58 (0.49) Service Provider 3.29 (0.54) aSample size for each stakeholder is as follows: TAI-ASD = 19, caregiver = 13, service provider = 40. Feasibility – Consumers For the consumer group (TAI-ASD and parents/caregivers), feasibility was measured across four domains: desirability, understanding, collaboration, and applicability (see Table 4.5 for domain results and Appendix D for item-level results). Both TAI-ASD and caregiver group means met the benchmark (4 on a 6-point scale) on the desirability domain (TAI-ASD: M = 4.21, SD = 1.17; caregiver: M = 4.52, SD = 1.10; d = 0.27), suggesting that on average both 62 groups were excited to try the ESSS, they liked the assessment and would volunteer to take it again, they felt the ESSS is a good way to help people, and they would recommend it to others who were having trouble finding or keeping a job. Both group means also met the benchmark for the understanding domain (TAI-ASD: M = 4.75, SD = 0.96; caregiver: M = 4.88, SD = 0.68; d = 0.15), suggesting that on average they understood how to complete the ESSS, they felt they were able to do so correctly with little or no assistance, and the assessment helped them understand skill areas that may benefit from more training or practice. The TAI-ASD group mean (M = 2.53, SD = 1.53) met the benchmark (3 or lower on a 6-point scale) for the collaboration domain, suggesting that on average they did not think it was necessary to collaborate and have regular communication with other professionals and caregivers to effectively use the assessment. However, the caregiver group mean (M = 3.09, SD = 1.58; d = 0.36) was slightly above the benchmark, therefore an item-level evaluation was initiated to identify how perspectives specifically differ. While caregivers on average suggested that regular communication with professionals was not necessary to complete the ESSS (M = 2.75, SD = 1.65), collaboration with professionals may be needed to use the ESSS effectively (M = 3.44, SD = 1.97). Both the TAI-ASD and caregiver groups did not meet the benchmark (4 on a 6-point scale) for the applicability domain (TAI-ASD: M = 3.97, SD = 1.35; caregiver: M = 3.98, SD = 1.07; d = 0.01). While both means are marginally below the benchmark, further investigation at the item-level will provide more clarity about group perspectives. On average, the TAI-ASD group means suggested that while completing the ESSS did not get in the way of doing other things (M = 4.70, SD = 1.38) and it was not too much work for them (M = 4.30, SD = 1.63), they agreed on average there may have been too many steps on the ESSS (M = 3.55, SD = 2.11) and that it took longer than expected to complete (M = 3.35, SD = 1.84). Whereas, the caregiver 63 group means suggested that while completing the ESSS did not get in the way of doing other things (M = 4.31, SD = 1.70) and it did not take longer than expected to complete (M = 4.19, SD = 1.47), they suggested it may have been too much work for them (M = 3.88, SD = 1.31) and there were too many steps to complete (M = 3.56, SD = 1.67). Table 4.5 Consumer Feasibility Results for the ESSS Feasibility Domaina Stakeholderb M (SD) d TAI-ASD 4.21 (1.17) 0.27 Desirability Caregiver 4.52 (1.10) TAI-ASD 4.75 (0.96) 0.15 Understanding Caregiver 4.88 (0.68) TAI-ASD 2.53 (1.53) 0.36 Collaboration Caregiver 3.09 (1.58) TAI-ASD 3.97 (1.35) 0.008 Applicability Caregiver 3.98 (1.07) Note. Bolded M(SD) indicates the feasibility domain did not achieve the benchmark. a Desirability, understanding, and applicability domain benchmarks are 4 or higher on a 6-point scale; collaboration domain benchmark is 3 or lower on a 6-point scale. bSample size for each stakeholder is as follows: TAI-ASD = 20, caregiver = 16. Feasibility – Service Providers For the service provider group (paraprofessionals and professionals), feasibility was measured across six domains: desirability, understanding, applicability, collaboration, system climate, and system support (see Table 4.6 for domain results and Appendix D for item-level results). Both paraprofessional and professional group means met the benchmark (4 on a 6-point scale) for the desirability domain (paraprofessional: M = 4.67, SD = 0.68; professional: M = 4.58, SD = 0.71; d = 0.13), suggesting that on average both groups agreed the ESSS is an effective and fair way to assess soft skill development; it would not disrupt the students/consumers they work with from engaging in other employment-related activities; the assessment procedures easily fit in with their current practices; and they are interested in using 64 the ESSS and would do so with enthusiasm and positive attitudes. Both service provider group means also met the benchmark for the understanding domain (paraprofessional: M = 4.86, SD = 0.61; professional: M = 4.86, SD = 0.62; d = < 0.001), suggesting that on average they understand how to use the ESSS; they would be able to implement the assessment; the procedures (including scoring) are not too complicated; and the items are understandable and easy to read. Both service provider group means also met the benchmark for the applicability domain (paraprofessional: M = 4.95, SD = 0.42; professional: M = 4.84, SD = 0.75; d = 0.18), suggesting that on average both groups agreed that the ESSS is not too complex to carry out accurately; material resources are reasonable with minimal preparation needed; the time required for implementing and record keeping is manageable; and they would be able to allocate time to complete the assessment with students/consumers. Regarding the collaboration domain, both service provider group means did not meet the benchmark of 3 or lower (paraprofessional: M = 4.19, SD = 1.53; professional: M = 4.05, SD = 1.22; d = 0.10). Further evaluation at the item level suggested that both groups on average agreed that collaboration (paraprofessional: M = 3.90, SD = 1.73; professional: M = 4.27, SD = 1.24) and regular communication with other professionals and caregivers (paraprofessional: M = 4.48, SD = 1.57; professional: M = 3.82, SD = 1.47) would be required to effectively use the ESSS. System climate and system support were two additional feasibility domains specific to service provider perspectives. Both paraprofessional and professional group means met the benchmark (4 on a 6-point scale) for the system climate domain (paraprofessional: M = 4.69, SD = 0.99; professional: M = 4.70, SD = 0.70; d = 0.01), suggesting that on average both groups agreed that using the ESSS would align with the mission of their work setting; their work environment is conducive to implementing such an assessment; their administrator/supervisor 65 would be supportive of using the ESSS; and the procedures would be consistent with how assessments are implemented in their school/company. Both service provider group means also met the benchmark (3 or lower on a 6-point scale) for the system support domain (paraprofessional: M = 2.55, SD = 1.30; professional: M = 2.39, SD = 1.01; d = 0.14), suggesting that on average neither group would need additional professional development, consultation, or resources to implement the ESSS, and they also would not need additional consultative support to score the assessment. Table 4.6 Service Provider Feasibility Results for the ESSS Feasibility Domaina Stakeholderb M (SD) d Paraprofessional 4.67 (0.68) 0.13 Desirability Professional 4.58 (0.71) Paraprofessional 4.86 (0.61) < .001 Understanding Professional 4.86 (0.62) Paraprofessional 4.95 (0.42) 0.18 Applicability Professional 4.84 (0.75) Paraprofessional 4.19 (1.53) 0.10 Collaboration Professional 4.05 (1.22) Paraprofessional 4.69 (0.99) 0.01 System Climate Professional 4.70 (0.70) Paraprofessional 2.55 (1.30) 0.14 System Support Professional 2.39 (1.01) Note. Bolded M(SD) indicates the collaboration domain did not achieve the benchmark. a Desirability, understanding, applicability, and system climate benchmarks are 4 or higher on a 6-point scale; collaboration and system support domain benchmarks are 3 or lower on a 6-point scale. bSample size for each stakeholder is as follows: paraprofessional = 21, professional = 22. Similarities and Differences in Stakeholder Perspectives on Social Validity and Feasibility The following results align with research question #2 (what are the similarities and differences in social validity and feasibility of the ESSS across different types of stakeholders?). A series of multivariate analysis of variance models (MANOVA, main effects) were conducted to determine the degree to which stakeholder perspectives aligned and where they differed across social validity and feasibility domains (see Appendix E for detailed results). A parallel 66 qualitative content analysis was also conducted to further illuminate similarities and differences across stakeholder perspectives. Outlined below are the results of this inquiry organized by social validity outcomes, consumer and service provider feasibility outcomes, and qualitative responses. Social Validity A one-way MANOVA was conducted to evaluate whether there were significant differences in stakeholder perspectives (TAI-ASD, caregivers, service providers) regarding the social validity of the four structural components of the ESSS assessment (introduction section, rating of items section, scoring your answers section, and understanding your answers section). Stakeholder perspectives on the combined social validity dependent variables were not significantly different, F(8, 132) = 1.16, p = .33, partial 2 = .07, power = .52 (with  = .05). In other words, while mean differences exist across stakeholder perspectives (as reviewed previously), these differences are marginal and not statistically significant. Feasibility – Consumers A main effects MANOVA was conducted to evaluate whether there were significant differences in consumer perspectives (independent variable: TAI-ASD and caregivers) across four feasibility domains (dependent variables: desirability, understanding, collaboration, applicability). Geographic area (rural/suburban vs. urban) and school status (attending secondary/post-secondary education or vocational training vs. not attending) were significantly correlated with the applicability domain (Spearman’s r = -.36, p = 0.04 and Spearman’s r = -.43, p = 0.02, respectively) and thus were included as fixed factors in a three-way main effects MANOVA model. Multivariate tests indicated significant main effects for the stakeholder group, F(4, 24) = 4.59, p = .007, partial 2 = .43, power = .89 (with  = .05), and geographic area, F(4, 67 24) = 3.59, p = .02, partial 2 = .37, power = .80 (with  = .05), but not for school status F(4, 24) = .88, p = .49, partial 2 = .13, power = .24 (with  = .05). Thus, four univariate three-way main effects ANOVA models with Bonferroni corrections (p < .05  4 = .0125) were then conducted to determine which perspectives were significantly different across dependent variables. Univariate main effects ANOVA results indicated there were significant differences in the desirability domain of the ESSS between TAI-ASD and caregivers, F(1, 27) = 7.36, p = .011, partial 2 = .21, power = .74 (with  = .05). For example, TAI-ASD on average found the ESSS to be less desirable when compared to caregiver perspectives (M = 4.13, SD = 1.17 and M = 4.73, SD = .82, respectfully). In fact, TAI-ASD means across all seven items in the desirability domain were lower than caregiver means, with differences ranging between .21 and .54 with the greatest difference occurring for the item, “I liked taking this assessment.” For the other three feasibility domains (understanding, collaboration, applicability), there were no significant differences (p < .0125) between TAI-ASD and caregiver perspectives. Univariate main effects ANOVA results also indicated significant differences in the understanding domain between stakeholders living in rural/suburban and urban areas, F(1, 27) = 8.84, p = .006, partial 2 = .25, power = .82 (with  = .05). For example, stakeholders living in rural/suburban areas (vs. urban areas) on average had more difficulty understanding the purpose of the ESSS, how to complete the assessment, and how to use the assessment to understand their (child’s) strengths and where they might need more training or practice (rural/suburban grand M = 4.50, SD = .99 and urban grand M = 5.13, SD = .53, respectively). In fact, rural/suburban area means across all six items in the understanding domain were lower than urban area means but also had the greatest variance in responses (see Table 4.7); mean differences ranged between .32 and 1.13 with the greatest differences in understanding how to complete the assessment, and 68 how to use the assessment to understand perceived soft skill strengths and skills that may need more training or practice. However, both rural/suburban and urban groups meet the feasibility benchmark (4 on a 6-point scale), except for the item, “this assessment helped me understand my (child’s) strengths.” For the other three feasibility domains (desirability, collaboration, applicability), there were no significant differences (p < .0125) between stakeholders living in rural/suburban vs. urban areas. Table 4.7 Rural/Suburban and Urban Group Means for Consumer Feasibility: Understanding Domain Consumer Feasibility: Groupa M (SD) d Understanding Domain Item I understand why this assessment was picked to Rural/Suburban 4.35 (1.58) 0.43 help me/my child. Urban 4.93 (1.00) Rural/Suburban 4.47 (1.59) 0.52 It is clear what I had to do. Urban 5.14 (0.77) I was able to do every step of the assessment Rural/Suburban 4.82 (1.78) 0.22 with little or no help. Urban 5.14 (0.95) This assessment helped me understand my Rural/Suburban 3.94 (1.64) 0.86 (child’s) strengths. Urban 5.07 (0.73) This assessment helped me understand where Rural/Suburban 4.41 (1.54) 0.48 I/my child might need more training or practice. Urban 5.07 (1.14) Rural/Suburban 5.00 (1.32) 0.41 I was able to use this assessment correctly. Urban 5.43 (0.51) aSample size for each group is as follows: non-urban = 17, urban = 14. Feasibility – Service Providers A one-way MANOVA was conducted to evaluate whether there were significant differences in service provider perspectives (paraprofessionals and professionals) across six feasibility domains (desirability, understanding, collaboration, applicability, system climate, system support). Service provider perspectives on the combined feasibility dependent variables were not significantly different, F(6, 36) = .16, p = .99, partial 2 = .025, power = .08 (with  = .05). In other words, while mean differences exist across stakeholder perspectives (as reviewed previously), these differences are marginal and not statistically significant. 69 Qualitative Perspectives Stakeholders were offered an opportunity to provide feedback across four open-ended questions about how the ESSS assessment could be improved overall, and specifically how each of the structural components of the ESSS could be improved (instructions and rating of items section, scoring section, and understanding your answers sections). A total of 12 TAI-ASD, 8 caregivers, and 34 service providers provided qualitative responses to these questions (67.5% response rate). A content analysis generated three primary categories of responses: strengths, challenges, and suggested improvements for the ESSS. The following review discusses each of these categories in succession and highlights both individual and collective voices. Strengths. Although the open-ended questions specifically asked about improvements, eight service providers (SP) and one caregiver (C) offered comments about how much they liked the ESSS assessment and did not think it needed to be improved. For example, some of their specific comments stated: • “The assessment is spot on for evaluating an individual on the Autism Spectrum. This assessment could be coupled with a scoring assessment our company uses when doing situational assessments with individuals on job sites.” (SP) • “It is nice that the assessment is short and somewhat comprehensive.” (SP) • “I love how this is set up.” (SP) • “No improvement needed. Easy to follow, fill out and score.” (SP) • “This is a good assessment for soft skills.” (SP) • “This [is a] good assessment for our clients to learn more about them and job match.” (SP) 70 • “These skills are really important for employment, so I am glad to see you researching this!” (C) Seven service providers and one caregiver commented on the importance of gathering multiple perspectives on TAI-ASD soft skill development, indicating that the individual with autism themselves should take the ESSS along with others working closely with them (e.g., family members, teachers, job coaches, supervisors, etc.). While stakeholders did not indicate they were aware the ESSS already includes self-report and informant-report forms to gather different perspectives, their feedback positively reinforces the assessment development decisions thus far and the importance of offering these options to the community. Their perspectives are highlighted in the following comments: • “I think this is an assessment that could be used in conjunction with others; it is opinion based, so having multiple perspectives would also be important (the individuals, and others who support them).” (SP) • “The involvement of e.g., parents, family, significant others, caregivers, and/or support staff and on-the-job observation would be beneficial to accurately completing this assessment.” (SP) • “… I would love to see the supervisor or a job coach take the second column data, perhaps incorporate it into a job review - someone who is actually at the job site to give the data.” (C) Challenges. Self-scoring the ESSS was an overwhelming challenge that nine TAI-ASD, five caregivers, and 13 service providers discussed. Comments suggested difficulties with understanding how to self-score, how to correctly use the scoring crosstabulation table, and how to translate the scores into understandable results. While one individual suggested the color- 71 coding on the scoring crosstabulation table was helpful, comments generally suggested the process of self-scoring was complicated and confusing when having to combine numbers and letters, and that the letters generated by the scoring table were not intuitive. Numerous individuals suggested this section of the ESSS should be automated and/or would be easier to complete using a “paper and pencil form.” The following comments illuminate these perspectives: • “The scoring rubric was sometimes confusing.” (TAI-ASD) • “The operation is a little cumbersome.” (TAI-ASD) • “The biggest challenge that I had was the end, with computing the letter for a given score. I wonder if you could automate that part. I think I'm a smart person, and it took me a while to figure out what I was supposed to do haha. If you're able to automate the computation of it, I think that would help the assessment a lot-- the assessment would be easier to complete and take less time.” (PC) • “I would rather the table lookup was automatic, with a summary that told me that a set of skills was well developed, another set was able to be developed, etc.” (PC) • “Create an easier rating system, time consuming and letters and numbers can be easily missed or switched.” (SP) • “There should be a way you could program the assessment to automatically give the corresponding letter score for each question without the need for participants to do so. That letter score could then be transferred to the explanation section so that each answer could be interpreted by the young person or their support staff.” (SP) Additionally, five service providers and two parents expressed concerns about some TAI- ASD likely having difficulty scoring the ESSS independently, but that it may depend on their 72 reading level and cognitive abilities. For example, one caregiver said, “I think especially if you have autistic individuals with an ID [intellectual disability], they're not going to be able to complete this independently,” and a service provider said, “It may be slightly difficult for an autistic youth to score this on their own, depending on their reading skill level.” Three caregivers and two service providers expressed difficulty understanding the difference between measuring (a) their confidence that a specific TAI-ASD they were thinking about could do the soft skill without help or reminders, and (b) how frequently this same TAI- ASD performed the skill at work without help or reminders. For example, a caregiver said, “My confidence is usually based on my experience with them doing or not doing the thing. A better explanation of how they differ might be useful to tease apart differences there, if they exist.” A service provider furthermore reflected, “I would need a little help finding out how to answer the two columns differently. It seems like many of my answers on both sides were the same.” Lastly, two service providers commented on the possibility of reducing items that seemed redundant so the assessment could be simplified and more easily accessibility. One TAI-ASD commented on how they thought the ESSS was condescending and how they felt infantilized. Suggested Improvements for the ESSS. In addition to numerous suggestions about automating the self-scoring section, four stakeholders (one TAI-ASD, one caregiver, two service providers) commented about how some of the items on the ESSS were broad, abstract, and ambiguous. To provide more clarity and consistency of interpretation, a TAI-ASD and parent both suggested expanding each item to include examples of possible work situations to help the individual assess their soft skills more accurately. Two caregivers additionally suggested expanding the understanding your answers section to include weblinks to employment resources (e.g., workforce development, vocational rehabilitation, independent living centers, etc.) and 73 online and local soft skills training programs to help improve skills. Regarding accessibility, a TAI-ASD suggested making sure the ESSS can be used with text-to-speech functioning, and a caregiver suggested having mobile-friendly options. A TAI-ASD and a service provider suggested the ESSS should include additional items related to mental health, using work tools and materials, absenteeism, time off task, self-management, and self-awareness. Regarding the G-V-S-E labeling on the scoring table, a TAI-ASD and a service provider recommended using a labeling convention that is more intuitive (such as developing-confidence-skills-strengths) to minimize the possibility of misinterpretation and further confusion. A service provider additionally suggested the Likert anchors on the frequency scale adjust to almost never and almost always (instead of the current absolute condition of never and always). Lastly, a service provider recommended the confidence and frequency scales are answered separately (instead of simultaneously), indicating that “it was a bit too much going from left to right with each question.” Preliminary Reliability and Factor Structure of the ESSS Assessment The following results align with research question #3 (what is the preliminary reliability and factor structure of the ESSS?). Preliminary reliability is discussed first, followed by the preliminary factor structure of the ESSS assessment. Preliminary Reliability In this study, internal consistency (Cronbach’s alpha) is the primary reliability indicator and was evaluated for the confidence scale and frequency scale on the self-report (TAI-ASD) and informant-report (caregivers and service providers). Overall, the confidence scale demonstrated very good reliability for both the self-report ( = .881) and informant report (parent  = .855, service provider  = .916). Similarly, the frequency scale also demonstrated 74 very good reliability for both the self-report ( = .913) and informant report (parent  = .828, service provider  = .919). Appendix F shows individual ESSS item means by stakeholder group (TAI-ASD, caregiver, service provider), item-scale correlations, and instrument reliability if the item was deleted from the measure. For the TAI-ASD group on the confidence scale, the average item mean was 3.03 (SD = .47, range = 2.32-3.84); for the caregiver group, the average item mean was 2.51 (SD = .43, range = 2.00-3.13); for the service provider group, the average item mean was 2.52 (SD = .54, range = 1.86-3.12). Across each stakeholder group, the confidence scale item mean spread ranged between 1.12-1.52 points, indicating adequate item variance. Individual item means were generally close to the center of the range of possible scores, especially for the caregiver and service provider groups. However, 14 individual item means for the TAI-ASD group were above 3.0 (with 4 of these items exceeding means of 3.3), suggesting these items tended to gather closer to the extreme end of the agreement range. Regarding item-scale correlations, while no correlations were below 0.2 across both self- and informant-reports (caregiver and/or service provider), items 7, 10 and 18 were below this benchmark for the caregiver group, and item 1 was below this benchmark for the TAI-ASD group. With respect to instrument reliability upon individual item deletion, Cronbach alphas remained stable across items per stakeholder group. For the TAI-ASD group on the frequency scale, the average item mean was 2.88 (SD = .51, range 2.32-3.63); for the caregiver group, the average item mean was 2.53 (SD = .39, range 1.94-3.13); for the service provider group, the average item mean was 2.46 (SD = .50, range 1.90-3.02). Across each stakeholder group, the frequency scale item mean spread ranged between 1.12-1.32, indicating adequate item variance. Similar to the confidence scale, frequency scale item means were generally close to the center of the range of possible scores, with greater 75 variation within the TAI-ASD group. For example, 10 individual item means gathered closer to the extreme end of the agreement range above 3.0 (with 2 of these items exceeding means of 3.3). While no item-scale correlations were below 0.2 across both self- and informant-reports (caregiver and/or service provider), items 4, 5, 7 and 23 were below this benchmark for only the caregiver group. Regarding instrument reliability upon individual item deletion, Cronbach alphas remained stable across items per stakeholder group, except for item 6 for the caregiver group ( = .793, but remains within an adequate alpha range; DeVellis, 2017). Preliminary Factor Structure – Confidence Scale The scree plot displaying the factor eigenvalues from the exploratory factor analysis using principal axis factoring with direct oblimin rotation is shown in Figure 4.1. The plot suggests there may be two inflection points, one point at factor 2 and another point at factor 4, suggesting the possibility of retaining 1, 2, or 3 factors. A parallel analysis was conducted for further consideration of factor retention (Patil et al., 2007, 2017). Table 4.8 compares the eigenvalue results of the parallel analysis (eigenvalues extracted from randomly generated correlation matrices) with the real dataset eigenvalues. Patil and colleagues (2008) suggest “retaining as many factors as there are eigenvalues in the dataset that are greater than the corresponding eigenvalues provided by web-based parallel analysis” (p. 168). In this comparison, only the first factor in the real dataset exceeds the corresponding parallel analysis eigenvalue, thus suggesting the confidence scale of the ESSS may be a single-factor scale presumably measuring the global latent construct that represents soft skills. 76 Figure 4.1 Scree Plot for ESSS Confidence Scale Table 4.8 Comparison of Parallel Analysis and Real Dataset Eigenvalues for the Confidence Scalea Real Dataset Parallel Analysis Factor Eigenvalue % of Variance Cumulative % Eigenvalue 1 2.367 8.213 35.707 35.707 2 2.075 1.870 8.132 43.839 3 1.892 1.538 6.686 50.525 4 1.711 1.491 6.483 57.008 5 1.615 1.176 5.113 62.121 6 1.475 1.072 4.660 66.782 7 1.378 .909 3.951 70.733 8 1.294 .867 3.770 74.503 9 1.212 .799 3.473 77.976 10 1.123 .703 3.057 81.033 11 1.067 .636 2.767 83.800 12 .985 .553 2.406 86.206 13 .911 .541 2.354 88.560 14 .834 .490 2.132 90.692 15 .776 .366 1.591 92.283 16 .711 .349 1.518 93.802 17 .646 .298 1.294 95.095 77 Table 4.8 (cont’d) 18 .591 .257 1.116 96.212 19 .547 .228 .993 97.204 20 .493 .211 .918 98.122 21 .427 .160 .696 98.818 22 .378 .142 .617 99.436 23 .302 .130 .564 100 aInput for parallel analysis included 23 variables, n = 77, 100 randomly generated correlation matrices, and 95th percentile of random eigenvalues. In addition to performing a visual scree plot analysis and parallel analysis, Howard (2016) suggests further considering how items load on their primary and alternative factors using the .40-.30-.20 rule. This rule states that satisfactory variables have (a) primary factor loadings above 0.40, (b) alternative factor loadings below 0.30, and (c) a minimal difference of 0.20 between primary and alternative factor loadings. Applying this rule, an evaluation of 1-, 2-, and 3-factor solutions were conducted to further explore which model best describes the preliminary structure of the confidence scale on the ESSS. In the 1-factor solution, only one item fell below the 0.40 benchmark (item 4: “start a conversation and keep it going”); in the 2-factor solution, three items did not meet the .40-.30-.20 rule (items 1, 4, 14); in the 3-factor solution, six items did not meet the rule (items 4, 5, 7, 13, 14, 16). Furthermore, the interpretability of the 2- and 3- factor solutions are complicated by items with low factor loadings and/or cross-loadings. See Table 4.9 for the 1-factor item loadings and Appendix G for a comparison of 1-, 2-, and 3-factor item loadings. Based on the above collective considerations, it appears at this stage of scale development the confidence scale on the ESSS is best described as a single-factor soft skills scale. 78 Table 4.9 Confidence Scale Item Loadings for a 1-Factor Solutiona Item Item Factor # Loadings C9 Show respect for others. 0.778 C11 Respond to feedback with a positive attitude. 0.727 C19 Be in charge of my/their emotions. 0.689 C14 Shift my/their attention from one task to another when being asked. 0.664 C6 When someone is talking to me/them, I/they listen without 0.652 interrupting. C20 Use different ways to cope with stress (e.g., take a break, deep 0.649 breathing). C21 Take responsibility when I/they have made a mistake. 0.640 C15 Show flexibility when things change. 0.622 C2 Use an appropriate voice volume based on the location and situation. 0.605 C7 Use appropriate social manners (e.g., please and thank you). 0.587 C13 Plan my/their time to get tasks done. 0.584 C16 When I/they say they are going to do something, I/they follow through 0.582 and do it. C8 Work well with others. 0.567 C22 Show a positive attitude towards work. 0.542 C17 Notice when there is a problem or conflict. 0.541 C18 Fix a problem or conflict. 0.521 C12 Use different ways to speak up for myself/themselves depending on 0.490 the situation. C10 Are ok with people having different opinions. 0.488 C23 Show confidence in my/their skills and abilities. 0.448 C3 Use appropriate personal space based on the type of conversation. 0.445 C5 When talking with coworkers and supervisors, I/they do not share 0.429 things that are too personal. C1 Have good hygiene (e.g., bathe & brush teeth to get ready for work). 0.410 C4 Start a conversation and keep it going. 0.303 a4 iterations required. To note, while item 4 appears problematic across all 1-, 2-, and 3-factor solutions, at this stage of scale development, it will be retained in the instrument for face validity reasons. For example, during the first phase of this multiphase study, subject matter experts suggested and collectively consented on this item. Furthermore, during the current second phase, all stakeholder groups agreed this item was a relevant soft skill (TAI-ASD: M = 3.40, SD = .74; caregiver: 79 M = 3.60, SD = .51; service provider: M = 3.56, SD = .50). However, the item represents a double-barrel construction and may benefit from further narrative adjustments (e.g., splitting the item into two items) prior to proceeding with the next phase of instrument development (scale evaluation). Preliminary Factor Structure – Frequency Scale A parallel exploratory factor analysis using principal axis factoring with direct oblimin rotation was conducted for the frequency scale. The following describes the results of this analysis that mirrors the same steps and considerations from the confidence scale. The scree plot displaying the factor eigenvalues is shown in Figure 4.2. Unlike the confidence scale, the frequency scale plot suggests there is only one inflection point (at factor 2), suggesting the possibility of retaining only 1 factor. A parallel analysis was conducted for further consideration of factor retention (Patil et al., 2007, 2017). Table 4.10 compares the eigenvalue results of the parallel analysis with the real dataset eigenvalues. In this comparison, only the first factor in the real dataset exceeds the corresponding parallel analysis eigenvalue, thus suggesting the frequency scale of the ESSS may also be a single-factor scale (similar to the confidence scale) and presumably measures the same global latent construct that represents soft skills. 80 Figure 4.2 Scree Plot for ESSS Frequency Scale Table 4.10 Comparison of Parallel Analysis and Real Dataset Eigenvalues for the Frequency Scalea Real Dataset Parallel Analysis Factor Eigenvalue % of Variance Cumulative % Eigenvalue 1 2.345 8.102 35.226 35.226 2 2.108 1.914 8.322 43.548 3 1.912 1.492 6.485 50.032 4 1.761 1.384 6.015 56.048 5 1.633 1.232 5.356 61.404 6 1.496 1.073 4.664 66.068 7 1.423 0.929 4.039 70.108 8 1.301 0.855 3.717 73.824 9 1.200 0.811 3.526 77.351 10 1.120 0.794 3.452 80.803 11 1.033 0.698 3.035 83.838 12 0.963 0.592 2.572 86.410 13 0.904 0.526 2.287 88.696 14 0.837 0.503 2.187 90.883 15 0.776 0.368 1.600 92.483 16 0.716 0.314 1.367 93.850 17 0.669 0.305 1.325 95.175 18 0.596 0.248 1.080 96.255 19 0.535 0.217 0.944 97.199 81 Table 4.10 (cont’d) 20 0.483 0.191 0.829 98.028 21 0.425 0.172 0.750 98.778 22 0.368 0.159 0.691 99.469 23 0.297 0.122 0.531 100 aInput for parallel analysis included 23 variables, n = 76, 100 randomly generated correlation matrices, and 95th percentile of random eigenvalues. As both the scree plot and parallel analysis suggested the frequency scale is composed of one factor, further consideration of multiple factors using the .40-.30-.20 rule was not pursued. In the 1-factor solution, two items fell below the 0.40 benchmark (item 17: “notice when there is a problem or conflict,” and item 4: “start a conversation and keep it going”; see Table 4.11 for factor loadings). In terms of item reduction at this stage of scale development, both items will be retained in the instrument for face validity reasons. For example, during the first phase of this multiphase study, subject matter experts suggested and collectively consented on both items. Furthermore, during the current second phase, all stakeholder groups agreed both items were relevant soft skills (item 17: TAI-ASD, M = 3.40, SD = .82; caregiver: M = 3.67, SD = .49; service provider: M = 3.48, SD = .51; item 4: TAI-ASD, M = 3.40, SD = .75; caregiver: M = 3.60, SD = .51; service provider: M = 3.56, SD = .50). Table 4.11 Frequency Scale Item Loadings for a 1-Factor Solutiona Item Item Factor # Loadings F11 Respond to feedback with a positive attitude. 0.729 F9 Show respect for others. 0.686 F19 Be in charge of my/their emotions. 0.675 F3 Use appropriate personal space based on the type of conversation. 0.663 F20 Use different ways to cope with stress (e.g., take a break, deep 0.651 breathing). F14 Shift my/their attention from one task to another when being asked. 0.641 F21 Take responsibility when I/they have made a mistake. 0.635 F6 When someone is talking to me/them, I/they listen without 0.628 interrupting. F2 Use an appropriate voice volume based on the location and situation. 0.603 82 Table 4.11 (cont’d) F22 Show a positive attitude towards work. 0.581 F16 When I/they say they are going to do something, I/they follow 0.576 through and do it. F8 Work well with others. 0.571 F13 Plan my/their time to get tasks done. 0.562 F7 Use appropriate social manners (e.g., please and thank you). 0.559 F15 Show flexibility when things change. 0.542 F1 Have good hygiene (e.g., bathe & brush teeth to get ready for work). 0.535 F10 Are ok with people having different opinions. 0.524 F18 Fix a problem or conflict. 0.480 F12 Use different ways to speak up for myself/themselves depending on 0.478 the situation. F23 Show confidence in my/their skills and abilities. 0.436 F5 When talking with coworkers and supervisors, I/they do not share 0.429 things that are too personal. F17 Notice when there is a problem or conflict. 0.389 F4 Start a conversation and keep it going. 0.326 a4 iterations required. 83 CHAPTER 5: DISCUSSION The following topics are discussed in this final chapter: an overview of the current study; a discussion about the social validity and feasibility of the Employment Success: Soft Skills (ESSS) assessment across stakeholder perspectives; a review of the preliminary reliability and factor structure of the ESSS; stakeholder-informed improvements to the ESSS; limitations of the current study; suggestions for future research; and implications for research and practice. This study represents the second phase of a three-phase, multistep scale development process for measuring employability skills (Boateng et al., 2018). The first phase focused on item development (Strain & Sung, 2019) and scale creation that aligns with social cognitive career theory’s performance model framework (Lent et al., 1994). This two-step process generated 72 essential skills for employment success (23 soft skills, 25 employment preparation skills, and 24 employment retention skills) that were subsequently translated into self-report and informant-report forms across three respective assessments: Employment Success: Soft Skills (ESSS), Employment Success: Preparation Skills (ESPS), and Employment Success: Retention Skills (ESRS). Building on these previous steps, the current study used a mixed-methods, participatory- informed research approach to specifically advance the Employment Success: Soft Skills (ESSS) assessment for transition-age individuals with autism spectrum disorders (TAI-ASD). This second phase of scale development incorporated four steps: testing scale items and assessment administration with key stakeholders to obtain their feedback about the feasibility and social validity of the assessment (i.e., clinical utility); evaluating possible item reduction using inter- item and item-total correlations; exploring latent constructs through factor analysis; and evaluating the preliminary internal consistency (reliability) of the ESSS. The following 84 successively discusses each of these four topics as well as stakeholder-informed improvements to the ESSS, study limitations, and research and practice implications. Social Validity and Feasibility of the ESSS Assessment Social Validity In general, the ESSS assessment appears to be a socially valid instrument but will require a few improvements to further strengthen the social validity. For example, out of 11 social validity domains evaluated across the four sections of the ESSS assessment, only three domains did not meet the benchmark but were < 0.2 below the benchmark (e.g., introduction section understandability and clarity, and overall ease of scoring). Furthermore, all stakeholders generally agreed the 23 items on the assessment were understandable, were written clearly, and are all considered relevant soft skills. These results further validate prior studies’ evaluation of soft skills necessary for employment success (Agran et al., 2016; Ju et al., 2012, 2014; Liu et al., 2014; Robles, 2012; Sarfraz et al., 2018). Stakeholders also generally agreed the ESSS helped them identify perceived soft skill strengths and areas for improvement. While mean differences between stakeholder perspectives were not significantly different, it is clear from the quantitative and qualitative results that (a) the instructions in the introduction section of the ESSS will need to be modified for clarity, (b) some items on the ESSS are too abstract and/or vague and will also need to be reworded for clarity, and (c) simplifying and streamlining the scoring process is necessary to improve the ease of scoring and interpreting the results for future users of the ESSS assessment. An interesting finding worth noting are differences in perspectives about how easy it would be in general for autistic youth to score the assessment on their own. This specific item did not meet the social validity benchmark across all stakeholder groups and was the lowest rated 85 item across all social validity items. Although average TAI-ASD ratings for this item were 0.35 below the benchmark with caregiver responses closer to TAI-ASD responses (0.08 difference), service providers in general rated this item the lowest (0.42 difference from TAI-ASD responses). While this study did not ask caregivers or service providers to describe detailed characteristics of their child or the autism communities they serve and did not collect paired responses between stakeholders, further exploration into these notable differences in perspectives is needed. For example, negative impacts on employment outcomes when family members and service providers carry low expectations and perceptions of TAI-ASD capabilities and abilities is well documented in the literature (Hagner et al., 2012; Kuo et al., 2018; Webster & Garvis, 2017). Furthermore, from an historical perspective often framed around deficit-based models (Shogren & Raley, 2022), informants may have an unconscious bias towards minimizing TAI- ASD’s self-awareness and ability to validly report their skills, abilities, capabilities, and preferences (Niles & Harkins Monaco, 2019). While it is unknown whether unconscious bias may have influenced informants in this study, current research has reinforced that individuals with ASD have insights to and awareness of their skills, abilities, capabilities, and preferences (Hume et al., 2018; Webster & Garvis, 2017). This study further supports this notion. For example, when rating the 23 items as relevant soft skills, TAI-ASD ratings were quite similar to caregiver and service provider ratings. TAI-ASD also demonstrated insights between their confidence in performing a soft skill and the frequency that they perform the skill; the individuals in this study on average rated themselves higher on the confidence scale and lower on the frequency scale. Lastly, caregivers made the most errors when self-scoring their ESSS results, which suggests they had more difficulty scoring the assessment on their own than the TAI-ASD group. 86 Feasibility In general, the ESSS assessment appears to be a feasible instrument across a variety of settings and a variety of users but will require additional adjustments to further strengthen the overall feasibility. For example, consumers and service providers across a variety of academic and community settings agreed the ESSS assessment is a desirable instrument and a fair way to evaluate soft skill development; they understood how to independently use the assessment to identify perceived soft skills strengths and areas for improvements; and they would volunteer to complete the assessment again. Service providers also agreed they could feasibly implement the ESSS assessment in their respective settings with little difficulty; the ESSS aligns with the mission of their work settings; the systems in which they work are conducive to and would be supportive of them using the ESSS; and they would be able to carry out the assessment without needing additional resources, professional development, or consultative supports. Although stakeholders expressed overall positive feedback about the usability and generalizability of the ESSS assessment, three domains intimated that additional adjustments may be necessary to further strengthen the overall feasibility for different stakeholders. First, while consumer ratings in the applicability domain were marginally under the benchmark ( 0.03), quantitative and qualitative results suggested the ESSS was somewhat taxing for consumers; the self-scoring process definitively caused the greatest challenge and some consumers expressed difficulty with how to interpret their results. Second, although the desirability domain for consumers met the overall feasibility benchmark, the TAI-ASD group ratings in this domain were significantly different than the caregiver group ratings, suggesting the ESSS was less desirable for the TAI-ASD group. Third, although the understanding domain for consumers met the overall feasibility benchmark, consumers living in rural/suburban 87 communities had significantly different ratings than consumers living in urban communities, suggesting consumers living in rural/suburban communities had more difficulty understanding key features of the assessment. Conceivably, to further strengthen the applicability, desirability and understanding of the ESSS for all potential users of the assessment, simplifying and streamlining the scoring process and providing more context and clarity for interpreting results will be necessary. Higher ratings on the collaboration domain were noted for caregivers and service providers, suggesting that collaboration and communication with others is necessary to effectively use the ESSS assessment; this is not a surprising finding. In fact, this finding further supports the call for improved interagency and multidisciplinary collaborations across research and practice settings (Cumming et al., 2020; Kuo et al., 2018). For example, caregivers and service providers maintain supportive roles in the lives of TAI-ASD across different contexts (home, school, community); when considering an individual’s soft skill development, it is important to incorporate different perspectives and especially TAI-ASD (Hume et al., 2018). Numerous caregivers and service providers in this study offered a variety of comments further validating this importance, and although their comments do not suggest they were aware the ESSS includes both self-report and informant-report versions, the two versions were designed to specifically offer opportunities for multidisciplinary collaborations that incorporate multiple perspectives for data-driven decision-making. The feedback from stakeholders is also consistent with transition planning frameworks that strongly recommend the coordinated collection and use of assessment data for decision-making across several key transition practices, including self- directed and individualized services and supports, interagency and multidisciplinary collaboration, and family engagement (Kohler et al., 2016). 88 Preliminary Reliability and Factor Structure of the ESSS Assessment Overall, preliminary reliability (internal consistency) was very good across stakeholder groups for both the confidence scale and the frequency scale of the ESSS. Conceivably, the combined approaches of applying a Delphi method with subject matter experts during the first phase of item development, and participatory action research approaches during the second phase of scale development provides strong supporting evidence for the effectiveness of these strategies when designing and developing new instruments (Boateng et al., 2018). Arguably, these intentional approaches that incorporate multidisciplinary perspectives impacted the high reliability and content validity of the ESSS. In terms of item reduction considerations, evidence is inconclusive. First, while two items on the self-report confidence scale suggested possible multicollinearity, there were no indications of multicollinearity on the informant-report confidence scale for either caregivers or service providers. Second, item-scale correlations were generally above .30, but there were some inconsistencies across stakeholders. Third, no item if deleted from either the confidence scale or frequency scale across stakeholders significantly impacted the overall reliability below an acceptable Cronbach’s alpha (0.7). Fourth, although there were two items that fell below 0.4 on the exploratory factor analysis, both items were retained for face validity and one of the items needs additional evaluation due to the double-barreled construction. Lastly, some stakeholders suggested item reduction to further simplify the ESSS; while reducing items would simplify the ESSS, it comes at a risk of losing valuable information about specific soft skill development that may be beneficial to understand in both research and practice settings. The exploratory factor analyses suggested both the confidence and frequency scales may be single-factor scales. However, caution is warranted with these hypothesized interpretations at 89 this stage of scale development. For example, in the ESSS’s current construction, only approximately 35% of the variance is explained by this one factor model for each of the confidence and frequency scales. The sample size in this study also does not meet minimum standards of practice to determine a confident factor analytic solution (Beavers et al., 2013; DeVellis, 2017). All this to say, while the combined aforementioned preliminary evidence offers a baseline for the ESSS assessment in its current construction and are promising results, further studies with larger sample sizes will be necessary to more rigorously determine whether item reduction is warranted and to thoroughly evaluate additional psychometric properties of the ESSS self-report and informant-report forms. Stakeholder-Informed Improvements to the ESSS Assessment To answer the fourth research question in this study (What updates are necessary to improve the ESSS assessment?), quantitative and qualitative responses from stakeholders were merged to inform which aspects of the ESSS need improving. The following outlines the aggregated stakeholder-informed adjustments that will be applied before proceeding with the third phase of instrument development (scale evaluation). • In response to comments about some items being too abstract and/or vague, items will be expanded to include short examples (e.g., …) that provide further context for the item. The intention of this adjustment is to provide the end-user with a consistent framework for interpreting each item and to offer a more concrete understanding of the specific soft skill. To note, however, this adjustment may increase the overall reading level of the ESSS, which may reduce the accessibility of the assessment for some users to complete the ESSS independently. While there are benefits and limitations to adding short examples to each ESSS item, thoughtfully balancing the benefits and limitations will be 90 necessary to ensure the ESSS remains accessible to a wide range of users while also ensuring users have context to accurately interpret and respond to each item. • Item #4 (start a conversation and keep it going) on both confidence and frequency scales is double-barreled. This item will be broken into two items so only one aspect of the soft skill is being measured within each item (e.g., start a conversation with others at work, and maintain a conversation at work; for example, you know when to keep the conversation going and how to end the conversation politely). • In response to caregiver and service provider comments about differentiating the differences between the confidence scale and the frequency scale, additional descriptions will be provided. For example, the explanation may say: “Use the confidence scale to rate what you believe your child / the individual can do. Use the frequency scale to rate how often your child / the individual demonstrates the skill while at work.” • In response to a comment about how the end-user completes the confidence and frequency scales (e.g., simultaneously vs. successively), in the web-based version of the ESSS the scales will be split so users complete the confidence scale first followed by the frequency scale. This adjustment may help users better differentiate the difference between the two scales as completing the two scales simultaneously may be more taxing to shift back and forth between perspectives. • In response to a comment about the anchors on the frequency scale being absolute (never, always), these will be expanded to never/almost never and always/almost always. This adjustment may further encourage variability in responses and enhance measurement sensitivity. 91 • In response to comments about confusion related to the G-V-S-E lettering on the scoring table, these labels will be replaced with relevant descriptors that are easier to interpret. For example, G-V-S-E may be replaced with Developing, Confidence, Practice, and Strengths. Developing refers to skills that are in-development and may benefit from both confidence-building and extra practice; confidence refers to skills that are well-practiced but may benefit from additional confidence-building; practice refers to skills that individuals have confidence in doing but may benefit from extra practice; and strengths refers to skills that are self-perceived strengths where the individual is confident they can do the skill and frequently performs the skill on their own without help or reminders from others. • In response to stakeholder comments about the laborious and sometimes confusing process of self-scoring and generating results for interpretation, the web-based version of the ESSS will be developed on a platform that automates scoring. The intention is for the user to input their scale responses which will then be processed using an algorithm to generate an output of skills organized by those that are in-development, skills that may benefit from confidence-building, skills that may benefit from additional direct practice, and skills that are self-perceived strengths. • In response to comments about accessibility, future web-based versions of the ESSS will be developed on platforms that have text-to-speech capabilities and are mobile-friendly for smart phones, tablets, etc. • Multiple parents specifically recommended the ESSS link individual results to training programs and community resources that may further support soft skill development. To best support individuals, families, and service providers, links to training programs 92 should incorporate programs that use different modalities of training (synchronous and asynchronous including in-person, hyflex, and online) as well as links to local, regional, national, and global resources. All this to say, these aspirational steps are necessary but may require multiple iterative steps over time with guidance and support from various transition-focused communities of practice. • Some stakeholders offered comments about adding skills to the ESSS (e.g., mental health, using work tools and materials, absenteeism, time off task, self-management, and self-awareness). Some of these skills (using work tools and materials, absenteeism, time off task, and self-management) are already represented in the other Employment Success collection of assessments, namely the Employment Success: Preparation Skills and Employment Success: Retention Skills. The other suggested skills (mental health and self- awareness) may require further inquiry to determine where and how best to incorporate these into the Employment Success collection of assessments. Limitations This study has limitations that may impact a complete interpretation and application of the ESSS assessment. First, under-powered sample sizes may have impacted the ability to detect subtle but meaningful differences in stakeholder perspectives about the clinical utility of the ESSS. This limitation was considered a priori; to plan for this potentiality, open-ended questions were strategically placed throughout the social validity and feasibility surveys to further evaluate qualitative differences that may be difficult to capture quantitatively. However, it is possible the qualitative responses did not reach saturation as there were individual perspectives not able to be captured in this study (explained further below). The under-powered sample size also warrants cautious interpretation of the factor structure of the ESSS due to possible differences in 93 stakeholder perspectives that could potentially meaningfully impact the factor structure. Future studies should conduct exploratory factor analysis and confirmatory factor analysis with larger sample sizes for each of the self-report and informant-report versions of the ESSS for a more accurate and stable interpretation of the factor structure of the assessment (Beavers et al., 2013). Second, diverse racial, ethnic, gender, and heterogenous autistic perspectives were underrepresented in this study. While participant demographics in this study resemble similar demographics across social service and educator workforces (predominately white females; U.S. Bureau of Labor Statistics, https://www.bls.gov/cps/cpsaat11.htm), individuals with autism (predominately white males who are their own legal guardians; Aylward et al., 2021; Barnard- Brak et al., 2019), and parents/caregivers who participate in research (predominately white mothers/females; Braunstein et al., 2013), the results of this study cannot be broadly applied across diverse communities of individuals with multiple marginalized identities. This study specifically selected recruitment strategies that would maximize outreach to various communities across the U.S.; however, it is possible that ecological systems may have impacted whether individuals with multiple marginalized identities felt comfortable and safe participating in research (Levine & Breshears, 2019). Voices of individuals with multiple marginalized identities have a right to be heard, elevated, and included in the development and evaluation of the ESSS; while a smaller proportion of diverse voices were included in this study, future ESSS studies should use weighted sampling strategies (Cohen et al., 2011) for more representative samples that incorporate proportionally higher percentages of multiple marginalized identities. Third, when conducting a comparative analysis between self- and informant-report forms, samples should ideally include paired dyads of TAI-ASD and informants (parents/caregivers, service providers) so detailed comparisons between the two forms can be evaluated. This type of 94 comparative analysis with paired dyads was beyond the scope of this study. Future studies conducting scale evaluations of the ESSS should include paired dyads for a more thorough understanding of the consistency and stability of the self-report and informant-report forms. Fourth, this study evaluated the clinical utility of an electronic version of the ESSS. While hard copies of the ESSS were available if stakeholders did not have access to a computer/tablet and the internet, no stakeholders made requests to complete a hard copy version of the study. It is possible end-users of the ESSS may have different experiences depending on whether they engage with a hard copy version or the electronic version. Care was taken when transferring the hard copy ESSS into an electronic version to align them as similarly as possible, however there were structural constraints to the online software program that limited an exact replica of the hard copy version in a digital space. Thus, in addition to implementing the suggested improvements to the ESSS affiliated with the electronic version, future studies could also evaluate end-user experiences of the hard copy version to determine what (if any) improvements could be applied. Lastly, this study focused on evaluating the clinical utility of the ESSS from perspectives within the United States, therefore applicability and relevancy of the ESSS across global communities is unknown at this time. For example, it is possible that different cultures value different soft skills, or different cultures may have different operational definitions for the same soft skill (e.g., show respect for others). Expanding the ESSS to global communities would first require validating the generalizability of the ESSS from both content and process perspectives; additional studies could then conduct scale evaluations to determine the psychometric structure and how it compares to the U.S. version of the ESSS. 95 Implications for Research and Practice Improving overall employment outcomes for TAI-ASD communities requires ongoing collaborations between TAI-ASD, their families, educators, service providers, employers, and researchers; the key to these collaborations is centering TAI-ASD interests, goals, priorities, abilities, capabilities, resources, and needs (Kosciulek, 2004). Sometimes, TAI-ASD may not have considered how these areas influence their employment decisions and/or they may have an incomplete picture of these influences. In times like these, it is helpful to utilize other sources of knowledge within the TAI-ASD’s sphere of influence that may help clarify these areas: for example, nuclear and extended family members, friends, educators, service providers, mentors, religious/spiritual leaders, community elders, etc. As Kohler and colleagues (2016) further emphasize, gathering data from various sources of influence across time can provide important context for TAI-ASD-driven decision-making. Part of the array of data to consider includes understanding current employment skill development (e.g., hard skills and soft skills). It is necessary that TAI-ASD understand their employment skill strengths and skills that are continuing to develop so they can make informed decisions about how to leverage their strengths to scaffold continued development through targeted interventions (Lee et al., 2018); standardized employment skills assessments can bridge this understanding but very few assessments validated with autistic communities currently exist. Thus, as accentuated in the research and practice literature, there is an immediate need for employment skills assessments that are theory-aligned, psychometrically sound, socially acceptable, and clinically applicable. The Employment Success: Soft Skills assessment is one step closer to answering this call to action with the following clinical and research implications. 96 The results of this study suggest the ESSS is socially acceptable across different stakeholders and meets clinical utility standards in terms of its overall relevancy, feasibility, and practicality. Of the existing employability assessments for autistic communities, the ESSS appears to be the first assessment that has demonstrated social and clinical utility. However, automated scoring and results generation will be necessary to maintain the feasibility and social acceptability of the ESSS. Transition-related decisions should never be conducted based on the ESSS alone, but as some stakeholders reinforced, the ESSS can be used in conjunction with other data gathering resources, assessments, and informants to offer TAI-ASD a holistic understanding of their current soft skills skillset. Furthermore, the availability of self-report and informant-report versions of the ESSS align with various recommendations in the literature that specify the importance of collecting perspectives from multiple informants when assessing skill development (Hume et al., 2018; Hillier et al., 2011). To ensure the accessibility of the ESSS across heterogenous autistic communities, people within the TAI-ASD’s sphere of influence (caregivers, educators, service providers, and others) may wish to be available to support TAI- ASD who can benefit from the ESSS being read aloud to them or to offer additional content or process guidance. While the electronic version of the ESSS will have a text-to-speech function, the hard copy version should also be compatible with a variety of assistive technologies that have text-to-speech capabilities. Across clinical, academic, and research settings, the ESSS may be used for multiple purposes. For example, it may be used as a counseling and guidance tool to help TAI-ASD identify their current self-perceived soft skill strengths, and to guide the exploration of interventions that can scaffold skill development that may benefit from additional confidence- building and/or direct practice. Informant-report versions may also be used in conjunction with 97 self-report versions to compare similarities and differences in perceived soft skill strengths. However, it is extremely important that the self-report and informant-report results are never used pejoratively, but instead should be an opportunity to explore the personal, contextual, and environmental factors that may have influenced the respective results (Lent et al., 2000). The ESSS may also facilitate academic and vocational transition program planning to measure TAI- ASD skill development outcomes as well as evaluate overall programming successes and content gaps that may be missing from programming. For example, the ESSS may be completed at multiple timepoints throughout programming (e.g., before/during/after work-based learning experiences, before/after an employment readiness training intervention, etc.). Results can be viewed at a single point in time or across time, and at the individual level or aggregated across multiple TAI-ASD to understand programmatic trends. Lastly, the results of this study suggest the ESSS has potential to be expanded into multiple versions that may accommodate different reading levels and may be applicable across different populations of individuals with and without disabilities. With all that said, while this study suggests promising results for the applicability of the ESSS across clinical, academic, and research settings, careful consideration should be taken when using the ESSS as additional scale evaluation is necessary for a more complete understanding of the psychometric properties of the assessment. Furthermore, as the ESSS has yet to be translated into different versions, careful consideration should also be taken when deciding whether to use the ESSS with non-autistic populations and with individuals whose reading level is below 6th grade. From a research perspective, this study appears to be the first to apply career theory to soft skill scale development and to consider the social validity of the assessment with various stakeholders who identify as and serve within autism communities. Furthermore, this study 98 represents the first assessment tool to directly measure key domains of the Social Cognitive Career Theory’s performance model. While these efforts hold merit, attempting to build a tool that assists in bridging research and practice fields requires many steps. As Boateng and colleagues (2018, p. 2) highlight, “Scale development is not, however, an obvious or a straightforward endeavor.” This study represents steps 3-6 in the second phase of a 3-phase, 9-step scale development process; there are still three remaining steps in the last phase before a complete understanding of the ESSS can be ascertained from a psychometric perspective. The combined results of (a) testing ESSS items, (b) administering with projected end-users, (c) evaluating potential item reduction, and (d) surveying preliminary factor extraction and reliability show promising results at this stage of scale development. These aggregated results suggest that proceeding to the next scale evaluation phase—testing scale dimensionality, reliability, and psychometric validity (criterion and construct validity, such as predictive, convergent, and discriminant validity)—is reasonable after implementing the stakeholder- informed improvements to the ESSS. In the meantime, however, researchers may wish to cautiously use the ESSS assessment when measuring soft skills in research studies as the psychometric structure of the assessment has only been hypothesized and requires further testing (e.g., the ESSS could be used as a supplementary assessment in a series of other assessments). Lastly, in addition to moving forward with scale evaluations with the ESSS, future research is also needed to further explore the clinical utility and psychometric structures of the Employment Success: Preparation Skills and Employment Success: Retention Skills assessments. Together with the Employment Success: Soft Skills assessment, the collection of Employment Success assessments may offer tools for understanding and measuring perceived employment strengths 99 and skills that could benefit from additional confidence-building and direct practice across hard and soft skill domains. Closing Remarks This dissertation had three overarching objectives: (1) to fill a research-practice gap in employability assessments that promote multidisciplinary collaborations and data-driven decision making across systems that serve TAI-ASD; (2) to further scaffold previous scale development efforts with the end-goal of creating a practical soft skills assessment that can smoothly translate across multiple settings and is effective and useful for multiple users; and (3) to develop a quantitative tool for measuring key constructs related to Social Cognitive Career Theory’s performance model. While these pragmatic, aspirational, and theoretical objectives will continue to guide future developments of the Employment Success: Soft Skills assessment, this dissertation represents a stepping-stone towards achieving these goals. Unequivocally, these aspirations could not be possible without the guidance, feedback, and thoughtful suggestions shared by key stakeholders – transition-age individuals with autism, parents and caregivers, educators, and service providers; their voluntary contributions to the development of the ESSS assessment are paramount. While there is still much work to do to improve employment opportunities, experiences, and outcomes for autistic communities, it is the hope that the Employment Success assessments may offer the community useful tools to further scaffold these efforts. 100 REFERENCES Agran, M., Hughes, C., Thomas, C. A., & Scott, L. A. (2016). Employment social skills: what skills are really valued? Career Development and Transition for Exceptional Individuals, 39, 111–120. https://doi.org/10.1177/2165143414546741. Alverson, C. Y., & Yamamoto, S. H. (2017). Employment outcomes of vocational rehabilitation clients with autism spectrum disorders. Career Development and Transition for Exceptional Individuals, 40(3), 144-155. https://doi.org/10.1177/2165143416629366 Anderson, A., Moore, D. W., Rausa, V. C., Finkelstein, S., Pearl, S., & Stevenson, M. (2017). A Systematic Review of Interventions for Adults with Autism Spectrum Disorder to Promote Employment. Review Journal of Autism and Developmental Disorders, 4, 26–38. Aylward, B. S., Gal-Szabo, D. E., & Taraman, S. (2021). Racial, ethnic, and sociodemographic disparities in diagnosis of children with autism spectrum disorder. Journal of Developmental and Behavioral Pediatrics, 42(8), 682-689. https://doi.org/10.1097/DBP.0000000000000996 Balcazar, F. E., Keys, C. B., Kaplan, D. L., & Suarez-Balcazar. Y. (1998). Participatory action research and people with disabilities: Principles and challenges. Canadian Journal of Rehabilitation, 12(2), 105–12. Bandura, A. (1986a). Social foundations of thought and action: A social cognitive theory. Prentice-Hall. Bandura, A. (1986b). The explanatory and predictive scope of self-efficacy theory. Journal of Clinical and Social Psychology, 4(3), 359-373. https://doi.org/10.1521/jscp.1986.4.3.359 Bandura, A. (1999). Self-efficacy: Toward a unifying theory of behavioral change. In R. F. Baumeister (Ed.), The self in social psychology. Key readings in social psychology (pp. 285-298). Psychology Press/Taylor & Francis. Barnard-Brak, L., Richman, D., & Almekdash, M. H. (2019). How many girls are we missing in ASD? An examination from a clinic- and community-based sample. Advances in Autism, 5(3), 214-224. https://doi.org/10.1108/AIA-11-2018-0048 Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research, and Evaluation, 18(6), 1-13. https://doi.org/10.7275/qv2q-rk76 Bennett, K. D., & Dukes, C. (2013). Employment instruction for secondary students with autism spectrum disorder: A systematic review of the literature. Education and Training in Autism and Developmental Disabilities, 48, 67-75. 101 Bennett, A. E., Miller, J. S., Stollon, N., Prasad, R., & Blum, N. J. (2018). Autism spectrum disorder and transition-aged youth. Current Psychiatry Reports, 20(103), 1-9. Bishop, M., Chapin, M. H., & Miller, S. (2008). Quality of life assessment in the measurement of rehabilitation outcome. Journal of Rehabilitation, 74(2), 45-54. Blanca, M. J., Alarcón, C., Arnau, J., Bono, R., & Bendayan, R. (2017). Non-normal data: Is ANOVA still a valid option? Psicothema, 29(4), 552-557. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Public Health, 6(149), 1-18. https://doi.org/10.3389/fpubh.2018.00149 Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., Bakken, S., Kaplan C. P., Squiers, L., Fabrizio, C., & Fernandez, M. (2009). How we design feasibility studies. American Journal of Preventive Medicine, 36(5), 452-457. Braunstein, V. L., Peniston, N., Perelman, A., & Cassano, M. C. (2013). The inclusion of fathers in investigations of autistic spectrum disorders. Research in Autism Spectrum Disorders, 7(7), 858-865. https://doi.org/10.1016/j.rasd.2013.03.005 Briesch, A. M., & Chafouleas, S. M. (2009b). Exploring student buy-in: Initial development of An instrument to measure likelihood of children’s intervention usage. Journal of Educational and Psychological Consultation, 19, 321-336. https://doi.org/10.1080/10474410903408885 Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Harvard University Press. Brown, D., & Cinamon, R. G. (2016). Contribution of personality to self-efficacy and outcome expectations in selecting a high school major among adolescents with learning disabilities. Career Development and Transition for Exceptional Individuals, 39(4), 237- 248. https://doi.org/10.1177/2165143415587689 Buettgen, A., Richardson, J., Beckham, K., Richardson, K., Ward, M., & Riemer, M. (2012). We did it together: a participatory action research study on poverty and disability. Disability & Society, 27(5), 603-616. Carter, S. L., & Wheeler, J. J. (2019). The social validity manual: Subjective evaluations of interventions (2nd ed.). Elsevier, Inc. Chen, J. L., Leader, G., Sung, C., & Leahy, M. (2015). Trends in employment for individuals with autism spectrum disorder: A review of the research literature. Journal of Autism and Developmental Disorders, 2, 115-127. 102 Coghlan, D., & Brydon-Miller, M. (2014). The SAGE encyclopedia of action research (Vols. 1- 2). SAGE Publications Ltd. https://doi.or/10.4135/9781446294406 Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). Routledge. Connor, A. (2017). Beyond skills to pay the bills: Effects of a vocational social skills intervention on psychosocial functioning among young adults with autism. PhD Thesis, Michigan State University, East Lansing, MI. (Order No. 10277019). Connor, A., Sung, C., Strain, A., Zeng, T., & Fabrizi, S. (2019). Building skills, confidence, and wellness: Psychosocial effects of soft skills training for young adults with autism. Journal of Autism and Developmental Disorders, 50, 2064-2076. https://doi.org/10.1007/s10803-019-03962-w Corbin Dwyer, S., & Buckle, J. L. (2009). The space between: On being an insider-outsider in qualitative research. International Journal of Qualitative Methods, 8, 54-63. Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Sage Publications, In Fawcett, S. B. (1991). Social validity: A note on methodology. Journal of Applied Behavior Analysis, 24(2), 235-239. Cumming, T., Strnadová, I., & Danker, J. (2020). Transitions of students with autism and intellectual disabilities in inclusive settings: The nexus between recommended and actual practice. Australasian Journal of Special and Inclusive Education, 44(1), 28-45. https://doi.org/10.1017/jsi.2020.1 da Silva Cardoso, E., Dutta, A., Chiu, C.-Y., Johnson, E. T., Kundu, M., & Chan, F. (2013). Social-cognitive predictors of stem career interests and goal persistence in college students with disabilities from racial and ethnic minority backgrounds. Rehabilitation Research, Policy, and Education, 27(4), 271-284. https://doi.org/10.1891/2168-6653.27.4.271 DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Sage Publications. Etikan, I., Abubakar Musa, S., & Sunusi Alkassim, R. (2016). Comparison of Convenience Sampling and Purposive Sampling. American Journal of Theoretical and Applied Statistics, 5, 1-4. https://doi.org/10.11648/j.ajtas.20160501.11 Farrow, C. (2016). Career decision-making patterns of college students with high functioning autism spectrum disorders, learning disabilities, and college students without disabilities (Order No. 10163848). Available from ProQuest Dissertations & Theses Global. (1844987431). Fawcett, S. B. (1991). Social validity: A note on methodology. Journal of Applied Behavior Analysis, 24(2), 235-239. 103 Gal, E., Landes, E., & Katz, N. (2015). Work performance skills in adults with and without high functioning autism spectrum disorders (HFASD). Research in Autism Spectrum Disorders 10, 71–77. Gal, E., Meir, A. B., & Katz, N. (2013). Development and reliability of the autism work skills questionnaire (AWSQ). American Journal of Occupational Therapy, 67, e1–5. https://doi.org/10.5014/ajot.2013.005066 Gibbons, M. M., Hyfantis, J., Cihak, D. F., Wright, R., & Mynatt, B. (2016). A social-cognitive exploration of the career and college understanding of young adults with intellectual disabilities. Professional School Counseling, 19, 80-91. Gilson, C., & Carter, E. (2016). Promoting social interactions and job independence for college students with autism or intellectual disability: a pilot study. Journal of Autism and Developmental Disorders, 46, 3583–3596. Gorenstein, M., Giserman-Kiss, I., Feldman, E., Isenstein, E. L., Donnelly, L., Wang, A. T., & Foss-Feig, J. H. (2020). Brief report: A job-based social skills program (JOBSS) for adults with autism Spectrum disorder: A pilot randomized controlled trial. Journal of Autism and Developmental Disorders, 50(12), 4527–4534. https://doi.org/10.1007/s10803-020-04482-8 Grob, C. M., Lerman, D. C., Langlinais, C. A., & Villante, N. K. (2019). Assessing and teaching job‐related social skills to adults with autism spectrum disorder. Journal of applied behavior analysis, 52(1), 150-172. https://doi.org/10.1002/jaba.503 Hackett, G., & Betz, N. (1981). A self-efficacy approach to the career development of women. Journal of Vocational Behavior, 18, 326-339. Hagner, D., Kurtz, A., Cloutier, H., Arakelian, C., Brucker, D. L., & May, J. (2012). Outcomes of a family-centered transition process for students with autism spectrum disorders. Focus on Autism and Other Developmental Disabilities, 27(1), 42-50. https://doi.org10.1177/1088357611430841 Hampton N., & Mason E. (2003). Learning disabilities, gender, sources of efficacy, self-efficacy beliefs, and academic achievement in high school students. Journal of School Psychology, 41, 101–12. Harley, D. A., Alston, R. J., & Middleton, R. A. (2007). Infusing social justice into rehabilitation education: Making a case for curricula refinement. Rehabilitation Education, 21(1), 41- 52. Hasan, A., Yasin, S. N. T. M., & Yunus, M. F. M. (2016). Technical competency for diploma in mechatronic engineering at polytechnics Malaysia. International Review of Management and Marketing, 6(4S), 96–199. 104 Hedley, D., Uljarevic, M., Cameron, L., Halder, S., Richdale, A., & Dissanayake, C., (2016). Employment programmes and interventions targeting adults with autism spectrum disorder: A systematic review of the literature. Autism, 21, 1-13. Hendricks, D. (2010). Employment and adults with autism spectrum disorders: challenges and strategies for success. Journal of Vocational Rehabilitation, 32, 125-134. Hergenrather, K. C., Geishecker, S., McGuire-Kuletz, M., Gitlin, D. J., & Rhodes, S. D. (2010). An introduction to community-based participatory research. Rehabilitation Education, 24(3&4), 225-238. Hillier. A., Campbell. H., Mastriani. K., Vreeburg Izzo, M., Kool-Tucker, A. K., Cherry, L., & Beversdorf, D. Q. (2007). Two-year evaluation of a vocational support program for adults on the autism spectrum. Career Development for Exceptional Individuals, 30, 35–47. Hillier, A., Fish, T., Siegel, J. H., & Beversdorf, D. Q. (2011). Social and Vocational Skills Training Reduces Self-reported Anxiety and Depression Among Young Adults on the Autism Spectrum. Journal of Developmental and Physical Disabilities, 23, 267-276. Howard, M. C. (2016). A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve? International Journal of Human- Computer Interaction, 31, 51-62. https://doi.org/10.1080/10447318.2015.1087664 Hutchinson N. L, Versnel J., Chin P., & Munby H. (2008). Negotiating accommodations so that work-based education facilitates career development for youth with disabilities. Journal of Prevention, Assessment, and Rehabilitation, 30, 123–136. Iacomini, S., Berardo, F., Cavallini, F., & Dipace, A. (2021). Assessment tools for the career planning of adolescents and adults with neurodevelopmental disorders: A systematic review. Journal of Clinical & Developmental Psychology, 3(3), 34-55. https://doi.org/10.13129/2612-4033/0110-3200 Ju, S., Zhang, D., & Pacha, J. (2012). Employability skills valued by employers as important for entry-level employees with and without disabilities. Career Development for Exceptional Individuals, 35, 29–38. https://doi.org/10.1177/0885728811419167 Ju. S., Pacha, J., Moore, K., & Zhang, D. (2014). Employability skills for entry-level employees with and without disabilities: A comparison between the perspectives of educators and employers. Journal of Vocational Rehabilitation, 40, 203–212. https://doi.org/10.3233/JVR-140685 Katz, N., Dejak, I., & Gal, E. (2015). Work performance evaluation and QoL of adults with high functioning autism spectrum disorders (HFASD). Work, 51, 887–892. Kidd, S. A., & Kral, M. J. (2005). Practicing participatory action research. Journal of Counseling Psychology, 52(2), 187-195. https://doi.org/10.1037/0022-0167.52.2.187 105 Kiener, M., & Koch, L. (2009). Action research in rehabilitation counseling. Journal of Applied Rehabilitation Counseling, 4(3), 19-26. Kim, J. H., & Choi, I. (2021). Choosing the level of significance: A decision-theoretic approach. Abacus, 57, 27-71. https://doi.org/10.1111/abac.12172 Kohler, P. D., Gothberg, J. E., Fowler, C., and Coyle, J. (2016). Taxonomy for transition programming 2.0: A model for planning, organizing, and evaluating transition education, services, and programs. Western Michigan University. Kosciulek, J. F. (2004). Theory of informed consumer choice in vocational rehabilitation. Rehabilitation Education, 18(1), 3–11. Krumboltz, J. D. (1979). A social learning theory of career decision making. In A. M. Mitchell, G. B. Jones & J. D. Krumboltz (Eds.), Social learning and career decision making (pp. 19-49). Carroll Press. Krumboltz, J. D. (1994). Improving career development theory from a social learning perspective. In M. L. Savickas & R. W Lent (Eds.), Convergence in career development theories: Implications for science and practice (pp. 20-22). CPP Books. Kuo, A. A., Crapnell, T., Lau, L., Anderson, K. A., & Shattuck, P. (2018). Stakeholder perspectives on research and practice in autism and transition. Pediatrics, 141(s4), S293- S299. https://doi.org/10.1542/peds.2016-4300F Lattimore, L. P., Parsons, M. B., & Reid, D. H. (2002). A prework assessment of task preferences among adults with autism beginning a supported job. Journal of Applied Behavior Analysis, 35, 85–88. Lattimore, L., Parsons, M. B., & Reid, D. H. (2003). Assessing preferred work among adults with autism beginning supported jobs: Identification of constant and alternating task preferences. Behavioral Interventions, 18, 161–177. http://dx.doi.org/10.1002/bin.138 Leahy, M. J., Del Valle, R. J., Landon, T. J., Iwanaga, K., Sherman, S. G., Reyes, A., & Chan, F. (2018). Promising and evidence-based practices in vocational rehabilitation: Results of a national Delphi study. Journal of Vocational Rehabilitation, 48, 37-48. https://doi.org/10.3233/JVR-170914 Lee, G. K., & Carter, E. W. (2012). Preparing transition-age students with high-functioning autism spectrum disorders for meaningful work. Psychology in the Schools, 49(10), 988- 1000. https://doi.org/10.1002/pits.21651 Lee, G. K., Chun, J., Hama, H., & Carter, E. W. (2018). Review of transition and vocational interventions for youth and adults with autism spectrum disorder. Journal of Autism and Developmental Disorders, 5(3), 268-284. 106 Lent, R. (2012). Social cognitive career theory. In S. Brown & R. Lent (Eds.), Career development and counseling: Putting theory and research to work (2nd ed., pp. 115-146). Wiley. Lent, R. W., & Brown, S. D. (2013). Social cognitive model of career self-management: Toward a unifying view of adaptive career behavior across the life span. Journal of Counseling Psychology, 60(4), 557-568. https://doi.org/10.1037/a0033446 Lent, R. W., Brown, S. D., & Hackett, G. (1994). Toward a unifying social cognitive theory of career and academic interest, choice, and performance. Journal of Vocational Behavior, 45(1), 79-122. https://doi.org/10.1006/jvbe.1994.1027 Lent, R. W., Brown, S. D., & Hackett, G. (2000). Contextual supports and barriers to career choice: A social cognitive analysis. Journal of Counseling Psychology, 47, 36-49. https://doi.org/10.1037/0022-0167.47.1.36 Lent, R. W., Hackett, G., & Brown, S. D. (1999). A social cognitive view of school-to-work transition. The Career Development Quarterly, 47, 297-311. Lerman, D. C., White, B., Grob, C., & Laudont, C. (2017). A clinic-based assessment for evaluating job-related social skills in adolescents and adults with autism. Behavioral Analysis Practice, 10, 323-336. https://doi.org/10.1007/s40617-017-0177-9 Levine, A., & Breshears, B. (2019). Discrimination at every turn: An intersectional ecological lens for rehabilitation. Rehabilitation Psychology, 64(2), 146-153. http://dx.doi.org/10.1037/rep0000266 Lindsay, S., Adams, T., McDougall, C., & Sanford, R. (2012). Skill development in an employment-training program for adolescents with disabilities. Disability and Rehabilitation, 34(3), 228–237. https://doi.org/10.3109/09638288.2011.603015 Liu, S., Huang, J., & Wang, M. (2014). Effectiveness of job search interventions: A meta- analytic review. Psychological Bulletin, 140(4), 1009-1041. https://doi.org/10.1037/a0035923 Lynas L (2014) Project ABLE (Autism: Building Links to Employment): A specialist employment service for young people and adults with an autism spectrum condition. Journal of Vocational Rehabilitation, 41, 13–21. McDonald, T. A., & Machalicek, W. (2013). Systematic review of intervention research with adolescents with autism spectrum disorders. Research in Autism Spectrum Disorders, 7, 1439-1460. McLaren. J., Lichtenstein, J. D., Lynch. D., Becker, D., & Drake, R. (2017). Individual placement and support for people with autism spectrum disorders: A pilot program. Administration and Policy in Mental Health, 44, 365–373. 107 Malterud, K. (2001). Qualitative research: Standards, challenges, and guidelines. The Lancet, 358, 483-488. Mercer, N., & Howe, C. (2012). Explaining the dialogic processes of teaching and learning: The value and potential of sociocultural theory. Learning, culture and social interaction, 1, 12-21. Miller, F. G., Neugebauer, S. R., Chafouleas, S. M., Briesch, A. M., & Riley-Tillman, T. C. (2013). Examining innovation usage: Construct validation of the Usage Rating Profile – Assessment [Poster presentation]. American Psychological Association Annual Convention, Honolulu, HI. https://production.wordpress.uconn.edu/educationurp/wp- content/uploads/sites/965/2014/09/2013-APA-Miller-Neugebauer-Chafouleas-Briesch- Riley-Tillman.pdf Milner, U. C., Rogers, E. S., Bloch, P., Costa, W., Pritchett, S., & Woods, T. (2015). Exploring the work lives of adults with serious mental illness from a vocational psychology perspective. Journal of Counseling Psychology, 62(4), 642-654. https://doi.org/10.1037/cou0000109 Murray, N., Hatfield, M., Falkmer, M., & Falkmer, T. (2016). Evaluation of career planning tools for use with individuals with autism spectrum disorder: A systematic review. Research in Autism Spectrum Disorders, 23, 188-202. Nicholas, D. B., Mitchell, W., Zulla, R., Solomatin, E., & Qi, S. (2019). A review of CommunityWorks Canada®: Toward employability among high school-age youth with autism Spectrum disorder. Global Pediatric Health, 6, 1-12. https://doi.org/10.1177/2333794X19885542 Niles, G. Y., & Harkins Monaco, E. A. (2019). Privilege, social identity and autism: Preparing preservice practitioners for intersectional pedagogy. Journal of the Division on Autism and Developmental Disabilities, 6, 112-123. Ochs, L. A., & Roessler, R. T. (2001). Student with disabilities: How ready are they for the 21st century? Rehabilitation Counseling Bulletin, 44, 170-176. Patil, V. H, Singh, S. N., Mishra, S., & Donavan, D. T. (2008). Efficient theory development and factor retention criteria: A case for abandoning the ‘eigenvalue greater than one’ criterion. Journal of Business Research, 61, 162–170. https://doi.org/10.1016/j.jbusres.2007.05.008 Patil, V. H, Singh, S. N., Mishra, S., & Donavan, D. T. (2017). Parallel analysis engine to aid in determining number of factors to retain using R [computer software], available from https://analytics.gonzaga.edu/parallelengine/ 108 Pongsaksri, M., Lersilp, S., & Suchart, S. (2017). Development and reliability of the basic skill assessment tool for adolescents with autism spectrum disorder. The Scientific World Journal, 2017, 1-7. https://doi.org/10.1155/2017/2403943 Rahmat, N., Ayub, A. R., & Buntat, Y. (2016). Employability skills constructs as job performance predictors for Malaysian polytechnic graduates: a qualitative study. Malaysian Journal of Society and Space, 12(3), 154–167. Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. Routledge. Reid, D. H., Parsons, M. A., & Green, C. A. (1998). Identifying work preferences among individuals with severe multiple disabilities prior to beginning supported work. Journal of Applied Behavior Analysis, 31, 281–285. http://dx.doi.org/10.1901/jaba.1998.31-281 Robles, M. M. (2012). Executive perceptions of the top 10 soft skills needed in today’s workplace. Business Communication Quarterly, 75, 453-465. Roux, A. M., Shattuck, P. T., Rast, J. E., Rava, J. A., & Anderson, K. A. (2015). National Autism Indicators Report: Transition into Young Adulthood. Life Course Outcomes Research Program, A.J. Drexel Autism Institute, Drexel University. https://drexel.edu/~/media/Files/autismoutcomes/publications/National%20Autism%20In dicators%20Report%20-%20July%202015.ashx Sarfraz, I., Mohan, M., Hewege, C., & Rajendran, D. (2018). An exploration of global employability skills: A systematic research review. International Journal of Work Organisation and Emotion, 9, 63-88. https://10.1504/IJWOE.2018.10012435 Schwartz, I. S., & Baer, D. M. (1991). Social validity assessments: Is current practice state of the art? Journal of Applied Behavior Analysis, 24(2), 189-204. Scott, M., Milbourn, B., Falkmer, M., Black, M., Bölte, S., Halladay, A., Lerner, M., Lounds Taylor, J., & Girdler, S. (2019). Factors impacting employment for people with autism spectrum disorder: A scoping review. Autism, 23(4), 869-901. https://doi.org/10.1177/1362361318787789 Seaman, R. L., & Cannella-Malone, H. I. (2016). Vocational Skills Interventions for Adults with Autism Spectrum Disorder: A Review of the Literature. Journal of Developmental and Physical Disabilities, 28(3), 479–494. https://doi.org/10.1007/s10882-016-9479-z Shattuck, P. T., Narendorf, S. C., Cooper, B., Sterzing, P. R., Wagner, M., & Taylor, J. L. (2012). Postsecondary education and employment among youth with an autism spectrum disorder. Pediatrics, 129(6), 1042-1049. Shattuck, P. T., Lau, L., Anderson, K. A., & Kuo, A. A. (2018). A national research agenda for the Transition of youth with autism. Pediatrics, 141(s4), s355-s361. https://doi.org/10.1542/peds.2016-4300M 109 Shattuck, P. T., Garfield, T., Roux, A. M., Rast, J. E., Anderson, K., McGhee Hassrick, E., & Kuo, A. (2020). Services for adults with autism spectrum disorders: A systems perspective. Current Psychiatry Reports, 22, 1-13. Shogren, K.A., & Raley, S.K. (2022). Positive psychology and disability: Creating a context for self-determination. In M. L. Wehmeyer & K. A. Shogren (Eds.), Positive psychology and disability series: Self-determination and causal agency theory: Integrating research into practice (pp. 3-12). Springer. https://doi-org.proxy1.cl.msu.edu/10.1007/978-3-031- 04260-7_1 Snell-Rood, C., Ruble, L., Kleinert, H., McGrew, J. H., Adams, M., Rodgers, A., Odom, J., Wong, W. H., & Yu, Y. (2020). Stakeholder perspectives on transition planning, implementation, and outcomes for students with autism spectrum disorder. Autism, 24(5), 1164-1176. https://doi.org/10.1177/1362361319894827 Sosnowy, C., Silverman, C., & Shattuck, P. (2018). Parents’ and young adults’ perspectives on transition outcomes for young adults with autism. Autism, 22, 29-39. Strain, A., & Sung, C. (2019, April). Assessing soft skills and employment-readiness skills for young adults with autism spectrum disorder [Concurrent presentation]. National Council of Rehabilitation Educators Spring Conference, San Diego, California. Strickland, D. C., Coles, D. C., & Southern, L. B. (2013). JobTIPS: A transition to employment program for individuals with autism spectrum disorders. Journal of Autism and Developmental Disorders, 43, 2472-2483. https://doi.org/10.1007/s10803-013-1800-4 Suchart, S., Lersilp, S., & Pongsaksri, M. (2015). Development of a basic work skills assessment tool for adolescents with autism. International Journal of Child Development and Mental Health, 3(2), 7-14. Sung, C., & Connor, A. (2017). Social-cognitive predictors of vocational outcomes in transition youth with epilepsy: Application of social cognitive career theory. Rehabilitation Psychology, 62(3), 276-289. https://doi.org/10.1037/rep0000161 Sung, C., Connor, A., Chen, J., Lin, C.-C., Kuo, H.-J., & Chun, J. (2019). Development, feasibility and preliminary efficacy of an employment-related social skills intervention for young adults with high functioning autism. Autism, 23(6), 1542-1553. https://doi.org/10.1177/1362361318801345 Sung, C., Connor, A., Strain, A., & Nasamran, A. (2021, under review). Short report: Feasibility and preliminary efficacy of an employment-readiness intervention for transition-age individuals with autism spectrum disorder. Autism. Terry, L. & Kelley, K. (2012). Sample size planning for composite reliability coefficients: Accuracy in parameter estimation via narrow confidence intervals. British Journal of Mathematical and Statistical Psychology, 65, 371–401. 110 United States Government Accountability Office. (2016). Youth with autism: Roundtable views of services needed during the transition into adulthood (GAO-17-109). https://www.gao.gov//assets/690/680525.pdf Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (M. Cole, V. John-Steiner, S. Scribner, & E. Souberman, Trans.). Harvard University Press. Walsh, L., Lydon, S., & Healy, O. (2014). Employment and Vocational Skills Among Individuals with Autism Spectrum Disorder: Predictors, Impact, and Interventions. Journal of Autism and Developmental Disorders, 1(4), 266-275. Webster, A. A., & Garvis, S. (2017). The importance of critical life moments: An explorative study of successful women with autism spectrum disorder. Autism, 21(6), 670-677. https://doi.org/10.1177/1362361316677719 Wehman, P., Lau, S., Molinelli, A., Brooke, V., Thompson, K., Moore, C., & West, M. (2012). Supported employment for young adults with autism spectrum disorder: Preliminary data. Research and Practice for Persons with Severe Disabilities, 37(3), 160–169. Wehman, P., Schall, C., Carr, S., Targett, P., West, M., & Cifu, G. (2014). Transition from school to adulthood for youth with autism spectrum disorder: What we know and what we need to know. Journal of Disability Policy Studies, 25, 30-40. https://doi.org/10.1177/1044207313518071 Wehman, P., Schall, C., McDonough, J., Molinelli, A., Riehle, E., Ham, W., & Thiss, W. R. (2013). Project SEARCH for youth with autism spectrum disorders: Increasing competitive employment on transition from high school. Journal of Positive Behavior Interventions, 15(3), 144–155. https://doi.org/10.1177/1098300712459760 Wehman, P., Schall, C. M., McDonough, J., Graham, C., Brooke, V., Riehle, J. E., Brooke, A., Ham, W., Lau, S., Allen, J., Avellone, L. (2016). Effects of an employer-based intervention on employment outcomes for youth with significant support needs due to autism. Autism, 21(3), 276-290. https://doi.org/10.1177/1362361316635826 Wehman, P., Schall, C., McDonough, J., Sima, A., Brooke, A., Ham, W., Whittenberg, H., Brooke, V., Avellone, L., & Riehle, E., (2019). Competitive employment for transition- aged youth with significant impact from autism: A multi-site randomized clinical trial. Journal of Autism and Developmental Disorders, 50, 1882–1897. https://doi.org/10.1007/s10803-019-03940-2 Wei, X., Yu, J. W., Shattuck, P., & Blackorby J. (2015). High school math and science preparation and postsecondary STEM participation for students with an autism spectrum disorder. Focus on Autism and Other Developmental Disorders, 32, 83-92. 111 Willis, S. (2002). The relationship of social cognitive variables to outcomes among young adults with emotional disturbance. Unpublished dissertation, University of Maryland, College Park. Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203-214. Yang, H., Cheung, C. & Fang, C. C. (2015). An empirical study of hospitality employability skills: perceptions of entry-level hotel staff in China. Journal of Hospitality & Tourism Education, 27(4), 161–170. 112 APPENDIX A: EMPLOYMENT SUCCESS: SOFT SKILLS ASSESSMENT Figure A1.1 Employment Success: Soft Skills Assessment (self-report, page 1) 113 Figure A1.2 Employment Success: Soft Skills Assessment (self-report, page 2) 114 Figure A1.3 Employment Success: Soft Skills Assessment (self-report, page 3) 115 Figure A1.4 Employment Success: Soft Skills Assessment (self-report, page 4) 116 Figure A2.1 Employment Success: Soft Skills Assessment (informant-report, page 1) 117 Figure A2.2 Employment Success: Soft Skills Assessment (informant-report, page 2) 118 Figure A2.3 Employment Success: Soft Skills Assessment (informant-report, page 3) 119 Figure A2.4 Employment Success: Soft Skills Assessment (informant-report, page 4) 120 APPENDIX B: SOCIAL VALIDITY & FEASIBILITY SURVEYS Figure B1.1 Social Validity Survey for All Stakeholders (page 1) 121 Figure B1.2 Social Validity Survey for All Stakeholders (page 2) 122 Figure B1.3 Social Validity Survey for All Stakeholders (page 3) 123 Figure B2.1 Feasibility Survey for TAI-ASD (page 1) 124 Figure B2.2 Feasibility Survey for TAI-ASD (page 2) 125 Figure B2.3 Demographics for TAI-ASD (page 1) 126 Figure B2.4 Demographics for TAI-ASD (page 2) 127 Figure B3.1 Feasibility Survey Parents/Caregivers (page 1) 128 Figure B3.2 Feasibility Survey Parents/Caregivers (page 2) 129 Figure B3.3 Demographics for Parents/Caregivers (page 1) 130 Figure B3.4 Demographics for Parents/Caregivers (page 2) 131 Figure B4.1 Feasibility for Service Providers (page 1) 132 Figure B4.2 Feasibility for Service Providers (page 2) 133 Figure B4.3 Feasibility for Service Providers (page 3) 134 Figure B4.4 Demographics for Service Providers (page 1) 135 Figure B4.5 Demographics for Service Providers (page 2) 136 APPENDIX C: CORRELATION TABLES Table C1 Correlations Between ESSS Sections and Demographic Variables (Social Validity) Stakeholdera Ethnicityc n  p n rs p Introduction Section 76 .167 .35 72 - .038 .75 Rating of Items Section 74 .173 .33 70 - .118 .33 Scoring Your Answers Section 74 .164 .38 71 - .058 .63 Understanding Your Answers Section 71 .228 .16 68 .014 .91 Gender Identityb Geographic Aread Introduction Section 69 .187 .31 72 .120 .31 Rating of Items Section 67 .239 .15 70 .191 .11 Scoring Your Answers Section 68 .173 .37 71 .160 .18 Understanding Your Answers Section 65 .179 .36 68 - .130 .29 Note. df = degrees of freedom;  = eta; sig = significance; n = sample size rs = Spearman’s rho. a Stakeholder = TAI-ASD, parents/caregivers, service providers. bGender Identity = woman, man, non-binary. cEthnicity = white, BIPOC. d Geographic Area = urban, rural/suburban. Table C2 Correlations Between ESSS Sections to Evaluate Multicollinearity (Social Validity) Introduction Scoring Your Scoring Your Rating of Items Section Answers Section Answers Section Introduction Section -- Rating of Items Section .404*** -- Scoring Your Answers Section .652*** .498*** -- Understanding Your Answers Section .393*** .319** .557*** -- Note. Listwise N = 71. **p < .01. ***p < .001. 137 Table C3 Correlations Between Consumer Feasibility Domains and Demographic Variables Stakeholdera Ethnicityb Geographic Areac (n = 36) (n = 31) (n = 31) rs p rs p rs p Acceptability .019 .91 -.058 .76 -.363 .04* Desirability .159 .35 -.045 .81 .174 .35 Collaboration -.174 .31 -.300 .10 -.048 .80 Understanding .014 .94 -.153 .41 .346 .06 Work Statusd School Statuse Gender Identityf (n = 31) (n = 31) (n = 30) rs p rs p  p Acceptability .232 .21 -.425 .02* .228 .49 Desirability .302 .10 .040 .83 .265 .38 Collaboration .027 .88 -.297 .10 .114 .89 Understanding .119 .52 .095 .61 .329 .22 Note. rs = Spearman’s rho; sig = significance;  = eta. a Stakeholder = TAI-ASD, parents/caregivers. bEthnicity = white, BIPOC. cGeographic Area = urban, rural/suburban. dWork Status = working or interested in working, not working or not interested in working. eSchool Status = attending, not attending. fGender Identity = woman, man, non-binary. *p < .05. Table C4 Correlations Between Consumer Feasibility Domains to Evaluate Multicollinearity Acceptability Desirability Understanding Collaboration Acceptability -- Desirability -.363* -- Understanding -.143 .713*** -- Collaboration .164 -.257 -.375* -- Note. Listwise N = 36. *p < .05. ***p < .001. 138 Table C5 Correlations Between Service Provider Feasibility Domains and Demographic Variables Stakeholdera Ethnicityb Geographic Areac (n = 43) (n = 42) (n = 42) rs p rs p rs p Acceptability .013 .93 -.049 .76 -.085 .59 Desirability -.051 .75 .060 .71 -.265 .09 Collaboration .010 .52 .217 .17 .095 .55 Understanding .061 .70 .058 .71 .124 .43 System Climate -.087 .58 -.035 .83 -.056 .73 System Support .011 .94 -.067 .67 -.014 .93 Gender Identityd Work Settinge (n = 41) (n = 42) rs p  p Acceptability .067 .68 .348 .30 Desirability .035 .83 .315 .41 Collaboration -.049 .76 .155 .92 Understanding .161 .31 .315 .41 System Climate .256 .11 .332 .35 System Support .021 .90 .374 .22 Note. rs = Spearman’s rho; sig = significance;  = eta. a Stakeholder = paraprofessionals, professionals. bEthnicity = white, BIPOC. cGeographic Area = urban, rural/suburban. dGender Identity = woman, man, non-binary. eWork Setting = college/university, not-for-profit, private organization, public agency, school district. 139 Table C6 Correlations Between Service Provider Feasibility Domains to Evaluate Multicollinearity Acceptability Desirability Understanding Collaboration System Climate System Support Acceptability -- Desirability .668*** -- Understanding .549*** .338* -- Collaboration - .012 -.118 -.185 -- System Climate .404** .726*** .416** -.154 -- System Support -.363* -.341* -.426** .441** -.297 -- Note. Listwise N = 43. *p < .05. **p < .01. ***p < .001. 140 APPENDIX D: SOCIAL VALIDITY & FEASIBILITY MEANS TABLES Table D1 Social Validity Domain Means Social Validity Domain Stakeholder n M SD f p Understandability TAI-ASD 20 2.87 1.09 .18 .33 (Introduction Section) Parent/Caregiver 15 3.17 0.62 Service Provider 40 3.13 0.45 Clarity TAI-ASD 20 2.89 1.07 .16 .40 (Introduction Section) Parent/Caregiver 15 3.14 0.74 Service Provider 40 3.15 0.51 Understandability TAI-ASD 20 3.63 0.56 .18 .29 (Rating of Item Section) Parent/Caregiver 15 3.65 0.44 Service Provider 40 3.45 0.46 Clarity TAI-ASD 20 3.65 0.49 .21 .20 (Rating of Item Section) Parent/Caregiver 15 3.62 0.45 Service Provider 40 3.44 0.47 Relevance TAI-ASD 20 3.56 0.47 .11 .67 (Rating of Item Section) Parent/Caregiver 15 3.62 0.42 Service Provider 40 3.51 0.43 Understandability TAI-ASD 20 3.20 0.95 .14 .50 (Scoring Your Answers Section) Parent/Caregiver 14 3.43 0.65 Service Provider 41 3.17 0.59 Clarity TAI-ASD 20 3.26 0.96 .14 .51 (Scoring Your Answers Section) Parent/Caregiver 14 3.43 0.65 Service Provider 41 3.17 0.59 Ease of Use TAI-ASD 20 2.81 0.82 .16 .40 (Scoring Your Answers Section) Parent/Caregiver 14 3.07 0.47 Service Provider 41 2.81 0.59 Understandability TAI-ASD 19 3.47 0.61 .17 .36 (Understanding Your Answers Parent/Caregiver 13 3.65 0.47 Section) Service Provider 40 3.25 0.59 Clarity TAI-ASD 19 3.47 0.61 .25 .13 (Understanding Your Answers Parent/Caregiver 13 3.62 0.51 Section) Service Provider 40 3.28 0.55 Utility TAI-ASD 19 3.29 0.65 .20 .26 (Understanding Your Answers Parent/Caregiver 13 3.58 0.49 Section) Service Provider 40 3.29 0.54 141 Table D2 Social Validity Item Level Means for Introduction Section & Ease of Use Sectiona Social Validity Item Stakeholder n M SD The instructions section is easy to understand. TAI-ASD 20 2.89 1.12 (Introduction Section) Parent/Caregiver 15 3.20 0.77 Service Provider 40 3.18 0.50 How to rate items is easy to understand. TAI-ASD 20 2.84 1.09 (Introduction Section) Parent/Caregiver 15 3.13 0.64 Service Provider 40 3.08 0.57 The instructions section is written clearly. TAI-ASD 20 2.95 1.10 (Introduction Section) Parent/Caregiver 15 3.13 0.74 Service Provider 40 3.18 0.50 How to rate items is written clearly. TAI-ASD 20 2.84 1.09 (Introduction Section) Parent/Caregiver 15 3.15 0.74 Service Provider 40 3.13 0.56 The scoring table is easy to use. TAI-ASD 20 2.75 0.97 (Scoring Your Answers Section) Parent/Caregiver 14 3.21 0.58 Service Provider 41 3.05 0.67 Recording scores is easy to do. TAI-ASD 20 2.70 0.98 (Scoring Your Answers Section) Parent/Caregiver 14 3.21 0.58 Service Provider 41 2.92 0.72 In general, it would be easy for autistic youth to TAI-ASD 20 2.65 0.88 score the assessment on their own. (Scoring Parent/Caregiver 14 2.57 0.85 Your Answers Section) Service Provider 41 2.23 0.96 In general, it would be easy for others to score TAI-ASD 20 3.15 0.93 the assessment on their own e.g., parents, Parent/Caregiver 14 3.29 0.47 teachers, paraprofessionals, job coaches, etc. Service Provider 41 3.05 0.50 (Scoring Your Answers Section) a Understandability and Clarity (Introduction Section), and Ease of Use (Scoring Your Answers Section) were the only domains that did not meet the social validity benchmark (3 on a 4-point scale), thus further analysis was conducted at the item level. 142 Table D3 Consumer Feasibility Item Level Means Item Stakeholder n M SD 3. I could see myself taking this assessment TAI-ASD 20 4.35 1.42 again. (desirability) Parent/Caregiver 16 4.56 1.26 4. This assessment is a good way to help TAI-ASD 20 4.45 1.40 people. (desirability) Parent/Caregiver 16 4.81 0.98 *5. I would not want to take this assessment TAI-ASD 20 3.75 1.55 again. (desirability) Parent/Caregiver 16 4.00 1.41 8. If I knew a friend/young adult was having TAI-ASD 20 4.10 1.55 trouble finding or keeping a job, I would tell them to try this assessment. (desirability) Parent/Caregiver 16 4.31 1.20 15. I was excited to try this assessment. TAI-ASD 20 4.20 1.28 (desirability) Parent/Caregiver 16 4.63 1.41 16. I would volunteer to take this assessment TAI-ASD 20 4.45 1.43 again. (desirability) Parent/Caregiver 16 4.63 1.31 19. I liked taking this assessment. TAI-ASD 20 4.15 1.53 (desirability) Parent/Caregiver 16 4.69 1.30 2. I understand why this assessment was picked TAI-ASD 20 4.35 1.57 to help me/my child. (understanding) Parent/Caregiver 16 5.19 0.75 6. It is clear what I had to do. TAI-ASD 20 5.00 0.92 (understanding) Parent/Caregiver 16 4.37 1.54 9. I was able to do every step of the assessment TAI-ASD 20 4.90 1.59 with little or no help. (understanding) Parent/Caregiver 16 5.31 1.08 13. This assessment helped me understand my TAI-ASD 20 4.40 1.64 (child's) strengths. (understanding) Parent/Caregiver 16 4.56 0.89 17. This assessment helped me understand TAI-ASD 20 4.55 1.61 where I/my child might need more training or practice. (understanding) Parent/Caregiver 16 4.69 1.30 18. I was able to use this assessment correctly. TAI-ASD 20 5.30 1.13 (understanding) Parent/Caregiver 16 5.13 0.81 10. Collaboration with others is needed in TAI-ASD 20 2.60 1.57 order to use this assessment, e.g., professionals, parents/caregivers. Parent/Caregiver 16 3.44 1.97 (collaboration) 14. Regular communication with others is TAI-ASD 20 2.45 1.64 needed in order to complete this assessment e.g., professionals, parents/caregivers. (collaboration) Parent/Caregiver 16 2.75 1.65 143 Table D3 (cont’d) Item Stakeholder n M SD *1. This assessment was too much work for TAI-ASD 20 4.30 1.63 me. (acceptability) Parent/Caregiver 16 3.88 1.31 *7. This took too long to do. TAI-ASD 20 3.35 1.84 (acceptability) Parent/Caregiver 16 4.19 1.47 *11. There are too many steps to complete. TAI-ASD 20 3.55 2.11 (acceptability) Parent/Caregiver 16 3.56 1.67 *12. Taking this assessment got in the way of TAI-ASD 20 4.70 1.38 doing other things. (acceptability) Parent/Caregiver 16 4.31 1.70 Note. * indicates the item is reverse scored; means and standard deviations reflect the reverse score. Feasibility domains are displayed in parentheses for each item (desirability, understanding, collaboration, acceptability). 144 Table D4 Service Provider Feasibility Item Level Means Item Stakeholder n M SD 1. This assessment is an effective choice for Paraprofessional 21 4.48 1.12 understanding employment skills. (Desirability) Professional 22 4.41 0.85 6. This assessment is a fair way to evaluate Paraprofessional 21 4.38 1.32 young adult’s employment-related skills. (Desirability) Professional 22 4.50 0.80 *8. I would not be interested in using this Paraprofessional 21 4.52 1.12 assessment. (Desirability) Professional 22 4.41 1.26 10. I would have positive attitudes about Paraprofessional 21 5.00 1.27 implementing the assessment. (Desirability) Professional 22 4.86 0.64 11. This assessment is a good way to assess Paraprofessional 21 4.67 1.02 young adult's employment-related skills. (Desirability) Professional 22 4.45 0.96 17. I would implement this assessment with a Paraprofessional 21 4.81 0.81 good deal of enthusiasm. (Desirability) Professional 22 4.50 0.80 20. Using this assessment would not disrupt Paraprofessional 21 4.76 1.00 young adults engaging in other employment- related activities. (Desirability) Professional 22 4.82 1.01 21. I would be committed to carrying out this Paraprofessional 21 4.71 1.01 assessment. (Desirability) Professional 22 4.77 0.81 23. The assessment procedures easily fit in Paraprofessional 21 4.67 0.97 with my current practices. (Desirability) Professional 22 4.45 1.18 4. I understand how to use this assessment. Paraprofessional 21 5.05 1.02 (Understanding) Professional 22 5.09 0.68 5. I am knowledgeable about the assessment Paraprofessional 21 4.95 0.97 procedures. (Understanding) Professional 22 4.95 1.00 *13. Scoring the assessment is too Paraprofessional 21 4.14 1.35 complicated. (Understanding) Professional 22 4.05 1.21 25. I understand the procedures of this Paraprofessional 21 5.24 0.70 assessment. (Understanding) Professional 22 4.96 0.79 30. The items on this assessment are Paraprofessional 21 4.90 0.83 understandable and easy to read. (Understanding) Professional 22 5.27 0.70 15. Collaboration with others is required in Paraprofessional 21 3.90 1.73 order to use the assessment, e.g., professionals, parents/caregivers. (Collaboration) Professional 22 4.27 1.24 145 Table D4 (cont’d) Item Stakeholder n M SD 28. Regular communication with others is Paraprofessional 21 4.48 1.57 needed to implement the assessment procedures, e.g., professionals, Professional 22 3.82 1.47 parents/caregivers. (Collaboration) 3. I would be able to allocate my time to Paraprofessional 21 4.90 0.89 implement the assessment. (Acceptability) Professional 22 4.64 1.09 7. The total time required to implement the Paraprofessional 21 5.19 0.51 assessment procedures would be manageable. (Acceptability) Professional 22 4.73 0.94 12. Preparation of materials needed for the Paraprofessional 21 5.19 1.12 assessment would be minimal. (Acceptability) Professional 22 5.18 0.66 16. Material resources needed for this Paraprofessional 21 4.95 1.07 assessment is reasonable. (Acceptability) Professional 22 5.18 0.66 *18. This assessment is too complex to carry Paraprofessional 21 4.70 1.00 out accurately. (Acceptability) Professional 22 4.68 1.17 27. The amount of time required for record Paraprofessional 21 4.76 0.54 keeping would be reasonable. (Acceptability) Professional 22 4.64 0.95 9. My administrator/supervisor would be Paraprofessional 21 5.00 1.23 supportive of my use of this assessment. (System Climate) Professional 22 4.86 0.77 14. Using this assessment would be consistent Paraprofessional 21 4.86 1.01 with the mission of my work setting. (System Climate) Professional 22 4.90 0.87 19. The assessment procedures are consistent Paraprofessional 21 4.10 1.14 with the way things are done in my school system/company. (System Climate) Professional 22 4.18 1.18 26. My work environment is conducive to Paraprofessional 21 4.81 1.17 implementation of an assessment like this. (System Climate) Professional 22 4.86 0.71 *2. I would need additional resources to carry Paraprofessional 21 4.29 1.62 out the assessment. (System Support) Professional 22 4.18 1.44 *22. I would need consultative support to score Paraprofessional 21 4.62 1.50 the assessment. (System Support) Professional 22 4.86 1.08 146 Table D4 (cont’d) Item Stakeholder n M SD *24. I would need consultative support to Paraprofessional 21 4.45 1.60 implement the assessment. (System Support) Professional 22 4.64 1.29 *29. I would require additional professional Paraprofessional 21 4.43 1.63 development to implement the assessment. (System Support) Professional 22 4.77 1.34 Note. * indicates the item is reverse scored; means and standard deviations reflect the reverse score. Feasibility domains are displayed in parentheses for each item (desirability, understanding, collaboration, acceptability, system climate, system support). 147 APPENDIX E: MANOVA MULTIVARIATE & UNIVARIATE TABLES Table E1.1 Social Validity - Box’s Test of Equality of Covariance Matrices Design: Intercept + Stakeholder Group Box’s M 61.07 F 2.71 df1 20 df2 5331.17 Sig < .001 Table E1.2 Social Validity MANOVA – Multivariate Results Partial Observed F df p Eta-Squared Powera Pillai’s Trace: Intercept 1173.94 4 < .001 .99 1.00 Pillai’s Trace: Stakeholder Group 1.16 8 .33 .07 .52 a Computed using alpha = .05. Table E1.3 Social Validity MANOVA – Univariate Results Standard 95% CI Partial Eta- Observed ESSS Section Stakeholder Group B t p Error (lower, upper) Squared Powera Introduction Intercept 3.14 0.11 29.84 < .001 2.93, 3.35 0.93 1.00 TAI-ASD -0.16 0.18 -0.88 0.38 -0.53, 0.21 0.01 0.14 Parent/Caregiver 0.15 0.21 0.72 0.48 -0.27, 0.57 0.01 0.11 Service Provider - - - - - - - Rating of Items Intercept 3.48 0.07 49.59 < .001 3.34, 3.62 0.97 1.00 TAI-ASD 0.18 0.12 1.46 0.15 -0.07, 0.42 0.03 0.30 Parent/Caregiver 0.17 0.14 1.24 0.22 -0.11, 0.45 0.02 0.23 Service Provider - - - - - - - 148 Table E1.3 (cont’d) Standard 95% CI Partial Eta- Observed ESSS Section Stakeholder Group B t p Error (lower, upper) Squared Powera Scoring Your Intercept 2.94 0.09 32.10 < .001 2.76, 3.12 0.94 1.00 Answers TAI-ASD 0.11 0.16 0.71 0.48 -0.21, 0.43 0.01 0.11 Parent/Caregiver 0.26 0.18 1.44 0.15 -0.10, 0.63 0.03 0.30 Service Provider - - - - - - - Understanding Your Intercept 3.28 0.09 38.16 < .001 3.11, 3.45 0.96 1.00 Answers TAI-ASD 0.10 0.15 0.67 0.50 -0.20, 0.40 0.01 0.10 Parent/Caregiver 0.32 0.17 1.88 0.07 -0.02, 0.67 0.05 0.46 Service Provider - - - - - - - a Computed using alpha = .05. Table E2.1 Consumer Feasibility - Box’s Test of Equality of Covariance Matrices Design: Intercept + Stakeholder Group + Geographic Area + School Status Box’s M 71.48 F 1.49 df1 30 df2 904.05 Sig .046 149 Table E2.2 Consumer Feasibility MANOVA – Multivariate Results Partial Observed F df p Eta-Squared Powera Wilks’ Lambda: Intercept 276.10 4 < .001 .98 1.00 Wilks’ Lambda: Stakeholder Group 4.59 4 .007** .43 .89 Wilks’ Lambda: Geographic Area 3.59 4 .020* .37 .80 Wilks’ Lambda: School Status .88 4 .489 .13 .24 a Computed using alpha = .05. *p < .05. **p < .01. Table E2.3 Consumer Feasibility MANOVA – Univariate Results Independent Standard 95% CI Partial Eta- Observed Feasibility Domain B t p Variables Error (lower, upper) Squared Powera Acceptability Intercept 3.32 0.50 6.58 < .001 2.28, 4.35 0.62 1.00 TAI-ASD 0.03 0.50 0.06 0.96 -1.00, 1.06 0.00 0.05 Parent/Caregiver - - - - - - - Rural/Suburban 0.36 0.60 0.60 0.55 -0.88, 1.60 0.01 0.09 Urban - - - - - - - Attending school 0.86 0.52 1.64 0.11 -0.21, 1.92 0.09 0.35 Not attending school - - - - - - - Desirability Intercept 5.49 0.45 12.20 < .001 4.57, 6.42 0.85 1.00 TAI-ASD -1.22 0.45 -2.71 0.01* -2.14, -0.30 0.21 0.74 Parent/Caregiver - - - - - - - Rural/Suburban -1.26 0.54 -2.35 0.03 -2.37, -0.16 0.17 0.62 Urban - - - - - - - Attending school 0.66 0.46 1.41 0.17 -0.30, 1.61 0.07 0.28 Not attending school - - - - - - - 150 Table E2.3 (cont’d) Independent Standard 95% CI Partial Eta- Observed Feasibility Domain B t p Variables Error (lower, upper) Squared Powera Understanding Intercept 5.79 0.35 16.37 < .001 5.07, 6.52 0.91 1.00 TAI-ASD -0.83 0.35 -2.36 0.03 -1.56, -0.11 0.17 0.62 Parent/Caregiver - - - - - - - Rural/Suburban -1.26 0.42 -2.97 0.01* -2.13, -0.39 0.25 0.82 Urban - - - - - - - Attending school 0.37 0.37 1.02 0.32 -0.38, 1.12 0.04 0.17 Not attending school - - - - - - - Collaboration Intercept 3.29 0.71 4.65 < .001 1.84, 4.75 0.45 0.99 TAI-ASD 0.95 0.71 1.34 0.19 -0.50, 2.40 0.06 0.25 Parent/Caregiver - - - - - - - Rural/Suburban 0.01 0.85 0.01 0.99 -1.73, 1.75 0.00 0.05 Urban - - - - - - - Attending school 0.77 0.73 1.05 0.30 -0.73, 2.26 0.04 0.17 Not attending school - - - - - - - a Computed using alpha = .05. *Bonferroni corrections, p < .05  4 = .0125. Table E3.1 Service Provider - Box’s Test of Equality of Covariance Matrices Design: Intercept + Stakeholder Group Box’s M 39.14 F 1.57 df1 21 df2 6152.12 Sig .047 151 Table E3.2 Service Provider Feasibility MANOVA – Multivariate Results Partial Observed F df p Eta-Squared Powera Wilks’ Lambda: Intercept 544.35 6 < .001 .99 1.00 Wilks’ Lambda: Stakeholder Group .16 6 .99 .03 .08 a Computed using alpha = .05. Table E3.3 Service Provider Feasibility MANOVA – Univariate Results Standard 95% CI Partial Eta- Observed Feasibility Domain Stakeholder Group B t p Error (lower, upper) Squared Powera Acceptability Intercept 4.84 0.13 37.16 < .001 4.58, 5.10 0.97 1.00 Paraprofessional 0.11 0.19 0.59 0.56 -0.27, 0.49 0.01 0.09 Professional - - - - - - - Desirability Intercept 4.58 0.15 30.81 < .001 4.28, 4.88 0.96 1.00 Paraprofessional 0.09 0.21 0.43 0.67 -0.34, 0.52 0.00 0.07 Professional - - - - - - - Understanding Intercept 4.87 0.13 36.94 < .001 4.60, 5.13 0.97 1.00 Paraprofessional -0.01 0.19 -0.04 0.97 -0.39, 0.37 0.00 0.05 Professional - - - - - - - Collaboration Intercept 2.96 0.29 10.04 < .001 2.36, 3.55 0.71 1.00 Paraprofessional -0.15 0.42 -0.34 0.73 -1.00, 0.71 0.00 0.06 Professional - - - - - - - System Climate Intercept 4.70 0.18 25.84 < .001 4.33, 5.07 0.94 1.00 Paraprofessional -0.01 0.26 -0.04 0.97 -0.54, 0.52 0.00 0.05 Professional - - - - - - - System Support Intercept 4.61 0.25 18.70 < .001 4.12, 5.11 0.90 1.00 Paraprofessional -0.17 0.35 -0.47 0.64 -0.88, 0.55 0.01 0.08 Professional - - - - - - - a Computed using alpha = .05. 152 APPENDIX F: RELIABILITY TABLES Table F1 Confidence Scale Item Means, Standard Deviations, Item-Total Correlations, and Cronbach Alpha Adjustments Item-Total Cronbach Alpha Confidence Scale Item Stakeholder M SD Correlation if Item Deleted C1. Have good hygiene (e.g., bathe & brush TAI-ASD 3.42 0.69 0.66 0.87 teeth to get ready for work). Parent/Caregiver 2.94 0.77 0.66 0.84 Service Provider 2.98 0.87 0.17 0.92 C2. Use an appropriate voice volume based TAI-ASD 3.11 0.81 0.63 0.87 on the location and situation. Parent/Caregiver 2.75 1.00 0.67 0.84 Service Provider 2.71 0.86 0.51 0.91 C3. Use appropriate personal space based on TAI-ASD 3.05 0.78 0.66 0.87 the type of conversation. Parent/Caregiver 2.50 1.03 0.68 0.84 Service Provider 2.98 0.75 0.28 0.92 C4. Start a conversation and keep it going. TAI-ASD 2.47 0.70 0.41 0.88 Parent/Caregiver 2.69 1.14 0.23 0.86 Service Provider 2.52 1.07 0.37 0.92 C5. When talking with coworkers and TAI-ASD 3.16 1.07 0.44 0.88 supervisors, I/they do not share things that are Parent/Caregiver 2.75 1.00 0.37 0.85 too personal. Service Provider 2.45 1.02 0.32 0.92 C6. When someone is talking to me/them, TAI-ASD 3.05 1.03 0.40 0.88 I/they listen without interrupting. Parent/Caregiver 2.69 0.95 0.67 0.84 Service Provider 2.36 0.91 0.68 0.91 C7. Use appropriate social manners (e.g., TAI-ASD 3.32 1.11 0.49 0.88 please and thank you). Parent/Caregiver 3.00 0.73 0.15 0.86 Service Provider 2.79 1.00 0.68 0.91 C8. Work well with others. TAI-ASD 3.05 0.97 0.38 0.88 Parent/Caregiver 2.75 0.86 0.54 0.85 Service Provider 2.83 0.91 0.60 0.91 153 Table F1 (cont’d) Item-Total Cronbach Alpha Confidence Scale Item Stakeholder M SD Correlation if Item Deleted C9. Show respect for others. TAI-ASD 3.84 0.50 0.81 0.87 Parent/Caregiver 3.13 0.89 0.58 0.84 Service Provider 3.12 0.74 0.72 0.91 C10. Are ok with people having different TAI-ASD 3.11 1.10 0.42 0.88 opinions. Parent/Caregiver 2.50 0.82 0.09 0.86 Service Provider 2.52 0.86 0.48 0.91 C11. Respond to feedback with a positive TAI-ASD 3.11 0.81 0.48 0.88 attitude. Parent/Caregiver 2.50 0.89 0.66 0.84 Service Provider 2.50 0.92 0.72 0.91 C12. Use different ways to speak up for TAI-ASD 2.79 1.08 0.29 0.88 myself/themselves depending on the situation. Parent/Caregiver 2.25 0.93 0.51 0.85 Service Provider 2.19 0.89 0.49 0.91 C13. Plan my/their time to get tasks done. TAI-ASD 2.74 0.99 0.51 0.88 Parent/Caregiver 2.00 0.89 0.25 0.86 Service Provider 2.10 0.98 0.56 0.91 C14. Shift my/their attention from one task to TAI-ASD 2.79 0.86 0.41 0.88 another when being asked. Parent/Caregiver 2.50 0.63 0.44 0.85 Service Provider 2.24 0.93 0.72 0.91 C15. Show flexibility when things change. TAI-ASD 2.74 0.81 0.40 0.88 Parent/Caregiver 2.13 0.72 0.41 0.85 Service Provider 2.19 0.99 0.63 0.91 C16. When I/they say they are going to do TAI-ASD 3.16 0.69 0.63 0.87 something, I/they follow through and do it. Parent/Caregiver 2.63 0.72 0.27 0.85 Service Provider 2.74 1.06 0.55 0.91 C17. Notice when there is a problem or TAI-ASD 3.11 0.88 0.54 0.87 conflict. Parent/Caregiver 2.13 0.50 -0.22 0.86 Service Provider 2.33 0.90 0.49 0.91 154 Table F1 (cont’d) Item-Total Cronbach Alpha Confidence Scale Item Stakeholder M SD Correlation if Item Deleted C18. Fix a problem or conflict. TAI-ASD 2.32 0.89 0.49 0.88 Parent/Caregiver 2.06 0.85 0.18 0.86 Service Provider 1.86 0.78 0.58 0.91 C19. Be in charge of my/their emotions. TAI-ASD 3.00 1.20 0.52 0.88 Parent/Caregiver 2.31 1.01 0.65 0.84 Service Provider 2.36 0.96 0.68 0.91 C20. Use different ways to cope with stress TAI-ASD 3.11 0.81 0.28 0.88 (e.g., take a break, deep breathing). Parent/Caregiver 2.06 0.85 0.73 0.84 Service Provider 2.33 0.90 0.59 0.91 C21. Take responsibility when I/they have TAI-ASD 3.53 0.70 0.61 0.87 made a mistake. Parent/Caregiver 2.50 0.97 0.31 0.85 Service Provider 2.31 0.95 0.57 0.91 C22. Show a positive attitude towards work. TAI-ASD 2.95 0.85 0.41 0.88 Parent/Caregiver 2.44 0.89 0.43 0.85 Service Provider 2.93 1.00 0.62 0.91 C23. Show confidence in my/their skills and TAI-ASD 2.89 0.81 0.29 0.88 abilities. Parent/Caregiver 2.50 0.97 0.20 0.86 Service Provider 2.57 0.80 0.51 0.91 Note. TAI-ASD (n = 19), parent/caregiver (n = 16), service provider (n = 42). 155 Table F2 Frequency Scale Item Means, Standard Deviations, Item-Total Correlations, and Cronbach Alpha Adjustments Item-Total Cronbach Alpha Frequency Scale Item Stakeholder M SD Correlation if Item Deleted F1. Have good hygiene (e.g., bathe & brush TAI-ASD 3.11 0.88 0.60 0.91 teeth to get ready for work). Parent/Caregiver 2.69 0.95 0.44 0.82 Service Provider 2.76 0.92 0.47 0.92 F2. Use an appropriate voice volume based on TAI-ASD 3.00 0.75 0.51 0.91 the location and situation. Parent/Caregiver 2.50 0.97 0.40 0.82 Service Provider 2.73 0.81 0.65 0.91 F3. Use appropriate personal space based on TAI-ASD 3.05 1.03 0.60 0.91 the type of conversation. Parent/Caregiver 2.38 0.96 0.47 0.82 Service Provider 2.70 0.78 0.73 0.91 F4. Start a conversation and keep it going. TAI-ASD 2.37 1.01 0.52 0.91 Parent/Caregiver 2.63 0.96 0.18 0.83 Service Provider 2.41 0.87 0.35 0.92 F5. When talking with coworkers and TAI-ASD 2.68 1.11 0.69 0.91 supervisors, I/they do not share things that are Parent/Caregiver 2.75 1.06 0.16 0.83 too personal. Service Provider 2.51 0.93 0.39 0.92 F6. When someone is talking to me/them, TAI-ASD 3.05 0.62 0.30 0.91 I/they listen without interrupting. Parent/Caregiver 2.44 1.03 0.91 0.79 Service Provider 2.24 0.70 0.52 0.92 F7. Use appropriate social manners (e.g., TAI-ASD 3.11 0.99 0.64 0.91 please and thank you). Parent/Caregiver 2.94 0.77 0.10 0.83 Service Provider 2.66 0.91 0.56 0.92 F8. Work well with others. TAI-ASD 2.74 0.87 0.56 0.91 Parent/Caregiver 2.93 0.93 0.62 0.81 Service Provider 2.63 0.86 0.59 0.92 156 Table F2 (cont’d) Item-Total Cronbach Alpha Frequency Scale Item Stakeholder M SD Correlation if Item Deleted F9. Show respect for others. TAI-ASD 3.63 0.68 0.44 0.91 Parent/Caregiver 3.13 0.89 0.61 0.81 Service Provider 3.02 0.72 0.70 0.91 F10. Are ok with people having different TAI-ASD 3.16 0.90 0.44 0.91 opinions. Parent/Caregiver 2.63 0.72 0.33 0.82 Service Provider 2.59 0.77 0.46 0.92 F11. Respond to feedback with a positive TAI-ASD 3.21 0.71 0.73 0.91 attitude. Parent/Caregiver 2.25 0.77 0.49 0.82 Service Provider 2.39 0.83 0.67 0.91 F12. Use different ways to speak up for TAI-ASD 2.47 1.02 0.69 0.91 myself/themselves depending on the situation. Parent/Caregiver 1.94 0.85 0.23 0.83 Service Provider 2.00 0.55 0.35 0.92 F13. Plan my/their time to get tasks done. TAI-ASD 2.63 0.96 0.31 0.92 Parent/Caregiver 2.19 0.91 0.28 0.83 Service Provider 2.17 0.86 0.65 0.91 F14. Shift my/their attention from one task to TAI-ASD 2.74 0.81 0.48 0.91 another when being asked. Parent/Caregiver 2.44 0.63 0.53 0.82 Service Provider 2.24 0.83 0.63 0.91 F15. Show flexibility when things change. TAI-ASD 2.37 0.90 0.49 0.91 Parent/Caregiver 2.19 0.75 0.68 0.81 Service Provider 2.22 0.96 0.51 0.92 F16. When I/they say they are going to do TAI-ASD 3.05 0.71 0.42 0.91 something, I/they follow through and do it. Parent/Caregiver 2.94 0.68 0.38 0.82 Service Provider 2.63 0.99 0.61 0.91 F17. Notice when there is a problem or TAI-ASD 2.58 0.90 0.51 0.91 conflict. Parent/Caregiver 2.19 0.54 -0.28 0.84 Service Provider 2.39 0.83 0.45 0.92 157 Table F2 (cont’d) Item-Total Cronbach Alpha Frequency Scale Item Stakeholder M SD Correlation if Item Deleted F18. Fix a problem or conflict. TAI-ASD 2.32 0.89 0.58 0.91 Parent/Caregiver 2.19 0.91 0.21 0.83 Service Provider 1.90 0.77 0.46 0.92 F19. Be in charge of my/their emotions. TAI-ASD 2.79 0.85 0.63 0.91 Parent/Caregiver 2.56 0.96 0.50 0.82 Service Provider 2.49 0.87 0.70 0.91 F20. Use different ways to cope with stress TAI-ASD 2.95 0.85 0.40 0.91 (e.g., take a break, deep breathing). Parent/Caregiver 2.25 0.68 0.59 0.82 Service Provider 2.17 0.77 0.64 0.91 F21. Take responsibility when I/they have TAI-ASD 3.58 0.69 0.40 0.91 made a mistake. Parent/Caregiver 2.31 0.79 0.27 0.83 Service Provider 2.24 0.94 0.66 0.91 F22. Show a positive attitude towards work. TAI-ASD 2.89 0.74 0.60 0.91 Parent/Caregiver 2.75 0.77 0.71 0.81 Service Provider 2.88 0.87 0.55 0.92 F23. Show confidence in my/their skills and TAI-ASD 2.74 0.87 0.77 0.91 abilities. Parent/Caregiver 2.94 0.85 -0.06 0.84 Service Provider 2.49 0.81 0.41 0.92 Note. TAI-ASD (n = 19), parent/caregiver (n = 16), service provider (n = 41). 158 APPENDIX G: EFA FACTOR LOADING COMPARISON TABLES Table G1 Confidence Scale 1-Factor Loadings Item # Confidence Scale Item Factor Loading 9. Show respect for others. .778 11. Respond to feedback with a positive attitude. .727 19. Be in charge of my/their emotions. .689 14. Shift my/their attention from one task to another when being asked. .664 6. When someone is talking to me/them, I/they listen without .652 interrupting. 20. Use different ways to cope with stress (e.g., take a break, deep .649 breathing). 21. Take responsibility when I/they have made a mistake. .640 15. Show flexibility when things change. .622 2. Use an appropriate voice volume based on the location and situation. .605 7. Use appropriate social manners (e.g., please and thank you). .587 13. Plan my/their time to get tasks done. .584 16. When I/they say they are going to do something, I/they follow .582 through and do it. 8. Work well with others. .567 22. Show a positive attitude towards work. .542 17. Notice when there is a problem or conflict. .541 18. Fix a problem or conflict. .521 12. Use different ways to speak up for myself/themselves depending on .490 the situation. 10. Are ok with people having different opinions. .488 23. Show confidence in my/their skills and abilities. .448 3. Use appropriate personal space based on the type of conversation. .445 5. When talking with coworkers and supervisors, I/they do not share .429 things that are too personal. 1. Have good hygiene (e.g., bathe & brush teeth to get ready for work). .410 4. Start a conversation and keep it going. .303 Note. Extraction Method: Principal Axis Factoring; 4 iterations required. 159 Table G2 Confidence Scale 2-Factor Loadings Item # Confidence Scale Item Factor 1 Factor 2 2. Use an appropriate voice volume based on the location .715 - and situation. 19. Be in charge of my/their emotions. .713 - 20. Use different ways to cope with stress (e.g., take a .648 - break, deep breathing). 13. Plan my/their time to get tasks done. .646 - 21. Take responsibility when I/they have made a mistake. .600 - 17. Notice when there is a problem or conflict. .592 - 12. Use different ways to speak up for myself/themselves .580 - depending on the situation. 6. When someone is talking to me/them, I/they listen .579 -.128 without interrupting. 3. Use appropriate personal space based on the type of .555 - conversation. 5. When talking with coworkers and supervisors, I/they do .506 - not share things that are too personal. 7. Use appropriate social manners (e.g., please and thank .476 -.165 you). 18. Fix a problem or conflict. .461 -.103 16. When I/they say they are going to do something, I/they .446 -.192 follow through and do it. 14. Shift my/their attention from one task to another when .422 -.314 being asked. 1. Have good hygiene (e.g., bathe & brush teeth to get .360 - ready for work). 22. Show a positive attitude towards work. -.130 -.804 8. Work well with others. - -.775 9. Show respect for others. .218 -.687 11. Respond to feedback with a positive attitude. .201 -.645 23. Show confidence in my/their skills and abilities. - -.547 15. Show flexibility when things change. .193 -.525 10. Are ok with people having different opinions. - -.523 4. Start a conversation and keep it going. - -.294 Note. Extraction Method: Principal Axis Factoring. Rotation Method: Oblimin with Kaiser Normalization; rotation converged in 5 iterations. 160 Table G3 Confidence Scale 3-Factor Loadings Item # Confidence Scale Item Factor 1 Factor 2 Factor 3 17. Notice when there is a problem or conflict. .747 - - 21. Take responsibility when I/they have made a mistake. .680 - - 12. Use different ways to speak up for myself/themselves .646 - - depending on the situation. 20. Use different ways to cope with stress (e.g., take a .507 - .222 break, deep breathing). 19. Be in charge of my/their emotions. .493 - .298 18. Fix a problem or conflict. .476 -.119 - 16. When I/they say they are going to do something, I/they .389 -.205 .118 follow through and do it. 13. Plan my/their time to get tasks done. .372 - .351 14. Shift my/their attention from one task to another when .357 -.333 .120 being asked. 22. Show a positive attitude towards work. -.171 -.803 - 8. Work well with others. - -.751 - 9. Show respect for others. .163 -.687 .103 11. Respond to feedback with a positive attitude. .179 -.649 - 15. Show flexibility when things change. - -.548 .184 23. Show confidence in my/their skills and abilities. .197 -.538 -.219 10. Are ok with people having different opinions. - -.531 .139 4. Start a conversation and keep it going. .118 -.286 - 2. Use an appropriate voice volume based on the location .100 - .797 and situation. 3. Use appropriate personal space based on the type of - - .603 conversation. 1. Have good hygiene (e.g., bathe & brush teeth to get -.106 -.138 .538 ready for work). 6. When someone is talking to me/them, I/they listen .141 -.185 .533 without interrupting. 5. When talking with coworkers and supervisors, I/they do .254 - .306 not share things that are too personal. 7. Use appropriate social manners (e.g., please and thank .257 -.200 .282 you). Note. Extraction Method: Principal Axis Factoring. Rotation Method: Oblimin with Kaiser Normalization; rotation converged in 10 iterations. 161