EFFECTS OF TEACHING ACCEPTANCE AND COMMITMENT THERAPY-BASED STRATEGIES ON IMPROVING ACADEMIC ENGAGEMENT AND PARTICIPATION FOR STUDENTS IN AN APPLIED BEHAVIOR ANALYSIS MASTER’S PROGRAM Michigan State University in partial fulfillment of the requirements for the degree of Applied Behavior Analysis -- Master of Arts 2021 Mikeya Renee Dunnigan By A THESIS Submitted to ABSTRACT EFFECTS OF TEACHING ACCEPTANCE AND COMMITMENT THERAPY-BASED STRATEGIES ON IMPROVING ACADEMIC ENGAGEMENT AND PARTICIPATION FOR STUDENTS IN AN APPLIED BEHAVIOR ANALYSIS MASTER’S PROGRAM By Mikeya Renee Dunnigan The field of applied behavior analysis (ABA) comes with rigorous work expectations and demands. ABA practitioners who are also graduate students must balance the extensive workload of educational and work expectations. Given these demands, graduate students providing ABA services to clients with autism spectrum disorder (ASD) need strategies to maintain their own positive mental health in order to maintain high academic achievement and to prevent burnout. Using a multiple baseline design this study examined whether teaching 6 graduate students enrolled in an ABA master's program who were also working as part-time practitioners providing ABA services to individuals with ASD to use Acceptance and Commitment Therapy (ACT)- based strategies impacted their engagement and active participation during an online synchronous class. Changes in psychological flexibility, stress, and values-behavior were also examined. Although there was no functional relation between the training on using ACT-based strategies and the dependent variables, the lack of effects is likely because the participants enrolled in the study did not have baseline levels that indicated a need for intervention. Additionally, it is hypothesized that competing factors and stressors related to the COVID-19 pandemic likely outweighed the impact of the training to use the ACT-based strategies. The results are discussed more in as well as implications for future research. Keywords: graduate students, behavior technician, Acceptance and Commitment Therapy, psychological flexibility, stress, value driven behavior TABLE OF CONTENTS LIST OF TABLES ................................................................................................................ v LIST OF FIGURES ............................................................................................................... vi KEY TO SYMBOLS AND ABBREVIATIONS .................................................................. vii Introduction ........................................................................................................................... 1 Acceptance and Commitment Therapy ........................................................................... 3 The Current Study ............................................................................................................ 7 Method ................................................................................................................................... 9 Setting and Participants .................................................................................................... 9 Materials ........................................................................................................................... 10 Dependent Variables and Response Measurement ........................................................... 10 Observed Variables…………………………………………………………..……..10 Engagement. ....................................................................................................... 10 Active Participation ........................................................................................... 12 Self-Report Measures ……………………………………………………………....13 Psychological Flexibility .................................................................................... 13 Stress .................................................................................................................. 13 Values Behavior ................................................................................................. 14 Social Validity ........................................................................................................... 14 Experimental Design ........................................................................................................ 15 Procedures ........................................................................................................................ 15 Pre-Surveys .............................................................................................................. 15 Baseline .................................................................................................................... 15 Training on Using ACT-Based Strategies ................................................................ 16 Post-Training Observations ...................................................................................... 18 Post-Surveys ............................................................................................................. 18 Interobserver Agreement .................................................................................................. 18 Procedural Integrity .......................................................................................................... 19 Data Analysis .................................................................................................................... 19 Results. .................................................................................................................................. 20 Observed Variables ......................................................................................................... 20 Engagement ............................................................................................................. 20 Group 1………………………………………………………………………...20 Group 2………………………………………………………………………...20 Group 3………………………………………………………………………...21 Active Participation .............................................................................................................. 23 Group 1 ………………………………………………………………………..23 Group 2………………………………………………………………….……..23 iii Group 3…………………………………………………………………….…..23 Self-Report Measures ........................................................................................................... 25 Psychological Flexibility .......................................................................................... 25 Stress ........................................................................................................................ 25 Values behavior ....................................................................................................... 25 Social Validity ....................................................................................................................... 29 Discussion. ............................................................................................................................. 30 Observed Variables ......................................................................................................... 30 Self-Report Measures ...................................................................................................... 32 Social Validity ................................................................................................................. 33 Limitations ........................................................................................................................ 34 Conclusion ........................................................................................................................ 37 APPENDICES ....................................................................................................................... 38 Appendix A ....................................................................................................................... 39 Appendix B ....................................................................................................................... 40 Appendix C ....................................................................................................................... 41 Appendix D ....................................................................................................................... 42 Appendix E ....................................................................................................................... 43 Appendix F ........................................................................................................................ 44 Appendix G ....................................................................................................................... 45 REFERENCES ...................................................................................................................... 46 iv LIST OF TABLES Table 1: Engagement Definition with Examples and Non-Examples ................................... 11 Table 2: The Active Participation Rating Scale as Developed by Instructor ........................ 12 Table 3: Total Scores for Participants on the Self-report Measures ...................................... 27 Table 4: Pre- and Post- Importance and Action Rating-VLQ Changes ................................ 28 Table 5: Mean Rating for Each Item of the Social Validity Questionnaire .......................... 29 v LIST OF FIGURES Figure 1: Percent of 5 min Intervals Each Participant was Engaged for Each Observation Period ..................................................................................................................................... ……22 Figure 2: Active Participation Scores for each Participant During Baseline and Post- Intervention… ........................................................................................................................ ……24 vi KEY TO SYMBOLS AND ABBREVIATIONS ACT Acceptance and Commitment Therapy ASD Autism spectrum disorder BCBA Board Certified Behavior Analyst DSP Direct Support Professional IOA Interobserver Agreement PF/PI Procedural Fidelity/Integrity vii Introduction Autism Spectrum Disorder (ASD) is a diagnosis used to describe persistent deficits in social communication and interactions, repetitive patterns of behavior, and social and occupational impairments on a varying scale of severity (American Psychiatric Association, 2013). Some of the primary characteristics of ASD include having socially unusual behaviors and mild to severe disruptive behaviors (Lord et al., 2018). Providing therapeutic services to individuals with ASD is a career that many find rewarding, though also very rigorous. Most evidence-based practices for the treatment of ASD are based on the principles of applied behavior analysis (ABA). ABA is “the science in which the tactics derived from the principles of behavior are applied systematically to improve socially significant behavior and experimentation is used to identify the variables responsible for behavior change” (Cooper et al., 2019, p 19). The discipline of ABA was distinguished from the experimental analysis of behavior by its focus on social impact (i.e., solving socially important problems in socially important settings) (Foxx, 2008). ABA, then, is a universal set of principles that can impact multiple fields through the use of systematic approaches to develop individualized treatments for a broad range of people and disabilities with the goal of increasing behaviors that can improve quality of life and decreasing harmful or maladaptive behaviors (Luoma & Vilardaga, 2013). According to Slocum and colleagues “ABA has produced remarkably powerful interventions in fields such as education, developmental disabilities and autism, clinical psychology, behavioral medicine, and organizational behavior management” (2014, p 42). ABA programming and interventions are designed and supervised by a board-certified behavior analyst (BCBA). Pursuing the BCBA certification is a rigorous educational and professional pursuit that not many take on and complete successfully (Foxx, 2008). First, 1 individuals must obtain a master's degree to be eligible to sit for the behavior analyst certification exam (BACB, 2021). Continuing education beyond a bachelor's degree can be difficult yet rewarding in any field of study; many individuals that pursue higher education find the few years of struggle leads to a better quality of life and numerous opportunities (Oswalt & Riddock, 2007). Additionally, individuals pursuing BCBA certification must also provide ABA services to clients and obtain the required fieldwork hours to be eligible to apply for certification (between 1500 and 2000 hours; BACB, 2021). Often, students choose to complete the coursework and obtain the fieldwork hours simultaneously. The combination of pursuing higher education and working with intense caseloads to meet the requirements to be eligible to take the BCBA exam can lead to student stress and lower psychological flexibility (Hyun et al., 2006). Due to the high intensity of the treatment that is needed for clients receiving ABA therapy, burnout amongst the professionals who provide ASD treatment has become an issue within the field (Pingo et al., 2019). Student stress and burnout may impact the quality of therapy they provide to clients and can lead to poor academic performance (American Health Care Association, 2010). The combined stress may cause students to lose sight of their values and engage in dysfunctional behaviors such as skipping class, bingeing (food, media, etc.), recreational drug use, and a variety of other behaviors to cope (Wilson & Groom, 2002). Strategies that reduce stress and promote psychological flexibility among students pursuing the requirements to become a BCBA, including taking master’s level courses and simultaneously working as a practitioner providing ABA services to individuals with ASD, are needed to improve quality of life during graduate education and to sustain careers long- term. 2 Acceptance and Commitment Therapy Acceptance and Commitment Therapy (ACT; Hayes et al., 2006) is a relatively new therapeutic intervention that seeks to increase psychological flexibility and promote values behavior. ACT is defined as “a psychological intervention based on modern behavioral psychology, including RFT [relational frame theory], that applies mindfulness and acceptance processes, and commitment and behavior change processes, to the creation of psychological flexibility” (Hayes et al., 2006; p 89). ACT is derived from behavioral psychology and includes behavioral principles by using language consistent with traditional behavior objectives associated with skill acquisition, goal setting, and skill training (Luoma et al., 2007). The main goal of ACT is to help participants increase psychological flexibility by developing their full awareness of conscious and unconscious thoughts (Harris, 2008). Within ACT, psychological flexibility is defined as “contacting the present moment fully as a conscious human being, and based on what the situation affords, changing or persisting in behavior in the service of chosen value” (Hayes et al., 2006, p 59). In more behavioral terms “psychological flexibility consists of engaging in skillful behavior in the presence of aversive experiences in the service of living a rich and meaningful life” (Tarbox et al, 2020, p 2). The main objective of ACT focuses on willingness to accept and change behavior through its six core principles: 1) value-driven behavior, including diffusion, 2) expansion, 3) connection, 4) the observing self, 5) values, and 6) committed action (Hayes et al., 2006). These core principles have objectives and exercises that build upon the acceptance and promotion of psychological flexibility. In addition to psychological flexibility, ACT can be a useful tool to help individuals promote their own goal-oriented behavior. ACT teaches that rather than trying to change or eliminate negative thoughts, individuals should work on changing the impact of thoughts by 3 changing the mental context in which they occur (Luoma et al., 2007). This method is exemplified through its illumination of values. Hayes and colleagues (2006) define values as “verbally constructed, global, desired, and chosen life directions that are defined by clients and what give life meaning” (p. 68). The identification of personal values helps the individual connect the changes in life, problems, and uncontrollable events to their personal values. This value centered mindset promotes positive outcomes as opposed to covering up negative thoughts (Harris, 2008). Thus, ACT is most effective when it is practiced by individuals in personal contexts and specifically developed to promote individual fluency and practice in its core components (Hayes et al., 2006). Finally, ACT can be used to increase values-behavior and decrease stress. Values- behavior is defined to “encompass intentional and effective methods that support and grow beliefs” (Harriss, 2008). Stress is defined as a natural physiological feeling that can be emotional or physical. Stress can develop in positive and negative ways based on an individuals' reaction to demands (Hayes et al., 2006). Despite the promise of ACT and its grounding in the principles of behavior analysis, little research has examined the effects of ACT in the behavior analytic field, particularly related to its effectiveness to relieve stress and improve psychological flexibility for ABA graduate students who also work as practitioners. Within the existing research, however, studies have examined the impact of ACT on other direct support professionals (DSP) and graduate students. First, Pingo and colleagues (2019) examined the effects of an ACT-based training for DSPs working with individuals with ASD in community and recreational settings. They evaluated change in DSP’s frequency and competency of treatment as well as their workplace stress and job satisfaction after a verbal and written feedback condition and then after a combined ACT-based training with 4 verbal and written feedback condition. After the feedback intervention was introduced, each participant demonstrated an increase in the amount of time they were engaged with active treatment and fidelity of treatment. Performance remained at high levels when the feedback and ACT-based training condition was implemented, indicating the addition of the ACT training condition did not improve performance beyond that of the feedback condition (e.g., there was no more than a 10% difference for each participant between the two phases). Overall, the study found improvements in engagement and treatment fidelity; however, the inclusion of the feedback condition does not allow for conclusions of the impact of the ACT-based training alone. Moyer and colleagues (2017) evaluated the usefulness of ACT-based interventions for psychology practitioner doctoral students. The authors implemented the ACT-based intervention as a part of a course requirement, teaching students to utilize ACT-based techniques and then evaluating changes personally and professionally in psychological flexibility, emotional dysregulation, and stress, while also examining performance on ACT-knowledge quizzes. The results indicated that students’ ACT-related knowledge increased from the beginning to the end of the course. Eight of the10 participants also displayed significant improvement in emotional dysregulation. Although all students improved in psychological flexibility and stress measures, these changes were not statistically significant; still the students collectively agreed that the ACT-based interventions were beneficial to their learning and professional development. This study provides an example of the potential an ACT-based intervention can have on graduate students and the development of their mental health and career. The researchers recommended future studies should explore the effects of ACT on psychological flexibility and stress 5 withinother professional fields alongside conducting research using a multiple baseline design to stagger intervention implementation among participants. Finally, similar to Moyer and colleagues (2017), Paliliunas and colleagues (2018) evaluated the effects of ACT on psychological flexibility, stress, values-behavior, and academic measures of midterm and final exam scores for students enrolled in a graduate level psychology course where ACT was incorporated into 6 weeks of the semester through values assignments. The class was split into two groups, control and intervention. The control group completed ACT vocabulary words and course materials, whereas the intervention group completed a series of values activities and ACT-related scenarios and questions that allowed them to work through the materials and apply it to their personal lives. Academic performance was assessed through 4 pre- quizzes about ACT principles and their relation to the course materials. Then a midterm exam and final exam scores were used to assess academic performance and changes in knowledge about ACT. Finally, self-report measures were used to assess changes in psychological flexibility, stress, and values-behavior. At the end of the study, a social validity survey was given to assess the students’ perceived usefulness of ACT. Findings show that 95% of participants who received the ACT-based intervention improved in psychological flexibility, stress, and values- behavior as well as academic performance, compared to the control group who reported no changes in the self-report measures. Many students from the intervention group also felt that the intervention was helpful to them as a student. The findings from this study indicate that an interactive ACT-based training can improve graduate student’s mental health in relation to psychological flexibility, stress, and values behavior. Overall, the limited research on ACT-based trainings for graduate students or practitioners indicate that ACT may be effective in improving psychological flexibility, 6 decreasing stress, and increasing values behavior. ACT may also improve academic and work performance and ACT-based knowledge. Although Pingo and colleagues (2019) were not able to conclude that ACT improved work performance beyond the feedback condition, this was one of the few studies to evaluate the effects of an ACT-based training in practitioner performance. Alternatively, Moyer and colleagues (2017) and Paliliunas and colleagues (2018) both incorporated the ACT-based training into the coursework for graduate students and reported improvements in psychological flexibility and values behavior, decreases in stress, and increased academic performance. The Current Study Graduate students who also work as practitioners in the field experience multiple types of stress. There are the usual stressors of higher education on top of stress that can come from working with clients with intensive behavioral needs. The students often work with clients during the day and then have classes in the evening. Both require heavy commitments and time. Students may be facing burnout after long shifts and may lack energy when schoolwork arises. This constant pull of the dual roles may lead to decreased academic engagement and participation and/or increased stressed and reduced of values-driven behavior. The stress of dual roles can have an impact on internal non-observable behaviors and external observable behaviors. Although ACT is a long and intensive therapeutic approach that might not be feasible to deliver to graduate students, the previous research has indicated that teaching students and practitioners to use ACT-based strategies can help relieve stress and improve their everyday academic and work performance. To expand the research on the effectiveness of using ACT-based strategies for graduate students and practitioners, the purpose of the current study was to examine the effects and impact 7 of teaching ACT-based strategies to graduate students who are also working as ABA practitioners. Specifically, this study sought to examine whether teaching the students to use ACT-based strategies improved engagement in class sessions and active participation during class lecture. Further, the study assessed changes in unobservable factors that may be impacted by life as a graduate student and practitioner, including psychological flexibility, stress, and values behavior. Extending the procedures described by Pingo and colleagues (2019) to practitioners who were also graduate students, the specific research questions were: 1) Does student engagement and active participation during online class lectures increase following a training on ACT-based strategies? 2) Do self-reported psychological flexibility, stress, and values behavior improve after a training on ACT-based strategies? and 3) Do participants' report that the training on ACT-based strategies is a socially valid and useful way to address psychological flexibility, stress, and values behavior? 8 Setting and Participants Method This study took place virtually through a large public university in a mid-western state with students enrolled in an ABA master’s program. Due to the COVID-19 pandemic all face-to- face classes and research activities were moved online and offered synchronously through the online conferencing platform Zoom. All students were required to complete the training on using ACT-based strategies and self-report questionnaires for course credit and were given the option to allow the data to be used for research purposes. All students agreed to allow their data to be used for research and provided their informed consent. This study included 6 participants who were first year students in a master’s in ABA program. All participants were enrolled in 2 content-focused ABA courses and 1 practicum course (total of 9 credit hours). Two of the courses met online through Zoom one time per week for at least 1.5 hours; the third course was offered asychronously. Students were also assigned to an ABA practicum site where they worked at least 4 hours a day to obtain at least 20 required fieldwork hours per week in a clinic, home, or work-based setting or through the delivery of telehealth therapy supporting children (ranging from toddler age to school age) with ASD or young adults (ages 18-26) with intellectual or developmental disabilities. Jhene was a 23-year- old White female who worked for an early intensive behavior intervention (EIBI) center for over 3 years. Mona was a 23-year-old White female who had worked for an EIBI center for less than 6 months. Lora was a 25-year-old White female who had worked at a university-based job skill development program for less than 6 months. Tori was a 22-year-old White female who had worked for an ABA therapy center for less than 6 months. Raynell was a 23-year-old White female who had worked at university-based job skill development program for less than 6 9 months. Amber was a 22-year-old White female who had worked for an ABA therapy center for less than 6 months. Materials The trainings took place over the Zoom platform with screenshare and utilized Google Classroom for written activities. Materials used for the training to teach the use of ACT-based strategies were PowerPoint presentations with talking points adapted from Pingo and colleagues (2019). Other study materials for the participants were posted to a Google Classroom, including activities and assignments corresponding with the training, as well as blank documents for taking notes. Homework assignments were assigned and turned in through Google Classroom. All participants and the researcher needed basic computer and internet access to participate. Materials used for data collection included the Engagement Time Chart (Appendix A) and a pen. The video recordings had time stamps embedded, which were used for the interval time sampling procedure. The materials for the self-report survey included computer access for the Qualtrics survey system where consent to participate, demographic information, the self- report questionnaires, and the social validity survey were completed. Dependent Variables and Response Measurement Observed Variables There were 2 observed dependent variables in the present study. The first variable, engagement, was used to examine students’ engagement in class sessions over Zoom. The second observed variable was active participation in class discussion, evaluating the number and quality of class contributions. Engagement. Engagement was defined as when the student 1) vocally responded to the lecturer or a peer, including responding to comments or asking on topic questions; 2) emitted 10 non-vocal reactions or used the reaction buttons in Zoom, including physical head or body movement in response to a comment or question from the instructor or a peer (e.g., head nod, laugh, thumbs up, etc.); or 3) had their camera on and was looking at the screen or was interacting with class materials (e.g., typing, writing notes, reading documents when instructed). Table 1 below describes examples and non-examples of engagement in class lecture. Table 1: Engagement Definition with Examples and Non-Examples Engagement Definition: When the student is responding to the lecturer or peer vocally, including responding to comments or asking questions on topic or physically, including head physical movement (head nods, thumbs up, etc.), uses reaction buttons in zoom, screen is on actively focusing on class materials (i.e., typing, writing notes, reading documents) Examples • Talking • Looking at the Screen • Physical movement (head nods, thumbs up, hand raise) • Using Reaction buttons in zoom • Using chat feature • Writing notes • Reviewing class materials • Presenting in class Non-Examples • Absent • Reading magazine • On cellphone • Camera turned off • Doing Laundry • Shopping • Sleeping • Playing with animals • Cooking Engagement was measured through three 5-min observations per class session using a 10- s whole interval time sampling procedure. Specifically, each class was approximately 90-min long, from 5:00pm to 6:30pm EST. Researchers divided each weekly class into 3 equal segments and randomly selected a 5-min time sample within each of the 3 segments to collect engagement data for every participant. For example, for the first class (i.e., the first three data points), the researcher coded minutes 15:00-20:00 from the first 30-min segment (5:00pm-5:30pm); for the 11 second 30-min segment (5:31pm-6:00pm), the researcher coded minutes 35:00-40:00, etc. Times selected for each week were consistent across participants. Engagement was indicated as a (+) for every interval in which the participant was actively engaged for the entire 10-s interval. The Engagement Time Chart can be found in Appendix A. Active Participation. Active participation was defined by the course instructor as the student’s meaningful contribution to class lecture through vocal responses (number of times they spoke). Each week, the instructor gave each student a score from 1 to 7 based on the student’s contribution to the class lecture that week (see Table 2 for the rating scale); the student received a score of 0 if they were absent from class. The course instructor who developed the participation rating was not a part of the research study and was independent of the research process. To reduce bias, the instructor was not aware of the order in which students completed the training on ACT-based strategies. After each class, the instructor shared the students’ participation ratings with the researchers for graphing and visual analysis. Table 2: The Active Participation Rating Scale as Developed by Instructor Select the score that best describes your participation during today`s class meeting. If you score within 1 pt of what the instructor gives you, you will receive a bonus point. Student absent from class Student did not talk during entire discussion (seemed to be falling asleep, too) 0 1 2 Student did not speak but seemed to be paying attention (nods, looks, avoids 3 Student spoke only when instructor asked them a question 4 Student spoke on their own but what they said did not promote discussion 5 Student spoke once and made a good point that contributed something to the 6 Student spoke more than once and made at least one good point 7 Student spoke more than once and made more than one good point distraction) discussion Note. This scale was created by course instructor and results were only reported to researchers. 12 Self-Report Measures The present study also examined change in the participants’ self-reported psychological flexibility, stress, and values behavior through pre- and post-intervention questionnaires. Psychological Flexibility. Psychological flexibility was measured using the Acceptance and Action Questionnaire-II (AAQ-II; Bond et al., 2011). The AAQ-II consisted of 7 questions related to past experiences and emotional factors related to feelings. Items were rated on a 7- point scale from 1 (never true) to 7 (always true). The AAQ-II is most valuable in evaluating issues, such as experiential avoidance and unwanted inner thoughts and events (Bond et al., 2011). The AAQ-II is scored by calculating the total score with higher scores indicating less psychological flexibility and lower scores indicating greater psychological flexibility. Change in scores from pre- to post-intervention were evaluated. See Appendix C for the AAQ-II survey and scale system. Stress. Factors related to stress were measured through the Perceived Stress Scale (PSS; Cohen et al., 1983). The PSS was designed to assess the frequency of how often certain feelings of stress occur in life, as well as their perceived level of interference in life. Ten items are rated on a 5-point scale from 0 (never) to 4 (very often) (see Appendix D). The PSS asks participants to look at their life and rate their specific feelings in the last month to evaluate how they perceive their current levels of stress. The scale is unique in the sense that it does not measure a lifetime of stress but specific time periods to allow for more accurate responding. PSS scores are obtained by reverse scoring (e.g., 0 = 4, 1 = 3, 2 = 2, 3 = 1, 4 = 0) the four positively stated items (items 4, 5, 7, & 8) and then summing across all items. Individual scores on the PSS can range from 0 to 40 with higher scores indicating higher perceived stress. The PSS was evaluated by examining the changes in scores from pre- to post-surveys. 13 Values Behavior. Values behavior was measured through the Valued Living Questionnaire (VLQ; Wilson & Groom, 2002). The VLQ consists of 2 parts. Participants first rate 12 domains for the general importance they have in their lives on a scale of 1 to 10, with 1 being not important at all and 10 being very important. Then they rate the same 12 domains related to how much the behaviors that they currently exhibit align with the domain on a scale of 1 to 10, with 1 being that your actions have been completely inconsistent with your values in this area and 10 being that your actions have been completely consistent with your values. The 12 values and domains include, 1) family (other than marriage or parenting); 2) marriage/couples/intimate relations, 3) parenting; 4) friends/social life; 5) work; 6) education/training; 7) recreation/fun; 8) spirituality; 9) citizenship/community life; 10) physical self-care (diet, exercise, sleep); 11) environmental issues; and 12) art, creative expression, and aesthetics. Participants respond based on their behavior within the last week. Scoring for the VLQ is completed by multiplying the two numbers from the first and second parts for each domain. Once all domains are calculated, the total of all applicable domains (not all domains were relevant to each participant) is added and then averaged by the applicable number of domains to get a composite score. The VLQ scores were evaluated by examining total and domain changes from pre- to post-surveys. See VLQ in Appendix E. Social Validity. The social validity survey was developed to assess the relevance and usefulness of the training on using ACT-based strategies for the participants. The survey was given post-intervention and consisted of 9 questions rated on a 6-point scale with a comment section for additional comments and suggestions. See Appendix G for the social validity survey. 14 Experimental Design This study was a multiple baseline design across participants. This design was chosen to control for outside school and job factors beyond the training on using ACT-based strategies that may have influenced student behaviors. The training on using ACT-based strategies was delivered in groups of two with participants assigned to group based on their availability. The design had three tiers of participants. The first tier included Jhene and Mona, the second tier included Lora and Tori, while the third tier included Amber and Raynell. Following standard multiple baseline design procedures, all groups entered baseline at the same time. When tier one entered intervention, the remaining two tiers remained in baseline. One week after the first tier completed the training on ACT-based strategies, the second tier entered intervention, with the third tier remaining in baseline. One week after the second tier completed the training on ACT- based strategies, the third tier entered intervention. Procedures Pre-Surveys Prior to baseline data collection or intervention, all participants were asked to complete the three surveys related to psychological flexibility, stress, and values behavior. Specifically, a link to the pre-treatment surveys was sent to all participants 1 week prior to the researcher conducting any observations. Participants were asked to complete the surveys during class and completion was part of their course grade. Baseline After all pre-treatment surveys were completed and before intervention, baseline data was collected for at least 3 weeks. Specifically, the instructor recorded each class session and then sent the recording to the researcher for coding after class was completed. The researcher then 15 divided the class session into 3 equal parts and randomly selected one 5-min segment from each of the 3 parts to code for engagement. Participants were not aware when the observation was being conducted or what variables were being measured. Observations were conducted once per week. Following each class session, the instructor also shared each student’s participation rating with the researcher for graphing and visual analysis. Training on Using ACT-Based Strategies The training on using ACT-based strategies was provided to two participants at a time. The training on using ACT-based strategies consisted of a 2-part virtual interactive workshop with each training conducted exactly 1 week apart; day1 was 2-hrs and day 2 was 1-hr. Group 1 trained on two Saturday mornings, while groups 2 and 3 trained on two Wednesday evenings. The training was adapted from an ACT-based training developed and shared with the researcher by John Pingo and colleagues at Southern Illinois University-Carbondale (2019). John Pingo shared the training presentation with the primary researcher and then two had email correspondence that explained the training and how to use it (Personal communication, 2020). The researcher then edited the training by consulting the texts The Happiness Trap (Harris, 2008) and ACT in Practice: Case Conceptualization in Acceptance & Commitment Therapy (Bach & Morgan, 2008) to include specific ACT metaphor examples related to the six principles of ACT, as well as interactive activities. The final training consisted of a 67-slide PowerPoint presentation accompanied with a script for each slide to maintain consistency across sessions. The structure of the training used the first day as a knowledge builder about ACT and its core principles, as well as identification of participant values. The training covered a basic history of ACT in ABA and its origin in Relational Frame Theory. The participants were then given homework assignments and 1 week to engage with the materials. The homework 16 assignments included a ping pong ball activity where participants identified important factors in their life. The second activity was a mindfulness activity where students logged instances of mindfulness throughout the week. The third activity was a values activity were the participants identified 3 of their top values and barriers that are in place to them fulfilling actions that align with those values. The homework assignments gave participants time to utilize skills learned in the training and to reflect on their values and behaviors. Values clarification is a major aspect of ACT, so allowing them to work through it was a good practice. The mindfulness activity promotes the utilization of diffusion and acceptance techniques. These activities strengthened the core lesson of the ACT principles by allowing participants to apply them in everyday life. During the second session, the participants discussed their experiences with the homework activities and then participated in a series of interactive activities about ACT. The participants were given real life examples where they could apply the skills they learned to practical scenarios that could happen in their practice as graduate students or ABA practitioners. The specific strategies were highlighted through scenarios that directly related to situations they could possibly encounter. The first scenario utilized psychological flexibility skills and involved an example of a first-year master's student. The second scenario utilized acceptance skills in the workplace and involved an example of new ABA practitioner. The third scenario utilized the principles of committed action and involved an example of an overwhelmed doctoral student. The last scenario discussed values clarification again in reference to the same doctoral student. These scenarios and discussions directly highlighted how ACT-based strategies can be used in their life and how they can help with daily challenges that arise. 17 Post-Training Observations Following the completion of the 2-part training on using ACT-based strategies, post- training observations of engagement were again conducted weekly to evaluate whether there was a functional relation between the training on using ACT-based strategies and increases in engagement. Observations were conducted the same as in baseline. No observations were conducted for any participants in the week between ACT workshops. As the instructor was unaware of the intervention timeline, they continued to provide participation scores to the researchers each week. Data collection continued for two weeks after all participants had completed the training. Post-Surveys The link to the post-treatment survey was sent to the participants at the end of the academic semester after all 6 participants completed the training. Similar to the pre-surveys, the instructor provided the link for the students to complete the surveys in class. Interobserver Agreement Interobserver agreement (IOA) was collected by a secondary researcher for 20% of observations across phases of the study and for every participant. The secondary researcher was trained by using practice segments of class sessions to code for engagement until both researchers met 100% agreement for every trial of the practice segments across 6 recordings. The researcher calculated point-by-point IOA by tallying the number of intervals both coders had in agreement over the total number of applicable intervals. Total IOA for the study was 97%. IOA for Jhene, Mona, Lora, Tori, and Raynell was 100% across all intervals in both baseline and post-intervention. IOA for Amber was 100% agreement of intervals in baseline and 95% agreement of intervals post-intervention. 18 Procedural Integrity To assess the primary researcher’s fidelity of the delivery of the training on using ACT- based strategies, procedural integrity (PI) was taken by secondary researcher. Specifically, the secondary researcher observed a recording of the training and used a checklist to mark whether each aspect of the training was conducted. The secondary researcher assessed PI for 2 of the 6 trainings on using ACT-based strategies and both sessions were scored 100% for completing all components of the training. The primary researcher also used the checklist to assess her own PI during each training. The primary researcher assessed procedural fidelity for 6 of 6 trainings and scored 100%. The PI checklist for the training on using ACT-based strategies is in Appendix F. Data Analysis Observed data was evaluated using visual analysis of the engagement and participation data. Because of the small sample size, statistical analyses were not conducted to evaluate change on the pre-post surveys. Rather, descriptive statistics were used to examine whether there were improvements in scores after the training on using ACT-based strategies. 19 Results Observed Variables Engagement Participant scores for engagement were averaged in two segments, the first score reflected baseline responding and the second score reflected post intervention responding. See Figure 1 for all engagement scores for each participant. Group 1. Jhene was engaged for an average of 86.3% (range: 63% to 100%) of intervals during baseline and was engaged for an average of 95.8% (range: 90% to 100%) of intervals across observations after intervention. In baseline, Jhene had a high level of engagement in the first three observations then her performance level significantly dropped beginning in observation four. For the remainder of baseline, her performance remained at the lower level and was moderately variable. After intervention, her engagement returned to a high and relatively stable level. During baseline, Mona was engaged for an average of 93.7% (range: 70% to 100%) of intervals across observations showing high variability. After the intervention, she was engaged for an average of 96.5% (range: 73% to 100%) of intervals across observations. The variability of her engagement decreased, and she maintained a high level of performance, excluding observations 16 and 17. Group 2. During baseline, Lora was engaged for an average of 97.6% (range: 93% to 100%) of intervals across observations and was engaged for an average of 77.3% (range: 0% to 100%) of intervals across observations after intervention. In baseline, Lora had high and stable levels of engagement. The first observation period after intervention, Lora was absent from class, resulting in 0% engagement for observations 13 to 15. After Lora returned, her engagement returned to the high level seen in baseline. During baseline, Tori was engaged for an average of 20 82.2% (range: 46% to 100%) of intervals across observations, and engagement was significantly variable. After intervention, she was engaged for an average of 80.7% (range: 13% to 100%) of intervals across observations, and engagement remained variable. Group 3. During baseline, Raynell was engaged for an average of 95.5% (range: 76% to 100%) of intervals across observations. Her engagement was at a high level with moderate variability. After intervention, she was engaged for an average of 98% (range: 86% to 100%) of intervals. After intervention, Raynell`s engagement remained at a high level and the variability in her engagement decreased. During baseline, Amber was engaged for an average of 45.1% (range: 0% to 100%) of intervals across observations with significant variability. After the intervention, she was engaged for an average of 18.7% (range: 0% to 80%) of intervals across observations. After intervention, her engagement initially improved with an immediate increase in level but then engagement returned to low levels. 21 Figure 1: Percent of 5 min Intervals Each Participant was Engaged for Each Observation Period 22 Active Participation Participant scores for active participation were averaged in two segments, the first score reflected baseline responding and the second score reflected post intervention responding. Group 1. Out of seven points, Jhene’s average participation score was 6.2 (range: 5 to 7) in baseline and 6.6 (range: 6 to 7) after intervention. In baseline, Jhene`s active participation had a decreasing trend. After the intervention, Jhene’s participation maintained at a high level with slight variability. Mona’s average participation score was 6.3 (range: 6 to 7) in baseline and 6.6 (range: 6 to 7) after intervention. During baseline, Mona`s active participation was at a high level and had slight variability. Post-intervention, her participation maintained a consistently high level with slight variability. Group 2. Lora’s average participation score was 6 (range: 6) in baseline with a high level and stable performance. After intervention, her active participation score was 5.2 (range: 0 to 6). Excluding the first session after intervention in which she was absent, her level of performance remained consistent with her baseline performance. In baseline, Tori ’s average participation score was 6.5 (range: 6 to 7) with little variability. After the intervention, her average performance was 7 (range: 7) with her performance maintaining a high and stable level. Group 3. Raynell's average participation score was 6.75 (range: 6 to 7) in baseline. Throughout baseline, her performance was high with low variability. After intervention, her average participation score was 7 (range: 7) with high performance with no variability. In baseline, Amber’s average participation score was 4.1 (range: 0 to 7) in baseline with her performance having high variability, due to absences. After intervention, her average participation score was 6 (range: 6) with stable high performance. 23 Figure 2: Active Participation Scores for each Participant During Baseline and Post-Intervention 24 Self-Report Measures Three self-report measures were given at the beginning and end of the study. Given the small sample size, statistical analyses were not conducted to evaluate change on the pre-post surveys. Rather, descriptive statistics were used to evaluate the impact of the training on using ACT-based strategies. Psychological Flexibility Three participants (Jhene, Mona, and Lora) increased their scores on psychological flexibility from pre- to post-intervention, indicating a worsening of psychological flexibility. The psychological flexibility scores of two participants (Raynell and Tori) decreased, indicating improved psychological flexibility. Amber’s post-survey score could not be calculated because not all questions were answered. See table 3 for each participant’s pre- and post-AAQ-II score. Stress Four participants (Jhene, Lora, Amber, and Raynell) reported an increase in perceived stress after intervention, whereas two participants (Mona and Tori) reported decreased stress after intervention. See table 3 for each participant’s pre- and post-PSS score. Values Behavior Jhene’s pre-survey score on the VBQ was 41.67 and the post-survey score was 46.8. She displayed improvements in the following domains after intervention: family, parenting, friends, education, and citizenship. Mona’s pre-survey score on the VBQ was 41.1 and the post-survey score was 41.92. Despite the relatively stable score, Mona had improvements in every domain post intervention, with the action's category increasing for: marriage, education and physical self-care. Lora’s pre-survey score was 32.58 and the post-survey was 35.6. Lora had improvements in parenting, art, and environmental issues. While her results indicate that she values those domains, little actions were reported with them. Tori’s pre-survey score was 67.42 25 and the post-survey score was 45.36. Tori had changes in the following domains: parenting, work, spirituality, citizenship, and recreation. Post-intervention, Tori showed an increase in the value of work and her actions increased as well. The domains parenting, spirituality, and citizenship showed increases in their value to her, but no increases in actions were reported. For her category recreation, she showed an increase in behavior after intervention. Raynell’s pre- survey score was 64.63 and the post-survey score was 65.6. Raynell had changes in recreation and spirituality domains. Raynell indicated post-intervention, that recreation became a higher priority, and she reported an increase of those behaviors. Her results also indicated that spirituality became more of a priority to her, but her actions did not increase. Amber’s pre- survey score was 28.67 and the post-survey score was 25.5. Amber had changes in the following domains: family, marriage, parenting, work, education, and environmental issues. Amber’s responses indicated that all the previously stated domains decreased in value to her, though the domains work and education increased in values-behavior. See Table 4 for a more detailed look at participant changes. 26 Table 3: Total Scores for Participants on the Self-report Measures PSS 7 14 19 26 13 27 17.67 (7.84) 16 15 20 25 29 27 21.40 (6.35) (4.49) Pre-Intervention Post Intervention AAQ-II 20 16 26 N/A 15 22 19.80 VLQ 41.67 41.1 32.58 28.67 64.63 67.42 46.01 (16.30) Participant Measure AAQ-II Jhene Mona Lora Amber Raynell Tori VLQ 46.8 41.92 35.6 25.5 65.6 45.36 43.46 Mean (13.35) Note. AAQ-II = Acceptance and Action Questionnaire–II; PSS = Perceived Stress Scale; VLQ = Valued Living Questionnaire. The AAQ-II measures psychological flexibility with scores ranging from 7 to 49 and higher scores indicating lower psychological flexibility. The PSS measures perceived stress with scores ranging from 0 to 40 and high scores indicating higher perceived stress. The VLQ is a measure of values-behavior with scores ranging from 12to 100 and higher scores indicating higher instances of values-behaviors. PSS 16 8 25 28 27 14 19.67 (8.17) 27 Table 4: Pre- and Post- Importance and Action Rating-Valued Living Questionnaire (VLQ) Changes Family Marriage Parenting Pre Post Pre Post Pre Post Pre Friends Pos t A I Pre 8 5 5 5 6 3 8 8 Pos t A 6 10 4 3 I 6 6 9 7 10 10 10 10 9 7 8 8 Physical Self-Care Post Pre I 5 6 8 9 8 8 A 2 8 7 3 5 5 I 5 10 7 10 9 8 A 4 10 7 4 4 3 Work Education Pre Post Pre Post I 6 7 9 8 10 10 A 10 9 9 5 10 9 I 6 2 9 8 10 10 A 10 10 10 7 10 10 Environment Post Pre A I 6 2 2 1 -- -- 4 2 A 2 2 2 2 I 6 2 7 6 8 -- 6 -- 10 2 7 1 I 6 9 9 9 10 10 A 9 10 9 4 10 10 Pre I 8 -- 5 5 8 -- A 3 -- 2 1 4 -- A 10 10 8 5 10 10 I 8 7 9 6 10 10 Art Post I 7 1 -- 5 A 6 7 -- 2 6 2 4 1 Participant Jhene Mona Lora Amber Raynell Tori Participant Jhene Mona Lora Amber Raynell Tori 8 1 0 I A 1 0 1 0 1 0 9 1 0 1 0 5 6 9 8 I A I A 9 7 1 0 8 9 1 0 6 1 0 5 4 1 0 7 10 8 8 7 10 8 7 6 1 6 1 0 8 I A 1 0 10 8 7 7 1 0 1 0 1 1 6 10 9 6 3 8 -- -- -- I A I -- 6 7 9 -- A -- 10 1 1 -- 10 10 1 -- -- Spirituality Pre Recreation Pre I A 4 8 3 8 6 8 7 3 Post I A 7 8 7 8 5 8 9 2 1 0 6 4 8 8 6 8 7 I 1 5 6 8 5 -- A 10 1 1 6 5 -- -- -- -- Citizenship Post A 4 5 1 1 Pre I A 5 8 1 10 1 6 8 3 I 2 6 5 5 Post I A -- -- 5 10 1 4 8 4 -- 2 -- 1 3 -- 4 -- 9 5 4 1 Note: I = Importance Rating; A = Action Rating 28 Social Validity See Table 5 for the average rating for each item on the social validity questionnaire. All the participants found the intervention somewhat acceptable. The participants all felt that the intervention was at an appropriate level for master’s level students. Most participants agreed that they would at least suggest the training on using ACT-based strategies to peers or colleagues, as well as use the strategies in an academic setting. Similarly, most participants agreed they would use the strategies in a practicum setting. All the participants agreed that the intervention was reasonable and most said that they liked the procedures. Overall, most participants found the intervention beneficial but better for a future time. No further comments or changes were suggested towards the intervention in the survey. Table 5: Mean Rating for Each Item of the Social Validity Questionnaire Statements The training on using ACT-based strategies was an acceptable intervention for Master`s level students. Most graduate students would find this intervention helpful to them. I would suggest the training on using ACT- based strategies to a peer or colleague. I would be willing to use the strategies taught in academic settings. I would be willing to use the strategies taught in work or practicum settings. This intervention was reasonable. I like the procedures used in the ACT-based training. Overall, the intervention was beneficial. N Min Max Mean (SD) 5 4 3 3 3 4 3 2 7 6 7 7 7 7 7 7 5.67 (.82) 5.17 (.75) 5.17 (1.47) 5.33 (1.37) 5.5 (1.52) 5.83 (1.17) 5.17 (1.33) 5.17 (1.72) 6 6 6 6 6 6 6 6 29 Discussion The purpose of the present study was to teach graduate students who also worked as ABA practitioners to use ACT-based strategies and to evaluate the impact of the training on their engagement and active participation in class, as well as their self-reported psychological flexibility, stress, and values behavior. ABA is an intense and rigorous field; working with clients who have high needs can lead to ABA practitioner stress and depleted work performance (Pingo et al., 2019). ABA practitioners who are also full-time graduate students in ABA often experience the pressure of graduate education on top of working with intensive clients (Foxx, 2008). Strategies that mitigate burnout and improve mental health are needed for students in general, and for those working with clients who have extensive support needs. Although no functional relation was demonstrated, implications for the results are discussed below. Observed Variables Overall, four of the six participants displayed high levels of engagement before intervention. Specifically, these four participants were engaged in over 85% of intervals during baseline. Three of the four participants with high baseline levels of engagement continued to display high levels of engagement after intervention and these levels stabilized to remain above 95% engagement across intervals. The fourth participant, Lora, was diagnosed with COVID-19 and missed class during the first post-intervention observation, resulting in engagement scores of 0. After that initial decline in engagement, Lora returned to high and stable responding for every class period in which she was present. It is clear from the data that these participants did not need to learn ACT-based strategies to improve their academic engagement and there was no functional relation between the training and their engagement during the online lecture. 30 Similarly, five out of the six participants displayed high levels of active participation before intervention and all participants scored 6 or above for active participation after intervention. Again, these data indicate there was little need for an intervention teaching ACT- based strategies to address active participation during an online course lecture. Two participants, Tori and Amber, displayed a different pattern of engagement and active participation. Although both had variable and low engagement during baseline and these levels decreased and became more variable post-intervention, Tori had high participation scores and Amber had low participation scores during baseline. Despite the training to use the ACT-based strategies, it appears that various competing factors likely impacted their engagement and participation, and these participants did not change their behavior after the training. For example, Tori was present for every class but was often multitasking and engaging with other things during the lecture. For example, she would often participate in the class lecture (resulting in a high participation score) but was also engaging with her pet, watching tv, using her cell phone, or sleeping during the lecture (resulting in a low engagement score). Amber, on the other hand, tested positive for COVID-19 and experienced multiple other difficulties with the virtual semester, resulting in her missing at least 3 class sessions and receiving engagement and participation scores of 0 for those nine observations during baseline. Post-intervention, Amber was present for class and participated (resulting an almost 2-point increase in participation scores) but was often multitasking (e.g., cleaning, doing laundry, and reading novels) when she was present in class (resulting in continuously low engagement scores). Thus, it appears that for Tori and Amber, competing distractions led to lower levels of engagement during online lectures, and training on the ACT-based strategies did not change this 31 behavior. Additionally, given that four out of six and five out of six participants already had high levels of engagement and participation, respectively, it is not possible to draw conclusions on the impacts of the training of using ACT-based strategies. Self-Report Measures Similar to the observed variables, this study found no significant improvements in psychological flexibility, stress, and values behavior from pre- to post-intervention. Although two of the participants had improved psychological flexibility, while 3 had worsened psychological flexibility, these changes were minimal and cannot be attributed to the training on ACT-based strategies. Similarly, two participants had decreased perceived stress post intervention and 4 of the 6 participants reported behaviors indicating they were more aligned with their values. The variable responding, however, makes it hard to attribute changes to the ACT-based intervention. More specifically, Jhene and Lora had increased scores in psychological flexibility and stress indicating they became more stressed and less flexible over the course of the study. Mona’s flexibility did not really change but she became more stressed. Amber reported slightly more stressed, which could have been related to missing class because of COVID-19. Finally, while Raynell became more flexible and more stressed, Tori became more flexible and less stressed. Given that the post-survey was administered at the end of the semester, the increase in stress scores may be attributed to the workload around final exams and it being the participant’s first semester of graduate school. These and other competing factors may indicate that a training on using ACT-based strategies could not help participants manage the competing stress they were experiencing. 32 The values behavior variable had relative improvements compared to the other self-report measures. Four of the 6 participants showed minor increases from pre- to post-survey scores, indicating their behaviors became more aligned with their values. The same two participants who had poor engagement variables (Amber and Tori), also did not report improvement their values- behavior; again, their behaviors were very different from one another. Amber’s drop in scores could have resulted from her health challenges and other stressors of work and school. She missed a lot of time in class due to COVID-19 and this could have possibly made her reassess her values and the actions she took towards them. Tori’s change in scores was a more significant drop which could possibly indicate that a significant change in life domains occurred throughout the study, or that she may have had additional external stressors that were not reported. Like the other participants, Tori was in her first semester of graduate school and working as a practitioner in an ABA clinic with children with ASD. The balance of this new life may have shifted her priorities. Even though the values behavior variable showed promising indications, no clear correlation between the ACT-based intervention and the measures could be made. Social Validity Finally, the results of the social validity survey indicated that although participants were generally satisfied with the training overall and would recommend it to peers and colleagues, they were not completely sold on the value of the training. The results indicate that the procedures in place were well-founded, but the impact of the training was not great for all participants. Future researchers could adapt the training to fit the specific needs of participants and the multiple competing factors taking place in the world. The methodology and implementation of the training were socially valid, but the perceived value was not consistent across all participants. 33 Limitations Despite the promise of ACT in previous research, there were several limitations that impacted the current study and its findings. First, the study design and participant selection were restricted because of competing factors related to the COVID-19 pandemic. Specifically, in March 2020 the University closed all in person operations, including moving all face-to-face courses to be offered online and pausing all in-person research. These restrictions were still in place during the Fall semester of 2020; as a result, the parameters for the current study changed, leading to a smaller participant pool and to only the availability of online data collection. Originally, the study intended to recruit from a pool of about 30 participants but was limited to the current six participants. As is evident from the baseline data of the observed variables, most of these six participants already had high levels of engagement and active participation and were not necessarily in need of a training on how to use the ACT-based strategies to improve academic performance. Future research should use baseline data to identify those participants who would benefit the most from learning ACT-based strategies. Additionally, because all classes were offered online and in person research was restricted, all data were collected from recorded class lectures that occurred over Zoom. This limited the ability to collect engagement data, as the engagement definition was not able to account for all behaviors that could not be seen off screen. For example, researchers were not able to control or identify if someone was scrolling through social media or doing other work unrelated to class. To address this limitation, the present study defined the engagement variable to be very specific to body movements and actions that could more readily be identified as actions that would occur if lectures were in person. Future research should examine engagement during in-person lectures to better measure this variable. It was also not possible to change the 34 participation variable, as this scoring system was determined from the course instructor’s requirements and graded by the instructor independently from the study. While the instructor had clear guidelines for participation within the course syllabus, the researchers had no control over the measure. As a result, the measure only addressed vocal contributions and did not account for specific actions of the students that could relate to active participation. A future study could look at incorporating more behaviors into the active participation definition. Second, the timing of the study may have impacted the potential benefits of learning to use ACT-based strategies. Given the requirements of the ABA master’s program, this study was required to be completed during the Fall semester. As a result, the participants were first year masters' students enrolled in their first ABA courses and (for most) just beginning work at their ABA practicum placement. The training on using ACT-based strategies was also a course requirement (with the option to allow data to be used for research). When they enrolled in the program, the students expected to attend courses in person. Thus, in addition to the natural stressors of attending the first courses of graduate school, transitioning to new practicum expectations, and adjusting to new coursework requirements, the students also experienced the added stressor of attending classes online and being isolated from their cohort members (to mitigate the spread of COVID-19). All of these factors may have impacted their scores on the observed and self-report measures. Further, due to scheduling, the post-survey was administered during finals week. Final's week is historically a stressful time for students as the workload can be heavy as deadlines approach. This is a naturally stressful time that was pressured by a global pandemic and virtual learning (Daniel, 2020) and the post-intervention stress scores may have been exacerbated because of this. The timing of these things was a major limitation of the current study; however, 35 time constraints for the class the participants were in and the time period the researcher had to complete the study prioritized the intervention in this semester. Third, some of the participants in the present study were unfortunately diagnosed with COVID-19 and faced health challenges alongside the expectation to maintain class and practicum work. These challenges also resulted in reduced engagement and active participation scores because they were absent from class. The physical and psychological barriers that come with illness are incomparable and must be respected as such. Even those who did not contract COVID-19 were likely impacted by the fear of contracting the illness and the mitigation strategies that were in place to slow the spread of the virus. The pandemic has been a very stressful time for the nation as a whole and for students as they navigate the new virtual world (Daniel, 2020). The new normal has impacted how educators, students, and staff work and learn the new expectations for school. The pandemic and student illnesses made the research with this small group of participants even more challenging because everyone was battling COVID-19 in some way. Finally, it is possible that the self-report measures were influenced by a social desirability bias, in that participants may have responded in ways to please the researcher or their instructor. When using self-report measures there is always a question of accurate responding that researchers cannot control. Participants were told throughout the study that the answers to the measures were not going to affect their grade or be shared with the instructor; however, researchers cannot be sure that this did not impact responses. Participants could have possibly responded a certain way because they wanted to avoid embarrassment or did not want to upset the course instructor or researcher. This is a natural problem that occurs when measuring internal 36 factors through self-report; there is a level of uncertainty of accuracy and bias that researchers must account for when interpreting results. Conclusion The results of the current study are not able to assume a functional relation between the training on using ACT-based strategies and graduate students’ engagement, active participation, psychological flexibility, stress, or values behavior. Even amid a global pandemic, students’ engagement and participation behaviors were relatively high before the training on using ACT- based strategies and they remained high after intervention. Future research should evaluate the effects of ACT under more “normal” conditions and with students who display low baseline levels of engagement and active participation to truly evaluate the impact of ACT for graduate students who are also working as ABA practitioners. Again, ABA practitioners who also serve as full-time graduate students often feel the pressure of graduate education on top of working with intensive clients. Although the current study did not find improvements in engagement in class sessions, active participation during lecture, or changes in psychological flexibility, stress, and values behavior after the training on using ACT-based strategies, this is still an important area to address, and future research is needed to identify ways to mitigate the challenges of being a graduate student and an ABA practitioner. 37 APPENDICES 38 APPENDIX A Engagement Time Chart Participant: _____________ Date: __________________ Session #: _______________ Coder: __________________ Definition: When the student is responding to the lecturer or peer vocally, including responding to comments or asking questions on topic or physically, including head physical movement (head nods, thumbs up, etc.), uses reaction buttons in zoom, screen is on actively focusing on class materials (i.e., typing, writing notes, reading documents) Whole Interval: Observe single student in video class for a total of 5 minutes. Mark (+) if the student was engaged in for the WHOLE interval and a (-) if student was not engaging at any point during the interval. Time stamp: Total Times Intervals Bx Occurred :10 :20 :30 :40 :50 1:00 1:10 1:20 1:30 1:40 1:50 2:00 2:10 2:20 2:30 Code: + or - 2:40 2:50 3:00 3:10 3:20 3:30 3:40 3:50 4:00 4:10 4:20 4:30 4:40 4:50 5:00 ____ 30 39 APPENDIX B Demographic Information: Name: Age: Race: Gender: Level of Education: Job Position: 40 APPENDIX C Acceptance and Action Questionnaire-II (Psychological Flexibility) Scale: 1- never true 2- very seldom true 3- seldom true 4- sometimes true 5- frequently true 6- almost always true 7- always true Questionnaire: Below you will find a list of statements. Please rate how true each statement is for you by selecting a number next to it. Use the scale below to make your choice. 1. My painful experiences and memories make it difficult for me to live a life that I would value 2. I'm afraid of my feelings 3. I worry about not being able to control my worries and feelings 4. My painful memories prevent me from having a fulfilling life 5. Emotions cause problems in my life 6. It seems like most people are handling their lives better than I am 7. Worries get in the way of my success 41 APPENDIX D Perceived Stress Scale (Stress) Scale: 0-Never 1-almost never 2- sometimes 3- fairly often 4- very often Questionnaire: The questions in this scale ask about your feelings and thoughts during the last month. In each case, you will be asked to indicate how often you felt or thought a certain way. Although some of the questions are similar, there are differences between them and you should treat each one as a separate question. The best approach is to answer fairly quickly. That is, don’t try to count up the number of times you felt a particular way; rather indicate the alternative that seems like a reasonable estimate. 1. In the last month, how often have you been upset because of something that happened unexpectedly? 2. In the last month, how often have you felt that you were unable to control the important things in your life? 3. In the last month, how often have you felt nervous and stressed? 4. In the last month, how often have you felt confident about your ability to handle your personal problems? 5. In the last month, how often have you felt that things were going your way? 6. In the last month, how often have you found that you could not cope with all the things that you had to do? 7. In the last month, how often have you been able to control irritations in your life? 8. In the last month, how often have you felt that you were on top of things? 9. In the last month, how often have you been angered because of things that happened that were outside of your control? 10. In the last month, how often have you felt difficulties were piling up so high that you could not overcome them? 42 APPENDIX E Valued Living Questionnaire (Values Behavior) Part 1: Below are domains of life that contain values for some people. We are concerned with your quality of life in each of these areas. One aspect of quality of life involves the importance one puts on different areas of living. Rate the importance of each area (by circling a number) on a scale of 1-10. 1 means that area is not at all important. 10 means that area is very important. Not everyone will have notable values all of these areas, or care about all areas the same. Rate each area according to your own personal sense of importance. Questions: 43 Education/training 1. Family (other than marriage or parenting) 2. Marriage/ Couples/intimate relations 3. Parenting 4. Friends/Social Life 5. Work 6. 7. Recreation/fun 8. Spirituality 9. Citizenship/community life 10. Physical self-care (diet, exercise, sleep) 11. Environmental issues 12. Art, creative expression, and aesthetics Family (other than marriage or parenting) Part 2: In this section, we would like you to give a rating of how consistent your actions have been with you values in each of these domains. We are not asking about your ideal in each area. We are also not asking what others think of you. Everyone does better in some areas than others. People also do better at sometimes than at others. We want to know how you think you have been doing during the past week. Rate each area (by circling a number) on a scale of 1-10. 1 means that your actions have been completely inconsistent with your values in this area. 10 means that your actions have been completely consistent with your values. Questions: (1-12 again) 13. 14. Marriage/ Couples/intimate relations 15. 16. 17. Work 18. 19. 20. 21. 22. 23. Art, creative expression, and aesthetics Education/training Recreation/fun Spirituality Citizenship/community life Physical self-care (diet, exercise, sleep) Environmental issues Parenting Friends/Social Life APPENDIX F ACT Training PI Checklist Date: ___________________ Session #: __________________ Coder: _____________________ The following PI form was used for primary researcher conducting the ACT based training. Codes: + behavior occurs, - behavior does not occur, N/A- not applicable Target: Day 1 Training: 1. Overview of training provided 2. ACT is defined 3. Psychological Flexibility is defined 4. Stress is discussed and reviewed 5. Fusion is discussed and reviewed 6. The ACT Model is reviewed 7. Present Moment Awareness is reviewed 8. Mindfulness is reviewed with activity 9. Acceptance is reviewed with activity 10. Diffusion is reviewed with activity 11. Perspective taking is reviewed with activity 12. Committed action and barriers to committed actions is reviewed 13. Values are reviewed with activity 14. Values are distinguished from goals 15. Conclusion 16. Questions 17. Homework is assigned Day 2 Training: 18. Review of Homework 19. ACT and ABA reviewed 20. Psychological flexibility and the workplace reviewed with example 21. Acceptance in the workplace reviewed with example 22. Committed Action in the workplace reviewed with example 23. Values in the workplace reviewed 24. Aligning personal values and workplace values reviewed 25. Discussion of workplace values with examples 26. General Discussion 27. Conclusion 28. Questions 29. Remarks Observation Notes: 44 Code: Total Applicable: Total Correct: APPENDIX G Social Validity Survey The Purpose of the present survey is to discuss the training on using ACT-based strategies intervention. Please rate the items below in reference to the training on using ACT-based strategies on a scale of 1-6 with 1 being strongly disagree and 6 being strongly agree. (Martens, Witt, Elliott, & Darveaux, 1985) Strongly Strongly Disagree Agree Questions: 1 2 3 4 5 6 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 4 3 4 3 4 3 4 3 4 3 4 3 4 5 5 5 5 5 5 5 6 6 6 6 6 6 6 1. The training on using ACT-based strategies was an acceptable intervention for Master`s level students. 2. Most graduate students would find this intervention helpful to them. 3. I would suggest the training on using ACT- based strategies to a peer or colleague. 4. I would be willing to use the strategies taught in academic settings. 5. I would be willing to use the strategies taught in work or practicum settings. 6. This intervention was reasonable. 7. I like the procedures used in the ACT-based training. 8. Overall, the intervention was beneficial. 9. Comments: 45 REFERENCES 46 REFERENCES American Health Care Association. (2010). Results of the 2009 AHCA survey of nursing staff vacancy and turnover in nursing homes. Health Services Research and Evaluation. http://publish.ahcatech.org/research_data/staffing/Documents/Vacancy_ Turnover_Survey2002.pdf American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). https://doi.org/10.1176/appi.books.9780890425596 BACB. Behavior Analysis Certification Board. (2021). Board Certified Behavior Analyst Handbook. Bond, F. W., Hayes, S. C., Baer, R. A., Carpenter, K. M., Guenole, N., Orcutt, H. K., Waltz, T., & Zettle, R. D. (2011). Preliminary psychometric properties of the acceptance and action questionnaire– II: a revised measure of psychological inflexibility and Experiential avoidance. Behavior Therapy, 42(4), 676–688. https://doi.org/10. 1016/j.beth.2011.03.007. Bowling, N. A., Wagner, S. H., & Beehr, T. A. (2017). The facet satisfaction scale: An affective measure of job satisfaction facets. Journal of Business Psychology, 33, 383- 403. Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 386–396. Cooper, J. O., Heron, T. E., & Heward, W. L. (2019). Applied Behavior Analysis (3rd Edition). Hoboken, NJ: Pearson Education. Daniel, S. J. (2020). Education and the COVID-19 pandemic. Prospects, 49, 91–96. Foxx, R.M. (2008). Applied Behavior Analysis treatment of autism: The state of the art. Child Harris, R. (2008). The Happiness Trap: How to stop struggling and start living. Shambala And Adolescent Psychiatric Centers of North America, 17, 821-834. Publication, Inc. Boulder, Co. Hayes, S. C., Luoma, J. B., Bond, F. W., Masuda, A., & Lillis, J. (2006). Acceptance and commitment therapy: Model, processes and out- comes. Behavior Research and Therapy, 44(1), 1–25. https://doi.org/10.1016/j.brat.2005.06.006. Hyun, J. K., Quinn, B. C., Madon, T., & Lustig, S. (2006). Graduate student mental health: needs assessment and utilization of counseling services. Journal of College Student Development, 47(3), 246–266. https://doi.org/10.1353/csd.2006.0030. 47 Kahn, R. L., Wolfe, D. M., Quinn, R. P., Snoek, J. D., & Rosenthal, R. A. (1964). Organizational stress: Studies in role of conflict and ambiguity. New York, NY: John Wiley. Lord, C., Elsabbagh, M., Baird, G., & Veenstra-Vanderweele, J. (2018). Autism Spectrum Disorder. Lancet, 392, 508-20. Luoma, J. B., Hayes, S. C., & Walser, R. D. (2007). Learning ACT: An acceptance & commitment therapy skills training manual for therapists. Oakland, CA: New Harbinger Publications. 549-552. Luoma, J. B. & Vilardaga, J. P. (2013). Improving therapist psychological flexibility while training acceptance and commitment therapy: A pilot study. Cognitive Behavior Therapy, 42 (1), 1-8. Moyer, D., Murrell, A., Conally, M., & Steinberg, S. (2017) Showing up for class: Training graduate students in acceptance and commitment therapy. Journal of Conceptual Behavior Science, 6(1), 114-118. Oswalt, S. B., & Riddock, C. C. (2007). What to do about being overwhelmed: graduate students, stress and university services. College Student Affairs Journal, 27(1), 24–43. Paliliunas, D., Belisle, J., & Dixon, M. R., (2018). A randomized control trial to evaluate the use of acceptance and commitment therapy (ACT) to increase academic performance and psychological flexibility in graduate students. Behavior Analysis in Practice, 11, 241- 253. Pfefferbaum, F. B. & North, C.S. (2020). Mental health and the COVID-19 pandemic. New England Journal of Medicine,383, 510-513. doi: 10.1056/NEJMp2008017 Pingo, J.C., Dixon, M.R., & Paliliunas, D. (2019). Intervention enhancing effects of Acceptance and Commitment Training on performance feedback for direct support professional work performance, stress, and job satisfaction. Behavior Analysis in Practice. 13(1), 1-10. https://doi.org/10.1007/s40617-019-00333-w Slocum, T.A., Detrich, R., Wilczynski, S. M., Spencer, T. D., & Lewis, T. The evidence-based practice of applied behavior analysis. Behavior Analyst, 37, 41–56. https://doi.org/101007/s40614-014-0005-2 Tarbox, J., Szabo, T.G. & Aclan, M. (2020). Acceptance and Commitment Training Within the Scope of Practice of Applied Behavior Analysis. Behavior Analysis Practice. https://doi.org/10.1007/s40617-020-00466-3 Wilson, K.G. & Groom, J. (2002). The valued living questionnaire. (Unpublished document, available from the author at the Department of Psychology). University of Mississippi, Oxford, MS. 48