THE USE OF EQUIVALENCE-BASED INSTRUCTION TO TEACH GRADUATE STUDENTS SIMPLIFIED DEFINITIONS OF BEHAVIOR ANALYTIC TERMINOLOGY By Grace Medlin A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Applied Behavior Analysis—Master of Arts 2022 ABSTRACT THE USE OF EQUIVALENCE-BASED INSTRUCTION TO TEACH GRADUATE STUDENTS SIMPLIFIED DEFINITIONS OF BEHAVIOR ANALYTIC TERMINOLOGY By Grace Medlin Individuals who are pursuing their Board-Certified Behavior Analyst (BCBA) certification are required to have an effective understanding of applied behavior analysis (ABA) terminology. Hemwell (2020) used Equivalence Based Instruction (EBI) and Match-to-Sample (MTS) training to teach technical definitions of ABA terminology to first-year ABA students. Participants displayed limited emergent intraverbal responding and maintenance. Hemwell (2020) speculated this was due to the length and complexity of the definitions. The current study extended on Hemwell (2020) to determine if using simplified definitions of ABA terminology was effective in promoting the acquisition, and maintenance of ABA terminology. Using a multiple probe across behaviors design replicated across participants, participants were exposed to 30 simplified ABA terms, along with definitions and examples via a MTS teaching method. Participants were assessed on their performance of untaught relations in a selection-based pre-post assessment, as well as through written intraverbal probes. Despite text simplification, emergent intraverbal responding was variable and did not maintain for all terms. However, emergent selection-based responding across all relations from pre to post test was evident. Following text simplification, participants displayed emergent selection based responding with more complex definitions from the Cooper et al. (2020) textbook. Overall, future instructors may consider the use of EBI and text simplification in their courses to teach ABA terminology, examples, and principles. Keywords: Equivalence-based instruction, Match-to-Sample, conditional discrimination, selection-based responding, topography-based responding, text simplification TABLE OF CONTENTS LIST OF TABLES ......................................................................................................................... iv LIST OF FIGURES ........................................................................................................................ v Introduction ..................................................................................................................................... 1 Equivalence-Based Instruction ................................................................................................. 1 Text Simplification ................................................................................................................... 5 Methods........................................................................................................................................... 8 Participants & Setting ............................................................................................................... 8 Materials & Experimental Stimuli ............................................................................................ 8 Response Measurement & Dependent Variables .................................................................... 10 Interobserver Agreement ........................................................................................................ 12 Experimental Design ............................................................................................................... 13 Procedures ............................................................................................................................... 14 Pretest...............................................................................................................................................................................................14 Intraverbal Probes ......................................................................................................................................................................14 Equivalence Based Instruction (EBI) Teaching Sessions ...........................................................................................16 Remedial Training (All Participants) ..................................................................................................................................17 Posttest .............................................................................................................................................................................................18 Procedural Fidelity .................................................................................................................. 18 Social Validity ........................................................................................................................ 19 Results ........................................................................................................................................... 20 Intraverbal Probes (BA and CA Relations) ............................................................................ 20 Andy...................................................................................................................................................................................................20 Alice ...................................................................................................................................................................................................22 Eliza ...................................................................................................................................................................................................24 Kelly ...................................................................................................................................................................................................26 Rachael .............................................................................................................................................................................................28 Julia ....................................................................................................................................................................................................30 Sarah..................................................................................................................................................................................................32 Pretest to Posttest Improvement (All Relations)..................................................................... 34 Generalization to Cooper et al. (2020) Definitions................................................................. 34 Discussion ..................................................................................................................................... 36 Text Simplification ................................................................................................................. 38 Limitations .............................................................................................................................. 40 APPENDIX ................................................................................................................................... 42 REFERENCES ............................................................................................................................. 54 iii LIST OF TABLES Table 1. List of Terms and Definitions for Stimulus Set 1 (Verbal Behavior) 43 Table 2. List of Terms and Examples for Stimulus Set 1 (Verbal Behavior) 44 Table 3. List of Terms and Definitions for Stimulus Set 2 (Generalization and Maintenance) 45 Table 4. List of Terms and Examples for Stimulus Set 2 (Generalization and Maintenance) 46 Table 5. List of Terms and Definitions for Stimulus Set 3 (Equivalence-Based Instruction) 48 Table 6. List of Terms and Examples for Stimulus Set 3 (Equivalence-Based Instruction) 49 Table 7. Results of Social Validity Survey by Participant 51 Table 8. Pretest and Posttest Data Final Scores, All Stimulus Sets 52 Table 9. Pretest and Posttest Data by AD Relation (Percent Correct) 52 Table 10. Pretest and Posttest Data by Relation for Stimulus Set 1 (Percent Correct) 52 Table 11. Pretest and Posttest Data by Relation for Stimulus Set 2 (Percent Correct) 53 Table 12. Pretest and Posttest Data by Relation for Stimulus Set 3 (Percent Correct) 53 iv LIST OF FIGURES Figure 1. Examples of Emergent Relations 2 Figure 2. Example of Equivalence Based Instruction Teaching Session (Google Slide) 12 Figure 3. Graphical Display of Andy’s Probe, EBI, and Remedial Training Data 21 Figure 4. Graphical Display of Alice’s Probe, EBI, and Remedial Training Data 23 Figure 5. Graphical Display of Eliza’s Probe, EBI, and Remedial Training Data 25 Figure 6. Graphical Display of Kelly’s Probe, EBI, and Remedial Training Data 27 Figure 7. Graphical Display of Rachael’s Probe, EBI, and Remedial Training Data 29 Figure 8. Graphical Display of Julia’s Probe, EBI, and Remedial Training Data 31 Figure 9. Graphical Display of Sarah’s Probe, EBI, and Remedial Training Data 33 v Introduction There are currently over 30,000 Board Certified Behavior Analysts (BCBAs) worldwide, whose role is to use the principles of applied behavior analysis (ABA) to provide effective treatment to children with autism and other disabilities, and support to their families (Behavior Analyst Certification Board, 2022; Carr & Nosik, 2016). Practicing behavior analysts must be board certified, which is obtained by completing required coursework, obtaining a minimum number of fieldwork hours, and passing the Behavior Analyst Certification Board (BACB) Exam. To be successful on the BACB exam, individuals must have the ability to comprehend and explain principles of ABA terminology, scenarios, and applied examples (Critchfield et al. 2017). Equivalence-Based Instruction Equivalence-Based Instruction (EBI) is a potentially quick and effective method to teach vocabulary terminology to those studying to be behavior analysts. EBI uses principles of stimulus equivalence to teach that two or more stimuli are the same, related, and/or functionally similar (Critchfield et al., 2018). EBI involves the principle of emergent relations in which some relations are taught and then the learner develops a new relational understanding without direct teaching (O’Neill & Rehfeldt, 2014). One method of using EBI is by dividing stimuli into different classes or categories and teaching certain relations between stimuli with the expectation other relations emerge (Fienup et al., 2011). For example, when teaching vocabulary terms, the categories may include 1) the vocabulary terms, 2) their definitions, and 3) examples of each word. Using conditional discrimination skills, individuals are likely to form relations between the stimuli. 1 Emergent relations typically develop from selection-based to topography-based (O’Neill & Rehfeldt, 2014); once individuals are taught to match stimuli with similar relations, they will then develop emergent topography-based (intraverbal responding) understanding of the relations. Examples of this relational framework are reflexivity, symmetry, and transitivity. An example of reflexivity would be if presented with a picture of a cup, and provided three comparison stimuli (e.g., individual pictures of cup, table, spoon), the learner would select the picture of the cup. Another emergent relation would be the selection of the word CUP upon seeing the presented stimuli (e.g., picture of a cup), establishing a relation between the two stimuli, this is referred to as symmetry (Cooper et al., 2020). Last, transitivity builds further in developing relations. For example, when presented with the written word CUP a learner would select from an array, a picture of a cup, and vocally identify the presented stimuli (e.g., “cup”). See Figure 1 for a visual example of the three relations. Figure 1. Examples of Emergent Relations Note. Reflexivity is a standard example of basic matching, so these two pictures of a cup are identical and match. Symmetry is where you would establish a relationship between the picture of a cup and say the written word cup. Transitivity builds even further on this. For example, the written word Cup is related to the picture of a cup and the vocal statement “cup”. Matching-to-Sample (MTS) is a typical approach within EBI used to teach these relations (Brodsky & Fienup, 2018). MTS involves the presentation of a sample stimulus (e.g., a term), and the selection of a related comparison stimulus in an array of two or more stimuli (e.g., a 2 definition that corresponds with the targeted term, and at least one other word that does not). For example, when presented with the sample stimulus “cup,” and the comparison stimuli are “cylinder drinking container,” “flat saucer for food,” or “circular hole w/ bottom for food,” the learner would select the comparison stimulus “cylinder drinking container.” Because MTS procedures require the individual to select a comparison stimulus based on the presented stimulus, this type of responding is referred to as selection-based responding (Polson & Parsons, 2000). Previous research studies have shown the effectiveness of using EBI to teach ABA topics to graduate students, such as functions of behavior and single subject designs. Albright et al. (2016) used computer-based MTS training to evaluate the emergence of selection-based and topography-based responding with ABA graduate students using a pretest-train-posttest design. Topography based responding involves the teaching of specific topography (e.g., intraverbal) based on a specific controlling variable (e.g., “Ready, set, _”; Polson & Parsons, 2000). Specifically, ten graduate students were taught four equivalence classes representing the four functions of behavior (e.g., attention, escape, access to tangibles, and automatic reinforcement). Albright & colleagues (2016) established stimuli within each class including a textual label (A), descriptions (B), graphical representations (C), and clinical vignettes (D). Using topography- based responding, the oral pretest assessed BA, CA, and DA relations; a selection-based multiple-choice pretest then assessed all relations, and a computer pretest assessed only the relations to be trained. Participants then completed MTS training on AB, AC, and AD relations, and completed tests of symmetry and transitivity. Following this training, participants completed posttests identical to the pretests. All participants in this study showed improvements from their pre- to posttest scores and demonstrated maintenance of knowledge two weeks after completing 3 the posttest. This study demonstrated the effectiveness of selection-based training on emergent topography-based responding; however, only four relevant ABA topics were taught all within a single day. Walker & Rehfeldt (2012) used EBI to teach distance learning graduate students single subject designs using a pretest-train-posttest-maintenance design. This study consisted of topography-based tact and intraverbal pretests assessing whether participants could name the correct design when presented with the definition (BA), graph (CA), and clinical vignette (DA). Following the pretest, participants completed selection-based intraverbal instruction where they were taught AB, AC, and BD relations using multiple choice questions. Last, participants completed the posttest, identical to the pretest, and a test at 16-weeks to measure maintenance. Performance varied by participant; however, all participants demonstrated the emergence of CA tact relations following AC intraverbal instruction. All participants improved in scores from the pre- to posttest. However, it is important to note one limitation of this study is that participants were only exposed to one example and therefore did not demonstrate generalization to novel clinical vignettes. Finally, Hemwell (2020) used EBI to prepare individuals pursuing their BCBA® certification for the BACB® exam and careers in the field of ABA by improving their understanding of key behavior analytic concepts. Hemwell (2020) assessed the effectiveness of EBI on the acquisition of knowledge of behavior terminology and examples using intraverbal probes and MTS procedures through a pretest-train-posttest design. Stimulus sets taught to participants were based on terms in three chapters of the Cooper et al. (2020) textbook, Applied Behavior Analysis, 5th edition. Each stimulus set consisted of 10 terms from each of the following chapters: 18 (“Verbal Behavior”), 19 (“Equivalence-Based Instruction”), and 30 4 (“Generalization and Maintenance of Behavior Change”), for a total of 30 terms. “A” stimuli were the MTS stimuli, or the terms. “B” stimuli were the definitions of the terms taken verbatim from the Cooper et al. (2020) textbook. “C” stimuli were examples of each term developed by the experimenter. Participants completed a selection-based responding 180 question pre-test that assessed knowledge of all relations of stimuli. Next, participants completed topography-based responding through intraverbal probes, consisting of 60 open-ended questions with 20 questions per stimulus set. Intraverbal probes were completed prior to teaching stimulus sets (e.g., 3 per participant) as a baseline measure. Participants were then assigned to complete EBI training for the first stimulus set. Following mastery of a stimulus set, participants completed intraverbal probes again. If a participant did not pass an intraverbal probe by meeting the 80% correctness mastery criteria, they completed remedial training (e.g., additional EBI training until they were able to pass the intraverbal probe). This sequence of training and assessment was repeated for all three stimulus sets. Once training on all sets was complete, the participants completed a posttest, identical to the pretest. Results of Hemwell’s study show that EBI produced emergent selection- based responding, but EBI training did not result in participants reaching mastery criteria for intraverbal responding. Hemwell hypothesized that the reason for these results was due to the length and difficulty of the definitions that were taught in her study and stated further research should be conducted to investigate this limitation. Text Simplification The overarching goal of text simplification is to rewrite complex text into simpler language that is easier to understand for most individuals (Xu et al., 2015). Studies have shown that text simplification is an effective method to increase reading comprehension and understanding of material. Several studies have been conducted to evaluate the methods used for 5 automatic text simplification and how text difficulty impacts comprehension (Siddharthan, 2014). For example, Xu et al. (2015) evaluated effective methods for measuring and developing simplified text, specifically looking at Wikipedia. Results of the study showed a variety of components to take into consideration when simplifying text. Specifically, individuals must consider the target audience and who the text is for when simplifying text. In addition, using elements such as paraphrasing and splitting to simplify text are effective methods. Still, Siddharthan (2014) discussed the need for further research on how examples of simplified language generalize to the real world, concluding that more systematic evaluations are needed that measure text comprehension by end users. As previous studies have shown that simplifying definitions may help with comprehension of material, the current study aimed to replicate and extend Hemwell (2020) to determine if using simplified definitions of ABA terminology was effective in promoting the acquisition, understanding, and maintenance of ABA terminology. Specifically, the present study assessed whether simplifying ABA definitions and terminology impacted results and maintenance of emergent responding within intraverbal probes and multiple-choice responding. The specific research questions were: 1) Does EBI training with simplified definitions lead to acquisition of selection-based responding for AB and AC relations? 2) To what extent does EBI training with simplified definitions impact emergent written topography-based responses (i.e., intraverbal probes) for BA and CA relations? Does this performance maintain over time? 6 3) To what extent does EBI training with simplified definitions impact emergent selection- based responses with behavior terminology (AB, AC relations), definitions (BA, BC), and examples (CA, CB)? 4) Does performance with simplified behavior terminology, definitions, and examples generalize to the definitions provided by Cooper et al. (2020) (AD relations)? 7 Methods Participants & Setting Seven students in their first semester of a hybrid Master of Arts in ABA program located in the Midwest participated in the current study. Participants' ages ranged from 22 years-old to 25 years-old and all spoke English as their first language. Participants’ prior experience with and training in ABA varied. Andy was a 24-year-old White male with a Bachelor of Arts (BA) in psychology and no ABA experience prior to the study. Alice was a 23-year-old White female with a BA in psychology and 1- 2 years of ABA experience working as a behavior technician prior to the study. Eliza was a 25-year-old White female with a Bachelor of Science (BS) in behavioral neuroscience and 1 - 2 years of ABA experience working as a Registered Behavior Technician (RBT). Kelly was a 24-year-old White female with a BS in kinesiology and no ABA experience prior to the study. Rachael was a 22-year-old White female with a BA in psychology; she had 1 - 2 years of ABA experience as a behavior technician and had taken ABA courses prior to the study. Julia was a 22-year-old female with a BS in kinesiology and 1 - 2 years of ABA experience as a behavior technician; she was actively studying for RBT certification prior to the study. Sarah was a 22-year-old White female with a BA in psychology, and 1 - 2 years of ABA experience as a behavior technician. All of the intraverbal probe sessions, pretest, and posttest took place online via Desire2Learn® (D2L®), and all teaching sessions occurred online via Google Slides . TM Materials & Experimental Stimuli Participants were required to have access to a computer with internet and a webcam; however, if a participant did not have a reliable computer with internet and a webcam, they were loaned a laptop for the duration of the study. Throughout the experiment participants used D2L® 8 and Google Slides to complete the pretests, teaching sessions, posttests, and intraverbal probes TM (described in more detail below). The link to the appropriate teaching sessions was sent to participants via university email once the appropriate intraverbal probe was completed. The intraverbal probe tests, located on D2L®, consisted of fill-in-the-blank questions. The teaching sessions, pretests, and posttests consisted of an MTS teaching format. When completing the intraverbal probes, pretest, and posttest, participants were required to use Respondus Lockdown Browser® and Respondus Monitor® features within D2L®. The Respondus Lockdown Browser® is a browser that prevents individuals from accessing outside resources from the internet browser during testing. Additionally, Respondus Monitor® uses the individuals’ webcams and requires them to show videos of their surroundings (Respondus LockDown Browser & Monitor, 2020). The experimental stimuli consisted of the same three stimulus sets of behavior analytic vocabulary terms used by Hemwell (2020). Each stimulus set contained 10 terms each from the following chapters in the textbook Applied Behavior Analysis (Cooper et al., 2020): 18 (“Verbal Behavior”), 19 (“Equivalence-Based Instruction”), and 30 (“Generalization and Maintenance of Behavior Change”), for a total of 30 terms. The same examples for each term, developed by Hemwell (2020), were also used. Extending Hemwell (2020), a simplified definition of each ABA term (operationally defined as a statement of 30 words or less that describes the term without the use of jargon) was developed. To simplify each definition, the experimenter first consulted various introductory to behavior analysis and verbal behavior textbooks to identify an alternative definition for each term. She then compiled the terms and definitions into a survey for behavior analysis faculty, BCBAs, and trainees and asked respondents how they would simplify the definition for someone 9 not familiar with ABA. Responses to the surveys were used to create new definitions for each term, with the goal of making the term easier to understand while also maintaining the specific components that were considered necessary to keep the definition accurate. Upon reviewing the new definitions, it was determined the definitions were still too complex, containing various ABA principles and jargon. After further discussion, the experimenter removed all ABA jargon and further shortened each definition. The resulting simplified definitions were all 5 to 29 words in length and did not contain jargon. All final definitions were reviewed by two BCBAs and considered acceptable for introductory ABA learners. A list of the terms, definitions, and examples used for each stimulus set can be found in Tables 1-5. “A” stimuli were the MTS stimuli, or the terms. “B” stimuli were simplified definitions of the identified terms from the textbook. “C” stimuli were examples of each term developed by the experimenters. “D” stimuli were Cooper et al. (2020) definitions measured in the pre and posttests. Response Measurement & Dependent Variables Pretest, intraverbal probe, posttest, and teaching data were collected and scored as a percentage of correct responses. Correct responding during the intraverbal probes was defined as the participant typing in the term that corresponded to the definition or example provided. Correct responses had to include at least 75% of the words contained within the term. For example, if the definition of “higher order operant class” was provided and the participant typed “higher order class”, this was scored as correct. However, if the definition of “response generalization” was provided and the participant typed “generalization,” this was scored as incorrect. Responses were also scored as correct if they included a synonym of one of the words in the concept that did not change the meaning of the word. For example, when presented with 10 the definition of naturally existing contingency, the response “naturally occurring contingency” was scored as correct. Responses were also scored as correct regardless of whether the participant typed the plural or singular word, and regardless of tense or part of speech. For example, when presented with an example of a tact, “tact,” “tacts,” or “tacting” were all scored as correct. Minor spelling errors were also scored as correct. For example, if a participant typed, “transititivity” instead of “transitivity,” this answer was scored as correct. Correct responding during the pretests and posttests was defined as selecting the term, definition, or the example (depending on the relation being trained or tested) that corresponded to the sample stimulus presented at the top of the screen. Correct responding during the EBI teaching sessions was defined as selecting the comparison stimulus that corresponded to the sample stimulus presented at the top of the screen by using a mouse pointer to click on the correct definition or example. For example, during MTS AB relation training, if the word “echoic” was presented as the sample stimulus, the correct response would be selecting the stimulus that displayed the definition of the word “echoic.” See Figure 2 for an example of an MTS AB relation. 11 Figure 2. Example of Equivalence Based Instruction Teaching Session (Google Slide) Note. An example of what the participant views during an EBI teaching session; Match-to- sample (MTS) for AB relation (term-definition). Interobserver Agreement Interobserver agreement (IOA) was assessed for scoring on the EBI teaching sessions and the intraverbal probes. To assess for IOA, a second-year master’s student used a scoring key to score 30% of each participants’ EBI teaching sessions and intraverbal probes. Specifically, to assess IOA for EBI teaching sessions, the second observer watched a recording of the EBI teaching session and marked the response selected by the participant for each question. The responses recorded by the second observer were compared to those recorded by the experimenter. An agreement was counted as when the experimenter and second observer indicated the same response; a disagreement was counted when the experimenter and second observer indicated different responses were selected. IOA was calculated by dividing agreements by agreements plus disagreements multiplied by 100 to yield a percentage. To assess IOA for intraverbal probes the second observer accessed an intraverbal probe via D2L, and scored the response entered by the participant as correct or incorrect. The responses recorded by the second 12 observer were compared to those recorded by the experimenter. An agreement was counted as when the experimenter and second observer indicated the same response; a disagreement was counted when the experimenter and second observer indicated different responses were selected. Total count IOA was calculated as the number of correct responses calculated by the experimenter divided by the number of correct responses calculated by the second observer multiplied by 100 to yield a percentage (Cooper et al., 2020). IOA for all participants’ EBI performance on stimulus sets 1 was 100%, while for stimulus set 2 it was 99%. IOA for MTS performance on stimulus set 3 was 95% (range, 95% to 100%) across all participants. IOA for all participants except Julia’s intraverbal probe performance on all stimulus sets was 100%. IOA for Julia’s intraverbal probe performance was 99%. Finally, because D2L® automatically scored the pretest and posttest, IOA was not calculated. Experimental Design A multiple probe design with probe conditions (Ledford & Gast, 2018) across stimulus sets was replicated across participants and embedded in a pretest-train-posttest design to evaluate the effectiveness of the EBI on the acquisition of simplified definitions and examples of behavior analytic vocabulary. This design was appropriate for the current study because the skills taught were unlikely to emerge during baseline in the absence of direct instruction. All participants first completed a pretest and then started in baseline for all three stimulus sets until they demonstrated stable responding. Then participants completed EBI teaching sessions (described below) for an assigned stimulus set until they reached mastery criterion. Next, participants completed an intraverbal probe. If they demonstrated mastery, they moved on to EBI teaching for the next assigned stimulus set. If they did not demonstrate mastery criteria, they completed remedial 13 training (described below). Participants followed this format until they completed EBI and intraverbal probes for each stimulus set, then finally took a posttest. Hemwell (2020) had all participants complete each stimulus set in the order in which the content was introduced in Cooper et al. (2020). However, to control for level of difficulty within the stimulus sets, the present study assigned stimulus sets randomly across participants. Procedures Pretest Prior to any training, a pretest was administered to assess baseline knowledge of each relation and to be used as a comparison to test emergent symmetrical and transitive relations on the posttest. Each participant completed a 210-question multiple choice pretest that assessed all relations of stimuli (AD, AB, AC, CA, BA, BC, CB) with no time limit. Participants were given instructions to select the option that corresponded to the term, definition, or example presented and they were told to guess if unsure of the answer. The examples used in the pretest were novel, meaning they were different from the ones used in the EBI teaching sessions and intraverbal probes. D2L® was programmed to randomize the order of questions for each participant. No feedback was provided for correct or incorrect answers during the pretest. The test format consisted of the target word, definition, or example as the question and four answer choices listed below the question. Each stimulus sets’ terms, definitions, and examples were kept separate from each other; when an individual was given a stimulus from stimulus set 1 they were only given answer options that related to stimulus set 1. Intraverbal Probes Each participant completed three initial intraverbal probes following the pretest and prior to EBI teaching sessions. Participants also completed intraverbal probes following mastery of 14 each stimulus set in the EBI teaching sessions and remedial training sessions. This was done to ensure experimental control by showing that change in scores only occurred when the intervention was applied (Ledford & Gast, 2018). Each intraverbal probe consisted of 60 fill-in- the-blank and/or open-ended questions—20 questions per each of the three stimulus sets. There were two different intraverbal probes, “Intraverbal probe A” and “Intraverbal probe B,” which each had the exact same format and definitions but included one of the two examples used in the EBI teaching sessions (see Tables 4, 5, and 6). The order of intraverbal probes (A or B) completed by participants was randomized. Each intraverbal probe was taken on a different day and on a different day from the pretest. Data were only collected on BA and CA relations during intraverbal probes due to the nature of the intraverbal test format. All intraverbal probe conditions were identical. Intraverbal probe questions included either the definition or example presented at the top of the screen and one box below for the participant to write in the corresponding term. The participants were instructed to type the term that corresponded to the example or definition presented. If they did not know the answer, they were instructed to type “I don’t know”, “Idk”, or some other indicator that showed they attempted to answer the question. Participants had unlimited time to complete each intraverbal probe but were required to use the lockdown browser when completing the intraverbal probe. After completing the last question on the probe, the participants clicked “Submit” and results were saved for the experimenter to score. Mastery criteria on the intraverbal probes were set to 80% correct responding or higher on both BA and CA relations for the corresponding stimulus set. No feedback was provided to participants concerning their performance on the intraverbal probes or which questions they answered correctly or incorrectly 15 If the participant did not pass the probe, the experimenter sent an email instructing them to complete remedial training (described below). The email also included brief feedback stating that responses are expected to include the full term. Following remedial training, participants completed the alternate intraverbal probe. If participants did not pass the second probe for the same stimulus set, they were provided with additional feedback (e.g., slow down during remedial training, and to read through each question and answer). Following remedial training, the participant completed a third probe. Equivalence Based Instruction (EBI) Teaching Sessions During EBI teaching sessions, participants were taught AB and AC relations via MTS procedures. Teaching sessions consisted of 40 questions total per stimulus set. The first 10 teaching trials were term-definition (AB) relations. The last 10 teaching trials were term-example (AC) relations, and the training included two different examples across the two AC relation sections. EBI teaching sessions were delivered via Google Slides . Prior to the session, TM participants were given the instructions via email to screen-record their participation either in presentation mode or over Zoom for researchers to score and determine when they reached mastery criterion. The training also included an introductory slide describing the general procedure of the EBI teaching sessions, including information about the reinforcement slides and error correction procedure. The EBI target stimuli (A = term) were presented simultaneously with three comparison stimuli below. The instructions for the EBI teaching sessions were to select the definition or example that corresponded to the term presented at the top of the screen. Correct selections were hyperlinked to a slide with a visual that read “CORRECT!” in green font. Incorrect responses were hyperlinked to a visual that read “INCORRECT” in red font and 16 resulted in the delivery of the error correction procedure where participants were presented with the same question again with the correct answer prompted by an arrow above the term. Participants would then select the prompted answer that was hyperlinked to a screen that read: “Correct.” in grey font. Next, the same term was presented one more time without the arrow to transfer stimulus control from the prompt to the term. Correct selections were then hyperlinked to a slide that said “Correct,” in green font. Any errors during the error correction procedure started the procedure over from the initial “INCORRECT” in red font feedback slide. In order to continue after feedback, participants selected a button that read “Next,” which was hyperlinked to either the next term or the next slide in the error correction procedure depending on whether the participant selected the correct stimulus. Participants were then instructed to either upload the video of their screen recording to D2L®, or to provide a mediaspace link of the recording via email. The experimenter then watched the recorded video and scored participants’ performance, marking whether each selection was correct or incorrect. Mastery criterion was set to 90% correct responding or higher across three consecutive teaching sessions for each stimulus set. The experimenter emailed participants once they reached mastery criterion and instructed them on which intraverbal probe to take located in D2L®. An identical procedure was followed for all stimulus sets. The assignment of stimulus sets was randomized for each participant. With the goal to measure the complexity of terminology within each stimulus set based on EBI teaching session performance. Remedial Training (All Participants) If participants scored below an 80% on either BA or CA relations on the intraverbal probe for the current stimulus set, they were instructed to complete remedial training. The first round of remedial training for all stimulus sets consisted of three additional EBI teaching 17 sessions, followed by an email with feedback to be sure to write down the full term for every response during the intraverbal probe. Due to time constraints, participants were allowed to complete up to two remedial training sessions a day. Intraverbal probes could not be taken on the same day a participant completed training(s); however, after completion of an intraverbal probe, participants were allowed to complete the next assigned training on the same day. If participants did not reach mastery criteria on the probes following remedial training, they were assigned to complete two additional remedial training sessions; these could both occur on the same day. Prior to the completing the two additional remedial training sessions, the experimenter stated for the participants to slow down when completing the training. Following this, if mastery criteria was still not met, no additional remedial training was provided, and participants moved on to the EBI teaching sessions for the next randomized stimulus set. Posttest Each participant completed a 210-question multiple choice posttest that tested all relations of stimuli (AD, AB, AC, CA, BA, BC, CB) following completion of teaching sessions and intraverbal probes of all three stimulus sets. The test format was identical to the pretest administered at the beginning of the study and no feedback was provided for correct or incorrect answers during the posttest. Procedural Fidelity Data were collected on procedural fidelity (PF) of the EBI teaching sessions. The experimenter used the same GoogleSlides© created by Hemwell (2020), who based the presentations on a checklist created by Cummings & Saunders (2019). The experimenter assessed the method in which each EBI teaching session was presented to assess for PF. Each EBI teaching session consisted of 40 trials within each set and each trial consisted of seven 18 necessary fidelity components that described to which slide each stimulus was to be hyperlinked. To assess for PF, each slide was scored as correct if the experimenter incorporated all necessary components or incorrect if the experimenter missed any of the necessary components when creating the slide. Fidelity was calculated as the number of steps scored as correct by the total number of checklist steps multiplied by 100 to yield a percentage (Cooper et al., 2020). All three stimulus sets of EBI teaching sessions were created with 90% fidelity. Social Validity Following completion of the study, participants were emailed a survey to assess the social validity of this intervention. Participants were sent a Qualtrics link via university email that included questions regarding their knowledge and comfort of the behavior analytic terms before and after participation in the training, time commitment, significance of the information they learned, and overall satisfaction with the teaching method. Participants were instructed to rate each question on a five-point Likert scale with the following options: strongly agree, agree, neutral, disagree, and strongly disagree. The results of the social validity survey can be found in Table A7. All participants reported that the information they learned was relevant to their schoolwork and fieldwork. The majority of participants reportedly felt more confident in their understanding of the targeted terms following their participation in the study. The majority of participants agreed that EBI teaching sessions and intraverbal probes were easy to navigate. 19 Results Intraverbal Probes (BA and CA Relations) Andy Data for Andy’s intraverbal probes and EBI teaching sessions can be found in Figure 3. Andy’s scores on BA and CA relations ranged from 0% to 20% across stimulus sets on the first three intraverbal probes in the baseline condition. Andy started the study with stimulus set 3. Andy met mastery criteria within eight EBI teaching sessions for stimulus set 3. He then scored a 50% on BA relations and 40% on CA relations on the intraverbal probe for stimulus set 3; scores on both relations for stimulus sets 1 and 2 remained at baseline levels (range, 0% to 20%). Andy completed three remedial training sessions for stimulus set 3 and improved his BA score to a 70% and CA score to a 60%. Because this still did not meet mastery criteria for stimulus set 3, Andy completed two additional remedial training sessions. Andy improved his BA score to an 80% and CA score to an 60% and moved onto the next randomly assigned stimulus set (set 1). Andy completed EBI teaching sessions and reached mastery criterion for stimulus set 1 after three sessions. He then scored 90% on BA relations and 80% on CA relations on the intraverbal probe for stimulus set 1. His scores for the previously taught stimulus set, stimulus set 3, remained above baseline levels at 60% for both relations and his scores for the untaught stimulus set (set 2) remained at baseline levels (0%). Since Andy met mastery criteria for stimulus set 1, he moved onto the final randomly assigned stimulus set (set 2). Andy met mastery criterion for stimulus set 2 within three EBI teaching sessions. His scores on BA and CA relations for stimulus set 2 were 80% and 60%, respectively. Scores for stimulus set 3 dropped slightly (60% BA, 50% CA) and scores for stimulus set 1 remained at 20 mastery criteria level (90% BA, 80% CA). Andy would have required remedial training sessions for stimulus set 2, however, the study ended before he could complete those sessions. Figure 3. Graphical Display of Andy’s Probe, EBI, and Remedial Training Data Note. Diamonds depict BA relations. Opened circles depict CA relations. Squares depict EBI teaching sessions. Solid lines depict a phase change. Dotted lines depict a probe phase. 21 Alice Data for Alice’s intraverbal probes and EBI teaching sessions can be found in Figure 4. Scores on BA and CA relations ranged from 0% to 40% across stimulus sets on the first three intraverbal probes in the baseline condition. Alice started the study with stimulus set 3. It took Alice twelve EBI teaching sessions to reach mastery criterion for stimulus set 3. She then scored a 50% on BA relations and 20% on CA relations on the intraverbal probe for stimulus set 3. Scores on relations for both stimulus sets 1 and 2 remained at or above baseline levels (range, 0% to 50%). Alice completed three remedial training sessions for stimulus set 3 and improved her BA score to a 70% and CA score to a 40%. This still did not meet mastery criteria for stimulus set 3, therefore, Alice completed two additional remedial training sessions. Alice improved her BA score to 90% and CA score to 80% (scores on relations for stimulus sets 1 and 2 continued at baseline levels) and moved onto the next randomly assigned stimulus set (set 2). Alice reached mastery criterion for stimulus set 2 after five EBI teaching sessions. She scored 100% on BA relations and 90% on CA relations on the intraverbal probe for stimulus set 2. Her scores for the previously taught stimulus set 3 remained above baseline levels (80% BA, 60% CA) and her scores for stimulus set 1 remained below baseline levels (BA 20%, CA 0%). Alice then completed EBI teaching sessions for stimulus set 1 and met mastery criterion within 3 sessions. Her scores on BA and CA relations for stimulus set 1 were 90% and 90%, respectively. She scored 90% on BA relations and 60% on CA relations for stimulus set 3 and scored 90% on BA relations and 90% on CA relations for stimulus set 1. 22 Figure 4. Graphical Display of Alice’s Probe, EBI, and Remedial Training Data Note. Diamonds depict BA relations. Opened circles depict CA relations. Squares depict EBI teaching sessions. Solid lines depict a phase change. Dotted lines depict a probe phase. 23 Eliza Eliza’s intraverbal probes and EBI teaching sessions can be found in Figure 5. Scores on BA relations ranged from 0% to 100% across stimulus sets on the first three intraverbal probes in the baseline condition. The CA relations scores across stimulus sets ranged from 0% to 60% in the first intraverbal probe condition. Eliza started the study with stimulus set 2. It took Eliza three EBI teaching sessions to reach mastery criteria for stimulus set 2. She then scored a 60% on BA relations and 80% on CA relations on the intraverbal probe for stimulus set 2. Scores on both relations for stimulus sets 1 and 3 remained at baseline level (range, 0% to 100%). Eliza completed three remedial training sessions for stimulus set 2 and improved her BA score to a 100% and CA score to a 90% (with sets 1 and 3 remaining at baseline levels). This met mastery criteria for stimulus set 2, therefore, Eliza moved onto the next randomly assigned stimulus set (set 1). Eliza reached mastery criterion for stimulus set 1 after three EBI teaching sessions. She scored 100% on BA relations and 100% on CA relations on the intraverbal probe for stimulus set 1. Her scores for the previously taught stimulus set 2 remained above baseline levels at 70% for BA relations and 90% for CA relations, and scores for stimulus set 3 remained low. Eliza then completed four EBI teaching sessions for stimulus set 3 to reach mastery criterion. Her scores on BA and CA relations for stimulus set 3 were both 100%. She scored 80% on both relations for stimulus set 2 and scored 100% on both relations for stimulus set 1. 24 Figure 5. Graphical Display of Eliza’s Probe, EBI, and Remedial Training Data Note. Diamonds depict BA relations. Opened circles depict CA relations. Squares depict EBI teaching sessions. Solid lines depict a phase change. Dotted lines depict a probe phase. 25 Kelly Data for Kelly’s intraverbal probes and EBI teaching sessions can be found in Figure 6. Scores on both BA and CA relations ranged from 0% to 10% across stimulus sets on the first three intraverbal probes in the baseline condition. Kelly started the study with stimulus set 1. It took Kelly three EBI teaching sessions to reach mastery criteria for stimulus set 1. She then scored a 50% on BA relations and 30% on CA relations on the intraverbal probe for stimulus set 1. Scores on both relations for stimulus sets 2 and 3 remained low (0%). Kelly completed three remedial training sessions for stimulus set 1 and improved her BA score to a 100% and CA score to a 90%. This met mastery criteria for stimulus set 1, therefore, Kelly moved onto the next randomly assigned stimulus set (set 2). Kelly reached mastery criterion for stimulus set 2 after four EBI teaching sessions. She scored 90% on BA relations and 70% on CA relations on the intraverbal probe for stimulus set 2. Her scores for the previously taught stimulus set 1 remained above baseline levels at 70% for both relations. Kelly completed three remedial teaching sessions for stimulus set 2 and improved both her BA and CA relation scores to 90%. Scores for stimulus set 1 remained at mastery criteria at 80% for both relations and scores for stimulus set 3 were below baseline levels. Kelly then completed ten EBI teaching sessions for stimulus set 3 to reach mastery criterion. Her scores on BA and CA relations for stimulus set 3 were 70%. She scored 80% on both relations for stimulus set 1 and scored 90% on BA relations and 100% on CA relations for stimulus set 2. Kelly completed three remedial training sessions for stimulus set 3 and both her BA and CA relation scores remained at 70%. This still did not meet mastery criteria for stimulus set 3, therefore, Kelly completed two additional remedial training sessions. Kelly maintained her BA score of 70% and CA score decreased to 60%. Kelly did not meet mastery criteria for the last 26 stimulus set before the study ended. However, BA and CA relations for previously taught stimulus set 1 remained above baseline levels for both relations (range, 70% -100%), and scores for stimulus set 2 remained above baseline levels (range, 60% - 100%) for both relations. Figure 6. Graphical Display of Kelly’s Probe, EBI, and Remedial Training Data Note. Diamonds depict BA relations. Opened circles depict CA relations. Squares depict EBI teaching sessions. Solid lines depict a phase change. Dotted lines depict a probe phase. 27 Rachael Data for Rachael’s intraverbal probes and EBI teaching sessions can be found in Figure 7. Scores on BA relations ranged from 0% to 50% across stimulus sets on the first three intraverbal probes in the baseline condition. The CA relations scores across stimulus sets ranged from 0% to 60% in the baseline condition. Rachael started the study with stimulus set 3 and it took Rachael eight EBI teaching sessions to reach mastery criterion. She then scored a 50% on both BA and CA relations on the intraverbal probe for stimulus set 3. Scores on both relations for stimulus sets 1 and 2 remained at baseline levels (range, 0% to 60%). Rachael completed three remedial training sessions for stimulus set 3 and improved her BA score to 80% and CA score to 50%. This still did not meet mastery criteria for stimulus set 3, therefore, Rachael completed two additional remedial training sessions. Rachael improved her BA score to 80% and CA score to 90% and moved onto the next randomly assigned stimulus set (set 1). Next, Rachael reached mastery criterion for stimulus set 1 after three EBI teaching sessions. She scored 100% on BA and CA relations on the intraverbal probe following EBI teaching sessions for stimulus set 1. Her scores for the previously taught stimulus set 3 remained above baseline levels at 80% for BA relations but fell to 50% for CA relations. Rachael then completed EBI teaching sessions for stimulus set 2 and met mastery criterion within 3 sessions. Her scores on BA and CA relations for stimulus set 2 were 100% for BA relations and 80% for CA relations. She scored 70% on BA relations and 40% on CA relations for stimulus set 3 and scored 80% on BA relations and 90% on CA relations for stimulus set 1. 28 Figure 7. Graphical Display of Rachael’s Probe, EBI, and Remedial Training Data Note. Diamonds depict BA relations. Opened circles depict CA relations. Squares depict EBI teaching sessions. Solid lines depict a phase change. Dotted lines depict a probe phase. 29 Julia Data for Julia’s intraverbal probes and EBI teaching sessions can be found in Figure 8. Scores on BA relations ranged from 0% to 70% across stimulus sets on the first two intraverbal probes in the baseline condition. The CA relations scores across stimulus sets ranged from 0% to 40% in the baseline condition. Julia started the study with stimulus set 1. It took Julia three EBI teaching sessions to reach mastery criteria for stimulus set 1. She then scored a 100% on BA relations and 90% on CA relations on the intraverbal probe for stimulus set 1. Scores on both relations for stimulus sets 2 and 3 remained at baseline levels (range, 0% to 10%). Next, Julia completed EBI teaching sessions for stimulus set 2 and reached mastery criterion after three sessions. She scored 10% on BA relations and 10% on CA relations on the intraverbal probe for stimulus set 2. Her scores for the previously taught stimulus set 1 remained above baseline levels at 90% for BA relations and 60% for CA relations. However, a 10% for both BA and CA relations for stimulus set 2 did not meet mastery criteria. Julia then completed three EBI teaching sessions and improved her BA score on the intraverbal probes to 80% and CA score to 90% Julia then completed four EBI teaching sessions for stimulus set 3 to reach mastery criterion. Her scores on BA and CA relations for stimulus set 3 were 50% and 60%, respectively. She scored 50% on BA relations and 60% on CA relations for stimulus set 1 and scored 70% on BA relations and 100% on CA relations for stimulus set 2. Julia then completed three remedial training sessions, as she did not meet mastery criteria for stimulus set 3. Julia’s scores decreased in both relations to 30% for stimulus set 3. This still did not meet mastery criteria for stimulus set 3, therefore, Julia completed two additional remedial training sessions. Julia improved her BA and CA scores to an 80% and moved on to the posttest. 30 Figure 8. Graphical Display of Julia’s Probe, EBI, and Remedial Training Data Note. Diamonds depict BA relations. Opened circles depict CA relations. Squares depict EBI teaching sessions. Solid lines depict a phase change. Dotted lines depict a probe phase. 31 Sarah Data for Sarah’s intraverbal probes and EBI teaching sessions can be found in Figure 9. Scores on BA relations ranged from 0% to 30% across stimulus sets on the first three intraverbal probes in the baseline condition. The CA relations scores across stimulus sets ranged from 0% to 40% in the baseline condition. Sarah started the study with stimulus set 2 and met mastery criterion within four EBI teaching sessions. She then scored an 80% on BA relations and 60% on CA relations on the intraverbal probe for stimulus set 2. Scores on both relations for stimulus sets 1 and 3 remained at baseline levels (range, 0% to 40%). Sarah completed three remedial training sessions for stimulus set 3 and improved her BA score to a 100% and CA score to a 70%. This still did not meet mastery criteria for stimulus set 2; therefore, Sarah completed two additional remedial training sessions. Sarah decreased her BA score to a 70% and increased CA score to an 80% and moved onto the next randomly assigned stimulus set (set 1). Next, Sarah reached mastery criterion for stimulus set 1 after three EBI teaching sessions. She scored 100% on BA relations and 100% on CA relations on the intraverbal probe for stimulus set 1. Her scores for the previously taught stimulus set 2 remained above baseline levels at 70% for BA relations and 60% for CA relations. Sarah then completed three EBI teaching sessions for stimulus set 3 to reach mastery criterion. Her scores on BA and CA relations for stimulus set 3 were 20% and 0%, respectively. She scored 70% on BA relations and 80% on CA relations for stimulus set 2 and scored 80% on BA relations and 70% on CA relations for stimulus set 1. Sarah completed three remedial training sessions for stimulus set 3 and improved her BA score to 60% and CA score to 70%. This still did not meet mastery criteria for stimulus set 3, therefore, Sarah completed two additional remedial training sessions. Sarah improved her BA score to 70% and CA score to 70% and moved on to the posttest. 32 Figure 9. Graphical Display of Sarah’s Probe, EBI, and Remedial Training Data Note. Diamonds depict BA relations. Opened circles depict CA relations. Squares depict EBI teaching sessions. Solid lines depict a phase change. Dotted lines depict a probe phase. 33 Pretest to Posttest Improvement (All Relations) On average, out of a total of 210 possible points participants scored 114.43 (SD= 36.66) on the pretest, and significantly increased their scores on the posttest to an average of 188.57 (13.75), t(6) = -5.75, p < .001. Andy scored a 37% on the pretest and improved his score to 85% on the posttest. Alice scored a 56% on the pretest and improved her score to a 78% on the posttest. Eliza scored a 75% on the pretest and improved her score to a 98% on the posttest. Kelly scored a 25% on the pretest and improved her score to a 91% on the posttest. Rachael scored a 63% on the pretest and improved her score to a 91% on the posttest. Julia scored a 61% on the pretest and improved her score to a 95% on the posttest. Sarah scored a 64% on the pretest and improved her score to a 91% on the posttest (see Table A8). Examining scores for each set more closely, responding was slightly more variable. On average, for set 1 participants scored 46.29 (13.29) out of a possible 70 points on the pretest and significantly increased their scores to 65.14 (4.84) on the posttest, t(6) = -3.91, p = .004. For set 2, participants scored 42.39 (17.03) on the pretest and significantly increased their scores to 66.29 (4.03) on the posttest, t(6) = -3.79, p = .005. And for set 3, participants scored 25.86 (8.47) on the pretest and significantly increased their scores to 57.14 (8.01) on the posttest, t(6) = - 11.42, p < .001). All participants’ pretest and posttest data across the three stimulus sets can be found in Tables 9, 10, and 11. Generalization to Cooper et al. (2020) Definitions The majority of participants (6 out of 7) displayed generalization of the simplified definitions of the terms to the standard definitions in the Cooper et al. (2020) textbook across all stimulus sets. On average, for set 1 participants scored 47.14 (17.04) on the pretest and increased their scores to 70.00 (11.55) on the posttest, t(6)= -3.55, p = .012. For set 2, participants scored 34 60.00 (9.00) on the pretest and significantly increased their scores to 82.86 (20.59), t(6)= -1.99, p = .047. Last, for set 3 participants scored 30.00 (8.16) on the pretest and increased their scores to 71.43 (13.45), t(6)= -6.97, p < .001). See Table A12. 35 Discussion EBI involves the principle of emergent relations in which some relations are taught and then the learner develops a new relational understanding without direct training (O’Neill & Rehfeldt, 2014). MTS is a common approach used to teach these emergent relations within EBI (Brodsky & Fienup, 2018). The present study assessed whether simplifying definitions of ABA terminology impacts results and maintenance of emergent responding within intraverbal probes and multiple-choice responding. There were four main findings from this study. First, participants were able to learn the relations between the terms, definitions, and examples using EBI via MTS teaching. These results provide further evidence of the effectiveness of EBI in selection-based responding (AB and AC relations). Instructors of individuals working towards their BCBA® certification may consider using EBI teaching via MTS format when teaching ABA terminology, and examples. However, despite the success of EBI training, the present study also highlighted issues in the transfer to and maintenance of topography-based responding. Second, despite acquisition of AB and AC relations during EBI, performance was variable when assessing BA and CA relations on post-training intraverbal probes. All participants required remedial training for at least one set in the study. Further, once participants met mastery criterion on intraverbal probes, not all participants-maintained criterion level performance throughout the study. These results are similar to those reported by Hemwell (2020), where EBI teaching was effective in producing emergent selection-based responding across AB and AC relations but responding did not transfer to written topography-based intraverbal responding (BA and CA relations), nor did it maintain over time. Still, although several participants were unable to reach mastery criterion for intraverbal probes, all participants 36 showed improvement from their baseline scores. Further research on EBI and its effects on topography-based responding is needed. Despite text simplification, not all participants in the present study were able to reach mastery criterion and maintain intraverbal responding. Hemwell (2020) also found limited efficacy in EBI teaching via MTS promoting emergent topography-based responding. While previous studies have demonstrated emergent topography-based responding when teaching approximately 15 concepts or less (Albright et al. 2016; O’Neill & Rehfeldt, 2014; Sella et al., 2014; Walker et al., 2010), Hemwell (2020) hypothesized that the larger the number of terms in her study may have been too cumbersome for the participants to learn and maintain overtime. Future research may look into the acquisition and maintenance of emergent topography-based responding with ABA terminology with less material or material taught over a greater time. Third, all participants demonstrated emergent symmetrical and transitive selection based responding across stimulus sets from pre-to-posttest (e.g., AD, AB, AC, BA, CA, CB, BC). The finding that untrained relations emerge following the training of some relations is similar to results of previous EBI studies evaluating EBI as an effective teaching method for untrained relations to emerge (Albright et al., 2016; Brodsky & Fienup, 2018; Greville et al., 2016). Interestingly, improvement differed by stimulus set. Although performance improved across all three stimulus sets from pretest to posttest, performance on stimulus set 3 was still close to chance responding at posttest. This difference in performance could have occurred due to external variables (e.g., previous experiences, previous training); on the other hand, variability may have occurred because of the terms and definitions within the given stimulus set. It is possible that the terms, definitions, and/or examples within stimulus set 3 were still more difficult to comprehend due to the complexity and difficulty of the concepts. For example, terms 37 in stimulus set 2, such as “response generalization” and “response maintenance” may not be as difficult concept compared to those in set 3 (e.g., XYZ). Fourth, all participants improved performance on AD relations, matching the term to the Cooper et al. (2020) definitions, indicating generalization of the simplified definitions. Previous studies have shown that text difficulty impacts comprehension of material (Siddharthan, 2014). The findings from the current study indicate that after participants were taught simplified definitions of ABA terms, they were able to generalize these terms to more complex, untaught definitions from the Cooper et al. (2020) textbook. Further research can be conducted to further evaluate the implications of text simplification and EBI. For example, it would be beneficial for future instructors to know when text simplification is most appropriate for their students. In addition, future researchers may investigate the most effective and evidence-based approach to text simplification. Perhaps developing a task analysis for text simplification would be a beneficial tool for future instructors when teaching abstract concepts. Last, because participants were able to generalize these simplified definitions to more complex principals, instructors should consider how to simplify definitions when first teaching terms, such as removing complex jargon, to help students better understand and comprehend the overall principle. Text Simplification An important component of the current study was the inclusion of simplified definitions to teach ABA terminology. Results from the present study indicated that participants were able to generalize simplified definitions to more complex definitions from the Cooper et al. (2020) textbook. These findings highlight that the use of jargon or overly nuanced definitions may not always be necessary when first introducing ABA terms and concepts. However, despite simplifying the text, some terms were still more difficult than others for participants to learn. 38 Participants took more EBI teaching sessions to complete stimulus set 3 terms compared to stimulus sets 1 and 2. Future research should examine a more systematic way of simplifying terms to ensure difficulty is consistent across concepts and relations. Future research may also examine how the simplification of ABA definitions translates to applied settings. Studies may examine how this training impacts performance on the BCBA® certification exam or how the simplification of definitions impacts the use of jargon by BCBAs in their everyday role. Finally, future research should examine under what circumstances it is appropriate to use text simplification. Understanding the implications of text simplifications and when it is most important would help future instructors effectively teach individuals preparing for the BCBA exam. In addition, given the variable responding, lack of transfer to intraverbal responding, and lack of maintenance, text simplification may not have been enough. Future research may examine additional strategies that could be added to the EBI teaching to enhance the emergence of untrained relations. For example, future research could examine whether performance on intraverbal probes is further improved if participants are provided a tool/aid (e.g., word bank, cheat sheet) for assistance. Such an aid could help participants to remember the terms during topography-based responding, and the experimenter could systematically fade the use of the aid as the participants progress through the study. Another consideration for future replication is to incorporate a more systematic feedback system following intraverbal probes. Participants in the current study did not receive any feedback on their performance; it is possible that performance may have improved if they received feedback on which terms they scored incorrectly on, and why. 39 Limitations It is important to note this study is not without limitations. First, midway through their participation in the study, some participants were assigned to read the Cooper et al. (2020) chapters that contained the terms being taught in this study for an ABA course. This occurred due to the study timeline extending further than anticipated because of the length of time it took some participants to complete a stimulus set. The introduction of this material may have impacted performance on some participants' EBI teaching sessions or posttest results. While participants had access to Cooper et al. (2020) material during the study, the use of Lockdown Browser ® limited the ability for participants to look up the terms during the intraverbal probes or posttest. Second, only two baseline probes were obtained for Julia before she began EBI training on her first stimulus set. and moved Julia into the next study phase. Ideally, Julia would have completed a total of three baseline probes to establish a stable baseline, but she was moved to intervention before the third probe in error. Third, the current study was conducted during the COVD-19 pandemic. Some participants contracted the virus and were displaying mild symptoms of COVID-19 while completing the EBI teaching sessions, intraverbal probes, or posttest. It is possible their illness may have impacted performance. Despite these limitations, the present study provides further evidence on the effectiveness EBI has on teaching ABA terminology, and examples in a selection-based format. In addition, this study provided further explanation on the effects of text simplification and how it can be paired with EBI training to develop untrained relations. It is a requirement from the BACB® for individuals pursuing their BCBA certification to have an effective and comprehensive understanding of ABA terminology, definitions, and examples. Upon receiving a BCBA 40 certification, it is also expected for the BCBA to effectively simplify ABA terminology when discussing programming with parents, clients, or students at the beginning stages of learning ABA. Understanding and comprehending ABA terminology is an important skill and can be difficult for first time students to understand and learn. The results of the current study indicate that the use of EBI and MTS, with the addition of text simplification, may lead to acquisition of the skill. 41 APPENDIX 42 APPENDIX Tables Table 1. List of Terms and Definitions for Stimulus Set 1 (Verbal Behavior) Term (A) Definition (B) Echoic When the learner repeats what the other person says. Mand When the learner asks for what they want. Tact When the learner labels objects, events, or actions in the environment. Intraverbal When a learner responds to a question or a fill in the blank statement. Taking When a learner writes down exactly what a speaker says Dictation Textual When a learner reads written words out loud. Generative When a learner gains new skills due to previous learning. Learning Copying Text When a learner rewrites written words. Private Events The learner’s internal thoughts you don’t see or hear. Verbal behavior about verbal behavior. Autoclitic 43 Table 2. List of Terms and Examples for Stimulus Set 1 (Verbal Behavior) Term (A) Example 1 (C) Example 2 (C) Example 3 (C) Intraverbal probe A Intraverbal probe B Pretest/Posttest Client says “train” after Child says “cup” after You say “book” and your Echoic hearing therapist say mom says “cup.” child says “book.” “train.” Saying “cookie,” and Saying “I want to watch Your client hands you a Mand someone gives you a TV” and someone turns onpicture icon of an iPad and cookie. the TV. you give her the iPad. Seeing an apple Pointing to cows on the Telling someone you’re Tact and saying, “There’s an side of the road and saying sad. apple.” “cow.” Client says “bus” after Friend asks you when You ask your client “what Intraverbal hearing “The wheels on your birthday is, and you TV show do you like?” and the...” respond “September 2nd.” they say, “PJ Masks.” Writing down the time of Typing a direct quote from Taking Writing down someone’s an appointment when told your professor during Dictation name after they tell you it. over the phone. class. Seeing the written word Seeing “C-A-T” and Seeing the number 5 and Textual “bowl” and saying saying “cat.” saying “Five.” “bowl.” Client labels a picture of a Your child selects a picture Client asks for the ball Generative dog after teaching him to of juice from an array after after teaching her to label a Learning select a picture from a dog teaching her to request ball. from an array of 3 stimuli. juice. Writing down titles of Writing down a Writing down key terms Copying books you want to read restaurant’s address after from the textbook you’re Text while looking at a list of finding it on the website. reading. recommendations. Private Thinking about what you Thinking about which Having a headache. Events want for dinner. movie you want to watch. “I think I’m going on “I know tomorrow is Autoclitic “It might be snowing.” vacation.” Tuesday.” 44 Table 3. List of Terms and Definitions for Stimulus Set 2 (Generalization and Maintenance) Term (A) Definition (B) Response Generalization When a learner knows how to do something because they learned how to do something similar. Multiple Exemplar Training Using a lot of examples in your teaching. Indiscriminable When the learner does now know if their behavior will result Contingency in reinforcement. Teach Loosely Changing things in the environment to help teach. Response Maintenance When the learner continues to perform the behavior after being taught. Generalization probe Assessing to see if the behavior occurs in an untaught situation. Naturally Existing When reinforcement or punishment occurs naturally. Contingency Program Common Stimuli Making the teaching setting similar to other settings, so the behavior occurs in more places. Setting/Situation When the behavior occurs in a context other than in the one Generalization that it was taught. Instructional Setting The environment where the behavior was originally taught to occur. 45 Table 4. List of Terms and Examples for Stimulus Set 2 (Generalization and Maintenance) Term (A) Example 1 (C) Example 2 (C) Example 3 (C) Intraverbal probe A Intraverbal probe B Pretest/Posttest Response Teaching someone to Teaching someone to wave You teach your child to Generalization answer “cookies” when to say bye and they also flip a light switch to turn asked their favorite say “bye” it on. Without any food, and they answer teaching they press a “crackers” the next time button that also turns the the question is asked light on Multiple Saying “do this” “copy Showing a picture of an Running one step Exemplar me” and “do the same” eagle, a pigeon and a imitation with objects Training during imitation training hummingbird with a ball, a car, and a during tact training of the train word “bird” Indiscriminable Providing tokens on a Allowing a break from the Giving a child access to Contingency VR-3 schedule of table when client asks for a an iPad on a VI-5 minute reinforcement break on average every 5x schedule of they ask. reinforcement Teach Loosely Setting up a child’s Teaching someone to form Using different stimuli workplace in one area of patterns with red, blue and every time you teach a the classroom during green blocks during one client to identify shades one session and moving session and purple, green of green to a new location in the and yellow beads during classroom for the next another session session. Response You begin teaching a You teach a child to play You learned how to say Maintenance child to recite his phone Pop the Pig. A month after hello in French (Bonjour) number by showing a mastery/discontinuing in high school. 5 years visual prompt of all teaching, you play pop the later you are still able to of the numbers. Over pig with the child and they say “Bonjour” when time you fade the independently perform all someone asks you how to numbers shown until of the steps correctly. say hello in French you do not show any numbers. You ask the child their phone number and they recite it without the visual prompt; 46 Table 4. (cont’d) Generalization Having a new therapist Asking a client to tact a Having a client mand for probe ask a client how old she novel picture of her sister. information on the is after being taught this playground after teaching skill by another therapist this skill in the classroom Naturally A girl is leaning back in Putting on a sweater when You feel an itch on your Existing her chair and the chair it’s cold out makes you arm and scratch it and it Contingency falls resulting in injury. feel warm. goes away Program Using a bar of soap Teaching a student to read Using a picture of the Common Stimuli instead of a bottle to their name in print and client’s dog when teach client to wash cursive because you know teaching him to label a their hands because you future teachers will display dog know the family uses their name using different bars of soap at home. fonts Setting/Situation A learner is taught to Teaching a client to brush You taught your child to Generalization complete a photographic their teeth at home, and ask for a fork for their activity schedule in their then they independently dinner at home. Your classroom and brush their teeth when child then independently independently completes sleeping over at their asks for a fork when you one in the gym. grandparents’ house. are at a restaurant Instructional The basement of a The client’s classroom The treatment space at Setting client’s home where where ABA therapy the clinic you work at ABA therapy sessions sessions are conducted are conducted. 47 Table 5. List of Terms and Definitions for Stimulus Set 3 (Equivalence-Based Instruction) Term (A) Definition (B) Symmetry When the directions of matching relationship can be reversed (if AB = BA). Reflexivity Matching two identical things (A = A). Transitivity When the learner understands that item A equals item B and that an item equals item C, then without teaching, when shown item A the learner selects item C. A = B, B = C, then A = C. Conditional Discrimination A discrimination that depends on the stimulus context. Simple Discrimination When the learner responds differently in the presence of different things. Higher-Order Operant A group that includes within it other groups that can Class themselves be their own class. Class-Specific Where a learner is taught they must respond differently Reinforcement when shown different things, and will receive different reinforcement based on their response. Class Expansion Adding an additional item to a set of items to increase the number of items in a group. Contextual Control The function of a behavior can change based on the context or situation. Training Structure The process of procedural set-up used when teaching how things differ in different contexts. 48 Table 6. List of Terms and Examples for Stimulus Set 3 (Equivalence-Based Instruction) Term (A) Example 1 (C) Example 2 (C) Example 3 (C) Intraverbal probe A Intraverbal probe B Pretest/Posttest Symmetry A learner is taught that a A learner is taught that picture of a dog is the the written word A learner is taught that same as a real dog and is “phone” is the same as picture of a toothbrush is able to understand a picture of a phone and related to a picture of without teaching that a understands that a toothpaste and knows that real dog is the same as a picture of a phone is the a picture of toothpaste is picture of a dog same as the written related to a toothbrush word ”phone” in the without any direct absence of any teaching reinforcement Reflexivity Matching a picture of a Matching the written Matching a pencil to slide to another identical word ”ball” to another another identical pencil picture of a slide without written word “ball” without any direct teaching any previous teaching or without any direct or reinforcement reinforcement teaching Transitivity Teaching that the Teaching that the spoken spoken word “pen” is Teaching that a picture of a word “tree” is the same the same as a picture of nickel is the same as a real as a picture of a a pen. Teaching that a nickel. Teaching that a a tree. Teaching that a picture of a pen is the real nickel is the same as picture of a tree is the same as the written the written words “5 same as the written word word pen. Client knows cents”. Client knows that a tree. Client knows that that the spoken word picture of a nickel is the the spoken word "tree” is "pen” is the same as the same as the written words the same as the written written word pen “5 cents” without direct word tree without direct without direct teaching. teaching. teaching. Conditional Someone points to a Someone points to a A child places a picture of Discrimination picture of a cookie in an blue card in an array of a car on top of another array of three different picture of a car in an array three different pictures colored cards when of 3 when told to “put with when shown a 3-D toy they hear the word same” cookie “blue” Simple You put a single ball in You clear a table and Your friend tells you to Discrimination front of your client put a train in front of ring the doorbell when you and say “hand me the your client arrive at her house. There ball.” They hand you the and say “point to the is only one button next to ball train.” They point to the her door. You press the train button to ring the doorbell. 49 Table 6. (cont’d) Higher-Order Generalized instruction- Operant Class Generalized imitation Generalized manding following Class-Specific When selecting a picture When selecting a When selecting a picture Reinforcement when given the written picture when given the when given the spoken word, you receive a spoken word, you word, you receive a toy veggie straw, and when receive a skittle, and car, and when selecting the selecting the written when selecting the picture when given the word when given the written word when written word, you receive a spoken word you receive given the picture you chip. a high five. receive social praise. Class Teaching that the spoken Teaching that the Teaching that a picture of a Expansion word “candle” is the written word “lamp” is hairbrush is the same as the same as the written word the same as the spoken written word “hairbrush” and a picture of a candle, word and an actual and an actual hairbrush, then introducing an lamp, then introducing then introducing the actual candle and a picture of a lamp and spoken word “hairbrush” teaching that it is the teaching that it is the and teaching that it is the same as the picture, the same as the actual item, same as the actual item, the spoken word and the the spoken word and picture and the written written word. the written word word Contextual Matching a red car to a Matching a hat to the word Matching an apple to the Control picture of a vehicle “clothing” when presenting word “fruit” when when presented with with “clothing,” presented with “fruit,” vehicles, animals and “silverware,” and “vegetable” and “grain,” furniture, but matching “electronics,” but matching but matching an apple to a red car to other red a hat to a picture of snow the word “food” when items when presented when presented with presented with “food,” with red, blue and pictures of snow, sun and “toy,” and “appliance.” green items rain Training Structure One-to-many Linear Series Many-to-one 50 Table 7. Results of Social Validity Survey by Participant Participant Question Andy Alice Eliza Kelly Rachael Julia Sarah The information I Strongly Strongly Strongly Strongly Strongly Strongly Strongly learned is relevant to Agree Agree Agree Agree Agree Agree Agree my schoolwork. The information I Strongly Strongly Strongly Strongly Strongly Strongly learned is relevant to Agree Agree Agree Agree Agree Agree Agree my fieldwork. I feel more confident in my understanding of the terms Strongly Strongly Strongly Strongly Strongly Agree Disagree following Agree Agree Agree Agree Agree participation in the study. Participation in this Strongly study was too time Disagree Disagree Disagree Agree Agree Neutral Agree consuming. The EBI teaching sessions (i.e., Google Strongly Strongly Strongly Agree Agree Agree Agree Slide trainings) were Agree Agree Agree easy to navigate. I had a strong understanding of Strongly Disagree Strongly Strongly Disagree Strongly these terms before Neutral Disagree Disagree Disagree Disagree participating in the study. The probes were Strongly Strongly Strongly Neutral Agree Agree Agree easy to navigate. Agree Agree Agree 51 Table 8. Pretest and Posttest Data Final Scores, All Stimulus Sets Pretest Posttest Participant Score Percent Score Percent Andy 68/210 33% 178/210 85% Alice 117/210 55% 164/210 78% Eliza 157/210 75% 205/210 98% Kelly 52/210 25% 191/210 91% Rachael 133/210 63% 191/210 91% Julia 128/210 61% 200/210 95% Sarah 140/210 66% 191/210 91% Table 9. Pretest and Posttest Data by AD Relation (Percent Correct) Set 1 Set 2 Set 3 Participant Pre Post Pre Post Pre Post Andy 70 60 60 90 30 70 Alice 30 60 70 40 40 60 Eliza 60 90 90 100 40 90 Kelly 30 70 20 90 30 70 Rachael 50 80 80 100 30 60 Julia 60 70 40 80 20 90 Sarah 30 60 60 80 20 60 Table 10. Pretest and Posttest Data by Relation for Stimulus Set 1 (Percent Correct) Andy Alice Eliza Kelly Rachael Julia Sarah Relation Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post AD 70 60 30 60 60 90 30 70 50 80 60 70 30 60 AB 100 100 70 90 100 100 30 100 90 100 100 100 80 100 BA 70 100 70 100 90 100 50 100 80 100 50 100 90 100 AC 50 100 80 80 60 80 20 100 70 100 80 100 60 100 CA 30 90 50 80 80 100 10 90 100 100 70 100 80 100 BC 40 100 60 90 90 100 20 100 100 100 80 100 100 100 CB 50 90 70 80 90 100 40 100 90 100 80 100 100 100 52 Table 11. Pretest and Posttest Data by Relation for Stimulus Set 2 (Percent Correct) Andy Alice Eliza Kelly Rachael Julia Sarah Relation Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post AD 60 90 70 40 90 100 20 90 80 100 40 80 60 80 AB 20 100 80 100 100 100 20 100 90 100 90 90 90 90 BA 40 100 90 100 90 100 20 100 70 100 80 100 60 100 AC 20 80 50 100 90 100 40 90 70 100 40 100 60 100 CA 10 90 60 80 70 100 30 90 70 100 90 100 70 100 BC 10 100 50 80 90 100 20 90 80 100 70 100 80 100 CB 50 100 90 80 90 100 10 100 60 100 60 100 70 100 Table 12. Pretest and Posttest Data by Relation for Stimulus Set 3 (Percent Correct) Andy Alice Eliza Kelly Rachael Julia Sarah Relation Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post Pre Post AD 30 70 40 60 40 90 30 70 30 60 20 90 20 60 AB 20 80 20 100 80 100 10 80 30 100 50 100 90 90 BA 10 90 30 80 50 100 30 90 40 80 40 100 50 80 AC 20 60 30 60 60 100 10 90 20 80 50 100 50 100 CA 20 60 50 70 40 100 40 90 50 70 50 100 40 90 BC 40 60 60 50 70 100 20 90 30 70 50 100 50 80 CB 20 60 30 60 50 90 20 80 30 70 20 70 40 80 53 REFERENCES 54 REFERENCES Albright, L., Schnell, L., Reeve, K. F., & Sidener, T. M. (2016). Using stimulus equivalence- based instruction to teach graduate students in applied behavior analysis to interpret operant functions of behavior. Journal of Behavioral Education 25, 290-309. Brodsky, J., & Fienup, D. M. (2018). Sidman goes to college: A meta-analysis of equivalence- based instruction in higher education. Perspectives on Behavior Science, 41(1), 95-119. Carr, J. E., & Nosik, M. R. (2016). Professional credentialing of practicing behavior analysts. Policy Insights from the Behavioral and Brain Sciences, 4(1), 3–8. Critchfield, T. S., Barnes-Holmes, D., & Dougher, M. J. (2018). What Sidman did – Historical and contemporary significance of research on derived stimulus relations. Perspectives on Behavior Science, 41, 9-32. Critchfield, T. S., Doepke, K. J., Kimberly Epting, L., Becirevic, A., Reed, D. D., Fienup, D. M., Kremsreiter, J. L., & Ecott, C. L. (2017). Normative emotional responses to behavior analysis jargon or how not to use words to win friends and influence people. Behavior Analysis in Practice, 10(2), 97–106. Cummings, C., & Saunders, K. J. (2019). Using PowerPoint 2016 to create individualized 49 matching to sample sessions. Behavior Analysis in Practice, 12, 483-490. Greville, W. J., Dymond, S., & Newton, P. M. (2016). The student experience of applied equivalence-based instruction for neuroanatomy teaching. Journal of educational Evaluation for Health Professions, 13(32). Fienup, D, M., Covey, D. P., & Critchfield, T, S. (2010). Teaching brain-behavior relations economically with stimulus equivalence technology. Journal of Applied Behavior Analysis, 43, 19-33. O’Neill, J., & Rehfeldt, R. A. (2014). Selection-based instruction and the emergence of topography-based responses to interview questions. The Analysis of Verbal Behavior, 30, 178-183. Polson, D.A., & Parsons, J.A. (2000). Selection-based versus topography-based responding: An important distinction for stimulus equivalence? The Analysis of Verbal Behavior, 17, 105-128. Sella, A. C., Ribeiro, D. M., & White, G. W. (2014). Effects of an online stimulus equivalence teaching procedure on research design open-ended questions performance of international undergraduate students. The Psychological Record, 64, 89-103. 55 Siddharthan, A. (2014). A survey of research on text simplification. Recent Advances in Automatic Readability Assessment and Text Simplification, 165(2), 259–298. Walker, B. D., & Rehfeldt, R. A. (2012). An evaluation of the stimulus equivalence paradigm to teach single-subject design to distance education students via Blackboard. Journal of applied behavior analysis, 45(2), 329–344. Xu, W., Callison-Burch, C., & Napoles, C. (2015). Problems in current text simplification research: New Data can help. Transactions of the Association for Computational Linguistics, 3, 283–297. 56