A FORMATIVE, UTILIZATION-FOCUSED EVALUATION OF USA SWIMMING’S FOUNDATIONS OF COACHING PROGRAM By Andrew Patrick Driska A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Kinesiology—Doctor of Philosophy 2015 PUBLIC ABSTRACT A FORMATIVE, UTILIZATION-FOCUSED EVALUATION OF USA SWIMMING’S FOUNDATIONS OF COACHING PROGRAM By Andrew Patrick Driska Coach education programs serve an essential function for sport governing bodies by providing coaches with coaching knowledge and evidence-based practice recommendations. Despite the proliferation of formal coach education programs, where participants complete a structured course and pass a knowledge test, there has been little work to evaluate programs that educate a large number of coaches (McCullick et al., 2009), especially a recent wave of online coach education programs. One such online coach education program is USA Swimming’s Foundations of Coaching program, a requirement for new coach members, consisting of two sequential online courses. More than 2,000 new swimming coaches take this course every year. To examine the effectiveness of the Foundations of Coaching program, the author worked with seven staff members at USA Swimming who oversaw the program’s development and implementation (i.e., key program decision-makers). The author was guided by the practice of utilization-focused evaluation (Patton, 2011), a theoretical framework for evaluating program effectiveness that simultaneously engages key program decision-makers in the evaluation, with the intended goal of promoting the use of the evaluation findings and process. The author engaged key program decision-makers at USA Swimming to generate a preliminary logic model, i.e., a sequential map detailing how activities in the course led to specific learning and coaching outcomes for participants. In addition, this process generated five purposes for continued evaluation: examining coach perceptions of the program, exploring program effects on coaches’ knowledge, attitudes, and behaviors, examining program delivery, identifying areas for improvement, and exploring the utilization of evaluation findings and process by USA Swimming. Following this stakeholder engagement process, the author studied existing quantitative data (i.e., coach demographics and test scores) and conducted interviews with 21 coaches to elicit their perceptions of the course in relation to the five evaluation purposes. Results from the coach interviews revealed positive coach perceptions of the course’s value, with 19 of 21 coaches stating that the course had helped them in their work as coaches. The online program was delivered with minimal technical complications. Program effects were broken into knowledge, attitudes, and behaviors. The strongest effects on coach knowledge were in the delivery of knowledge about drills and technique, long-term athlete development (i.e., a philosophy of holistic and developmentally-appropriate training), and planning of training. The attitudes most likely to be shaped by the course related to increasing the coach’s conscientiousness (i.e., the degree to which they put deliberate thought into their coaching practices), as well as increasing value for long-term athlete development. The behaviors most likely to be influenced included writing age-appropriate workouts, incorporating fun into practice, and engaging in goal setting with athletes. Program improvement areas largely focused on increasing opportunities for learning and continued dialogue about best practices in coaching after the course had been completed. To facilitate evaluation use, the author provided continual support for staff as they worked to implement changes. The evaluation process helped to build capacity for USA Swimming to plan future courses and outreach initiatives. This manuscript presents both specific findings and recommendations for USA Swimming, as well as findings generalized for researchers and practitioners of coach education. ABSTRACT A FORMATIVE, UTILIZATION-FOCUSED EVALUATION OF USA SWIMMING’S FOUNDATIONS OF COACHING PROGRAM By Andrew Patrick Driska Coach education programs serve an essential function for sport governing bodies by providing coaches with coaching knowledge and evidence-based practice recommendations to improve the experience of athletes involved in the sport. Despite the proliferation of formal coach education programs, there has been little work to evaluate programs that educate a large number of coaches (McCullick et al., 2009), especially a recent wave of online coach education programs. Grounded in the framework of utilization-focused evaluation (Patton, 2011), this manuscript presents results from a program evaluation of USA Swimming’s Foundations of Coaching program. Foundations of Coaching consists of two separate online courses (101 and 201), is part of the requirements for new USA Swimming coach members, and is taken by more than 2,000 new swimming coaches each year. A stakeholder engagement process led by the author engaged seven staff members of USA Swimming in the process of exploring the program’s objectives and intended effects on program participants, which generated a preliminary logic model that described anticipated program outcomes. In addition, this process generated five purposes for continued evaluation: examining coach perceptions of the program, exploring program effects, examining program delivery, identifying areas for improvement, and exploring utilization of evaluation findings and evaluation process by USA Swimming. Following this stakeholder engagement process, the author designed a sequential exploratory mixed methods design, using qualitative inquiry as the primary means to elicit perceptions from 21 coaches that had completed the course to address the evaluation purposes. Results of the qualitative inquiry revealed positive coach perceptions of the course’s utility, with 19 of 21 coaches stating that the course had helped them in their work as coaches. Program effects were broken into knowledge, attitudes, and behaviors. The strongest effects were in the delivery of knowledge about drills and technique, long-term athlete development, and planning training (knowledge), increasing conscientiousness and value for long-term athlete development (attitudes), and increasing the degree to which coaches wrote age-appropriate workouts, incorporated fun into practice, and engaged in goal-setting with athletes (behaviors). The program was delivered with minimal technical complications, although the effects of the material presented in the Foundations 101 course appeared to be stronger. Coaches identified the need for continual access to course materials and clear opportunities for continuing education as areas for improvement. In addition, the author presented the need for continuing the conversation of good coaching practice with coaches after the formal coach education experience had ended. The author addressed a fifth purpose, evaluation use by USA Swimming administrators, by conducting a follow-up meeting to present interim results and solicit perceptions from these key program decision-makers, and by providing continual support for staff as they worked to implement changes. The evaluation provided an opportunity for USA Swimming to develop its evaluative capacity with relation to planning courses and future outreach initiatives for coach development. This manuscript discusses results of this evaluation first in the context of recommendations for USA Swimming, and second in relation to the contribution of this inquiry to the broader coach education literature. ACKNOWLEDGEMENTS I would first like to acknowledge my parents, Steve & Janie. Both of my parents are educators, and from a very early age, instilled the importance of the intrinsic value of learning. While my path has been a winding one, and has certainly taken its time, I appreciate that their support has never decreased. To the many teachers I have had along the way. To Mr. Wagman, who delivered a subtle kick in the pants in 2006 when, after not seeing him for 10 years, he stated that I was overqualified for a day-job I was applying for. To Dr. Cameron, my undergraduate adviser, who during a critical breakdown in my undergraduate education, helped me realize and accept that people can reinvent themselves in their careers, and helped me graduate. To Naeem Inayatullah, who threw a critical-thinking monkey wrench into the works at a very important time. To Bud Reiner, Steve Duncheskie, and Nick Militello, my first coaching mentors, who provided the backbone of my informal education as a coach, and kept my feet on the ground and kept my emotions cool when the pressure kicked up. To Dr. Cindra Kamphoff, my masters degree adviser, who flat-out told me one day that I needed to pursue a Ph.D. To Jay Morini, my friend who told me, “never pay for graduate school.” That turned out to be the how-to advice that made graduate school a possibility in the first place. To my brother, Brian, who started his graduate program at the same time that I did, and provided a great scratching post with whom I could commiserate. To the members of this dissertation committee. Dr. Smith for your patience, and for being the positivist anchor in the room! Dr. Mishra, for being a creative inspiration, ten steps iii ahead of conventional wisdom. Dr. Campbell, for your encouragement and for showing me the world of evaluation – such a great way to link my career interests. To Dan. Lieutenant Columbo. Silently engineering endless possibilities for me and countless others with whom you have worked. Thanks for the opportunity, trust, and above all, your patience. To the staff at USA Swimming who worked so well with me on this project and who permitted me this opportunity in the first place. To students in the Department of Kinesiology and Institute for the Study of Youth Sports at Michigan State University: driven people… and willing to engage in conversation, knowing that it was going to go off into many different directions. Jed Blanton – great perspective. Missy Wright – you know why. Ian Cowburn – bet I get this submitted before you. iv TABLE OF CONTENTS LIST OF TABLES ......................................................................................................................... ix! LIST OF FIGURES ........................................................................................................................ x! KEY TO ABBREVIATIONS ........................................................................................................ xi! Chapter 1 - Introduction .................................................................................................................. 1! Chapter 2 - Review of Literature .................................................................................................... 8! History of and Rationale for Coach Education ................................................................... 8! Coach Education Research ............................................................................................... 12! Informal, coach-driven methods. .......................................................................... 12! Formal, curriculum-driven methods. .................................................................... 12! Mentorship ............................................................................................................ 13! Evaluation of Coach Education Programs ........................................................................ 14! Coach Education for USA Swimming .............................................................................. 16! Foundations of Coaching program........................................................................ 16! Benefits and drawbacks to formal education for swimming coaches ................... 18! The Experience of the User: Coach Reactions ................................................................. 21! Program Evaluation .......................................................................................................... 23! Evaluation methods............................................................................................... 25! Needs assessment. ..................................................................................... 25! Evaluability assessment. ........................................................................... 25! Logical framework or theory of change.................................................... 25! Program process. ....................................................................................... 26! Program outcomes. ................................................................................... 26! Evaluation theories................................................................................................ 27! Utilization-focused evaluation .................................................................. 28! Matching evaluation theory to the evaluation context .............................. 29! Rationale for this Evaluation ............................................................................................ 31! Chapter 3 - Methods...................................................................................................................... 33! Phase I: Preparing to Conduct the Evaluation ................................................................. 35! Phase II: Framing the Evaluation ...................................................................................... 36! Situation analysis .................................................................................................. 36! Building rapport with key program decision-makers ........................................... 37! Conduct focus group meeting with key program decision-makers ...................... 38! Production of interim report and evaluation agreement ....................................... 41! Phase III: Data Collection, Preliminary Analysis, and Presentation of Interim Findings 41! Creating a sample of coach participants ............................................................... 42! Analytical framework for qualitative inquiry ....................................................... 43! Preliminary interviews and analysis ..................................................................... 46! Interviews and deductive analysis ........................................................................ 48! v Interim findings meeting ....................................................................................... 49! Phase IV: Inductive Analysis, Description, and Interpretation of Findings ..................... 50! Chapter 4 – Results and Discussion .............................................................................................. 52! Description of the Foundations of Coaching Program ..................................................... 52! Course content ...................................................................................................... 52! Course pedagogy and technology ......................................................................... 53! Framing the Evaluation ..................................................................................................... 58! Situation analysis .................................................................................................. 59! Objectives of the Foundations of Coaching program ............................... 60! Uncertainties about the Foundations of Coaching program ..................... 61! Potential barriers to the Foundations of Coaching program ..................... 62! Addressing skeptical viewpoints about the Foundations of Coaching program ............................................................................................... 64! Logic model .............................................................................................. 65! Purposes of the evaluation ........................................................................ 67! Purpose 1: Program utility and perceptions ................................ 67! Purpose 2: Program effects .......................................................... 67! Purpose 3: Program delivery........................................................ 68! Purpose 4: Areas for improvement ............................................... 68! Purpose 5: Utilization of evaluation findings and process........... 68! Quantitative Exploration of Demographics and Test-Score Data..................................... 69! Demographics ....................................................................................................... 69! Examining test-score differences between groups .................................... 71! Interview Participants ........................................................................................... 77! Preliminary analysis .............................................................................................. 80! Interim findings meeting ....................................................................................... 83! Reaction to evaluation interim findings .................................................... 83! Evaluation process effects......................................................................... 85! Brainstorming possibilities ....................................................................... 88! Inductive Analysis to Address Purposes of the Evaluation .............................................. 90! Needs and Challenges ....................................................................................................... 90! Swimmer management.......................................................................................... 91! Coach development ............................................................................................... 92! Team management ................................................................................................ 93! Facility management ............................................................................................. 93! Program Utility and Perceptions ....................................................................................... 93! General appraisal of the course. ............................................................................ 94! Salient features of the course ................................................................................ 98! Limitations of the online course ........................................................................... 99! Program Effects ................................................................................................................ 99! Knowledge from the course ................................................................................ 100! Useful topics ........................................................................................... 100! Teaching and coaching principles .............................................. 101! Developmental principles and considerations ............................ 103! Planning training ........................................................................ 103! Reaffirming and reinforcing my stance as a coach .................... 104! vi What motivates athletes? ............................................................ 105! Nutrition and health .................................................................... 105! Communication ........................................................................... 106! Providing a library of resources ................................................. 106! Useful topics summary ................................................................ 107! Already knew it or relied on sport experience ........................................ 109! Didn’t understand.................................................................................... 110! Burning questions ................................................................................... 111! Knowledge implemented from the course .......................................................... 112! Translating and implementing changes .................................................. 113! Planning and workout design ..................................................... 113! Swimmer management and pedagogy ......................................... 114! Add-ons – goal setting and nutrition education.......................... 116! Restructuring age-groups based on PFAD model – a miniature case analysis ......................................................................... 116! Reflecting ................................................................................................ 120! Reflection induced by the Foundations course ........................... 121! Reflection induced by situations arising from the coach’s environment........................................................................... 122! Quest for new knowledge ....................................................................... 124! Communities of practice ......................................................................... 126! Barriers to knowledge implementation ................................................... 129! Coach attitudes .................................................................................................... 132! Professional mindset ............................................................................... 132! Conscientiousness ................................................................................... 134! Long-term athlete development .............................................................. 135! Valuing education and professional development .................................. 139! Other coach attitudes............................................................................... 139! Coach behaviors .................................................................................................. 141! Use of video analysis .............................................................................. 142! Writing age-appropriate workouts .......................................................... 144! Incorporating fun into practices .............................................................. 146! Goal setting ............................................................................................. 148! Being a role-model .................................................................................. 154! Developing the athlete as a person.......................................................... 156! Coach mindsets and philosophies towards developing the athlete as a person ............................................................................ 157! Efforts to develop the athlete as a person ................................... 158! Other coach behaviors............................................................................. 162! Program Delivery ............................................................................................................ 166! Operating systems and user complications ......................................................... 167! Timing and pacing of courses ............................................................................. 167! Interaction with the course .................................................................................. 170! Interpretation and judgment of program delivery ............................................... 171! Areas for Improvement ................................................................................................... 173! Concerns or improvements to the Foundations of Coaching course .................. 174! vii Content (inclusion, organization, depth, clarity) .................................... 174! Access to course materials ...................................................................... 177! Continuing education after the course .................................................... 178! Functionality ........................................................................................... 179! Evaluator interpretations ......................................................................... 179! USA Swimming service enhancements or additions .......................................... 181! Education ................................................................................................ 182! Networking and in-person clinics and conferences ................................ 183! Curating information............................................................................... 184! Revised Logic Model ...................................................................................................... 185! Chapter 5 – General Discussion .................................................................................................. 189! Evaluation Purposes ........................................................................................................ 189! Program utility and perceptions .......................................................................... 189! Program effects ................................................................................................... 190! Program delivery ................................................................................................. 199! Areas for improvement ....................................................................................... 202! Utilization of evaluation process and findings.................................................... 207! Use of evaluation findings ...................................................................... 207! Use of evaluation process ....................................................................... 211! Formative Judgments and Recommendations ................................................................ 213! Limitations ...................................................................................................................... 215! Future Evaluation Directions .......................................................................................... 219! Contributions to Coaching Science Knowledge ............................................................. 221! Future Coaching Science Research Directions ............................................................... 231! APPENDICES ............................................................................................................................ 236! Appendix A: Original Proposed Project Timeline .......................................................... 237! Appendix B: Assessment of Evaluator Competencies.................................................... 238! Appendix C: Evaluator Statement of Reflexivity ........................................................... 240! Appendix D: Solicitation E-mail .................................................................................... 242! Appendix E: Coach Interview Guide .............................................................................. 243! REFERENCES ........................................................................................................................... 246! viii LIST OF TABLES Table 1 - Evaluation timetable ...................................................................................................... 34! Table 2 - Skeptical viewpoints on the Foundations of Coaching program ................................... 65! Table 3 - Foundations of Coaching Preliminary Logic Model (January 2014) ............................ 66! Table 4 - Hypothesis Tests of Demographic and Test-Score Data ............................................... 72! Table 5 - Demographic Statistics for Interview Participants ........................................................ 79! Table 6 - Parent Code Numbers, Titles, and Descriptions ............................................................ 81! Table 7 - Deductive Code Application ......................................................................................... 82! Table 8 - Original Proposed Project Timeline ............................................................................ 237! ix LIST OF FIGURES Figure 1 - Screenshot from Foundations of Coaching course showing Pool Pointers ................. 54! Figure 2 - Screenshot from Foundations of Coaching course showing video integration ............ 55! Figure 3 - Self-check quizzes as formative assessment ................................................................ 56! Figure 4 - Open-ended formative assessment question ................................................................ 56! Figure 5 - Revised Program Logic Model .................................................................................. 188! x KEY TO ABBREVIATIONS USA Swimming Coach Member – the membership status required for a coach with a USA Swimming affiliated program. In order for a club to be included as a USA Swimming program, and be subject to its insurance, all personnel who carry on the duties of a coach must have attained the status of coach member. LSC – Local Swimming Committee; one of the 59 administrative units of USA Swimming that are typically at a state or regional level. Key program decision-makers – term used in place of the term “primary intended-users” that is espoused by the utilization-focused evaluation literature; specifically in this evaluation, they were staff members at USA Swimming involved with the evaluation process xi Chapter 1 - Introduction Sport is a widespread and essential feature of culture in the United States and throughout the world. It is enjoyed in both unstructured, recreational formats, as well as in structured, competitive formats (typically driven by adult organizers). At the hub of the structured sport experience is the coach, who is responsible for planning training, organizing competition, and in general, providing a quality experience from which athletes can learn and develop as players and people. Although coaching may appear easy from the spectator’s point of view, it is a profession that requires extensive knowledge from formal and experiential sources. The path to becoming an expert coach is often highly individualized and contextually based, and as such, there is little agreement over what processes are required to develop coaches effectively. Coach education provides an important service to coaches and sport participants. Over the past 40 years, sport psychology and coach education researchers have explored the effectiveness of coach education programs with regards to their ability to improve sport experiences and sport outcomes for athletes. Extensive research by Smith, Smoll, and colleagues (1977; 1983; 2002) has examined coach behavior and its impacts on athletes. Their Coach Effectiveness Training program (Smith, Smoll, & Curtis, 1979), rooted in social cognitive theory and self-efficacy (Bandura, 1977), uses a modeling approach to increase the frequency of coach behaviors (e.g., reinforcement, encouragement, contingent praise) related to positive sport outcomes for athletes (e.g., increased enjoyment, retention). Evaluations of CET training workshops have demonstrated increased coach adoption of positive coaching behaviors, and have shown their relationship to increased positive outcomes for sport participants. A limitation of these experiments (termed evaluations by the authors) is that they assessed isolated, researcher-driven intervention studies. Although these results were disseminated and adopted by 1 large-scale coach education programs (e.g., American Sport Education Program, Program for Athletic Coach Education), these large-scale programs have not been evaluated at a system-level. Recent research has focused on the sport motivational climate, specifically with regards to the provision of a task-oriented climate (rooted in Achievement-Goal Theory; Ntoumanis & Biddle, 1999) and meeting athlete basic psychological needs for competence, autonomy, and relatedness (rooted in Self-Determination Theory; Deci & Ryan, 2000). Additional research has examined the effects of a caring sport climate (Iwasaki & Fry, 2013). Much of this research is grounded in evidence showing that when athletes experience a task-oriented motivational climate that meets the three basic psychological needs, there is a greater likelihood for positive sport outcomes, such as increased enjoyment, participation, and goal striving. These interventions typically seek to use the coach as the primary actor in shaping the motivational climate. Despite their success in improving sport outcomes for athletes, these interventions do not constitute comprehensive coach education, as the motivational climate is only one facet of the duties that a coach must attend to in their work. Thus, “evaluations” of motivational climate interventions do not constitute true evaluations of a comprehensive coach education program. Canadian researchers have examined Sport Canada’s National Coach Certification Program extensively over the past 15 years. Sport Canada requires all coaches to be certified by this formal coach education program, which consists of attending in-person courses in the early phases of certification. These researchers have provided an extensive debate over the methods by which coaches learn best, advancing reflective practice (Gilbert & Trudel, 1999), communities of practice (Culver & Trudel, 2008), and a range of learning methods (Lemyre, Trudel, & Durand-Bush, 2007) as the preferred way to educate coaches. Notably absent in these lines of inquiry is the exploration of changes in coaching behavior, as well as a lack of 2 examination of outcomes for sport participants. As different lines of coach education and motivational climate research thrive, it becomes apparent that little research has examined the coach as the central actor in the provision of sport experiences for youth. Coaching is a comprehensive discipline, requiring knowledge from technical, tactical, physical, developmental, and psychosocial domains. The process of becoming a competent coach requires both formal and informal educational experiences, as well as significant amounts of experiential learning. The process by which the immense amount of evidence is internalized by the coach and translated into coaching practice, and how that process leads to beneficial outcomes for sport participants is a complex process. Intervention studies rooted in logical positivism (Smith & Smoll, 2014) may be able to demonstrate pieces of this conceptual model, but capturing its sequential and longitudinal nature may require a form of inquiry that is better suited to making sense of a messy context. Program evaluation is a form of social science research that examines programs in their context, and adapts inquiry methods to demonstrate program effectiveness (or ineffectiveness). The highly contextualized nature of a program evaluation enables greater use of evaluation findings so that evidence can drive practice. Program evaluations serve many functions; some seek to render summative judgment on a program’s ability to affect change (or not), while others seek to describe and explain the processes by which a complex program creates change in its participants. The variety of research methods available to a program evaluator are as wide as those available to a typical social science researcher. The key difference between program evaluation and social science research is that a program evaluation works in the context of the program, seeking to explore or explain events or outcomes in situ, as generated by the actual participants in the program, rather than attempting to manipulate variables and participants for 3 the sake of experimental control. Utilization-focused evaluation (Patton, 2008; 2011; 2013) is a specific theoretical framework for program evaluation that prioritizes the use of findings by primary intended users (or key program decision-makers), and conducting the evaluation in such a way that primary intended users are engaged and well-positioned to use the evaluation findings. It also espouses the need for key program decision-makers to benefit from the use of the evaluation process, i.e., to engage in evaluative thinking to build organizational capacity to use evaluation data and engage in evaluative thinking. Although utilization-focused evaluation does not specify a particular method of inquiry, it does provide that the evaluator maintain a facilitative and flexible style that is part consultant, and part rigorous social science researcher. If the ultimate goal is evaluation use by key program decision-makers, then the evaluator must be skilled in ascertaining a strong understanding of the culture of the organization, the aims and purposes of the program, and the ways in which key program decision-makers value and use evidence to guide their decision-making processes. Program evaluation has the potential to help make sense of the messy context of sport coaching and coach education, a context where experimental researchers often have difficulty manipulating variables and experimental conditions. It also shows promise in its ability to evaluate widespread coach education programs, especially in the modern era, which has seen the recent proliferation of online coach education programs. The American Sport Education Program (ASEP), the National Federation of State High School Athletic Associations (NFHS), the Positive Coaching Alliance (PCA), as well as several national sport governing bodies (USA Volleyball, USA Football, United States Tennis Association, USA Swimming) have adopted online courses to educate tens of thousands of coaches. Online coach education offers 4 convenience, and with the proliferation of broadband internet, an increased ability to scale up to offer widespread coach education at an affordable price. Because online education has disrupted the traditional notion of face-to-face, teacher-driven education, many skeptics have doubted its ability to provide quality education for coaches, which has driven a need to evaluate these programs. Despite the clamor in the research community to evaluate online coach education programs, there is a concurrent need amongst sport governing bodies to demonstrate that coach education programs (already in place out of a demonstrated need to provide some form of education to coaches) have the ability to deliver effective educational programming that makes a difference for coaches, athletes, and sport parents. Amongst sport governing bodies offering online coach education, USA Swimming has expressed a need to evaluate the most recent iteration of its coach education program. USA Swimming (USAS) is the national governing body for the sport of swimming in the United States. It oversees over 300,884 athletes, 15,674 coaches (2,799 full-time coaches), and 3,009 affiliated swimming clubs (USA Swimming, 2012). Seeing the importance of an educated fleet of coaches for the sport of swimming, in 1996, USAS implemented a coach education policy. Coach education has evolved from a mail-based correspondence course to its current format of two sequential online courses, Foundations of Coaching 101 and 201, which became the official means of coach education in January 2013. The course is administered by ePath Learning, and educates more than 2,000 new coach members every year. Given a long and complex history of coach education efforts, including philosophical disagreement on the goals of coach education and how it should be administered, USA Swimming has a need for greater understanding of how this program works and whether or not it is effectively educating coaches. Evaluating online education can benefit from the use of a guiding framework for inquiry 5 into the effectiveness of course. The technological, pedagogical, and content knowledge (TPACK) framework (Mishra and Koehler, 2006) provides a useful tool for examining the three domains that make up an online course. Content can be assessed for its relevance and its use by program participants, and technological and pedagogical elements of the course can be assessed for their efficacy in delivering the content knowledge and helping to spur the reflective process that should ideally accompany coach education efforts. Previous studies have evaluated the effectiveness of small, researcher-driven coach education interventions, but little research has evaluated effectiveness of widespread coach education programs. Furthermore, this research has not evaluated the impact on the coaches, but instead has focused on outcomes for athletes. For sustainable coach education programs to deliver service effectively, evaluations must examine programs in situ and provide the program administrators with high quality evidence so that they can make needed changes and improvements. Although a utilization-focused program evaluation is designed in concert between the evaluator and key program decision-makers to serve the needs of the organization for high-quality evidence, there are convergent research aims that mix with the evaluation purposes. This manuscript reports on a utilization-focused evaluation conducted by the author to evaluate the Foundations of Coaching program for USA Swimming. Evaluation purposes were determined through a process of facilitation between the evaluator and key program decisionmakers, but these evaluation purposes also have benefit to the wider community of coach educators and sport psychology researchers. Therefore, given that online coach education programs have not been evaluated for their effectiveness, and given the need for greater understanding of how coach knowledge, attitudes, and behaviors are learned and subsequently 6 interact to guide coaching practice, specifically in the case of USA Swimming coach members, this evaluation had the following purposes: (a) explore coach perceptions of the Foundations of Coaching program’s utility; (b) explore program effects with regard to knowledge gained and implemented, and attitudes and behaviors adopted; (c) explore the efficacy of the online course to deliver the program as intended; (d) explore areas for program improvement to better meet coach needs. A related purpose was to facilitate the use of evaluation findings and evaluation process by the key program decision-makers, in order to increase the likelihood of evaluation use in everyday administrative decision-making. 7 Chapter 2 - Review of Literature Coaching sports is an interdisciplinary, problem-focused, applied practice. Coaches need to have a competent understanding of the sport: its purposes, challenges, tactics, techniques, norms, and rules. In addition to content knowledge of the sport, coaches need to learn peoplemanagement and teaching skills, so that they may effectively transfer their knowledge to players. A coach’s content knowledge means little unless they have the skills to transfer this knowledge to help their athletes master the sport. Thus, there is a tension between how much knowledge a coach needs to have, and the need for the coach to develop the skills to apply this knowledge. This literature review will provide an overview of past coach education efforts and the logic that guided them, frameworks of coach education research, evaluation and research on coach education programs, a history of coach education for USA Swimming, technological and pedagogical constraints of online learning, program evaluation, and a rationale for this specific evaluation. History of and Rationale for Coach Education Early research on effective coaching examined specific behaviors that were associated with desired outcomes (e.g., increased athlete achievement, decreased athlete anxiety). Rigorous observation and coding of coach behaviors identified 12 types of behaviors associated with effective coaching, which were later codified into an observational recording instrument called the Coaching Behavior Assessment System (CBAS; Smith, Smoll, & Hunt, 1977). The CBAS categorizes coach behaviors into reactive behaviors, such as reinforcement (i.e., positive reaction to good play or good effort), mistake-contingent technical instruction (i.e., instructing a player how to correct a mistake), punitive technical instruction (i.e., correction of a mistake done in a hostile manner), keeping control (i.e., reaction to restore order), as well as spontaneous behaviors 8 such as general technical instruction or general encouragement. Studies showed that children who played for coaches with high levels of reinforcement, encouragement, and technical instruction reported greater enjoyment in their sport experiences (e.g., had more fun, liked teammates and coaches more) when compared to coaches that engaged in more punitive or ignoring behaviors (Smith, Zane, Smoll, & Coppel, 1983). In summarizing their research, Smoll and Smith (2002) state that they employed a mediational model that governed many of these coach behavior studies; in this model, athletes’ perceptions and recall mediated the relationship between coach behaviors and athletes’ evaluative reactions. Although these studies clearly demonstrate the importance of specific behaviors, most notably contingent instruction and reinforcement, the psychosocial outcome measures in these studies focus almost exclusively on athletes, with minimal exploration of coaches’ learning and development resulting from the intervention. Thus, they do not present a comprehensive picture of how coaches learn and improve their skills. Following from research with the CBAS, Smith and Smoll developed Coach Effectiveness Training (CET) as an evidence-driven coach education program. Although CET attempts to deliver key philosophies about sport coaching (e.g., winning isn’t everything, separating athlete’s feelings of self-worth from game outcomes, teaching athletes that success equals striving for victory), it is primarily a behavioral intervention designed to increase coaching behaviors that lead to positive psychosocial outcomes for athletes, as well as increased behavioral monitoring by coaches (Smith & Smoll, 2002). Experimental designs trained one group with the CET model and another with a technical skills training workshop; coaches that had been trained with CET had athletes who reported that their coaches were better teachers, and these athletes reported greater sport enjoyment (Smith, Smoll, & Curtis, 1979). A similar 9 experimental design with swimming coaches explored the effects of CET on athlete self-esteem (Conroy & Coatsworth, 2006); this experiment demonstrated that CET’s effects were moderated by age and gender, in that young swimmers, and female athletes who began the season with low self-esteem were most likely to see increases in self-esteem if they were in the intervention group. CET formed the basis of three notable coach education programs: the American Sport Education Program (ASEP), the National Youth Sport Coaches Association (NYSCA) program, and the Program for Athletic Coaches’ Education, (PACE). In one of the PACE workshops, Malete and Feltz (2000) found that CET was able to increase coaching efficacy, representing an attempt to examine cognitive outcomes for coaches resulting from a CET intervention. With the possible exception of this article, most studies of CET are focused on athlete outcomes, rather than examine the outcomes on the coach. These studies’ bias towards athlete outcomes appear to treat the coach as little more than a variable that stands in the way of a positive sport experience for children. These studies do not examine the intervention itself, but instead examine the intervention’s efficacy to create a change in psychosocial variables in athletes. CET interventions are focused primarily on the coach’s behavior, which does not fit with cognitivebehavioral models of behavior prediction and adoption, such as the theory of planned behavior (Ajzen, 1991). The theory of planned behavior takes a wider view of the factors that impact and predict behavior, stating that attitudes (a summation of strength of beliefs or expectancies, and their concordant values), subjective norms (social pressures experienced by the coach, and the concordant motivation to comply with each social pressure), and perceived behavioral control (the belief that you can perform a behavior that will achieve a particular outcome). A coach education intervention model based on the theory of planned behavior would take account of 10 these three factors that can influence coaching behaviors, rather than simply trying to shape behaviors alone. Recent research has flocked to examining the impact of the motivational climate, or the degree to which the sport climate supports the three basic needs of competence, autonomy, and relatedness, part of the much broader self-determination theory (Deci & Ryan, 2000), spurred in part by a self-determination theory-based model of the coach athlete relationship (Mageau & Vallerand, 2003). Studies have examined the effects of autonomy-supportive coaching on need satisfaction and self-perceptions in swimmers (Coatsworth & Conroy, 2009), increased subjective vitality in sport participants (Adie, Duda, & Ntoumanis, 2008), and physical and psychological well-being (Reinboth, Duda, & Ntoumanis, 2004). In Europe, the multi-nation Promoting Adolescent Physical Activity project (PAPA, www.projectpapa.org) aims to help coaches and sport practitioners enhance the motivational climate (i.e., promote task orientation, autonomy-supportive coaching) in order to increase positive physical and psychosocial outcomes from youth sport participation. Although the motivational climate and autonomy-supportive coaching research has shown benefits for athletes in sport, and despite its greater engagement with the cognitive and affective components of coaching, its focus on athlete outcomes tends to ignore the coach as little more than a moderator variable in the delivery of a quality youth sport experience for children. This research overlooks the coach as an independent actor, as a person with needs and demands, but who also holds the key for shaping the motivational climate. Research that examines coach perceptions of coach effectiveness training, as well as autonomysupportive coaching and motivational climate interventions, could be particularly useful in helping to understand how these interventions work (and why they fail), as well as helping sustain their positive impacts long after the researchers have left the coach to work with their 11 team without intervention support. Coach Education Research Swimming coaches learn how to coach through a variety of methods, ranging from informal, coach-driven approaches (e.g., personal sport experience; vicarious coaching experience; internet searches; interactions with other coaches; experimentation and reflection; communities of practice) to formal, curriculum-driven approaches (e.g., undergraduate courses, Foundations of Coaching program). Mentorship is another method for coach education that can combine both formal and informal elements of coach education. Informal, coach-driven methods. Personal sport experience and vicarious coaching experience form the foundation of most coaches’ knowledge (Lemyre, Trudel, Durand-Bush, 2007). Internet searches can provide coaches with established knowledge and can help inspire creativity. Interactions with other coaches can allow transfer of established knowledge, and provide a starting point for critical problem solving and self-inquiry (Lemyre, Trudel, DurandBush, 2007). Gilbert and Trudel (1999) have advanced the idea that master coaches are reflective practitioners (Schön, 1987), i.e., they experiment with new coaching methods and then reflect, using an iterative process of knowledge creation. Coaches also use communities of practice – knowledge networks of similar coaches (typically internet-based user forums) – to transfer, test, and advance knowledge (Culver and Trudel, 2008). However, it is not known how many coaches learn in this way. It is thought that coaches engage in such learning efforts to varying degrees; coaches who do not explore different means of coaching generally continue to coach as they were coached. Formal, curriculum-driven methods. Coaches with undergraduate degrees often have experience in coaching courses, and some may even obtain a coaching minor or specialization. 12 Such programs generally present generalized sport science topics (e.g., exercise physiology, biomechanics, motor learning, psychology) and pedagogical methods (e.g., practice management, sport pedagogy). The Foundations of Coaching program for USA Swimming presents coaches with a two-part course over two years, which introduces basic management concepts in the first year (e.g., athlete development, working with parents, running an effective practice, teaching stroke mechanics), and introduces sport science topics in the second year. The Foundations program assumes that the coach has not had prior formal coach education, and is designed to present knowledge sequentially, so that beginning coaches can run effective practices before they are introduced to sport science concepts. Mentorship. Mentorship offers a hybrid approach to coach education, in that it can help to enhance a formal coach education program through reinforcing the concepts taught, or providing assistance developing the coaching skills to deploy that knowledge. Mentoring can also serve as a coach-driven education method, where the mentor helps to guide a mentee’s inquiries in such a way that the mentee is able to learn and refine their critical thinking skills. Mentoring added a dynamic component to a coach education program developed for the National Wrestling Coaches Association, in that it was able to provide a connection between novice head coaches and experienced head coaches. This connection allowed for knowledge introduced through online modules to be transferred and adapted to a coach’s specific coaching context. Mentoring has provided an effective compliment to this online coach education program, and it shows promise to both reinforce the official curriculum, as well as provide a go-to source for the mentee coach in their quest to improve (Gould, Lauer, Driska, et al., 2012). In summary, coach education can occur through informal, coach-driven methods, as well as formal, curriculum-driven methods. The coach is often the difference maker in the process. It 13 remains to be seen if the coach’s assets and approach to learning is an entity (fixed) or learnable quality, but mentorship shows promise as a means to augment the coach’s approach to learning from curriculum-driven approaches. In addition, the use of influential coaches and sport leaders to shape coach attitudes towards learning and continual improvement may be able to augment the coach’s willingness to learn and engage in continual professional development. Evaluation of Coach Education Programs What makes evaluation different from the previous research on coaching behaviors, coaching effectiveness, and the ways in which coaches learn, is that evaluation would attempt to pull back to take a much broader view, taking into effect the specific context in which all of these interactions take place. The role of researchers is often to prove theories or demonstrate the efficacy of particular interventions for generalization to other contexts. However, what happens on the ground in most sport programs is often a product of local expertise that is rarely theory-driven. Coach education programs are often designed in response to the lack of theory or evidence driving practice at the ground level. Evaluations seek a detailed description, interpretation, and judgment of these programs’ effectiveness, working within the political and social constraints imposed by the organization, demonstrating their unique effectiveness (or ineffectiveness) in context. Despite the proliferation of the term evaluation in many studies of coach education, the term evaluation in this dissertation refers to program evaluation (discussed in greater depth later in this chapter). Gilbert and Trudel (1999) first advanced the concept of evaluating large scale coach education programs, specifically, the National Coaching Certificaiton Program (NCCP), administered by Sport Canada. They employed a single case study design utilizing the six-stage model of evaluation, developed through the human resource management literature (Brinkerhoff, 14 1987). The evaluation aims examined program delivery, useful knowledge from the course that was implemented, and followed up with recall interviews and structured observations of coaching. The program demonstrated minimal effects: the instructor did not deliver it effectively, there was little change in knowledge, and behaviors remained similar to baseline measures. The authors do not report on provision of their findings to the NCCP; instead, dissemination of these findings was made through the scholarly literature. Thus, in this case, dissemination of results did not lead to measurable program change for the NCCP. Bean (2011) examined the effectiveness and use of content stemming from an online coach leadership course for the National Wrestling Coaches Association (NWCA). This evaluation involved stakeholder interviews to solicit purposes and intended outcomes for the program participants, breaking them into macro goals (e.g., developing a broad awareness of their coaching roles, increased reflection) and micro goals (e.g., improve communication skills, develop a program vision). Results showed that coaches perceived the online course videos met their needs and delivered useful information. Although responses of the organizational stakeholders (NWCA) to the evaluation results are not presented, implications for the continual development and design of the technical elements of the program were shared with future program developers (Bean, personal communication, 2011). Cassidy, Potrac, and Mackenzie (2006) reported on their development and evaluation of a coach education program for a rugby football club (Code 1 of Rugby). This article details a curriculum design process that engaged senior coaches in the club, as well as the program’s delivery, its evaluation through interviews with eight coaches in the program, and the subsequent reflection on the evaluation results. These authors explain that the delivery of the program, intended as a means for delivering scientifically based knowledge for use by coaches, in reality 15 became an opportunity for coaches to discuss problems; they liken this process to the formation of a community of practice. In concluding their article, the authors discuss how their program evolved into a community of practice, referencing the curriculum analysis framework of Fullan (1991). Although this article provides an in-depth study of the entire process of coach education, from curriculum development, to delivery, to outcomes, and finally to reflection by the program designers, it appears to be strongly rooted in educational literature, thus becoming more of a reflection on learning than on coaching outcomes. Finally, a review of 46 publications (McCullick et al., 2009) showed trends in coach education literature between 1995 and 2008. Among these trends included a vast number of position papers relative to data-driven publications, a vast array of publishing outlets (n = 21), an overload of “consumer-driven research” (i.e., coach experiences), and a lack of systematic evaluation of large-scale coach education programs. These authors conclude that researchers “need to get their hands dirty” and become more involved with sport governing bodies in order to evaluate and improve their coach education programs. Coach Education for USA Swimming Coach education has been a feature of USA Swimming policy since 1996. Education is a requirement for coach membership, along with a criminal background check, CPR/AED training, safety training for swimming coaches, and athlete protection training (USA Swimming website, Coach Membership Requirements, 2014). Once a coach has obtained coach membership status, they are entitled to conduct the duties of a coach. The coach education program used by USA Swimming to educate coach members is the Foundations of Coaching program (occasionally referred to as the Foundations program), which is the subject of this evaluation. Foundations of Coaching program. The Foundations of Coaching course is an online, 16 curriculum-driven approach that teaches fundamental knowledge and skills for novice coaches. It is the most recent iteration of coach education efforts for USA Swimming, which began in 1996 with paper correspondence course methods, later evolving into a DVD and test-based course, and then finally into its current form as an online course. The course is produced in conjunction with the American Swimming Coaches Assocation (ASCA), an organization devoted to the professional advancement of swimming coaches. The current version of the Foundations course represents a major revision from the previous version of coach education employed by USA Swimming. The previous version required a coach to pass a 180 question online exam after watching two DVDs that outlined a comprehensive set of sport-science topics (e.g., exercise physiology, biomechanics, motor learning, sport psychology). Previously, a coach could work as a coach member of USA Swimming for one year before completing the Foundations course; the test had to be passed before a coach began their second year of coaching. USA Swimming club development staff expressed concerns that the previous version did not provide any basic coaching essentials to coaches before they started their first year of work as a coach, that the previous course was too “test-based,” and that the amount of material presented was overwhelming for a new coach. The previous model was discontinued in December 2013, although there are some influential administrators and coaches who believe that the previous model’s difficult testing requirements were appropriate and should be retained (USA Swimming Programs and Services Director, personal communication, 2013). The course is presented through a combination of guiding avatars that teach the principles of these lessons through a lecture format, the use of hypothetical problems to stimulate critical thinking (e.g., “Is Ryan, a five-year-old swimmer, old enough to compete?”), and the use of “talking head” interviews with experienced swim coaches to deliver information in lecture 17 format. In addition, the course provides videos that model ideal swimming coach practices followed by a coach debrief. The course is broken into two parts, Foundations 101 and 201. Foundations 101 must be completed as part of a coach’s initial requirements for coach membership (in theory, before a coach ever begins their on-deck coaching work with a USA Swimming affiliated club). The Foundations 201 course must be completed any time before a coach begins their second year of coach membership with USA Swimming. Foundations 101 takes approximately 90 minutes to complete, while Foundations 201 takes approximately four hours to complete; the user is free to pause and then resume their progress at their own will. Several sources of greater information for each of the topics are linked to the course. For instance, during the teaching of stroke mechanics, the user can access a document that provides information on teaching stroke mechanics on all four competitive strokes. At the conclusion of the online course, the coach is assessed using a test (25-questions for the 101 course; 40 questions for the 201 course). Benefits and drawbacks to formal education for swimming coaches. Providing formal educational experiences for coaches yields benefits and detriments for sport organizations like USA Swimming. Formal education can direct coaches’ attention to important issues in the sport (e.g., desired pathways for athletic talent development, risks for overuse injuries) that they would otherwise not have awareness of. Formal education also teaches effective coaching methods and strategies, as well as age-appropriate methods of interacting with athletes. Formal education may serve the purpose of screening out individuals who are not willing to conform to accepted norms for coaching behavior, or individuals who are unfit to work with children, due to past histories of child abuse (USA Swimming coach certification also requires that a coach pass a background check and take a course in the recognition and prevention of child abuse). 18 Coach education could also have unintended detriments. Creation of an official curriculum often suggests that there is only one correct way of coaching, while in reality, there may be several correct ways of coaching that fall within acceptable guidelines. The official curriculum may then have the unintended effect of squelching coach creativity. It should be noted that coaches deviating from established training methods through careful experimentation with different forms of training methods have, in many ways, contributed to the explosion of swimming talent in the past decade (Rushall, 2013). One of the unique strengths of the American system of swimming is the enormous variance in coaching styles, and many coaches (e.g., David Salo, David Marsh, Michael Bottom) have been rewarded for creatively deviating from dogmatic principles of swimming training. Coaching systems in other countries that have more rigorous educational systems have tended to promote coaches who are very scientific in their applications, who possess the highest educational credentials, but may lack adequate people skills and situational intelligence that are required for the management of athletes, especially at the elite level (Australian Sports Commission, 2013). While formal swim coach education may have these drawbacks, it is possible to imagine a system that can successfully allow for formal certification, and yet still allow for the range of low-risk, creative approaches to training that have pushed the sport forward in recent years. However, three critical issues have yet to be resolved regarding the effects of coach education. The first issue is the precarious link between coach attitudes, knowledge, and skills, and actual coaching behaviors. The assumption that increased knowledge alone will lead to desired behaviors lacks evidence (Ajzen, 1991). A range of factors other than the coach’s own attitudes and knowledge influence behavior, such as social pressures (i.e., “Does my boss want me to develop long-term talent, or produce an age-group champion this year?”), and behavioral control 19 beliefs (i.e., “Do I have the knowledge and skills to develop athletic talent?”). The second issue is that recent evidence supports the notion that successful coaches learn according to their own intentions (Werthner and Trudel, 2006), and can obtain knowledge from a range of sources if they feel that this knowledge addresses an issue that they are experiencing. While this supports the notion that coach-driven educational methods have merit, it also suggests that official curricula may be “cherry-picked” for knowledge that provides immediate value to the coach. Coaches in many studies (Erickson, Bruner, MacDonald, and Côté, 2008; Lemyre, Trudel, and Durand-Bush, 2007; Wright, Trudel, and Culver, 2007) have indicated their willingness to use material from official curricula and formal educational sources, which provides evidence that formal education has value to coaches. One coach participating in a formal coach education program noted, “for me, if what I hear makes sense, then I am willing to try it and see how it might help” (Werthner & Trudel, 2006, p. 207). However, it is also quite apparent that the coach is a major variable in the educational process, and that a one-time, online educational program may not be adequate to shape coach attitudes about the importance of all topics in the official curriculum. The third issue is to ask a broader question about USA Swimming’s goals for professional development, summed up by the following questions: how is it that we want coaches to think and act, and what are the best ways to achieve this outcome? If we want coaches to possess certain knowledge, what methods can USA Swimming use, and how effective are those methods? For instance, can an online lesson with assessment deliver knowledge to 2,000 coaches nationwide, and then going a step further, can it change attitudes, skills, and ultimately, actual coach behaviors? Or, if we want coaches to possess refined peoplemanagement skills and situational intelligence, what methods can be employed, and how 20 effective can we expect them to be? For instance, could these types of advanced knowledge be delivered online, or would these efforts require face-to-face learning, through coaching clinics or mentorship? A hierarchy of coach learning objectives that matches coach-development goals to appropriate, evidence-based pedagogies could help to streamline and synergize coach education and development efforts, helping USA swimming to direct financial and human resources to areas where they are best utilized. Although this hierarchy could benefit USA Swimming, its creation is beyond the scope of this project; however, evaluating existing online methods for their effectiveness does fall within the scope of this project. The Experience of the User: Coach Reactions One of the critical assumptions of the Foundations of Coaching 101 program is that an online course can train a first-year swimming coach. Unpacking this assumption, several factors influence the learning experience for the coach, which calls for an investigation of user experiences in the course. Learning is not simply the cognitive acquisition of facts, knowledge, and skills; it is a complex process that involves memory, emotion, attention, and cognition. Users experience an emotional reaction to the content presented in Foundations 101, so how does this emotional reaction drive learning outcomes? For instance, if a coach completely disagrees with the model of athletic talent development that is taught overtly in Foundations 101, this disagreement is not experienced simply as a cognitive rejection of knowledge; it may be accompanied by an emotional reaction to the experience, which may lead to the coach rejecting the content in the course. Or, perhaps the coach might have an unpleasant reaction to content in the course that is taught by an avatar coach named John (an avatar is a digital representation of a person, who, in this online course, serves as a teacher). How will this emotional reaction shape learning in the course? 21 Conducting an inquiry into the user experience can be theoretically guided by the technology, pedagogy, and content knowledge (TPaCK) framework (Mishra and Koehler, 2006). The framework suggests that learning outcomes are shaped by content (e.g., effective swimming practice organization strategies, effective methods to teach swimming strokes), pedagogy (e.g., watch a video of an expert coach explaining a concept, and then answer an application question on your own), and technological knowledge (e.g., a user’s ability to successfully navigate the web-based layout of the online course). Two questions persist about a first-year coach taking the Foundations 101 course. First, what factors go into the online learning experience that enable one coach to be more successful than another coach, i.e., why does one coach learn and apply most of the coaching knowledge from the course, while another coach does not? Second, in the long-term, would success in the Foundations 101 course predict actual long-term coaching success (e.g., swimmer achievements)? These questions interest USA Swimming because they have invested a significant amount of financial and human capital into coach development efforts and the results of those efforts have important ramifications for both the growth and performance success of the sport. With all of this capital invested in coach development, USA Swimming has only two guaranteed official contacts with its member coaches, which are the Foundations 101 and 201 programs. Thus, there is a need to investigate how effectively this learning experience leads to beneficial coaching outcomes. At this point, it is necessary to delineate how such an investigation such as the one proposed in this dissertation might proceed. One approach to investigating the effectiveness of the Foundations 101 program is through the practice of program evaluation, a form of social science inquiry that is highly related to research, yet entails greater involvement of and 22 responsiveness to stakeholders in the program of interest. The following section provides a brief description of the broader field of program evaluation, as well as a description and rationale for the specific type of program evaluation (i.e., utilization-focused evaluation) employed for this dissertation. Program Evaluation “Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs in ways that are adapted to their political and organizational environments and are designed to inform social action to improve social conditions,” (Rossi, Lipsey, and Freeman, 2004, p. 16). Program evaluations have been used to examine the effectiveness of child development programs such as Head Start, after-school youth development programs, health enhancement and disease prevention programs, university cooperative extension programs, and government-run job training centers, to give a sense of the breadth of programs that are typically evaluated. Program evaluations often serve as an important step in the program funding process, as funders are eager to see that the program is effective before they continue to donate money. Evaluations have two basic components: an accurate description the program, and a judgment of the program’s performance in meeting the social need. Evaluations vary in their levels of description and judgment, and these levels are largely determined by the purpose of the evaluation (see following sections for greater explanation). In short, the judgment rendered in an evaluation, the evaluator’s responsiveness to the information needs of a program’s stakeholders (rather than being held to an externally-imposed methodology), and in some cases, the epistemological concerns and assumptions of the evaluator (i.e., privileging “local theory” over established social scientific theories), are the elements that separate an evaluation from applied 23 research or action research. Program evaluators must carefully consider the political and social context in which they are working, and must match the type of evaluation, especially the level of judgment, to the nature and status of the program being evaluated. Programs in their formative stages should be evaluated to improve function, or to provide useful information about the program to key program stakeholders. Evaluators focused on providing useful information to key program stakeholders typically fall into the utilization-focused evaluation camp (Patton, 2008; 2011). The dimension of “use” adds complexity to the “description plus judgment” definition of an evaluation given by Rossi et al. (2004). Michael Patton (2008; 2011), originator and practitioner of utilization-focused evaluation, espouses the idea that evaluations must provide important information to key program stakeholders who hold decision-making power in a program; these people are referred to as “primary intended users” of the evaluation. In utilization-focused evaluation, the end goal that the evaluator constantly works toward is producing an evaluation that will be used by the primary intended users. The theory and application of utilizationfocused evaluation will be discussed later in this chapter. While the primary focus of evaluation research is to provide useful information to those working in the programs being evaluated, this does not mean that evaluation studies cannot make contributions to scientific knowledge. The analysis methods employed give the same attention to rigor that would be established in a social scientific inquiry. Evaluation findings can be written into a scientific journal article, and findings can be extrapolated to other contexts (Patton, 2011). Hence, while this dissertation focuses on providing a utilization-focused evaluation for USA Swimming, it also anticipated that the knowledge gained can contribute the emerging body of knowledge on how coaches learn, and the effectiveness of online learning. 24 Evaluation methods. It is crucial to match the type of evaluation to the current status of the program and its stakeholders. For instance, it is not practical to conduct an efficiency assessment for a program that has just been created. Rossi, Lipsey, and Freeman (2004) present five steps in an evaluation hierarchy, where early steps represent formative or process evaluation methods (e.g., needs assessment, evaluability assessment, logical framework or theory of change, program process), and the latter steps represent methods used for outcome evaluations (e.g., program outcomes or impact assessments, efficiency assessments). Needs assessment. A needs assessment serves to define a problem (e.g., lack of educated coaches), its causes, the surrounding issues and politics, and if a program is needed to address the problem (Rossi, Lipsey, and Freeman, 2004). Needs assessments are not restricted to situations where no program exists to meet the need; they can be performed for existing programs, and can be a useful way to take a look at the problem that an existing program is designed to address (Witkin & Altschuld, 1995). Successful needs assessments prioritize needs and select solution strategies that are acceptable and financially feasible. Evaluability assessment. An evaluability assessment asks the question, “is the program ready to evaluate?” Evaluability assessment is crucial when there is a need to clarify the purpose of the program, a need to streamline and make programming consistent across wide geographic areas, or when there is a need to increase evaluation capacity (Wholey, 2004). An evaluability assessment can assess congruence between program theory and program design, i.e., can a program designed to change behaviors logically succeed, given its programming? Many of the techniques of evaluability assessment are built into Patton’s utilization-focused evaluation (2011), which will be discussed in the Methods chapter. Logical framework or theory of change. Effective programs operate with a set of 25 governing principles that theoretically guide the actions of program staff. The program theory may be explicitly defined and followed by staff members, or it may be an implicit set of assumptions and shared values held by a tight-knit staff. A logical framework (or logic model) is a linear sequence of program assets, activities, outputs, and participant outcomes that is descriptive and does not provide causal mechanisms (W. K. Kellogg Foundation, 2004). A theory of change differs from a logical framework in that it is typically non-linear and shows both interactions as well as causal mechanisms. Useful theories of change generally adhere to scientific principles of behavior change. For example, to use an established behavioral change model, the theory of planned behavior (Ajzen, 1991) states that attitudes (i.e., beliefs, values), subjective norms (i.e., social pressures), and perceived behavioral control (i.e., self-efficacy) both interact with each other, and directly effect behavioral intentions, which subsequently influence actual behavior. Program process. Program process evaluation is the process of identifying all of the factors that shape a the program that is “received” by the participant, including the program’s intended mechanisms as well as unintended effects. Process evaluation is the beginning of what Patton (2011) calls “reality testing,” i.e., “how close to the theorized program is the actual program that the participants receive?” Common measures to investigate in a program process evaluation include the reach of the program, the dosage delivered, the dosage received, program fidelity, program implementation (i.e., a quantifiable index of how well the program works), and participant satisfaction (Lianant and Steckler, 2002). A major portion of this evaluation will investigate this program process measures, in accordance with Patton’s theory of utilzationfocused evaluation (2011). Program outcomes. Program outcome evaluation is the process of measuring the 26 performance of a program in meeting its stated outcome goals, as defined by the program’s logic model or theory of change (Wholey, 1996). Performance measurement is a type of outcome evaluation that can provide continual feedback to program directors regarding the performance of the organization (Poister, 2004). An efficiency assessment examines a program’s effectiveness in producing stated outcomes relative to the capital (financial, human, otherwise) invested; this is sometimes referred to as a cost-benefit analysis. Evaluation theories. The field of program evaluation has grown out of the culture of inquiry in the social sciences, partially in response to the need for accountability and standards in the administration of social programs implemented in the 1960s. However, different traditions in evaluation have emerged to address a range of evaluation priorities and needs. Alkin & Christie (2004) propose an “evaluation theory tree,” where a trunk of social inquiry and accountability branches into three main branches: methods, valuing, and use. Into these three branches, the majority of evaluation theorists (and theories) can be divided, based on their view of the primary purpose of evaluation. Methods theorists are primarily concerned with executing sound inquiry methods. Valuing theorists believe that “what distinguishes evaluators from researchers is that evaluators must place value on their findings and, in some cases, determine which outcomes to examine,” (Alkin & Christie, 2004, p. 32). Notable valuing-theorist Michael Scriven states that society requires valuing: “Bad is bad and good is good, and it is the job of evaluators to decide which is which,” (Scriven, 1986, p. 19). Use theorists believe that the primary purpose of an evaluation is to assist key program stakeholders in the process of program decision-making and organizational change (Alkin & Christie, 2004). Therefore, the primary job of the evaluator is to work with key program stakeholders to provide the highest quality information. Michael Patton, a major advocate of use-theory, contends that evaluators should involve key stakeholders (called 27 primary intended users of the evaluation) throughout an evaluation, to determine the purpose of the evaluation, to select key evaluation questions, and to determine methods of inquiry, data collection, and data-analysis (Patton, 2011; 2013). Patton contends that despite conventional wisdom, most stakeholders do have strong opinions on the credibility of data collection methods, statistical sampling, and evidence. Use theorists end goal is promoting use of the evaluation by primary intended users; the underlying belief espoused by Patton (2011) is that the more the evaluation has meaning and value to the primary intended-users, the more likely they are to use its findings. Utilization-focused evaluation. To ensure propriety, evaluations should be guided by a methodological theory or framework; the framework of utilization-focused evaluation (U-FE) (Patton, 2008; 2011; 2013) guides this evaluation. The central tenet of U-FE is intended use by intended users, meaning the evaluator’s primary goal is producing an evaluation that will have utility for primary intended users, i.e., those individuals who make important decisions regarding the program and its implementation (Patton, 2011; 2013). When considering the five professional standards embraced by the American Evaluation Association (AEA, 2013; Yarbrough, Shulha, Hopson, and Carruthers, 2011), Patton says, “utility is the first standard for a reason,” (Patton, 2013). What makes a good evaluation theory, and how well does utilization-focused evaluation (U-FE) measure up? Miller (2010) proposed five criteria by which an evaluation theory (or theoretical framework) could be judged: operational specificity (i.e., does the theory provide clear guidance on what to do?); range of application (i.e., in what contexts can the theory be used, and are there any contexts where it should not be used?); feasibility in practice (i.e., does the evaluator have the skill and expertise to apply the theory?); discernible impact (i.e. if the 28 theory emphasizes a desired outcome that will result because of the evaluation process, does that desired outcome actually happen?); reproducibility (i.e., could another evaluation reproduce this evaluation?). U-FE performs well against these five criteria. Patton’s recent text (2011) has operationalized U-FE into a step-wise process, and U-FE can be applied to a range of contexts, especially those contexts where intended users are looking for increased understanding of their program. In this case, the evaluator possesses a strong knowledge of swimming and swim coaching, has an understanding of the political context of USA Swimming, and has a strong background group facilitation and “qualitative” data gathering techniques, all of which are required skills and knowledge to make a utilization-focused evaluation feasible in practice (see the methods section for a discussion of evaluator preparedness and reflexivity). The discernible impact of a utilization-focused evaluation is intended use by intended users, and this evaluation’s discernible impact (i.e., the degree of use and implementation by primary intended users) can be assessed. U-FE is somewhat problematic in its reproducibility; for instance, many of the methods used to build a program theory require the evaluator to interact and be responsive to the primary intended users, and therefore often go “off script.” Detailed notes recounting such facilitative exercises can aid reproducibility, but ultimately, one evaluator may have a different interpersonal style that makes certain exercises difficult to reproduce. To guard against reproducibility flaws, the evaluator should be clear and transparent when describing methods in published evaluation reports and subsequent write-ups for scientific journals. Matching evaluation theory to the evaluation context. Utilization-focused evaluation (U-FE) addresses four specific needs of this specific evaluation context. First, the evaluation stakeholders are not very familiar with the practice of program evaluation, although there is an increasing demand for “metrics” to demonstrate program effectiveness. Prioritizing use will 29 keep primary intended users involved, it will increase transparency and familiarity with the evaluation process, and will increase interest and value of data-driven decision-making. Second, most sport organizations are typically very pragmatic organizations, as many employees are former coaches (who are notoriously pragmatic individuals). The criterion of “use” fits best into this pragmatic mindset, as opposed to the criteria for other evaluation frameworks, such as participatory evaluation (Cousins and Whitmore, 1998), which prioritizes increasing participation from all levels of involvement with the program, and increasing the amount of stakeholder involvement in evaluating the program. Third, there is not a mandate for valuing or judging of this program, therefore, to use an evaluation framework that stresses a summative judgment on the value of this program would be misguided. Fourth, U-FE does not espouse rigid adherence to one methodological school of thought (i.e., quantitative vs. qualitative methods, randomized control trials vs. quasi-experimental methods); instead, it holds that evaluators should use methods that will present meaningful findings that primary intended users will actually use. Given the lack of mechanistic clarity for the Foundations 101 program, not to mention that it sits within the larger context of coach and program development that is not well articulated, being allowed a flexible methodology will allow the evaluator to probe at length into the mechanism of the program. Finally, evaluator reflexivity is an important characteristic, as the evaluator’s biases and assumptions will shape the evaluation. While the purpose of the evaluation should drive the methods employed by the evaluator, U-FE fits both the facilitative style of the evaluator, and is consistent with his core beliefs about the appropriate involvement of stakeholders and their “native knowledge.” Given that this is the first evaluation conducted independently by the evaluator, using an evaluation framework that is close to the evaluator’s own predispositions should aid consistency during difficult situations. 30 Rationale for this Evaluation There is a need for a greater understanding of how coach education meets the needs of coaches, of how it delivers knowledge that can be contextualized and applied by a coach, and for what purpose it serves in the continual professional development of a coach. There is also a need for a greater understanding of how online courses work, specifically, how effectively they transmit knowledge, and how they lead to desired changes in coach behavior. Understanding coaches’ needs for knowledge and professional development, as well as understanding their reactions to and experiences with an online course, are critical to understanding how this knowledge is delivered. USA Swimming has a need for understanding how some 2,000 coaches use this mandatory program, what they gain and retain from the experience, their attitudes and beliefs about it, how it leads to desirable coaching outcomes (i.e., employing coaching techniques that optimize athlete development), and what role it plays in their larger coach and organizational development strategy. Specific purposes for this evaluation will be discussed in chapter four, as they arise from focus group meetings with key program decision-makers at USA Swimming, but the general rationale for this evaluation is as follows. First, little is known about the efficacy of online coach education courses to deliver a program that is capable of changing coaches’ knowledge, skills, and attitudes. Second, little is known about how well coaches understand, accept, and retain the knowledge presented in an online course. Third, if knowledge is gained and attitudes and behaviors can be shaped, to what extent do coaches apply the knowledge presented in an online course? Fourth, if more can be learned about how an online coach education program shapes knowledge, attitudes, and behaviors, can specific learning outcomes in the course be useful predictors of actual coaching behaviors of first-year swimming coaches? 31 Beyond the rationale of this evaluation delivering critical knowledge to USA Swimming, this study has a greater merit within the field of coaching science. At the present time, a range of sport governing bodies (e.g., United States Tennis Association, USA Volleyball, USA Football) and coach education agencies (American Sport Education Program, Positive Coaching Alliance) have designed (or are in the process of designing) online coach education programs to reach a widespread audience of coaches. However, no published efforts have been made to evaluate these programs. The provision of evidence to guide practice would greatly assist these organizations, and ideally would improve the sport climate for children who participate in these sports. 32 Chapter 3 - Methods Program evaluations often differ from traditional social science research, in that methods may evolve in response to the evaluation contextual demands. Dissertation manuscripts for social science research typically follow a predictable formula as well, where literature is reviewed, a purpose is stated, a method is planned, revised, and then executed, results are reported and then discussed. Because this manuscript differs from this traditional format, the following overview will be of assistance to the reader. The method section (Chapter 3) will provide a general outline of the methods of gathering and analyzing data employed by this program evaluation; this outline will be presented in a temporal format to demonstrate the sequential nature of the program evaluation process. Due to the range of inquiry and analysis methods, specific methods of inquiry and analysis are presented in the results section (Chapter 4), and discussion of these results frequently appears in order to provide clarity for the reader. Throughout the method section, the terms author and evaluator are used interchangeably and refer to the author of this dissertation. The utilization-focused evaluation (U-FE) framework guided this evaluation (Patton, 2008; 2011; 2013). The evaluation occurred in four phases, which are diagrammed in Table 1. 33 Table 1 - Evaluation timetable Phase Stakeholder Engagement Research/Inquiry Process Phase I Prelim discussion with Program and Services Director Selection of stakeholders and primary intended users Understanding the evaluation context Preparing client for evaluation Risk assessment Evaluator readiness and competency exercises IRB exempt application and approval Successfully propose dissertation, and make needed revisions Phase II Clarify evaluation purposes Focus priority evaluation questions Goals clarification exercise Program objectives and desired outcomes Examining barriers and exceptions Simulate potential findings Negotiate methods Write interim report Sign evaluation agreement MSU IRB revisions as needed Phase III Deliver interim findings report in Colorado Springs Facilitate discussion of interim findings Negotiate any issues with the rest of inquiry Quantitative inquiry Examine demographic statistics and test score data for statistical relationships Phase IV Qualitative inquiry/analysis Solicit interviews (21 coaches) Conduct interviews Transcribe Prelim deductive analysis meeting with triangulation after 3 interviews Continued deductive analysis Prepare interim report Finish deductive analysis Complete inductive analysis Description of findings Write dissertation report Defend dissertation In phase I, the author prepared to conduct the evaluation, and gained a deep understanding of the program’s history and its context with the organization and the swimming community, which allowed him to plan appropriate evaluation methods. In phase II, the author engaged key program decision-makers (referred to as primary intended users in the utilizationfocused evaluation literature) through a process of informal in-person discussions and a structured focus-group meeting, which led to an agreement on how the evaluation was to be conducted. Phase III involved data collection, preliminary analysis of findings, and presentation of interim findings to key program decision-makers. Phase IV consisted of a deductive and 34 inductive analysis of qualitative data, the description and interpretation of major evaluation findings, and the formative evaluation (or judgment) of the program’s ability to meet its intended purposes. Phase I: Preparing to Conduct the Evaluation Phase I of the evaluation can be conceptually thought of as “tilling the soil,” preparing both the evaluator and USA Swimming (the client) for the evaluation. The author conducted several phone interviews with the Programs and Services Director and the Director of Field Services to gain an understanding of the Foundations program, its history, and the needs it fulfilled for the organization. Through this process, the author gained a working knowledge of the major concepts delivered by the program, and some preliminary questions of interest for the stakeholders in the organization. It is critical to note that the author did not foreclose on evaluation methods during this phase, although he did propose potential directions for how the evaluation could be used to gain more information about the program and its effectiveness in an effort to educate these key program decision-makers on what the evaluation could potentially entail. During Phase I, the author assessed his own readiness and competence with regards to conducting the evaluation relative to the five guiding principles for evaluators (American Evaluation Association, 2004), notably the four standards of evaluator competence: technical competence, cultural competence, explanation of evaluator limitations, and continual improvement (see Appendix B: Assessment of Evaluator Competencies). Throughout engagement with key program decision-makers, the author employed a facilitative style intended to encourage collaboration. This followed from a review of evaluation use (Cousins, Donohue, and Bloom, 1996) that suggested the degree to which the evaluator successfully employed 35 collaborative methods was the strongest predictor of evaluation use. Following from the U-FE framework, as well as the qualitative research tradition (Glesne, 2011), the author wrote a general statement of reflexivity, which allowed him to address potential biases and professional weaknesses that might otherwise compromise the quality and integrity of the evaluation. The general statement of reflexivity is contained in Appendix C: Evaluator Statement of Reflexivity. The author worked with the Programs and Services Director to identify key program decision-makers. The term key program decision-makers was used instead of the term primary intended users (commonly used by the U-FE literature) to clarify that the evaluation was intended to aid and guide future decision-making by key individuals with influence over the Foundations of Coaching program. These key program decision-makers included a total of seven individuals: the Programs and Services Director, the Director of Field Services, three Sport Performance Consultants, a Programs and Services Coordinator, and the Club Development Division Vice President. It should be noted that in order to best promote use of evaluation findings, a review of use found that the ideal median number of primary intended users is six (Cousins, Donohue, and Bloom, 1996). Phase II: Framing the Evaluation Situation analysis. The beginning of Phase II served to complete a situation analysis of the program in the context of the parent organization. The situation analysis, an ongoing process that began in phase I, addressed four areas: understanding the program (i.e., history, goals, participants, staff, budget), identifying the interests of stakeholders (i.e., interest alignments and conflicts, political context), evaluation history (i.e., prior experiences with evaluation, current monitoring/evaluation methods, evaluation capacity), and the decision-making context (i.e., purposes of the evaluation, future decisions to be based on the evaluation, program facing 36 uncertainties). The analysis was conducted through informal interviews with key program decision-makers, a process which also served the purpose of the author building rapport with these individuals. Building rapport with key program decision-makers. Before meeting with key program decision-makers, the author applied for IRB clearance, for which this project was deemed exempt. The author conducted a set of informal interviews with key program decisionmakers at the USA Swimming home office. These meetings were conducted with the purposes of building rapport with these individuals, in which the author leveraged his significant experiences as a former swimming coach to convey his knowledge of the sport, and discussed several aspects of research and evaluation methodology, so as to gain credibility in the eyes of key program decision-makers. These interviews did not follow a script, but instead focused on asking each individual to describe what he or she did in their day-to-day work. Through these discussions, the author ascertained a great deal of knowledge regarding the challenges faced by these individuals in their jobs. The individuals who were coach and club development consultants provided their unique insights into the types of problems frequently encountered by most USA Swimming clubs (e.g., coach fears about losing athletes to rival clubs if they cannot improve swimmer performance), and helped to provide a picture of the typical coach and typical program that the Foundations program is intended to reach. The individuals who worked to administer the program enriched the author’s working knowledge of how the program had been redesigned and reconfigured into the online format, as well as providing a picture of how data (e.g., scores from tests given at the end of each course) moved from the test results to the coach to USA Swimming. These interviews served to build the author’s rapport with the key program decision-makers and deepened his understanding of the Foundations program, and explained 37 USA Swimming’s broader commitment to coach and club development. Conduct focus group meeting with key program decision-makers. To elicit a greater program understanding and frame the evaluation, the author conducted a three-hour focus group meeting with six of seven key program decision-makers (the Division Vice President was unable to attend) at the USA Swimming main offices in Colorado Springs, Colorado. This meeting had four specific objectives: (a) identify the needs that the Foundations of Coaching program fulfills; (b) explore the mechanisms by which the Foundations of Coaching program works; (c) engage in contingency thinking and commit to reality testing; (d) solicit key program decision-makers’ perceptions of use of evaluation process, i.e., increased awareness of job functioning as a result of participating in the evaluative process (Patton, 1998). The author engaged key program decision-makers through the use of a participatory style of facilitation, a style of questioning that has received support in both the utilization-focused evaluation (Patton, 2011) and participatory evaluation (Cousins & Whitmore, 1998) literature. Participatory facilitation and decisionmaking strategies were planned using the help of a guide (Kaner et al., 2007). To address the objectives for the focus group meeting, the evaluator divided the meeting into three 45-50 minute sessions (with 10-15 minute breaks). To facilitate evaluator recall, the entire meeting was audio recorded (with the verbal consent of the meeting participants). The first 45-minute session of the focus group sought to answer the following question: What needs does the Foundations of Coaching program fulfill? The evaluator posed the following question to the key decision-makers: What is the purpose of the Foundations of Coaching program in the first place? Or, what is the “problem” that we are trying to fix by having a coach education program in the first place? Each participant wrote their own responses, then shared them using a structured-go-around (i.e., each participant reports their findings to the group in order, and 38 comments are held to the end) as a means of gathering all participant understandings quickly and without too much judgment. Answers were tabulated using a sticky-note poster. After looking at the comprehensive list generated through this process, the evaluator posed to the following question to the participants: To what extent does the Foundations of Coaching program solve this problem? Participants were asked to ponder their responses to this question aloud. The evaluator did not employ a structured response system for this exercise in order to allow for organic discussion as the participants pondered the nature of the problem. While some participants were more vocal than others, all participants contributed to the discussion. The evaluator recorded participant responses using a sticky-note poster. The purpose of the second focus group session was to provide an outline for how the Foundations of Coaching program works. To do this, the evaluator employed an activity to engage participants in “backwards thinking,” by envisioning how a coach would be different as a result of taking the course. Participants were divided into two groups. One group generated a list of changes in coach thinking, specifically, changes in knowledge, values, beliefs, or attitudes. The other group generated a list of changes in coach behaviors. Each group recorded their list on a large sticky-note poster, and took turns presenting back their major findings. Discussion was allowed to develop organically. The evaluator played the role of facilitator, occasionally challenging the participants to explain how a particular change might be created, or pointing out the underlying assumptions that were being made about a particular change. This information was used to generate a list of coach outcomes resulting from the Foundations course. Before taking a break after the second session of the meeting, the evaluator asked meeting participants to “picture the worst club that you have ever observed,” and consider whether or not these changes would still occur under those conditions. 39 The evaluator assumed a more directive style for the third session of the focus-group meeting, typically employing a question/answer format with constrained discussion, while recording major ideas on a whiteboard as the discussion ensued. That evaluator asked participants to consider a coach in a poor coaching environment (e.g., poor leadership, poor resources), and consider how the coach education process might be different when compared to a coach in a well-supported coaching environment. The second question asked the participants to ponder what a real coach (i.e., a coach who had recently taken the course) would say about the program. Would our assumptions from the previous two sessions be off-base? What would be different? Following these questions, the evaluator engaged key program decision-makers in contingency thinking by asking, “who are our biggest skeptics on this program, and what would they say about our conclusions today?” A second question provided a simulation of negative evaluation findings, one of the core tenets of evaluation planning with utilization-focused evaluation (Patton, 2011). Meeting participants responded to these negative findings in order to identify potential program weaknesses that would need to be explored in greater depth through the evaluation process. The activities undertaken in this final session of the focus group meeting was part of the process that Patton (2011) calls “committing to reality testing.” It puts the program in a negative spotlight, and puts key program decision-makers face to face with potential negative outcomes of the evaluation, allowing them to think about how they would respond to negative findings long before any are actually found. When key program decisionmakers are threatened with negative findings for which they have not been prepared, this can threaten the use of the evaluation, as key program decision-makers often try to find fault with the evaluator or the evaluation’s inquiry methods. In effort to better understand process effects (i.e., the effects on participants arising from being engaged in the evaluation process), at the close of 40 the focus group meeting, the evaluator asked each meeting participant to reflect on what had been eye-opening or particularly useful for them during the meeting. Each participant then presented their take-home findings in a structured-go-around format (Kaner et al., 2007). At the close of this meeting, the evaluator recapped findings from the meeting and outlined the next steps for the evaluation. Production of interim report and evaluation agreement. The final step of Phase II was for the evaluator to create a draft interim report, which summarized the major findings from the meetings and correspondence in Phases I-II, delineated uncertainties, prioritized evaluation questions, and proposed future directions for the evaluation. All key program decision-makers reviewed the draft interim report, providing comments and revisions as needed, which were addressed in a revision of this interim report. Because this report was meant for the internal consumption of USA Swimming decision-makers, and contained sensitive and confidential information, it will not be included in this dissertation. However, this document was critical in helping the author demonstrate consensus amongst key program decision-makers with regards to what the evaluation would entail, and served to keep the evaluator grounded, making sure that the evaluation addressed the concerns and purposes identified and agreed upon by all parties. These specific purposes of the evaluation will appear in the Framing the Evaluation section of Chapter 4. The interim report proposed a qualitative investigation into coach perceptions of the Foundations of Coaching course, and identified the support and resources necessary from USA Swimming in order to produce these evaluation results. Phase III: Data Collection, Preliminary Analysis, and Presentation of Interim Findings Once the author and USA Swimming decision makers were prepared for the evaluation and the evaluation itself was framed, the evaluator initiated phase III of the evaluation. Phase III 41 consisted of creating a sample of coach participants, developing an analytic framework to guide the study, conducting preliminary interviews and analyses, conducting the main interviews and analysis, and holding an interim findings meeting for key program decision-makers. Each process is outlined in the following sections; greater depth of detail is provided in chapter 4. Creating a sample of coach participants. USA Swimming provided the evaluator with two datasets. The first dataset came from the SWIMS database (the database used to maintain coach member information), and contained information for coaches who had completed their coach education requirements. The second dataset came from ePath Learning (the company which conducts the Foundations course), and included test-score data matched on the e-mail address and first name/last name of the coach who had taken the program. A total of 3,460 coaches had completed some aspect of the coach education requirements; however, this evaluation was focused on the entire Foundations of Coaching program, so the potential sample was restricted to coaches that had completed Foundations of Coaching 101 & 201, as well as the rules and regulations test. The author then matched coaches from the first dataset with their corresponding test-scores from the second dataset, creating one master dataset that was used for subsequent analysis. The master dataset included a total of 1,324 coaches who had completed all of their coach education requirements, and who had been matched to a corresponding score. Demographic data from USA Swimming appearing in the dataset (e.g., gender, ethnicity, employment level) was converted from a text-string into numeric codes. This dataset was then exported to IBM SPSS (Version 22). Descriptive statistics or frequency distributions were calculated for each variable to check for normality and homogeneity of variance. Eleven hypotheses were developed to explore potential relationships between demographic variables (i.e., age-bracket, gender) and outcome variables (e.g., test-scores); full explanation of these 42 hypotheses along with results appear in Chapter 4. Based on results from hypothesis testing, the evaluator created five sampling strata in the following order: coach age (18-23, 24-28, 29-35, 36-50, and 51+), then high (96%+) versus low (80-84%) test scores, and then coach gender. Within each strata, the evaluator purposively identified coaches that would maximize diversity based on race/ethnicity, and then solicited these coaches via e-mail to participate in a phone interview. Coaches who responded to the email were accepted on the basis that they helped to maximize diversity within each strata wherever possible. A copy of the solicitation e-mail can be found in Appendix D: Solicitation Email. Analytical framework for qualitative inquiry. The analytical framework that guided the design of the interview and data collection process was that of simultaneous deductive/inductive analysis, also called abductive analysis (Patton, 2002), which mixes aspects of traditionally separate deductive and inductive analytical methods. In this approach, a deductive method is used to provide an overarching framework for the data collection, by determining the categories of questions that are used to gather information about the program, driven by the purposes of the evaluation. In a traditional deductive analysis, the researcher asks questions and seeks answers only so far as to directly answer the questions. However, using the abductive approach, the questions are more open-ended, allowing for participants to paint a broader picture and answer interview questions freely, providing a much broader scope of how the program is actually used. For instance, the interview asked coaches what was useful in the Foundations course (or, “what stood out for you?”), instead of asking about each topic specifically (e.g., “tell me what you thought of the section on mental skills”). This simultaneous deductive/inductive approach allowed the evaluator to organize a large amount of interview data 43 around sensitizing concepts (e.g., salient knowledge from the program, attitudes and behaviors resultant from the program). Once collected, this data could be organized inductively, which enabled it to reflect the nature of how participants were actually using the program. The following analogy may be useful for describing the analytical procedure as it was planned an conducted. The interview was designed to produce excerpts of transcripts that were organized around different sensitizing concepts. Picture each excerpt as a colored marble, and imagine more than 1,000 colored marbles in an unorganized pile. The deductive process was used to gather marbles of similar colors into separate color-coded bins. Once sorted into colorcoded bins, an inductive process was used to organize them into themes that described their nature (e.g., size, shape, markings), allowing patterns in the data to emerge more naturally. This type of analytical framework is useful when the evaluator or researcher has a general sense of how the program works, or has developed a logic model that explains how the program works, but is seeking greater conceptual clarity on the particular nature and individual differences of how these program elements fit together for different participants (Patton, 2002). Using a concrete example from this inquiry, the evaluator used a sensitizing concept (or deductive bin) of salient knowledge (i.e., useful topics) from the course. The questions were specific enough to guide participants to provide examples of useful course content, but not so specific to ask about every content area from the course. Employing this process with the entire sample of coaches allowed the most meaningful topics to emerge and rise to the top. The deductive/inductive process allowed the evaluator to avoid the confirmation bias frequently observed in purely deductive qualitative interview methods, and at the same time, structured the analysis enough to answer the evaluation questions. Not all topics addressed by the interviews used this deductive/inductive approach. For 44 instance, in assessing particular coaching behaviors of interest that were identified in the preliminary logic model developed in Phase II, questions were directly leveled at participants regarding their adoption of those coaching behaviors, e.g., “do you use video analysis with your swimmers?” To guard against confirmation bias that would be expected from asking such a direct question, specific follow-up probes were employed to gather a greater amount of detail about how these behaviors were enacted by the participants, e.g., “could you tell me a little more about the ways in which you use video analysis with your swimmers?” These response excerpts were then inductively coded based on the responses given by the participants, which allowed the evaluator to present a nuanced picture of how these behaviors were being employed. The strength of the abductive (or deductive/inductive) approach is that it allowed the evaluator to focus a participant’s attention around specific topics of interest that were related to the purposes of the evaluation, while still allowing for participants to express a diversity of opinions that accurately reflected the reality of program implementation. Purely deductive analytical frameworks would have restricted the range of participant responses and the richness of response data that could be generated. Purely inductive analytical frameworks (e.g., phenomenology) would have generated responses that would be too idiographic, and could easily stray from the purpose of the evaluation, which was to assess the use and implementation of the course. The interview guide (see Appendix E: Coach Interview Guide) was developed around the following sensitizing concepts (or deductive bins): coach background; coaching context; coach needs and challenges; coach’s general impressions of the online course and the learning environment; useful topics from the course and how these topics were put into practice; coach attitudes, coaching behaviors; coach development. Most lines of questioning began with broader 45 questions, allowing the participant to speak generally about a topic and begin to explore their feelings about the topic, a process referred to as “getting words to fly” (Glesne, 2011). Specific follow-up questions allowed the evaluator to probe the meaning of statements made by the participant for clarification and specific meaning, generating rich statements that reflected the reality of how participants implemented the course material in their day-to-day work as coaches. In the solicitation email sent to coaches, and during the solicitation process prior to the actual interview, very little about the nature of the interview was revealed (e.g., what questions would be asked, specific items to recall), so as to capture a picture of the knowledge from the course that the coach actually retained and used on a day-to-day basis without forced recall. A semistructured interview approach was employed, which allowed the researcher to adhere to the deductive/inductive analytical framework. When a participant provided a response that was not anticipated, the evaluator was able to follow-up with questions to explore those topics. To account for the varied nature of some interviews, a larger than normal sample-size (n = 21) allowed for adequate coverage of all topics in the interview guide. Preliminary interviews and analysis. Prior to beginning the process of soliciting and interviews with coaches (individuals who were outside of the USAS structure), the author filed and obtained IRB clearance. The evaluator solicited a sample of nine high scoring coaches in the 51+ age group, and 14 high-scoring coaches from the 36-50 age group. One of the nine coaches from the 51+ age group, and two of the 14 coaches from the 36-50 age group replied and then participated in an interview. Coaches from these age groups were selected for solicitation following the recommendations of Patton (2011) to begin data collection with “data rich” individuals. In this case, both age and test-score were used as proxy variables for being “data rich,” in the sense that they demonstrated better mastery of knowledge as assessed by the 40-item 46 Foundations 201 test, and had a greater number of life experiences that might help set coaching and learning in perspective. Following from the digital native hypothesis (which suggests that all people born after 1985, during the era of widespread computer use, are fluent in digital technologies), the evaluator anticipated that older coaches might have experienced greater difficulty with the online learning environment, and therefore might present more detail regarding the limitations of the online environment. After a coach completed an informed consent form electronically, the evaluator arranged a time to conduct a telephone interview. Interviews ranged between 50-58 minutes, and were conducted over a five-day interval. After each interview, the evaluator wrote a statement of reflexivity in an audit trail document, which included reactions to the participant, and reflections on the nature of the dialogue (i.e., the effectiveness of questions, the degree to which the hurriedness of the interviewer may have restricted the participant’s willingness to share or discuss particular elements). These reflections also included the evaluator’s thoughts about the nature of the Foundations course, and the potential for other questions that could be raised about the effectiveness or the purpose of the course. The evaluator transcribed these interviews during the same five-day interval as when the interviews were conducted. Interview and demographic data for these coaches appears in Table 5. The evaluator developed a coding scheme based on the logical framework developed in Phase II, where numbers corresponding to the seven sensitizing concepts could be used to tag excerpts of text, making them easily visible during analytical procedures. After developing this coding scheme, the evaluator met with two other qualitative researchers not intricately involved with the project, who acted as peer-debriefers to the coding scheme, helping to improve weaknesses in the scheme. A list of major deductive codes is presented in Table 6. After hand- 47 coding the three preliminary interviews, the evaluator revised the “course knowledge” code into two related sub-codes: Useful Topics (topics from the course that coaches found immediately useful or applicable), and Processing (how coaches translated knowledge from the course). The preliminary analysis served as a check on the effectiveness of interview questions, notably the considerable social presentation bias that arose with one line of questioning: “do you consider yourself a role-model?” and “do you develop your athletes as people?” This line of questioning was refined to ask the following questions: “As a result of their swimming experiences, have you seen your swimmers change as people?” [Probe] “Do you think you play a role in that change?” In addition, some of the questions meant to assess changes in coach attitudes were reframed to take a devil’s advocate approach. For instance, after asking a coach to elaborate on the concept of developing a professional mindset, the follow-up question would ask, “do you think that if you hadn’t taken this course, these attitudes would have developed?” This placed the burden of proving the course’s effects on the coach, and proved useful for getting a coach to assess, retrospectively, the effects of the course on their own thinking and behaviors. Based on these questioning approaches, the evaluator revised the interview guide, which guided the remainder of the interviews. Interviews and deductive analysis. The evaluator solicited high-scoring coaches from the remaining age-groups, and purposively selected participants in order to keep the number of participants in the sample roughly proportional to the population of coaches, and in order to maximize the diversity of the sample using both the race/ethnicity and the club excellence status variables. This process was repeated with low-scoring coaches to round-out the sample. A total of 13 high scoring coaches and eight low-scoring coaches were interviewed for this study (n = 21). Before beginning the interview process, each coach participant completed the informed 48 consent form online, and was given the opportunity to ask the evaluator questions, both at the outset of the interview, and at any time within or after the interview. The evaluator and three research assistants transcribed all interviews verbatim, and the evaluator cleaned and edited each interview for clarity and consistency. Using Dedoose software (SocioCultural Research Associates, dedoose.com), the evaluator created a case for each participant, which included relevant descriptors (variables) as well as the transcript of the interview. Dedoose software allows for the creation of a code tree, which organized parent codes and child codes into a hierarchical framework. The deductive analytical process involved reading the transcript, highlighting passages of text (i.e., excerpts) that conferred a specific meaning related to a sensitizing concept (e.g., attitudes, behaviors, useful topics), and then applying the appropriate deductive code. Excerpts could be tagged with one code or multiple codes. A complete list of deductive codes with the total number of code applications appears in Table 7. In addition to the deductive analysis (i.e., code application procedure), the evaluator used the memo function of Dedoose to keep a record of developing themes that might not be accounted for the deductive analytical procedure. Interim findings meeting. The purpose of the interim results presentation was to give the key program decision-makers an opportunity to view and respond to interim findings, prompting discussion, clarification from the evaluator, and strategic thinking and planning with regard to the program. After conducting all 21 interviews, but before completing the deductive analysis of all interviews, the evaluator prepared to present interim findings to the key program decision-makers. Following the same analytical procedure outline previously in this section, the evaluator performed deductive analysis using a smaller sample of seven high-scoring participants, chosen to maximize diversity in club size, gender, age, and coaching experience. 49 Results of this deductive analysis were organized to show summaries of responses given by the coaches with regards to four broad sensitizing concepts: general perceptions of the course; knowledge from the course; coach attitudes; coaching behaviors. The evaluator presented findings using a power point presentation for key program decision-makers (the same staff members from USA Swimming who participated in the first focus group meeting), but did not create an interim findings document, following the recommendations of Patton (2011) to avoid the printing of interim documents, as they can often confer the finality of results. The evaluator presented these interim findings, fielding questions as necessary; structured critique of the findings was accomplished in groups of two, who then reported back to the whole group. The evaluator audio recorded the interim results meeting and wrote minutes afterwards in order to promote recall of the discussions. A more thorough description of the findings and outcomes from the interim results meeting occurs in Chapter 4. Phase IV: Inductive Analysis, Description, and Interpretation of Findings Following the interim findings meeting, the evaluator completed the deductive coding of the remaining 14 interviews in the same fashion as the first seven interviews. After completing the deductive coding procedure, the evaluator began the inductive analysis by creating an outline in a Word document that included each of the deductive codes. Then, the evaluator used Dedoose to screen for each deductive code, making each excerpt tagged with that code visible. The evaluator read each excerpt, and reduced it to a theme (i.e., brief statement or sentence encapsulating the meaning(s) expressed in a longer excerpt of text). Themes were clustered under their deductive codes, and then organized into representative categories. Because the analytical strategy varied for each deductive code (e.g., useful topics, age-appropriate workouts), a description of the inductive analytical strategy employed for each deductive code appears in its 50 corresponding section in Chapter 4. The themes that clustered under each deductive code were used to guide a descriptive write-up for each code. Using both the insights presented in the interim findings meeting, as well as being grounded in the data, the evaluator interpreted these descriptive findings, which are presented in Chapter 4 and summarized in Chapter 5. Using these interpretations, the logic model developed for the first interim report was revised to incorporate the increased knowledge gained about the Foundations program. Finally, the evaluator presents a formative judgment of the Foundations of Coaching program relative to its objectives. The evaluator will create a draft evaluation report, and aims to report full findings from the qualitative inquiry to key program decision-makers in February 2015. 51 Chapter 4 – Results and Discussion The results are presented temporally, in the order that they were carried out. However, it should be noted that there was often significant overlap, particularly with the main analysis and the interim findings meeting (conducted in early July 2014). Because this evaluation employed multiple forms of inquiry (e.g., focus groups with key program decision-makers, quantitative data exploration, qualitative interview analysis, interim findings meeting), analysis occurred throughout the data collection timeline. Therefore, to aid the reader in comprehending the lengthy results section in the same temporal fashion as the evaluation unfolded, results are discussed throughout this section. A general discussion of these results appears in chapter 5. Description of the Foundations of Coaching Program The Foundations of Coaching program is broken into two courses: Foundations 101, taken as part of a coach’s initial requirements for coach membership, and Foundations 201, taken before a coach begins their second year of coach membership. USA Swimming does not certify coaches; rather it educates coaches and imposes requirements in order for coaches to attain coach member status. An individual with coach member status can conduct the duties of a swimming coach. Course content. The content in the Foundations 101 course is designed to meet the needs of a novice coach. It is broken into four sections: the responsibilities of a coach (promoting the sport and athlete enjoyment of swimming); communicating with athletes, parents, and officials; athlete growth and development; coaching athletes at practices and meets. The 101 course is organized around specific tasks and responsibilities that a coach must perform. As opposed to being organized around specific coaching duties, the majority of content in the Foundations 201 course is arranged by subject. Foundations 201 includes the following ten 52 sections: coaching philosophy; swimming organization (governance); growth and development (psychosocial aspects); motivation and goal-setting; mental skills; biomechanics; physiology; season planning; planning a practice session; nutrition, hydration, and drug-free sport. Course pedagogy and technology. The course incorporates the following methods to communicate and assess content knowledge. The primary mode of content delivery is a voiceover slide-show, which includes an outline of the voice-over script as well as pictures and visual aids where needed. The coach must wait until the voiceover is complete before they can advance to the next slide. Pool pointers are interactive diagrams that a coach must click in order to reveal information. Although pool pointers organize and present new information to the coach, but they also serve the purpose of requiring the coach to click elements of the course in order to progress. This prevents the coach from pressing play and completely disengaging from the course, which had been a possibility with the previous DVD/test version of the course. An example of the course layout with Pool Pointers is shown in Figure 1. 53 Figure 1 - Screenshot from Foundations of Coaching course showing Pool Pointers The course includes video clips that feature expert coaches and sport scientists and integrate seamlessly into the course; most video clips are less than 75 seconds, and the coach must wait for the video to finish playing before they can move to the next screen. An example of a video is shown in Figure 2. 54 Figure 2 - Screenshot from Foundations of Coaching course showing video integration A third essential feature is the use of regular self-check quizzes that appear throughout the course in order for a coach to check comprehension. These quizzes give coaches two opportunities to select the correct answer(s) before allowing the coach to move forward. A screenshot of a typical self-check quiz appears in Figure 3. Another form of assessment is the notepad feature (shown in Figure 4). The notepad allows a coach to devise an open-ended response to a question posed in the course and then enter their response. An instructor does not assess these responses; instead, the coach is asked to compare their answer to the preferred response that is revealed when the coach clicks on the Pool Pointer. 55 Figure 3 - Self-check quizzes as formative assessment Figure 4 - Open-ended formative assessment question 56 Another pedagogical feature of the course is what ePath Learning terms Job Aids, which are links to separate resources that provide supplementary information or guidance to the coach as they take the course. The Job Aids for the Foundations 201 course include 33 pages of supplemental content that can be downloaded as individual documents, or one long, inclusive document. At several points throughout the course, a specific lesson will provide a link to a specific Job Aid, which will open a 1-2 page PDF document that provides the specific supplemental material for that lesson. The course will typically instruct the coach to print the document and use it to follow along with the lesson. For instance, in the coaching philosophy lesson in the Foundations 201 course, the Job Aid is a document that prompts coaches to answer three questions: (a) write down what your swimmers need to develop; (b) write down your personal needs as a coach; (c) now write a statement of your coaching philosophy. At the bottom of this document, the coach is instructed as follows: In order to continue to develop your personal coaching philosophy, read articles on coaching and talk to other coaches. Write down what you’ve learned. Revisit and revise your philosophy as you develop as a coach. Try to update your written philosophy statement annually. (USA Swimming, Reference Library, 2014) The final part of the Foundations course is the test that must be taken in order to pass the course. Coaches must score a total of 80 percent or higher to pass, and the test can be taken as many times as needed in order to obtain a passing score. The test uses a bank of test questions, so that the same questions are not presented for each coach. The 101 test has a total of 20 fixedresponse (e.g., multiple choice) questions, and the 201 test has a total of 40 fixed-response questions. The number of questions in this iteration of the Foundations course is a significant reduction over the previous version, which consisted of a 180-question online test, where the 57 user had to attain a score of 160 questions correct in order to pass. Framing the Evaluation It is common for many programs to arise from a demonstrated or historical need, and to be developed and implemented without a deliberate planning process that identifies ideal program outcomes. This can often create some distress to social scientists that are accustomed to a detailed planning process that leads to a systematically implemented program. The reality is that many programs designed to affect some sort of positive change are not designed nor implemented by social science researchers: they are designed and implemented by experts who work closely with their population, who have an intimate knowledge of their population, and who have a decent understanding of how their program works, but who may lack the formal knowledge that guides the development and implementation of programs that can effectively induce human behavior change. Evaluators are no strangers to such programs. The program evaluation may often serve to formulate the local theory (or theory-in-use) that describes how the program works into such a fashion that makes the implicit theory explicit; it could be thought of as a post-hoc description of program theory. Those more accustomed to traditional social science research methods should be aware that this is a relatively common practice in the field of program evaluation. The Foundations of Coaching program fits this mold. It arises from both a demonstrated and historical need for education of coach members. The most recent iteration represented a significant shift in the approach, as the course moved from a DVD + test model (180 questions) to a fully online model. The fully online model represented two major changes to the preexisting program format. First, as stated by the Programs and Services Director, the current iteration is focused on learning as opposed to testing. While the testing elements still remain 58 (albeit in much shorter increments), there are more formative assessments of coach learning and knowledge along the way that build to a final test. The test can be retaken until a coach attains a minimum score of 80 percent. Second, the educational process was shifted so that a coach would experience some type of coach education before setting foot on the pool deck as a coach member of USA Swimming. Not only was a small portion of essential, skill-based knowledge moved to the time before a coach begins to coach (e.g., responsibilities of a coach, communication, athlete growth and development, coaching athletes at practices and meets), but the means in which this information was presented to the student was also a departure from the previous course, in that it organized the knowledge around specific duties that the coach would need to perform. Specifically, the Foundations 101 course employs the use of several scenarios to organize the knowledge and principles being taught. This enables a coach who has yet to experience live, ondeck swim coaching to anticipate specific problems and issues that might arise in your first job as a coach. According to the Programs and Services Director, this change in pedagogical approach stemmed from a very deliberate process: to examine the core knowledge that a coach needs to have before they begin their job, and to present it in such a way that it can be more easily contextualized by the coach. Situation analysis. The purpose of the situation analysis, conducted January 2014 through the first focus-group with key program decision-makers, was to clarify the program’s purposes, identify what the program was doing particularly well, as well as to uncover doubts, uncertainties, and barriers to the program’s effectiveness. In addition, this analysis sought to uncover potential skeptical viewpoints on the Foundations of Coaching program, as well as potential skeptical viewpoints about the results of this evaluation. Patton (2011) states that identifying threats and skeptical viewpoints is an essential feature of a utilization-focused 59 evaluation, as it helps key program decision-makers to identify specific areas and topics on which they would like focused inquiry, in order to address the skeptics who may threaten this program. Helping key program decision-makers identify sources of skepticism and defend their program from attacks borders on program advocacy by the evaluator; as advocacy was not a goal for the evaluator, he strived to be fair, thorough, and honest in the assessment of the data when helping key program decision-makers address these sources of skepticism. This section of the chapter addresses the findings from this situation analysis, including the purposes of the Foundations program, identifying knowns and unknowns with regards to the program’s performance, concern from skeptics, and then proceeds to outline the purposes for the remainder of the evaluation. Objectives of the Foundations of Coaching program. The key program decision-makers identified seven objectives for the Foundations of Coaching program. First and foremost, the Foundations course delivers core coaching knowledge. Second, it introduces new topics with the hope of inspiring future learning. Third, it has a positive influence on the coach, so that the coach has a positive influence on their swimmers. Fourth, it reinforces USA Swimming’s dedication to responsible and safe coaching practices and coach conduct towards athletes. Fifth, it delivers knowledge about long-term athlete development (LTAD, i.e., long-term physical, psychosocial, and emotional growth and development of athletes and the matching of athletic programming appropriately to these developmental constraints). Sixth, it delivers knowledge about first-year issues for coaches (e.g., working with parents, coaching swimmers at meets). Seventh, it provides the coach with an outline for performing their coaching duties. To meet these objectives, the key program decision-makers identified four processes that enabled the Foundations program to function well. These processes had been identified both 60 intuitively and through formative, unsystematic reports from unsolicited coaches who had written thankful e-mails with regards to the Foundations program. First, the program introduces core coaching principles and knowledge, set within a coaching context, through the use of a scenario-based approach; this approach enables a beginning coach to get a good idea of how these principles actually apply to coaching swimmers. For example, the use of a scenario entitled, “Is five-year old swimmer Ryan old enough to start competing?” contextualized a common issue faced by coaches and provided information that coaches could learn to help guide their future decision-making. Second, the key program decision-makers believed that the intensive focus on athlete growth and development should have a positive impact on the sport experience for swimmers; this conclusion was developed intuitively, based on the significant expertise these professionals had gained in their careers working to develop programs and coaches. Third, the focus on athlete safety and on coaches being a positive influence was emphasized more directly than in the past, which reflected a need for USA Swimming to address the concern of athlete welfare in the wake of an abuse scandal (Goldman, 2013). Fourth, the current iteration of the program has a stronger minimum effect, because it forces the coach to sit through the message, as opposed to the previous DVD/test iteration of the program, which could not guarantee that a coach actually watched the educational DVD before taking the test. While not failsafe, the current iteration, with its interactive features and quizzes, forces the coach to engage with the program before taking the qualifying test. Thus, there are better assurances that at minimum, there is stronger program fidelity for coaches who wish to disengage from the actual educational portion of the course, and jump immediately to the test to try their luck with their current knowledge. Uncertainties about the Foundations of Coaching program. The evaluator helped the 61 key program decision-makers to identify uncertainties about how the program works, which fell into three categories. The first of these categories was uncertainties about the coach taking the course. For instance, the key program decision-makers had not done systematic analysis of the relationship between demographic variables and output measures, such as test scores. The course was designed for a first-year coach, but many coaches could bring experience coaching non-affiliated swimming programs (e.g., high school teams, summer recreational league teams) into their first-year experience being a USA Swimming coach member. Despite being designed for a novice coach, how would these experienced, first-time members of USA Swimming respond to the course? The second category included four unknowns about the learning process: (a) we lack a range of examples of how coaches use and apply the information from the course in their day-to-day work as coaches; (b) we lack an understanding of what coaches do when they have a question that arises from the course (as well as from other situations); (c) we don’t know if coaches embrace the idea that this course is about continual learning and development as a coach, rather than simply passing a test; (d) we don’t know if coaches have the perception that the current iteration of the course has dumbed-down the knowledge too much. The third category of uncertainties dealt with lack of knowledge about outcomes from the course. For instance, was the course capable of educating coaches about topics that can only be learned through more complex, dialectic processes, such as reflection on action, mentorship, or actual experience? There was additional uncertainty about the course’s ability to inspire future learning, and if so, what topics it prompts coaches to pursue in greater depth. Potential barriers to the Foundations of Coaching program. The evaluator facilitated the process of identifying potential barriers that might block the program from being delivered as intended. These barriers focused primarily on the human elements that could impact program 62 delivery, rather than specific technological barriers that could arise (although these barriers were addressed elsewhere in the dissertation). As with the uncertainties, these barriers were organized by the evaluator into three logical categories of concern. The first and largest set of potential barriers arose within the coach. These barriers included a poor attitude towards learning and professional development (manifested through statements like, “I already know all this stuff anyway”), or a poor attitude towards USA Swimming, e.g., the coach sees the course as brainwashing from the national office. These barriers within the coach could arise if the coach was turned-off by specific elements of the online course (e.g., avatars delivering knowledge) or by online learning in general, and additional complications were anticipated if a coach was attempting to complete the course right before a registration deadline. Finally, a barrier could be the coach’s own desire and personality traits; if these precluded the coach from being successful in the coaching profession, there was little that the course could do overcome these deficiencies. Barriers could arise from sources external to the coach. For instance, some coaches might be under significant influence of head program coaches, mentors, or other peers who have a low opinion of USA Swimming’s coach education program. These individuals are influencers with the ability to shape a new coach’s perceptions of the coach education program before the coach has an opportunity to experience it on their own. In addition, these coaches might communicate the position that anything worth learning in coaching can be learned on the job, through experience, and that everything else is a waste of time. The distance-learning model itself presents several barriers to the effective functioning of the program. First, the software could present problems that impede progress (e.g., videos fail to play, interactive features fail, quizzes assess unrelated content). But perhaps more importantly, there are potential limitations to what can be accomplished with a model where instruction travels from expert to novice 63 through a computerized medium, and where there is no opportunity for instructor-peer discussion, or peer-to-peer discussion. However, given that the previous model was not based upon a face-to-face class, there never was a baseline of face-to-face interaction that the online model is being compared against. This leads into another barrier, in that the course does not specify any type of a follow-up evaluation or observation after the course; the onus for coach development lies almost entirely with the coach and the coach’s team. Addressing skeptical viewpoints about the Foundations of Coaching program. The evaluator asked the key program decision-makers to identify their biggest skeptics and picture them sitting in the audience as positive results to an evaluation of the Foundations of Coaching program were being presented. What would these skeptics have to say? The roundtable discussion yielded six salient critiques of the Foundations program. Each point of skepticism is addressed in Table 2, with potential follow-up questions or points of dispute. The majority of skeptical viewpoints centered around the ability of a formal, online learning experience to provide any useful information to a coach, when coaching is clearly a profession that demands apprenticeship-style, on-the-job training with guidance from experts and mentors. There was also some anticipated skepticism towards the message about long-term athlete development, given that some established coaches have speculated publicly that an athlete-centered developmental approach will lead to a generation of “soft” swimmers. Responses to skeptical viewpoints vis-à-vis evaluation findings are discussed in Chapter 5. 64 Table 2 - Skeptical viewpoints on the Foundations of Coaching program Skeptical Viewpoint Potential Response(s) You can’t prove that a better educated coach is a better coach What is the definition of a better coach? How have great coaches become better? How much of great coaching comes from continual learning and application of knowledge? Is an uneducated coach worse, and does an uneducated coach create risk that an educated coach would not create? Too much coach education stifles innovation One of the unique strengths of American swimming is the innovation of swimming coaches and diversity of coaching approaches. There is rightfully some concern that an extensive system of coach education might stifle innovative thinking; this concern should be explored in the evaluation. Formal education does not work There is scientific evidence that formal coach education can be effective. This evaluation seeks to find out what specifically works well in the online format. Mentorship or apprenticeship is the preferred method to educate coaches. While there may be some truth to this statement, mentorship is a much more labor and cost-intensive means of educating coaches. The online course provides education to over 2,000 coaches nationwide at a very competitive price-point. The evaluation may show a need for increased mentorship to help coaches learn more difficult topic, which might help to pinpoint areas to address for sport performance consultants. It might also show what areas can be learned online, and do not require the attention of mentorship in order to be learned. The course lacks rigor and challenge – it has “dumbed-down” important knowledge. The evaluation intends to show coach perceptions of the course content, and its degree of application. Using a previous iteration of the coaching education program, it is not certain that a difficult test created a better coach; the link between passing a test and applying knowledge as a coach is tenuous. Too much focus on long-term athlete development leads to a generation of ‘soft’ swimmers. We believe that this could be the intuitive belief of some coaches. However, we believe that the course will help coaches to find ways to challenge swimmers in ways that do not put athlete safety at risk. Logic model. The evaluator used findings from the focus group with key program decision-makers to construct a preliminary logic model that would outline the sequence of events that occur in the program. This model was used in large part to structure the inquiry into the program’s utility and perceptions, as well as the program’s effects. Read from left-to-right, a logic model provides a linear framework that organizes program inputs (assets), activities (what the program participant does), outputs (products resulting from activities), outcomes (resultant changes in knowledge, attitudes, behaviors), and impacts (changes to context or society attributable to outcomes). The preliminary logic model that appeared in the interim report to key program decision-makers is displayed in Table 3. 65 Table 3 - Foundations of Coaching Preliminary Logic Model (January 2014) 66 The logic model underwent an iterative development process, resulting from a constant cross talk between the existing logic model and the data that the evaluator gathered and analyzed. Subsequent revisions of the logic model showed that information that coaches took from the course and translated into specific coaching changes constituted short-term outcomes. Information gathered about how knowledge was acquired and processed, as well as how coaches developed specific attitudes and behaviors led to the current iteration of the logic model, presented at the end of Chapter 4. Purposes of the evaluation. The final portion of the interim report identified five purposes for the evaluation; the first four were identified as priority evaluation purposes through collaborative work with the key program decision-makers, especially the Programs and Services Director and the Director of Field Services. A fifth purpose, utilization of evaluation findings and process, originated with the evaluator as a function of this being a utilization-focused evaluation (Patton, 2011). (Reader’s note: these purposes represent the purpose statement that would typically appear at the end of the introduction section of a research article). Purpose 1: Program utility and perceptions. The first purpose was to examine the utility of the program to coaches who took the course, including coach perceptions of the course and its ability to meet their needs and challenges of being a coach. This purpose was also conceived as a coach’s gut reaction to the course (i.e., their immediate, emotional appraisal of the course), noting that previous iterations of the course had received unfavorable reactions from coaches, which had impeded the flow of information. Purpose 2: Program effects. Determining the program effects constituted the largest and most extensive portion of this evaluation. As conceived, the evaluation examined the usefulness of individual topics presented in the course, their implementation into coaching practice, the 67 attitudes possessed and developed by coaches who took the program, and the practice and adoption of specific coaching behaviors that add value to the athlete’s experience. The evaluation did not focus explicitly on demonstrating the causal nature of the course, but instead sought to examine the associative nature between coach perceptions and utility of the program, use and implementation of course knowledge, and specific attitudes and behaviors. During an interim findings meeting (discussed later in this chapter), the evaluator clarified the difference between correlation and causation, and allowed for discussion around this purpose. The need for causality was not determined as essential at this time, although this evaluation might inform future studies and evaluations that attempt to demonstrate causality. Purpose 3: Program delivery. Given that this program is delivered online, there is a possibility for technological problems that inhibit the learning process. Therefore, this purpose sought to examine the degree of consistency in the user experience. This was deemed as important due to the fact that a significant number of new coaches were above the age of 40, and not so-called digital natives (people who have grown up with computing technology for the majority of their lifespan, and therefore are assumed to have a grasp on basic computer functionality), thereby invoking some trepidation that widespread technological difficulties could be problematic for a large number of coaches completing the course. Purpose 4: Areas for improvement. This evaluation had a formative nature in that it sought to explore areas for program improvement. Areas for improvement were not restricted to the Foundations course, but also to areas where USA Swimming could provide more direction and leadership to benefit coach and swimming program development. Purpose 5: Utilization of evaluation findings and process. A fifth purpose sprung primarily from the evaluator (although it does has value to the key program decision-makers). 68 Planning for the utilization of evaluation findings and evaluation process is a key facet of utilization-focused evaluation (Patton, 2011). Throughout the evaluation, the evaluator collected and analyzed data with an eye to utilization by key program decision-makers. The evaluator employed an interim findings meeting to present interim findings and to allow for comment and clarification by key program decision-makers. Use of the evaluation process was discussed at both the initial focus-group meeting (January 2014) and the interim findings meeting (July 2014), and a detailed evaluation report will be presented to the key program decision-makers at a meeting to be determined in the future (February 2015). With these five purposes in mind, the evaluator proposed a sequential exploratory mixedmethod inquiry with qualitative methods as the favored source of data (Creswell & Plano Clark, 2006). In this design, quantitative methods would be used first to explore relationships between coach demographic and test-score data at the population level. These findings would be used to guide primarily sampling of coach participants, and to a lesser extent, questions for the qualitative inquiry, which consisted of retrospective interviews with coaches who had completed the Foundations of Coaching program. These methods and their subsequent analysis are described sequentially. Quantitative Exploration of Demographics and Test-Score Data Demographics. USA Swimming provided the evaluator with a spreadsheet including a population of 3,460 coaches who had registered between January 1, 2013 (the date when the current Foundations of Coaching program became the preferred form of meeting coach education requirements) and February 9, 2014, and who had taken at least the Foundations 101 program. This population was narrowed to a sample of coaches who had fulfilled all of their coach education requirements, which included the Foundations of Coaching 101 and 201 courses, as 69 well as the Foundations of Coaching Rules and Regulations program (which was not a part of this evaluation). This yielded a sample of 1,336 coaches. The second step was to determine these coaches’ tests scores on the Foundations 101 and 201 tests. Twelve coaches did not have sufficient test-score data, and were thus removed from the sample, yielding the intermediate sample of coaches (n = 1,324). The next step was to examine patterns amongst demographics in the intermediate sample to explore meaningful variance amongst coaches. The mean percentage score for the 101 test was slightly higher (91.37, SD = 6.06) than the mean for the 201 test (90.24, SD = 5.78); these values were normally distributed (skewness and kurtosis values falling between ± 2). Frequencies were calculated for the remainder of demographic values (age, gender, race/ethnicity, club affiliation, Club Excellence level, full time/part time, USA Swimming Zone location). The age distribution showed a peak at age 23, with 59.5 percent of coaches being between 18-28 years old. There was a significant dip around age 30, followed by a slight rise in the age histogram from ages 35-45. Using the histogram of age distribution, the evaluator also employed his knowledge of swimming and coaching in order to create five different age brackets that represented natural breaks in age groups. Coaches 18 to 23 years of age (22.4 percent of the sample) represented coaches who were college age, who might be coaching as a way to make money during their college years. Coaches 24 to 28 years of age (27.1 percent) represented the next age group, logically coaches who were the age of recent college graduates and/or early career professionals. Coaches 29-35 years of age represented a smaller fraction of coaches (10.8 percent), and were differentiated from the 24-28 year-old age-bracket with the assumption that these coaches might be entering coaching as a second profession, and because their relatively low representation (a valley in the histogram) suggested that there could be different motives for 70 them entering the coaching profession. Coaches 36-50 years of age (23.5 percent) were seen as coaches who might be entering the profession in order to coach their children, while coaches aged 51 or older (6.5 percent) were seen as coaches who might have entered the profession as a second career. The reader is cautioned that there is not an empirical method being used to create these age-brackets, but rather a logical break to maximize the variance between groups so as to later explore this variance using qualitative methods. Given that a qualitative follow-up would be used to explore individual differences in coach experiences, and would serve as the main part of the analysis for this evaluation, the evaluator did not use advanced statistical methods to create groups. Instead, this represents the evaluator’s logic-driven estimation of natural age-group breaks, designed for the purpose of later soliciting coaches from these age groups based on a stratified systematic sampling method. Examining test-score differences between groups. Test score data served as a proxy variable for a coach’s knowledge gained from the course, and there was a need to explore if demographic variables (e.g., age, gender, race/ethnicity, club excellence) contributed to test performance. To explore statistical relationships between demographic variables and test scores, the evaluator employed basic hypothesis testing procedures to examine differences between groups of coaches on both the 101 and 201 test scores; results are presented in Table 4. 71 Table 4 - Hypothesis Tests of Demographic and Test-Score Data Hypothesis Statistical Test Results H1: Scores on FoC 101 and FoC 201 will be positively correlated Dependent t-test (paired samples) Pearson correlation Levene’s statistic Significant difference between mean scores of 101 and 201 (p = .000); Significant, low-positive correlation between 101 and 201 scores (.197, p = .000); Effect size (r=.14) H2A: Club excellence coaches will have higher scores than non club excellence coaches Independent t-test Factor: ClubExcel Levene’s statistic No significant difference between club excellence and non-club excellence status for BOTH 101 and 201. Levene’s stat – insignificant (equal variances assumed) H2B: Higher club excellence levels will predict higher scores ANOVA (4 way) Factor: ClubExcel Levene’s statistic No significant difference between club excellence levels for BOTH 101 and 201. Gold level is significantly lower (p < .1) Levene’s stat – insignificant (equal variances assumed) H3: Being affiliated with a specific club (ClubAffil = 1) will predict higher test scores than being unattached Independent t-test Factor: ClubAffil Levene’s statistic Significant differences between club affiliated coaches and non-affiliated coaches. Levene’s insignificant (equal variances assumed) 101 – Unattached scored 1.21 points higher (p=.034), 91.27 to 92.48 201 – Unattached scored 1.42 points higher (p=.006), 90.20 to 91.62 H4: Coaches who are full time will have higher scores than coaches who are part time Independent t-test Factor: FullTime_PartTime Levene’s statistic No significant difference between part time and full time coaches for BOTH 101 and 201. Levene’s stat – insignificant (equal variances assumed) H5: There will be a difference in Independent t-test scores between males and females Factor: Gender Levene’s statistic Significant differences on 201 only. 101 – No significant differences between men and women. 201 – Women scored 0.99 points higher on average (p=.001), 89.78 to 90.77 Levene’s stat – insignificant (equal variances assumed) H6: Former USAS athletes will Independent t-test have higher scores than non former Factor: RegType athletes Levene’s statistic Significant differences on 201 only. 101 – No significant differences between non former USAS athletes and former athletes. 201 – Coach members who were not formerly USAS registered athletes scored 0.89 points higher on average (p=.004), 89.97 to 90.86 Levene’s insignificant, eq. var. assumed H7: There will be differences in scores between the four zones ANOVA Factors: Zone Levene’s statistic No significant differences for 101 or 201. Levene’s insignificant, eq. var. assumed H8: Coaches with an interest in a community initiative will have higher scores Independent t-test Factor: Commun_Initiative Levene’s statistic No significant differences for 101 or 201. Levene’s insignificant, eq. var. assumed 72 Table 4 cont’d Hypothesis Statistical Test Results H9a: There will be differences in 101 scores between five age brackets ANOVA Factors: AgeBracket Levene’s statistic Use Tukey HSD for individual comparisons. 51&up scores highest on average, and is significantly different than 18-22, 23-28, and 2935 groups (p<.05). 36-50 scores second highest on average, and is significantly different from 18-22 (p=.017) Levene’s = .06 (approaching significance) Error bar chart reveals major differences for men and women in 51&up bracket (women lower), but women higher for 36-50 bracket. H9b: There will be differences in 201 scores between five age brackets ANOVA Factors: AgeBracket Levene’s statistic Use Tukey HSD for individual comparisons. 18-22 scores significantly lower than the rest of the age brackets (p<.01). 23-28 gender differences are highly pronounced, no overlap in error bars at all, women significantly higher. H10: Effects of being white vs. non- ANOVA white/mixed ethnicity on exam Factors: NonWhite_NEW score Levene’s stat No significant differences between three groups. Equal variances assumed (Levene’s not sig.) H11: Effects of ethnicity 1 on test scores No significant differences based on ethnicity. ANOVA Factors: Ethnicity1 Hypothesis 1 posited that there would be a significant positive relationship between 101 and 201 test scores; Pearson correlation showed a significant (p < .01) small positive correlation (r = .197, effect-size = .14) between the 101 and 201 test scores. This would suggest that testing effects (e.g., test-taking ability) explained a small percentage of the variance. Hypotheses 2A-4 sought to explore the relationship between the coach’s club membership and their test-scores. Hypothesis 2A posited that coaches that worked for teams that had attained any level (gold, silver, bronze) of USA Swimming Club Excellence status (a designation given to clubs with demonstrated history of meeting exceptional performance criteria) would have higher test scores than coaches for teams that had not attained Club Excellence status. However, no significant differences were found between these two groups of coaches for either the 101 or 201 tests, suggesting that Club Excellence status (any level) did not contribute to differences in coach test 73 scores. Hypothesis 2B explored differences in test scores between the four levels of Club Excellence (Gold, Silver, Bronze, none); there was a trend for coaches at gold level clubs to have significantly lower test scores, but only at a significance level of p < .1. This was noted as a trend to monitor in the future as more coaches go through the current version of the program. Hypothesis 3 posited that coaches with no club affiliation (unattached) would have higher test scores on both the 101 and 201 tests, because they would be more likely to be at smaller clubs, or independent programs with a small number of swimmers and coaches, and thus might have more use for an online course as a means of gathering knowledge. This hypothesis was accepted, as unattached coaches scored an average of 1.21 percentage points higher on the 101 test (p < .05) and 1.42 percentage points higher on the 201 test (p < .01). Hypothesis 4 posited that coaches with full-time status would score higher than coaches with part-time status, due to greater investment in coaching as a career; this hypothesis was rejected. Hypothesis 5 explored differences between male and female coaches; this hypothesis was accepted for the 201 test only, as women scored an average of .99 percentage points higher (p < .01). Hypothesis 6 posited that coaches who were former USAS athlete members would have higher scores than those who were not; this hypothesis was rejected for the 201 test only, as coaches who were not former USAS athletes scored an average of .89 percentage points higher than coaches who were former USAS athletes (p < .01). Hypotheses 7a and 7b employed separate one-way analyses of variance tests to explore differences in the 101 and 201 test scores amongst the four different USA Swimming zones (East, South, Central, West); no significant differences existed for either the 101 or 201 tests. Hypothesis 8 posited that coaches who had indicated an interest in participating in USA Swimming community initiatives would score higher than their counterparts who had not indicated this interest; this hypothesis was rejected. 74 One-way analysis of variance (ANOVA) testing was used to explore differences in test scores between the five different age-brackets and seven different race/ethnicity classifications (white; Hispanic or Latino; Asian; black or African American; Native Hawaiian & other Pacific Islander; American Indian & Alaska Native; some other race). Hypothesis 9a employed a oneway ANOVA to explore differences in 101 test scores between the five age brackets. Coaches in the 51 & over age bracket had the highest 101 test scores on average. Post-hoc t-tests revealed that this age-group had significantly higher (p < .05) test-scores than their counterparts in the 1822, 23-28, and 29-35 age-groups. Coaches in the 36-50 age-bracket had the second highest scores on average, significantly higher than the 18-22 age-bracket (p < .05). The Levene’s statistic revealed unequal variances in the age-brackets, with men in the 51 & over age-bracket scoring significantly higher than women. However, given the relatively small percentage of coaches in this age-bracket (n = 86) and the significance value (p = .06), caution is suggested in the interpretation of this difference. Hypothesis 9b employed a one-way ANOVA to explore differences in 201 test scores between the five age-brackets. Coaches in the 18-22 age-bracket scored significantly lower than coaches in the other four age-brackets (p < .01), and gender differences between coaches in the 24-28 age brackets were highly pronounced. This suggested that young coaches, particularly males, scored consistently lower than their peers in different age groups, a fact which warranted attention in the qualitative inquiry. The race/ethnicity demographics provided by USA Swimming were difficult to code, as USA Swimming allows coaches to indicate up to two different values for race/ethnicity. Two different one-way ANOVAs explored potential test score differences based on two reclassifications of race/ethnicity. Hypothesis 10 explored the effects of being in one of three categories of race/ethnicity (white, non-white, or mixed race/ethnicity) on test scores; this test 75 showed no significant differences in test scores amongst these three classifications of race/ethnicity. Hypothesis 11 examined differences between test scores based on the value that coaches entered for race/ethnicity choice 1 (with the assumption that this is the race/ethnicity with which the coach primarily identifies). No significant differences between test-scores existed based on these classifications. While this offered some reassurance that the test did not show signs of cultural bias, caution should be taken in over interpreting these results, for the following reasons. First, coaches who did not indicate any race/ethnicity value were not included (n = 137). Second, the percentage of white coaches far outweighs the number of nonwhite or mixed race/ethnicity coaches, and despite the Levene’s statistic for assumption of equal variances being significant, there is a strong possibility that real differences could exist; this should be monitored in the future. Third, this points to an issue in the self-reporting of race/ethnicity. If it is deemed as important for the accurate reporting of data, it may be prudent to ask coaches, “for the sake of statistical reporting, how would you classify your race/ethnicity?” and restricting the set of options to one choice. Ultimately, this depends on how USA Swimming wishes to classify data about their membership. Using the variables that demonstrated significant differences between age groups, the evaluator constructed a multiple regression prediction model for both the 101 and 201 test scores. Club affiliation and age (as a continuous variable) were able to significantly predict 101 test score; however, the R-squared value for explained variance was extremely low (.015), meaning that these two variables explained only 1.5 percent of the variance in the 101 test score. For the 201 test score, a model using club affiliation, gender, former USAS athlete, 101 test score (continuous), and age (continuous) was able to significantly predict 201 test score, but as with the prediction model for the 101 test score, the R-squared value was again extremely low 76 (.073), with half of this value accruing to a coach’s 101 test-score. Both of these prediction models suggest that demographic statistics were not particularly useful in predicting test scores. On one-hand, this provides evidence that the tests do not exhibit any systematic bias based on the coach’s demographic variables. On the other hand, it might suggest that the significant differences between groups of coaches do not warrant a stratified systematic sampling method. The point of this extensive analysis of demographic statistics was to explore relationships between demographic variables and test-scores, the best proxy variable available at this moment to approximate what a coach learned from the Foundations course. Test scores should be understood as just that… a proxy variable for what a coach got out of the course. There remained a very real possibility that there could be vastly different experiences in the course based on these demographic variables, and the decision was made to create a stratified systematic sampling method that would enable the evaluator to draw upon a greater variety of coach experiences with the Foundations course. Ideally, the exploration of these differences could uncover variables that could explain more variance in the course experience than the demographic variables that were available to the evaluator at the time of the interview participant solicitation. Using this information, the evaluator then moved to the next phase of selecting participants to solicit for interviews. Interview participants. The evaluator employed a stratified systematic sampling method to solicit interested coaches to participate in an interview about the Foundations of Coaching program. Within each age-bracket, the evaluator systematically sorted coaches based on Foundations 201 test scores, then gender, then deliberately oversampled non-white and Club Excellence coaches. In total, the evaluator solicited 323 coaches by e-mail. The evaluator selected participants from amongst those coaches who replied to the e-mail expressing an interest 77 in the interview. If a participant agreed to participate in an interview, they completed a secure, web-based informed consent form. Participants received a $25 electronic gift-card to Amazon.com to compensate them for their time. A demographic breakdown of the sample of coaches is presented in Table 5. 78 Table 5 - Demographic Statistics for Interview Participants ID Age Gender Ethnicity 201 score Club Affil. Club Excel. Employmt. Rank Years Exp. Min. Lines 1 54 F White/Asian High Yes No Part-Time Head 1-2 52 647 2 49 F White High Yes No Part-Time Ass’t 1-2 58 767 3 42 M White/Am. Indian High Yes No Part-Time Ass’t 1-2 58 712 4 25 F White High Yes No Full-Time Ass’t 1-2 54 674 5 24 M Hispanic High Yes Yes Full-Time Ass’t 1-2 48 566 6 24 F Caucasian High Yes No Part-Time Ass’t 5+ 95 1,182 7 27 F White High No No Part-Time Head 5+ 70 832 8 26 M White High Yes No Full-Time Head 1-2 68 711 9 20 M Caucasian High Yes No Part-Time Ass’t 1-2 44 465 10 19 F Caucasian High Yes Yes Part-Time Ass’t 3-5 56 629 11 27 M Caucasian High Yes No Part-Time Ass’t 1-2 72 977 12 23 F White High Yes Yes Part-Time Ass’t 1-2 60 736 13 35 M White High Yes No Part-Time Head 5+ 64 689 14 46 F White Low Yes No Part-Time Ass’t 5+ 66 740 15 42 M White Low Yes No Part-Time Head 3-5 51 616 16 38 M Caucasian Low Yes No Full-Time Head 5+ 81 961 17 55 F Caucasian Low Yes No Part-Time Ass’t 1-2 74 872 18 22 F White Low Yes Yes Part-Time Ass’t 5+ 63 786 19 21 M White Low Yes No Part-Time Ass’t 1-2 49 582 20 31 M Middle Eastern Low Yes No Part-Time Ass’t 5+ 50 579 22 19 M White Yes Yes Part-Time Ass’t 1-2 36 535 None A total of 21 interviews, conducted by phone, FaceTime, or in-person, and lasting on average 60.4 minutes (SD = 13.5) were audio recorded and transcribed verbatim, yielding a total of 15,318 lines of text (approximately 340 pages of single-spaced text), with an average transcript length of 729 lines (SD = 165.8). To record thoughts related to emerging themes, as well as maintain awareness of bias, the evaluator wrote a reflexive reaction to each interview in 79 an audit trail document. Each transcript was uploaded to Dedoose, a secure, web-based application that was used for deductive coding. The evaluator analyzed the 13 high-scoring coaches first, as they were taken to be data-rich individuals capable of providing a much richer and broader picture of the program (Patton, 2011). Preliminary analysis. The preliminary analysis served as a check on both interview design and the planned deductive coding scheme. The evaluator developed a list of deductive parent codes based on version 1 of the logic model; parent codes were designed to capture elements of the coach interviews related to the inquiry. Parent code numbers, titles, and their descriptions are listed in Table 6. The evaluator hand-coded interviews using these seven parent codes and associated child codes (see Table 7 for a list of all parent codes and child codes), then met with two separate peer debriefers to present and discuss the coding scheme. The parent codes remained largely intact, with the exception of the content knowledge code. To capture the nuance of how coaches were learning knowledge from the course, but also processing it and translating it into action, the evaluator created two child codes for the content knowledge code: Learning (4a), and Processing/Translating (4b). 80 Table 6 - Parent Code Numbers, Titles, and Descriptions Code # Code Title Code Description 1 Coach Factors that describe the coach (e.g., sport background, demographics, day job, career intentions, entry to coaching) 2 Context Factors that describe the coaching context (e.g., athletes, colleagues, club features, environment/setting, region) 3 Course Technology & Pedagogy Perceptions Factors related to the technological and pedagogical features of the Foundations of Coaching course 4 Content Knowledge Factors related to content knowledge presented in the Foundations of Coaching course 5 Attitudes Factors related to coach attitudes (specified and non-specified) 6 Coaching Behaviors Factors related to coaching behaviors (specified and non-specified) 7 Coach Development Factors related to the coach's professional development (e.g., unique experiences, 8 Miscellaneous Valuable excerpt, but unrelated to other codes 9 Good Quote Strong, representative passages - applied during inductive analysis phase The preliminary analysis showed that confirmation bias occurred when the evaluator asked participants to discuss their efforts at being a role model and developing athletes as people. As these assets are generally perceived in a positive light, the answers provided by coaches all confirmed these behaviors, but with very generic answers. The evaluator, on recommendation of a peer debriefer, changed this portion of the interview guide to a new line of questioning: “as a result of their swimming experiences, have your athletes changed?” with the follow-up, “do you think you’ve played a role in that change?” The deductive coding process occurred at the same time as preparation for an interim findings meeting with key program decision-makers. For the sake of clarity, this process is detailed in this section of the chapter, even though the majority of deductive coding happened after the interim findings meeting. Using Dedoose software, the evaluator read each interview transcript, applying deductive codes to excerpts that discussed the sensitizing concept connected to each deductive code. Table 7 shows a list of all deductive codes, the number of coaches for which that deductive code was applied to an excerpt, and the total number of excerpts tagged with that respective code. 81 Table 7 - Deductive Code Application Code Title # Coaches # Excerpts Coach 21 54 Context 20 39 Needs/Challenges 17 53 Course (Technology & Pedagogy) Global assessment of course How & where you took the course Concerns or Improvements 21 14 19 59 26 63 Salient features & reactions Other Technology/Pedagogy Limitations of Online Course 13 11 9 31 19 23 Content Knowledge 1 1 Learning from Course Useful topics Didn't understand Burning Questions Already Knew It 6 20 11 12 11 8 99 14 18 17 Rely on Sport Experience Recall Missing or Underemphasized Processing Translating & Implementing Changes 6 1 1 1 17 15 1 1 1 51 Reflecting Quest for new knowledge Community of practice Bricolage Outcomes from Implementation 17 13 16 6 4 42 23 43 13 9 Barriers to Implementation Other Content Knowledge 10 8 21 14 Attitudes Professional Mindset Conscientiousness 2 15 14 2 32 22 LTAD Valuing education & professional development Other Attitudes 13 9 13 35 22 33 Coaching Behaviors 1 1 Video Age-appropriate workouts Fun incorporated properly Goal Setting Be a role model 17 19 19 18 11 18 37 32 30 17 Develop athlete as a person Other Coach Behaviors 18 17 40 40 82 Table 7 cont’d Code Title # Coaches # Excerpts Coach Development Mentorship 12 7 34 15 USA Swimming Services 17 37 Miscellaneous 1 1 Good Quote 12 19 Totals 21 1,225* * Represents the total number of codes applied, not total number of excerpts Interim findings meeting. To provide key program decision-makers an opportunity to react to preliminary findings from the inquiry, the evaluator prepared an interim findings presentation using the deductive analysis results from seven high scoring coaches that provided the greatest variability of experiences with the course. The interim results addressed the first two evaluation purposes: program perceptions and utility; program effects. Specific findings were presented to the group as trending positive (i.e., interim results support this conclusion) or jury is still out (i.e., deeper understanding of this topic is required through further analysis). These designations reinforced the interim nature of the findings, and helped to provide fodder for discussion around topics where more understanding was required. Reactions and points of discussion from below provided points for the evaluator to clarify in the remainder of the analysis. Reaction to evaluation interim findings. The key program decision-makers expressed pleasant surprise at the coaches’ general satisfaction with the Foundations course, as well as the general acceptance of the importance of long-term athlete development, the proper incorporation of fun into practice, and the use of age-appropriate training for swimmers. The interim results showed that most coaches valued using fun, but that some clubs made it more of a priority than others, which generated some concern amongst the key program decision-makers. They noted 83 that there is a perception amongst the membership that “you can either be a high-performance club, or you can be the kind of club where athletes have fun,” stating on the contrary that many of the most successful clubs were places where both attributes were embraced. Better clarification of this topic was requested, given that USA Swimming had recently launched a new marketing campaign entitled The Funnest Sport, which was built around the concept that swimming is a sport where everyone can participate and enjoy their sport experiences. The key program decision-makers were not surprised to hear that video-use by coaches was spurious. One sport performance consultant acknowledged that coaches had the capability to do video analysis with swimmers, yet lacked practical strategies to manage time effectively. “It takes five seconds to shoot a video, but ten minutes to analyze it,” therefore, coaches tend not to make it a regular part of their repertoire. Three topics that the evaluator presented as jury is still out (adoption of a professional mindset, coach being a role-model, and developing the athlete as a person) received a great amount of attention. The key program decision-makers offered two potential explanations for professional mindset. First, it might simply be that coaches are getting buried in all of the other content. Second, it might be that there is not a clear connection between acting in a professional manner and the positive consequences, such as increasing your perceived value as a coach or better athlete performances. There are very clear links between the other behaviors taught in the course (e.g., teaching technique, age-appropriate training) and positive outcomes (e.g., faster swimmers, better athlete retention). Coaches may simply gravitate to the course content that they believe can get them the results they want. The evaluator agreed to explore the issue of professional development in greater detail. Questions about the best ways for coaches to develop the athlete as a person were also 84 raised. The relative merits of a deliberate life-skills instruction approach versus a reactive teachable moments strategy were discussed, but it was noted by one staff member that if developing the athlete as a person is not even on a coach’s radar, nothing will happen at all. The evaluator noted that these responses were complicated and that the interim report did not capture enough detail, vowing to explore them in greater detail for the final analysis. Another finding that was submitted for the committee to think about in greater detail was an acknowledged weakness of the program that most were aware of: that the Foundations program is the only guaranteed point of contact between USA Swimming and its coach membership. The evaluator detailed several accounts where the course had begun a conversation about a particular topic that was both interesting and highly applicable for the coach. However, unless the coach was extremely proactive, the idea might often be forgotten as the coach moved on to the next topic in the course. In particular, several coaches noted that PDF documents downloaded during the course were information-rich, but in only two cases did coaches note that they had saved these PDFs and returned to use them in their implementation of a new idea. Therefore, the evaluator suggested that some thought be given to strategies for continuing the conversation, to provide booster shots to the original dosage from the course. Evaluation process effects. This meeting was an opportunity for the key program decision-makers to learn more about how their program worked, by having program elements reflected back to them. In one striking example, three staff members discussed how the most significant effects appeared to be athlete development and the importance of fun, which were two implicit desired effects of the program: Staff member 1: You know, it’s interesting. I’ve just been thinking about it, because I was… I was the one who mainly worked with the curriculum, going back and 85 redesigning it, and I almost felt like… we were… overusing, or like… it [sic] was going to get tired of hearing about long-term development, because I was like pounding that into everything we did… Staff member 2: And here it is. Staff member 1: …and here it is! I mean, and so… it’s kind of like, what did we pound? And it… and it worked. Staff member 3: They heard it. Staff member 1: You know, and that’s good! And I think… fun was another thing that was pretty… that was pounded, although it wasn’t frivolous, everybody’s gotta do silly things fun, but the concept of it has to be enjoyable… was pounded pretty hard. So, the fact that those two emerged here is kind of… refreshing. And so maybe it’s like, one of the things… cause one of the things we talked about in the January meeting was that we didn’t do a real good job before we started this of… of what we want the outcomes to be. Um… we just kind of took our material, and put it in a new format. But, these were two of the definite outcomes that… that I was conscious of throughout. So, whatever we do in the future, of having those desired outcomes in front of us, before we design the course, is hugely important, because it did come through, which is good to see. Evaluator: Yeah… it’s something that you… you can almost think of it as dosage. Staff member 1: Yeah! Evaluator: You know? You gave them a big dose of fun and long-term athlete development, and it had an effect. This exchange shows evidence of the evaluation helping to show the key program decision- 86 makers how to make implicit desired effects of the program more explicit during the design process. A discussion that ensued later in the meeting discussed the merits of the previous approach to coach education, where information was laid out for coaches to take what they wanted, as opposed to a cookbook style course that stated there was one correct way to coach. The concept of dosage was used to illustrate why the current course, which forces the user to listen to the message before moving on to the test, may have stronger effects on coaches who would have otherwise skipped right to the test without hearing the message. The previous approach to coach education, where knowledge was presented for all takers, might have worked well for the best coaches who wanted to get all of the information that they could; but it didn’t have any safeguards against a coach who was not interested in learning or hearing the message from USA Swimming, and who was simply trying to pass the test to get certified. Thus, when compared to previous versions of the course, the current iteration does much more to ensure that a minimum dosage of programming reaches every coach. In addition to gaining a greater understanding of how the program worked through the process of evaluation, the key program decision-makers asked multiple questions about the nature of the inquiry process itself. Patton (2011) notes that most program administrators have their own notions of data collection and research, and these notions can shape their expectations of what constitutes significance. One staff member asked directly about the significance of these findings: were 21 interviews with coach participants enough to confidently state these findings? The evaluator responded by suggesting that with the research method employed, causality was not a goal, but then asked, “how certain do you need to be about this program? Do you need to demonstrate that it causes improvements in coaches? Or do you need to know that it is associated with improvements in coaches?” This was tabled as a potential direction to explore 87 after the final report was delivered. Another question later in the meeting addressed the extent to which the evaluator had contacted the total number of coaches who were eligible to do an interview. Such questions indicated the key program decision-makers’ general concern with inquiry methods and how they were essential for yielding a concrete description of the program. Brainstorming possibilities. Early in this meeting, the evaluator discussed the value of finding ways to continue the conversation between USA Swimming and the coach as a means of furthering the coach education process, and helping to support the implementation of any changes that the coach might make to their coaching practices or to their program as a result of taking the Foundations of Coaching course. Towards the end of the meeting, there was discussion around some form of follow-up method. For instance, following a coach’s completion of the Foundations course, they might receive an e-mail with a reminder or a checkin about a particular topic of interest. Social media, particularly Twitter, were thought to be potentially more useful, as they might allow for more openness in the dialogue, to allow for ideas to travel from USA Swimming to the coach membership, but also from the membership back to USA Swimming. This type of approach might allow for coaches to share their own best practices around specific topics of interest, e.g., share examples of how coaches are doing goalsetting with 10&under swimmers. The issue of getting coaches to embrace attitudes and behaviors that might have less immediate value to them, especially the notions of a professional mindset and developing the athlete as a person, was discussed at length. The evaluator noted a coach education program that he had helped to develop (Gould, Lauer, Driska, et al., 2012), and noted that a key component of that program had been the value of having well-known and respected coaches reinforce the message. This approach was discussed positively amongst the key program decision-makers, 88 noting several coaches that they could probably enlist. The embrace of long-term athlete development principles and age-appropriate training methods gave encouragement to the group, but it also helped to bring USA Swimming’s Progressions for Athlete Development (PFAD) model to the forefront. It produced fruitful discussion amongst the key program decision-makers, with one staff member comparing the PFAD model’s implementation to more vigorous attempts to change policy by USA Hockey and the United States Tennis Association (USTA), noting that, I think we think [PFAD is] a nice idea, and we want to promote it, but we’re not committed to… changing… materially changing anything… to impact it the way they’ve done with cross-ice hockey, with smaller tennis courts, and… we just don’t… it’s not… it’s not something we’re willing to make a stand on. Furthermore, the discussion consensus provided that the implementation of PFAD has not been as vigorous, because the longstanding philosophy of USA Swimming has been to support club independence and foster a diversity of coaching styles and programs, because this independence and diversity has produced the most successful swimming nation in the world (in terms of Olympic medal counts). But by respecting medal counts and choosing not to implement policy from the top-down, it might have allowed widespread poor coaching practices to persist at greater levels than they should have. The evaluator noted that there might be some advantage to rolling out a policy through coach education, allowing it to filter through the membership with each new wave of coach members that enters the field. This way, it could spread at the grassroots level, rather than being implemented in a top-down nature. There might be value in waiting and learning from the top-down implementations of long-term athlete development models by USA Hockey and USTA. Another staff member noted that the current version of 89 PFAD was going through revisions, and that the preliminary evidence supporting coach acceptance of PFAD principles might energize the PFAD principles’ revision and re-release. At the end of the meeting, there was some brief discussion about what resources would need to be mobilized if changes were to be made to the Foundations course. The evaluator brought up the potential for creating a survey that could be taken as part of the course that could provide useful feedback on the course and its effects. Two staff members discussed what would be necessary to do with ePath Learning (the company which maintains the course) in order to make needed changes. The interim report meeting served several functions: to provide an opportunity for key program decision-makers to react to interim findings, to request clarification on certain findings, to spark discussion around certain topics in advance of the full findings being released, thereby giving staff members time to react and prepare, and to continue the process of building value for the evaluation and inquiry processes amongst the key program decision-makers. Inductive Analysis to Address Purposes of the Evaluation Following the methods outlined in Chapter 3 of this document, the evaluator conducted an inductive analysis directed towards addressing the five purposes for this evaluation: (a) program utility, (b) program effects, (c) program delivery, (d) areas for improvement, and (e) utilization of process and findings. Due to the fact that details of the inductive analysis varied between different categories (e.g., useful topics, coach attitudes, coaches behaviors), a brief description of the analytical strategy is presented at the outset of each of the following sections, whenever it differentiated from the approach described in Chapter 3 of this document. Needs and Challenges To understand how coaches might perceive the utility of a coach education program, it is 90 important to understand their needs. Because new coaches come into the Foundations program with a high variety of backgrounds and experience, it is highly plausible that needs and challenges could vary significantly. For coaches with no experience, they may not have an idea what their needs will be when they start coaching. However, all of the coaches in this sample had been coaching for a minimum of four months, and thus had an adequate understanding of their working environment and the daily challenges that they experienced in their jobs. This line of questioning intended to serve as a preliminary needs assessment. Although it should be noted that a full needs assessment is much more contextually-based, this needs assessment was conducted for the purpose of demonstrating the congruence between coach needs and the programing provided in the Foundations course. The evaluator posed the question to coaches at the outset of the interview, phrased generally: “could you give me a sense of some of the day-today challenges you face in your job?” Responses were coded with a “needs/challenges” deductive code, and excerpts tagged with this code were read and reduced to themes. The evaluator then organized these themes into representative categories. Four categories encompassed the range of coach needs and challenges expressed by the sample: swimmer management; coach development; team management; facility management. Swimmer management. A total of 12 coaches discussed challenges with managing athlete behavior, keeping athletes focused, and challenging each athlete appropriately. Six coaches discussed the challenge of “getting the swimmer into the right lane.” This referred to the process of finding the optimal level of challenge, where a swimmer possessed enough skill to complete a workout, but also experienced enough challenge in the workout so that they could grow and continue to develop as a swimmer. Coaches discussed how a swimmer’s physical needs could be complicated by their social needs, which they recognized as an important and 91 integral part of why swimmers participated in the sport. Conflicts arose when a swimmer was moved to a higher level of training (i.e., different practice group), while their friends were swimming at a lower level of difficulty. Managing this constant tension between meeting the myriad needs of a swimmer was described as an ongoing challenge, and a challenge that they felt would never have a resolution. A second challenge of swimmer management discussed by six coaches was keeping swimmers focused on the process of training, racing, and continuous improvement. Improvements in swimming are typically measured by fractions of a second, and yet require a high amount of effort and attention must be focused on minute details of technique. Swimmers generally prepare to swim their fastest, or peak, at the final meet of the season (a process called tapering). The challenge reported by these coaches was to keep the swimmers focused on a goal that was three to four months in the distance, when their natural tendencies are to focus on what is immediately in front of them. A third challenge of swimmer management, discussed by five coaches, was managing behavioral problems, which ranged from athletes not listening to the coach to more extreme examples of fighting. Several coaches discussed the difficulty of working with athletes who demonstrated special needs, a process that was often complicated by parent behavior. Coach development. Nine coaches discussed challenges related to their own development as coaches. Three of the youngest coaches (between 19-23 years-old) discussed the challenge of differentiating yourself as a coach from the rest of the athletes. All three of these coaches worked for the program where they had competed as athletes, and thus knew many of the younger athletes as former teammates. Three coaches discussed their concerns relative to professional advancement. One of these coaches discussed a paradox in that there seemed to be very few opportunities to advance professionally, and that professional advancement for many 92 age-group coaches often involved moving to the intercollegiate level. She also discussed the fact that it was difficult to see her advancement as a female coach, given that so many program head coaches in her area were male. The other two coaches expressed concerns about how they would be able to make age-group coaching a viable career for themselves, while at the same time, they were putting so much of their energy into building their programs into viable and sustainable programs. Two coaches discussed the challenge of being confident in your decisions as a new coach, and two coaches discussed the need for having a mentor. Team management. Seven coaches discussed the challenges of managing a team, which were marked by three themes. Most prominent was the challenge of working with parents, such as finding appropriate ways to communicate with a dissatisfied or unreasonable parent of a swimmer, or handling situations that were created when a group of parents ganged-up on a coach after one parent was dissatisfied with a decision that a coach had made. Another challenge was that of taking over a new program, which was often complicated by parents or existing staff who were accustomed to the ways they had been working prior to the new head coach arriving. Finally, communication amongst the coaching staff was at times lacking or problematic, which created problems in workout administration, or limited effective communication with parents. Facility management. A small, but not insignificant theme of lack of lane space was mentioned by two coaches. It is mentioned here because the issues of swimmer management are often dependent upon lane space. If a coach has a limited number of lanes in which they can place different groups of swimmers based on their skill-level and training needs, more compromises must be made that often end up putting athletes into a lane that is too challenging, or not challenging enough, or into a lane where their social needs are not being met. Program Utility and Perceptions 93 The key program decision-makers at USA Swimming had little evidence of how the course was perceived amongst coach members, save for two sources of anecdotal information. First, the general number of complaints about coach education requirements decreased after the adoption of the current, online version of the Foundations of Coaching course. Second, they had received a number of complimentary emails from coaches, thanking them for developing the course, and noting how helpful it had been. Thus, a primary purpose of the evaluation was to present a much more detailed and systematic picture of how coach members perceived the course. This section will present findings on coach reactions to the course, specific features of the course that stood out for coaches, and some practical limitations of the course. The evaluator sought to get coaches most salient memories of the course, rather than to cue coaches to specific elements of the course (this was done later in the interview). Thus, the question was intentionally phrased broadly, i.e., “So… what did you make of the Foundations course?” The evaluator then coded these responses inductively, looking for common patterns of responses. General appraisal of the course. Three coaches stated unequivocally that they truly enjoyed the course. A more common response was a positive impression of the course that was qualified relative to their low expectations for the course. Typical of this category of response included sentiments like: “it was a long day of training, but I got a lot out of it,” (Coach 2) or, “compared to other courses I’ve had to take, this course didn’t seem like a waste of time,” (Coach 15) or, “all of the information isn’t, like, useless information,” (Coach 9). Not every coach response went to a like or dislike appraisal of course; instead, some coaches paused to consider the content from the course. Five coaches noted rather generically that they found the course helpful or informative, and thus had a moderately positive view of the course. More detailed responses indicated that the course had reinforced or resonated with the views of the 94 coach, or provided an important reminder of some topic related to coaching. The coach had been familiar with the information coming in to the course, and the course served to reinforce the knowledge, or promoted a viewpoint that resonated with the beliefs of the coach. During initial meetings with key program decision-makers, a concern that arose was the degree to which older coaches (as compared to new coaches ages 18-25) or experienced coaches (who had coached in high school or other non-USA Swimming affiliated programs) would be receptive to the knowledge and viewpoints presented in the course. This mattered because the science of swimming has made significant advancements over the past 15-20 years, and many formerly widespread and accepted coaching and training practices are now relatively out of favor. Addressing this concern, Coach 2 stated that after having been out of the sport for twentyfive years (having been a competitive swimmer in her teenage years, but returning now to coach as her children were moving through the sport), that the course reinforced several ideas she had learned as an athlete, although it challenged several other ideas she had. To her, the course seemed to say, “hey, you thought you knew this stuff, just because you swam once upon a time, but really, you don’t know as much as you thought you did.” (Coach 2) Another coach, who had extensive coaching experience for teams not affiliated with USA Swimming, and thus came into the course with a wealth of swimming knowledge, stated that the course had reinforced knowledge that his mentors had helped him build early in his career. These data support the notion that the course could resonate, reinforce, and to some degree, help coaches update and adapt existing knowledge when presented with the most current knowledge. Four coaches considered the layout and stylistic elements of the course when prompted to recall the course. These coaches reported that the course “had a nice flow,” that the user interface was intuitive, and that elements of the course “kept me engaged.” Coach 18 was 95 particularly complimentary, noting that she had taken the previous DVD-based version of the course. She stated that the design of the course makes a difference, noting that the course was well laid-out, the fonts were clear, and that it appeared “shiny and new. That kind of stuff makes a difference.” She also noted that the online version of the course fit much more easily into her workflow, making it easier to complete between tasks at her primary job. Not all reactions to the course were positive, although negative reactions were tempered and less common. Four coaches, notably older than 35 years, mentioned that they disliked online learning in general; however, they recognized that given the alternative of giving up a weekend to attend a face-to-face version of this course, online learning was the preferred method of taking this course. A second negative reaction was that the course was one more “box to tick” in order to become a coach. One coach suggested that, for him, the course had attained the level of a driver’s licensing test, i.e., something that he just needed to pass to earn his credentials. Another coach suggested that she had been rushed to complete all of her coaching credentials, and that this course was just one more credential she had to attain; her purpose was to “pass the damn thing and get it done.” Four coaches commented that the speed of voice-over audio seemed slow, but most recognized that they could see why the words were presented on the screen and read at the same time. One coach stated that she simply muted the audio and read the text instead, because she could read much faster than the voice-over. A common negative reaction was not necessarily a strong rejection of the course, but rather a casual indifference to the course. It was clear that for a few coaches, the course simply did not make much of an impression, as during recall, these coaches had a difficult time differentiating this course from other courses they had taken in the same time period. Almost all of the coaches who had difficulty recalling and differentiating the Foundations of Coaching 96 course from other online courses came from the group of coaches with lower scores on the Foundations 201 test. One young coach stated that the course “wasn’t very inspiring,” and that he had been “indifferent” to the course from the start, stating that he had learned everything he needed to know about swimming already, through his experience in the sport. He noted that learning on the job was the best way to learn, however, during the course of the interview, this coach discussed several instances from his recent coaching experience where content knowledge from the course would have been helpful to him. This instance suggests that being uninspired or indifferent to the message caused the message to be missed. Notably absent in these negative reactions are two concerns that were expressed by the key program decision-makers at USA Swimming. First, there were no wholesale rejections of the content knowledge presented in the course, or immediate negative reactions to some core element of the course. This had been the case with previous iterations of the course, which had employed a DVD with course materials, and an online exam with 180 questions (160 questions were required to pass and receive coaching credentials). Second, there were very few mentions of technical problems arising during the course. These problems were generally confined to coaches over 32 years-old (the threshold for so-called digital natives), who expressed the fear that their course results were not being saved as they progressed, or who expressed some technical problems. Coaches under the age of 32 also reported occasional technical glitches, but these users also reported that these were solved by logging-out and logging back in to the program. Thus, the course did not appear to generate a set of technical difficulties above the normal amount that could be expected, and that younger coaches were quick to employ a basic fix-all, whereas older coaches were more likely to place a phone-call to technical support. The generally positive appraisal of the course should not lead the reader to conclude that 97 coaches did not have a multitude of suggestions for improving the course. These concerns are addressed later in this chapter, in the section titled Areas for Program Improvement. Salient features of the course. Using the same broad questioning approach used to inquire about a coach’s general appraisal of the course, the evaluator sought to determine what features of the course were particularly salient in the mind of the coach. When a follow-up question such as, “So, what did you like about the course?” led to a discussion of some feature of the course, these interview excerpts were tagged with the salient features code for further inductive analysis. One feature of the course that received noteworthy attention were the videos that featured experts and coaches, both in talking-head style interviews, but also doing simulations and demonstrations of coaching. Videos of technique demonstrations were useful to coaches, but several coaches noted that being able to watch a coach interact with his or her swimmers was an effective modeling of ideal coaching behavior. The interactive questions and quizzes that were interspersed throughout the course also attracted attention. Of the eight coaches who discussed them, all seemed aware that they were placed in the midst of the course in order to retain your attention and reinforce content, but only one seemed genuinely annoyed by having to take all of them: “I’m going to read it and listen anyway, because I don’t want to fail the test!” (Coach 17) Two other coaches responded to a feature of the course where coaches are instructed to write answers to open-ended questions, which seemed useful at first, but as they progressed through the course, they began to wonder if anyone was actually reading their responses. One of these coaches stated that it would be interesting to see what other coaches had written to these open-response questions. The pacing of voice-over presentations attracted some reserved negative attention from four coaches, although all coaches acknowledged that they were fast readers and could read the 98 screen faster than the audio. Four coaches discussed using the PDF documents that could be downloaded throughout the course. However, when prompted to discuss how often they recalled and used these PDF documents, two of these coaches noted that they hadn’t read them since taking the course. The other two coaches had saved and organized them, using them extensively in their own work. Thus, while being perceived as quite useful by a minority of coaches, their actual use was moderated by the coach’s level of organization! Limitations of the online course. When prompted to discuss the limitations of the course, seven coaches discussed the idea that your experiences, both as a swimmer, and on-thejob as a coach, were major determinants of the type of coach you are (or would become). Coach 1 and Coach 17 had no competitive swimming backgrounds, but these coaches had vastly different experiences with the course, thus it was difficult to determine if swimming experience was the difference-maker. Coach 3 and Coach 13 suggested that the course had been successful for them, most likely because they had a strong support network of colleagues who valued coach education and professional development, and had created a culture where they could ask questions and discuss issues and difficulties, and work to solve problems together. Thus, the course was just another form of sharing and spreading useful ideas, and it was successful for these coaches because they were already in an environment that valued learning and development. This emphasizes the point that a one-shot coach education program cannot represent the whole of efforts to educate and develop coaches. Although coach development might share a common feature, such as an online course taken by all coaches, the coach development process can be incredibly distinct between two different coaches, and thus requires multiple inputs over the course of a coach’s career. Program Effects 99 Arguably the most important purpose of this exploratory analysis was to examine the effects of the program on the coaches who took the course. This section is presented in four subsections: knowledge from the course (i.e., useful topics); knowledge implementation from the course; coach attitudes; coach behaviors. Knowledge from the course. To understand more about what coaches got out of the Foundations course, the evaluator posed open-ended questions, such as, “so… what was your take-home message from the Foundations course?” or, “was there any topic that really struck you as useful information for you as a coach?” Although asking each coach about each course topic would have provided a wider range of perceptions on all of the course content, the intent was to see what coaches actually retained. The open-ended approach, rather than prompting the participant with recall of a topic before asking questions, allowed the evaluator to gain a better sense of what information had been retained by the coach; in some cases, the course had been taken more than 16 months before the interview. In all cases except for one, the participant could easily recall a useful topic from the course. Excerpts from these interviews that discussed useful knowledge from the course were tagged with one of five codes: useful topics, didn’t understand, burning questions, already knew it, and relied on sport experience. Excerpts tagged with these codes were reduced to themes, and the evaluator inductively categorized these themes to present a coherent picture of knowledge acquisition from the course. Useful topics. Twenty coaches reported that they had found some form of useful knowledge in the foundations course, and there were a wide range of responses. These responses were inductively coded in order to present a true picture of the knowledge gained by the coaches in this sample; the analysis did not attempt to fit the knowledge gained by the coaches into the predetermined topical categories of the course, as defined by the first version of the logic model. 100 Constructivist approaches to learning would predict that coaches took information from this course and reshaped it to fit into their existing knowledge about swimming and coaching; thus to retain the formal divisions of course knowledge along the topics delineated by the course’s organization would present a false picture of how the coaches in this sample had codified the new knowledge. While an argument could be made that a nomothetic approach of examining trends of knowledge acquisition across participants does not capture the highly idiosyncratic nature of learning, the intent of this evaluation was not to examine individual coach’s learning and development patterns. The purpose of the evaluation sought to look at the big picture of knowledge acquisition. The categories below represent how the coaches in this sample had parsed the knowledge from the course. Teaching and coaching principles. Fourteen coaches discussed learning some form of coaching or teaching principles from the course. The evaluator categorized these topics into four representative topics: drills and technique; swimming pedagogy; behavior management; goal setting. Eleven coaches discussed the value of the drills and technique videos that were presented in the course. These drills appealed to coaches with a wide range of swimming and coaching experience. For new coaches with little experience in the sport of swimming, they were new information. For new coaches with significant previous sport experience, these drills were not new to them; however, the videos had the effect of explaining why the drill was done. Even for experienced coaches (five or more years), they found some of these drills useful. In later discussions with key program decision-makers at USA Swimming, it was noted that drill and technique videos are extremely popular, and that drill and technique presentations at coaching conferences often attract a higher number of attendees. Six coaches discussed learning swimming pedagogy from the course, which could be 101 broadly defined as the interactive process that enabled information to flow from coach to athlete in such a way as to facilitate a change in the swimmer’s knowledge and/or behavior. For instance, three coaches discussed how they had learned more about the communication process, including the sandwich criticism approach for delivering a critique. Two of these same coaches also discussed how this approach was particularly helpful in their efforts to coach a swimmer through a race. For instance, Coach 11 stated: I thought that was quite helpful, um… but especially with a developmental swimmer, not overloading them with things about their race, you know? When they come to you, you um… I ask swimmers, “how do they think they’ve swam?” Even if they’re young, I say, “how do you think that went?” You know, and then I even go further, and I say, tell me a good thing you did and tell me something that you could have improved? Because, swimming… we always are… you know, looking to improve ourselves. There’s always something to improve. I tell them that, every single time. (Coach 11) Another useful feature was watching an actual age-group coach work with real swimmers to teach a specific skill. Finally, one coach who had recently immigrated to the United States described how the course demonstrated the “American way” of teaching technique to swimmers. This was particularly useful to him because technique had not been a major focus for the coaches that he had as an athlete growing up in his native country, and there was little modicum for how this process was intended to occur. Three coaches discussed behavioral management threads, such as understanding the individual differences between swimmers, and the importance of making a positive environment and the use of positive reinforcement techniques. One coach in particular cited a specific example from the course that stated that coaches should not have swimmers swim butterfly as 102 punishment for misbehavior. It should be noted here that in many other lines of questioning, coaches discussed issues related to behavior management, but these examples are meant to show where behavioral management principles were lifted directly from the course. In a similar nature, three coaches noted that the course section on goal-setting had been a useful topic for them. The adoption of goal-setting as a coaching practice is discussed later in this chapter. Developmental principles and considerations. Eight coaches noted that the course had taught them some key information about the different needs (biomechanical, physiological, psychosocial, emotional) of swimmers across the developmental continuum of childhood and adolescence. These needs were tied to USA Swimming’s long-term athlete development plan, called Progressions for Athlete Development. LTAD is a relatively new philosophy of athlete development, pioneered by Istvan Balyi and colleagues (1991; 2013). It is a prominent feature in sport governing bodies throughout the world, but has not taken hold in the United States as it has in other countries with more centralized systems of sport governance (e.g., Canada, United Kingdom, Australia, New Zealand). Coaches gained both a general framework of what is appropriate, and not appropriate, for each age group, and they also got specific strategies for writing workouts. These considerations are discussed in more depth later in this chapter, in both the coach attitudes section (long-term athlete development), as well as the coach behaviors section (writing age appropriate workouts, incorporating fun into practices). Planning training. Although the specific lessons about the planning of training varied across coaches, seven coaches reported that they had taken some useful information to inform their planning process from the Foundations course. One novice coach described how she had planned her season using a method outlined in the course. Another coach described how he had increased the amount of kick training that his swimmers completed after learning about this topic 103 in the Foundations course. Another novice coach stated that the knowledge about the physiological states induced by particular forms of training allowed him to understand what the swimmers would be feeling after engaging in a particular form of training. Reaffirming and reinforcing my stance as a coach. This may be the most informative find with regards to the topic of “useful information” in the course. As anticipated by the key program decision-makers, even new coaches come into a coach education program with existing knowledge about coaching that they have gleaned from their sport experiences, or through watching the coaches they had as athletes. Therefore, the knowledge presented in the course is not always very new to the coaches, and there was some concern that coaches would react negatively, and turn-off to the course because they felt that they already knew the information. While in a few instances, this may have been the case, it was more likely that coaches did not tune out the message of the course in its entirety. In fact, seven coaches reported that the course played the role of reaffirming and reinforcing their stance as a coach. This could be done through resonating with knowledge, or with philosophy and approach to coaching. It could also make a coach more confident. Coach 11 described how the process first, results second philosophy of USA Swimming resonated with his own beliefs about swimmer development. Another coach stated that after completing the course, she felt a greater confidence in her abilities as a coach: I think the course um, reassured me in my coaching ability, and I think I kinda, after I took it, you know, I packed it up nice and neat, and just said, like, okay I know that now…I’ve been reaffirmed, I…I knew my stuff in that course, I, I’m on a good path. Um, I learned, I learned stuff, from, you know, I learned new drills like I said, but, as a whole, I think it was mostly like…reassuring me that I had, I’ve been using the right drills, the 104 right techniques and that…I was confident in my coaching after I finished it. (Coach 10) What motivates athletes? Five coaches stated that they had a reaction to a section of the course that was entitled What Motivates Athletes? This had an important effect on Coach 8, a novice head coach, who subsequently reconfigured the age-group structure of his team in response to these principles. This was an eye-opening segment of the course for Coach 13: You know, maintaining a positive atmosphere, a lot of the positive reinforcement things. I mean, these are kids, and we’re adults, and we tend to come from a lot of backgrounds where, you know, in corporate America, that’s not the way to do things. (Laughter). You know, that’s not how things work. Um, those are just really good reminders that “hey you know what? First and foremost we’re here to have fun, you know, promote the sport as a lifelong opportunity,” not as, “we’re going to improve right away.” Um, you know, I think some of the…more of the psychological side of things was helpful for people who come from an environment working with adults. The effect on Coach 16 was different. He stated that this information “stuck in his brain somewhere,” after taking the course. After experiencing significant problems with swimmer interest, morale, and participation during his first season as a new coach, he said that something triggered him to remember the reasons why kids swim and the importance of fun. Thus, while not acted upon immediately, the course made a strong enough impression to have a latent effect on this coach. Nutrition and health. Five coaches discussed the value of the nutrition information presented by the Foundations course. For Coach 2, it updated her experience from being an athlete, when it was typical for athletes to binge on sugary foods before races. Coach 6 noted that she was familiar with good nutrition concepts, but that the information in the course 105 provided some new tips that she could pass along to swimmers and parents. Two coaches (#9, #18) discussed how the nutrition information was very helpful for them, but that it was difficult to shape the eating behaviors of their athletes. Although Coach 9 felt that his efforts might have been in vain, Coach 18 went to greater lengths to implement change. She realized that it wasn’t fair for her to expect busy parents to completely rearrange their eating habits to meet the needs of a swimmer, so she tried to put more responsibility on her swimmers, by conducting nutrition education sessions with her athletes while they stretched before practice. Communication. Five coaches stated that the course had provided them with specific information about communication with parents as well as communication with swimmers. Communication in this case is a general theme, but there were many instances throughout the analysis that referred to instances where communication played an important role (e.g., professional mindset, developing the athlete as a person). Interestingly, two of these coaches were immigrants to the United States, and had significant sport experience and coaching experience in different countries. Coach 20 summarized it as follows: My message from Foundations was communication. It was only communication. When I start with that, they know the techniques. I just need to fix their technique spots. If I didn’t have good communication with them, they wouldn’t care. With good communication they care and they listen to me. If they listen to you they will understand your tactics. This is communication for me, absolutely. Evaluator: Okay, so everything follows from communication? Coach 20: Absolutely! No communication, no improve on technique, it’s impossible. Providing a library of resources. Five coaches noted that the course was particularly 106 valuable source for documents that helped them to develop their programs and plan workouts. Throughout the course, coaches are prompted to open and/or download PDF documents by clicking a button called the job aid icon. Three of these four coaches said that these documents were helpful to them, but when prompted to discuss how they had used these documents recently, they stated that they hadn’t looked at them since taking the course. The other two coaches downloaded and stored these documents, and had used them on a regular basis since taking the course. These documents provided an obvious value to coaches, but the organized nature of their presentation throughout the course made it difficult for coaches to circle around and use this information to continually guide practice. The Areas for Improvement section of this chapter provides greater exploration of this topic. Useful topics summary. The coaches in this sample took a large variety of useful topics from this course, and only one coach in the sample did not report any useful knowledge gained from the course. Considering that the topics of greatest use were somewhat diffuse, this probably owes more to the interview approach than to the actual perceived usefulness of specific topics in the course. Recall that the interview process used an open-ended question that did not prompt coaches to provide their perceptions on each topic of the course. Instead, the approach was intended to gain a better sense of what topics were salient enough to be recalled with minimal prompting from the interviewer. Owing to the exploratory nature of this evaluation, this was acceptable for determining the relative merit of the different sections of the course. Had the evaluation found that only a few topics gained the attention of coaches who took the Foundations program, there could be cause for concern that specific topics were either not perceived as useful, not understood, or not valued by coaches. In the midst of interviews with participants, the evaluator developed the term buffet-style learning to describe the process by which coaches 107 found particular segments of the course to be useful. Participants confirmed this notion, stating that many topics were simply “common sense” knowledge. But many participants also suggested that they did not completely tune out the message when they were familiar with the content of the message. For instance, although most coaches are quite familiar with drills and technique elements in swimming, it was the most commonly referenced useful topic in the course. However, the second most frequently mentioned topic was developmental principles and considerations, somewhat to the surprise of key program decision-makers at USA Swimming. The message about developmentally appropriate coaching methods was woven throughout the Foundations course. In the 101 course, the issue was presented in a problem-based format, framed through a scenario entitled “Is five-year old Ryan old enough to start competing?” The Programs and Services Director had been responsible for this development in the course, notably the reconfiguration so that coaches would get some basic training before they ever set foot on the pool deck. The knowledge presented to coaches did not follow the traditional paradigm of learning, where principles are presented for the student to commit to memory, and then later find ways to generalize this knowledge to situations that they experience in their coaching work. The problem-based approach began by bringing this common question front and center, and then using this question as a modicum for the delivery of critical information pertaining to long-term athlete development. Thus, the distinction between these two useful topics is probably that for one topic (drills and technique), most coaches had enough sport experience that they could place the knowledge contextually. Even if a coach has no experience, he or she could observe another more experienced coach to figure out when and where to use drills and technique instruction within the 108 flow of a typical practice. In contrast, the knowledge about developmentally appropriate coaching practices might not have an obvious context for a new coach, and yet, “is my son or daughter old enough to start swimming competitively?” is a completely natural and common question for parents to ask, regardless of a coach’s experience level. This educational approach was completely intentional, according to the Programs and Services Director. The change in approach to a problem-based scenario arose based on conversations and feedback from new coaches that suggested that previous coach education efforts were not doing enough to address the needs of novice coaches. This analysis suggests that the change in teaching approach adopted by the Foundations 101 course had an impact on coach’s acquisition of both the factual knowledge related to developmentally appropriate practices for swimmers, but also the contextual intelligence to see where this information had value and could be used. It is notable that this approach merged elements of a traditional formal approach and a constructivist, problem-based approach to learning. The problem primed the learner to understand the context in which this knowledge could be applied, and the course presented several factual guidelines that the coach could use immediately to answer the question. Already knew it or relied on sport experience. In several instances, coaches discussed how they already knew or understood information that was presented by the Foundations course. Excerpts coded with this tag generally arose from a probe question, rather than a major focus of analysis, such as the useful topics code described previously in this section. The analysis organized themes into categories based on the source of the knowledge (e.g., learned from swimming experience), rather than the specific topic (e.g., exercise physiology, psychology). The intent of this analysis was to show sources of knowledge that could be potentially competing with the Foundations course. Once these sources had been identified, the evaluator sought to 109 determine if any one particular source was providing coaches with competing knowledge regarding a particular topic. The analysis showed that coaches learned from a wide variety of sources, including their swimming experience, previous coaching experience, other jobs, life, and college/other courses. Despite responses from 12 coaches, the consistency across participants was negligible. By cross categorizing for topical knowledge, there does not appear to be one topic that coaches are learning elsewhere. Despite the lack of emergent patterns in this category, two coaches discussed how the course had been an important refresher that challenged some of their preexisting knowledge. A concern of the key program decision-makers was the persistence of old coaching knowledge, or the problem of a coach who learns everything they claim to know through their own sport experience. For both of these coaches (2 and 16), the course had challenged some of their preexisting knowledge, and they had been receptive to the new message. Thus, they provide two examples of coaches whose knowledge was augmented by the Foundations course. Future research might examine the degree to which such a course augments existing knowledge structures; however, the focus of this evaluation was too broad to examine this question in significant depth. Didn’t understand. The evaluator sought to understand what coaches perceived as difficult in this course, which was done through the normal course of the conversation, or through the use of a probe question. The evaluator discussed this topic with 10 coaches, and five of these coaches noted that there was nothing confusing from the course. One coach pointed out that if anything from the course was too difficult, he could simply look it up on the internet. Another coach noted that nothing from the course was too difficult, and that considering he was not a native English speaker, the advantage of taking the course online was that he could “repeat, 110 repeat, repeat!” Four coaches mentioned that biomechanical and physiological principles from the Foundations 201 course were difficult to understand. Two coaches noted that they simply tuned out information that they felt was unrelated to their anticipated coaching positions working with novice swimmers. For instance, Coach 17 discussed how she tuned-out knowledge about the ATP-CP pathway. She suggested that it might be more useful, in her case, to take a course called “Foundations of Coaching the novice swimmer.” Burning questions. The evaluator sought to gain greater understanding of the persisting or burning questions that coaches had about swimming knowledge. This topic was typically addressed through a probe or follow-up question such as, “is there any one topic that you find yourself wanting to know more about?” From the list of themes generated from the coded excerpts of 12 coaches, one emergent pattern captured coach responses. Five coaches wanted to know more about what they could specifically do within their own programs to help swimmers make long-term progress. For Coach 20, this was about “writing my book” (figuratively) on swimmer development (i.e., a generalized set of knowledge on how to develop and progress a swimmer over time). For Coach 4, who was running a program with a participation-model (i.e., limited emphasis on competition) designed in deliberate response to the lack of alternatives for kids who did not want to specialize in swimming, her concern was how to keep kids involved in the sport given the threat of burnout. Coach 9 wanted to know what he could do to get swimmers in his program to step-up their level of commitment and involvement. Two other coaches discussed how they needed specific information on the types of intervals and training cycles that would provide the optimal level of challenge for their athletes. All of these coaches discussed their general satisfaction with the Foundations course, and its ability to help advance their knowledge, and yet, the persistent question was how to put this in the frame of season-long 111 or career-long athlete development. Knowledge implemented from the course. The evaluation sought to examine what knowledge from the course was actually translated and implemented in a coach’s day-to-day practice. Implementation shows a commitment to the value of knowledge, and while some knowledge has value regardless of whether or not the coach puts it into practice, there are specific topics that coaches should put into practice, and the evaluation sought to determine which topics these were. During the interview, the evaluator took a topic that the coach had previously mentioned as being a useful topic from the course, and then asked the coach to “walk me through how you have made that a part of your program,” or “tell me how you have implemented that knowledge in your day-to-day work.” Excerpts discussing this process were tagged with the translating and implementing changes code. Because implementation of knowledge often involves processing situational feedback, the evaluator also discussed topics related to the coach’s reflective practice and their use of communities of practice, as well as the coach’s quest for new knowledge. These three topics had their own deductive codes. Excerpts tagged with these four codes were reduced to themes and inductively categorized using the same process employed throughout the analysis phase. The intent of analysis is to present a coherent picture of how coaches translate and implement new knowledge, but also to examine how situational feedback (e.g., swimmer responses to new coaching methods) induces reflection, how coaches work in communities of practice, and how coaches search for new knowledge when a problem or question arises. These changes and processes are somewhat out of the purview of the Foundations course, and yet coaching science literature suggests that these are essential processes for coach development. An integrative analysis section will explore how the effects of the Foundations course interact with the effects and processes that occur in the coach’s 112 environment. Translating and implementing changes. The two areas of coaching addressed by the Foundations course that saw the greatest implementation were planning and workout design, followed by swimmer management and pedagogy. Scattered coaches implemented add-on strategies of goal-setting and nutrition education. One coach described how he had reorganized his team’s entire age-group structured based on recommendations from the USA Swimming Progressions for Athlete Development. Planning and workout design. Eleven coaches described changes to their planning and workout design that resulted from the course. Writing age-appropriate training sets, addressed by the course in both the Foundations 101 and 201 courses, was a simple change that coaches could make almost immediately, and was quite helpful for running more effective workouts. Coach 2 described how she used a guide from the course to help her structure the types of workouts that she ran. Coach 13, a coach with more than five years of experience at the time he took the Foundations course, described how he had changed the types of workouts that he planned for his swimmers between the ages of 7-12, notably through the inclusion of games and training sets that the swimmers perceived as fun. Three coaches discussed how the course had shaped how they addressed the physiological needs of their swimmers at particular ages based on the recommendations of the relative volume of specific types of training that athletes should complete based on their age and developmental stage. Coach 5 stated that he had deliberately increased the amount and proportion of kicking-specific training for his younger swimmers. He stated that practices were typically in the range of 1,400 yards, and that roughly 400 yards of each practice was kicking only. Three coaches discussed how the course had provided them with a practical means of planning an entire season. All three of these coaches had found this to be 113 particularly useful in jobs where they coached another team that was not a USA Swimming affiliated club (e.g., high school team) that had a relatively shorter season with a greater number of constraints, including a busy meet schedule, or swimmers with conflicting schedules who were unable to attend practice regularly. Coach 1, a novice coach, described how she followed the recommendation of breaking the season into thirds, and how this had helped her design a taper to help athletes prepare for end-of-season competition. Coach 1 also made an interesting observation about the value of the course for framing coaching knowledge. She realized that the course was not a formula for producing successful swimmers. Rather, she stated, “the course provides a framework… you fill in the blanks.” Coach 16, a coach with more than five years of previous coaching experience, stated that throughout his career, he had emphasized the fundamentals of swimming (e.g., proper technique, appropriate physiological training). The Foundations course had emphasized the importance of these aspects of his coaching philosophy, which he had felt were not necessarily reinforced by other coaches he knew and had worked with in the past. This validation of his coaching philosophy reinforced his will to continue coaching in this manner. Swimmer management and pedagogy. Specific skills and strategies for teaching and coaching swimmers were directly implemented by three coaches, who stated that videos from the Foundations course had been particularly helpful for showing them how to communicate with a group of swimmers, as well as how to instruct a swimmer in a one-on-one context. Coach 6, a coach with more than five years of previous coaching experience, stated that it was very helpful to see videos of coaches instructing age-group swimmers, and to watch the ways in which swimmers responded to this instruction. Coach 10 talked about how many demonstration videos freely available on the internet often showed demonstrations with elite athletes, but watching 114 videos in the Foundations course that demonstrated these skills with age-group swimmers was particularly helpful for her in understanding the unique ways in which children learn. Two coaches, notably both who were immigrants, described that it was particularly useful to see communication strategies demonstrated, such as the sandwich criticism approach. Coach 20 continually described this approach as the American style of coaching, which he particularly admired and believed to be much more successful than methods employed in his native country. Coach 13, an experienced coach, described how the course had changed the way he had responded to swimmers who were experiencing emotional upset following a poor performance, and the importance of understanding that no two swimmers would handle the situation in the exact same way. The course had sensitized him to the importance of knowing the difference between athletes, and he had then developed different approaches to communicating with them. Two coaches described how the course had influenced their approach to communicating with parents. Coach 5 made it a regular feature of his coaching to talk to parents when they arrived to drop-off or pick-up their children from practice. This was an opportunity for him to get to know parents better, and for him to learn more about his swimmers as people. Coach 22 stated that the course had increased his awareness of biting his tongue when a parent criticized him, because he understood more about the importance of parents in the process of developing a swimmer. In a good illustration of how the course worked in concert with the influence of experienced coaches and supervisors in his program, he stated that he had learned some very practical strategies about how to conduct himself when he received parent criticism, but that the course had also sensitized him to the importance of parents, and more acceptable ways of working with parents: I have a parent telling me… like today, I had a parent come in, and she’s like, “my kid’s 115 in C group.” And I’m like, “Okay, C group is in this lane today,” and she’s like, “no, it’s not.” I made the lanes… yes, it is. Um, just… and, you know, obviously I didn’t say that to her. But, just like, kind of, sometimes having to bite your tongue. Um, and knowing… knowing the appropriate thing to say to a paying parent. Add-ons – goal setting and nutrition education. Goal-setting and nutrition education were presented in the Foundations 201 course as practical strategies for helping swimmers to advance towards improvement and goal achievement. Five coaches made use of one of these strategies as a means of enhancing the performance of their swimmers. Goal-setting, discussed by three coaches, was identified as a coaching behavior of interest at the outset of the program evaluation by the key program decision-makers, and an analysis of goal-setting behaviors of coaches is presented later in this chapter. It is noted here because three coaches identified goalsetting as a practical take-away from the course during an open-ended question (without being prompted with the question, “is goal setting something that you do with your athletes?”). Two coaches described how they had made nutrition education a facet of their coaching. Both coaches discussed the practical limitation of talking about nutrition when they had little control over what types of nutritional decisions athletes made. Coach 18 took a reasoned strategy, noting that by talking about goal setting with her swimmers, who were primarily between the ages of 10-12, she was able to increase their knowledge so that when they moved up to the next age group and had more control over some of their nutritional habits, they would be primed to make better decisions about the type of food they consumed. Restructuring age-groups based on PFAD model – a miniature case analysis. Coach 8 described a profound effect that the Foundations course had upon the design of his program. After inheriting the program in an abrupt coaching change, he decided to reconfigure the age- 116 group structure of the team to bring it inline with the Progressions for Athlete Development model presented in the Foundations course. To do this, he employed a PDF document that he downloaded while taking the Foundations course, which outlined the ideal progression of a swimmer from initial age-group swimming involvement to high-performance training at the senior level (ages 14 and up). The PFAD document outlines the different training needs that athletes have particular sensitivities to at different developmental stages, as well as the major motivators for athletes at each age. Implementing these changes proved to be a challenge for this coach. Most prominent was the fact that arranging training groups based on swimmer’s demonstrated abilities and needs broke up an age-group structure that had been in place for several years, and in so doing, he disrupted the social bonds for some swimmers who were no longer practicing with their friends. This caused some complaints from parents and swimmers alike. A second problem was that sorting athletes into the new age groups was generally an easy process, but several athletes were right on the threshold of group membership based on his criteria, and the question was whether or not to keep a swimmer with athletes who were of a similar age but higher ability, or to group the swimmer with younger swimmers of a similar ability. Inevitably, the coach decided to keep some of these athletes with other athletes of a similar age – maintaining the social grouping at the expense of providing physical training that was well beyond the ability of what the athlete could handle. He acknowledged that the only way he could have implemented the new system perfectly would have been to completely start from scratch, which was not possible. He did note that athletes who had started in this system at a younger age, and had followed the system for a year, were in a better position than the athletes who were in older age groups when the change was implemented. This case presents many of the aspects of coach learning that are crucial to understanding 117 how coaches implement knowledge gleaned from a formal learning experience (e.g., online training). The course presented a factual, evidence-based framework that he could employ to improve the experiences of athletes on his team. Implementing the new framework provided situational backtalk (i.e., unforeseen constraints that must be renegotiated; Schön, 1987); in this case, the situational backtalk included instances where the framework did not fit with the experiences and needs of athletes on the team. He had to handle that situational backtalk in the form of athlete and parent complaints. He found a way to accommodate these athletes in such a way that enabled the framework to survive. This is particularly illustrative of the many types of dilemmas that face coaches making necessary program changes. This coach showed a willingness to commit to an idea, but also a willingness to commit to its implementation. The process of implementation was something that he had to figure out for himself, for the most part. This prompts a question about the nature of online training, and the degree to which it can support a difficult transition such as revamping the entire age-group structure of a team. While this evaluation has shown ample evidence that coaches can implement some practical ideas from an online training, thereby demonstrating the efficacy of an online training for transferring knowledge that can be translated to everyday coaching situations, the question remains if an online course is able to support the more complex process of implementing a major revision to the way in which a team functions. The situational backtalk from this implementation prompted questions that were not addressed in the Foundations course, and in all fairness, it might be beyond the capability of an online course to provide all of the answers to those questions, many of which could not be foreseen. Thus, the challenge appears to be not so much the dissemination of a new evidence-based practice, but support in the implementation of a difficult change to an existing program. 118 Although this case does not represent the majority of the experiences of coaches in this sample, the evaluator chose to include this case analysis because it was representative of the strength that the course could have. The reader is cautioned not to over-interpret the inclusion of one case as representative of the causative effects of the course on all coaches who take the program. However, it should be noted that a major program overhaul is generally not something that can be initiated and undertaken by assistant coaches, and assistant coaches constituted the majority (n = 16) of the coaches in this sample. Second, the case provides an interesting background for future research in the adoption and implementation of new policy and evidence based coaching. Previous studies (Black and Holt, 2009; Lang & Light, 2010) have demonstrated the difficulty of adopting and conforming to the LTAD model, and this case provides an interesting perspective on the use of an online coach training model as the catalyst for initiating the adoption of new evidence-based strategies. It is particularly interesting to note that an online training seems to have had as much success in facilitating the implementation of an LTAD model as other methods, such as top-down policy changes. This case illustrates that the idea might arise from an online training, but the implementation process requires a lot of diligent work and follow-through on the part of the coach. Thus, it might be a more effective strategy to use an online training to introduce the LTAD strategy, while human resources (e.g., coaching consultants, mentors) might be devoted to helping a coach handle the difficulties that arise during the implementation process. It might also be noted that younger coaches had a strong sensitivity to the topics associated with long-term athlete development, particularly regarding the prevention of athlete burnout and overtraining (addressed later in this chapter in the coach attitudes/long-term athlete development section). The combination of this sensitivity to the principles of long-term athlete development, coupled with detailed knowledge and practical 119 strategies presented in the Foundations course, may provide an effective model for the dissemination of evidence-based practice that will benefit the experience of swimmers. Given the attention to long-term athlete development in both the research and coach education domains, the adoption of long-term athlete development principles by coaches and programs, particularly through the use of online training programs as a catalyst, could be a potential topic of research interest and attention. Reflecting. Reflection has been identified by coach education literature as a preferred method of coach learning (Trudel and Gilbert, 2006; Gilbert & Trudel, 1999), and has been primarily based on the work of Donald Schön (1987), whose work identified the process of reflection employed by experienced practitioners in a range of professions. In the initial focus group with key program decision-makers, the group indicated that getting coaches to “think more about what they do” was an objective of the course that is consistent with the research on the importance of reflection for coach development. This phenomenon was addressed through the use of the conscientiousness code (see Coach Attitudes section later in this chapter), but the evaluator also tied this concern to a wide body of literature discussing the importance of reflection for coach learning. Throughout the interview process, the evaluator asked direct questions and follow-up questions to assess the degree of reflection induced by the course (e.g., “Tell me more about how that part of the course impacted your thought process.”), as well as situations arising in the coach’s environment (e.g., “So when you experienced that problem/setback, how did you come to a resolution?”). Given that a purpose of the evaluation was to explore the effects of the course, themes were organized into two main categories: reflection induced by the Foundations course, and reflection induced by situations arising from the coach’s environment. Instances of reflection are presented in this section, but the reader is 120 referred to the Coach Attitudes – Conscientiousness section (later in this chapter) for further evidence of the development of this attitude in this sample of coaches. Reflection induced by the Foundations course. The course induced reflection on three different topics: technical elements of swimming, athlete psychosocial/emotional needs, and coaching style/philosophy. With regards to the technical elements of swimming, Coach 10 stated that the course was particularly helpful in making the connection between different drills and how they were actually linked to the fundamentals of the stroke. Coach 2 noted that reflection on the course put her knowledge from her own swimming experience (25 years previous) into question, noting that many of the training methods espoused by the course went against the methods that her coaches had used when she was training and competing as an athlete. These two examples are particularly useful in answering a concern raised by the key program decision-makers early in the evaluation process: does the course effectively challenge and compliment the knowledge that a coach has gained as a former (or current) competitive swimmer? These examples illustrate how new knowledge from the course was integrated into existing knowledge structures, and how this knowledge was rearranged in a way that was more useful for the coach, but was more in line with current evidence-based practices. Coaches 8 and 13 both provided stories detailing how the course got them to think more about how the psychosocial and emotional needs of swimmers were important to address. Coach 19, although at first somewhat reluctant to note that the course had any salient effects on his coaching practice, later noted in the interview that the course had illustrated the difference between how boys and girls respond to different experiences. In this case, it was somewhat apparent that he had experienced an issue in his work as a coach, and the interview process had triggered his memory that the course had addressed this point. While this may not seem at first 121 to support the efficacy of the Foundations course to deliver the message about gender differences, it does illustrate the effect that a conversation can have on recall of facts presented by the course. Thus, it lends support to the notion that the online course in itself is not sufficient to deliver the message, and that there could be potential value in human follow-up after the course. Finally, three coaches provided evidence that the course had gotten them to rethink the way they coached, or perhaps more generally, their philosophy as a coach. Coach 3 referred to the Foundations course as a “starting point” that had pushed him to look for new ideas as a coach, rather than showing him that there was “one correct way to coach.” This coach described working with a staff of coaches who were all very supportive in this venture, who shared ideas frequently using a Facebook group, and who worked together to solve problems that were being faced by swimmers on the team. While Coach 3 did not state that he would have been any better for having taken the Foundations course, he did suggest that the course had played a role in getting him to think more and be more open-minded to the problem-solving process. Coach 18 stated simply that the course had motivated her to see how much she could improve as a coach. Coach 12 stated that the course section devoted to the coaching philosophy had a profound effect on her, in particular, one question that asked, “what kind of a coach do you want to be?” This had prompted her to reconsider past coaches she had as a swimmer, reflecting on what the best coaches had done to enable them to make a difference. Reflection induced by situations arising from the coach’s environment. Situational backtalk from the coach’s environment induced reflection on different topics: interpersonal relations, training and racing, and coaching style/philosophy. With regards to interpersonal relations, a variety of examples were presented. Coaches 10 and 12 both detailed situations 122 where they had encountered an athlete having an inappropriate emotional outburst following a poor performance, and they had reacted quickly in order to confront the swimmer. Both coaches described how afterwards, they asked themselves if they had handled the situation properly, and what they might do differently the next time a similar outburst occurred. Coach 10 talked about how she circled-back to have a conversation with the athlete to explain why she had reacted the way she did. Another area where this reflection came into play was in coach’s reactions to hostile parents. Coaches 4, 6, and 22 discussed instances where they had experienced an unpleasant experience receiving parent feedback, and they detailed the processes by which they handled this feedback, and then reflected on how they might avoid this type of interaction in the future. Coach 4 stated that as a staff they had discussed this issue and had adopted a policy where, instead of trying to provide a parent with a hasty answer to their question, they asked instead, “can I get back to you about that tomorrow?” In this mean time, they were able to discuss the issue as a group, and novice coaches were able to run this issue by their superiors, and this process allowed the group to process the issue, and be more prepared should it arise in the future. Coach 6 stated that she had learned how to work with parents the hard way, having been “thrown into the fire” in her first head coach job coaching a summer-league team as an 18 year old. When she experienced trouble, she contacted her previous coach, who was able to help her understand why parents got upset, and what she could do to best handle it. One of the lessons she learned through this relationship was that she could not make everyone happy. Coach 22 also illustrated the immense value of learning from more experienced coaches at his team in handling irate parents. I mean, I think… I think I realized when you take a coaching job, I think that it helped that it was as big as [club name omitted] because you know you kind of have to be more 123 professional than other clubs, but I think it helps you realize when you take a coaching job for a big club that you’re part of a big group, you know. You can’t really be unprofessional or you are not going to keep your job, or you’re going to get in trouble or… So I think I realized before the course that you always have to be professional, but I think the course just reaffirmed that. Quest for new knowledge. This process, arising from the key program decision-makers, was described as follows: ideally, a coach who takes the Foundations course will be set onto a quest for new knowledge. In short, the course will help to trigger and cultivate the coach’s intellectual curiosity. Twelve coaches discussed their quests for new knowledge. These coaches were typically asked this question in different points of the interview, often in response to when a coach had mentioned a particular problem they had experienced; this became an opportunity for the evaluator to pursue the method in which the coach had worked to solve the problem. The evaluator did not ask this question of every coach – it was a probe question. With such a line of questioning, it was disruptive to the flow of the coach’s narrative to continually ask, “would you attribute this to taking the course or not?” Thus, the attempt at causality was not a specific aim of this probe question. However, two coaches discussed the causal nature of the course with regards to the quest for new knowledge. Coach 1 noted that the course clearly demonstrated that there is not “one right way to do something. You have to educate yourself.” Coach 12 stated that the psychology part of the course “sparked my interest, and got me thinking.” The analysis of this code was meant to describe the process by which coaches seek more information, and to provide a basic understanding of the sources of knowledge. Six coaches noted how they used the internet when they had questions, making it the most common source of new information discussed by coaches in this sample. Only two coaches made specific mention 124 of the USA Swimming website, although both of these coaches stated that the resources were immense and helpful. Coach 5 stated that when he visited the USA Swimming website, it was often when he was just looking for general knowledge, rather than when he was looking for an answer to a specific question; this process would last between 30-45 minutes. When looking for a solution, or a video, that search process would last less than 10 minutes. Coach 6 stated that access to the resources on the USA Swimming site was rather difficult, in that good resources were often hard to find. Another coach mentioned the difficulty of continued access to these resources, given that this coach’s day job employer used a secure network. If this coach wanted to consult a resource during the lunch hour while writing a workout, it was not possible to do so. Print media was not entirely dead, as three coaches noted that they enjoyed reading print sources, such as books and magazines. Two formal opportunities for new knowledge include professional development webinars conducted by USA Swimming, and continuing education that is done in affiliation with American Swimming Coaches Association (ASCA). Although only five coaches discussed these forms of new knowledge as a legitimate course of action, the content of their discussion sheds important light on potential gaps between coaches interest in continuing education and development, and their actual professional development behaviors. With regards to USA Swimming designed and coordinated webinars, only two coaches directly mentioned that they were aware of them. Although their attitudes towards them were positive, they had not crossed that divide between contemplation and action. Three coaches discussed how the quest for new knowledge should involve an easy path into formal education opportunities, although two of these coaches had more than five years of experience and did not state a specific interest in this pathway. However, one coach specifically stated that he was actively attempting to figure out 125 how to begin a path of continuing education with ASCA. After taking the Foundations 101 and 201 courses, he was surprised to find that ASCA conducted its courses in a different format, and found it very difficult to navigate the ASCA website. Communities of practice. The term communities of practice has been used extensively in the teacher education literature, and more recently, in the coach education literature (Culver & Trudel, 2008). A community of practice refers to a group of individuals that deliberately comes together for the purpose of problem solving. This rather loose definition of a community of practice was intentional, given the descriptive nature of this evaluation. The evaluator’s intent was to investigate and describe the process by which coaches gathered around a problem and worked together to solve it. Studies that examine communities of practice have stringent criteria for examining the processes why which problems are brought to the group, and the structure of conversation and problem-solving processes. This evaluation, broad in its scope, simply sought to assess whether coaches engaged in group problem-solving, and provide cursory details about this process. A total of 15 coaches discussed some form of coming together with other coaches to discuss dilemmas they had encountered in their practice, or for the general purposes of professional development (e.g., discussing hypothetical issues that might occur, and how they might be handled). Eight coaches mentioned that their community of practice consisted of in-house staff meetings or informal discussions with coworkers. Two coaches discussed instances where coworkers, particularly coaches on staff with greater experience, had been go-to sources for questions. Coach 12 noted that through this relationship, a few coaches on their large staff shared articles with each other. Three coaches discussed having staff meetings where they discussed how to handle difficult parent questions or situations where a parent might need to be 126 confronted. Coach 4 noted how they trained to handle difficult parent inquiries, and Coach 5 discussed how these staff meetings were an opportunity for the head coach to address professional decorum that was expected of the coaches on staff. Coach 22 stated that staff meetings were the place where he had learned expectations on how to interact with “a paying parent” who was upset. While these meetings appeared to generally be top-down, driven by the head coach, there appeared to be room for assistant coaches to raise concerns and topics of discussion. These formal and organized meetings contrasted with a decentralized community of practice that was described by three coaches. The most illustrative example, provided by Coach 3, is provided in anecdotal form, to show its essence, and to show how the Foundations course integrated into this strong community of practice. This coach described how he and his coworkers had formed a very unique and effective community of practice, and how this had enabled their staff to work more effectively. They employed a Facebook group to communicate with each other, which provided a simple way to quickly share useful resources found on the internet. They had also acknowledged their coaching strengths and weaknesses, and they actively delegated tasks based on those strengths and weaknesses. Coach 3 noted that the Foundations course had simply been one more source of knowledge that fed into their community of practice, one more source of topics they could use as a conversation started. This was an interesting finding, because it seemed like the primary unit of continuing education and development was the community of practice, and the Foundations was just data that was imputed into this existing system. Coach 3 backed this up, stating that the course had provided him with some interesting and useful tools, rather than being the primary framework or source of his coaching knowledge. 127 Coaches described how they had formed and developed local networks of coaches, typically through conversations with other coaches at meets. However, these meetings tended to be brief and very topical. Six coaches mentioned that they saw immense value in local clinics, but interestingly, many stated that they would appreciate the ability to discuss issues with coaches who were in similar situations, rather than learning exclusively from top-down, expertdriven sources. Forming these networks with other coaches served a variety of purposes. For newer coaches (1, 5, 9, 19), it seemed to be simply that they preferred to learn from other coaches. This finding discredits the assumption that younger coaches are digital natives, and therefore more likely to prefer and benefit from learning online. Several novice coaches under the age of 25 stated that they saw immense value in having clinics, specifically local and smaller clinics, where they could engage with other coaches. They were not very specific about what they hoped to learn at clinics, other than stating that they wanted the opportunity to learn from other coaches. Two coaches with greater amounts of experience stated that there would be a lot of value to hearing from other coaches how they had solved difficult problems. For Coach 6, this mention came up in the context of discussing how to handle difficult parent situations. For Coach 7, she had greater concerns about how she could advance as an age-group coach, given that there were few female coaches her age who were continuing to coach at this level: I don’t know how it is in other LSCs [local swimming committees] from a head coach perspective, but when you walk on the pool deck and you’re one of the youngest ones there, you know, sitting there in, you know, [Junior Olympics], in a room with the other coaches, you’re not as well…recepted [sic]. Less people are willing to talk to you than the coach that, you know, they may have never talked to, but you know, is 50 something 128 years old, and has been doing this for 15 years. Evaluator: Right. Yeah…yeah its interesting cuz…you know, its almost sort of like, how do you handle yourself, or carry yourself or, how do you make that leap, from…you know, being, yeah, the novice coach, to, to being the, the one that…everybody knows and… Coach 7: Are asking about your swimmers. Not going, “oh, you have a nice little swimmer there.” … So, I, I, I would like to see something like that because…not that I feel…like I’m drowning, but, you know, sometimes when you go to those meets, that’s kinda, how you can feel, and I feel that maybe the younger coach retention maybe isn’t quite what it should be. Barriers to knowledge implementation. Through both discussion and follow-up questions, the evaluator investigated the barriers to implementation of changes in programming, typically resulting from the Foundations course. Ten coaches discussed topics related to these barriers, and through inductive analysis, the evaluator developed four categories for the barriers experienced by coaches. The first category was other coaches, generally the coach’s superiors. Coach 12 discussed how, despite their supportiveness, the head age group and head program coaches were quite intent in the way that they trained their athletes, and disinterested when it came to her ideas of making practice more fun and enjoyable for swimmers. Another coach stated that his boss was critical of any efforts he made to “have fun,” although he acknowledged that part of this critique may have been directed towards him in an effort for him to differentiate himself from the athletes and define himself as a coach. It should be noted that in both of these cases, the coaches were younger than 25 years, and it may be possible that in these two cases, older head age group coaches were not resistant to the idea of fun, but may have been concerned 129 first and foremost with the young coach’s ability to assert their control over the group. Two coaches noted that implementation of changes required a certain level of confidence or assuredness, sometimes lacking in a new coach, and in such cases the coach themself may have been the barrier to implementation. A second category of barriers was entitled inevitabilities. The evaluator humorously wishes to advance the following swim coaching corollary: kids like junk food, you will never have enough lane space, you can never wipe the slate clean when you take over a new program, and people are inherently resistant to change. These were barriers described in some form by four different coaches, and showed how several inevitable realities of swim coaching will continue to provide significant barriers to change. Coach 9 illustrated the uphill battle of getting kids to make smarter nutritional choices: Evaluator: You talked about promoting good nutrition and, umm, having coached kids, I know that it’s important that you promote nutrition. I also know that kids really like Snickers bars. Coach 9: Yes, me too. Evaluator: (Laughter) So do you get a sense of like how well kids embrace that… nutrition? Do you know what I mean? Coach 9: Yesss, they hate it, they don’t like to eat healthy. (Laughter) And umm, it’s terrible, but I mean, they eat stuff, and you just… keep feeding [your message] to them, and getting, like, in their head that, like, they need to start making better food choices. And they click slow, but not as much as I liked them to click, but, baby steps. Parents constituted a third barrier, specifically when coaches attempted to make broad changes, such as age-group assignments for different athletes. Coach 13 discussed how parents 130 might attempt to accelerate the progress of their child if they had demonstrated exceptional performance. Coach 8 noted an occasion where a distressed parent complained about how his changes to the team’s age group structure had interrupted the social connections that her child had built on the team. Coach 6 recounted a similar example where a group of parents ganged up on the coaching staff after they had made significant changes to the age-group structure, resulting in swimmers being shifted to different training lanes. The coach described the difficulty of communicating the coaching staff’s point of view to parents who were set in their beliefs. Finally, in one instance, the Foundations course itself was cited as a barrier to implementation of a goal-setting program. Coach 12 noted that about one month into her first season of coaching, she felt that goal-setting would be beneficial for the swimmers. She stated that she went back to the Foundations course, but struggled to log-in to the course, and gave up because she didn’t have time to pursue the technical issue. Instead, she relied on her memory of goal-setting from the course, as well as her own experience with goal-setting as an athlete. This example is included as an illustration of how knowledge goes from the Foundations course to the coach and then to implementation. The barrier in this case arises from the fact that the course material was not easily accessed. Although it may not seem like an insurmountable barrier to reestablish one’s login credentials in order to find course material presented in an online class, it is often a question of how much executive functioning a coach wants to devote to an information search. Before the advent of mobile devices, and if the coach had knowledge of goal-setting from a previous educational experience, that knowledge might be stored in a notebook. Now that the course has moved online, the information may have never been taken down into some concrete form by the coach. This is particularly troublesome for information that might have 131 latent value (as opposed to immediate usefulness) to the coach. When it comes time to implement the changes, if the information is not close at hand, it may be difficult for the coach to find a way to implement these changes as they were presented in the program. Coach attitudes. Attitudes (i.e., mindsets, habitual ways of thinking) can influence the types of behavior exhibited by coaches, and the provision of knowledge from the course (Ajzen, 1991), therefore, a goal of the evaluation was to identify and assess the types of coach attitudes related to successful learning and implementation of best-practices for coaching swimmers. The preliminary focus group with key program decision-makers at USA Swimming identified four desired coach attitudes that should result from taking the course. Attitudes were defined as habitual mindsets or ways of thinking that coaches who took the Foundations of Coaching course would ideally demonstrate. The deductive analysis employed five child codes to highlight these five sensitizing concepts: professional mindset, conscientiousness, long-term athlete development, valuing education and professional development, and other attitudes. After deductively coding excerpts, an inductive analysis process converted excerpts into themes. The evaluator organized themes by employing a typology (Patton, 2002) to determine the origin of coach attitudes. On one end of the spectrum were attitudes that were clearly attributed to the course, and on the other end of the spectrum were attitudes clearly attributed to factors arising outside of the course (e.g., head program coach, previous coaches, other courses). In some cases, themes fell somewhere between the two end-points, where the attitude was jointly attributed to the course and to factors arising outside of the course. A description of common themes regarding coach attitudes is provided in the remainder of this section. Professional mindset. A professional mindset was defined as knowing and accepting the responsibility that a coach has in the proper execution of their coaching duties. In the deductive 132 coding process, the evaluator coded instances related to this broad definition with the Professional Mindset deductive code, yielding a total of 32 codes, referenced by 15 coaches. Excerpts were converted to themes, and then sorted according to the typology of attitude origin. Overall, the data do not strongly suggest a common origin for attitudes related to a professional mindset. For some coaches (n = 6), the course influenced the development of a professional mindset. There were two pathways of influence; first, by modeling professional behaviors through the use of videos of experienced coaches interacting with swimmers and parents, and second, by prompting coaches to reflect on their own professional philosophy by asking, “what kind of coach do you want to be?” Two different coaches (notably both male coaches between 19-24 years old) suggested that the completion of the course conferred that a professional standard had been met; one coach took a positive spin, stating that the course was a valuable credential as he sought jobs and professional advancement; the other coach took a more derogatory approach, stating that the course was akin to passing a driver’s examination, little more than a barrier that must be overcome in order to obtain a necessary credential. There was also speculation by three coaches (notably all older than 25 years) that the course could have little influence on a coach’s professional mindset unless that coach already had some sense of professionalism prior to beginning their coaching job. Coach 7 stated: Um…I really feel that… how you start out as a coach, your beginning years as a coach, is really formulated after the coaches that you had. Whether it be, you know, your club coach, your high school coach, your college coach, coaches that you’ve worked with, um, I, you know, I feel that, that has a big part of your development, um, not saying that someone who’s a recent college graduate would be immature…but…you know…I think that…its, it’s a little bit, it’s a little bit hard to, you know, change, you know, change the 133 color of a horse. You can’t, you can’t remake them… um, just by taking a 60-minute course. It’s not, it’s not gonna change them. This sentiment expressed by Coach 7 resounded with four coaches who expressed that their professional mindset was developed through experiences they had with colleagues, previous coaches they had as an athlete, and other life experiences. However, the data did not suggest that developing a professional mindset was an either/or proposition, as two coaches suggested that while their primary models of professionalism were colleagues and mentors, the course reinforced the importance of a professional mindset. Conscientiousness. Conscientiousness was defined as thinking more about what you are doing as a coach (similar to being more reflective), being more deliberate with your actions, and not just doing the “same old thing you have always done.” The course appeared to be moderately effective (10 of 21 coaches) at getting coaches to become more conscientious about the way they do their work, and this was accomplished in two ways. The first way, embraced by a subset of seven coaches, was by specifically causing the coach to be more attentive to swimmer’s needs, skills, and abilities. For three coaches, this meant being attentive to swimmer performance during practices, and making adjustments based on how swimmers performed. Two coaches shared that the course made them more attentive to the psychosocial and emotional needs of their athletes; Coach 13 had significant experience before taking the Foundations course, and specifically cited the course as a causal agent. Coach 16 suggested that course had created cognitive dissonance regarding his coaching philosophy and approach to the purpose of training; this coach reported having a reflective moment six months after taking the course, where in the midst of a difficult season and facing adversity from athletes and parents, he let go of some of his old beliefs on the rigid nature of training, and began to implement some of the 134 philosophies and ideas behind age-appropriate training. Finally, Coach 8 suggested that the course provided a “map” that helped him to understand how each daily practice linked to the next, and how these linked to future outcomes like taper/championship meets as well as longterm swimmer development. The course increased conscientiousness through introducing specific knowledge and skills for some coaches. However, for a set of four coaches, notably all female, the course increased their conscientiousness by getting them to think more about the type of a coach they wanted to be. Two of these coaches discussed how they came to the realization that being a coaching was about “more than just running a practice.” One coach directly attributed this to the course section on coaching philosophy that prompted her to think back to the coaches she had as an athlete, and to think about the coaches who were most effective, who made the most difference in her life. The lack of items that demonstrate conscientiousness arising from factors other than the course should suggest only a lack of counterexamples or negative cases. It does not suggest that a conscientious mindset only arose from the coach taking the course. The possibility exists that these coaches were exceptionally conscientious people to begin with, and as such, might have been more sensitive to the message. However, given the specific examples attributed to the course, it is reasonable to conclude that for half of the coaches in this sample, the course had some impact on their conscientiousness as a coach. Long-term athlete development. Long-term athlete development (LTAD; Balyi, Way, & Higgs, 2013) is a general model of holistic athlete development that specifies the types of skill instruction and activities that are developmentally appropriate for every age. Different sports have developed a specific LTAD plans that meet the needs of athletes and coaches in their 135 respective sports. USA Swimming developed Progressions for Athlete Development, a longterm athlete development plan. Implementation of the LTAD model has been done on a voluntary basis for USA Swimming affiliated teams. This is in contrast to sport governing bodies that have more assertively marketed and pushed LTAD models, such as the United States Tennis Association (Quick-Start/10 & under tennis), or USA Hockey (American Development Model), as well as Sport Canada, which has oversight and funding control of sport programs, and has required every sport to implement LTAD principles at the team level. The Foundations of Coaching course places a significant amount of attention on introducing LTAD principles, and thus the key program decision-makers identified the adoption of LTAD principles as a key attitude that would be embraced by a coach having taken the Foundations program. Ten coaches reported that the course had directly influenced their attitudes towards longterm athlete development. For four coaches, the programming directly changed something about the way they thought about swimmer development or the purpose of swimming training. For instance, one coach discussed how the course had helped him to understand the psychosocial and emotional needs of younger swimmers, as well as how athletes might respond to competition at different ages. Coach 2 stated that she had an “aha! moment” when she watched the segment about how the practices for young swimmers should not simply be a reduction in volume for what the senior age-group’s practice, but instead needed to have an entirely different structure and address different athlete needs (much in the same way that a mathematics lesson for students in third grade is not a watered-down version of what students would do in tenth-grade). Coach 16 noted how the ideas and principles surrounding LTAD, notably, the importance of properly incorporating fun into practices, got “stuck in my brain somewhere,” and six months later, after experiencing a difficult season, he engaged in some deep reflection. He had been operating 136 under the assumption that swimming had been a difficult experience for him, in terms of being required to participate in “almost militaristic” training as an adolescent, and therefore, it was simply a right-of-passage for all swimmers. After experiencing significant parent and athlete backlash, he suddenly revisited the idea of increased fun during practice and realized the importance of that message. This coach then revamped practices to incorporate fun into practice. He noted that holistic athlete development (his term for LTAD) was important in the modern era, because “30 years ago, athletes or parents would not question a coach, but that has changed.” For the other six coaches, the course resonated with their beliefs and personal experiences with burnout. At least three coaches mentioned the importance of teaching fundamentals and proper technique to swimmers when they were younger, reinforcing the LTAD model. Adoption of the LTAD model of training and preparing swimmers is not simply measured by a coach’s adoption of positive attitudes of acceptance towards LTAD; it might occur through the adoption of specific behaviors, notably the use of age-appropriate forms of training, and the proper incorporation of fun into workouts (these two behaviors are discussed at length in the following section on coach behaviors). Thus, it might also be possible for a coach to engage in these behaviors without having first embraced LTAD as a guiding principle. A number of coaches spoke about their experience implementing these behaviors, and they are presented here, as they greatly inform some of the nuances and difficulties of implementing the LTAD plan. Implementing a LTAD plan can, in many cases, go against the established norms regarding athlete training; it is a classic case of an idea that works in theory, but struggles in practice. Coaches detailed several instances where they experienced situational backtalk as they implemented LTAD best practices. As discussed in the previous section regarding implementation of changes, one coach rearranged his team’s entire age-group structure following 137 the detailed map provided by Progressions for Athlete Development. This alleviated some common coaching problems, such as having athletes with poor stroke proficiency engaging in difficult endurance training that they were not capable of completing. It also provided specific criteria for advancing from a lower age-group to a higher age-group. In its implementation, however, the coach ran into several problems. For instance, rearranging age-groups based on skill competencies meant that some athletes were moved from higher age-groups to lower agegroups, much to the dismay of some parents, who felt that this reflected poorly on their children. It also had the effect of separating swimmers from their friends. The coach acknowledged that at certain ages, establishing and maintaining social bonds with peers is a very important part of keeping swimmers involved in the sport; here, the new LTAD plan was breaking social bonds in order to meet the physical needs of the athletes. The coach was experiencing situational backtalk from the implementation of a program that, in theory, was supposed to improve the experience of athletes on his team. The coach endured most of the criticism, and made some exceptions to the group structure in order to appease some parents, but in the end, was able to establish this model of swimmer development. Although this coach had the resolve to see through these changes, it was apparent that it created a lot of job strain for him; it is entirely possible that, facing this type of parent and athlete resistance, a new coach would be content to maintain the status quo rather than implement the LTAD model. Furthermore, the LTAD problem could be delineated into two camps. The first would be those coaches who work for programs where there is a lack of any type of long-term plan for swimmer development and management. These coaches were quite interested in getting guidance on how to build this plan. The second camp would be those programs that had an established development model (note that not all programs had a model in-line with LTAD 138 principles), where the difficulties arose in terms of articulating the specifics of the plan, or of individualizing training (i.e., getting swimmers into the proper training group), or making the transition from the old to new program resulted in problems like those described above. In some cases, teams had delineated specific criteria for advancing swimmers from lower age-groups to higher age-groups, whereas some teams had not. The data provided in this example paint an incomplete picture of LTAD adoption, but provide a preliminary picture of coach actions with respect to LTAD policies. Valuing education and professional development. The key program decision-makers identified the idea of being a life-long learner as central to becoming a successful coach. However, it was unclear where the course helped to reinforce this idea. Nine coaches discussed themes related to how they valued their education and professional development, and for seven of these coaches, it was apparent that this was a personal value that they brought with them into the course. Four coaches noted that their club had a strong value for professional development. One coach, having recently immigrated to the United States, provided an interesting perspective that perhaps may have been overlooked by the other coaches in the sample (all natives of the USA). This coach stated that the course made it apparent to him that he needed to keep learning more, and that the attitude towards continual learning and professional development is much better in the USA when compared to his native country. These findings suggest that for the coaches in this sample, professional development or continual learning is something that arises within the individual or the swim team’s organization. It could also suggest that the value of lifelong learning was not communicated very strongly by the course. There is a strong possibility that the message about continual learning must be complemented by other sources. Other coach attitudes. Coaches were asked with an open-ended question if this course 139 had influenced the way they think, or their general mindset towards coaching (e.g., “do you think you view yourself any differently, or think differently about your position as a coach, than if you hadn’t taken this course?”). A number of participant responses did not fall into the four predetermined attitudes, and thus the evaluator inductively analyzed these themes into categories. The strongest attitude to emerge from this analysis was the idea of embracing the athletecentered philosophy, i.e., understanding the individual differences between swimmers. Two coaches noted that the course improved their confidence in their abilities to coach, and two different coaches noted that the course improved their attitudes towards USA Swimming as an organization. The course had demonstrated that USA Swimming embraces the importance of educated and professional coaches who make a difference for their swimmers. Coach 3 provided a unique insight that might be overlooked in a nomothetic (across-participants) analysis. This coach noted that by watching a video of an experienced coach teaching and demonstrating skills to swimmers, he learned some important guidelines for how skills were communicated and demonstrated in swimming. This coach had been a competitive wrestler throughout high school and college, and had coached wrestling and several other sports in addition to swimming. He stated that in wrestling, it was essential for a coach to make contact with the athlete during skill instruction, otherwise the skill could not be learned. Contact, therefore, was normalized and acceptable within the sport culture of wrestling. However, in swimming, contact with athletes was not generally within the sport norms, and had been discouraged by the athlete protection training (part of the initial training requirements for coach membership). This point was not overtly emphasized by the coaching video, but instead was something that he had picked out. This point is especially salient given that many coaches may come from backgrounds in other sports, and the instructional methods that are taken for granted within the swimming community 140 might be completely unfamiliar to coaches from other sports. Making these differences more evident in the videos may have merit. Six coaches expressed uncertainty about their knowledge and attitudes towards swimmers with special needs (e.g., ADHD, autism spectrum disorders, behavioral management problems, cognitive impairments). The significant finding is not that a consistent attitude towards these swimmers exists, but that coaches expressed uncertainty on how to think. All of these coaches provided vivid examples of how they had struggled with an athlete’s behavior, and in some cases, how these situations had been made more complicated due to a parent’s response to the coach’s concern or actions. The issue of special needs athletes went largely unaddressed in the Foundations course, and yet it seemed to be a pressing need for several coaches in this sample. Coach behaviors. Given the exploratory nature of this evaluation, the assessment of coach behaviors was done on a self-report basis. Although coaching behavior assessments (Smith, Smoll, & Hunt, 1977) provide a more direct measure of coach behavior, these instruments were not employed for two reasons. First, the behaviors of interest were defined more broadly than the behavioral paradigm that guides the CBAS. Second, the purpose of this evaluation was to gather exploratory and preliminary information about specific behaviors that the course aims to develop in coaches who take it, and thus self-reported behaviors was more useful to address this purpose of gathering broad information about the types of behavior, rather than a measure of adherence. For instance, one of the behaviors of interest was incorporating fun into workouts. Rather than assess the degree to which fun was incorporated, the purpose of the analytical framework allowed the evaluator to explore the multitude of ways in which coaches valued fun and incorporated fun, providing rich examples for the ways in which coaches were doing this. The exploratory nature of this evaluation would allow future evaluations and studies 141 to examine these specific types of fun in greater depth, perhaps through the use of quantitative methods of behavioral observation or athlete surveys. Given that causation of behaviors often arises from a range of cognitive and perceptual influences (Ajzen, 1991), the analytical framework did not seek to determine if the course was a causal agent in these behaviors or behavior changes. However, in some interviews, coaches directly mentioned that the course caused them to adopt a particular behavior, or change an existing behavior, and this link was noted. The initial focus group with key program decision-makers at USA Swimming identified six coaching behaviors of interest were (a) the use of video analysis, (b) writing age-appropriate workouts, (c) incorporating fun into practices, (d) goal setting, (e) being a role-model, and (f) developing the athlete as a person. The evaluator asked coaches directly if they engaged in the behavior, and then used probe or follow-up questions as a means to gather a greater level of understanding about how the coach engaged in the behavior. Excerpts of these responses were coded deductively according to these six predetermined behaviors (as well as a seventh code for other behaviors), and a list of themes developed from these excerpts. These themes were categorized and organized inductively within these deductive bins, thus attempting to capture the range of responses around the sensitizing concept of a specific coach behavior. The inductive strategy for each coach behavior varied from one code to the next in order to best display the closest approximation of unifying themes that ran through the data. Use of video analysis. Eighteen coaches provided feedback about their use of video and video analysis. Some coaches initially interpreted this as the use of pre-recorded videos of swimmers demonstrating proper technique or racing strategies, but the analysis was focused on the use of video analysis, i.e., video recording a swimmer and then showing that video to the 142 swimmer with the purpose of making some change to the swimmer’s stroke. Three coaches described intentional use of videotaping software or other video-based analytical software to improve technique. These coaches were rather intentional about making time for athletes to participate in these activities. The majority of coach video use (reported by 12 coaches) could best be described as rare to occasional use, and generally non-intentional or reactive use of a mobile phone camera to take video of a swimmer and show it to them for analysis. This might be done as a spur-of-the-moment decision for the coach, where the video analysis is directed to fix a problem. It is not intentional, meaning that these coaches did not describe plans to video analyze all of their swimmers, or deliberately devote practice time to video analysis. Four coaches simply stated that they did not use video analysis at all. One of these coaches had just gotten the team’s governing board to purchase new underwater video equipment; in contrast, another coach stated that they had an underwater camera, but it was simply too much of a hassle to use it. A smaller than expected number of coaches are making intentional use of video, given the ease of video analysis made possible by devices like an iPad with a waterproof, submersible case, as well as the proliferation of affordable app-based software for analysis (e.g., Coach’s Eye, Dartfish). The fact that a number of coaches make rare or occasional use of video analysis would suggest that the low degree of adoption of video analysis is not due to a lack of skills on the part of the coach, or a lack of equipment on the part of swim teams. Instead, there appears to be a lack of intention to use video analysis. It should also be noted that most coaches did video analysis on an individual basis, rather than using a systematic approach where all swimmers of a team, or within an age-group or training group were video analyzed. Video analysis is a timeconsuming process, and coaches may simply not be prepared to manage the large volume of 143 video content that would be generated by having 30 swimmers engage in video analysis. Thus, it might be useful to develop and teach strategies for coaches to integrate team-wide video analysis into their day to day practices. Writing age-appropriate workouts. A major concern addressed by key program decision-makers at USA Swimming was that coaches developed separate workouts that addressed the specific developmental needs of swimmers at each age, rather than simply watering down the workout given to the senior age-group, by decreasing yardage or increasing the time for interval repeats. Developmentally appropriate training methods show that practices for younger swimmers are not only quantitatively different (e.g., less total yardage, shorter distances for repeat swims, longer time intervals for repeats) but also qualitatively different (e.g., greater emphasis of fundamentals and proper technique for younger swimmers, different approach of the coach, incorporation of different types of games and fun training activities). Thus, the analysis was designed to capture the types of changes in training coaches made for younger and older swimmers. Nineteen coaches discussed the concept of age-appropriate workouts. Twelve of these coaches discussed the planning process for developing these workouts, and for seven of these 12 coaches, the planning process was guided by an overarching philosophy (of the coach or club) that helped the coach to determine the appropriate form of workouts at each level. Eight of these 12 coaches discussed how the decision of workout planning largely relied upon their own interpretation of swimmer needs and abilities, meaning that they had developed their own guidelines for what constituted age-appropriate (and in the view of the evaluator, these standards were appropriate). This is meant to illustrate that there was not a high level of supervision over what constituted an “age-appropriate” workout, rather, it was left up to the individual coach. 144 One coach summed up this issue as purely administrative; he had conducted a few conversations with his assistant coaches over what constitutes an ideal, age-appropriate workout for the ages of athletes they worked with. However, given that he was coaching another age-group simultaneously, the amount of supervision that he gave over workout construction and administration was minimal. Therefore, the guiding supports of the team’s age-group structure and developmental plan served as a backstop that helped to assure that coaches had a framework for workout construction. A second category of themes that arose was that coaches needed to develop the skill of accommodating the skills and abilities of swimmers during the midst of a practice. Five coaches described the importance of this skill, but it was not entirely apparent that this skill was enhanced by the Foundations course. Coach 5 suggested that by teaching very novice swimmers how to swim during swim lessons, he had learned about the individual differences that govern how some swimmers respond to a particular method of skill acquisitions while others do not. Coach 6 stated that she “knew her swimmers well,” and therefore could adjust practices as needed; however, this coach had been trained as an elementary school teacher and possessed several years of coaching experience prior to taking the Foundations course. Despite the lack of a causal link between the Foundations course and the skill of adjusting and adapting practices to meet swimmer needs, several coaches attributed increased conscientiousness to the course, and this raised awareness might help coaches be more aware when their swimmers are not experiencing success in their practices. Two coaches stated that they were not involved in the process of writing age-appropriate workouts. In both cases, the head coach was writing a generic workout for the older swimmers that was adapted for younger athletes. Both of these coaches, earlier in their interviews, reported 145 being indifferent to the Foundations course, and both coaches scored low on the Foundations 201 test. Both coaches described team environments where the head coach wished to maintain control over what the assistants were doing, but both of these coaches noted several situations where the plan the head coach wished to implement was not working effectively. In these two cases, it is difficult to ascertain where responsibility for writing age-appropriate workouts lies, as they both did not appear to have much control over the writing of workouts. On the whole, it appears that the coaches in this sample reported writing age-appropriate workouts, and that the Foundations course had introduced or reinforced the concept. The skill of accommodating workouts to each swimmer’s needs and abilities was cultivated through personal experience and reflection. In situations where coaches had little control over the workouts, it appeared that despite their knowledge that the workouts were not age-appropriate, they were not in a situation where they felt they could challenge the will of the head program coach who wrote the workouts. Thus, the course had some influence on this behavior, but implementation was dependent on the context. Incorporating fun into practices. A critical coach behavior identified by key program decision-makers was integrating fun into practices. The analysis explored the ways in which coaches integrated fun into workouts, such as through the use of games at the end of practice, or challenging drills and practice sets that athletes perceived as fun. Fun might also be perceived as the coach having an attitude or approach to coaching that was lighthearted and made the athletes happy. This has particular relevance for decision-making as USA Swimming recently began a marketing campaign entitled The Funnest Sport, which draws upon the inclusionary nature of the sport. Inductive analysis generated four categories of fun: fun built-in to practice or linked to training elements (14 coaches), games during practice (6 coaches), coach’s attitude towards 146 swimmers (5 coaches), and making fun contingent on hard-work (2 coaches). It should be noted that no coach in the sample vigorously denounced the idea that swimming should have elements of fun. The majority of coaches discussed specific ways that they built fun into the training that they did on a day-to-day basis. One way to do this was through providing more opportunities for swimmers to race. Coaches might set up situations where swimmers had to race each other, or where one swimmer had to race and meet a difficult time standard that would then result in the early end of practice or the entire group getting to play a game (commonly referred to as a “getout swim,” or a “hero set”). Two coaches discussed how they used relays (and relay based games) in order to stoke athlete interest. Five coaches mentioned how they used fun drills in order to teach and reinforce particular stroke technique elements. Six coaches talked about using a variety of games, but also activities that the swimmers had identified as fun (e.g., kick set disguised as a game) throughout all parts of practice. For instance, two of these coaches stated that their swimmers (between 9-12) really liked games that forced them to think during practice, such as a categories game, or a game of hangman. In the game of hangman, when swimmers guessed a letter correctly, they got to guess another letter towards solving the puzzle. If they guessed incorrectly, they had to swim another lap before guessing the next letter. Six coaches discussed how they used games at the end of a practice or during the offseason as a way to keep swimmers interested and engaged. The distinction between these games located at the end of training (or in place of normal training) and the other types of games that were integrated into practice, is that these games were specifically demarcated as something done at the end of practice, or in place of the normal practice. Setting up such a dichotomy between practice and games can often give the impression that swimming is work, not play, 147 whereas mixing games into the normal training routine can have the effect of telling swimmers that swimming is fun in itself. Coaches may hesitate to play games at the end of practice because their swimmers may anticipate or expect that they will get to play a long game at the end of every practice, which may not be the case. Nonetheless, the coaches who employed these types of games supported the notion that fun was a critical part of swimmer engagement and involvement, as well as a critical factor for retaining swimmer interest. Finally, two coaches insisted that fun in swimming was related to mastery and goal-achievement, which was contingent on hard work. They expressed strong beliefs that swimmers needed to find ways to make this hard work fun, understanding that rewards would be experienced in the future. One of the coaches who espoused this belief (Coach 8) noted that he saw an importance in making practice activities fun, but that there was just an inherent amount of work involved in the sport of swimming, and swimmers needed to find their own personal ways to adapt to this part of the sport culture. On the whole, coaches heard the course’s message about incorporating fun into workouts. More could be done to emphasize how challenges could be made fun, rather than simply providing a game at the end of practice as a reward. This concept was discussed with key program decision-makers at the interim findings meeting, where there was a suggestion of making use of social media to help share and disseminate examples of fun and challenging workouts that could be easily integrated into workouts for different age-groups. Given the range of interesting approaches to fun reported by the coaches in this sample, fun is something they are actively trying to cultivate, which would suggest that there would be a multitude of ideas to share via social media. Goal-setting. Coach interviews yielded a variety of responses to the question of “do you 148 do goal-setting with your swimmers?” Fifteen coaches described how they engaged in some type of a goal-setting or reinforcement process designed to help swimmers achieve goals they had set for themselves, goals that had been set for them by the coaching staff, or milestones that had been determined by the coaching staff. Three coaches stated that they did not engage in any form of goal-setting, and three coaches did not discuss the topic. The coaches in this sample expressed a wide-range of interpretations of goal-setting, including many that strayed somewhat from the best-practice understanding of goal-setting that is clearly expressed in the sport and performance psychology literature (Locke & Latham, 2002). Therefore, the intent of the analysis of this code was to show the range of goal-setting behaviors expressed, rather than to express the degree of adherence to the behavior, or the attribution of this behavior. Thirteen coaches engaged in a goal-setting process with their athletes. Of these 13 coaches, seven did this by providing generic advice about goals to their swimmers. This could be done by asking swimmers to set goal times for each event they wanted to swim, or by directing athletes to specific elements of their stroke technique or racing tactics. It might also be done through giving a swimmer encouragement when they had stated a goal, such as, “I want to qualify for the state championship meet,” followed by praise after they had met their goal. One coach stated that he told swimmers to have goal times (i.e., performance goals), but that he didn’t really follow through on tracking them or reinforcing any part of the process, a common problem with the use of goal setting by coaches. Another coach said that he and the staff frequently talked to their adolescent swimmers about the importance of the process of improvement, but did not have the swimmers identify any performance or outcome goals that they wanted to achieve. These two coaches are particularly illustrative of the issue surrounding an incomplete application of the goal setting process. The Foundations course emphasizes the 149 importance of setting goals, and the sport of swimming is replete with examples of the viability of setting goals. Swimming, compared to other sports, is particularly well designed for goal setting, because the feedback of time-improvement is very specific and measurable. However, the principles of goal setting are quite labor-intensive, involving individual meetings between coach and athlete, which can often be quite time consuming. Thus, making effective goal-setting a part of your program can be a major investment of coach time and resources. Despite the time-consuming nature of effective goal-setting, five coaches described the processes by which they conducted one-on-one goal meetings with athletes, through formal goal meetings or informal discussions on the pool deck. Three of these coaches described a very specific process, where swimmers would set performance goals (or “competitive goals” in the words of one coach), but would also set process goals, such as technique corrections they needed to make in order to achieve those performance goals. Coach 8 described how he had done goalsetting in the past as an athlete, but that the course gave him a better idea of how to do it with his swimmers. He used a PDF from the course that helped him to design a goal sheet for swimmers, and provided a template for conducting a goal meeting. Coach 6 used a range of kid-friendly goal setting sheets that she introduced during practice, and then sent home with her swimmers to complete as “homework.” These worksheets were designed using phrases and words that were appropriate to the comprehension abilities of the athletes she worked with. Coach 18 recounted a very similar story, in that goal setting had always been an important part of her swimming career, but that the Foundations course had reinforced the importance of using the goal-setting process. She had created a goal sheet that included competitive goals at different levels (e.g., win a “heat-winner” ribbon at a meet, make a regional time standard). The goal-sheet also included a question that asked the swimmer, “tell me one 150 thing about yourself.” The coach stated that this question often provided her with an opportunity to learn more about her swimmers as people with lives outside of the swimming context. This stronger connection also improved the quality of conversation around the types of goals that the swimmer was trying to achieve. This may be an important feature of a goal-setting meeting that could increase the adoption of goal-setting behaviors amongst coaches. For instance, it may be that the immense amount of time required to conduct one-on-one goal setting meetings represents an opportunity cost; there could be other ways to spend this time more effectively (e.g., swimming more laps, practicing under-developed skills, one-on-one technique instruction). There might also be parental pressure to devote swimming practice time to actual swimming, rather than devoting it to talking about swimming. However, the experience of both of these coaches shows that goal-setting meetings have been a great opportunity to strengthen the communication between coach and athlete, enabling the coach to learn more about what motivates the athlete, both as a person and as a swimmer. Given that developing the athlete as a person is a coaching behavior of interest for the key program decision-makers from USA Swimming, goal setting might be a crucial component of this process. The examples of these coaches lend support to this notion. Amongst the three coaches who stated that they did not use goal setting, a common reason did not emerge. One coach stated that they simply “did not get around to it,” while another stated that although the team did not do goal setting, they did “offer technique sessions.” This might suggest that goal-setting is equated with individual efforts that a swimmer takes on, above and beyond their normal commitments to the team, in order to improve. Finally, another coach said he preferred not to use goal setting because it put too much emphasis on goal times, and his major emphasis as a coach was technique improvement. When pressed about using 151 “technique goals,” the coach stated that they had not used technique goals either. Five coaches discussed their use of well-developed reinforcement systems. These systems were largely linked to milestones that the team and/or coaching staff had identified as important, and thus were important to the staff and the swimmers. Milestones were different from goals, in that they were often set by adults, or tied to the USA Swimming age-group time standards (i.e., time standards for each event in each age-bracket that correspond to a national percentile ranking; USA Swimming Age Group Time Standards, 2014). One coach stated that their team had a system of beads that represented the achievement of particular technique and time-based milestones. Another coach described how his program used a stroke skills chart to track the progress of swimmers; this chart measured a swimmer’s achievement of particular technique milestones. This coach stated he believed that goals were important, but that tracking progress was equally as important, if not more important. A third coach described a less formal system of providing feedback and reinforcement to athletes. In this case, swimmers were often given small technique goals to focus on improving during particular races (e.g., take three dolphin kicks off of each wall), and the swimmers were provided with a small piece of candy when they came to speak with coaches after the race, at which time the coach could reinforce the goals. Two of the five coaches that engaged in these developed reinforcement system also simultaneously engaged in one-on-one goal setting with athletes. One coach stated that their coaching staff carried a list of swimmer goals to all competitions; this enabled coaches to provide instant positive feedback and reinforcement when a swimmer had met a goal, which he believed made the value of the goal more immediate to the swimmer. Another coach stated that she used a sticker system to indicate swimmer progress against specific technique milestones; 152 however, it was not apparent that this sticker system was intricately tied to the novel goals that each swimmer had set during the goal setting process. In summary, the Foundations course introduces goal-setting to coaches quite effectively, and causes some coaches to adopt goal-setting as a practice. The actual process of goal-setting varies significantly between contexts, due to a range of constraints (e.g., goal-meeting approaches, lack of time, lack of follow through). Coaches also reported that reinforcement of milestones appears to have merit as well, although it appears that coaches conflated reinforcement of preset milestones with goal-setting. The two approaches of goal-setting and reinforcing milestones prompt an interesting debate that goes to the core of a long-standing debate between cognitive and behavioral approaches to behavior modification (Smith & Smoll, 2011). Advocates of goal-setting and cognitive interventions to shape behavior would argue that having the swimmer engage in the process of setting goals would increase the desire of the swimmer to comply with behaviors that were directed towards achieving the goal (Burton, 1989). The behavioral approach would suggest that reinforcement is what matters most; it does not matter if the swimmer is invested in the goal they have set (Smith & Smoll, 2011). The true answer, as with most issues in psychology, probably lies somewhere in between the two approaches. An important consideration is the age of the athlete. For instance, a perennial topic of debate is the degree of autonomy which athletes should be provided in the determination of goals and directions. Another critique of goal-setting is that the time between when the goal is set and identified by the athlete, and the time when it is reinforced can often be quite long. Coach 13 noted this issue, when he described the importance of having a swimmer’s goal times available for coaches so that they could provide immediate reinforcement for achieving a goal. 153 The argument about the effectiveness of a cognitive strategy or a behaviorist intervention might ignore that the goal-setting process might be a proxy for the effect of building a coachathlete relationship. Notable research by Gould and colleagues (2002) has shown that the quality of the coach athlete relationship is an important predictor of athletic success. The coach athlete relationship is developed through conversation and interaction, and a one-on-one goal setting meeting provides an opportunity for that conversation to take place. One coach discussed how an open-ended question on the goal-setting form instructing swimmers to “tell me one thing about yourself” was an excellent avenue towards having greater interaction, and helped to promote better understanding between coach and athlete. Another significant process or effect that could be missed in the debate between cognitive and behavioral approaches is the potential for a Hawthorne effect, a common augmentation of treatment and manipulation first noted by industrial psychologists (Landsberger, 1958). A Hawthorne effect occurs when the person being manipulated by an experimental treatment responds not to the treatment itself, but responds in a positive manner because they recognize and appreciate the experimental manipulation. In the case of goal-setting meetings or behavioral reinforcement, either strategy may be effective because the swimmer is responsive to the fact that the coach has taken an interest in him or her. Seven of the eight coaches who employed a rigorous goal-setting process or behavioral reinforcement strategy also demonstrated that they had a strong interest in the development of their swimmers, both as athletes but also as people. Given that coaches may often work with 40 or more different athletes, the type of treatment employed (one-on-one goal-setting meetings, or rigorous behavioral reinforcement) may not matter; instead, the effect may come from the fact that the coach makes the extra effort to connect and recognize the swimmer’s progress. Being a role model. Interview questions were purposefully structured to prevent 154 anticipated social-desirability bias, given that the topic has a positive and responsible connotation, and that coaches would be expected to present themselves as positive and responsible people. The first question asked coaches to explore if their swimmers had changed as a result of their experiences in swimming, and a follow-up question asked the coach what role they had played in that transformation. The analysis sought to determine if the Foundations course was capable of teaching a coach to adopt role-modeling behaviors. Six coaches talked about how they acted as role models, but responses varied amongst participants. Coach 11 immediately discussed how he led by example, doing dry-land exercises with the group of 14-18 year-old swimmers that he coached. Another coach stated that he had used swimming as a way to go places in life, and the athletes on the team could see him as an example of a person who had moved on to bigger things: “that’s the role I play and I’m happy to do it.” Coach 9 noted that his former high school coach had been a role model for him by focusing on his improvement and enabling his success. When pressed about taking up a similar role for the athletes he was now coaching, he stated that he wanted to be a role model, but not the “main factor in their development.” Coach 13 somewhat reluctantly stated that some of his former swimmers had told him that his strong encouragement had helped their development. Coach 10 was the only coach who clearly stated that role modeling was important because athletes were always watching how you were interacting with your adult peers and other swimmers. Three coaches discussed how they were very ambivalent about being a role model for their athletes. Coach 2 stated that if she had to go out of her way to be a role model, and was no longer “being herself,” then she did not want to do that. She saw role modeling as being the domain for certain people with strong interpersonal skills, and took an entity viewpoint on 155 interpersonal skills (i.e., you either have them or you don’t; they cannot be developed). Coach 15 suggested that the course would have limited impact on a person who was not capable of being a role model for kids. Although the evaluator did not probe extensively on this topic, and because there was extensive overlap with other sensitizing concepts in the interview (e.g., developing the athlete as a person, professional mindset), it may be that role modeling was not explored in enough depth during the interview to provide sufficient stories. However, it does appear that many coaches either did not have a clear role-modeling strategy that was designed towards producing certain effects in athletes, were reluctant to claim any effects of their role modeling behavior (Coaches 9 and 13), or felt that role modeling was either something you were good at and you did, or something you were not capable of and therefore did not do (Coaches 2 and 15). Another possibility, raised in the interim findings meeting, was that there might not be a clear means-ends connection between acting as a role model and positive performance outcomes. Developing the athlete as a person. This sensitizing concept was probed using the same interview questions as the being a role model questions (e.g., “how have your swimmers changed as a result of their swimming experiences?”). Some coaches were able to infer that the sensitizing concept of interest to the evaluator was the coach’s efforts to develop the athlete as a person, whereas other coaches discussed the changes in the swimmers that were highly related to swimming (not personal development). Coach excerpts were reduced to themes, and then inductively categorized, falling into two broad categories. The first category of responses discussed coaches’ mindsets and philosophies regarding developing the athlete as a person, and the second category showed the range of ways in which coaches were acting, reacting, or not acting to develop the athlete as a person. 156 Coach mindsets and philosophies towards developing the athlete as a person. These mindsets were arranged using a typology, ranging from coaches who had little interest or awareness of this role, to those coaches who were hesitant to overtly develop the athlete as a person, to coaches who believed they had an important and defined role in the process. The coaches who showed little interest either had little awareness of this role, or had an entity belief that you’re either the type of coach who is a role model, or you’re not. Two coaches expressed hesitancy about engaging in efforts to develop the athlete as a person. One coach discussed how her strong religious beliefs guided her interactions with athletes, but that she was hesitant to engage in “character lessons,” because it might be perceived by the team membership that she was attempting to preach her religious values. Another coach acknowledged the role that his current boss and former coach as an athlete had played: “she’s like a second mom.” He also acknowledged the significant positive role that coaches can play in a swimmer’s development as a person. Yet, when pressed on whether he wanted to play a role in a swimmer’s development as a person, he stated that he “didn’t want to be, like, the driving factor in that.” Given this coach’s young age (under 23), it was perhaps not surprising his hesitancy to play an active role in swimmer development, as perhaps he had not differentiated himself from the swimmers as a coach. Nonetheless, the recognition between the importance of being a role model, as well as the direct impact his coach had had on him, conflicted with this coach’s hesitancy to develop the swimmer as a person. On the other side of the spectrum, three coaches shared that they had made a conscious choice that it was important to develop the athlete as a person. Coach 12 discussed how the section of the Foundations 201 course devoted to the coaching philosophy had influenced her to think about the kind of coach that she wanted to be. She thought back to previous coaches she 157 had as an athlete, compared the coaches that made a difference to the coaches who didn’t seem very interested in making a difference. When she made this decision, it was easier for her to make decisions on what types of behavior she could engage in when swimmers came to speak with her; notably, she stated that she didn’t engage in gossip when swimmers came to tell her or ask her questions about other swimmers and families on the team. This was part of the struggle of differentiating herself as a coach, and no longer an athlete on the team, as she was a former athlete for this program, and had competed with many of the swimmers’ siblings. Coach 11 was somewhat different in his mindset; for him, developing the athlete as a person was highly entwined with teaching the swimmer about how to enjoy the process and the “journey of swimming,” which he defined as his ultimate marker of success as a coach. Efforts to develop the athlete as a person. The efforts to develop the athlete as a person were organized into three distinct categories: use of the coach-athlete relationship, use of sport experiences, and use of behavioral management or other non-deliberate methods. Six coaches reported using the coach athlete relationship as a conduit for developing the swimmer as a person. This was done through deliberate efforts made by the coach to learn more about the athlete, either through goal-setting meetings, through talking to athletes about “school and stuff like that,” or by talking to parents of athletes. Coach 6 was very deliberate in her approach to teaching character, as she provided weekly character quotes and lessons for her athletes, and then provided them with a list of these quotes at the end of the season. She provided an example of an athlete she had coached over the past six years, who had grown into a leader through her work with the coach, and was now helping coach the younger swimmers. Coach 8 described a very specific instance where a goal meeting prompted a major life change for a swimmer, who wrote about this experience for a school essay: 158 It had something to do with, you know… apparently no one had ever told this kid that they believed he could be a better swimmer. Um… but I had a goal meeting with this kid his freshman year, and… we set these goals. He improved a lot. And I don’t think he ever had a coach tell him that, um… that he could be a better swimmer, or that there’s a process to… improving. And when he saw that he improved, apparently… he became a better student. Evaluator: Hmm! Coach: Because he… uh… in the paper, uh, it explained how his grades went up almost a letter grade per subject. Evaluator: Uh huh? Coach: Just because he believed in himself more. And then he went on to say, “I believe in myself more because… I have an adult… who believes in me, that wasn’t my parent.” The use of sport experiences to develop the athlete showed a range of responses. To promote clarity of understanding these different approaches, the evaluator arranged them using a typology (spectrum), from active or explicit approaches, where coaches engineered elements of the practice environment that were designed to teach life skills or develop the athlete, to reactive or implicit approaches, where the coach would use real experiences that naturally occurred in sport as a means to teach the swimmer about certain life lessons (this is frequently referred to as using teachable moments). A common active approach included the manipulation of the practice environment to develop the psychosocial needs of swimmers. Typically, the fastest swimmers lead the lane, and commonly a pecking-order can develop where one or two swimmers always lead the lane, and thus swimmers at the end of the lane develop a self-fulfilling prophecy that 159 they are slower than the lead swimmers. To counter this phenomenon, Coach 4 deliberately planned practice sets that would enable swimmers from the back of the lane to get out front and lead. Coach 5 stated that the use of games during practices had enabled some quieter, more reserved swimmers to open-up and become more expressive. Coach 15 stated that he had put athletes in back-to-back events during competitions, which afforded them little time for rest and recovery between maximal exertions. He stated that athletes often perceived this task to be more difficult than it was in actuality; the process of doing this often built athlete confidence, which carried into other venues outside of the pool. The reactive approach to using sport experiences to develop the athlete as a person would best be understood through the use of teachable moments (the exact language used by several coaches), best characterized by the coach waiting for a naturally occurring event during the context of a swimming practice or competition that would create emotional upset for a swimmer, and then doing an emotional debrief of the situation. For instance, Coach 12 discussed how she shared her recent personal experiences as an athlete with younger swimmers when they had negative emotional reactions after a poor race; her recent experience as an athlete was still salient for her, and she was able to use that as a strength for relating how athletes could learn to manage their disappointment. She stated that she had seen marked improvements in the emotional stability of her athletes over the course of her first year of coaching, after she had made this a feature of her coaching behaviors. Coach 10 also provided a strong insight to this process, stating that she wanted her athletes to learn life skills as a result of experiencing difficult situations in the sport. She stated that she often reacted to the emotional outbursts of swimmers by asking them, “why do you think you feel that way right now?” This became a conduit for helping athletes to understand and manage their emotions in the sport context. Coach 10 also set 160 some important boundaries for acceptable and unacceptable behavior. She clearly stated to athletes that certain behaviors were not tolerated in her group: With the younger ones who, you know, don’t necessarily have a respect for their coach yet, um, I just gotta be consistent in my group, you know, “I don’t know what is allowed at home, but I know in my group you gotta respect me so I can respect you.” Um… and just… seeing negative behavior, and nipping it in the bud, you know, just immediately taking them aside and explaining why that’s not acceptable, or, trying to figure out why they do it. So just immediately… reinforcing good behavior, and… um… you know… taking away any negative comments, negative actions or behavior right away. One coach noted how he had tailored his approach to match the two different coaching environments that he worked in. This coach had two coaching jobs; one coaching job was with a religious school, and the second coaching job was with a secular team. The difference in his approach between the two clubs was firmly grounded in the philosophy of the organization that he worked for. In the religious school environment, he taught character lessons overtly, linking them back to a Christian worldview. In the secular team environment, his approach was more reserved, as he felt that teaching “character lessons” might be perceived as overtly proselytizing, thus he adopted a teachable moments approach, waiting for situations to arise, and having those discussions without an overt religious worldview framing it. Coach 11 stated that learning the process of swimming (i.e., steady, deliberate work focused on improvement and not necessarily outcomes like times) was what developed the athlete as a person. Finally, some coaches exhibited a reactive, behavioral management approach, or were unable to articulate how participation on their swim teams would develop the athlete as a person. One caveat to these findings would be that they are expressed by the first three coaches 161 interviewed by the evaluator, which employed a different set of questions; however, the responses of these coaches were probed to sufficient depth during the interview process that they do not provide a biased presentation of the issue. Coaches 1, 2, and 3 all stated that they had more reactive approaches to developing the athlete as a person. Coach 1 stated that she walked the line between maintaining a “positive environment” and disciplining athletes when she had to. Coach 3 stated that “our kids are good kids,” and that his efforts were more along the line of role-modeling, not an intentional focus on “really building the kid up.” In summary, coach efforts to develop the athlete as a person ranged significantly. Although some coaches suggested that the Foundations course had emphasized the importance of developing character, there was not widespread conceptual agreement on what character development was supposed to look like. Coaches provided a range of strategies that were adapted to their own unique contexts and their own predispositions towards teaching character. The wide range of viewpoints and strategies suggests that greater conceptual clarity of character development might be introduced through the course, and the course might more explicitly illustrate the linkage between character development and better performance outcomes for swimmers. Other coach behaviors. Interview questions were purposefully left open-ended so that coaches could elaborate on other important behaviors that they engaged in that were differencemakers for the athlete experience (i.e., “is there anything else you do as a coach that makes a difference to the swimmers?” followed by, “Would you attribute that to the course?”). Other lines of questioning generated discussion that showed the types of behaviors coaches engaged in that were crucial for them performing their duties as a coach, but were not specifically tied to improving the athlete experience, or developing the swimmer as an athlete or person. Coaches 162 discussed a range of behaviors, which the evaluator analyzed inductively, fitting themes into four categories: developing the sport climate; accommodating individual differences; communicating with athletes; and managing parents. Developing the sport climate was divided into two subcategories to present a more accurate picture of the strategies employed to meet this end. It should be noted that the phrase sport climate was employed by the evaluator to describe the phenomenon described by coaches. Within the sport psychology literature, sport climate or motivational climate is used extensively to refer to the psychosocial environment created by coaches as perceived by athletes, which contribute to increased motivation amongst participants. This research has been rooted in basic psychological needs/self-determination theory (Adie, Duda, & Ntoumanis, 2012), taskorientation/achievement-goal theory (Ntoumanis & Biddle, 1999), and a caring climate (Iwasaki & Fry, 2013). Within the sport climate category, the first subcategory showed that the coach was guided by a defined philosophy that governed interaction with athletes. Coach 12 and Coach 16 both stated that the course had caused them to reflect upon the nature and importance of the athletes experiencing swimming as fun, which then guided their actions towards swimmers. Coaches 4, 8, and 11 all espoused athlete-centered development beliefs prior to taking the course, and in the case of Coaches 4 and 11, they also worked for clubs that had strong philosophical beliefs guiding the work of the team’s coaching staff. The second subcategory detailed the range of task-oriented reinforcement (Ntoumanis & Biddle, 1999) that coaches used in order to keep swimmers focused on the process of improvement and development. Examples included the use of a stroke-skills chart to track swimmer mastery of particular swimmingrelated skills (Coach 11); rewarding athletes with small, symbolic rewards for achieving specific milestones (Coaches 4 and 13); the use of delay-relays, a game that a coach described as 163 completely focused on relay exchange technique (Coach 10). Coach 11 discussed the range of approaches that he used for task-oriented reinforcement, including a caught-box on the whiteboard during practices. A swimmer would have their name written in this box when the coach caught them doing something well. This coach also gave out a coach’s cap (i.e., a special cap with the swimmer’s name on it) as an award to swimmers for larger achievements that were grounded in high amounts of effort, focus, or personal improvement. This coach noted that the process first, results second approach of USA Swimming, as presented in the Foundations course, truly resonated with his own personal philosophy on swimming and life. This case provides an illustrative example of how the Foundations course frequently influenced coach behavior, not through introducing new knowledge, attitudes, or strategies, but through reinforcing and resonating with the existing beliefs of coaches, and in some cases making those already existing strategies a more deliberate part of a coach’s approach. Five coaches discussed how learning to understand and then manage individual differences between athletes was an important part of learning how to coach. To a small degree, this was addressed by the Foundations course, through content designed to address differences at particular ages, but the lesson of individual differences was primarily learned through the process of coaching. Four coaches discussed communicating with athletes being an important behavior that they engaged in. For example, Coach 10 discussed how she helped athletes who were still learning how to handle tough emotional episodes: Well, when you see swimmers, you know, getting frustrated, you know, after their race, pounding their fist in the water, and throwing their cap and goggles, you know, you gotta get on that immediately, you gotta say, “You’re wearing a [team name omitted] cap, you’re representing the team, you cant be doing that stuff.” And then go to the core of 164 the problem, like “Why are you feeling this way? You have to understand that, even the best of swimmers at times, they’re not always gonna have the best race.” Coach 11 and Coach 16 both stated that the course section detailing how to communicate with a swimmer at a race had shaped the way they communicated with their athletes. Coach 11 stated that the approach from the course reinforced his own process first, results second beliefs, which meant that when communicating with a swimmer after a race, he asked about the process of improvement (e.g., “what did you do well, and what could you do differently?”). Coach 16 stated the sandwich approach to providing critical feedback (positive comment, corrective instruction, positive encouragement) was new to him, but that he had employed it throughout the season with success. The final category of other behaviors was managing parents. Coach issues with parents were identified as common needs and challenges for coaches, and they ranged in their complexity. With regards to specific coaching behaviors, a range of options were mentioned as strategies for working with parents. Coach 5 noted that the course emphasized the importance of communication with parents, and that he and the staff at his team made regular efforts to have informal conversations with the parents of their swimmers. Coach 14 talked about how another coach made regular efforts to send parents articles from USA Swimming. Coach 7 talked about the importance of thinking-through the process of conducting a parent meeting, with respect to time and place. She stated that although the course emphasized the importance of some basic strategies for working with parents, and that she had implemented the strategy of a meet-andgreet parent-coach gathering, she felt that the degree of complication with some parent issues was such that a simple strategy could not help to fix the problem. She stated that knowing more about how other similar coaches had handled difficult parent issues would have been useful. 165 Another coach stated that parents differed in their approach to communicating their children’s known behavioral issues to the coach, framing it as a dichotomy: That’s also a parent issue, some of the parents…don’t wanna talk about their kids’ issues, and it’s real hard to deal with things, if you don’t know what the problem you’re dealing with is. Evaluator: Hmm… so parents aren’t necessarily very forthcoming, and… Coach: It… it’s funny, you get two different types of parents. You get one parent whose kid has one very mild case of Asperger’s, who will tell you everything you need to know, and a hundred things you don’t. Evaluator: Right. Coach: And then you’ll get another parent who will tell you, “well, he’s just a little odd.” Well, you know… we’ve been dealing with kids for a long time, and there’s more [of an issue] than just, “he’s a little odd.” Given that managing and working with parents was noted as a consistent challenge for coaches, and given that these coaches suggested the potential limitations of parent strategies detailed in the course, there may be a need for greater instruction and discussion of the process of working through difficult scenarios that arise when working with parents. Program Delivery Through interviews with program participants, the evaluator sought to assess the degree to which the program was delivered as intended by its designers/administrators. The evaluator sought to gain an understanding of the program’s functionality, e.g., major glitches in its operation, timing and pacing issues, operating system coherence, and the degree of variability amongst participants in the program received. The evaluator asked coaches to describe how they 166 took the course, noting any glitches or persistent problems with the delivery of the program as intended. In addition, users were asked to describe the ways in which they interacted with the course while taking it. Operating systems and user complications. Coaches used a variety of operating systems to take the Foundations course. Although individual experiences with different platforms (e.g., laptop computer, tablet computer) varied considerably, no coaches reported experiencing a technical problem that prevented them from engaging with and completing the course. Given that a significant percentage of new coaches were older users and not digital natives, a concern raised by both the evaluator and the key program decision-makers was the usability of the course for coaches older than 45 years. The data from this evaluation suggest that for users over the age of 45 years, the coach was the primary variable in the user experience, not the Foundations course. Two coaches over 45 years expressed that they had significant anxiety that their work and progress had not been saved as they completed the course. Two other coaches over 45 expressed having no problems with the course. One of these coaches stated that she completed the course on an iPad while she waited to pick up her child at swim practice. Apart from some small usability issues that coaches suspected could be related to poor Wifi connections, or simple glitches that could be fixed by logging-out and then logging back in, these data suggest that the Foundations course operated consistently across platforms and for all users. According to the key program decision-makers, some glitches were common in the early rollout of the course, but since that time there have been fewer reported issues. Timing and pacing of courses. Users complete the Foundations 101 and 201 courses according to a flexible time-table. The 101 course must be completed before a coach becomes a coach member of USA Swimming (i.e., before they can begin coaching duties with a USA 167 Swimming affiliated team), but the 201 course can be completed at any point before the coach begins their second year as a coach member of USA Swimming. Due to these requirements the experience of taking the two courses are quite different. The effects of the Foundations 201 course varied considerably, whereas the Foundations 101 course had a consistent effect on coaches in this sample. The variability of the effects from the 201 course could be attributed to a range of sources, including its greater length (four hours compared to the 90 minutes of the Foundations 101 course), more difficult content, and its curricular design (i.e., organized by knowledge topics rather than coaching scenarios), or it could simply be the mindset of the coach as they take the course. Many coaches indicated that they took the Foundations 201 course in short succession after completion of the Foundations 101 course, long before the deadline for completing the course. The implicit sequence for coach progress was to take the Foundations 201 course at the close of the coach’s first season; however, the coach is not required to follow this timeline. Based on the reports from coaches, taking the two courses back-to-back could be problematic for a few reasons. First, students generally exhibit greater sensitivity to knowledge at the outset of a course. In addition, the Foundations 201 course introduces advanced knowledge with less contextual background. To make this knowledge more useful, a coach might place this knowledge in context by using specific examples they had experienced in their previous season. Without this experience, the knowledge from Foundations 201, being considerably more difficult than the information in Foundations 101, becomes disembodied, making it more likely that coaches will simply try to comprehend it for the sake of passing the test, as was the case with four coaches in this sample, rather than to actively strategize how to put it into practice. This was evident in that these four coaches had not shown much evidence of what knowledge they 168 had implemented from the course. The act of strategizing to put knowledge into practice is a critical piece of this program, but because it is automated and delivered from a distance, it relies on the content and the pedagogy to induce this strategizing on the part of the coach. What drives this strategizing is most likely the needs and challenges that the coach has experienced in their work as a coach, but if the coach has not worked sufficiently long enough to experience needs and challenges from the job, the coach may not see the need for the knowledge. Restricting coaches from taking the Foundations 201 course until the conclusion of their first year of coaching might not provide a simple answer, and it points to a common conflict in the world of online education and training. The advantage of online courses, as identified in the literature about online learning, as well as coaches in this sample, is that they provide a convenience above and beyond a comparable face-to-face learning option. The convenience factor outweighs the fact that they are not the preferred way for the coach to learn. Convenience has been a driving factor in the adoption of online learning across many disciplines. However, there comes a point where bowing to the user’s needs for convenience may compromise the integrity of the educational experience. In the case of the Foundations of Coaching course, the desire to promote flexibility and convenience for the user, although well founded due to the complications and complaints about the previous version of the course, may currently compromise the value and applicability of knowledge for some coaches who take the Foundations 201 course. This justifies a deeper exploration of coach experiences with the Foundations 201 course, in order to explore how this knowledge makes it way across the knowledge-practice divide, and what variables from the coach’s own context might moderate this relationship. More extensive data on this topic might be useful in guiding a redesign of the Foundations 201 course delivery, if this direction was deemed viable by key program decision- 169 makers. Potential directions for this approach are discussed at greater length in chapter five of this document. Interaction with the course. In a face-to-face course, variability between instructors and the students in the classroom can cause the program, i.e., the curriculum, to vary significantly from one classroom to the next. With an online course, every user gets the same program. Therefore, variability in this program’s effects lie almost completely on the program recipient side. Three coaches reported that having taken the course at the same time as other coaches in their program, they were able to discuss many of the topics that the course brought up, which helped deepen their comprehension and understand how the knowledge applied to their context. Three coaches (all high scoring on the 201 test) noted that they could read the material that was presented in the course faster than the voice of the person reading the material; one coach noted that she simply muted the volume and read the words, because she described herself as a visual learner. Coach 3 described his learning process in the course as “cherrypicking” knowledge: With as crazy and hectic that the schedule that I run, a lot of times, if I say, “okay, I’ve… I’ve got this part solved,” and I recognize that this part is not… not a weakness for me, um… okay, “I got it, I got it, I got it, I got it, I got it…” click, click, click, click… okay, this is where I need to get some… knowledge. AD: Okay, so it’s kinda like cherry-picking… knowledge… that’s like really important for you right now, at the moment? 3: Yeah. Yeah, so… um… I may not be in a situ--… I’m, I’m kind of in a situation right now where, honestly, a lot of time, time and, uh… time is something that’s probably… the thing that I lack the most of. So if, if I can figure out how to get through 170 something a little quicker, I will. Four coaches noted that the PDF documents in the course were of immense value, although not all of these coaches kept them organized in a place where they could easily access them. Coach 8 noted that the PDF documents served an important role as a refresher: That whole library of resources? I mean, like you said, it’s all in your face for five hours, so I… so some of this stuff I don’t remember what I learned, and some of the stuff, I just go back and I look at it, and I’m like, “wow, I don’t remember learning this, but this is really helpful.” In addition, where the course fit into the coach’s workflow was an important consideration to make as well. Two coaches noted that they were forced to take the course from home, due to the lack of Internet access or the use of a secure or local network at the coach’s day job. One coach noted that the online course integrated very easily into her workflow in her primary job. This suggests that the convenience factor was generally positive, but it also suggests that attention to the course could be split with other tasks. Interpretation and judgment of program delivery. Ultimately, program providers need to decide what aspects of the online course they wish to control as tightly as possible, and what controls they are willing to relinquish to the user. Users may feel controlled and manipulated by provisions that keep their volume from being muted, or prevent them from dividing attention between multiple tasks while taking the course, and these feelings may impact how they perceive the content of the course and its applicability to their jobs. Coaches in this sample understood why the course had quizzes and interactive activities that required them to engage with the course. With regards to the course itself, it appears that the provisions used to keep the user engaged and progressing through the course were adequate, and did not aggravate 171 the user beyond the occasional annoyance. Future evaluation work could examine the ways in which users interact and engage with the Foundations course, through the use of a think-aloud analysis, where participants take the course, using all of its features, while continuously thinking out loud (Nielsen, 2012). A thinkaloud analysis can be accomplished with a small number of participants, and could provide useful information about how coaches interact with talking head videos, scenarios, and other interactive features in the course. This could show how knowledge makes the leap to practice; for instance, if coaches watch videos of coaches coaching, then make notes about the behaviors they observed, and then state, “I could use that,” this provides a more accurate representation of the learning process. With regards to the timing of when coaches take the Foundations 201 course, future evaluation work could examine different approaches of delivering this content in order to increase its effects on coaches. Given the difficulties of managing the PDF documents from the course, and the difficulties that coaches experienced attempting to access resources (e.g., videos) after they had completed the course, there is merit in exploring ways to make the course more mobile and accessible after completion. For instance, finding a way to support coaches after they have completed the Foundations course could help facilitate the process of a coach implementing changes to their program that resulted from knowledge they learned in the Foundations course (this process was referred to as continuing the conversation in the interim findings meeting with key program decision-makers, and the term appears throughout this document). Continuing the conversation also enables USA Swimming to spread out the amount of time that it maintains contact with coaches (currently, the Foundations 101 and 201 courses are the only points of contact). A means of continuing the conversation might include any 172 number of digital technologies. One approach would be an e-mail follow-up. A second approach would be moving content to a mobile-device application (e.g., iPhone/iPad, Android). The advantage of an app would first be its convenience; this would allow coaches much easier access to course content after taking the course. A second advantage of an app would be the ability to customize education based on the needs of the user, and then push the appropriate courses to the coach to take for continuing education. Although this type of customizing could be done using the existing computer-based learning environment, an app would get used more, and because it is used more, it would be easier to push continuing education courses through this interface, as compared to pushing it via e-mail. The advantage of an app is that it provides a one stop shop for the coach; it is a resource for material they need every day, but it is also where they turn in order to meet requirements for continuing education and development. Another possibility for continuing the conversation is the use of a social media campaign that allows coaches to share their work under a common hashtag, or in a user group or forum. Such an approach has the potential to help create a community of practice that is centered around ideal coaching techniques, and representatives from USA Swimming would have the ability to highlight the best examples. This type of voluntary sharing of best practice ideas may have particular usefulness after compulsory coach education requirements have been met. Areas for Improvement The evaluation sought to identify areas where the Foundations course could be improved. Interviews and focus groups with the key program decision-makers at USA Swimming helped to identify administrative issues and past problems with coach perceptions of the course. Interviews with coaches provided feedback about the course’s functionality, its content, and where it fit into the coach’s professional development plan. To discern coach perceptions for 173 areas that needed improvement, the evaluator asked the question directly of the coach, e.g., “did you have any thoughts on how the Foundations course might be improved?” Results from coaches were coded using the deductive code Concerns and Improvements, and excerpts tagged with this code were categorized inductively. The results are presented along with the interpretations and judgments of the evaluator. A related code, USA Swimming Services, was frequently used along with this code, wherever suggestions for changes or additions to USA Swimming outreach services were suggested. These results are presented following the suggestions for improving the Foundations course. Concerns or improvements to the Foundations of Coaching course. Nineteen coaches discussed these concerns with the evaluator. Four coaches stated outright that they had no concerns or suggestions for improvement. Fifteen coaches provided feedback about some aspect of the Foundations course that they believed could be improved or changed. Excerpts from these interviews were converted to themes, and the evaluator inductively organized themes into the following categories that reflected the emergent nature of patterns in the data: course content (inclusion, organization, depth, clarity); access to course materials; continuing education after the course; functionality. Content (inclusion, organization, depth, clarity). The types of content that coaches wanted to learn more about did not fall predictably into one topical area, which suggests that the Foundations course does not have a specific weakness or lack of content. Ultimately, the data showed that a coach’s needs for continuing education is likely to be highly individualized and needs-driven, and suggested revisions to how the course is organized might be taken with a grain of salt. For instance, four coaches suggested that they struggled with elements of the course that did not apply to the age-group that they currently coached (or planned to coach). The suggestion 174 of organizing the Foundations course around the needs of specific age-groups might improve the learning experience for these coaches, but it might complicate the learning experience for other coaches who expressed no concerns with the course in its current form. Two coaches suggested that there were almost two different tracks of knowledge – the knowledge that was essential for assistant coaches, and the knowledge that head coaches needed in order to take-over and run a program. Coach 7 suggested that for an assistant coach, the Foundations 101 and 201 courses together were almost too much at the same time, but for a head coach, there were certain topics that were not addressed in sufficient depth. In addition, she noted that the course didn’t really address an important job of the assistant coach, which is to assist the head coach in their role. Although two coaches noted that the information in the course describing USA Swimming’s organizational structure (e.g., House of Delegates) had little value to them, Coach 7 noted that this information was important information for a head coach. Three different coaches suggested a need for specific content designed to address behavioral problems of swimmers. A brief assessment of coach needs and challenges previously presented in this section suggested that behavioral management was a major day-to-day challenge faced by coaches. Coach 14 noted that coaches are dealing with a much greater range of special behavioral needs amongst children, and yet the amount of knowledge and training that coaches have with regards to these needs is not addressed in the course. It remains to be seen if an online course could adequately provide training to coaches to help them understand and work with children that have special needs. Three coaches expressed concerns about parent issues that were not addressed in sufficient depth by the course; two of these coaches suggested that some form of reciprocal parent education course might be an essential next step. Four coaches expressed a concern that they wanted more information about advanced 175 swimming training science, specifically the different types of training available and where to place them in training cycles, as well as the process of advanced level swimming stroke analysis. With advanced training information, the coaches wanted to know more about how this training might be interspersed and arranged throughout the seasonal training cycle, but also where it fit in with a swimmer’s progression through the program. Although most coaches noted that the drills and stroke technique videos in the Foundations course were quite useful, Coach 11 told an interesting story that illustrated the limitations of the current iteration of the stroke technique instruction in the course. To begin, he described how the head age-group coach, his supervisor, had the ability to diagnose how a certain stroke technique was creating pain for a swimmer …so the swimmer comes up, she says, “my shoulder’s hurting.” Coach will say, “Point to where it’s hurting at,” right? And she’s like, “let me see your stroke,” and she’ll be able to identify what’s going wrong, you know, maybe in her… front anchor, or her catch, you know what I’m saying, and actually stop her, and pull her out, and say, “hey, listen, you are… not rotating even on to your side, and you’re in-sweeping under your body, and this is why you have this… you know, this is why you’re pulling your shoulder on your butterfly, you know, you’re not bringing your hands forward all the way, and…” You know what I’m saying? So, you’re able to give the swimmer this… good feedback, and… that kind of stuff takes a long time to learn, to really, truly identify. He noted that although the Foundations course provided instruction on the strokes, and showed videos of strokes from multiple camera angles, providing a “one minute commentary about what they could improve on their stroke… that’s not enough.” Instead, he suggested that video analysis could go into much more depth, to help coaches spot stroke inefficiencies: If you were to have, like, a true, like, online… there would be multiple videos for one 176 stroke, and there would be different proficiencies in each stroke. You know? You would like, okay, “this guy’s totally jacked-up,” you know? “This guy… he’s, you know, in fly, he’s got the two kicks down, but his timing’s off. Can you spot the timing?” And kinda make, like, a quiz out of it. This discussion is a good illustration of the benefit of a semi-structured interview approach, in that it allowed the evaluator to step into a conversational role rather than the role of the data collector. The conversation that followed generated an interesting possibility for future development of coach education efforts. The initial discussion with the coach seemed to suggest that this type of expertise could only be gained by years of experience. But this conversation led the evaluator to recall a study that once read about the visual search strategies employed by novice and expert swimming coaches when analyzing video of swimmers (Moreno et al., 2006). In addition, significant research has been done with the use of computer simulations to aid and augment the human expertise and decision-making process (Buchanan, Davis, & Feigenbaum, 2006). This led to the suggestion that an advanced stroke analysis program could be developed and run in an online course format, which would then enable younger coaches to develop their expertise in stroke analysis. The degree to which this program might help to improve coach expertise could be reflected immediately in the rates of overuse injury amongst younger swimmers. Access to course materials. This category represented the fact that education is a continual process, where exposure to an idea may occur while taking the Foundations course, but the recall and application of that knowledge might occur much later in the season. Recall and application often serve to improve comprehension of the knowledge initially learned in the course. However, the course is a fixed experience that often takes place in a rush before a 177 deadline, and the course content is not easily accessed after the educational experience. Five coaches noted that they would have liked to access some aspect of course content (e.g., videos, PDF documents) more easily after they had completed it. Only two coaches saved and organized the PDF documents while they were taking the course, but four coaches noted that they had thought little about those PDF documents while they took the course, but then found themselves in situations later in the season where the knowledge they contained would have been applicable. Two coaches noted that the Foundations course occasionally employed open-ended response questions, where the coach was presented with a scenario, had to devise a solution, and then had to contribute that solution in the open-ended response. These two coaches wondered what happened to these responses. The open-ended response may give the illusion that the questions are being evaluated, but both coaches quickly got the impression that nobody was reading these questions. However, the fact that the questions were not being evaluated did not temper their curiosity; both of these coaches expressed their curiosity in the responses of other coaches. With more than 3,000 coaches taking the Foundations course each year, finding a way to monitor, organize, and present the responses to these open-ended questions in such a way that they would help coaches to see a coherent picture of the thinking processes of other coaches is currently a difficult proposition. Continuing education after the course. Four coaches mentioned concerns they had with the official continuing education process following the Foundations course. One coach stated his desire to jump right into the next level of coach education, but was rather confused about why the coach education process shifted over to the American Swimming Coaches Association (ASCA) coach progressions program. Another coach remarked that the exact progression of coach education requirements was not very clear. Two coaches suggested that some form of 178 shadowing might become a requirement for higher levels of education, after a coach had completed the Foundations program. Functionality. Three coaches expressed that they had experienced some fear that their progress in the course would not be saved. Two of these coaches were taking the course on a short deadline; therefore, had progress been lost, the deadline would not have been met. In addition, coaches who expressed this concern were all older than 45 years. There were periodic mentions from a range of coaches that the course occasionally experienced a small glitch. One younger coach explained that when she experienced a minor issue, she simply logged-out and then logged back into the program, which resolved the issue. This may simply be the true effect of being a digital native, in that users under the age of 45 immediately apply a generic fix whenever encountering a problem. In addition, the key program decision makers noted that in the program’s launch phase in January 2013, such problems were common, but have been reduced significantly in the past year. Evaluator interpretations. Through the evaluator’s work interviewing both the key program decision-makers as well as coaches who took the Foundations course, two issues arose. The first idea was to examine the interface between ePath learning (the company which hosts the Foundations course) and USA Swimming. Currently, the interface is minimal. When the coach completes the Foundations course, they print out a certificate that states that they have completed the course, which is part of the application for coach membership that must be submitted to USA Swimming. The limitation of this approach is that USA Swimming is not privy to any educational data that might be useful for explaining the learning experiences of coaches in the program. For instance, even rudimentary access to individual item test-score data could show what knowledge topics coaches understand well, and what questions confuse coaches. Given 179 that over 5,000 coaches have taken this course, the sample size is quite large, and with respect to new coach members of USA Swimming, the data would represent real-time population sampling, not simply a representative sample drawn for analysis. This level of data analysis could provide normative statistics, or could be used to examine trends and relationships in the data, such as the effects of taking the 101 and 201 courses back to back. A second consideration would be to prime coaches at the outset of the course by stating what an ideal coach should do, in very specific terms, with relation to the goals and objectives identified in this evaluation. For instance, what are the mindsets of an ideal coach? What are the types of behaviors that ideal coaches demonstrate? Using narrative examples provided by real coaches who completed the course would be a way to help illustrate the value of the course, but also provide an example of how the knowledge leapt off of the computer and onto the pool deck to make real change. Given that many coaches stated that they appreciated watching videos of real coaches, sharing the experience of other novice coaches might be particularly valuable in helping younger coaches to make use particular segments of the course. Based on the feedback provided from coaches, the course presents a wide range of useful topics, but due to its design constraints, it is unable to easily support the implementation of this knowledge, due to the fact that its learning materials are not easily organized or accessed from the pool deck. It would be prudent to explore design options that would allow these learning materials to be mobile. A simple place to start would be the organization of the supplemental PDF documents into a single Coach’s Workbook; the electronic file could be downloaded for mark-up or printing. For coaches who preferred a printed-and-bound option, USA Swimming might make low-cost print copies of such a workbook available by mail, or explore the possibility of mail-order through a catalog company or Amazon.com. A coach workbook could 180 be more than supplemental information, it could also include exercises designed to promote reflective thought and planning by the coach. More importantly, it represents a single repository of the coach’s collected swimming knowledge. When analyzed within the frame of a logic model, the course has a very small number of outputs (i.e., immediate products of learning). Ideally, the outputs of a program should help to promote the development of both short-term and long-term outcomes by supporting and guiding the participant. Another consideration, given the popularity of the videos presented in the course, would be a streamlined app that could house these videos and deliver them to coaches more easily. An app could represent a one-stop-shop for practice ideas, videos, coaching tips, and new information. An app would enable USA Swimming to provide service after the sale, helping to support the knowledge provided in the course as coaches implement it into their daily practice. Finally, such an app might be the preferred mechanism for delivery of continuing education. Given the popularity of the current USA Swimming app (Deck Pass), exploration of the app as the central delivery point for USA Swimming services, including coach education, could be fruitful. USA Swimming service enhancements or additions. Whenever the topic of improvements and enhancements to the Foundations course were discussed, the evaluator asked coaches to suggest other avenues where USA Swimming could be of particular use for their development as a coach. The question was posed somewhat broadly, so not to restrict the potential options for enhancements to programs just like the Foundations course. The coaches were invited to identify any number of ways in which USA Swimming might provide assistance. Sixteen coaches described some aspect of service that USA Swimming could refine, enhance, or begin to offer anew. Following from the patterns that emerged in the data, the evaluator 181 organized these themes into three categories: education; networking and in-person clinics/conferences; curating information. Education. Eight coaches discussed the need for more educational opportunities, although the specific nature of these educational opportunities did not cluster around any one topic or set of topics. Two coaches mentioned the need for some form of an online course that swimming parents could take to learn more about the sport and best practices related to the sport. Two coaches suggested that courses could provide more depth on a few issues related to stroke technique: Coach 11 (discussed previously) suggested that videos demonstrating disqualifications vs. legal strokes and turns would be helpful. Coach 17 noted that she had limited experience as a competitive swimmer, and had only begun swimming recently as a masters swimmer, therefore she had poor knowledge of how to teach stroke techniques to a beginning swimmer. She suggested that she needed much more elemental help with a step-bystep tutorial on how to instruct some of the more difficult strokes. Coach 7 and Coach 14 both suggested the need for more comprehensive education about long-term coach development. Coach 7 noted that promotion for coaches at USA Swimming clubs often means moving to the intercollegiate level, and that if USA Swimming has an interest in developing great coaches for the long-term, first it would have to address the issue of coach retention. Coach 16 and Coach 20 both discussed the need for taking more advanced courses. Coach 20 said he was “very interested” in taking his level three course. Coach 16 noted that “swimmer management” was a concern, and he wished to take a course that could explain practical strategies or provide a framework for long-term swimmer development and progression. He stated that “it’s now fair game for parents and athletes to question coach decision-making,” and so coaches would need to step up their game with relation to answering questions about swimmer management. The 182 perceived need for more education, but the lack of specific content areas where that education needs to occur, suggests that there is a need for more data on what coaches want to learn about. Simple two-item surveys placed throughout a course (asking questions such as “how well do you think you learned this topic?” or “do you want to learn more about this topic?”) could easily collect data and show where the interest and need lies. Finally, Coach 15 noted that he had to the take the Foundations course as part of his requirements to be a high school coach, and he took the course along with two generic courses offered by the National Federation of State High School Athletic Associations (NFHS). He noted that the courses offered through NFHS-Learn (the online platform utilized by NFHS) were directed towards a generic coaching audience, for which the biggest two sports were basketball and soccer. Thus, the examples and scenarios provided in these courses did not generalize well to the swimming context. Exploring high school coaches as a potential audience for the Foundations course may help to increase the number of coaches served, thus increasing the effects and impact of this program on the sport of swimming. Networking and in-person clinics and conferences. Six coaches expressed the desire to have more opportunities for in-person, face-to-face learning opportunities, although the actual learning goals that these coaches had for these face-to-face sessions varied widely. Two coaches expressed the importance and value of watching Olympic level swimmers up close and inperson. Coach 3 stated that after watching Ryan Lochte swim at a clinic, “you see things you wouldn’t see on a video.” Coach 22 stated that there was immense value in watching experienced coaches, something that would be possible at a clinic, and while this suggestion did not necessarily suggest the value of a clinic, Coach 14 also advocated for the immense value in having a new coach shadow a master coach. Three coaches suggested that a local clinic would 183 have immense value to them; however, they noted that the traditional, expert-driven clinic wasn’t the experience they were looking for. Instead, these coaches were looking for an opportunity to improve their network of coaches. Coach 6 stated that it would be interesting to have a coach share a problem before an open room, and then have coaches in the room discuss how they might work to solve the problem. Coach 7 noted that the coaches in her LSC were particularly tightlipped about sharing information with each other, and some sort of clinic or forum might help to break the ice and get coaches to share information a little more. Coaches 6 and 7 both had more than five years of experience, and their statements resonate with what is known about the needs of adult learners: they want opportunities where they can drive the conversation, and they want to be able to generate problems from their own practice that can be resolved through discussion with like-minded professionals. Coach 9, a novice coach also expressed the value of a local clinic, where he could share information and discuss issues with coaches who likely shared some of the same problems that he had. Finally, Coach 18 stated that she would appreciate attending “topical seminars,” i.e., short, one-day coaching conferences that were organized around a common topic, e.g., psychological development of adolescent swimmers, working with troublesome parents. She suggested that in-person seminars had value for USA Swimming interms of promoting their visibility as a partner in coach’s professional development, and proving that “[USA Swimming is] not just sending coaches online to meet all their needs.” Curating information. Four coaches noted that USA Swimming could play a critical role in terms of curating and vetting information. Three of these coaches noted that they had used existing USA Swimming resources, notably the USA Swimming website, as a reference point for quality information, but they also noted that the website was not very easy to navigate and find what you want, nor did it function particularly well with a mobile device. Another 184 coach noted that due to a secure computer network at work, access to the USA Swimming website was not practical during a lunch break. Some coaches indicated that they read print media and e-mail newsletters that disseminated swimming knowledge. Thus, the USA Swimming website might not be the only hub or source of information. Given that many coaches in this sample indicated that their quest for new knowledge led them in a variety of directions, it would be prudent to take a more thorough accounting of the information search strategies employed by coaches. A large number of websites and information services have proliferated in the age of social media, and such an analysis could show how information (and misinformation) spreads through the population of coaches. Revised Logic Model After completing the analysis and write-up of the results and discussion chapters, the evaluator revised the initial version of the logic model developed at the outset of the evaluation. The current iteration reflects the findings from evaluation, especially the degree to which particular elements of the original logic model were demonstrated in the evidence gathered from the coaches in the sample. The current version appears in Error! Reference source not found.. This logic model maintains the sequential nature of the original logic model. However, it adds elements of coach reflection, communities of practice, and a quest for new knowledge, which were explored in greater depth during participant interviews. The logic model should be read from left to right. The first box (inputs) represents the unique coach assets that a coach brings into the program. These assets, broken into knowledge, skills, and attitudes, vary significantly from one coach to the next. Moving to the right, the Foundations of Coaching program is represented by the cylinder. It shows the different units of knowledge that are presented in the program, along with the pedagogy used to deliver this content. These elements 185 are not broken out into specific skills and competencies, as the program does not delineate them in that fashion. As a result of participating in the Foundations program, two classes of activities occur: reflection and communities of practice. The degree to which each of these activities occur appears to be dependent on the coach, but there was significant evidence to show that these activities occurred for many of the coaches in the sample. Coming out of the program (moving to the right), program outputs are presented, along with the quest for new knowledge. The dashed line around the Foundations program is meant to show that this is the only part of the process over which USA Swimming possesses tight control. This matters for the sake of future planning, in that attempting to affect changes outside of this boundary might become increasingly difficult as the effort moves further away from the proximity of USA Swimming. Note that the quest for new knowledge is partially embedded in the section of the program under control of USA Swimming. This placement is deliberate, and is meant to show that USA Swimming might be able to exert more influence over this quest. For instance, many coaches reported using a variety of sources in their quest for new knowledge; if USA Swimming were able to drive coaches back to the USA Swimming website, or to another social media source curated by USA Swimming, this would help to continue the conversation with coaches after they had completed their coach education requirements. Moving to the right again, the next box shows demonstrated short-term outcomes. Short-term outcomes were classified as changes in knowledge, attitudes, and behaviors/skills arising from taking the course. The box below these demonstrated short-term outcomes (Implemented into coaching practice) could represent shortto-medium term outcomes, as implementation may be initiated immediately after taking the course or later in the season; even immediate implementations may evolve over a longer period of time. Moving to the far right column, the long-term outcomes and impacts are hypothesized 186 outcomes that were not examined in-depth in this evaluation; however, these outcomes are likely of interest to USA Swimming. Future evaluation efforts may seek to demonstrate these longterm outcomes and impacts, so they are listed here with the caveat that they could (and probably should) evolve based on consultation with key program decision-makers. 187 Figure 5 - Revised Program Logic Model 188 Chapter 5 – General Discussion This chapter will discuss the findings from this program evaluation, but the reader is advised that it will differ from a conventional discussion section that typically follows a kinesiology study or experiment. The results of this evaluation will be discussed with regard to the five evaluation purposes, followed by judgments and recommendations, limitations of the evaluation, and future evaluation directions. Because there are findings from this evaluation that can be extracted and generalized to research contexts, these findings, along with future research directions, will be presented separately, after a discussion of the evaluation results. Evaluation Purposes This evaluation was undertaken with five purposes, determined through a process between the evaluator and the key program decision-makers at USA Swimming, and agreed upon by both parties. These purposes included: (a) examine coach perceptions of the program’s utility; (b) examine the effects of the program on coaches, with attention to changes in knowledge, attitudes, and behaviors; (c) examine the consistency of the program’s delivery; (d) examine potential areas for improvement. A fifth purpose was to examine the organization’s utilization of the evaluation process and the evaluation findings. Program utility and perceptions. The first purpose was to examine the program utility and perceptions of coaches that took the Foundations of Coaching course. The evaluation showed that the large majority of coaches interviewed had a generally positive appraisal of the course. These general appraisals represented the coach’s immediate recall of the course, and memories were tied to a general affect-based memory (e.g., “I liked it,” or, “compared to other courses, it wasn’t a waste of time”), the layout and design of the course, or the course resonating with the coach’s own beliefs and philosophies about swim coaching. What is important about 189 these general appraisals of the course is that they represent the coach’s most easily accessed memory of the course. These appraisals are not tied to any specific learning objectives or content presented by the course. This represents an important hurdle for the course to clear, as affective components of design help to ensure that cognitive objectives from the course can be achieved (Norman, 2002). While five coaches expressed their dislike for online learning, all five preferred it to the alternative of sacrificing a weekend for a face-to-face version of the Foundations of Coaching course. With regard to salient features from the course, coaches were most responsive to videos of experts, especially to videos that showed coaches coaching athletes, as they modeled specific behaviors that these coaches could adopt in their own work. Reactions to the quizzes and interactive features interspersed throughout the course were mixed, with some coaches reporting them to be valuable, but other coaches finding them to be an annoyance. However, most of the coaches that found the interactive features annoying understood that they existed for the purpose of keeping them engaged. Coaches also noted that the course could not make up for a lack of competitive swimming experience as an athlete, and that a coach’s colleagues could be a major difference-maker if they were supportive of the course. For the key program decision-makers, the evaluation had the purpose of helping them to better understand how their program affects change in coaches. It showed that coaches had heard and understood the message presented in the course, particularly the message regarding athlete development. It also allayed fears that widespread technical glitches could present an insurmountable goal that would sour the membership on any form of coach education program. Program effects. The program appeared to have the strongest impact on coach knowledge, but it also showed the ability to influence coach attitudes and behaviors, especially with relation to the theme of long-term athlete development. The primary course developer 190 stated that a major focus of the course redesign was to get across the message about long-term athlete development, age-appropriate training, and the importance of fun. To accomplish this, the Foundations course employs learning scenarios that present a problem to coaches, and then introduces facts that can be used to help the coach make the preferred decision. The first major scenario that coaches encounter in the Foundations 101 course asks, “Is five-year-old Ryan old enough to start competing?” This opens a conduit for knowledge about athlete growth and development. Coaches reported that the most useful topics were coaching and teaching principles (e.g., drills, technique, practice management, communication) followed by developmental principles (e.g., age-appropriate workouts, what motivates athletes). These topics are practical knowledge and skills that a novice coach can put into practice as soon as they begin to coach that will lead to better sport outcomes for their swimmers. However, coaches did not report finding much use in the topics of mental skills for racing or physiological principles of practice design, two topics that are critical for successful coaching of athletes. It is important to analyze why these topics were not perceived as useful. The default explanation may be that coaches may have understood these topics from their sport experience, and did not identify the Foundations course as the source of knowledge for these topics. This is possible because of the interview technique, which did not ask the coach to expound on each topic of the course; rather, it simply explored what coaches recalled from the course. One possibility is that these topics could be too difficult for a new coach to understand. However, in analyzing the course content, these topics are presented in a way that an educated person can understand, with an ideal mix of scientific principles, but also a reasonable amount of applicability to the swimming context. For instance, the four physiological training zones are explained, and examples of training sets for each training zone 191 are provided. Thus, it may not be an issue of comprehension that is blocking usefulness. A distinct possibility might be that most of the coaches in this sample were coaching novice or agegroup swimmers, and these topics are most useful for advanced age-group swimmers (11-14 years-old) and senior-level swimmers (ages 14&up). Thus, the coach may not have identified the need for these topics, and not having much value for this information, it may simply be an issue that the coach is simply trying to attain enough competence to pass the 201 test. Another consideration is the way these topics are taught in the course. Both of these topics are presented primarily through a voice-over slide show supplemented by content that the user must click before advancing. It is akin to a lecture in a face-to-face class without activities that support application of the content to the coach’s context. Although videos of experts and coaches bookend this lecture content, these videos only serve to deliver a motivational message, rather than demonstrating how these skills are put into practice. This approach was taken with skill instruction and practice organization videos much more frequently in the 101 course, so it is plausible that these live demonstrations had more of an impact on coaches (indeed, several coaches noted that these skill demonstration videos were quite valuable). Another consideration is that the course does not provide coaches with an opportunity to plan how they would implement these concepts into their practice. These concepts are more complicated than drill instruction or practice organization, and thus require more planning and reflection for a coach to put this knowledge into practice. For example, an activity that asked a coach to write a brief imagery script that they could use during practices to improve a common skill (e.g., streamline kicking, flip turns) would help this knowledge cross the knowledge-practice divide. This type of an activity would be considered an output in the program logic model, which is an immediate, tangible product resulting from the learning experience. The Foundations course lacks tangible 192 outputs that coaches can carry away from the course and put into practice when they begin coaching. Increasing outputs would logically help to bridge the knowledge-practice divide. A different consideration is that these two important topics appear in the midst of eight other important topics in the Foundations 201 course. Thus, there might simply be a problem of information overload, causing coaches to learn this content well enough to pass the test, but not spend enough time focused on its application, thus preventing it from becoming a part of a coach’s repertoire. After completing the analysis of program effects, it appears that topics in the Foundations 101 course had stronger effects on coaches than the topics spread throughout the Foundations 201 course. This may have happened because the topics in the 201 course are more difficult to understand when compared to the topics in the 101 course, which are easier to understand and more immediately applicable to the coaching contexts faced by most of the coaches in this sample. The length of the 201 course may also contribute to decreased effectiveness. Although the topics in the 201 course have important value for coaches, it may be more effective to present them in smaller pieces. This could rectify two seemingly opposing complaints about the 201 course. Some coaches stated that it was too much information so they tended to focus on just getting it done, while others said that it didn’t go into enough detail to answer some of their harder questions. Pursuing a competency-based approach, where groups of related competencies were organized into mini-courses (e.g., biomechanics and teaching stroke technique, designing workouts to maximize physiological training effects, using goal-setting effectively), coaches could develop competencies by completing these courses one at a time, rather than simply doing them all at once in order to meet their coach education requirements. These mini-courses could be chosen with the requirement that a coach complete a minimum of three competencies each successive year after completing the Foundation 101 course and 193 attaining their initial coach membership. Breaking topics into mini-courses would allow for coaches to choose the competencies they most need to address, thereby customizing education to meet the needs of courses. This approach would also allow new mini-courses to be developed on a needs basis. If new challenges were to arise from the membership, e.g., strategies for inclusion and instruction of swimmers with physical impairments, USA Swimming could be more responsive by adding a 30-minute course to address this need, rather than trying to fit it into the existing framework of the 101 and 201 courses. More customization would allow coaches to pursue a path that meets the needs and challenges that arise in their own context, while still maintaining a framework of competencies that all coaches must possess. The content for such mini-courses is, in most cases, already produced; these changes would affect the manner in which it is organized and delivered to the coach. Another concern was the program’s ability to shape coach attitudes with respect to developing a professional mindset and the value of being a lifelong learner (valuing professional development). The program appeared to have minimal effects on these two attitudes, with coaches stating that these are values most likely inherent in the coach at the outset of the course. An entity theorist’s view would suggest that if these attitudes matter, efforts should be made to recruit coaches who already demonstrate these attitudes, rather than spending valuable time in the Foundations course trying to develop them. However, as suggested by Coach 7, these attitudes may be highly dependent on previous coaches that the new coach had as an athlete, suggesting that the outward manifestations of these attitudes (e.g., decorum with athletes, continual learning) are modeled. In order to shape coach attitudes, different approaches might be taken that could have a stronger impact on coach adoption of these attitudes. For instance, a coach education program for collegiate wrestling coaches (Gould et al., 2012) employed 194 respected, veteran coaches to deliver important and emotive messages to new coaches about the importance of professionalism, promoting the sport of wrestling, protecting the welfare of athletes, and developing the athlete as a person. The logic guiding this decision was that these veteran coaches would be much more influential than coach educators could ever hope to be. In the interim findings meeting with key program decision-makers, the possibility of getting respected coaches to deliver the message about professionalism was outlined as a strategy to address this deficiency within the Foundations program. Because these topics do not attract much attention at coaching clinics, it would be more appropriate for these veteran coaches to deliver this message as part of a lecture that was mostly focused on popular topics of interest, such as athlete training and athlete talent development. A similar approach could be taken for the coaching behavior of developing the athlete as a person. The results showed that coaches had a variety of approaches to this behavior, but that it was not clearly adopted by a significant minority of coaches. As identified in the interim findings meeting, it is entirely possible that the expectancy of results for developing the athlete as a person is not very clear. To promote this behavior, one strategy would be to show more examples of coaches explaining the benefits of developing the athlete as a person, and how those benefits also translated into better swimming performances. It would also be prudent to tie this behavior to the concept of role-framing, which arises in the reflective practice literature (Schön, 1987). When role-framing, a coach clearly delineates their role in the process of developing the athlete as a person, which helps to guide the types of deliberate actions that they take, and the types of reactions that they have when swimmers experience setbacks or adversity. Many of the concepts introduced by the coaching philosophy section of the course address the concept of understanding your swimmers’ needs as well as your personal and professional needs as a coach, 195 but there is little content that addresses how a coaching philosophy translates into behaviors. Role-framing takes a more direct approach towards identifying how a coach’s personal and professional beliefs guide their behaviors in practice. Using the role-framing terminology espoused in the reflective practice literature to guide this lesson might help coaches to integrate their beliefs into a more concise and coherent statement. If a coach could complete the course having identified and written their intended role-frame, it would help to focus their actions towards the desired behaviors of professionalism and developing the athlete as a person. These discussions of strategy regarding character education provide an interesting insight into the nature and location of character development as a values-driven enterprise, as well as the perceptions of coaches on who needs to be positively developed and who does not. Character development has been categorized into two camps: deficit-reduction approach, and assets development approach (Benson, 1997). Deficit-reduction approaches are developed to reach athletes who live in contexts that put them at-risk for becoming involved in problematic behaviors. These programs are generally situated in economically depressed urban environments, where sport programs can often serve as a buffer against many of the risks that threaten young men (generally young men of color), such as gang involvement, school truancy or poor academic performance. The assumptions made by a few coaches in this sample suggest a perception that swimmers do not suffer from a lack of character, nor do they face environmental risks that might threaten their well being, which would then justify some form of character education or life skill development program that would enable kids to combat these risks. Such observations are rooted in a dialectic on character deficit that are primarily driven by an assumption that threats to moral behavior arise in contexts such as economically-depressed urban cores, and not in contexts 196 with economic stability. Swimming thrives in economically-stable locations. This assumption takes a rather narrow view on what constitutes immoral behavior. For instance, financial impropriety in financial markets frequently leads to downstream effects that impact the lives of people who have very little to do with financial markets. Such immoral behavior is not typically perpetuated not through the actions of urban gangs or drug dealers, but through the actions of employees working for financial companies, individuals from the very middle-to-upper class backgrounds where swimming thrives. Thus, the perception that character development is not necessary for children growing up in middle to upper class communities is inherently rooted in a benignly racist and classist assumption about the location from which immoral behavior arises. Another interesting finding was that talk of how athletes had changed as people, and the effects coaches made on these athletes naturally veered towards emotion management. This may be because swimming has a unique capacity for coaches to work individually with athletes, handling their own reactions to specific situations that only involved them. This is one of the unique aspects of swimming when compared to a team sport. It provides a unique context for coaches to help athletes learn about attribution, because most performance outcomes in swimming can be attributed directly to something that the swimmer has done, as opposed to other sports, where a teammate’s actions might be the attribution for a loss or mistake. Coaches often state that the clock does not lie, owing to the objective nature by which performance in swimming is measured, and the assumption that poor performance is directly attributable to poor preparation. Thus, when compared to interactive sports (e.g., basketball, soccer), swimming provides a unique context for athletes to directly experience the consequences of their actions. Although other psychological factors figured into the development themes that coaches espoused (e.g., goal setting, confidence), the greater amount of focus placed on emotional maturity and 197 coping with disappointment provides an interesting window into coach perceptions of what influence they can have. Then again, this may be due to the ages of swimmers being coached by the coaches in this simple, which with few exceptions, were children and adolescents, primarily between the ages of 8-14. Emotional maturity and learning to manage your emotions are particularly salient features of development at these ages, but these were not the only psychological changes that coaches facilitated. Several coaches noted their successful implementation of goal-setting practices and the receptive responses of their athletes. Recent work has suggested a greater need for understanding the specific ages where athletes are most sensitive and receptive to learning specific mental skills or developing psychosocial or emotional coping mechanisms (Driska et al., 2014). Developing such an understanding of the sensitivities of swimmers to specific mental skill and psychosocial/emotional development initiatives of coaches could provide coaches with a roadmap of what skills can be developed at what ages. This topic of character development (or life skill development) through sport is a source of vigorous research and commentary in the sport psychology and youth sport research domains (Gould & Carson, 2008; Turnnidge, Côté, & Hancock, 2014). Researchers have traditionally advocated two viewpoints on the question of how athletes develop character and learn life skills through their sport experiences. Active or deliberate approaches involve coaches who intentionally work to develop their athletes as people. These intentional efforts involve building the coach-athlete relationship as a conduit for the coach becoming a positive developmental influence, the use of deliberate meetings with athletes to teach an important life lesson, or through the manipulation of certain distinct features of the sport environment to create experiences that will develop the emotional coping strategies and other resources needed to handle life’s adversity. Researchers have also advocated the reactive approaches, including the 198 use of teachable moments, but also the belief that sport, on its own and without the intentional efforts of actors within the sport context (e.g., coaches, athletes, referees, parents), develops character in athletes. Perhaps unaddressed by this literature would be those coaches who may only engage in minimal, reactive behavior management strategies, rather than a more deliberate approach. The coaches in this sample express views that fall across this continuum. The literature that examines positive youth development, life skills, and character development through sport may be experiencing some sampling bias. Studies that solicit coaches for an interview or survey with a strong sensitizing concept such as positive youth development through sport or developing character in athletes can clearly attract coaches with stronger than average opinions on the process, rather than a truly random sample of coaches whose views represent the true distribution. This evaluation was not cast as a study on positive youth development or lifeskills acquisition study, and although it did not explore the topic of character development in depth, its sample may represent a much closer approximation of actual coaching behaviors. The process of a coach developing the athlete as a person appeared to be very idiosyncratic, and at times difficult to discern from where the motive to engage in this behavior arose. The evaluation design was exploratory, with the intent of gaining more information about several facets of coach perceptions, knowledge, skills, attitudes, and behaviors. It was not designed to make an accurate determination of the degree to which coaches are acting to develop the athlete as a person. Thus, these results are offered as an exploratoration of coaches’ views towards character development, warranting further study and attention; it is meant to demonstrate the bounds of the topic in a swimming context, and the range of viewpoints, rather than the predominance of one viewpoint over another. Program delivery. The Foundations of Coaching program was delivered with relatively 199 few technological impediments. Although the use of an online course assured consistency of program delivery to the user, the ways in which different coaches learned and implemented knowledge from the online course varied considerably. The coaches in this sample entered the program with wide variation in both previous swimming experience and their current coaching contexts, but when baseline effects (i.e., self-reported changes in knowledge, attitudes, or behaviors) of the course are considered, it successfully delivers information about ageappropriate training and long-term athlete development, and is able to spur a greater amount of reflection in coaches. For those topics that did not appear to have a great impact on coaches, e.g., mental skills, physiological training design (discussed previously), it may be worth reconsidering if there is a desired baseline effect for this information. Baseline effects could be identified through the creation of specific objectives for each module of the course (e.g., demonstrate the ability to write a simple imagery script and deliver it to their swimmers). Although the goal of the course may simply be to expose coaches to knowledge with the hope that they will use it, these findings suggest that if the material is not found immediately useful or applicable, the knowledge may not be retained for later use. The creation of application exercises may help coaches to see how the knowledge applies to coaching, and although coaches may not use this information later, the application exercise should strengthen retention of this knowledge. The decision of how strongly to push the application of knowledge is not simply a matter of education; it is surrounded by a larger political debate on how actively USA Swimming should advance a position of what an ideal swimming coach should do. As one key program decision-maker stated in the interim findings meeting, USA Swimming walks a tightrope with respect to coach education. Traditionally, USA Swimming has embraced a neutral approach 200 that does not advocate for a correct way of coaching swimming; instead the goal is to present quality knowledge and hope that coaches embrace and implement it into their daily coaching practices. This neutral approach allows for a diversity of coaching styles to persist, which is a strength of the swimming culture in the USA. However, the Foundations of Coaching 101 course adopts a stronger advocacy for teaching long-term athlete development, age-appropriate training, and incorporating fun into workouts, and it appears that this knowledge was perceived as more useful and more likely to be implemented by the coaches in this sample. The link between advocacy of a position and the subsequent stronger effects on coaches calls into question the mechanistic effect of the course: is it the advocacy that leads to the effect, or is it the pure usefulness and common-sense nature of topics like LTAD and age-appropriate workouts? This evaluation did not extensively probe the link between advocacy of knowledge and the adoption of coaching practices, but these findings raise an important question for key program decision-makers to consider in the design of future programming: to what degree do we wish to advocate a particular position? Another consideration would be to ascertain the opinions of the USA Swimming membership with regards to what topics they feel USA Swimming’s position to be the authoritative and acceptable position. Given the need to re-examine the effectiveness of some topics in the 201 course, it would be worthwhile to pursue the possibility of breaking out these topics into separate competencybased mini-courses. Smaller courses that focus in greater depth on one topic could increase the amount of educational time that could be devoted to each topic. Redesign might incorporate the use of greater video support from coaches and experts (e.g., talking head interviews) that walk the user through the process of applying the knowledge to everyday practice. A second essential feature would be pairing this increased video support with the use of a program output, such as a 201 guided exercise on how to write an imagery script or how to construct a 90-minute practice for an 11-12 year-old age group. Areas for improvement. There are some relatively simple improvements that could be made to the existing course that would require minimal effort from program staff, and could have profound benefits for the educational experience gained from the course. These improvements include consolidating and organizing the educational application activities in the current course, and the use of social media to help facilitate a stronger community of practice focused on good coaching. More advanced improvements to the course would be to explore the development of topical mini-courses that extend upon the topics presented in the Foundations 201 course. Finally, depending on the difficulty and cost, it might be feasible to obtain learning metrics from ePath Learning, as well as creating embedded surveys to assess the development of coach attitudes and behavioral intentions during the course. First and foremost, attention to some simple improvements to the existing course could help to organize a coach’s knowledge. Aggregating all of the existing PDF documents currently used to support the course into one workbook file could accomplish this goal rather easily. The workbook file could be provided as an electronic PDF that coaches could edit on a computer, or print. For some users, it might be worthwhile to offer a printed and bound copy for a small price. Aggregation of documents into a workbook also provides an opportunity to revise the activities that coaches complete as a result of taking the course. The addition of such activities (i.e., program outputs) that were focused on application of course content in the coach’s context would help to support the implementation of knowledge that is more abstract, such as mental skills training and the use of physiological training zones for workout design. Another solution to help support implementation of content knowledge at scale (2,000+ 202 coaches nationwide) would be the use of a social media campaign that could help facilitate the transfer of knowledge from USA Swimming to the coach membership, but would also allow for USA Swimming to share and disseminate best-practice knowledge that was produced by coach members. This would allow like-minded coaches to come together to share their ideas and discuss the practice of coaching, similar to the community of practice idea that has been advanced in the coach education literature (Culver & Trudel, 2008). Previous work examining smaller coach education initiatives have noted that meetings between coaches naturally devolved into communities of practice (Cassidy, Potrac, & MacKenzie, 2006); thus, there may be some value in bringing together coaches under the auspices of a formal program, but allowing it to evolve into a community of practice. USA Swimming has a unique ability to facilitate a quality community of practice amongst coaches, although Culver and Trudel (2008) note that cultivating a community of practice requires significant time and effort. Several coaches noted that the USA Swimming website held great resources, but these resources were often difficult to find, especially on short notice from a mobile device. The USA Swimming website is not enabled for easy social media sharing of information, nor is it compatible with mobile devices. With most website redesigns being configured for mobile use, any website configuration should consider the two design constraints of being easily mobilized and social-media enabled. Another possible medium for facilitating this conversation could be the Deck Pass app, which is currently designed to provide access to swimmer times from the national database, meet results, but that also allows users to earn badges for certain accomplishments, or for coaches to network with their peers. This app could be configured to offer resources such as stroke technique videos or even customizable mini-courses. The advantage of using the Deck Pass app would be its provision of a one-stop shop for coach needs, rather than the coach having to 203 maintain a working knowledge of different sources for the best information related to certain aspects of coaching (e.g., technique videos, training set ideas, challenges and games). A second major area for improvement, as discussed previously, would be the provision of topical mini-courses that delved into topics of interest in much greater detail. Given the findings that the course content from the 201 course tended to overwhelm some users, there is merit in developing stand-alone topics. This could be complicated by the fact that the Foundations course is a partnership between USA Swimming and the American Swimming Coaches Association (ASCA). The working agreement is that continuing education beyond the 201 course is handled through the ASCA system, with the next step being ASCA Level 3 certification (followed by Levels 4 and 5). The advanced topical knowledge in these courses is not delivered through an online course, but rather a paper/pencil format. Given that this evaluation has demonstrated the efficacy of an online delivery mechanism for coach education, there could be more incentive for ASCA to move its programming to an online format. However, before such a redesign was undertaken, it would be highly advisable to consider the user’s needs. A competency-based system for coaches might cut across the current levels of ASCA training and the current Foundations 201 course. Therefore, discussions between ASCA and USA Swimming would need to occur in order to explore the possibility of a universal, competency-based system. Such a redesign might not involve much change to the existing content knowledge, but a change in the mechanism and timing of its delivery to the coach. The advantage of a competency-based system would be its adaptability to the changing needs of today’s swimming coach. Currently, to add content knowledge to an existing course, significant redesign efforts must address pedagogical and technological methods for teaching new content, application activities, as well as assessment. With a competency-based system, a 204 learning module could be implemented from the ground-up, with little need to address existing content. Such a system also would allow for the creation of new modules to address the needs of coaches. For instance, several coaches discussed their concerns about working with athletes that have special needs (e.g., autism spectrum disorders, ADHD), or their need to learn about detailed stroke analysis. To be more responsive to its coach membership, USA Swimming could implement a coach-member survey that could discern coaches’ most pressing needs, and then design competency-based modules in these areas to respond to those needs. The disadvantage to a competency-based system is that without being prompted to make connections between different bodies of knowledge (e.g., the effect of physiological training on a swimmer’s mental state), a coach may have all the pieces of content knowledge, but not have them connected into a coherent body of coaching knowledge that can guide coaching practice. One resolution would be a reflective-practice competency that addressed the tenets of reflective practice (Schön, 1987; Trudel & Gilbert, 2006) as a means to facilitate a coach’s integration of knowledge across the respective competencies. Although all of the previous improvements could add vitality to current coach education efforts, perhaps the easiest and most critical improvement would be to improve the current system of data communication between ePath Learning and USA Swimming in order to monitor and assess the development of coach knowledge, attitudes and behaviors. Since the outset of this evaluation, there is now a provision that allows coaches to enter their USA Swimming member identification code into the ePath system before beginning the Foundations course. This simple improvement eases the burden of communicating test-scores after their completion, significantly reducing the paper chase and administrative load. However, this simple addition allows for more detailed monitoring of coach performance in the Foundations course. As noted previously by the 205 evaluator, test-score data and demographic data were in two different datasets, and had to be merged by hand in order to create the sample of coaches for this evaluation. This laborious step would have made monitoring efforts too burdensome to undertake. Now, with the provision of the coach ID prior to taking the Foundations course, monitoring efforts could be done much more easily. As discussed with key program decision-makers at the interim findings meeting, the incorporation of attitudinal and behavioral intention measures through an embedded survey instrument in the Foundations course could be used to examine development of coach knowledge and attitudes before, during, and after taking the Foundations course. Utilizing a theory of planned behavior design (Ajzen, 1991), such a survey could be used to explore how coach attitudes to a topic (e.g., long-term athlete development, developing the athlete as a person) evolved as a result of taking the course. Measures of behavioral intention could assess a coach’s likelihood of implementing changes in coaching practice in accordance with those attitudes, and could demonstrate how the course shapes behavior change. Such a measure could be informative to USA Swimming, as it would show the relative effectiveness of the instructional approaches of the course in shaping the coaching behaviors that matter. If it were shown that the course did not have the ability to shape coach attitudes or behavioral intentions, it could help direct coach development efforts towards methods that might have a greater effect. For instance, although coaches noted that the course impacted their attitudes regarding long-term athlete development, it could be that these attitudes were already primed through the coach’s previous experience as an athlete. Noting that a few coaches mentioned stories about how they or their friends had experienced burnout as competitive swimmers, it could be that this experience made them that much more sensitive to long-term athlete development principles. However, the 206 course may have been the deciding factor in taking this general concern and turning it into decipherable coaching behaviors (e.g., age-appropriate training, incorporating fun into workouts). For attitudes that the course was not able to shape significantly, such as the adoption of a professional mindset, data from an embedded survey might suggest that if USA Swimming wished to develop this mindset in their coach members, it might be more effective to encourage visible leaders and respected coaches in the sport to push this message whenever they spoke to the media or presented at coaching clinics. The bottom line on such monitoring efforts is that they would allow USA Swimming to be more strategic in the selection of methods to shape coach attitudes and behaviors. Utilization of evaluation process and findings. A fifth purpose of this evaluation was to provide findings that could be easily utilized by key program decision-makers at USA Swimming, as well as facilitate an evaluation that enables the organization to benefit from the process of evaluation, through the development of evaluative thinking and strategizing (Patton, 2013). To facilitate the future utilization of findings, it is important to present answers to the questions raised by key program decision-makers during the evaluation process. The first part of this section will detail responses to several questions raised by the key program decision-makers, and the second part of this section will detail how the evaluation process affected the key program decision-makers. Use of evaluation findings. Three potential barriers to program success were identified in the first focus group meeting: barriers from the online learning environment, barriers within the coach, and barriers from coach influencers. After conducting this evaluation, the online learning environment did not present significant barriers to coach learning; in fact it represented a substantial improvement over previous versions of the Foundations course. Although several 207 areas for improvement were discussed previously, the online course accomplishes the goal of delivering knowledge, and to a smaller extent, shaping attitudes and behaviors. Supporting the implementation of this knowledge is something that should be complimented through other outreach efforts (e.g., social media). The barriers to coach education appear to lie primarily on the coach side. Within the individual coach, the biggest barrier appears to be the belief that a coach’s sport knowledge is sufficient to be a coach, which did not arise in the majority of coaches, but was still present at a significant level. To overcome this barrier, it may be useful to show videos of both experienced and young coaches delivering the message that sport knowledge is important, but coaching knowledge is a separate entity that must be cultivated. To this end, the evaluator recalls a particular instance where this impression was made upon him, when Jim Steen (the highly-decorated coach for Kenyon College) stated at a coaching clinic that, “it’s what you learn after you know it all that matters.” Another significant barrier that was identified by a few coaches in this sample was the barrier imposed by experienced coaches who have little interest in changing the ways that they run their programs. This could impose a significant barrier to the adoption and implementation of long-term athlete development principles, which, as demonstrated in this evaluation, were particularly salient and influential concepts delivered by the Foundations course. To address this barrier, efforts to shape the beliefs and practices could be made through having respected and influential coaches and sport leaders speak about the value of long-term athlete development; especially convincing would be the cases of elite coaches who were skeptical about LTAD, but when they made the change, they found it was a major improvement. The key program decision-makers expressed some concern that other stakeholders in the swimming community might challenge the efficacy of the Foundations of Coaching course. 208 There was no evidence to support the claim that the Foundations course stifled innovation in coaches. If anything, the opposite was true, in that it provided novice coaches with a greater range of tactics and strategies with which they could experiment, which helped to foster creativity, notably through the adoption of practices like goal setting and incorporating fun and challenge into training. Second, there was no evidence to support the viewpoint that formal education does not work; although the Foundations course does not provide a coach with all of the information they will need in their career, all coaches in the sample with the exception of one reported that they gained something useful from the course. Third, there was little evidence that the current approach dumbed-down the information in the course. If anything, by setting the information in the coaching context, it made the information more immediately usable, especially for novice coaches with less sport experience. There was a demand from some coaches for more detailed knowledge on some topics (e.g., seasonal planning, physiology, mental skills). Finding ways to deliver more knowledge to those coaches who want it may come through formal education, but there are myriad ways for the delivery of advanced knowledge, and not all may be wedded to the traditional course-based learning approach. The remaining three skeptical viewpoints are a matter for future discussion and potential research avenues. First, the notion that a better educated coach is not necessarily a better coach raises an existential debate on the definition of a better coach. In order to show if education could make a better coach, there would have to be conceptual clarity on the exact definition of a better coach; getting coaching experts to agree on this definition represents a herculean task, but it is not beyond the realm of research, as significant efforts have been made recently to define coaching excellence in range of sports (Côté, Young, North, & Duffy, 2007). Second, the notion that coaching is a profession that is best learned through mentorship or apprenticeship does not 209 necessarily exclude formal education from contributing to the knowledge of a coach. Previous research (Gould et al., 2012) has demonstrated that mentorship has a key role in helping a coach to grow professionally through helping to contextualize a formal educational experience, as well as helping to create a framework for future professional development. Coaches in this sample identified the importance of mentorship while simultaneously indicating a value for formal education. They also indicated that the information from the course often created valuable opportunities to discuss best practices in coaching with peers and supervisors, thus demonstrating that the course has the ability to introduce new ideas into the field from the grassroots. Third, the contention that too much focus on long-term athlete development will lead to a generation of soft swimmers is a question to be answered by a completely different line of inquiry. Long-term athlete development is an idea on the verge of demonstrating empirical evidence of widespread success; unfortunately, at the moment, the evidence in its favor is primarily based on theory assimilated from several domains of human development. The guiding logic of long-term athlete development is that it keeps more athletes involved in the sport, thereby building the base of the athlete development pyramid, and increasing the odds that more talented athletes will be retained by the sport. Empirical evidence is needed to show that higher levels of competitive success can be explained by greater retention of swimmers. The possibility to demonstrate the effectiveness of long-term athlete development in swimming might be accomplished through population-level statistical modeling using the massive dataset of the SWIMS database, which contains the race times for every swimmer registered with USA Swimming extending back to 2003. Developmental progress could be measured objectively using swimming times and FINA Power Point rankings (normative time values) as a dependent 210 variable, and using number of meets per season as an independent variable. This would show the relationship between volume of competition and performance outcomes over time. Given the extent of the SWIMS database, this is a unique opportunity that USA Swimming has that could demonstrate the viability of long-term athlete development. Use of evaluation process. Given the adoption of the utilization-focused evaluation framework, the evaluator made special consideration of evaluation use by primary intended users over the entire course of the evaluation. Utilization is not only about producing results and findings that can be put into practice; it is also about building capacity in primary intended users to think in an evaluative fashion. Patton (1998) defines process use as “using evaluation logic and processes to help people in programs and organizations to learn to think evaluatively.” Through engaging key program decision-makers in the process of describing how the Foundations program created changes in coaches, the evaluator was able to facilitate an opportunity to apply evaluative thinking to an existing program. The most notable area of process use was uncovered during the initial focus group meeting in January 2014. Following this meeting, the evaluator asked each of the key program decision-makers to indicate their impressions and take-home messages from the meeting. The responses were categorized by two main viewpoints. The first was that thinking about desired outcomes of an educational package during the design phase would be extremely useful. One staff member raised the point that “we ended up with a good course” despite not going through the process of identifying desired outcomes prior to the course redesign; this insight demonstrated that the staff had a strong implicit sense of what knowledge coaches needed, and the best ways to present that knowledge. Findings from the evaluation partially support this notion. The organization of the Foundations 101 course around specific coaching duties (e.g., communicating with athletes, responsibilities of 211 a coach) had a stronger effect than the organization of the Foundations 201 course, which organized content around knowledge topics (e.g., physiology, mental skills) with less information on how to implement this knowledge into everyday coaching practice. With more attention to these topics, it is anticipated that revised programming of the course content from the Foundations 201 course could have similar effects as the programming in the Foundations 101 course. A second use of evaluation process was that the key program decision-makers expressed greater awareness of how their professional development strategy was serving the coach membership. They noted that the Foundations course needed to make better connections to the continual professional development options that USA Swimming offers its coach members (e.g., webinars, sport performance consulting). The staff members who made these observations all expressed the concern that the current efforts at continual professional development seemed to attract the same set of coaches and clubs, but had yet to penetrate a much larger share of the membership. It was suggested at this meeting that the Foundations course might do more to explore next steps in a coach’s professional development. Given that the evaluation found that the course did not have a strong effect on coaches adopting a value for continual learning and professional development, this might have occurred because the course did not specify avenues for professional development; thus, had a coach been motivated by the course to seek more education or development opportunities, it was not clear what that next step should be. Several coaches mentioned their intent to attend webinars, coaching clinics and conferences, and to take the ASCA Level 3 course. However, these coaches also mentioned that they had not attended any webinars, and that they had not enrolled for ASCA Level 3. One recommendation in regards to this finding would be to help support coaches who want to make the next step by providing a 212 common landing page on the USA Swimming website devoted to explaining continuing education opportunities, or through the provision of a phone number to call if you are interested in attending a webinar or having a sport performance consultant speak with you. With regards to this recommendation, the key program decision-makers were conscious of their capacity, and hesitant to provide an immediate solution, for fear that it might generate leads of interested coaches that they would not have the staffing power to pursue. However, finding more ways to continue the conversation in order to further professional development aims of coaches was an explicit goal arising from the interim findings meeting in July 2014, and a priority for key program decision-makers. Formative Judgments and Recommendations This evaluation is exploratory and formative in its design, which makes summative judgment of its merit and worth inappropriate at this juncture. However, formative judgment offers an opportunity to address the basic question that underlies the evaluation: does the Foundations of Coaching program make a difference? The evaluator offers the following formative judgments with the caveat that they represent progress towards understanding this program’s effectiveness in meeting objectives. First, the Foundations of Coaching program is significant in that it has made coach education a positive force, rather than a dreaded hurdle that stands in the way of a coach’s credentialing process. Coaches find useful information in this course, and they are receptive to the messages presented by USA Swimming. Second, this program is significant in that it is an effective mechanism for delivering basic coaching principles and skills that can be of immediate use to coaches as soon as they begin their work. Third, it is significant for introducing conceptual knowledge about long-term athlete development, for shaping attitudes related to 213 LTAD, and for shaping the adoption of coaching behaviors (e.g., age-appropriate training, incorporating fun into workouts) that support the implementation of the LTAD philosophy. The Foundations program has the potential to evolve with relatively few technological burdens so that coaches can learn more about advanced topics. In its current form, the Foundations of Coaching program struggles to deliver content knowledge that has not been contextualized. Relative to the needs and challenges reported by coaches in this sample, the Foundations course does not deliver adequate knowledge with regards to working with swimmers that have special needs (e.g., autism spectrum disorders, ADHD). The course does not enlist a concrete set of program outputs (e.g., workbook application exercises, implementation plans) that would help to facilitate the transfer of knowledge from the course to everyday coaching practice. Finally, for coaches who are not already sensitized to the importance of these topics, the Foundations program does not appear to shape attitudes and behaviors in relation to a professional mindset and developing the athlete as a person. Based on these judgments, the recommendations that were discussed in depth previously in this chapter will be summarized. First, continue to support online learning as a preferred method for delivering relevant knowledge and shaping attitudes and behaviors on a large scale for all new coach members. Second, explore ways to continue the conversation with coaches after they have completed their coach education requirements as a means of encouraging continual professional development and supporting coaches’ implementation of changes in coaching practices. Third, consolidate all of the supplemental PDF documents into a single workbook file, and revise content as needed so as to provide more concrete program outputs. Fourth, explore alternative ways to deliver advanced content knowledge, perhaps breaking the 214 201 course into smaller competencies that are earned through badge system using USA Swimming’s Deck Pass app. Fifth, develop a consensus statement on the position of USA Swimming relative to coaches working to develop the athlete as a person, and then develop strategies to help coaches meet this goal, either through coach education programs or other means of outreach. Limitations There are several limitations to this evaluation that the reader should consider in their interpretation of findings. First, a major strength of utilization-focused evaluation is its insistence on producing a usable findings and process for primary intended users (i.e., key program decision-makers). This strength is also a weakness in that by design, the evaluation may omit key stakeholders in the evaluation process, as they may complicate the evaluation and decrease the likelihood that utilization will occur (Patton, 2011). Two key stakeholders in online swimming coach education that were not included in this evaluation were the American Swimming Coaches Association, a partner in developing the course and the official provider of continuing education for coaches after they have completed Foundations of Coaching 101 & 201, and ePath Learning, the company which houses and delivers the Foundations program. ASCA formerly handled the delivery of the first two levels of coach education for USA Swimming coach members. The new online delivery approach used by the Foundations of Coaching courses was a departure from the model employed when ASCA facilitated coach education. The findings of this evaluation may be of use for ASCA, especially with regards to the efficacy of using an online course to deliver knowledge. The evaluator made a decision to let the key program decision-makers at USA Swimming be the arbiters of these findings to ASCA, due to their long established working relationship. The decision to not include ePath Learning in the 215 evaluation was based primarily in the logic that the content creation was primarily under the control of the key program decision-makers at USA Swimming, and could be adjusted based on findings from the evaluation. This was done with the caveat that findings could be shared with ePath in such a way that they might be useful for other courses that they host. A second limitation was the amount of time spent with key program decision-makers in developing a logic model during the first focus group meeting in January 2014. The intent was to create the beginnings of a logic model and explore how specific processes in the course were linked to anticipated coach outcomes, then to have the evaluator continue to develop this logic model over the course of the evaluation. Given the time constraints of the first focus group meeting, there was not enough time to fully engage in this process, leaving the creation of the first draft to the evaluator. This logic model was revised slightly at the outset through e-mail correspondence with two key program decision-makers. A more thorough face-to-face meeting could have yielded a better preliminary logic model, however, this was not practical given time and travel constraints. Although this limited logic-model building process did not significantly compromise the quality of this evaluation, engaging in this process could have had beneficial effects for the key program decision-makers, as it would have provided an opportunity to deepen critical reflection on the processes by which the Foundations of Coaching program works to effect change in coaches. A deeper understanding of the process could have had an impact on future planning and design of related programs. This is regarded as a missed opportunity, but in the judgment of the evaluator, this lack of process did not significantly compromise the evaluation findings. The take-home lesson for the evaluator was to budget more time for the logic modeling process in initial focus group meetings with key program decision-makers. A third (although minor) limitation was the retirement of the evaluation committee leader 216 at USA Swimming while the evaluation was still being conducted. In order to present interim findings before her retirement, the evaluator had to rush analysis, thus the interim findings were still descriptive in nature. However, because the evaluator presented the findings as interim, the meeting opened-up considerably, and key program decision-makers were able to explore possible causal linkages between program elements and coach outcomes. For instance, the suggestion that there is not a clear link between developing the athlete as a person and expectancy of improved swimming performances emerged purely out of discussion at this meeting; it is unlikely that the evaluator would have drawn this connection. Patton (2011) reminds evaluators that primary intended users (i.e., key program decision-makers) often have a strong working knowledge of how their programs operate to effect change in participants. The preliminary nature of the data presented at this meeting provided enough evidence that key program decision-makers were able to make connections that were not possible prior to the evaluation process, and thus the facilitation of this process enabled the richness of their experience to complement the inquiry process. A fourth limitation lies in the method of inquiry chosen by the evaluator (with the advisement of key program decision-makers) to generate evidence about the effectiveness of the program. Given that the purposes of the evaluation were exploratory and formative in their nature, a qualitative, interview-based inquiry was chosen as the dominant form of research and analysis, in order to facilitate a broad range of viewpoints on the Foundations course. The inquiry method of interviewing coaches was able to ascertain coach perceptions of the course, as well as coach-reported changes in knowledge, knowledge implementation, as well as attitudinal and behavioral changes. Self-report data, especially that which is collected in a retrospective interview, relies upon the strength of the interviewer to pursue counter-factual evidence, such as 217 asking the coach to speculate on whether or not these changes would have occurred had they not taken the Foundations course. Although the interviewer was vigilant in pursuing these counterfactual inquiries, this method cannot be assumed to demonstrate the causal nature of the course. In addition, particular lines of questioning lent themselves to considerable interpretation by the participants. For instance, the definition of age-appropriate training could vary widely. The evaluator asked coaches to explain age-appropriate training through the use of specific examples, but there exists the possibility that the examples provided by coaches might not be representative of typical training that they give their athletes. Given the exploratory nature of this evaluation, however, the intent of this approach was to help gain a better understanding of the effects of the program and association between elements of the course, much in the same way that an elicitation study is often the precursor to a randomized experimental design. This limitation was discussed with key program decision-makers at both the initial and interim findings meetings, and they expressed their understanding that this evaluation (in its current state) is designed to show an association between taking the course and desired coaching outcomes, as opposed to attempting to demonstrate that the Foundations course is the causal agent of these outcomes. Thus, to claim that the Foundations course was the sole cause of changes reported by coaches in this sample would be an overstatement of the evidence. This method of inquiry relied upon the evaluator’s ability to stay grounded and immersed in the data (Patton, 2002), a method which can introduce some bias into the research. To counter this bias, the evaluator conducted peer-debriefings with other researchers with expertise in qualitative inquiry, as well as using the interim findings meeting as a source of peer debriefing with key program decision-makers. However, having another member of the research team who was equally immersed in the inquiry process might have strengthened the inquiry, particularly 218 with regards to increasing the level of causal inference drawn by these data. A future bestpractice for continued evaluation would be the addition of a second member to the evaluation team who could fulfill this role. Future Evaluation Directions This evaluation represents a formative effort to explore how the Foundations of Coaching program effects change in program participants. The next step towards garnering more evidence of the program’s effectiveness would be to develop methods that would allow for causal explanation of how specific elements from the program caused changes in the coach. Attempts to demonstrate causality of the Foundations of Coaching course could be informed by the model of program theory developed by this evaluation. One approach would be to use the program theory to devise a quantitative survey instrument to measure changes in knowledge, knowledge implementation, attitudes, and behaviors amongst coaches who take the course. Because the entire population of new USA Swimming coaches completes the course, it offers an interesting possibility for research at a population level, rather than selecting representative samples. Using a pre/post/follow-up quasi-experimental design, the administration of this survey could examine how attitudes change as a result of the course. With a large number of coaches completing the survey, it would be possible to see how specific attitudes cluster at the outset of the course, perhaps showing how attitudes were distributed in the population of new swimming coaches. Over time, this survey could be revised and developed into a smaller instrument through psychometric analysis. If it could be used to reliably predict coaching outcomes, it could be particularly useful in identifying coaches who might need more follow-up education, or who might be ideal candidates for some form of mentoring, or a follow-up visit from a sport performance consultant. 219 If the aforementioned survey could reliably demonstrate specific coaching outcomes, another potential evaluation (and research) direction would be an experimental design. The population of swimming coaches in the United States is split between coaching USA Swimming affiliated club teams that require the Foundations of Coaching program, and non-affiliated programs (e.g., interscholastic programs, summer league teams) that may not require the Foundations program. Coaches that do not take the Foundations program form a natural control group; thus the possibility for demonstrating causal effects of Foundations course exists. Such studies would need to go beyond self-reported behaviors, and use other methods to triangulate coach knowledge, attitudes, and behaviors. A range of validated survey instruments could be used to gain athlete perceptions of how well the coach was addressing athlete psychosocial needs. The Perceived Motivational Climate in Sport Questionnaire (Newton, Duda, & Yin, 2000) and Caring Climate Scale (Newton et al., 2007) are two measures that use athlete perceptions of the coach-created psychosocial climate. Another source of data triangulation would be coach behavioral observation measures, such as the Coaching Behavior Assessment System (CBAS; Smith, Smoll, & Hunt, 1977). The CBAS uses observational methods to count the frequency of 12 behaviors related to successful coaching. The limitation of the CBAS is that the behaviors are coded using a behaviorist psychology paradigm, falling into predetermined categories of reinforcement, punishment, or general communication. Reducing a complicated behavior identified in this evaluation (e.g., coach using video analysis to help a swimmer learn proper stroke technique) so that it could be coded into one of 12 predetermined behaviors would not adequately capture the complexity of this behavior. Nonetheless, use of the CBAS would add an element of data triangulation to an experimental design that could help to capture the variance in coaching behaviors and their relationship to the psychosocial climate created by the 220 coaches. Two additional surveys that measure athlete perceptions of coaching behaviors, the Coaching Behavior Questionnaire (CBQ, Williams et al., 2003) and the Coaching Behaviour Scale for Sport (Côté, et al., 1999), could also provide viable avenues for behavioral analysis. Contributions to Coaching Science Knowledge Previous research has examined coach perceptions and use of formal educational experiences, particularly through the use of formal coach certification programs in the Sport Canada system, where coach education is a nationwide requirement for all sports. Significant debate on the merits of different forms of coach learning has shown that coaches generally prefer to learn through less-formal and more coach-driven means, e.g., speaking with other coaches (Lemyre, Trudel, Durand-Bush, 2007). However, research has also indicated that the mindset of the coach towards learning matters more than the type of educational experience provided (Werthner & Trudel, 2006). Coaches that want to learn more gain insight from both formal, curriculum-driven approaches (required coach education courses), as well as from informal, coach-driven approaches (other coaches, internet searches). The findings from this evaluation support the notion of the coach being the primary variable in the learning equation. Twenty of 21 coaches in this sample reported learning something useful from the Foundations course, with 17 coaches providing evidence of how they implemented a useful topic into their everyday coaching practice. The presentation of coaching knowledge in a contextualized format appeared to be particularly successful, as coaches were more likely to implement content that was centered around the performance of specific coaching duties. The use of coaching scenarios also appeared to be particularly effective for delivering this knowledge. Knowledge that was less contextualized, or did not have a practical plan for application and implementation into coaching practice (e.g., physiological training zones, mental skills), was not as useful for coaches. Thus, it 221 appears that formal coach education delivered online is more useful and more likely to be applied and implemented when the instructional design puts this knowledge into context for the learner. Placing knowledge into context can often be a time-consuming process for the instructional designer. Creating scenarios around which knowledge can be delivered takes significant thought and planning, and often breaks knowledge out of its traditional disciplines. For instance, instead of delivering knowledge about biomechanics and psychological development as separate lessons, delivering this information simultaneously in response to a scenario (e.g., “what is the best way to provide stroke instruction for novice swimmers?”) helps a coach understand the appropriate ways to teach stroke drills and proper technique to swimmers of different ages. If knowledge is presented in an interdisciplinary, scenario-based format, it appears to have a more profound effect on the learner. Therefore, despite the significant time that must be invested into creating instruction based around inter-disciplinary scenarios, this approach has particular value in helping coaches to contextualize difficult content knowledge. Although this approach appears to be particularly useful in a course designed for meeting coach education requirements, it remains to be seen if a scenario-based learning approach would be the appropriate method to educate coaches in credit-based (i.e., university-level) coach education courses, where there is often a greater expectation for higher level content knowledge, and where small class sizes allow a greater opportunity for interaction between an expert coach educator and coaches in the course. A recent study demonstrated that while showing promise as a viable instruction method, online, problem-based learning had limitations in its ability to stimulate deep reflection (Driska & Gould, 2014). Another consideration is the degree of generalizability that is learned in scenario-based education. Although the scenario-based approach appears to have a 222 stronger initial effect on coaches using information, it remains to be seen for the coaches in this group if they have been able to deepen the complexity of that knowledge, and generalize it to related situations. Thus, there might be a limitation on the degree to which a scenario can promote learning of advanced topics. However, for an online course with the purpose of delivering basic competencies, the scenario-based approach shows promise. Skepticism over the efficacy of online coach education programs has brought a spate of interest in evaluating coach education programs, whereas before the proliferation of online coach education programs, there appeared to be little appetite for evaluation of coach education. Instead, research on coach education appeared to focus on the debate between the different approaches to coach education (e.g., reflection, formal education, communities of practice), as opposed to the more practical evaluation question of do these programs make a difference? Thus, there is a lack of knowledge on the efficacy of widespread coach education programs, and even less knowledge with regards to online coach education programs (McCullick et al., 2009). This evaluation contributes meaningful evidence that an online coach education program can make a difference in coach knowledge, attitudes, and behaviors. Noting the call for researchers to get their hands dirty examining the efficacy of existing coach education programs (McCullick et al., 2009), there should be some consideration given to the paradigm of inquiry. Although some established names in the field of coach education advocate for the paradigm of logical positivism (Smith & Smoll, 2014), this approach may not consider the myriad stakeholders involved in existing coach education programs, which may or may not be guided by evidence-based intervention theories. This work demonstrates the value and efficacy for using a program evaluation paradigm to guide inquiry into an existing coach education program. Embracing a program evaluation paradigm had two clear benefits. First, the 223 program evaluation paradigm was uniquely suited to examine existing programs in their native context (Rossi et al., 2004). Second, a program evaluation framework, notably the utilizationfocused framework (Patton, 2011), facilitated stakeholder involvement in the inquiry process. Increased stakeholder involvement in the inquiry process is more likely to keep the inquiry focused on answering questions that arise from the field, rather than questions that arise from the literature. The debate between advocates of reflection and formal education represents inquiry with immense value for scholars of coach education, but may have little practical value to the program administrators in sport governing bodies. Program administrators in this evaluation clearly demonstrated a need to understand practical questions, such as whether or not the coach education program improves coaching performance, or improves the sport experience for athletes and their families. The challenge for the sport science researcher or evaluator is maintaining an equal footing between the contributions to the scientific literature and the specific inquiry questions raised by the stakeholders in a coach education program. These two priorities are not mutually exclusive, but Patton (2011) warns about the tendency for a social science researcher to focus on interesting research findings from an evaluation (e.g., comparative studies of LTAD implementation, diffusion of ideas). Patton (2014) recently expounded on this tendency, discussing how research on heuristics and biases might inform the utilization-focused evaluator. A heuristic is a cognitive process by which a person filters an immense amount of information in order to make a decision. Patton suggests that a utilization heuristic can be an essential tool for the utilization-focused evaluator to develop and employ during the evaluation process: Basically, the utilization heuristic for managing situational complexity in utilizationfocused evaluation is to stay focused on use. For every issue that surfaces in evaluation negotiations, for every design decision, for every budget allocation, and for every choice 224 among alternatives, keep asking, “How will this affect use in this situation?” (Patton, 2014, p. 240; author’s emphasis) With regards to future inquiries on coaching education and its effects, it is important for researchers and/or evaluators to consider such a heuristic in an effort to keep the purpose of the evaluation (or study) in constant focus. A reflexivity statement, constantly read and revised, is one way for a researcher to revisit the purpose of the inquiry, which can help the focus to remain on questions of interest to program administrators and stakeholders. Another approach would be to have one member of an evaluation team focused on the purpose of the evaluation, while another member of the team would be allowed to think more about productive tangents that result from the inquiry. Previous research on coach education has focused largely on the preferred methods of education, and smaller studies have examined the efficacy of brief interventions on coach efficacy. Studies have also examined the effects of coach education on psychosocial outcomes for athletes (e.g., athletes’ perceptions of the motivational climate). These studies have been heavily outcome-focused, in that they have sought to demonstrate the efficacy of a particular coach education intervention on outcomes for athletes, rather than being process-focused, i.e., examining the process by which the coach changed as a result of the educational experience. Such bias towards outcomes might be rooted in a need to demonstrate the effectiveness of an intervention designed by a researcher, rather than to describe the process of learning and change that the coach undergoes. However, a major limitation of researcher-driven intervention studies is that they may be unsustainable in the sport climate after the researcher ends their involvement. Another limitation is that we have little knowledge from intervention studies that describes the process of change undergone by coaches as a result of the intervention, thus limiting 225 generalizability of an intervention. Using theoretical guidance to understand the process by which coaches are affected by a coach education experience was one aim of this evaluation. The evaluator aimed to guide this evaluation by using both local theory (i.e., the key program decision-makers’ collective understanding of how the Foundations of Coaching program worked) and scientific theory about the nature of behavior adoption (i.e., theory of planned behavior). This evaluation contributes a new approach to coach education, in that it examined the adoption of attitudes and behaviors as a result of the Foundations course, as these were hypothesized to have relationships with the content presented by the course. Specifically, this evaluation has shown that the adoption and implementation of long-term athlete development principles involves comprehension of knowledge and principles, attitudinal agreement with the conceptual approach of long-term athlete development (e.g., valuing fun and age-appropriate training), as well as the adoption of specific coaching behaviors (e.g., writing age-appropriate workouts, incorporating fun into training). Within the program evaluation literature, this approach is often referred to as the knowledge, attitude, and skills framework (Patton, 2011). This approach conceives of learning as more than a simple cognitive process of the accumulation of facts and principles. Instead, learning is a multifaceted approach, in that attitudinal changes often involve an affective component, and that the development of skills (i.e., discrete psychomotor processes) is given particular attention during the planning phase. Teaching skills involves a different set of educational constraints than teaching abstract content knowledge that is to be generalized by the learner to the various situations they encounter in their work. Best practice in the program planning literature now advocates that program designers identify learning objectives using the knowledge-attitudes-skills framework. This evaluation demonstrates that describing program 226 effects in terms of changes to knowledge, skills (behaviors), and attitudes provides a greater understanding of how the Foundations course impacted coaches. Future evaluations and studies of coach education programs should embrace a framework similar to the knowledge/skills/attitudes framework in order to provide a greater understanding of how coaches change as a result of the program. This evaluation has made a minor but not insignificant contribution to the exploration of gender differences with regard to how coaches learn, encode, and apply coaching knowledge. Several coaches in this evaluation reported a knowledge acquisition strategy akin to the strategy of bricolage (Turkle & Papert, 1992), which posits that knowledge is constructed in such a way that a person can use it to resolve dilemmas that arise in their environment in accordance to the constraints presented by the environment, as opposed to implementing a top-down ordered framework of knowledge to solve all of the problems that might arise. Turkle and Papert (1992) suggest that the predisposition to a bricolage approach to knowledge may reside within the individual, noting in their research that women were more likely (3-to-1) to favor the bricolage approach, whereas men were more likely to prefer a top-down ordered framework of knowledge by a similar magnitude. The evaluator engaged in a preliminary coding of interview data related to the concept of bricolage in the midst of this evaluation. Because this evidence was generated after the interviews with participant coaches, it is not presented as definitive evidence in the results chapter. However, this evaluation gives preliminary evidence to the existence of these knowledge organization and construction preferences between male and female coaches. The Foundations course, which operates with an implicit theory that there is not one correct way to coach swimmers, but rather a range of more-acceptable versus less-acceptable approaches, may be more beneficial for coaches who are predisposed to the bricolage approach to knowledge 227 construction. In fact, many online courses operate in this manner, because they often lack the affective component of a compelling authority figure (in the form of a classroom teacher) imposing their biases on the best way to coach. Specifically, this might mean that female users, being more likely to employ the bricolage approach to knowledge construction, may be more receptive to the online approach. This might also partially explain the significant differences in Foundations 201 test scores between male and female coaches in the 23-28 year-old age category. Four notable female coaches provided extensive evidence that the Foundations course was useful in that it presented a range of solutions (e.g., drills, workouts, parent communication strategies); four male coaches spoke extensively in favor of aspects of the course that provided knowledge that could be applied more systematically. The male coaches also spoke about how they valued learning from mentors, while female coaches did not speak extensively about the value of mentorship. This might suggest that female coaches would value an online approach to learning more than male coaches, while male coaches might prefer learning through apprenticeship or mentorship. These gender differences were not hard and fast, but the preliminary existence of differences between men and women in how they use the course warrants further investigation into the preferred coach learning styles between new female and male coaches. A recent radio feature by technology scholar Alexis Madrigal (2014) illustrated the difference in social media preferences between men and women. The social media platform Pinterest, which allows users to upload, share, and tag (code) pictures organized around topics of user interest (e.g., cooking, wedding planning, vacations), rose to prominence primarily amongst female users. Users can search for a topic of interest (e.g., data visualization) and see a range of high-quality data visualizations that have been uploaded and shared by Pinterest users. This 228 same feature can be accomplished by Google image search, but the major difference is that Google uses a top-down computer algorithm to search and display images based on userdetermined keywords. Madrigal commented that, “if Google is great when you know exactly what you want, Pinterest can help you figure out what you want” (Madrigal, 2014). Technological implementations have a way of resembling the human cognitive structures of the people who have designed them. Google’s inherent logic is that there is an algorithm that can answer any question, whereas the inherent logic of Pinterest is that its flexibility, given enough user input, can generate a range of solutions from which an acceptable solution can be deployed. Returning to the concept of coaching knowledge, perhaps users who are predisposed to the bricolage framework prefer a Pinterest approach to coaching techniques, whereas users who embrace the top-down approach prefer the Google-has-the-answer approach to coaching. This might suggest the existence of significant gender differences in coaching epistemology, or even differences in coaching epistemology that exists independently of gender. Future organization of coaching knowledge through websites that enable the searching and sharing of knowledge useful to swimming coaches should take these differences in knowledge organization into account. Finally, this dissertation advances the science of program evaluation as a viable inquiry method for both sport psychology researchers and a viable process for sport governing bodies. A recent systematic review of the literature on program evaluation in sport contexts (Driska, 2014) revealed a difference in the practices employed by sport psychology researchers in articles described as evaluations, and the practice of evaluation as conceptualized by advocates for program evaluation (Patton, 2011; American Evaluation Association, 2004). Most notable was the lack of guiding evaluation theoretical frameworks (e.g., utilization-focused evaluation, participatory evaluation) in the evaluation of the effects of various sport-based programs (e.g., 229 youth development programs, mental skills training, LTAD implementation). Published articles where the authors had employed a theoretical framework to guide the evaluation were more likely to create and use a logical framework that described the program in order to guide the rest of the inquiry. The logical framework has value for showing the process of the program and how it affects change in the participant, but it also is significant in its ability to reflect the mechanism of the program back to the program stakeholders for their validation. Stakeholder involvement is one of the defining features that sets program evaluation apart from social science research. The advantage of stakeholder involvement is that it often increases the social science researcher’s ability to leverage change. For instance, in an evaluation of implementation of a long-term athlete development policy by alpine ski coaches (Black & Holt, 2009), the stakeholders were engaged only for the purposes of delineating the problem of failed LTAD implementation. The stakeholders were not engaged in the problem-solving process of understanding how LTAD might be better implemented. This is not meant to fault the inquiry methods employed by this study, however, because it was using a research paradigm (rather than a program evaluation paradigm), the findings stopped short of their ability to affect change. Program evaluation offers a useful paradigm to connect two parties who could mutually benefit from partnership. Sport governing bodies face a range of dilemmas, from producing evidence-based coaching and administration knowledge for coaches and teams, to the same types of complex organizational problems that face large corporations and government agencies. They have a need for the evaluative thinking and capacity building that program evaluators can help them to develop. Sport psychology researchers in the United States have not formed extensive partnerships with sport governing bodies as have their colleagues in countries like Canada, Great Britain, Australia, and New Zealand. Sport governing bodies offer a host of opportunities for 230 sport psychology researchers to get more involved in studying and evaluating the processes and outcomes of sport involvement for millions of American youth. This partnership could be forged through program evaluations, where the aim is to bring the sport science researcher in closer partnership with the governing body, with the desired aim of evaluating and potentially resolving pressing issues being faced by the sport. Evaluations also produce a multitude of data, which are essential currency for researchers in the modern university setting. Both parties have the potential to benefit when program evaluation is a means for connection between sport psychology researcher and sport governing body. Future Coaching Science Research Directions There are several future research directions stemming from this evaluation that could be beneficial to the coaching science research as a whole. The first direction would be to explore the relationship between program outputs (e.g., application exercises, implementation plans, developed skills) and short-term outcomes, long-term outcomes, and impacts. Outputs from the Foundations course were notably limited, and a recommendation from this evaluation was to explore outputs that could help coaches to implement the course knowledge into everyday practice. However, little research (Driska & Gould, 2014) has explored how program outputs in coach education settings link to program outcomes for the coach and their program. Research in this area could demonstrate how a program output (e.g., writing an imagery script) might link to a short-term outcome (e.g., the coach using imagery to enhance athlete performance), a longterm outcome (e.g., athletes adopting imagery use to enhance performance and manage sportrelated anxiety), and even program impacts (e.g., more enjoyable sport experiences for athletes, athlete retention, improved health-related outcomes). A second direction would be further exploration of the relationships between knowledge, 231 attitudes, and skills/behaviors, especially as they relate to the implementation of evidence-based coaching practices. Previous research has examined this relationship with regards to concussion awareness and reporting in coaches (Glang et al., 2010) and athletes (Register-Mihalik et al., 2013). These studies originated within health behavior research circles, where the relationship between knowledge, attitudes, and skills/behaviors has been well established (Godin & Kok, 1996). This research and program design framework has not made much headway in the coaching science literature. However, if evidence-based coaching is to take hold amongst sport coaches, a stronger understanding of how scientific knowledge bridges the research-practice divide is essential. Future studies might take a longitudinal approach to explore how coaches respond to an initial educational experience (e.g., Foundations of Coaching course) by assessing changes in knowledge, attitudes, and behavioral intentions, and then assessing how they respond to follow-up efforts to support implementation and deepen their understanding of the topic. Behavioral observations and triangulation of effects through the collection of data from athletes, parents, and colleagues could provide a concrete picture of how evidence-based coaching practices are adopted and proliferate throughout a sport. A third research direction would be greater exploration of gender differences and the use of bricolage as a form of learning and knowledge construction. Much general coaching research has been conducted with male coaches, as they tend to dominate the coaching profession at the American intercollegiate level (77% of all collegiate coaches are men; Acosta & Carpenter, 2014), however, in the population of coaches for this evaluation, 55.8% of the coaches were female. Previous research examining the advancement opportunities for female coaches has identified social norms (i.e., gender-role expectations), political realities (i.e., athletic department cultures), and economic opportunity (i.e., lack of well-paying coaching jobs) as unique barriers 232 to the advancement of female coaches (Cunningham & Sagas, 2005; Knoppers et al., 1991). However, one potential cause of differing career outcomes for male and female coaches could be the differing ways in which male and female coaches learn, construct and apply coaching knowledge. Following from the ideas of Turkle and Papert (1992), female coaches may construct coaching knowledge differently from male coaches. As a few notable female coaches reported in this evaluation, the Foundations course provided them with ideas that they could implement to resolve coaching dilemmas as they arise. They did not discuss how the Foundations course provided them with a top-down solution that could resolve all of their coaching problems. Knowing more about how female coaches work to affect change in teams and individual athlete behavior, and noting that it might be different from the ways in which male coaches work to affect these changes, could inform the quality of knowledge around the topic of culture change when a new coach takes over an existing program. Media accounts of culture change when new male head coaches take over sport programs often detail how the new coach is bringing a new system to the team; players that don’t fit the new system are often traded or cut from the program if they cannot adapt. Such changes are often applauded when these male coaches take over teams that have experienced turmoil stemming from a lack of leadership; thus, top-down culture changes appear to be valued in the sport world. Female coaches, if they have constructed coaching knowledge following the tenets of bricolage, may be less likely to implement top-down culture change, but instead, be more likely to work within the constraints provided by the existing players in the program. Future research might examine the differences in managing styles amongst male and female head coaches using coaching knowledge organization as a conceptual framework. This research could examine the ways in which male and female coaches resolve a range of coaching dilemmas to see if marked differences exist. 233 The one-size-fits-all management approach has fallen out of fashion in corporate environments, and yet often persists in collegiate athletic departments. This lack of flexibility may stand as an unrealized barrier to the advancement of female coaches. Such studies might help to provide a more concrete understanding of the ways in which male and female coaches work, which would be helpful to athletic administrators. A fourth avenue for future research centers around the concept of long-term athlete development. This evaluation provided evidence that when presented with content knowledge about LTAD, and content designed to shape coach attitudes about the importance of LTAD, coaches were receptive to the philosophy of LTAD and implemented some of the principles into their everyday practice. Despite this encouraging evidence of LTAD adoption, it represents coach perceptions and self-reported behaviors, which do not carry the evidential weight of behavioral observations and data triangulation from athletes and parents that provide a comprehensive picture of how thoroughly the LTAD principles have been implemented. Implementation evaluation could provide a complete picture of how this concept has been integrated into existing coaching practices, specifically within the sport of swimming, but also within other sports. A multiple-case analysis design across a range of representative teams, incorporating a range of data collection and triangulation methods (e.g., athlete and parent surveys and interviews, coaching behavioral observations), could be used to conduct an implementation evaluation. With regard to the proliferation of the LTAD philosophy, future research could examine the most effective ways to disseminate new ideas and policy. This evaluation has demonstrated preliminary success in the use of online education for new coach members as a means of introducing LTAD principles into the sport. Different sport governing bodies (United States 234 Tennis Association, USA Hockey, Sport Canada) have used top-down policy implementation strategies with varying levels of success. As their strategies have differed, the lessons-learned have been different (and in some ways, highly contextualized), however, there would be value in research that both described the implementation strategies and compared the relative success of these approaches. Such research could be beneficial to other sport governing bodies that have yet to adopt an LTAD implementation strategy. This research could also examine the sport outcomes for athletes in each of these sports, such as sport retention, psychosocial development (e.g., mental skill development, learning and transfer of life skills), and health outcomes (e.g., physical activity levels, rates of injury and burnout). 235 APPENDICES 236 Appendix A: Original Proposed Project Timeline Table 8 - Original Proposed Project Timeline Time Stakeholder engagements Dec ‘13 Phase I Prelim discussion with Program and Evaluator readiness and competency Services Director exercises selection of stakeholders and primary intended users understanding the evaluation context preparing client for evaluation risk assessment Jan 6-10, ‘14 Phase II Evaluator independent work MSU obligations IRB exempt application and approval Successfully propose dissertation, and make needed revisions Monday (all PIUs) Evaluation preparation & risk assessment Face-to-face Clarify evaluation purposes meetingsFocus priority evaluation questions Colo. Springs Goals clarification exercise Thursday (all PIUs) Theory of change work - goals clarification revisited, causal mechanisms, model decomposition, examining barriers and exceptions Friday (all PIUs)Negotiate inquiry into theory of change- what would coaches say about this theory of change; what else would we want to know from coaches; Simulate potential findings; Negotiate methods Tuesday – Wednesday (evaluator only) Interim analysis of goals clarification exercise, prepare presentation for PIUs, adapt new exercises for remaining meetings Jan-Feb ‘14 Phase III Qualitative inquiry/analysis Solicit interviews (12-20 coaches) Conduct interviews Transcribe Prelim analysis meeting with triangulation after 4-6 interviews Continued analysis Prepare interim report Engage with research assistant for assistance with transcription and analysis (arrange for pay through MSU) Share interim findings with dissertation committee Need to get from USA-Swimming Records from coaches who took Foundations 101 in Fall 2013 – need test scores, demographics (age, race, gender, LSC), contact info Report back to committee with an update, proposed changes into the inquiry, revisions needed MSU IRB revisions as needed Evaluator regroups and interprets any issues from Phase II Mar-Apr ‘14 Phase III Share interim findings with PIUs Virtual meeting(s) Discuss process use of interim findings Revise theory of change as needed Discuss methods of quantitative inquiry Quantitative inquiry/analysis Design survey- novel items are validated with PIU input; IRB approval; pilot survey (??); revise as needed Select and solicit sample of 100-150 coaches Administer survey Analysis of results for variable relationships IRB approval of survey instrument May-Aug ‘14 Phase IV Present findings with PIUs Face-to-face, Colorado Springs Follow-up meetings (Virtual) promote use negotiate possibilities for ongoing evaluation efforts monitoring strategies Prepare findings report Analysis, interpretations, judgments, recommendations Consider approach of written report vs. a digital-only presentation Write-up dissertation Choose defense date Defend dissertation Close IRB as needed *Note that potential quantitative studies were suggested as future directions at time of proposal. 237 Appendix B: Assessment of Evaluator Competencies A. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries about whatever is being evaluated B. Competence: Evaluators provide competent performance to stakeholders. 1. Evaluators should possess (or ensure that the evaluation team possesses) the education, abilities, skills and experience appropriate to undertake the tasks proposed in the evaluation. 2. To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence. Cultural competence would be reflected in evaluators seeking awareness of their own culturally-based assumptions, their understanding of the worldviews of culturally-different participants and stakeholders in the evaluation, and the use of appropriate evaluation strategies and skills in working with culturally different groups. Diversity may be in terms of race, ethnicity, gender, religion, socioeconomics, or other factors pertinent to the evaluation context. 3. Evaluators should practice within the limits of their professional training and competence, and should decline to conduct evaluations that fall substantially outside those limits. When declining the commission or request is not feasible or appropriate, evaluators should make clear any significant limitations on the evaluation that might result. Evaluators should make every effort to gain the competence directly or through the assistance of others who possess the required expertise. 4. Evaluators should continually seek to maintain and improve their competencies, in order to provide the highest level of performance in their evaluations. This continuing 238 professional development might include formal coursework and workshops, selfstudy, evaluations of one's own practice, and working with other evaluators to learn from their skills and expertise. C. Integrity/Honesty: Evaluators ensure the honesty and integrity of the entire evaluation process. D. Respect for People: Evaluators respect the security, dignity and self-worth of the respondents, program participants, clients, and other stakeholders with whom they interact. E. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of interests and values that may be related to the general and public welfare. 239 Appendix C: Evaluator Statement of Reflexivity Written at the outset of the evaluation. There are two life and career experiences that will create sources of competing bias. First, the evaluator was formerly a coach, prior to beginning his doctoral studies. As a coach, he was largely self-taught at first, and used a variety of coach-driven learning methods, such as internet searches, independent readings, attending clinics, speaking with other coaches, and writing his own philosophy and beliefs about training. The second source of bias is that now the evaluator is approaching the program with the belief that, on paper, the program is structured soundly and serving a good purpose. This bias needs to be addressed – it is the bias of good intentions. Just because a program has good intentions and is designed in such a way that makes sense in terms of educational theory, it does not mean that it presents important and useful information to coaches. To address the tendency to reward good intentions, the evaluator will continually play “devil’s advocate” to tease out sources of meaning that might show where the program has good intentions but fails to deliver useful information. Written at the close of the evaluation. With regard to the first potential bias, the evaluator’s coaching knowledge was extremely helpful in speaking with coaches. It helped to facilitate an open exchange of ideas because of the common understanding of swimming concepts. Having been somewhat skeptical of coach education efforts as a self-taught coach, the evaluator was able to develop a greater understanding of how coach education is perceived differently by different coaches in different contexts. Some coaches are busy and can’t make time to devote to attending clinics or reading Swimming Even Faster cover to cover, nor do some have that level of interest; however, that does not mean they are bad coaches. In many ways, it was a helpful way to learn about the diversity of coaching experiences that exist in one sport. 240 With regard to the bias of good intentions, this was overcome by the strong rapport the evaluator was able to build with key program decision-makers. The facilitative style required for utilization-focused evaluation was also a style that he had learned through close work with his doctoral program advisor, and this style was instrumental in accelerating the process of trustbuilding with key program decision-makers. During meetings, key program decision-makers were very forthcoming with their views, and were able to talk about potential weaknesses in the program, because there was an eye to its eventual improvement. During the initial focus group meeting with key program decision-makers, the design of specific exercises, such as “picture the worst program you have observed,” or “picture your biggest skeptics sitting in the room,” allowed the evaluator to assume a critical role, but the criticism was located within another entity (e.g., program skeptics), thus allowing the evaluator to act more like a referee than a critical adversary. This allowed critical thought to flourish without the threat of premature judgment. 241 Appendix D: Solicitation E-mail Subject: Interview Request - USA Swimming Foundations of Coaching Evaluation E-mail body: Hi Coach, I am hoping that you can help me with a study that I am conducting at Michigan State University. I am trying to learn more about the effectiveness of USA Swimming’s Foundations of Coaching program, and to identify areas for its improvement. I have selected you as a possible participant in this study because you have recently completed the current online version of the Foundations of Coaching program. Participation in the study involves a phone interview, which would last approximately 45 minutes. Participation in this study is completely voluntary. Declining to participate will not affect your standing with USA Swimming, and your contributions to the study would remain confidential. If you are interested in participating, please reply to this e-mail. I will then contact you to make the necessary arrangements. If you’re interested, but have questions before you agree to participate, I would be happy to discuss them by e-mail (e-mail address withheld) or phone (phone number withheld). Thanks for your time. Sincerely, Andy Driska Participant Consent Form (provides more information about participating in the study): [URL removed] 242 Appendix E: Coach Interview Guide Coach Experiences with USA Swimming’s Foundations of Coaching Program Interview Guide Michigan State University Department of Kinesiology Key 1. Main idea of questions a. Probe question or follow-up question Do you have any questions about this study before we begin? 1. Sport background & demographics a. How did you get into coaching? b. Have you coached before becoming a coach member of USA Swimming? c. How would you describe your race or ethnicity? d. Gender? e. Age? f. Other work experience? g. Tell me about the club where you are currently coaching. h. What level of athletes do you currently coach? i. What is your career plan as a coach? Are you full time, or part-time? How long will you coach? 2. Learning to be a coach a. Can you tell me a little about how you have learned how to be a swimming coach? b. Did you learn from your experience as an athlete in the sport? Do you have a background in the sport? 3. Learning experience in the online course a. What were your impressions of the Foundations of Coaching online course? b. Overall, did you like it? Not like it? You’re not going to hurt my feelings… c. Tell me a little more about the technology in the course… any issues for you? d. How did you feel about how the lessons were presented? Did this add, or take away from the course? What might be done differently? 4. Application of learning experience a. Is there a topic from the Foundations course that really stood out for you? b. Do you apply something from the course frequently? c. How have you applied it? Could you tell me a story… 5. Difficult topics from online course 243 a. Is there a topic from the Foundations course that was just way over your head… that you needed more help with? b. Are there certain things that just have to be learned through experience or mentorship? c. Do you have any experience with mentorship, i.e., have you sought a mentor? Have you been mentored? 6. Coaching behaviors a. What coaching behaviors have you adopted as a result of the course? Leave open-ended, but hit the following checklist if the answers don’t come naturally. b. Using video and underwater video c. Writing age-appropriate workouts d. Proper incorporation of fun into workouts e. Being a role-model f. Developing the athlete as a person g. Goal-setting 7. Professional mindset a. Has this course influenced your mindset, the way you think about yourself as a professional? b. Are you more aware of your role(s) as a coach, the importance of how you do your job? c. Does it influence the way that you interact with athletes, parents? d. What do you make of continual improvement as a coach – is that a priority for you, how do you go about getting better as a coach? e. Can you tell me specifically about what you have done to improve as a coach? 8. Barriers a. I’m interested to learn a little more about your thoughts on coach education… does it have value for you? Do you feel that it might stifle coach innovation? b. Important others… is coach education and professional development a priority at your club? 9. Improvement in courses and service delivery a. What from this course might work better for you, could be more helpful? Any ideas on how we might deliver those services? 10. Further knowledge a. As a coach, what do you want to know more about? b. What are the big questions for you? c. What’s missing from the whole coach education experience for you? d. What’s your go-to source (or process) when you have a question? e. Did this course inspire you to constantly seek education as a coach? 244 REFERENCES 245 REFERENCES Acosta, R. V., & Carpenter, L. J. (2014). Women in intercollegiate sport: A longitudinal, national study: 37-year update. Retrieved from www.acostacarpenter.org Adie, J. W., Duda., J. L., & Ntoumanis, N. (2012). Perceived coach-autonomy support, basic need satisfaction and the well- and ill-being of elite youth soccer players: A longitudinal investigation. Psychology of Sport and Exercise, 13, 51-59. Alkin, M. C., & Christie, C. A. (2004). An evaluation theory tree. In M. C. Alkin (ed.), Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211. American Evaluation Association. (2004). American Evaluation Association Guiding Principles For Evaluators. Retrieved November 2013, http://www.eval.org/p/cm/ld/fid=51. American Evlauation Association. (2013). The Program Evaluation Standards. Joint Committee for Standards in Educational Evaluation. Retrieved October 2013, http://www.eval.org/p/cm/ld/fid=103. Australian Sports Commission. (2013). Independent review of swimming. Retrieved from http://www.ausport.gov.au/__data/assets/pdf_file/0003/517512/Independent_Swimming_ Review.pdf Balyi, I., Way, R., & Higgs, C. (2013). Long-term athlete development. Champaign, IL: Human Kinetics. Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191-215. Benson, P. L. (1997). All kids are our kids: what communities must do to raise caring and responsible children and adolescents. San Francisco, CA: Jossey-Bass. Black, D. E., & Holt, N. L. (2009). Athlete development in ski racing: Perceptions of coaches and parents. International Journal of Sports Science and Coaching, 4(2), 245-260. Brinkerhoff, R. O. (1987). Achieving results from training. San Francisco, CA: Jossey-Bass. Buchanan, B. G., Davis, R., & Feigenbaum, E. A. (2006). Expert systems: A perspective from computer science. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 87-103). New York, NY: Cambridge University Press. Burton, D. (1989). Winning isn’t everything: Examining the impact of performance goals on collegiate swimmers’ cognitions and performance. The Sport Psychologist, 3, 34-47. 246 Cassidy, T., Potrac, P., & McKenzie, A. (2006). Evaluating and reflecting upon a coach education initiative: The CoDe of rugby. The Sport Psychologist, 20, 145–161. Coatsworth, J. D., & Conroy, D. E. (2009). The effects of autonomy-supportive coaching, need satisfaction, and self-perceptions on initiative and identity in youth swimmers. Developmental Psychology, 45(2), 320. Conroy, D. E., & Coatsworth, J. D. (2006). Coach training as a strategy for promoting youth social development. The Sport Psychologist, 20, 128– 144. Cousins, J. B., Donohue, J. J., & Bloom, G. A. (1996). Collaborative evaluation in North America: Evaluators’ self-reported opinions, practices, and consequences. Evaluation Practice, 17(3), 207-225. Côté, J., Yardley, J., Hay, J., Sedgwick, W., & Baker, J. R. (1999). An exploratory examination of the coaching behaviour scale for sport. Avante, 5(3), 82-92. Côté, J., Young, B., North, J., Duffy, P. (2007). Towards a definition of excellence in sport coaching. International Journal of Coaching Science, 1(1), 3-16. Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5-23. Culver, D., & Trudel, P. (2008). Clarifying the Concept of Communities of Practice in Sport. International Journal of Sports Science and Coaching, 3(1), 1–10. Cunningham, G. B., & Sagas, M. (2005). Access discrimination in intercollegiate athletics. Journal of Sport & Social Issues, 29(2), 148-163. Deci, E. L., & Ryan, R. M. (2000). The what and the why of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227-268. Driska, A. P. (2014). Advancing program evaluation for sport psychology researchers: a systematic review and narrative analysis. Manuscript in preparation. Driska, A. P., & Gould, D. R. (2014). Evaluating a problem-based group learning strategy for online, graduate-level coach education. Kinesiology Review, 3, 227-234. Driska, A. P., Gould, D. R., Pierce, S. W., & Cowburn, I. H. J. (2014). A mixed-methods analysis of mental skills acquisition, retention, and application from a two-week intensive wrestling camp. Manuscript in preparation. Erickson, K., Bruner, M., MacDonald, D., & Cote, J. (2008). Gaining insight into actual and preferred sources of coaching knowledge. International Journal of Sports Science and Coaching, 3(4), 527–538. Fullan, M. (1991). Curriculum change. In A. Lewy (Ed.), The international encyclopedia of curriculum (pp. 279-281). Oxford: Pergamon. 247 Gilbert, W., & Trudel, P. (1999). Framing the construction of coaching knowledge in experiential learning theory. Sociology of Sport Online, 1–6. Retrieved from http://physed.otago.ac.nz/sosol/v2i1/v2i1s2.htm Gilbert, W., & Trudel, P. (1999). An evaluation strategy for coach education programs. Journal of Sport Behavior, 22(2), 234-250. Glang, A., Koester, M. C., Beaver, S., Clay, J., & McLaughlin, K. (2010). Online training in sports concussion for youth sport coaches. International Journal of Sports Science and Coaching, 5(1), 1-12. Glesne, C. (2011). Becoming qualitative researchers: An introduction. Columbus, OH: Pearson. Godin, G., & Kok, G. (1996). The theory of planned behavior: a review of its applications to health-related behaviors. American Journal of Health Promotion, 11(2), 87-98. Goldman, T. (2013, August 27). USA Swimming Faces Lingering Doubts Over Sexual Abuse. National Public Radio, retrieved from http://www.npr.org/blogs/thetwoway/2013/08/27/216188040/usa-swimming-faces-lingering-doubts-over-sexual-abuse Gould, D., & Carson, S. (2008). Life skills development through sport: Current status and future directions. International Review of Sport and Exercise Psychology, 1(1), 58-78. Gould, D., Guinan, D., Greenleaf, C., & Chung, Y. (2002). A survey of US Olympic coaches: Variables perceived to have influenced athlete performances and coach effectiveness. The Sport Psychologist, 16(3), 229-250. Gould, D., Lauer, L., Driska, A., Moyer, M., Bean, E., Johnson, D. (2012). Using Coaching Education to Help Save Intercollegiate Olympic Sports: The NWCA Coaching Leadership Program Example. Symposium given at the National Coaching Conference, Indianapolis, IN. Iwasaki, S., & Fry, M. (2013). The efforts of sport psychology professionals to assist sport administrators in evaluating youth sport programs. The Sport Psychologist, 27, 360-371. Kaner, S., Lind, L., Toldi, C., Fisk, S., & Berger, D. (2007). Facilitator’s guide to participatory decision-making. San Francisco, CA: Jossey-Bass. Kline, T. J. B. (2005). Psychological Testing: A Practical Approach to Design and Evaluation. Thousand Oaks, CA: Sage. Knoppers, A., Meyer, B. B., Ewing, M., & Forrest, L. (1991). Opportunity and work behavior in college coaching. Journal of Sport & Social Issues, 15(1), 1-20. Lansberger, H. A. (1958). Hawthorne revisited: Management and the worker: its critics, and developments in human relations in industry. Ithaca, NY: Cornell University Press. 248 Lemyre, F., Trudel, P., & Durand-Bush, N. (2007). How youth-sport coaches learn to coach. The Sport Psychologist, 21, 191–209. Linnan, L., & Steckler, A. (2002). Process evaluation for public health interventions and research. In A. Steckler & L. Linnan (Eds.) Process evaluation for public health interventions and research (pp. 1-23). San Francisco, CA: Jossey-Bass. Locke, E. A., & Latham, G. P. (2002). Building a practically use theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705-717. Madrigal, A. (2014, July 30). Can Pinterest compete with Google’s search? National Public Radio, Fresh Air podcast. Retrieved from http://www.npr.org/templates/transcript/transcript.php?storyId=332317807 Mageau, G. A., & Vallerand, R. J. (2003). The coach–athlete relationship: A motivational model. Journal of Sports Science, 21(11), 883-904. McCullick, B., Schempp, P., Mason, I., Foo, C., Vickers, B., & Connolly, G. (2009). A scrutiny of the coaching education program scholarship since 1995. Quest, 61(3), 322-335. Miller, R.L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31, 390-399. Mishra, P., & Koehler, M. J. (2006). Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge. Teachers College Record, 108(6), 1017–1054. Moreno, F. J., Saavedra, J. M., Sabido, R., Luis, V., & Reina, R. (2006). Visual search strategies of experienced and nonexperienced swimming coaches. Perceptual and Motor Skills, 103(3), 861-872. Newton, M., Duda, J. L., & Yin, Z. (2000). Examination of the psychometric properties of the Perceived Motivational Climate in Sport Questionnaire-2 in a sample of female athletes. Journal of Sports Sciences, 18(4), 275-290. Newton, M., Fry, M., Watson, D., Gano-Overway, L., Kim, M., Magyar, M., & Guivernau, M. (2007). Psychometric properties of the caring climate scale in a physical activity setting. Revista de Psicología del Deporte, 16(1), 67-84. Nielsen, J. (2012). Thinking aloud: the #1 usability tool. Retrieved from http://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/ Norman, D. A. (2002, July/August). Emotion and design: Attractive things work better. Interactions Magazine, 36-42. Ntoumanis, N., & Biddle, S. J. H. (1999). A review of motivational climate in physical activity. Journal of Sports Sciences, 17(8), 643-665. Patton, M. Q. (2002). Qualitative Research & Evaluation Methods. Thousand Oaks, CA: Sage. 249 Patton, M. Q. (2008). Utilization-focused evaluation. Thousand Oaks, CA: Sage. Patton, M. Q. (2011). Essentials of utilization-focused evaluation. Thousand Oaks, CA: Sage. Patton, M. Q. (2013). Utilization-focused evaluation: Workshop. Presented at 2013 conference for the American Evaluation Association, Washington, D.C. Patton, M. Q. (2014). What brain sciences reveal about integrating theory and practice. American Journal of Evaluation, 35(2), 237-244. DOI: 10.1177/1098214013503700 Poister, T.H. (2004). Performance monitoring. In J.S. Wholey, H.P. Hatry, & K.E. Newcomer (Eds.), Handbook of practical program evaluation (2nd edition) (pp. 98-125). San Francisco: Jossey Bass. Register-Mihalik, J. K., Linnan, L. A., Marshall, S. W., McLeod, T. C. V., Mueller, F. O., & Guskiewicz, K. M. (2013). Using theory to understand high school aged athletes' intentions to report sport-related concussion: Implications for concussion education initiatives. Brain Injury, 27(7-8), 878-886. Reinboth, M., Duda, J. L., & Ntoumanis, N. (2004). Dimensions of coaching behavior, need satisfaction, and the psychological and physical welfare of young athletes. Motivation and Emotion, 28(3), 297-313. Rushall, B. (2013). Swimming energy training in the 21st century: The justification for radical changes. Swimming Science Bulletin (self-published), 39, 1–55. Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach. Thousand Oaks, CA: Sage. Schön, D. A. (1987). Educating the reflective practitioner: toward a new design for teaching and learning in the professions. San Francisco, CA: Jossey-Bass. Scriven, M. (1991). Evaluation Thesaurus. Newbury Park, CA: Sage. Smith, R. E., and Smoll, F. L. (2011). Cognitive–behavioral coach training: a translational approach to theory, research, and intervention. In J.K. Luiselli, D.D. Reed (eds.), Behavioral Sport Psychology. New York: Springer Science + Business Media. Smith, R. E., & Smoll, F. L. (2014). Logical positivism. In L. Nelson, R. Groom, & P. Potrac (Eds.), Research Methods in Sport Coaching (pp. 18-30). New York, NY: Routledge. Smith, R. E., Smoll, F. L., & Curtis, B. (1979). Coach Effectiveness Training: A cognitivebehavioral approach to enhancing relationship skills in youth sport coaches. Journal of Sport Psychology, 1(1), 59-75. Smith, R. E., Smoll, F. L., & Hunt, E. (1977). A system for the behavioral assessment of athletic coaches. Research Quarterly. American Alliance for Health, Physical Education and Recreation, 48(2), 401-407. 250 Smith, R. E., Zane, N. W. S., Smoll, F. L., & Coppel, D. B. (1983). Behavioral assessment in youth sports: Coaching behaviors and children’s attitudes. Medicine and Science in Sport and Exercise, 15, 208-214. Smoll, F. L., & Smith, R. E. (2002). Coaching behavior research and intervention in youth sports. In F. L. Smoll & R. E. Smith (Eds.), Children and Youth in Sport. Dubuque, IA: Kendall-Hunt. Turkle, S., & Papert, S. (1992). Epistemological pluralism and the revaluation of the concrete. Journal of Mathematical Behavior, 11(1), 3-33. Turnnidge, J., Côté, J., & Hancock, D. J. (2014). Positive youth development from sport to life: Explicit or implicit transfer? Quest, 66(2), 203-217. DOI: 10.1080/00336297.2013.867275 USA Swimming. (2012). Membership demographics. Retrieved from http://www.usaswimming.org USA Swimming. (2014). Age-group time standards. Retrieved from http://www.usaswimming.org/DesktopDefault.aspx?TabId=1488 USA Swimming. (2014). Coach membership requirements. Retrieved from http://www.usaswimming.org/DesktopDefault.aspx?TabId=1677 USA Swimming. (2014). Reference library (members-only access). Retrieved from http://www.usaswimming.org/DesktopDefault.aspx?TabId=2159 “USA Swimming To Review Sexual Misconduct Prevention Program.” Morning Edition. National Public Radio. Washington, D. C. 27 Aug. 2013. Radio. Transcript retrieved from http://n.pr/16I0drb. Werthner, P., & Trudel, P. (2006). A New Theoretical Perspective for Understanding How Coaches Learn to Coach. The Sport Psychologist, 20, 198–212. Wholey, J.S. (1996). Formative and summative evaluation: Related issues in performance measurement. Evaluation Practice, 17, 145-149. Wholey, J. S. (2004). Evaluability assessment. In J. S. Wholey, H. P. Hatry, K. E. Newcomer (eds.), Handbook of Practical Program Evaluation. San Francisco, CA: Jossey-Bass. Williams, J. M., Jerome, G. J., Kenow, L. J., Rogers, T., Sartain, T. A., & Darland, G. (2003). Factor structure of the coaching behavior questionnaire and its relationship to athlete variables. The Sport Psychologist, 17(1), 16-34. Witkin, B.R., & Altschuld, J.W. (1995). Planning and conducting needs assessments: A practical guide. Thousand Oaks, CA: Sage. W. K. Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI. 251 Wright, T., Trudel, P., & Culver, D. (2007). Learning how to coach: the different learning situations reported by youth ice hockey coaches. Physical Education & Sport Pedagogy, 12(2), 127–144. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage. 252