ADAPTIVE EXPERTISE IN POLICE WORK: THE ROLE OF IN-SERVICE TRAINING By Charles Scheer A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Criminal Justice – Doctor of Philosophy 2014 ABSTRACT ADAPTIVE EXPERTISE IN POLICE WORK: THE ROLE OF IN-SERVICE TRAINING By Charles Scheer The task of strengthening police training regimens to match expanding duties and technologies, while simultaneously budgeting for such training, has become more urgent in recent years (Wilson et al., 2010a). To serve contemporary community and organizational needs, there is an increased need to make police officers more adaptable to unforeseen and rapidly-unfolding situations, and to work with increasingly diverse populations. Additionally, training for expert performance has been lacking in police organizations due to organizational inattention to the importance of such training, as well as its increasing cost (Glenn et al., 2005). Industrial and organizational psychologists have touted the need for training for adaptive expertise, or a skill encompassing both skills sets in a singular theoretical perspective, yet training tasks and programs to develop this skill have been the subject of debate. Despite promising results in similar fields, these training programs have not been attempted in police workforce situations. Moreover, evaluations of training programs designed to target skills for adaptive expertise are few. In this case study, a training regimen targeting adaptive expertise skills was initiated, developed, delivered, and evaluated in conjunction with a mid-size police agency in a large Midwestern town over the course of one year. The project encompassed a variety of critical tasks related to both the training delivery and evaluation of training for adaptive expertise. Project goals were developed by the host agency with researcher assistance, and were derived from prerequisites for adaptive expertise outlined in taxonomical literature. The training tasks and outcomes were created by training division staff with researcher oversight, and observation of training provided data used in evaluation. Additionally, a quasi-experiment was embedded in the training modules themselves, which took three months to deliver and targeted distinct skills related to the development of adaptive expertise. Finally, pre- and post-training interviews with training staff and officer participants provided qualitative data to contrast with quasiexperimental results. The process of evaluation is outlined and described in the document alongside research findings. Since the project consisted of a new research-practitioner partnership between university researchers and the police agency’s training division, the evaluation includes additional comment on the process by which this project developed. Because training of this nature normally occurs in police work during in-service training of experienced individuals, project findings are discussed in light of professional development objectives present in police organizations. Specifically, the impact of the training on the development of adaptability skills was seen as strong, but the training’s effect on the development of expert knowledge was not observed. This result has important implications not only for the continuing study of adaptive expertise as a singular entity, but for the renewed emphasis on skills development in police work for those persons seeking to further their careers. Project outcome measures are detailed and statistical analyses of the quasi-experimental portion are contrasted with qualitative findings to support these conclusions. Suggestions for future study, including theoretical exploration of adaptive expertise and the importance of this theory for police organizations, are included. Copyright by CHARLES SCHEER 2014 It is with respect and gratitude that this project is dedicated to the memory of PSO Eric E. Zapata of the Kalamazoo Department of Public Safety. I sincerely hope that in some way this project makes a positive contribution to the lives, careers, and well-being of police officers in his memory. v ACKNOWLEDGEMENTS I would like to thank the officers, patrol supervisors, chief, command staff, and support staff of the Kalamazoo Department of Public Safety for their support of this project. A special note of thanks goes to the staff of the Training Division for making this project possible. I would also like to thank Jeremy Wilson, Steven Chermak, Edward McGarrell, and Kevin Ford for their involvement, advice, direction, and encouragement as this project grew. This project would not have been possible without these people and many others, and I am eternally grateful. vi TABLE OF CONTENTS LIST OF TABLES…………………………………………………………………………xi LIST OF FIGURES………………………………………………………………………xiii INTRODUCTION…………………………………………………………………………1 The Structure of Police Training………………………………………………………...4 The Risks of Cutting In-Service Training……………………………………………….6 Study Research Agenda and Research Questions……………………………………….8 CHAPTER 1 THEORETICAL BACKGROUND………………………………………………………11 The State of Knowledge About Police Training………………………………………..11 Theories of the Changing Nature of Work and Its Relationship to Training…………...14 Theories of Organizational Behavior…………………………………………………...19 Theories of Training for Adaptive Expertise…………………………………………...28 Theories of the Role and Importance of In-Service Training…………………………...33 Summary: Theoretical Perspectives and the Examination of Adaptive Expertise……...38 CHAPTER 2 METHODOLOGY………………………………………………………………………..41 Background of the Study Site and Training History…………………………………...43 The Training Study: Structure and Content……………………………………………47 Research question one: The effect of training on performance…………………...48 Research questions/hypotheses associated with the pretest/posttest……...49 Design issues pertaining to experimental training evaluations……………55 Nonequivalent Dependent Variable (NEDV) analysis……………………56 Nonequivalent Dependent Variables in the model………………………..59 Training content…………………………………………………………...62 Module A: Firearms training………………………………………63 Module B: Active shooter planning……………………………….65 Module C: MILO simulations……………………………………..66 The use of three consecutive models explained…………………………...69 Evaluation…………………………………………………………………70 Research question two: How training targets organizational needs………………71 Qualitative procedure #1: Advisory panel meetings………………………75 Qualitative procedure #2: Officer and training staff interviews..…………76 Qualitative procedure #3: Observation of training………………………...78 Analysis of data…………………………………………………………………………81 Obtaining and analyzing data patterns……………………………………………..82 Matching patterns to theoretical criteria……………………………………………85 vii CHAPTER 3 PROJECT IMPLEMENTATION…………………………………………………………87 Initial Meeting with Kalamazoo DPS Training Staff…………………………………..89 Themes Related to Training Implementation…………………………………………...90 Theme 1: Training processes were informal and flexible………………………….91 Theme 2: Training content itself reflected a need for flexibility…………………..92 Theme 3: Deviations from planning are typical given these informal processes….93 Theme 4: In this environment, the training project that was implemented remained theoretically sound and methodologically rigorous……………………..93 Organizational Training Climate………………………………………………………..94 Observations of Training in Progress as Implemented…………………………………95 July 2012 Tactical Firearms training………………………………………………95 August 2012 Mobile Active Shooter Pre-Plans……………………………………97 September 2012 Subject Control Decision-Making training……………………..100 Posttraining Ride-along Interviews with Officers and Supervisors……………………103 Officers felt the training helped them develop adaptability skills………………...106 Officers perceived a further need for adaptability training……………………….107 Officers saw changes in department culture related to training…………………..107 Officers were fearful of depleted resources for training………………………….108 Officers were concerned with supervisors’ roles in the training process…………109 Other Structured Posttraining Interviews……………………………………………...110 Supervisors’ perspectives were supportive of training……………………………110 Officers viewed training as positive example of change………………………….111 CHAPTER 4 PROJECT OUTCOMES…………………………………………………………………114 Number of Cases and Nonresponse…………………………………………………....114 Descriptive Demographic Statistics...………………………………………………….116 Pretest and Posttest Data Analysis……………………………………………………..118 Findings by training-relevant items……………………………………………….120 Findings by elements within each construct………………………………………124 Training-irrelevant means differences…………………………………………….125 Training-relevant models: Ordinal logistic regression…………………………….127 Model parameters………………………………………………………………….128 Model results and interpretation…………………………………………………..139 Response outcome categories as compared to highest possible response...139 Demographic attributes……………………………………………………141 Experience variables……………………………………………………….143 Sensitivity check for non-response items………………………………………….147 Summary of pretest-posttest results………………………………………………..147 Open-Ended Survey Questions Disclosed Further Support for Training……………….149 Answers to Subordinate Questions Illustrate Support for Training……………………..151 CHAPTER 5 DISCUSSION……………………………………………………………………………..154 Conclusions Regarding the Training of Prerequisites for Adaptive Expertise…………155 viii Conclusions Regarding the Training Project Itself…………………………………….158 Conclusion 1: Adaptability and expertise may be two related goals……………...158 Conclusion 2: Training implementation affected outcomes………………………162 Conclusion 3: The training experience was enhanced by strategies for adaptive expertise…………………………………………………………………………...165 Conclusion 4a: The project represented a significant organizational leap………...167 Conclusion 4b: Organizational commitment to sustaining training improvements long-term training is unknown…………………………………………………….168 Conclusion 5: The training project exposed further training needs……………….170 Project Design Strengths……………………………………………………………….172 Project Design Limitations……………………………………………………………..176 Survey validity and reliability issues upon implementation………………………176 Internal validity threats……………………………………………………………177 Construct validity concerns………………………………………………………..180 Statistical conclusion validity concerns…………………………………………...182 External validity concerns…………………………………………………………182 Other concerns……………………………………………………………………..184 CHAPTER 6 IMPLICATIONS FOR FURTHER RESEARCH………………………………………..189 Implications for the Future Study of Adaptive Expertise……………………………...189 Adaptive expertise can be examined as two related phenomena…………………190 Adaptive expertise training can be articulated as related learning proficiencies….191 The project provides a sense of urgency to the need for adaptive expertise training…………………………………………………………………………….191 Implications for Police Training………………………………………………………..192 Formalizing the study of in-service training………………………………………194 Studying in-service training can lead to the development of organizational learning…………………………………………………………………………….195 Implications for Aligning Theories of Police Training With Police Practice…………..196 Developing training modules reflective of theoretical constructs…………………197 Demonstrating methods of increasing local control over police training………….197 Demonstrating benefits of increasing local control over in-service training………198 Implications for Research-Practitioner Partnerships……………………………………199 Tips for researchers studying police in-service training…………………………...200 Research-practitioner partnerships can be reiterated as an empirical strategy…….200 Clarifying the cliché of “evidence-based practice”………………………………..201 APPENDICES…………………………………………………………………………….203 Appendix 1. Sample Kalamazoo Department of Public Safety training schedule……..204 Appendix 2. Sample pretest measurement instrument for Module A…………………..205 Appendix 3. Sample posttest measurement instrument for Module A…………………207 Appendix 4. Observation protocol……………………………………………………...209 Appendix 5. Interview protocol………………………………………………………...210 Appendix 6. July lesson plans…………………………………………………………..211 Appendix 7. August lesson plans……………………………………………………….214 ix Appendix 8. September lesson plans…………………………………………………...215 Appendix 9. Kalamazoo Training Passport…………………………………………….218 Appendix 10. Sample training activity submitted……………………………………...219 Appendix 11. Sample training activity submitted……………………………………...221 Appendix 12. Sample training activity submitted……………………………………...223 REFERENCES…………………………………………………………………………....224 x LIST OF TABLES Table 2.1. Research hypotheses and variables…………………………………………………53 Table 2.2. Expansion of dependent variables into observable behaviors……………………...54 Table 2.3. Diagram of one-group pretest / posttest design…………………………………….58 Table 2.4. Inclusion of nonequivalent dependent variables for internal referencing strategy (IRS) …………………………………………………………………………………………..60 Table 2.5. Items and dependent variable measures for IRS collected in Modules A-C……… 68 Table 2.6. Insights gained from 2011 informal training needs assessment……………………72 Table 2.7. Project data collection and observation timeline…………………………………...81 Table 3.1. Emergent themes arising from project implementation process……………………91 Table 3.2. Posttraining interview sample characteristics……………………………………..104 Table 4.1. Demographics of officers completing all training modules (n=109)……………...119 Table 4.2. Paired sample t-test results, training-relevant and irrelevant items (n=109)………121 Table 4.3. Ordinal regression for demographic and control variables on performance on stressrelated posttest questions (N=109) …………………………………………….……………...131 Table 4.4. Ordinal regression for demographic and control variables on performance on problemsolving posttest questions (N=109) …………………...…………………….………………...132 Table 4.5. Ordinal regression for demographic and control variables on performance on crisisrelated posttest questions (N=109) ……………………………………….…………………...133 Table 4.6. Ordinal regression for demographic and control variables on performance on complexity-related posttest questions (N=109) ……………………….……………………....134 Table 4.7 Ordinal regression for demographic and control variables on performance on errormanagement posttest questions (N=109) ……………………………………………………...135 Table 4.8 Ordinal regression for demographic and control variables on performance on contingency-related posttest questions (N=109) ……………………………………………...136 xi Table 4.9. Summary of ordinal logistic regression models and statistically significant relationships (n=109)………..………………………………………………………………...137 Table 4.10. Responses to subordinate questions about training (n=109)..……………………152 Table 5.1. Conclusions regarding Kalamazoo training project outcomes…………………….159 xii LIST OF FIGURES Figure 3.1. Hypothesized comparison of training relevant and irrelevant items in IRS……….84 Figure A.1. Kalamazoo Training Passport…………………………………………………….218 Figure A.2. Sample Training Activity Submitted……………………………………………..223 xiii INTRODUCTION The contemporary landscape of police operations with regard to workforce maintenance and training consists of a patchwork of conflicting information, goals, resource availability, and adjustments to challenges that vary across agencies. The task of strengthening police training regimens to match expanding duties and technologies, while simultaneously budgeting for such training, has become more urgent in recent years (Wilson et al., 2010a). A 2010 survey by the Police Executive Research Forum reported that 68 percent of responding agencies (N=608) have cut or discontinued training as a result of recessionary budget cuts (PERF, 2010). Little empirical evidence has been collected to inform decisions to modify or terminate training protocols to generate cost-savings. Nevertheless, police agencies continue to invest in more esoteric training technologies, strategies, and improvements, often making choices based on scant information as to whether these innovations will necessarily lead to desired goals. Making crucial decisions in a time of fiscal uncertainty is risky; to do so without necessary information as to their long-term impact could be potentially disastrous for many departments (PERF, 2011). Police in-service training has long been an entrepreneurial endeavor, as departments have often sought external assistance from private sources whose training advice and specialized expertise have often replaced department-specific resources, usually at high cost that can no longer be sustained (Buhrmaster, 2009; PERF, 2010). Departments have historically not given in-service training the attention needed to bring instructional goals in line with departmental objectives, the long-term goals of police operations (such as community policing), and principles of organizational change and organizational learning (Bradford & Pynes, 2000; Chappell, 2007; Geller, 1997). The 2008 recession has performed a peculiar function with respect to police 1 training: agencies are often being forced to turn inward to satisfy training needs (Buhrmaster, 2009; Mayer, 2009). Taking responsibility for the training of their own officers can further the development of learning organizations long debated in police contexts (Geller, 1997), and develop adaptable training schemata which can align in-house training with institutional and organizational objectives and goals. In the course of this debate, two long-standing challenges facing police organizations have resurfaced: Bradford and Pynes (2000) lamented that training was not keeping up with practice, and Geller (1997) asked when police departments were going to be serious about becoming learning organizations. These concerns can be addressed with the development of police training reforms to content and process, which provide an opportunity to remake and reinvent operations at reduced cost. The promise of this existing opportunity comes with a renewed focus and understanding as to how police training operates. In policing, training is delivered at specific points throughout the officer’s career, targeting distinct outcomes. Because of this structure, the component potentially most susceptible to elimination or restriction due to budget cuts and lack of resources is in-service and supplementary training. However, very little is known about the nature of this training among the nation’s police departments. To date, no comprehensive survey of the state of police in-service training exists. Absent a thorough portrait of the state of in-service training in policing, police departments make decisions as to how to train officers beyond their initial academy and field training on an at-needs basis, often with monetary costs of training as the sole decision point. Such activity exposes a problem in contemporary police operations, one that is somewhat familiar to departments in other areas such as personnel planning (Wilson et al., 2010a). Comprehensive assessments to determine organizational training needs are rarely conducted, and departments rely on instinct and reactivity to guide post-academy training 2 forecasts and scheduling (Geller, 1997). Similarly, few departments assess their training capacities with regard to internal knowledge which may invigorate in-service training and propel organizational growth. Together with the tremendous uncertainty of post-recessionary budgeting, such an environment constitutes a “perfect storm” of missed opportunities: agencies need to turn inward to train their officers, but may be ill-prepared to do so because no model exists to capture their organizational strengths which could enhance such a new training scheme. Rethinking in-service training at this critical time could open up a variety of training opportunities, even with the external assistance once allowed for under more robust budgeting. Specifically, training literature across a number of related fields such as military science and organizational psychology have long recommended that in-service training targeted towards experienced persons include training for adaptive expertise, a variety of training which enhances application of knowledge and skills for rapidly-changing environments and situations through a number of targeted strategies (Chi, 2011; Lazzara et al., 2010; Pulakos et al., 2000; Sonnentag, 2004). Although studies have examined how this training works in military tactical environments, to date no application of principles of training for adaptive expertise has been attempted in police contexts. Applying these principles may expand the current state of police in-service training to encompass a variety of previously untrained phenomena, as well as provide a model for conducting quality training in a time of budget restrictions. The following case study answers urgent contemporary questions facing police organizations with respect to strengthening in-service training regimens in an atmosphere of budget restrictions, and how it may be accomplished. Utilizing a mixed-method approach, this study examined a training program targeting adaptive training behaviors as defined by industrialorganizational psychology literature (Hellen, 2009; Lazzara et al., 2010; Pulakos et al., 2006), as 3 well as the training needs identified by the site agency itself. The project developed and provided a model for police agencies intending to shift in-service training from traditional external venues to an internal approach. The project demonstrated that agencies can proactively assess the strength of their training in an effort to fine tune deficiencies, while ensuring that expert knowledge is being taught to the right classification of employees for whom it is designed and targeted. The Structure of Police Training In somewhat of a blanket statement, the 2010 PERF survey determined that police training is being sacrificed due to budget cutting; clarifying this statement requires a more precise knowledge of how police training operates. Training in policing is a three-tiered concept: a new recruit receives initial training in an academy setting, followed by a field training mentorship, with additional in-service or post-academy training after hire to reinforce experiential knowledge and introduce new technologies and evolving strategies (Alpert et al., 2006; McCampbell, 1986). In-service training is often conducted outside the department, and is increasingly privatized and entrepreneurial in nature (Aveni, 2005; Buhrmaster, 2009). Little is definitively known about the extent to which agencies control the quality of training their officers receive at any of the three stages, but the bulk of in-service training is traditionally under autonomous control of the individual agency. The high costs of running a police academy, as well as the presence of state mandates which guide academy curriculum and instruction, suggest that when agencies self-report that they are cutting training because of budget restrictions, they are referring to post-academy training. Although the lack of empirical data on in-service police training may suggest that risks undertaken by cutting such training are relatively unknown, in-service training is mainly 4 designed to develop the skills of experienced officers beyond that which is trained in the police academy. Because of the three-tiered police training structure, in-service training is the stage which represents the most appropriate opportunity to train experienced employees for expertise (Glenn et al., 2003). Thus, when departments say they are cutting training, it suggests that inservice training is the most at-risk, and what is potentially lost is the type of training that would most affect the development of expertise among experienced police officers. The variety of training that is being cut by agencies is theoretically the variety that best allows them the opportunity to expand the individual and organizational knowledge base and prepare employees for more advanced critical-thinking situations which frequently occur in police contexts. The three-tiered structure of police training ensures that the content of training is compartmentalized in domains targeting officers’ career stages. As the first training encountered, academy training intends to provide the officer with a basic working knowledge of specific topics in the police tactical, physical, and ideological environment so that these may be expanded upon later in the officer’s career. For instance, one such topic, firearms competence, may be divided into distinct knowledge and skills sets so that future training may enhance higher degrees of accuracy, or produce a more adaptable technique wherein the officer may apply the basic knowledge gained in the academy setting to more abstract or unforeseen circumstances. Similarly, field training mentorships intend to familiarize newly-hired officers with departmentspecific procedures, allowing them to apply skills learned in the academy in a realistic field environment (Alpert et al., 2006). As training is staggered throughout an officer’s career, inservice training becomes more valuable to both an officer’s professional development and the long-term stability and health of the agency, because it targets more advanced skills sets while invigorating the careers of officers over the long term (Geller, 1997). 5 The Risks of Cutting In-Service Training Due to the lack of empirical study, the implications for police organizations of eliminating in-service training can only be suggested by theory. In-service training is usually supplementary in nature, and involves the acquisition of new knowledge by experienced personnel, familiarization with advancing technologies, and strategies for career enhancement that strengthen organizational commitment and employee engagement (Brand, 1999; Clarke, 2001; Macey & Schneider, 2008; Tews & Tracey, 2008). This suggests that eliminating such training could fracture employee trust in the organization, leading to turnover and retention problems. Additionally, organizational forgetting could occur, a phenomenon where the organization’s knowledge base deteriorates over time, leading to organizational stagnation (Tsang & Zahra, 2008). Ultimately, if training designed to foster adaptive expertise in experienced officers is sacrificed by agencies, organizational risk is accentuated further by stunting the development of adaptive behavior and expert-level knowledge in its employees. Industrial-organizational psychology literature details an additional risk of sacrificing inservice training. Organizations are living entities; their trajectories are dependent upon multiple factors operating in tandem, such as personnel development, continued adaption of new technologies, and the assumption of new tasks and challenges (Applebaum & Gallagher, 2000; Clarke, 2001; Carson & Carson, 1997). When the opportunities for personnel to develop themselves professionally are limited, and stagnation occurs, the organization may reach the end of its theoretical life course (Paxton, 2009). Failing to meet the challenges of external forces or stimuli over time, these organizations become unresponsive to demands which drive organizational change (DeHolan & Phillips, 2004; Masuch, 1985). Such a risk could potentially mean that organizations which fail to reach set goals can potentially become obsolete. In 2011, 6 PERF held a conference where police managers discussed if operating budgets had interfered with their departments’ abilities to provide basic services to the community (PERF, 2011). Training was detailed as a mechanism to ensure the long-term vitality and relevancy of not only their departments, but the professional model of policing as a whole. In light of this concern, it can be echoed that training for the long term plays a crucial role in long-term organizational health. Despite this fear, many agencies do not face such extinction, but the role of training in acting as an invigorating force allowing them to fulfill longitudinal gals and perpetuate organizational learning has been underexamined. The inability or unwillingness of police agencies to train for adaptive knowledge, or train for organization-specific needs may be especially acute in contexts where training is not maximized for various reasons that are accentuated in the current economic context (PERF, 2010). Departments seldom set in-service training goals or standards of their own, preferring to handle in-service training needs on a crisis-management basis unless the topic is a required or mandatory recertification, or exceptional case (Schwartz & Yonkers, 1991). The result is a personnel profile for each agency that differs from organizational or community expectations. This lack of agency-specific training goal-setting may work in tandem with the misidentification or lack of identification of expert-level performers or performance to produce organizational forgetting (Carson & Carson, 1997; Tsang & Zahra, 2008). Similarly, employees’ careers become plateaued, a state where the career expectations and goals of an organization’s personnel arrive at a standstill (Carson & Carson, 1997; Smith-Ruig, 2009; Stout et al., 1988). Often, employees whose careers are plateaued remain hidden in various subgroups within the organization: they may be nested in work groups, where they can be cancerous and their plateaued situations become endemic; or they may hide in subcontexts where their true talents 7 are underutilized, such as when they are tasked with work that is beneath their true talent level. Potentially, this trend leads to conflict where the plateaued and non-plateaued employees are asked to perform at the same level; unless the organization has a consistent and current profile of their employees’ strengths, the result is training that is inappropriate to the employees’ professional needs (Smith-Ruig, 2009). Study Research Agenda and Research Questions Because police in-service training has rarely been examined in context, decisions made with regard to the development of police expertise through in-service training have often taken place without any specific empirical guidance, leaving police departments to navigate the dilemma of exploring this variety of training on their own (Glenn et al., 2003). This project placed a solid footing under those decisions. The study research agenda was derived from the site agency’s needs assessment as well as theoretical background about adaptive expertise training. This project has two overarching research goals: first, the study constructed, delivered, and assessed training for adaptive expertise through a number of prerequisites derived from learning taxonomies in an initial application of this theory to a unique police setting. Second, the study investigated how this variety of in-service training may be developed, implemented, evaluated, and incorporated into departments’ overall training regimens efficiently to serve specific training objectives. The sources of literature that assist in crafting research questions from this agenda are multidisciplinary, originating in organizational psychology, education, and criminal justice. Since very little analysis of the role of post-academy training has been conducted in police contexts, and literature about training strategies for police expertise is scant, this agenda serves as a starting point for a larger conversation about the role of in-service training in bringing about 8 multiple police training goals. However, the study focused specifically on establishing a knowledge base about training for adaptive expertise in a police setting. The study’s research questions furthered an understanding of the mechanics of adaptive expertise training in context. Sonnentag (2000) suggested nearly a decade ago that an experimental study where specific components of adaptive expertise training could be linked to performance objectives would definitively resolve many long-standing questions about the content of adaptive performance training. This study was inspired by that suggestion and isolated specific components of adaptive expertise to determine their impact on the establishment of prerequisites for adaptive performance. The project’s first research question is: what is the effect of structured practice, pre-training orientation, simulation-based training, self-evaluation, planning and forecasting, and training for diversity on prerequisites for adaptive performance? Secondly, how may in-service training be delivered in accordance with departmental objectives, resources, and existing strengths to achieve these same prerequisites? Two sources are used to inform the research methodology to answer these questions. The first source is theoretical literature on the mechanics of adaptive expertise and how it may be recognized and utilized as a goal in training settings. The second is an inventory of officer behaviors that the project site agency feels it would like to see in officers who receive such training. These two sources, prior literature and the department-specific inventory, assisted in the development of principles of adaptive performance which were used to determine training techniques, goals, and measurement outcomes. The task of creating these principles was also a part of the research project itself. The study site is the Kalamazoo Department of Public Safety (KDPS), a full-service consolidated public safety department located in Kalamazoo, Michigan, servicing approximately 75,000 residents. Prior to the initiation of this project, the site agency had been engaged in 9 addressing these multiple research questions and subquestions, albeit in a limited and anecdotal manner. Upon the promotion of a new training director, the department’s training division had begun to assess many aspects of its training delivery by identifying issues they felt may be considered training principles. In a training assessment, the department assessed if in-service training processes were a hindrance to producing better-trained officers. Answers to this question revealed variables which were used in the eventual research model. Among these variables, time spent in training was considered as a potential independent variable was seen to affect officer performance. Additionally, practice time was a variable which needed further examination for impact on department-specific outcomes. Specifically, the department wanted to know the effect of additional training time on officer performance. The following further research sub-questions are also derived from the agency’s training needs assessment. What KDPS training may reflect adaptive expertise criteria, and act as a foundation for further development of such training? What may be the impact of their particular policing model (consolidated public safety), as well as minimum staffing requirements, on training delivery? How can the department best identify experts within to assist in delivering the best possible training to achieve agency training goals? Constructing these research subquestions required the site agency to articulate the informal processes which had governed training decisions in the past. It was hoped that this project would not only assist KDPS in making training decisions more empirically-based, but also support a model by which other agencies can do the same, and deliver in-house in-service training at a level designed to bring about adaptive performance in police officers. 10 CHAPTER 1 THEORETICAL BACKGROUND The following section details the theoretical backgrounds upon which the study of adaptive performance training in a police context can be based. Because no single theory of police training has evolved to illuminate the task of placing a largely untested training technique in a new context, four theoretical perspectives are interwoven to create a more comprehensive viewpoint of the project. These theoretical perspectives speak about the nature of work, organizational behavior, training for expert-level performance, and the role of in-service training. Each of these perspectives has something to offer the project by providing a foundation from which practical questions can be asked and addressed. Before outlining these theoretical perspectives, it is helpful to consider the shortcomings of existing empirical studies of police training, illustrating why this project will be helpful in expanding theory as a whole. The State of Knowledge About Police Training The bulk of police training literature, specifically with regard to original survey research on the state of training, has traditionally focused on the first of three tiers of training, the police academy. A COPS-funded census of police academies provides descriptive data about the over 600 police academies operating in the United States, with a focus on organizational structure and relationships with sponsoring state agencies (Reaves, 2009). However, few questions probing the content of academy curricula and methods of instructional delivery are contained in the survey instrument, information which would assist in describing the state of in-service training in contemporary policing by examining topics covered and the extent of focus. Moreover, questions about hours devoted to training specific topics may be misinterpreted by respondents in surveys regarding academy behavior, especially questions about how many hours an academy 11 devotes to training community policing, since that topic is often infused into other curriculum topics and not taught as a stand-alone topic of instruction (Reaves, 2009; Rojek et al., 2007). The 2007 South Carolina Law Enforcement Census (Rojek et al., 2007) took a different approach: all 50 states’ police standards and training boards were contacted and mandated hours for licensure broken down by curriculum topic, resulting in a large database displaying the topics instructed in each state with corresponding hours for each. However, there were limitations to these data: each state licensing board may suggest a minimum number of training hours for licensure which may be surpassed in academies for reasons related to local or participating agency need. Rojek et al.’s report, at best, illustrated the minimum amount of attention given to specific topics as suggested by boards of training standards as opposed to what was actually happening at the instructional level. Nonetheless, neither report captures information about curriculum content or instructional delivery, nor do they focus on training beyond the police academy. Although theoretically academy training provides a skills and knowledge foundation upon which future in-service training may be built, the two training stages differ in other respects. Both the process and content of police training at the academy level differ from training that occurs at future points in an officer’s career (Bayley & Bittner, 1984; Chappell, 2007). Academy training is portrayed in literature as a structured socialization process, unique from training that occurs afterward because of the unique characteristics of the classroom experience that serve as the venue for the academy itself (Birzer, 2003; Bumbak, 2010; Haarr, 2001). The second phase of police training, the field-training mentorship, offers opportunities to model behavior through one-on-one instruction in the field. Field training has undergone a radical transformation in the age of community policing, from a static model based on checklists 12 and commands (Codish, 1996; McCampbell, 1986) to the incorporation of problem-based learning strategies and modeling techniques developed from andragogy, or adult learning (Birzer, 2003; Chappell, 2007; Hoover, 2002). However the use of case studies to examine field training has limits to generalizability given the diverse nature of field training approaches in practice (Chappell, 2007; Galloway, 2010; Glenn et al., 2003). The most critical phase of training in our prospective analysis of expertise and its utility for organizational growth is the least studied phase. Police post-academy training or in-service training has been discussed mostly in passing (Bumbak, 2010), its shortcomings explored anecdotally or in relation to broader theories about organizational behavior such as its role in fostering organizational learning (Crank & Giaccomazzi, 2009). No comprehensive survey of post-academy training content or curricula has yet been conducted; the few empirical studies examining this training phase are internal departmental audits conducted to assess compliance with state-mandated recertification procedures (City of Austin, 2007; City of Toronto, 2010; Glenn et al., 2003). What these audits reveal is that post-academy training normally constitutes two varieties: training conducted on a compulsory basis (e.g., for recertification or departmental mandate due to perceived deficiencies), and on a voluntary basis (e.g., professional development, acquisition of skills to foster career advancement). Post-academy training may also take place in different venues: it may be internally conducted by the department, or externally conducted by an outside body such as a private organization (Bumbak, 2010). Using this very basic descriptive knowledge about post-academy training, it may be possible to construct a model of post-academy training based upon training venues and intended impact. The research on police in-service training is scant, but suggests that a basic starting point for conceptualizing police training in post-academy settings could evolve from these basic descriptors. 13 The following theoretical backgrounds range from broad and conceptual narration (the nature of work and organizational behavior) to a more specific focus (the role of in-service training in organizations and training for expert-level performance). Links are made between the theoretical perspectives and the current project. Each of these related conceptual frameworks contributes to an overall theoretical perspective of in-service training for adaptive expertise in police work. Theories of the Changing Nature of Work and Its Relationship to Training Technological adaptations, shifting workforce demographics, and the decentralization of business have contributed to profound changes in the world of work in the new millennium (Karoly & Panis, 2004). Workers have not only needed to be more skilled than ever before, but they have been called upon to adapt their existing skills to rapidly-evolving and changing complex environments (Hall & Chandler, 2005; Hellen, 2009; Pulakos et al., 2004; White et al., 2005). In certain fields, these expectations have been foreseen for decades: for instance, in the world of business, this may mean learning new languages to be able to communicate with new global markets, and in the field of medicine, technological advances have fueled rapid change (Karoly & Panis, 2004). But some fields have been slow to internalize and react to these developments, or predict and plan for them in advance. In the field of policing, although the demands placed upon officers are traditionally very high, the added uncertainty of rapid change to existing social structures, cultures, and economic environments has required police officers to be more skilled than ever (Jensen & Levin, 2007; Schafer, 2007). Simultaneously, officers have had to make their skills more adaptable to circumstances that may fluctuate rapidly, creating a new definition for what constitutes police expertise (Glenn et al., 2003). Additionally, Wilson et al. (2010a) detail the expansion of police responsibilities in their report on recruitment and 14 retention of officers: problem-solving skills, global knowledge to fulfill homeland security responsibilities, communication skills for individual and social diversity, and analytical and technical knowledge suggest the accentuated need for adaptive expertise. Although these new demands are not foreign to most police agencies, the ability to train officers for such a high level of service has fallen short (Bradford & Pynes, 2000; Bumbak, 2010; Cleveland & Saville, 2007). Interestingly, one dimension of training that appears to have been forgotten in allowing training to lag behind other organizational demands in police contexts is that training acts as a direct response to social problems and trends (Haccoun & Saks, 1998). The importance of continuous training in acting as a direct response to the above changes, in producing a trainee or employee more prepared for these societal demands, has been underemphasized in contexts where training is not a part of organizational policy approaches. This constitutes a paradox: as the costs of matching workforce abilities to social trends increase, organizations are pressed to get more “bang for their buck” in training programs, which raises a host of issues with regards to the applicability and relevance of the training that is received (Haccoun & Saks, 1998). Policing is a field that is familiar with this paradox, as police agencies rarely invest in in-service training with data demonstrating its efficacy, or cost-benefit analyses showing what exactly is gained when officers are sent to external in-service training. Training is rarely used as an intervention, but it could be if data demonstrated what specifically may be of benefit. This dilemma calls to mind past discussions of the inadequacy of police training in inculcating values and belief systems necessary for the adoption of changes such as community policing, yet it is more related to the technical and proficiency shortcomings of an entire field of work in failing to deliver services to the public, or “client”, in this instance. Specifically, while police departments have long adopted new technologies and trained their employees accordingly, 15 they have often done so in a manner which treats expertise at using such technology as an end state (Glenn et al., 2003), as opposed to a tool that could be adapted to assist in navigating rapidly-evolving contexts and events. It has long been believed that once an officer enters a state of mastery, such as with firearms or other hard skills, he or she has reached a plateau of knowledge beyond which application of these skills occurs in circumstantial subcontexts. However, in their evaluation of the Los Angeles Police Department’s training program, Glenn et al. (2003) cautioned that the diverse nature of Los Angeles’ population demanded a new understanding of the role of expert knowledge, and made suggestions as to how the transition from officer to expert could occur. As work becomes more complex, police departments require their officers to not only be more skilled, but be able to wield those skills in a manner similar to any other “adopted” technological tool; however, training in order to perform at this level is lacking. The theoretical backdrop against which this shift occurs has its origins in both the fields of economics and business administration. First, the theory of work role adjustment has been used to detail the relationship between individuals and their social surroundings in times where changes in work roles take place (Berger, 2011; Nicholson, 1984). Over time in work contexts, organizational expectations fluctuate with regard to what roles their employees may play in achieving different organizational goals. This phenomenon is occurring now with regard to increased expectations of police officers: their work roles are changing because society expects police to perform more complex tasks, often in unforeseen and rapidly-changing contexts (Wilson et al., 2010a). As individuals, workers process these transitions as “adjustments”. Work role adjustment theory expands this term by breaking it into four components: the different requirements of the roles themselves, the individual’s motivational orientation, past occupational 16 socialization experiences, and current organizational efforts to affect the transition (Nicholson, 1984). The fourth predictor could include organizational efforts to train employees to handle the new roles expected of them, and assisting in adjustment to a newer, more complex work role and environment. The utility of this theory is to conceive of training provided by organizations as having a unique predictive value when examining the adjustment workers make to new work roles, while also considering the individual’s desire to adjust to the new role as an equally important variable. The study considers both of these theoretical orientations in constructing the eventual model. Second, researchers in the field of business administration have theorized about the impact of technological growth on the changing nature of work, focusing primarily on changes in cognitive abilities brought about by technology and their relationship to organizational goals (Heerwagen et al., 2010; National Research Council, 1999). As more complex and rapidlyevolving situations stress traditional work roles, a greater skills and knowledge base is required of workers themselves. Career development theorists have examined these phenomena in the context of changes in mid-career pathways, similar to work life adjustments, called a theory of cognitive competence (Heerwagen et al., 2010). The National Research Council’s (1999) analysis of Army work life, training, and career development partially enumerated this theory by focusing on flexibility, diversity, and occupational change as drivers of a redefinition of work content and structure, including training. In the new millennium, work situations are inherently more complex, demanding that individuals with higher-level skills operate in rapidly-fluctuating environments; training for these environments necessitates cognitive competencies that surpass traditional work role demands and expectations (Heerwagen et al., 2010; National Research Council, 1999; Zunker, 2008). Cognitive competence helps establish a base of knowledge in 17 individuals (and potentially, in group situations) where a number of skill sets may be used in expectation of the new economic demands of work: complex work environments, increasing demands on worker skills sets, and what Kanter (2000) terms “kaleidoscope thinking”, or broadbased thinking that leads to considerations of alternative methods of problem-solving. The important assumption in this theory is that change is constant and presents continual challenges to workers, and continuous and evolving competency and skills development is needed (Heerwagen et al., 2010). These two theoretical backgrounds can be directly related to similar experiences already taking place in the field of adaptive performance training. By linking them, one can effectively ground training behaviors already in progress in a conceptual apparatus in reverse fashion. Training for adaptive expertise involves, as White et al. (2005) describe in military science contexts, high-impact learning experiences designed to foster a more global understanding and kaleidoscopic view of the function of skills in on-the-job settings, mirroring Kanter’s perspective on the function of training as a tool by which workers are prepared and trained to think more globally. This also leads to more adaptive behaviors in specific on-the-job exercises, as explored by Ford and Schmidt (2000) in emergency response settings. One of their strategies to enhance skills development for actual emergency situations is to immerse the learner in active learning, reflected through realistic scenarios, expert-led problem solving, and simulated crises; these reflect Heerwagen et al.’s (2010) desire to prepare workers for the unexpected and increasingly complex real-world events that take place in multiple 21st century job settings. Finally, these theoretical perspectives support many of the training schemata in the field of disaster preparedness: Chen and Borodzicz (2009) look at simulation-based games as potentially providing opportunities for experiential learning, which assists in acquisition, processing, and 18 information to assist in rapid decision-making. Disaster preparedness training exercises as envisioned by Chen and Borodzicz assist the worker in transitioning one’s performance from routine expertise (the application of expert knowledge to routine situations) to adaptive expertise (the application of these same skills to unforeseen and emergency circumstances) (Chen & Borodzicz, 2009; Sonnentag, 2000). This reflects Nicholson’s theory of work role adjustment in that transitions from one expected role (in this case, the use of expert knowledge in routine environments) to another (adapting such expert knowledge to chaotic and unforeseen environments) can be trained for, practiced, and assessed. Theories of Organizational Behavior As evidenced by the above theoretical perspectives explaining the complexity of the world of work, human behavior is inherently unpredictable and often chaotic in nature. This may hold true especially in police work contexts, where unforeseen and often dangerous behaviors may require split-second decisions to be made in chaotic circumstances. This same environment impacts organizational behavior. When considering the straining effect of the 2008 recession on police agencies, the chaotic environment which organizations are expected to navigate is especially acute (PERF, 2010; Wilson et al., 2010a). Theories of organizational behavior contribute insight to how police organizations may assess needs, specifically training needs, in the context of balancing present resources and future organizational growth potential. While multiple theoretical perspectives exist which address the issue of organizational behavior in complex environments, the two most closely related to organizational change in this area are institutional theory and systems/complex systems thinking, also called ecodynamics theory. Both share some similarities with regard to how organizations behave, plan, and carry out their objectives and goals, but both differ markedly in their suggestions for organizational growth and 19 the manner by which institutions conform to formal structural procedures in order to achieve similar ends. Institutional theory can be used to explain the status quo of the contemporary police training environment. The 2010 PERF report outlining the state of training cuts portrays police organizations as responding to the new budgetary landscape with traditional modes of behavior (i.e., eliminating in-service training because of costs) and seeming incapable of rethinking how such training can be delivered (i.e., internalizing such training as a pathway to furthering organizational learning) (PERF, 2010). Conversely, systems and ecodynamics theory can be used to explain the opposing suggestions, strategies, and efforts to resolve the crisis of police training to construct a new institutional reality from the ashes of the old. These theoretical perspectives are complementary in this context. Open systems theory states that the loosely coupled nature of social systems reinforces what systems and ecodynamics theory implies about the nature of police decision-making: the boundaries of organizations, their structures and behaviors, are malleable and subject to frequent change (Scott, 1992). In a cyclical process, systems filter energy from outside as “throughput”, or the adaptability of resources in constructing new cycles of behavior which result in an integrated feedback-looping mechanism (Katz & Kahn, 1978). This feedback loop results in dynamic growth for the organization in an unsure and constantly changing environment. Argyris (1982) sees the looping mechanism as dual: while external forces and energies continually transform organizational goals, internal forces and feedback processing act as a goal-setting and evaluative device. This “double-loop” method of learning, where learning from errors shapes both short-term strategies to deal with future errors and changes in the overall conceptual apparatus used to envision such strategies, is directly related to systems and ecodynamics theory. The link between these theories is Weick’s interpretation of the role of sensemaking in organizations, a strategy which 20 assists collective and communicative processing of information and stimuli to discern patterns which may be used to generate responses to these often uncertain events (Rutledge, 1999; Weick, 1995). Using this linkage, sensemaking becomes a strategy that appears derived from the very chaotic environment that systems and ecodynamics theory states is prevalent in the modern organizational climate. Institutional theory assists in “reading” the activities of organizations in context of their operating environments: their definitions of legitimacy and the ways in which this designation is attained, their ability to handle crises and the tools they use, their stability and trajectory, and their accepted operational methods. Institutional theorists examine these structures as products of myth: the environment in which organizations operate engender a number of institutionalized and ritualized beliefs which reflect legitimacy in the eyes of external agents (Crank & Langworthy, 1992). To the organization, sustaining these myths is crucial to survival and a key to establishing legitimacy in the public view; it is an act of perpetuating and sustaining itself (Mastrofski & Uchida, 1996). In light of the desire of police departments to maintain legitimacy through the construction and perpetuation of myth, institutional theory would consider training as an avenue by which police agencies “ceremonially demonstrate their moral legitimacy” by engaging in “myth-building” (Crank, 2003, p. 188). This exercise has traditionally taken place, in post-academy and in-service contexts, as a ritualized and entrepreneurial exercise. Multiple websites, organizations, and private firms exist in a “police training industrial complex” to offer courses and certifications on an at-needs basis. 1 The potential purposes and functions of this training apparatus are largely anecdotal. It can be surmised, using institutional theory, that the availability of special training topics, offered at entrepreneurial cost and in varying geographic 1 Among these websites, PoliceTraining.net is one of the most popular, offering training opportunities in all fifty states and on topics ranging from closed-skills and tactical maneuvers to police management and leadership. The website reads as a calendar, with offerings posted by many private training firms in locations nationwide. 21 locales, serves the purpose to perpetuate myths about police work and disseminate values about the profession to outside agents (Crank, 2003; Crank & Langworthy, 1992). Crank (2003) suggests that the values inherent in myths police agencies perpetuate about themselves have remained largely separate from economic considerations; for instance, society rarely is asked to put a price on public safety or the health and well-being of the population that police protection provides. In the case of the police training apparatus described here, the issue of how much externally-offered training costs the department has rarely been questioned in a healthy economy, and as theory suggests, the values inherent in such an apparatus become engrained and institutionalized in police organizations, remaining unquestioned. Potentially, departments felt it acceptable to send their officers to costly training provided by private “experts” because other departments were doing the same, regardless of cost, and it reinforced the image of legitimacy to accumulate such training because it established the semblance of respect and ceremony. This mechanism shares much in common with theories of institutional isomorphism which have been applied to police contexts to examine police organizational structures and behavior. Giblin (2006) extracts three varieties of isomorphic behavior from DiMaggio & Powell’s (1983) theoretical framework of institutional pressures resulting in the adoption of shared behavior. Coercive isomorphism results from pressures placed upon organizations by external entities such as government (in the guise of mandates) or private groups (such as lobbying measures) in order to bring organizations more in line with perceived environmental legitimacy; mimetic isomorphism occurs when organizations, when faced with external change or uncertainty, model themselves after similar or peer organizations (DiMaggio & Powell, 1983). The third, normative isomorphism, is an appropriate lens to examine police training, as it is used by Giblin to describe how “training programs tend to highlight appropriate practices” which are 22 passed on from one agency to another through channels of legitimacy, such as publications and professional organizations (Giblin, 2006, p. 646). Agencies communicate practices with regard to training strategies and standards which are absorbed by their peers, and adopted as standard behavior. This phenomenon pertains explicitly to the current project, as in the case of a survey of the content and nature of Michigan law enforcement agencies’ in-service training procedures conducted in 2006 by the regulatory body which governs the state’s training standards (MCOLES, 2006). The survey gauged patrol officers’ impressions of the breadth of their inservice training regimens including their impressions of the strength of training curricula in a number of content areas; the results were widely disseminated throughout the state to Michigan’s police departments to provide feedback as to how training may be strengthened. In tandem with this assessment, the International Association of Chiefs of Police has provided a series of “Training Keys”, or short supplementary in-service curricula, training strategies, and objectives which departments may use in order to bolster training where it is deemed lacking (IACP, 2012). These two resources reflect Giblin’s examinations of normative isomorphism in that much of the current state of training in Michigan (and elsewhere) reflects uniform behaviors disseminated as legitimate and appropriate by accreditation agencies that are an arm of the state. Normative isomorphism is also directly related to the research questions involved in the current project, as programs are adopted in an effort to appear homogeneous and to satisfy external demands for accreditation, often without empirical tests for efficacy (Giblin, 2006). The current realities of municipal budgeting have challenged institutional theory’s claim that stability and growth are inherent byproducts of social norms and community function. Notably ironic is Crank and Langworthy’s statement that police departments are so engrained in the “natural order of things” that “the elimination of a municipal police department is… 23 unthinkable” (Crank & Langworthy, 1992, p. 360). This assumption has been reversed in the context of the 2008 recession and its aftermath, where the contemporary model of policing has been seen by some as unsustainable (Matarese, 2011; PERF, 2010). In the contemporary environment, the disbanding of police agencies as a result of either budgetary crises or institutional disorder is a looming reality, not at all unthinkable (King, 2009). For a select number of police organizations, the 2008 recession has endangered their very survival, and forced them to eliminate long-standing attributes of their traditional claims to legitimacy, such as the ability to train officers for the length of their career (Buhrmaster, 2009; PERF, 2010). The elimination of many of these central components of what once established police departments as “legitimate” in the eyes of the public necessitates a new conceptual apparatus with which to examine how police departments can sustain themselves, and how they can best operate in chaotic times and turbulent economic environments. Simply put, police agencies can no longer afford to perpetuate many of the long-standing myths about themselves that have sustained their existence for decades. Complex systems theory explains how this training apparatus may be viewed in the context of economic challenges. Organizations are dynamic in nature, with external forces (called “strange attractors”) perpetually straining their performance (Gleick, 1987; Walker, 2007). These forces can take multiple forms: they may be social, economic, or political in origin, and require the organization (called a “complex adaptive system”) to continually adapt to their threat (Mink et al., 1991). These complex organizations have four foci embedded within them which validate and shape their behavior, appearance, and trajectory of growth: context, valid information, relationships, and shared meaning (Dietz & Mink, 2005). The context focus is described as the social, physical, and political features that comprise an organization’s 24 environment, such as the existence of college campuses nearby, or having a community with a specific ethnic background. The information focus refers to the semantic descriptors used to frame and describe the context at work. The relationship focus refers to the ever-changing dynamic between an organization’s members and those in its environment, and the meaning focus consists of the totality of understanding between an organization’s members and outside agents about responses to events that take place (Dietz & Mink, 2005). These four foci interact and self-reinforce, filtering information from outside the system and processing their influence, redefining the organization’s shape and trajectory on a continually nonlinear basis (Dietz & Mink, 2005; Walker, 2007). Consistent with open systems theory, the organization grows and changes over time according to how these foci process these external forces: for instance, the cultural context in which the organization exists may impact its needs and responses to crisis, and the shared meaning of what the organization’s role, purpose, and influence may have on other entities in turn determines its structure and ability to process external information (Mink et al., 1991). Predictability and tradition are replaced by anticipated but unknown chaos and complexity; the organization assumes continuous change is present, and the context in which these events take place more accurately resembles a living ecosystem (Walker, 2007). Walker (2007) uses the phrase ecodynamics theory to explain the state in which living, unpredictable behaviors comprise social systems and environments in which humans and organizations behave and survive. It is helpful here to clarify an important feature of complex systems theory and how it relates specifically to the project. Dietz and Mink (2005) place complex systems theory in a police organizational context by focusing on relationships that police agencies build across different groups: their members, individuals and groups within the community, and power 25 structures that bear influence on the police agency itself. These relationships emanate from the four foci as described above in a manner that even the simplest occurrences produce complex and larger results (what the authors refer to as the “butterfly effect”) (Dietz & Mink, 2005, p. 8). An individual example of this process would be that training delivered to a group of officers can have myriad impacts and be disseminated, retained, and utilized in different and unforeseen ways. This constantly fluctuating pattern of relationships and events can be described as nonlinear in nature: that is, its growth trajectory, progress, and structure appear random and formless. At first glance, describing a police agency as having nonlinear characteristics appears to be contradictory given prior classic research into the stable and bureaucratic nature of police agencies themselves. However, Dietz and Mink portray organizations as adaptable: the path these relationships and events take may display complexity (the “complex” part of their designation “complex adaptive system”), but the events, and the organization itself, operate wholly in an industrial or governmental context, and must adapt to that context as they emerge (the “adaptive” component of the phrase “complex adaptive system”). By describing organizations themselves as adaptive, yet operating bureaucratically, complex systems theory lays the groundwork for examining adaptive expertise itself: training, as well as knowledge, can be disseminated in a rigid and formal manner, but may be adapted to varying effect and in different contexts by the trainee. Complex systems theory ensures that “adaptive” and “systematic” are not mutually exclusive concepts. Complex systems theory could explain how police organizations respond to the current budgetary challenge in redefining not only training goals, but as institutional theory posits, the myths that police agencies use to define training needs. Emergent patterns which exert an influence on organizational growth, planning, and strategy, suggest the need for police 26 organizations to form internal structures to deal with their continual threat and influence (Dietz & Mink, 2005). In one example, the creation of specialized units in police work acts as a response: citing community need and the external threat of forces such as community diversity, tactical procedure, or terrorism, police agencies form specialized units to deal with these continual challenges. The training environment currently at risk could likewise force police agencies to redefine, internalize, and fulfill their training needs through the creation of an internal and agency-specific response to this challenge. After the 2008 economic crisis, agencies were forced to reassess growth opportunities despite the “new normal” of budget challenges. Using complex systems theory, the establishment of an internally-driven structure or strategy to deal with the emergent pattern of economic chaos could constitute an opportunity to engage these forces and establish a new organizational focus, discarding long-held myths about the role of training in the process. It is important to note that to complex systems theory, chaotic environments hide a form of order that may deceive organizations attempting to evaluate responses to them. The words “formless” and “chaos” often inspire a misunderstanding of the true function of the context in which complex systems behave: simply because organizations may follow nonlinear growth patterns does not necessarily mean that responses are improvised or devoid of structure entirely. In a sense, the hidden order of complexity brings organizational theory full-circle back to the institutional perspective by holding that individual acts on the part of a system’s interdependent components, no matter how random they may appear, constitute an orderly progression that may be unintentional (Caldart & Ricart, 2004). Consequently, levels of analysis cannot be isolated to the extent that smaller decisions that make up an organization’s behavioral trajectory are seen as separate from the larger system in which they are embedded (James et al., 1982). This issue will 27 inform the methodology of the project itself and will be discussed in more detail later, but it is important to point out theoretically that complex systems respond to seemingly random events in a pattern that mirrors the ecosystem in which they operate. Theories of Training for Adaptive Expertise Before examining the landscape of theories about training for adaptive expertise it is helpful to consider that the phrase “adaptive expertise” implies a dual process. As identified by Chi (2011), the concept of adaptive expertise contains two concurrent themes: the study of exceptional individuals (expertise) and their ability to flexibly generalize and use their skills in non-routine circumstances (adaptability). These two fields have only recently been studied in tandem, and as the world of work grows increasingly more complex, differentiations between what constitutes expertise in routine contexts and the level of performance necessary to handle more exigent circumstances have been explained by theory (Bransford, 2001; Endsley, 2006; Schwartz et al., 1999; Sonnentag, 2000). Consistent with Chi (2011), other industrial psychologists assert that adaptive performance acts as the foundation for adaptive expertise, which may be considered a high (i.e., expert) level of adaptive performance (Endsley, 2006; Sonnentag, 2000). The dominant paradigm used to explain the expectations of adaptive performance is called situation awareness (Endsley, 2006). 2 Situation awareness indicates an “up-to-date understanding of the world around” (Endsley, 2006, p. 633), and can be anecdotally compared to the abilities of professional athletes who can “read” an entire field of play, identify where and how potential actions will evolve in advance of their happening, and anticipate where 2 A skill often trained in police contexts is “situational awareness”, which is theoretically distinct from situation awareness. Situational awareness, derived from the Marine Corps’ Special Forces Field Manual, is a tactical skill which involves the rapid assessment of combative individuals in tactical situations so that an accurate decision can be made with regard to a soldier’s safely diffusing tactical threats (US Marine Corps, 1997). 28 one needs to be at the precise moment they occur. It is theoretically similar to Kanter’s (2000) kaleidoscopic thinking processes. Reflecting the definition of adaptive expertise as a dual process, situation awareness involves the development of five distinct features: perceptual processing of relevant information, limited working memory, efficient processing of information for goal acquisition, expectation, and the construction of mental models that may disclose behavioral patterns (Endsley, 2006). Perceptual processing implies that limited mental capacities of learners, especially in complex environments, restrict the amount of information that may be processed, and limited working memory exacerbates this problem by reducing the ability of the learner to store and integrate processed information. Goal-driven processing involves the reconciling of learner goals with the knowledge that both their memory and ability to process information are limited: an individual selectively processes what information is consistent with their learner goals. Finally, expectations and the construction of mental models guide the interpretation of selected information and the modeling of that information into behavior. The five-fold process supports the development of adaptive expertise by acting as a cyclical and iterative feedback loop: individual learning goals drive the search for new information and its selection, storage and formation into models (Endsley, 2006). This expert level, found in what Chi (2011) terms “exceptional individuals”, is not an end state: it is a continuous process that occurs in the context of rapidly-evolving and potentially chaotic situations, ranging from driving through traffic (Johannsdottir & Herdman, 2010) to military tactical environments (White et al., 2005). Theorists of adaptive expertise disagree with respect to what differentiates an expert from a non-expert in this sense. While situation awareness is the key to adaptive performance, its existence in individuals does not imply that they may be able to generalize these skills to novel 29 and unforeseen situations (Schwartz et al., 1999). An individual with situation awareness but no ability to apply these skills in an adaptive manner has been labeled a “routine expert” (Hatano & Inagaki, 1986; Smith et al., 1997). The method by which routine experts may become adaptive experts is the point of disagreement among theorists of expert performance. Sonnentag (2000) focuses on developing five strategies to bridge this gap: knowledge, problem comprehension, goal setting, feedback processing, and communication. Cianciolo et al. (2006) isolate the development of tacit knowledge, or the interplay between individuals and environmental stimuli, as the crucial step in advancing expert performance beyond the routine; this level of knowledge is distinct from that which is transmitted through routine classroom instruction. Renwick & Burrows (2008) examine the role of metacognition and feedback processing more closely, finding that industry-specific knowledge can assist in developing capabilities for advanced research capacities in teachers. Hellen (2009) looks specifically at information processing and organizing as the key to developing evaluative skills that may advance adaptive performance. Ericsson and Lehmann (1996) focus on practice as the mechanism by which routine experts become superior expert performers, advancing the point where superior experts are able to establish a more conceptual understanding of the utility of their knowledge through continual practice in differential contexts over time. Each of these foci broadens the definition of adaptive expertise by contrasting it with the simple acquisition of knowledge. It is helpful here to begin to conceptualize, in conjunction with taxonomies of adaptive expertise, some prerequisites the literature identifies for the development of this skill. Adaptive expertise is a continual process grounded in a variety of broader skills sets, and is summarized in taxonomies of learning. Prerequisites for adaptive performance include the ability to organize complex information (Hellen, 2009; Pulakos et al., 2000), self-confidence and self-efficacy 30 (White et al., 2005), the ability to recognize cues which aid in problem identification (Lazzara et al., 2010; White et al., 2005), metacognitive ability and the ability to self-correct errors (Ericsson & Lehmann, 1996; White et al., 2005), experience and experiential learning (Pulakos et al., 2000; Sonnentag, 2000; White et al., 2005), the ability to negotiate stressful environments (Lazzara et al., 2010; Pulakos et al., 2000), conflict resolution and negotiation skills (Pulakos et al., 2000; White et al., 2005), and the ability to anticipate and negotiate crises (Pulakos et al., 2000; Pulakos et al., 2006). In the absence of a comprehensive listing of prerequisites for adaptive performance, this base of prerequisites crafted from literature serves as a foundation not only for the development of adaptive expertise, but as potential dependent variables in the project analysis. By targeting these dimensions of adaptive performance, training can accentuate skills needed for the development of adaptable behaviors at higher performance levels. Researchers agree that both experience and practice are prerequisites for the development of adaptive expertise. Pulakos et al. (2002) found a positive relationship between experience and the development of adaptive expertise when the new situations encountered required the expert to adapt to novel environmental stimuli. Wiedemann (1995) found a positive correlation between prior experience and adaptive performance, determining that the translation of knowledge into action in adaptive situations was particularly acute in those who had a wealth of experience to draw upon. Sonnentag (2000) theorized that problem comprehension and knowledge organization, two of the five crucial strategies for adaptive expertise development, are enhanced in experienced persons, and that increasing this experience leads to individuals utilizing their knowledge differently than the lesser or non-experienced. But experience and practice must vary for this component to have its greatest effect: Smith et al. (1997) determined that the way to get experts to perform in uncertain environments is to provide experiences in such contexts 31 repeatedly. Similarly, structured practice suggests a path beyond mere proficiency in trained experts. For Ericsson and Lehmann (1996), experts cannot learn to adapt their knowledge to unforeseen environments without first engaging in frequent applications to novel contexts in training and practice situations. Not practicing holds powerful implications, as certain experts may choose to “remain mediocre… [and are] satisfied in reaching an acceptable level of performance and continue in maintaining that level of performance with minimal effort for years on end” (Chi, 2011, p. 28). The construct of this training is largely theoretical, but Chi suggests that it requires experts, mentors, and instructors to be passionately engaged in providing creative and unorthodox training, pushing and demanding experts to perform at sustained levels “beyond their zone of proximal achievement” (Chi, 2011, p. 29). By providing opportunities for increased practice in novel situations, trainers push experts to think creatively about solving problems, apply existing knowledge more meaningfully, think globally about potential solutions, and process feedback in a manner that perpetually helps improve performance through reflection (Chi, 2011; Ericsson & Lehmann, 1996; Hatano & Inagaki, 1986). It is through practice that theories of adaptive expertise are illuminated: practice that challenges experts to go beyond routine performance allow for reflection, generalization, and a deeper understanding (Chi, 2011). What this research suggests is that adaptive experts are more experienced at what they do, practice more often at applying their tacit knowledge to non-routine contexts, continually develop their knowledge by treating performance as a training exercise, process and evaluate feedback on their performance, and are able to translate these experiences into data in a metacognitive way. Research suggests that without the proper instruction, training, practice, and application to potentially volatile and real-world circumstances, expertise becomes an end state, and while routine experts are valuable to organizations, their inability to be adaptive to the 32 demands of complex work settings hinders organizational performance (Chi, 2011; Cianciolo et al., 2006; White et al., 2005). These strands of expertise research also suggest strategies for training in the form of taxonomies of adaptive expertise (Lazzara et al., 2010; Pulakos et al., 2006). The differences between experts and nonexperts, and between routine and adaptive experts, informed the current project’s design by illustrating independent variables that may be used to vary performance levels (Pulakos et al., 2002; Pulakos et al., 2006). Theories of the Role of and Importance of In-Service Training The previous three theoretical perspectives contribute individual components to a larger understanding of how adaptive expertise works in particular settings. The fourth theoretical paradigm unites the previous theories into a common focus: how in-service training assists individuals and organizations in reaching long-term goals, and fosters individual and organizational growth and learning. Theories of in-service training are largely synergistic, providing a visual identity of the endgame of training. These theories complement theories of organizational behavior and the role of expert knowledge by suggesting how training contributes to this end result. Theories of in-service training focus on two specific features: the process of training itself, which involves delivery, maintenance, and evaluation, and the outcomes of training, which involve performance and goals. Both of these features of training theory add to a greater understanding of how training for adaptive expertise may benefit police organizations. Moreover, training theories assist in the design of the current training project by specifying potential variables that will be important to incorporate. These theories have implications for measurement of an eventual model of police in-service training, primarily by providing suggestions as to how training may be measured in a field where the opportunities to perform skills learned in expert-level training may be unexpected and rare. 33 In-service training is defined as training targeting individual growth and development of already-existing skills sets, normally in the context of a work situation where the individuals have already attained a degree of experience at specific tasks to be developed (Clarke, 2001; Pollard, 1969). It is intended to supplement training that has already been performed at an earlier career stage, and in being supplementary, involves a different set of training objectives, strategies, tasks, and outcomes than training that would traditionally be delivered to individuals who have no experience in a given task (Aguinis & Kraiger, 2009; Tews & Tracey, 2008). In police training contexts, in-service training (often referred to as “post-academy training”) would normally target individuals with an unspecified degree of prior experience and knowledge, delivered as a supplement to training they may have received in the police academy (Kramer & Barr, 1974). Its purpose is to expand skills sets to attain a level of performance beyond that which one may be able to attain based solely on academy training and experience (Pollard, 1969; Schwartz & Yonkers, 1991). Because this variety of training theoretically serves specific ends in workplace settings, examining its utility and function requires a conceptual apparatus derived from multiple disciplines which have been used to describe those same goals, namely the fields of adult education, educational psychology, and organizational psychology. In-service training presents unique opportunities for organizations to expand and further both individual and organizational performance (Della, 2004; Schwartz & Yonkers, 1991; Tews & Tracey, 2008). Due to their synergistic qualities, many of the features of in-service training (metacognition, transfer of training, adult learning) are difficult to disentangle from one another in order of delivery and product (Royer, 1979); a clearer picture of how this process takes place will emanate once specific variables in the project are detailed in the next section. Prior literature has been specific about which features of in-service training have been able to have a 34 positive impact on certain desired training goals. In-service training may foster the transfer of learned skills to on-the job behaviors by acting in part as a “dosage” of a specific skills set, encouraging reflective thinking and metacognition through strategies such as upward feedback and self-coaching (Tews & Tracey, 2008). In-service training has also been seen to impact transfer of skills through realistic and contextual practice that reflects expected job situations (Chen & Borodzicz, 2006; Ford et al., 1992; Kluge et al., 2010). Metacognitive goals have also been positively related to the use of in-service training strategies such as feedback processing and the encouragement of reflective thinking, due in part to opportunities provided by selfcoaching to recognize one’s own improvement over time (Baldwin, 1992; Tews & Tracey, 2008). In-service training also may develop critical-thinking and decision-making skills through the use of high-impact learning experiences such as realistic modeling strategies (Baldwin, 1992; Helsdingen et al., 2010), adult learning techniques such as andragogy (Birzer, 2003; Birzer & Tannehill, 2001; McCoy, 2006), and information processing strategies (Chen & Borodzicz, 2006). Andragogy in particular has been shown to assist in the identification of unforeseen problems encountered in work situations when adult learning strategies are a part of in-service training delivery (Birzer, 2003). Most importantly, mastery of skills as an outcome has been shown to be positively affected by realistic practice opportunities (Kluge et al., 2010) and continuous upward feedback (Tews & Tracey, 2008). In reinforcing transfer of skills, in-service training can be related directly to adaptive performance. For instance, strategies such as errormanagement training may reinforce metacognition, or the awareness of one’s mental process, as well as the formation of improvement strategies in challenging situations (Heimbeck et al., 2003; Kluge et al., 2010; Smith et al., 1997). Ultimately, many in-service training strategies, such as a classroom component, feedback opportunities, or realistic situational practice, have been 35 considered by adaptive expertise theorists as ways in which more adaptive behaviors may be instructed (White et al., 2005). In-service training ultimately targets individual development by assisting workers in expanding their skills sets and benefiting on-the-job performance (Aguinis & Kraiger, 2009), and in transferring trained skills more adequately to job contexts (Barnett & Ceci, 2002). However, the need to measure the impact of in-service training on individual development in order to test this theoretical proposition often requires researchers to utilize prerequisites for training performance as a proxy for future performance. In their review of training transfer research, Baldwin et al. (2009) state that measuring the transfer of skills over time may involve linking individual self-efficacy to future performance due to the difficulty in measuring specific changes in individual performance characteristics brought about by training supplements. Similarly, Bass (1998) uses transformational and transactional leadership development as a measure of the impact of in-service training. This challenge is presented in the current training project, specifically because actual performance opportunities to test the development of trained skills in policing are often rare. Prior research in examining this unique problem is used here to illustrate why individual development as a result of in-service training cannot be assumed as leading to improved on-the-job performance. In-service training has been theorized to have a positive effect not only on individuals and their performance, but also on organizational growth and performance. Although many of the contexts used to examine the effect of training on organizational performance involve productivity and profit development, other effects consistent with public organizations have been demonstrated such as improved employee attendance, employee engagement, employee empowerment, and a positive work climate (Aguinis & Kraiger, 2009; Clarke, 2001; Ford, 36 2007). Most importantly, a positive relationship has been established between continued training on-the-job and the development of learning organizations, reflecting two attributes of adaptive expertise training from Pulakos et al.’s (2000) taxonomy of adaptive performance. The method by which this transaction occurs is the furthering of organizational learning by training for expert-level skills and knowledge in experienced personnel within organizations. First, organizational learning requires the development of leadership through the transmission of cultural and organizational values which are transformational in individuals (Vera & Crossan, 2004). By developing new skills in individuals, in-service training systematically engages the entire workforce through the creation of an environment of learning and expectation of individual growth (Crank & Giacomazzi, 2009). Interestingly, as this leadership is developed over time in individuals throughout the organization, the ability of these persons to adapt to continual challenges is enhanced (Senge, 1990). As employees adapt to rapidly-unfolding change, so too does the organization become “adaptive”, much in the same way adaptive expertise is theorized to work in training contexts (Applebaum & Gallagher, 2000). Second, organizational learning is also furthered when individuals are able to engage in continual information processing (Brown & Brudney, 2003). Argyris (1982), in an early theoretical explication of organizational learning, stated that individual ability to detect and correct errors allowed for insightful and affective transformations that could permeate the organization. Information processing allows the entire organization to collectively deviate from routine, resulting in a feedback-processing loop where continual learning, self-evaluation, and growth through change become institutionalized (Alarid, 2000; Bushe, 2009; Senge, 1990). This closely resembles the information-processing and self-coaching strategies that have been theorized to develop adaptive expertise in individuals (Hellen, 2009). Argyris (1982) and Senge 37 (1990) state that as individuals within organizations are trained to become more adaptable and learn to expect the unexpected, organizations similarly grow and become flexible in the face of rapid changes brought about by external forces. One additional relationship between theories of training and principles of adaptive expertise expands the definition of practice (Ericsson & Lehmann, 1996). Deliberate practice as a training strategy is distinct from other forms of practice in that it must allow for reflection and deliberation (Tews & Tracey, 2008) as well as fostering extensive task repetition (Kluge et al., 2010). It is not sufficient to simply allow for one or the other, but both in tandem must operate to allow for extensive metacognitive reflection on the act of practicing skills. The practice opportunities must simulate realistic work experiences that a trainee may encounter (Ford et al., 1992) in order for the repetition to transfer not only to actual job performance, but performance in contexts not yet encountered, a pillar of adaptive performance (Ericsson & Lehmann, 1996; Kluge et al., 2010). However, limitations to the KDPS study do not allow for extensive task repetition associated with deliberate practice, but will instead focus on trainee reflection and maximization of practice opportunities. Summary: Theoretical Perspectives and the Examination of Adaptive Expertise The four theories summarized above contribute to a larger understanding of how adaptive expertise is developed through targeted training efforts, how prerequisites for their development may be embedded in training settings, how instructional techniques leading to the development of this skill may be used effectively, and how their impacts cascade from individual to organizational levels by acting as a model of how knowledge may be used adaptably. Each of these four theoretical perspectives assists in an initial research foray to place adaptive expertise in a new setting – police in-service training – by addressing what specific components will work 38 in establishing adaptive expertise in police officers. Reviewing these theoretical components, gaps in the literature can also be seen which the project hopes to address. To summarize, as the world of work becomes increasingly more complex, new strategies for performing at higher levels in an environment of rapid and unforeseen change are needed: theories of the changing nature of work have trumpeted the need for new training techniques to keep workers apace with these demands (Hall & Chandler, 2005; Karoly & Panis, 2004). Workrole adjustments, technological achievements, and an increasing palette of demands make police work especially susceptible to these rapidly changing needs (Wilson et al., 2010a). In the field of military science, adaptive expertise has been utilized as a platform for training military personnel to perform at high levels in such rapidly-fluctuating tactical environments (Vandergriff, 2008; White et al., 2005), but the benefits of such training for non-tactical environments have been theoretically linked to broader performance metrics (Chi, 2011; Lazzara et al., 2010; Pulakos et al., 2000; Sonnentag, 2000). Adaptive expertise is rooted in adaptive performance: developing the prerequisites for the development of adaptive performance requires a concerted organization-wide effort from every level (Pulakos et al., 2000; Vera & Crossan, 2004). Taxonomies of adaptive performance point the way to training strategies which may lead to the development of adaptive expertise in individuals (Endsley, 2006; Mangos & Johnston, 2008). Yet while theories of organizational behavior state that organizations themselves are often required to grow and behave in similar turbulent and chaotic environments, the link between organizational adaptations and the adaptive skills of the individuals within these organizations is somewhat unclear (Dietz & Mink, 2005; DiMaggio & Powell, 1983; Giblin, 2006; Gleick, 1987; Walker, 2008). 39 This is one shortcoming of prior literature on adaptive expertise: the lack of organizational context which may illuminate how the adaptive skills of an organization’s members are related to the ability of the organization to adapt to turbulence. By examining adaptive expertise training at work in an organization and examining its effect on the development of specific skills, a wider lens may be used to examine the effect of these strategies on the organization as a whole. When faced with uncertainty and rapid change, individuals strive to make sense of their surroundings in myriad ways: they reflect on past experiences, process information, initiate plans for action based on decisions made in the thick of battle, adapt behaviors that they have been instructed are appropriate for tasks at hand, and accomplish all of this quickly and effectively (Rutledge, 2009; Weick, 1995). Training individuals to perform using these abilities is assisted by multiple theoretical perspectives from related fields of study (Kanter, 2000). Individual training is thus related to organizational learning when employee performance enhances the ability of organizations to adjust to rapid change and adapt to external environments (Klein et al., 2006a; Klein et al., 2006b). At this stage of individual and organizational performance, the expectation of change ceases to become an end state and emotional responses of both individuals and organizations themselves may assist in transition to unexpected and turbulent events (Bartunek et al., 2006). This study tests how police organizations may train their officers to expect such turbulence, adapt to it as it occurs, and potentially contribute to organizational learning. 40 CHAPTER 2 METHODOLOGY The following chapter details the methodology of the research project as planned and developed by the researcher and host agency. The planning phase of the project is detailed in order to reflect how the agency and researcher engaged in the task of creating a research partnership from a perceived need arising from an internal assessment. This partnership was essential in surmounting many methodological challenges which have previously prevented a more thorough examination of adaptive expertise in field settings. Over a decade ago, Sonnentag (2000) lamented that, despite a large degree of promise generated by adaptive expertise as a pathway to superior adaptable performance in individuals in work contexts, methodological problems prohibited further exploration. A primary difficulty was that prior examinations did not contrast expert-level performers with non-experts, nor detail how the organization identified such experts. Particularly important for Sonnentag (2000) was the delineation between expertise based upon either performance or experience, and the relationship between both in crafting organizational definitions of “expert”. Also, no comparison group of non-experts or untrained performers was used in any prior case study to determine how training for expertise may become embedded in experienced or excellent performers, and what the role of such training may be. Sonnentag described data collection methods that have been attempted in the past to detail processes by which expert performers adapt to differential training tasks, including processtracing verbalizations out loud (later analyzed for patterns and themes using various qualitative assessment techniques), third party observations, self-report questionnaires, and retrospective recall. None of these prior attempts have yielded “definitive conclusions” because design flaws in each study do not reveal if high performance in experts is related to either experience or 41 training, and that “more experimental studies are needed” (Sonnentag, 2000, p. 256). Suggestions included providing expert-level training to non-experts to determine if performance improves using outcome measurements derived from taxonomies of adaptive expertise. The following study attempted to make more definitive conclusions utilizing a more rigorous research design. In doing so, the project methodology also took into account some inherently problematic obstacles endemic to training measurement and evaluation in field settings. Previous discussions of these flaws have centered around two primary difficulties: the obstacles preventing the use of control groups and random assignment of trainees in training evaluations conducted in applied settings, and the risks to scientific credibility and validity associated with the use of training designs to account for the lack of control groups and random assignment (Cigularov et al., 2008; Haccoun & Saks, 1998; Quinones & Tonidandel, 2003; Sonnentag, 2000). These challenges are both a product of organizational fear and the demands of methodological rigor: although dynamic field situations present proper evaluation settings, logistics, lack of organizational expertise at performing training evaluations, and ethical considerations often prevent organizations from performing analyses at all. Salas et al. (2003) described this situation as being particularly acute with respect to military and tactical environments similar to policing, and outlined core misconceptions about training evaluations that could be addressed through differential and triangulated evaluation techniques. This project employed a case study approach to determining the effect of multiple components of training for adaptive expertise on prerequisites for adaptive performance, and in examining how such training can be delivered in a police department in order to best utilize resources and existing strengths. The site chosen was the Kalamazoo Department of Public 42 Safety (KDPS), serving a mid-size city in southwest Michigan. The study employed a mixed methodology of qualitative observation and interviewing, as well as a quantitative evaluation of performance using pre and posttest self-report data generated from questionnaires administered to trained personnel. Data obtained will eventually inform a model of in-service training for adaptive expertise which can be used in different settings as a guide to implementing internal training for superior performance. The method used to conduct the training implementation and evaluation was partially derived from a needs assessment conducted by KDPS in August, 2011. The needs assessment allowed the department’s training division to reflect on a number of issues which incorporate Sonnentag’s (2000) suggestions to translate prior research into adaptive performance training. Some of the questions KDPS attempted to answer in the needs assessment were: what sort of tasks may the department evaluate to determine whether officers are performing at expert levels? If expertise is based upon specific training strategies, what tasks can the department use to evaluate the transmission of knowledge and skills required for adaptive performance? How does the department differentiate between routine and adaptive expertise, if they do at all? Finally, how can the department identify experts qualified to train their peers? These questions are also relevant to the research agenda of the study. Background of the Study Site and Training History The city of Kalamazoo is a medium-sized city in south-central Michigan with an urban population of approximately 76,000 (US Census, 2008). The public safety department serves the city’s urban, industrial center and outlying neighborhoods. There were approximately 205 sworn officers in the department at the beginning of the study (June 2012). Unlike other police agencies nationwide, the department is a consolidated public safety department, where officers 43 are cross-trained at both police and fire duties and provide both services to residents. This special designation gives the training project additional qualifications but also raises validity concerns with respect to training that will be discussed later. The city has seven zones which are organized geographically, with patrol distribution of approximately 5 officers per zone per shift. The department has its own staffed training division, consisting of a captain, sergeant, and three sworn officers who work full-time at the division. KDPS training division runs its own internal fire academy to augment the police training normally conducted externally prior to hire. The division is responsible for administering periodic in-service training and compulsory training to maintain certification in various police and fire skills. Active and current training files are maintained for each sworn officer in both electronic and hard-copy formats, consisting of past department-administered training as well as documenting any external training they have received. The department conducts periodic assessments of training strength, mostly on an informal basis due to agency size and familiarity with patrol behavior and current needs. For example, the training captain recently requested that patrol lieutenants meet with their platoons to gain input on future training ideas, and respond by email as to what topics the patrol officers would like to see on their upcoming training agenda. Responses by email contained suggestions that were incorporated quickly into the department’s training plans for the following year. The division itself has undergone a number of changes since 2011 which have impacted their training strength and commitment to in-service training. The captain of the training division is new to the position, having transferred from another division within the department, infusing the training division with new leadership and laying out a new vision for the future of the department’s officers. Also, multiple other personnel are new to the training division, bringing a new sense of creativity to a division that has been tasked with designing accelerated 44 training on an internal basis. One of the new additions to training protocol is the creation of a training calendar that reflects the department’s training plans months in advance. The calendar is a visible and symbolic indicator of change within the division towards a more proactive, planned training method, and includes input from middle-managers, line-level officers, and training division officers. Additionally, the department has invested funds in new technologies for training, such as a new interactive computer-based system and a life-sized simulation-city used for fire and police tactical training at a remote location. Two other developments are important in terms of organizational history at KDPS. In the summer of 2011, the division put the department’s most recently hired field training officers (FTOs) through a refresher course, and initially considered sending the officers to an outside course which would have cost the department. However, after meeting with a senior field training officer, it was determined that the refresher could be conducted internally at nominal cost, and reflect agency-specific goals. Without guidance from external agents or experts, the course targeted the development of several meta-competencies and adult learning attributes familiar to adaptive expertise, such as asking the field training officers to “name your top five attributes as an FTO” and using andragogical learning as the footprint for many in-class activities and tasks. The officers were also asked to consider failures on the part of their trainees as potential learning opportunities, mirroring many components of error management training. Additionally, the officers were required to make inventories of skills they wanted to enhance, as well as attributes they wanted to discard such as “ten bad habits of listening”. The course appeared to develop adaptive expertise among field training officers, yet without the instructor knowing what such training actually was. It demonstrates that training innovation, as well as the foresight to identify internal experts in order to supply such training, is not foreign to KDPS. 45 A second and more tragic circumstance occurred in the spring of 2011 as an officer at KDPS was killed in the line of duty, the first such incident in Kalamazoo’s history. As a result of this event, with no formal mandate, the training division began to reassess its firearms and tactical shooter training in order to infuse it with more high-impact learning experiences. One such training required officers on a firing line to empty their weapons of ammunition and “dry fire”, using standard firearms training procedures, on a live person, in this case a training officer. This training task was devised in response to the recent disclosure that some officers in the department had never pointed a weapon at a live person before, and may potentially feel stress doing so in high-stakes tactical situations. Again, this anecdotal experience demonstrates a commitment to change within the training division. In both instances, the department performed a number of tasks informally that are formalized in the training project, suggesting organizational familiarity with methods of training for adaptive performance. The identification of internal experts, assessment of needs, and implementation of fresh ideas in training were articulated by the agency as investments in organizational growth, calling to mind many theories of organizational learning. With the addition of new leadership, the training division wanted to conduct more empirical assessments of training strength leading to more evidence-based training designs. The articulation of the informal processes normally used in the training division was at the heart of the study. Such informal processes have rarely been examined in the context of an actual police workplace training scenario. Additionally, pre and posttest measurements were used to more accurately determine the strength of training in inculcating knowledge and skills in officers. Another potential improvement was to strengthen and expand opportunities for practice of trained skills in structured contexts through the training division. In an earnest effort to make 46 these changes more responsive to officers’ needs, the division conducted ride-alongs with officers throughout the time period after the loss of their officer in the spring and summer of 2011 as an informal needs assessment in order to inform the future training calendar. The department as a whole has shown a willingness to invest necessary resources in training improvements, including engaging with external academic assistance to implement new training ideas. The Training Study: Structure and Content The project was a within-site case study using a mixed method approach to measuring adaptive performance training over a three-month period, using the KDPS training structure as a naturally-occurring format for a pretest-posttest design. The project has two stages: a qualitative case analysis portion, conducted both before and after a three-month pretest-posttest evaluation where a different training module is tested in each of the three months. The case analysis portion precedes the distribution of the pretest in order to outline principles of adaptive expertise. By laying out the two stages in detail, research questions, variables, and testable hypotheses, as well as evaluation techniques, are discussed in this summary. A mixed methodology was used for the project to provide multiple forms of data to answer interrelated research questions as a pragmatic approach to evaluating training (Johnson & Onwuegbuzie, 2004). This strikes a balance between the more theoretical demands of educational and training research and the more practical considerations faced by KDPS. As stated, the study used two interrelated research questions addressed at various points by the mixed methodology. Rather than each research question being targeted by a single specific methodology, data gained from the overall project contributed to a data pool from which conclusions about research hypotheses were drawn. The following section details the manner by 47 which the methodology addressed research questions, both to assess training strength and to portray agency effort to initiate a training program. The first research question, asking the effect of specific training components on prerequisites for adaptive performance, was answered by collecting both quantitative data detailing changes in perceptions of performance as well as qualitative data regarding trainee performance. Assessing the impact and effectiveness of training for adaptive expertise entailed looking at pre- and posttest survey data derived from training experiences, as well as observing training in action, and conducting interviews with trainees to determine impressions of training effect and context of training changes. The second research question, asking how in-service training may be delivered to fulfill macro-level organizational needs, was addressed by placing the Kalamazoo project in the context of specific organizational changes over time, linking the addition of principles of adaptive expertise to more long-term trends in training behavior department-wide. To answer this question, the project conducted an advisory panel which revealed organizational commitment to training changes, and posttest interviews were conducted to reveal impressions of training changes over time. Additionally, data gathered from the surveys, observations, and interviews allowed for further context of identified organizational commitments to change. The project reiterated Johnson and Onwuegbuzie’s (2004) statement that reinforcing experiments with interview and observation components may result in an enhanced sense of meaning, broader context, and a more credible assessment of causality. Research question one: The effect of training on performance. Although there is overlap with regard to which research questions are addressed by distinct methodologies, the main focus of the project was on training effectiveness, primarily targeted by a pretest-posttest exercise involving training for adaptive expertise. Three distinct training modules, delivered 48 over three months, were designed to test different components of the adaptive expertise taxonomy, linking each component to their effect on expert-level performance measures. The pretest-posttest design conformed to the in-service training structure used by KDPS in order to take advantage of a naturally-occurring training opportunity to test the training modules. KDPS requires at least two hours of in-service training be given to each of its 205 sworn officers each month using a staggered schedule so that each officer receives training in a given topic area at some point in a given month. (An example of this schedule is provided in Appendix 1.) Different topics of instruction are taught each month, such as firearms training one month, computer simulation training the next month, and so on. The decision to provide two hours of in-service training per month accounts for work scheduling and backfill restrictions as training occurs when officers are on duty. Some training topics are augmented by suggested practice and supplement, such as in the summer of 2011 when the training division held rifle qualification and recertification and suggested to each rifle-carrying officer that they practice on their own while on duty, providing them a shooting range to do so. Research questions/hypotheses associated with the pretest-posttest. Over a three-month period, three distinct training modules (tactical firearms, active shooter planning, and a computer-based simulation training evolution) were delivered which reflect different components of the adaptive expertise taxonomy. Each training stage added strategies for adaptive performance as selected by training division staff in accordance with taxonomies of adaptive expertise and department training objectives and principles. Modules targeted a variety of both open and closed skills, allowing for comparison between strategies training both skills sets with principles of adaptive expertise. The project embedded specific training strategies in each of the three consecutive training modules to test these associations, such as whether open 49 skills (preparing trainees for rapidly-evolving environments using more perceptive skills) or closed skills (used in stable and predictable environments in self-paced situations) were affected by adaptive training. Research questions and hypotheses pertaining to the pretest-posttest design are outlined below. Specific variables used in modeling strategies for adaptive performance are indicated. These strategies were identified as having potential merit in inculcating the prerequisites for adaptive performance in training environments by Lazzara et al. (2010) and Pulakos et al. (2000), and are strategies familiar to trainers in the KDPS training division. Specific research questions are: 1. What is the effect of realistic simulation-based training on one’s ability to handle work stress? In police contexts, technology has assisted in creating more realistic scenarios for police officers to train for a variety of skills. KDPS wanted to extend those technologies further to create simulative environments that may prepare officers for unexpected situations. 2. What is the effect of structured practice on one’s ability to respond to and process errors? The opportunity to practice trained skills allows for application, repetition, and reflection (Chi, 2010). By implementing structured practice, the KDPS training division fulfilled both its informal assessment of needs and prerequisites for adaptive performance. 3. What is the effect of pre-training orientation on one’s ability to recognize cues and identify problems? The use of pre-training orientation has been theorized to improve metacognition, assist recognition of mental cues and models, and frame information 50 consistent with sensemaking (White et al., 2005). It allows for conceptual as opposed to procedural focusing, and aids in the processing of information (Hellen, 2009). 4. What is the effect of planning and forecasting on one’s ability to develop contingency plans? Modeling using planning and forecasting for unforeseen circumstances allows for the development of multiple potential solutions to complex problems (Lazzara et al., 2010). The KDPS training incorporated these opportunities at intervals throughout the training regimen to allow officer development of contingency plans. 5. What is the effect of training for interpersonal, cultural, or physical adaptability on one’s ability to anticipate unforeseen crisis situations? Taxonomies of adaptive expertise indicate that training individuals to anticipate encounters with diverse people enhances flexibility, communication skills, and objectivity (Pulakos et al., 2000). 6. What is the effect of self-evaluation on one’s ability to process complex information? Self-evaluation aids in metacognition by allowing the trainee to reflect, analyze, and create workable alternatives to failure (Pulakos et al., 2000; Tews & Tracey, 2008). These self-evaluation methods may take multiple forms when translated into training strategies: they may be verbal or written, or occur at various stages of the training (Pulakos et al., 2006). In each individual research hypothesis, dependent variables derived from theoretical sources were embedded in the department’s training objectives for that month. Mangos and Johnston’s (2008) confirmatory evaluation model (which will be discussed in a later section) states that knowledge, skills, abilities, and other characteristics are measurable, observable taskrelated events that take place in the course of training scenarios. Performance measures were embedded within the individual training scenarios in the confirmatory measurement model 51 suggested by Mangos and Johnston (2008), and the process of generating these measures themselves informed the dependent variables in the five training modules. Table 2.1 and 2.2 outline these variables and their relationship to observable behaviors and training strategies fulfilled in each month’s training exercise. The training division had previously implemented some features of these designs on an ad-hoc basis: for instance, a past training module allowed for feedback and reflection only because time allowed. However, for the purposes of this project, these modules were formalized in order to adhere to the pretest-posttest design. Tables 2.1 and 2.2 demonstrate how training strategies were crafted to reflect research hypotheses, and how they were translated into outcome measurements derived from prerequisites for adaptive performance and taxonomies of adaptive expertise training (Ericsson & Lehmann, 1996; Hellen, 2009; Lazzara et al., 2010; Pulakos et al., 2000; Pulakos et al., 2006; White et al., 2005). The first column in each table lists research hypotheses consisting of strategies and outcomes desired. In Table 2.1, the second column identifies what specific training activity was used to test each research hypothesis. The training module associated with these tests is shown in column 3, with literature sources shown. In Table 2.2, Mangos and Johnston’s (2008) confirmatory performance measurement guidelines were used to craft three specific dependent variables reflected on the measurement instrument (a sample of which is provided as Appendices 2 and 3). The specific relationship between the dependent variables and the training objectives for each module are detailed in the third column to visualize how the dependent variables were operationalized. In the fourth column, the research hypotheses are explained fully as associations between training strategies used and the dependent variables are outlined. 52 Table 2.1. Research hypotheses and variables. Research hypotheses Training strategy as independent variable Training module(s) associated with this test Source H1: Realistic simulation training has a positive effect on officer perception of one’s ability to handle work stress. Realistic simulation activity Module A (July): Firearms training Lazzara et al., 2010; Pulakos et al., 2000 H2: Structured practice opportunities supported by reflection have a positive effect on officer ability to respond to and process errors. H3: Pre-training orientation has a positive effect on officer ability to recognize cues and identify problems. Structured practice component Module A (July): Firearms training Ericsson & Lehmann, 1996; White et al., 2005 Pre-training orientation (advance lesson objectives and activities) Module B (August): Active Shooter Planning Lazzara et al., 2010; White et al., 2005 H4: Planning and forecasting exercises have a positive effect on officer ability to develop contingency plans. H5: Training strategies for diversity have a positive effect on officer ability to anticipate unforeseen crisis situations. Planning activity Module B (August): Active Shooter Planning Lazzara et al., 2010 Training strategy targeting cultural, interpersonal, or physical adaptability Module C (September): MILO Pulakos et al., 2000 H6: Self-evaluation exercises have a positive effect on officer ability to process complex information. Self-evaluation activity Module C (September): MILO Hellen, 2009; Pulakos et al., 2000 53 Table 2.2. Expansion of dependent variables into observable behaviors. Research hypotheses Dependent variables Relationship between DV and training objectives Association between DV and training strategies H1: Realistic simulation training has a positive effect on officer perception of one’s ability to handle work stress. Understanding how to reduce stress, knowledge of stressreduction strategies, confidence in reducing stress Understanding the role of errors, knowledge of errormanagement, confidence in ability to regulate errors Understanding problem recognition, knowledge of problem-solving strategies, confidence in problem solving The department must craft objectives which in part target stress-coping mechanisms in officers The department must craft objectives which target error recognition and management Simulation activities create stressful environments, allowing trainees to develop stress-coping mechanisms Structured practice assists trainees in determining errors and managing them in field situations The department must craft objectives which target the recognition of cues and identification of problems Orientation about upcoming training provides trainees with a blueprint for recognizing cues and identifying problems Understanding the use of contingencies, knowledge of contingency plans, confidence in using contingency plans Understanding diversity, knowledge of crisis management strategies, confidence in ability to respond to unexpected crises The department must create objectives which target the development and use of contingency plans The department must craft objectives to develop crisismanagement skills using physical, cultural, and interpersonal adaptability Forecasting “worst-case scenarios” allows trainees to develop contingency plans for such events Understanding complex information, knowledge of information-processing, confidence in ability to process information The department must craft objectives which target the ability to process complex information Training an officer to use selfevaluation as a tool assists them in processing complex information H2: Structured practice opportunities supported by reflection have a positive effect on officer ability to respond to and process errors. H3: Pre-training orientation has a positive effect on officer ability to recognize cues and identify problems. H4: Planning and forecasting exercises have a positive effect on officer ability to develop contingency plans. H5: Training strategies for diversity have a positive effect on officer ability to anticipate unforeseen crisis situations. H6: Self-evaluation exercises have a positive effect on officer ability to process complex information. 54 Training an officer to recognize and deal with diverse individuals assists them in dealing with unexpected crises Design issues pertaining to experimental training evaluations. Experimental training evaluations conducted in the field often contain inherent design issues that the pretest-posttest component of this study attempts to mitigate. A primary concern regarding the use of experimental and quasi-experimental designs is the ethical limitation of administering differential training programs in the field, raising concerns about differential treatment regarding later replications of training given to control groups (Cigularov et al., 2008; Haccoun & Saks, 1998; Quinones & Tonidandel, 2003; Salas et al., 2003). Both concerns arose in the original study plan for this project. Using the site as a setting for the case study necessitated making compromises in the study design. While there is a naturally-occurring opportunity for a control group given the staggered in-service training schedule used by KDPS (it takes 30 days for the entire department to be fully trained, and by the mid-way point of each month about half of officers have not yet received training), there did not exist an opportunity to replicate training for future control groups (given the limited attention given to specific topics over the calendar year) without prohibitive disruptions to the training schedule. If officers received different trainings as treatment and control comparisons over the course of one month, there would have been no opportunity to deliver the same training received by the treatment group to the control group. As with other past training evaluations conducted in field settings, it is also possible that there are legal obstacles to withholding beneficial training from certified law enforcement personnel, despite assurances that training would be delivered at a later date (Cigularov et al., 2008). This limitation required negotiation with KDPS training staff in the earliest planning stage in order to resolve study design issues. A one-group pretest-posttest design, while not as strong as a control-group design, constituted a compromise in order to conduct the case study in a field setting. 55 The pretest-posttest component adopted a method utilized by Cigularov et al. (2008) in an evaluation of suicide prevention training, and Hauck et al. (2011) in their evaluation of fatigue countermeasures training directed to flight attendants. Both of these studies use a one-group pretest-posttest design, yet additional measures were captured which accounted for the lack of control groups. Haccoun and Saks refer to this method as “piggy-backing measures” (Haccoun & Saks, 1998, p. 36), an appropriate visual descriptor of the exercise of capturing reaction measures in pre- and posttests that reflect cognitive, skill-based, and affective domains as well as nonequivalent dependent variables used to strengthen conclusions. Nonequivalent Dependent Variable (NEDV) analysis. and the study used a nonequivalent dependent variable analysis, also called an internal referencing strategy (IRS) (Haccoun & Hamtiaux, 1994), mimicking Cigularov et al.’s (2008) efficacy study of suicide education training in a public school setting, which similarly could not provide for control groups. The approach was also partially based upon Kirkpatrick’s (1998) criteria for training evaluations. The criteria categorized training evaluation questions according to what specific information about the training process researchers wanted to know. Kirkpatrick’s criteria 3 served as the underpinning for the use of nonequivalent dependent variables, as elaborated in a subsequent section outlining research questions and their relationship to data collection. The use of pretest-posttest measurements, therefore, was a blanket technique wherein the evaluation strategy addresses specific questions asked in each of the assessments; the evaluation strategy was therefore embedded in the pretest and posttest instruments. 3 A third criteria of Kirkpatrick’s (1998) evaluative paradigm, how training affects the organization, is not captured in the pretest-posttest, but will be measured through the qualitative component. 56 The internal referencing strategy (IRS) measured trainee achievement on items that are both included and not included in the training design; this provided not only a portrait of how much change occurred as a result of training, but used untrained items as a control for trained items. The IRS incorporated nonequivalent dependent variables (NEDV) which served multiple purposes. NEDV guard against specific internal and construct validity threats which may distort findings, and adhere to the principle of pattern matching that allows for causal inference (Shadish et al., 2002; Trochim, 1985). By eliminating a number of plausible alternative hypotheses, namely those expressed in NEDV relationships, the model can clarify the treatment effect (in this case, the effect of training on outcomes) (Haccoun & Hamitaux, 1994; Shadish et al., 2002). In program evaluation research, NEDV are used to construct patterns of relationships between program components and outcomes, often suggested by theory (Coryn & Hobson, 2011). Adaptive expertise taxonomies do not indicate whether or not one or more components are needed to obtain suggested outcomes: training for adaptive expertise is seen as a progressive and cumulative program of interrelated components (Pulakos et al., 2006). The project intended to test each of these components separately. Therefore, the use of NEDV further clarified the relationship between the adaptive expertise taxonomic agenda and outcomes, barring alternative explanations that may exist in training processes. The effectiveness of training is indicated if pretest-to-posttest changes on trained items are greater than pretest-to-posttest changes on untrained items. The mean differences from pretest to posttest on trained material should exceed those from untrained material (Cigularov et al., 2008). The specific NEDV that were used in the model will be discussed in the next section. There were other advantages to using an internal referencing strategy. The use of nonequivalent dependent variables strengthened assumptions regarding results on items that are 57 trained, specifically in this small-sample study. Haccoun and Saks (1998) discuss the likelihood that the use of control groups in small-number evaluations may potentially lead to inaccurate statements about findings, and that pretest-posttest designs might actually be a preferred method of evaluation if type-I error threats are accounted for in the initial design. Quinones and Tonidandel (2003) state that experimental training conditions are at best a synthetic approximation of applied field contexts, a blanket statement that Salas et al. (2003) agree often constitutes a fallacy in organizational assessment of training. Haccoun and Saks acknowledge that techniques such as an internal referencing strategy are “far from perfect… [but] point to the necessity of developing models that are realistic for use in real organizational contexts” (Haccoun & Saks, 1998, p. 35). The design and training strategy, including pretests and posttests, is graphically depicted in Table 2.3. Each row indicates a different training module, depicted as X1, X2, and X3. The diagram shows that the pretests collecting different measures O1A and O1B were used prior to training as a baseline for comparison. The posttest (O2A, O2B) was given at the end of the completed three-month training regimen. In the case of subscripts, A represents a vector of variables to trained items, and B represents a vector of NEDV, or untrained items. Table 2.3. Diagram of one-group pretest / posttest design. Module A (July): [O1A , O1B] X1 ---------------------------------------------------------------------------------------------------------Module B (Aug.): X2 ---------------------------------------------------------------------------------------------------------Module C (Sept.): X3 [O2A , O2B] 58 Nonequivalent dependent variables in the model. In order to enhance the strength of the internal referencing strategy, three nonequivalent dependent variables were used: officer ability to utilize officer safety strategies, officer familiarity with community policing, and officer familiarity with reporting strategies (each is operationalized similarly with the use of the confirmatory evaluation model as “understanding”, “knowledge”, and “confidence” in each). There was theoretical and practical justification for selecting these variables as NEDV, as they are conceptually related to the dependent variables in the model, yet were not assumed to be affected by training. Each variable, however, was likely to be affected by factors outside of training, such as officer familiarity with trained strategies, attending external training on one’s own, experience in the field, motivation to train, and history and maturation. These internal validity threats (which are discussed in detail in a subsequent section) intended to clarify pattern relationships between variables. Table 2.4 shows how NEDV were included in the model and how their presence was expected to deliver a clearer interpretation of associated relationships between the training and outcomes. The hypothesized relationships between variables are shown in column 3 of Table 2.4, illustrating the anticipated impact of introducing each NEDV. 59 Table 2.4. Inclusion of nonequivalent dependent variables for internal referencing strategy (IRS). Research hypotheses Dependent variables Hypothesized relationships with nonequivalent dependent variable included H1: Realistic simulation training has a positive effect on officer perception of one’s ability to handle work stress. Understanding how to reduce stress, knowledge of stressreduction strategies, confidence in reducing stress Understanding the role of errors, knowledge of errormanagement, confidence in ability to regulate errors Realistic simulation training has a positive effect on officer perception of one’s ability to handle work stress, but no effect on officer understanding, knowledge, or confidence in officer safety strategies, community policing, or reporting procedures Structured practice opportunities supported by reflection have a positive effect on officer ability to respond to and process errors, but no effect on officer understanding, knowledge, or confidence in officer safety strategies, community policing, or reporting procedures Pre-training orientation has a positive effect on officer ability to recognize cues and identify problems, but no effect on officer understanding, knowledge, or confidence in officer safety strategies, community policing, or reporting procedures Planning and forecasting exercises have a positive effect on officer ability to develop contingency plans, but no effect on officer, understanding, knowledge, or confidence in safety strategies, community policing, or reporting procedures Training strategies for diversity have a positive effect on officer ability to anticipate unforeseen crisis situations, but no effect on officer understanding, knowledge, or confidence in safety strategies, community policing, or reporting procedures H2: Structured practice opportunities supported by reflection have a positive effect on officer ability to respond to and process errors. H3: Pre-training orientation has a positive effect on officer ability to recognize cues and identify problems. H4: Planning and forecasting exercises have a positive effect on officer ability to develop contingency plans. H5: Training strategies for diversity have a positive effect on officer ability to anticipate unforeseen crisis situations. H6: Self-evaluation exercises have a positive effect on officer ability to process complex information. Understanding problem recognition, knowledge of problem-solving strategies, confidence in problem solving Understanding the use of contingencies, knowledge of contingency plans, confidence in using contingency plans Understanding diversity, knowledge of crisis management strategies, confidence in ability to respond to crises Understanding complex information, knowledge of information-processing, confidence in ability to process information 60 Self-evaluation exercises have a positive effect on officer ability to process complex information, but no effect on officer understanding, knowledge, or confidence in officer safety strategies, community policing, or reporting procedures The selection of officer safety as a NEDV in the IRS model was supported in multiple ways. The KDPS training division does not isolate officer safety strategies as a stand-alone topic of instruction, mirroring the belief that such strategies should instead be infused into curriculum for all instructional topics, and is included as an overall instructional goal throughout an officer’s training regimen. For instance, when KDPS conducts a training course dealing with domestic violence situations, it would be customary for officer safety strategies to be discussed in debriefing, but the strategies would not be an explicit part of the instructional plan. The department feels that officer safety should permeate all police-public encounters, yet is not taught in isolation from other content topics. Because of this, officer familiarity with safety strategies, experience, disposition to train on one’s own, and history could possibly impact the ability of officers to regulate the use of officer safety strategies. The KDPS experience of not isolating officer safety as a stand-alone topic of instruction is also supported by Rojek et al. (2007), which examined each of the fifty states’ POST requirements and found that the methodology employed by KDPS is consistent with state POST board requirements for licensure on similar tactical requirements. Therefore, the use of officer safety as a NEDV has solid theoretical background. Similarly, the use of community policing as a NEDV in the model was reinforced by organizational familiarity with infusion of this concept into the department’s in-service training regimen across several topics. Community policing strategies are mirrored in training topics as diverse as tactical firearms training (which often includes contact-and-cover scenarios), administrative reporting, cultural diversity, and incident preplan. As with officer safety, this variable meets the expected criteria for NEDV by not constituting a stand-alone instructional 61 topic, but an item that was conceptually to trained topics but would not likely be affected by changes in training. The final NEDV chosen was reporting procedures, which were embedded in other training topics. It is unlikely that special training in this topic would take place beyond the initial academy stage at KDPS, as on-the-job practice in such reporting techniques, as well as field training, grant officers an opportunity to practice reporting procedures. However, the need to create written after-action reports is emphasized in in-service training as a debriefing exercise (as noted in 2011 observations of firearms training by the researcher). As Rojek et al. (2007) also pointed out, it is rare that reporting procedures are taught as isolated topics of instruction at the in-service level, partially due to the expectation that experienced officers will have mastered this critical communications skill early in their careers. Again, this NEDV was utilized to demonstrate that changes in training may not affect untrained phenomena. Training content. The structured training activities outlined in Table 2.1 reflected lesson plans and training objectives crafted by the KDPS training division for that month. The department’s initial familiarity with principles of adaptive expertise was strengthened through the use of the advisory panel, which further informed training division staff about how specifically such training was delivered beyond the written lesson plan. One training division officer was responsible for delivering the training, as is normally the case at KDPS. The written lesson plan reflected strategies that the officer created. Instructor discretion and logistical changes may therefore have altered the training process at the point of delivery. Any fluctuations or inconsistencies may have constituted an implementation threat to internal validity: these were documented in the observation notes for each training task. Observation 62 assisted in the verification process to ensure consistency with the implementation adaptive expertise strategies. Each training module is summarized below. Module A: Firearms training. The first module tested both a realistic simulation component and a practice component. The training topic for Module A was tactical firearms training, where realistic simulated experiences were created by the training officer as evolutions which each trainee “experienced” based upon given information, such as situational encounters and live firearms use. Also in this module, a structured practice component involved setting time aside within firearms training evolutions for structured practice, deliberation, and discussion with training staff. The department has both an indoor and outdoor shooting range used periodically for practice. The training design formalized this by embedding practice as a training task within the module, allowing time for training staff to discuss questions on a case-by-case basis with trainees, and allowing for posttraining reflection on practice and errors. As described in Tables 2.1 and 2.2, Lazzarra et al. (2010) and Pulakos et al. (2000) state that realistic simulation training allows trainees the ability to develop stress-coping mechanisms in an effort to regulate stress, contributing to adaptive expertise. Also in Tables 2.1 and 2.2, Ericsson & Lehmann (1996) and White et al. (2005) state that practice opportunities have a positive effect on trainee ability to respond to and process errors, contributing to the development of adaptive expertise. The first research hypothesis (H1) examined is: realistic simulation training has a positive effect on officer perception of one’s ability to handle work stress. The second research hypotheses associated with Module A (H2) is: structured practice opportunities supported by reflection have a positive effect on officer ability to respond to and process errors. As with Modules B and C, pre- and posttest instruments (Appendices 2 and 3) were used to collect data pertaining to hypotheses H1 and H2. Formatting and question 63 arrangement were consistent across all modules, and utilized Schaeffer and Presser’s (2003) survey criteria, Mangos and Johnston’s (2008) confirmatory performance measurements, and contain NEDV. As illustrated in Table 2.2, Module A training curricula developed by KDPS training staff map to dependent variables in the research model through graduated stress application, a variety of realistic simulation training in which the gradual introduction of stressful events, scenarios, and environments is used to elevate trainees’ situation awareness (Endsley, 1995; Feldman, 2004). It is theorized that the addition of stressors in a gradual manner will expand working memory capacities of trainees, leading to increased self-perceptions in one’s ability to regulate stress and learn from mistakes. The survey instrument asked trainees to self-report changes in self-efficacy to respond to stress in tactical environments, and changes in self-efficacy in processing and responding to errors encountered as stress was gradually increased (Frensch & Runger, 2003). The dependent variable in this section measures changes in self-reported ability to manage stress and process errors in accordance with graduated stress application training strategies. The measures themselves, although created for this specific project by the researcher and not replicated verbatim from previous studies, patterns graduated stress application training objectives in elevating stress-management and error processing (Feldman, 2004). Survey measure quality across all 3 modules was in part evaluated through the computation of reliability coefficients which are discussed later, but an additional gauge of measure quality is the close association with the measures with curriculum items, assessed by either a review panel or subject-matter expert. Since no review panel was used, the partnership between KDPS training staff and the researcher to create items reflective of graduated stress application objectives was a strong indicator of measure association with curriculum components. 64 Module B: Active shooter planning. Active shooter planning refers to pre-plans devised by trainees in order to visually and tactically assess physical spaces (such as schools, retail stores, and public space) where a potential active shooter may strike. KDPS adopted this as a training plan, as explained later in the section on project implementation, as a response to current events that transpired during the actual research project. In this module, trainees were assigned the task of producing physical assessments of such spaces in their patrol area in the event of an active shooter, and were asked to view a video describing many of the considerations for this activity. Trainees were sent an email prior to training containing lesson objectives, familiarization activities such as recall exercises, and questions for the trainees to think about to stimulate preparedness for the training exercise. According to Lazzara et al. (2010) and White et al. (2005), pre-training orientation exercises assist trainees in recognize cues which aid in problem-solving. Also incorporated in this module, Lazzara et al. (2010) state that planning and forecasting techniques enhance trainee ability to develop contingency plans. The second hypothesis involves the testing of planning and forecasting as a tool to the development of contingency plans, which was the content of this exercise. The first research hypothesis associated with Module B (H3) is: pre-training orientation has a positive effect on officer ability to recognize cues and identify problems. The second research hypothesis associated with Module B (H4) is: planning and forecasting techniques have a positive effect on officer ability to develop contingency plans. Pre- and posttest instruments, reflected in Appendices 2 and 3, were used to collect data pertaining to these research hypotheses. The survey questions were similar in format to Module A, and were likewise derived from Schaeffer and Presser’s (2003) survey research criteria, Mangos and Johnston’s (2008) confirmatory performance measurements, and contain NEDV. 65 Module C: MILO simulations. MILO (Multiple Interactive Learning Objectives) is a computer simulation training device used by KDPS training division approximately biannually which produces simulated video-based tactical encounters that officers react to in a controlled training environment (the system is installed in a room at the training division). In Module C, each officer encountered five MILO video simulations in order to strengthen decision-making skills. In the training module, trainees encountered individuals from a variety of physical, cultural, and interpersonal situations, such as a mentally challenged person, a non-English speaker, and an uncooperative person. Also in this module, trainees were given an opportunity to evaluate their own performance on simulated exercises as a follow-up debriefing exercise. In Tables 2.1 and 2.2, Pulakos et al. (2000) state that strategies that train for the expectation of cultural, physical, and interpersonal diversity have a positive effect on trainee ability to anticipate unforeseen crisis situations, which contributes to the development of adaptive expertise. Also according to Pulakos et al. (2000) and Hellen (2009), self-evaluation exercises have a positive effect on trainees’ abilities to process complex information. The first research hypothesis associated with this module (H5) is: training strategies for diversity have a positive effect on officer ability to anticipate unforeseen crisis situations. The second research hypothesis associated with Module C is: self-evaluation exercises have a positive effect on officer ability to process complex information. Pre- and posttest instruments assessing these and other subsequent hypotheses are also in Appendices 2 and 3. The manner by which Module C curriculum maps onto survey measures is again associated with the development of situation awareness in trainees, except in this module there is a greater emphasis on the specificity of the measures with regard to what constitutes “training for 66 diversity”. Diversity training has become somewhat of a cliché in recent years in American policing, but there is a clear indication in literature that, when such training contributes to the diffusion of apathetic or complacent states of awareness, there can be clear benefits for enhanced crisis management skills for trainees (Feldman, 2004; Stewart, 2012). Measures in Module Crelated questions probe trainee changes in self-reported abilities to anticipate crisis situations. Again, measure quality and association with curriculum components was evaluated by KDPS training staff in addition to reliability coefficients which are detailed later. Taken in its entirety, the instrument contains questions which asked trainees to reflect on the training’s perceived ability to affect the performance of tasks related to adaptive expertise. Mangos and Johnston’s (2008) confirmatory performance measurement was used to frame trainees’ readiness to perform specific tasks: the instrument measured if the trainees understand what was involved, had knowledge of specific strategies, and had confidence or self-efficacy in performing actions consistent with adaptive expertise behaviors. These measurements were reported on a seven-point Likert scale. The instruments also contained affective and utility measures designed to elicit trainee reactions to whether or not they enjoyed the training, whether or not they actually performed training tasks, their impressions of trainers’ approaches to training, and perceived utility of training. Training-irrelevant measures (NEDV) were included in the instrument and followed the same methodology as training relevant measures in order to fulfill requirements for the IRS analysis. All data collected from pretest and posttest instruments for the internal referencing strategy in modules A-C are listed in Table 2.5. 67 Table 2.5. Items and dependent variable measures for IRS collected in Modules A-C. Demographic variables and levels of measurement: Years employed at KDPS Years of total police service (Less than 5, 5-10 years, 11-15 years, 16-20 years, over 20 years) Age (21-30 years, 31-40 years, 41-50 years, over 51 years) Ethnicity (White, African-American, Hispanic, Asian/Pacific Islander, Native American, self-reported as mixed) Gender (Male, Female) Highest level of education attained (High school, some college, Associates’ Degree, Bachelor’s Degree, Graduate Degree or other advanced) Hours of training practice on-your-own per week (0, 1-4, 5-8, 9-12, 14+) Dependent variable measures: 7-point Likert scale used (1= strongly disagree, 7=strongly agree) Module A: Training-relevant measures H1: Understanding how to reduce stress H1: Knowledge of stress-reduction strategies H1: Confidence in reducing stress H2: Understanding the role of errors H2: Knowledge of error-management H2: Confidence in ability to regulate errors Module B: Training-relevant measures H3: Understanding problem recognition H3: Knowledge of problem-solving strategies H3: Confidence in problem solving H4: Understanding the role of contingencies H4: Knowledge of contingencies H4: Confidence in ability to use contingencies Module C: Training-relevant measures H5: Understanding crisis management H5: Knowledge of crisis management strategies H5: Confidence in ability to respond to crises H6: Understanding contingency plans H6: Knowledge of contingency plans H6: Confidence in contingency planning All 3 modules: Training-irrelevant measures H7: Understanding community policing strategies H7: Knowledge of community policing strategies H7: Confidence in using community policing strategies H8: Understanding purpose of safety strategies H8: Knowledge of safety strategies H8: Confidence in using safety strategies H9: Understanding how to complete reports H9: Knowledge of reporting procedures H9: Confidence in completing reports Affective and utility measures from first posttest 7-point Likert scale used (1= strongly disagree, 7=strongly agree) Modules A-C: Does trainee like the training received Does trainee feel training is headed in the right direction Does trainee find the training useful and applicable Did trainee read all materials in preparation for training Open-ended qualitative questions, posttest What was most/least valuable from the training modules (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) (pre & post) Sources: Haccoun & Saks, 1998; Mangos & Johnston, 2008; Schaeffer & Presser, 2003 68 The use of three consecutive modules explained. As demonstrated by Table 2.5 and reiterated in a previous section, the use of three consecutive modules of training served multiple purposes in disentangling components of adaptive expertise in a field setting. First, measuring distinct features of adaptive expertise over a three-month period allowed the project to open the proverbial “black box” of training content to examine the specific cognitive processes at work by which learners construct schema needed to process complex information (Chen & Rossi, 1984; Sweller et al., 1998). This ability is directly related to prerequisites for adaptive expertise. “Adapting” from one context to another requires the learner to apply learned information to complex problems through the use of schema which offsets cognitive load, keeping working memories of trained individuals sufficiently balanced to make applications of learned skills to novel situations (van Merrienboer and Sweller (2005). The ability of individuals to process complex information in working memory is limited to the construction of mental models which are dependent upon multiple factors, including familiarity with content and training process (Sweller et al., 1998; van Merrienboer & Sweller, 2005). In the study design, examining the ability of the trainee to construct schemas which process complex information can lead to a better understanding of which training strategies assist in the transformation of principles of adaptive expertise into tangible training outcomes. Second, the use of three months of training topics reflected Heggestad and Kanfer’s (2005) assertion that training is a cumulative process. In a training program, tasks can be interrelated because training is designed to build upon prior knowledge, even if recently trained. Therefore, performance itself can be related to “transient factors” which are the amalgamation of training processes over time (Heggestad & Kanfer, 2005, p. 85). Specifically when measuring self-efficacy of individuals as a result of training, it is imperative to differentiate between these 69 joint influences over time. By isolating principles and strategies of adaptive expertise over successive training operations, the order in which these cumulative processes occur is accounted for in the analysis, and the specific addition of modules over time reveals how individual components contribute to performance. Evaluation. As indicated in Table 2.5, pre- and posttest instruments were administered to trainees prior to and following training. Sample measurement instruments for each of the three modules are provided in Appendices 2 and 3. In the pretest instrument, items related to trainee demographics (years employed at KDPS, total years of total police service, respondent age, ethnicity, gender, respondents’ highest levels of education attained, and their self-reported hours of training on-their-own per week) were included. Trainees were asked specific questions regarding their ability to form cognitive constructs associated with the goals of adaptability in that specific training module. Mangos and Johnston’s (2008) confirmatory model measured trainee reactions to each module’s testable item, as well as the training-irrelevant nonequivalent dependent variable of officer safety. On the posttest, testable items were measured a second time using Mangos and Johnston’s (2008) confirmatory evaluation model. Also, a section on affective and utility measures was included (Haccoun & Saks, 1998). Affective and utility measures gauged trainee reactions to the training itself and perceived trainer competency, as well as whether the training was seen as useful on-the-job. These questions were used to deepen reactions to specific training components aside from the qualitative interviews that were conducted. Prior literature on adaptive performance training provided two important guidelines for the evaluation process. First, White et al. (2005) outline a sample training evaluation instrument used for tactical military environments that provides a blueprint for the creation of the 70 instruments used in the study. Using a Likert questionnaire as well as directed open-ended responses, questions evaluated the quality and applicability of training received; these formed the foundation of the sample instruments in Appendices 2 and 3. The instruments merged the White et al. (2005) suggestions with eventual KDPS training objectives as indicated on these appendices, and provided a template for future data-driven evaluation practices. Second, the study utilized Mangos & Johnston’s (2008) confirmatory performance measurement model for evaluating the complex interrelationships between training objectives, observable performance, and self-reported outcomes. Their evaluation protocol treats dependent variables in each of the five training modules in the study as latent variables from which related, measurable performance characteristics may be individually evaluated for training impact. White et al. (2005) perform a similar task by using a column of significant measurable behaviors (the far left column in the Appendix 2 and 3 examples) much in the same way Mangos and Johnston (2008) suggest embedding performance measures directly within training scenarios. An example of this would be scripting training scenarios specifically to target training objectives, such as developing a training simulation designed to foster creative problem solving. Research question two: How training targets organizational needs. In the summer of 2011, the KDPS training division conducted an informal training needs assessment consisting of two efforts: ride-alongs conducted with officers where suggestions for training topics were fielded, and follow-up calls and emails with middle managers soliciting suggested training topics for the department. This internal effort resulted in a list of training topics officers wanted to see the training division provide as in-service opportunities, suggestions for ways in which the training division might provide these topics given the limited scheduling opportunities provided by the once-a-month schedule, and special concerns for training that dealt with female officers, 71 incentives to train externally, and performance levels. Insights gained from this needs assessment are summarized in Table 2.6. The training division conducted this needs assessment in order to gain suggestions from line-level officers as to what training expectations were for the 2012 fiscal year, but many of the officers’ comments and suggestions went beyond these practical concerns. The division decided to utilize these insights to provide curriculum ideas for the training project. Officers’ perceived need for more adaptive training was included in the insights gained from this informal needs assessment. Officers did not feel as though they were practicing their newly-acquired skills after in-service training was delivered, and contributed ideas as to how such practice may occur. Practice techniques were suggested, such as practicing in pairs with a partner and using two-person Table 2.6. Insights gained from 2011 informal training needs assessment. Problems officers saw related to in-service training Scheduling interferes with training opportunities, training division is knowledgeable but understaffed, shift-level practice opportunities are limited, lack of baseline standard Themes officers deemed important in in-service training Equitability of opportunity, credibility of trainers, practical usefulness of training, clarity of expectations, the need to practice trained topics after formal training, training should target lapses in behavior Defensive tactics and ground-fighting, officer safety, contact and cover on traffic stops, searches of persons, communications skills, core skills Suggestions for training content Suggestions for training processes Hands-on techniques, training in pairs, use of videos for peer and self-evaluation, include officers and mid-managers in training division meetings, delegated training and practice Source: Kalamazoo Department of Public Safety Informal Needs Assessment, unpublished data, 2011. 72 contact-and-cover operations. Additionally, the need for self-evaluation, reflection, peer evaluation, and empowerment in training division decision-making was noted in the needs assessment. The training division sought to implement these suggestions to improve the greater KDPS training landscape through the training project. In that sense, the project is a tool by which qualitative data may be leveraged into actual suggestions for practice, and then implemented into daily training activities. An example of this would be the inclusion of line-level officers in training division meetings: once training division staff was aware of the desire of officers to attend training staff meetings, considerations as to who may be included were made immediately. This is one example of how qualitative data was utilized in enhancing training experiences. The project made qualitative data-gathering more systematic through the use of three primary tools: an advisory panel, interviews of both training staff and officers, and observational data (Creswell, 2007). These qualitative procedures reflect complex systems theory in that the pattern of behavior assumed in the training division, and in KDPS as a whole, was not bound to linear interpretation or planning. What one observed in one day at the training division was an orderly but often spontaneous learning landscape where unexpected realizations, occurrences, and challenges were handled procedurally but flexibly, with the day’s activities flowing from a schedule of tasks on the training calendar but susceptible to unforeseen events and influences. In a sense, the challenges faced by the training division mirrored the same challenges they are training line-level officers to expect in police work, and the same adaptable behaviors used to confront these challenges in the training division were being taught to trainees. This perspective also used sensemaking as its theoretical lens, where an individual or workgroup uses complex 73 problem-solving to face unforeseen challenges, crafting new approaches which are integrated into the larger pattern of behavior that may be drawn on in the future (Bartunek et al., 2006; Rutledge, 2009). Examining workgroup dynamics of the training division stimulated questions of workgroup behavior and sensemaking which informed the analysis of qualitative data. It is helpful at this point to detail the specific nature of KDPS’ informal qualitative data collection systems, and the improvements seen once projected systematic methods were implemented in the project. In an ad-hoc, informal manner, KDPS training division has conducted routine advisory and staff meetings as part of its weekly tasks, meetings which were expected to air ideas about future training approaches and develop new methods of instruction through a sharing of experiences. Often, mid-managers from the department (usually sergeants and lieutenants) were asked to participate in an effort to keep lines of communication and suggestion open. These casual data collection techniques were also conducted by telephone and email, as the training division’s captain kept in regular contact with patrol events to invigorate training division plans for in-service training. Also, training staff consistently conducted informal observations of trainee performance during the course of training delivery, but on an ad-hoc basis. Few records of training performance were kept; the maximum focus of recordkeeping has been on hours officers spend obtaining in-service training, topics trained, and where such training occurs. While this informal system has provided rich data in the past, training division staff sought to strengthen this process and make it more systematic. As stated, the project had two sets of interrelated research questions, and each was addressed by the methodologies employed. Although the qualitative methodologies were intended to be used to place quantitative data in an organizational context, the research question most associated with the qualitative portion of the project was as follows: What is the manner by 74 which in-service training may be delivered in accordance with departmental objectives, resources, and existing strengths to achieve prerequisites for adaptive performance? This research question compelled the researcher to examine multiple elements within the culture of KDPS’ training structure, to unearth its relationship to the trainee and organizational needs, and to construct a portrait of the longitudinal character of KDPS’ training operations in order to address changes that may have occurred as a result of the training project over time. Specifically, the questions were answered by three qualitative procedures outlined below. Qualitative procedure #1: Advisory panel meetings. Researcher participation made the current informal processes more systematic by providing empirical data used to make more rigorous assessments of training character. First, preliminary meetings between training division staff and officers, mid-managers, and command staff (consistent with findings from the needs assessment) assisted in devising plans for the project, similar to an advisory panel. Three participants were selected from available staff (officers and supervisors) working during the meeting dates, mimicking a quota sample. The panel provided input to the training calendar and assistance in formulating specific training needs in the pretest-posttest phase of the project. One outcome from these meetings was the construction of an inventory of learning principles the training division will eventually use as descriptive guidelines for training for adaptive performance. The division used training objectives normally in the course of lesson planning, but designing these principles solidified overall training themes and objectives the division wished to test. Although the meeting procedure mirrored the currently existing KDPS monthly meetings, researcher residency allowed for more data collection through systematic documentation and analysis of minutes of the meetings. Additionally, researcher input allowed for open-ended questions and follow-up as needed (Creswell, 2007). Data gathered at this stage 75 detailed the training division’s familiarity with and approach to training for adaptive expertise using the taxonomies of adaptive expertise from literature as a guide. Descriptive notes were collected using a standardized observational protocol, shown in Appendix 4. Data were analyzed for thematic content and patterns emerging between discussants (Creswell, 2007). Approaches to determining the training calendar, utilization of available resources, identifying experts within the department to assist with instruction, and validating needs determined from the 2011 needs assessment were all potential points of data that were evaluated. Qualitative procedure #2: Officer and training staff interviews. Interviews with training staff and officers offered ideas for training division planning as a follow-up to the informal needs assessment conducted in the summer of 2011, and perspective about the project during and after implementation. Interviews of training division staff officers took place at the training division complex, and patrol officer interviews were conducted as ride-along opportunities. These ridealongs took place over two weeks’ residency. Two subgroups (day- and night-shift employees) were identified, and from those groups, three each of patrol officers, special operations, and supervisory staff (resulting in 18 persons per week, a total of 36 interviewees) were interviewed, ensuring that individuals from a variety of experiential, patrol area, and demographic backgrounds were selected. This strategy allowed the researcher to probe a variety of viewpoints about the character of the training division, its perceived strengths and weaknesses, and specific topics the department should train, gathering information to update the 2011 needs assessment (Miles & Huberman, 1994). The interview protocol and questions were consistent across each shift, but the interview procedure for each officer allowed for the unstructured elicitation of comments and points that arose from casual prompting (Creswell, 2007). Data were collected confidentially, and notes 76 were taken by the researcher. The length of each ride-along interview was dependent on time constraints and situational incidents, and length was subject to the discretion of the shift supervisor in the event of emergencies. Additionally, researcher observation into the context of training for adaptive expertise was drawn from the interview and ride-along experiences. Finally, the interview and observational protocols were approved by training division supervisors to ensure that information elicited was valuable to the division as part of its qualitative datagathering procedure; the open-ended questions were developed by the researcher. Training staff interviews determined developmental background of prior training efforts, and past strategies employed by the training division to familiarize the department with newer techniques of learning consistent with adaptive expertise. As of the end of 2012, there had been attempts on the part of training staff to incorporate some of the initial techniques suggested by literature from taxonomies of adaptive expertise, such as sending trainees copies of training expectations in advance and providing opportunities for practice. Training staff interviews supplemented patrol officers’ descriptions of training activity by providing a clear trajectory of past to current events. Interviews were conducted with each training officer, and the protocol and open-ended questions were consistent across each interviewee. The preliminary interview protocol, developed in conjunction with training division staff after the advisory panel (procedure #1), is attached as Appendix 5. Interviews were conducted following conclusion of the pretest-posttest portion of the project to determine changes that have occurred with regard to the training climate at KDPS as a result of the new training tasks. These “debriefing interviews” informed a discussion of the perseverance or elimination of barriers to organizational learning which took place. While interviews prior to the project yielded data that were used to provide a sense of the structure and 77 commitment to training on the part of the organization, the debriefing interviews yielded data that were contrasted with these baseline constructs. Tan and Heracleous (2001) conducted a similar study in a national police context where a longitudinal assessment of organizational behavior was not possible. The authors offered that the elimination of barriers to organizational learning could be used as a proxy measure for the advancement of organizational learning if reinforced by data demonstrating the extent to which changes occurred in respondent attitudes over time. This project gauged changes in training culture as a similar proxy measure for organizational change, derived from the qualitative interview data. Qualitative procedure #3: Observation of training. As the three training modules took place, researcher observation of training provided data regarding the manner by which training was implemented. This data provided a sense of how training principles were put into practice during the training portion of the assessment. The observation component served as an implementation assessment, with measures included in the observation protocol measuring principles of adaptive expertise as they occurred throughout training. The observational component intended to increase program fidelity by providing a richer context to mechanisms of training. Preliminary measures of implementation were collected according to the observation protocol, which is included in Appendix 4. The “sample” of these observation points was broken down according to the KDPS training schedule for each of the three months that the researcher was in residence; often times, multiple officers were included in one single training time slot. Therefore, the exact number of training observation opportunities was dictated by who showed up for training at the particular time and date that it was offered. 4 4 The exact number of observations was 20 separate dates in July 2012, 4 separate dates in August 2012, and 18 separate dates in September 2012. Training blocks lasted approximately 60 minutes each, and began at different times throughout the day. Times were inconsistent from one day to the next and depended upon officer availability and training protocol. 78 As demonstrated by the observation protocol, specific information about the training session was recorded, followed by a summary of the training session and a listing of training strategies used. Care was taken to ensure that observer/researcher interaction was minimized during training. Kalamazoo’s training division had conducted training with observers in the past, and there was organizational and individual officer familiarity with observations of training as part of the training experience (reinforced during the initial interview stage of the qualitative portion). However, because of historical attention to the potential for observational difficulties stemming from systematic social observation in police settings, a number of qualifications, listed below, were utilized in the observation portion to ensure for reliable and accurate data (Mastrofski et al., 1998; Reiss, 1991). Although Mastrofski et al. (1998) provided guidelines for a more systematic observation component which is not qualitative in nature, many of their suggestions derived from the Project on Policing Neighborhoods informed the qualifications in the training project. First, researcher familiarity with KDPS’ training procedures, venues, personnel, and policies regarding observations were maximized through preliminary contact and groundwork to ensure a smooth logistical and coordinated effort. Researcher familiarity with the procedures of police in-service training was high due to the researcher being a police officer at one time, but special care was taken to ensure that contact with training was observational and not participatory in any way. Second, researcher cooperation with the department adhered strictly according to KDPS internal policies regarding observers, because training environments are fluid and often require flexibility on the part of researchers to be able to adapt to environments that may not necessarily be conducive to note-taking and reflection. Third, in accord with review board and normative organizational protocols, observations were kept confidential. Finally, as suggested by Mastrofski et al. (1998), confidentiality was used to 79 mitigate trainer and trainee reactivity issues, as any training situation presents individuals who are learning a new task, which may cause potential embarrassment or guardedness on their part. These four procedures provided content which was collected and catalogued into Microsoft Word files. The resulting text files were triangulated as data: interview notes, observational content, and reactions from panelists were contrasted to elicit themes and outstanding issues. Patterns of behavior undertaken by different agents were examined to clarify the training culture within KDPS. The data provided context to the complexity of challenges faced by all aspects of the training endeavor at KDPS, beyond the training division itself to the line-level officers and middle managers within patrol operations. Another purpose of the qualitative strategies was to provide a more formal follow-up to the KDPS’ 2011 training needs assessment, conducted partially to determine future training needs. The 2011 needs assessment had revealed specific strategies officers desired as part of their in-service training. But information gathered as part of the needs assessment was not as systematic as the data which was collected from the qualitative effort. For example, the study interviews strengthened the initial needs assessment interview sample strategy by identifying a cross-section of day- and night-shift officers, and interviewing within those two populations patrol, special operations, and supervisory staff, resulting in a wider representation of officer characteristics which provided more breadth to the data. Also, the advisory panel and content analysis deepened an understanding of training needs in order to place quantitative data in longitudinal context. The qualitative data solidified findings from past examinations of training character conducted by the department while allowing for new insights offering context to the quantitative data. 80 A timeline of the entire project, consisting of both qualitative and quantitative datacollection strategies, is pictured in Table 2.7 with pre and posttest assessments indicated. Table 2.7. Project data collection and observation timeline. Activity Jul 2012 Collection of qualitative data from advisory panel meeting, interviews of officers to follow up informal needs assessment, analysis of existing records, training observations Training module 1: Firearms training (pretest and observation) Aug 2012 Sep 2012 Oct 2012 Pretest Training module 2: Contact & cover (observation) Training module 3: MILO (posttest and observation) Posttest Post-training interviews Analysis of Data This section details analysis techniques used to evaluate data provided by the pre- and posttest instruments, and how these techniques assisted in answering research questions. As a starting point, Shadish et al.’s (2002) pattern matching principle stands as a methodological orientation to data gathering and analysis in this project. This principle implies a procedure of offering plausible explanations for the causality of specific patterns of behavior based upon the totality of collected data. Although all of the project’s methodologies were based upon this principle, the use of nonequivalent dependent variables in particular relied on the pattern matching principle to assess programmatic, measurement, and effect patterns in data, as the 81 complexities inherent in the theoretical model supporting the project placed training techniques, organizational and individual behavior, and the adaptability of training knowledge and skills into an equal light. The greater question answered by the data analysis was, did the techniques derived from adaptive performance frameworks and taxonomies, when translated into specific training tasks, have any impact upon trainee perceptions of their behavior or their attitudes about task performance? Specifically, the project determined which of the dependent variables were impacted by training strategies for adaptive expertise. Determining if these relationships existed entailed two basic steps: obtaining and analyzing data patterns, and matching those patterns to theoretical criteria for adaptive expertise performance to evaluate hypotheses (Cook & Campbell, 1979; Shadish et al., 2002; Trochim, 1985). Obtaining and analyzing data patterns. In their review of training evaluation research, Ford et al. (2007) asked a global question regarding the advancements in training measurement that, although basic, has often been ignored by practitioners seeking to study training outcomes in detail: how do we know learning has occurred in training? The authors stressed at times that a “profile of learning” may be obtained from both cognitive and affective learning outcomes. This was consistent with White et al.’s (2005) measurement inventory which reflected mental, physical, and interpersonal behaviors. The implication inherent in both suggestions was that by considering effects of learning, data analysis may be employed to tell a more accurate story about the manner by which training was internalized and modeled. Data were collected at distinct points throughout the project as detailed below, with the corresponding methodology in parentheses: - Pretest-posttest provided: pretest and posttest information on 3 different training modules (using one pretest and one posttest instrument). Responses to open-ended questions were compared with qualitative findings. 82 - Qualitative data provided: notes from advisory panel meetings (using notetaking procedures), interview notes (using interview protocol), and observational notes (using observation protocol. These data provided both organizational-level and individual-level information about the training project, and enriched a description of the evolution of training over time. Initial data patterns were established by following Cook and Campbell’s (1979) principle of pattern matching that supported both a qualitatively-driven trajectory of data and a one-group pretest-posttest design with a nonequivalent dependent variable: “…the design is best used when a fairly complex theoretical framework can be articulated allowing not just for specification of which variables will be affected or not, but more precisely the degrees to which a set of variables will be differentially affected.”(Trochim, 1985, p. 579) When Trochim (1979) spoke of “the design” in the above excerpt, he meant specifically the use of a nonequivalent dependent variable. However, the study plan reimagined the use of qualitative data as a support for the overall task of ruling out plausible explanations for relationships between variables, by augmenting predicted data patterns which illustrated the relative strength of findings (Cook & Campbell, 1979). Qualitative data were used in the study not only to add context and provide a supplementary description. In the absence of a control group, qualitative data confirmed findings from the posttest results (Campbell, 1975, Galloway, 2010). For instance, if descriptive statistics regarding trainee performance demonstrate application of the training to patrol contexts, but officer interviews suggest that officers paid little attention to training transfer, these findings are compared to more accurately portray patterns of behavior and change. As a feature, this step required the coding of quantitative data, screening responses for missing values or fields, and creating categories. Raw qualitative data were retyped into Microsoft Word files and categorized. Pre- and posttest surveys were matched 83 according to the individual’s unique identifier number, determined upon distribution of the pretest instrument. Data from the internal referencing strategy was analyzed in the following manner. First, descriptive statistics of each item were reported, consisting of means, standard deviations, and sample sizes. Paired sample t-tests between pre- and posttests to determine statistical significance of mean differences were conducted. Scores on both training relevant and irrelevant items were tabulated as pretest and posttest averages for each item; high average scores reflected strong support for principles of adaptive expertise, while low scores reflected low support. Differences were compared, providing a result for each item in the following example: Average score (posttest) minus average score (pretest) equals change (gain score) Changes were expected in each training-relevant item from a low aggregate score on the pretest to a higher aggregate score on the posttest, expected results resembling Figure 3.1. Figure 3.1. Hypothesized comparison of training relevant and irrelevant items in IRS. Training relevant Scores Training irrelevant Pretest Posttest 84 The anticipated results of the internal referencing strategy, showing improvements on trainingrelevant scores while training-irrelevant scores stabilize between pretest and posttest, tested hypotheses regarding the effect of training on trainee self-report on the measurable dimensions. Calculating the differences between averages on summative scores obtained through pretest and posttest ensured that relevant differences between constructs remained. Paired-sample t-tests indicated basic means differences, followed by models controlling for demographic characteristics of individual officers to determine potential effects on dependent and nonequivalent dependent variables. Hypotheses suggested statistically significant relationships between pretest to posttest on training-relevant factors in the model; these relationships were disclosed by analyzing to what extent posttest scores differed from pretest scores, controlling for all factors. Given that prior research on the use of regression models in pretest-posttest means comparisons stressed unique results from different models using gain scores as dependent variables, ordinal regression was utilized as a more statistically sound technique to support the means differences analysis obtained in paired sample t-tests (Dimitrov & Rumrill, 2003; Singer & Andrade, 1997). These multivariate models explained relationships between select control variables such as years of police and organizational experience, age, gender, self-identified ethnicity, and hours of self-initiated training with each ordinal dependent variable, portrayed as self-efficacy with respect to twenty-seven identified training constructs (including trainingirrelevant measures). Matching patterns to theoretical criteria. Once statistical analyses were performed, adaptive expertise training guidelines were reconsidered in light of findings to determine if patterns seen in descriptive data mirror what was occurring in training. For instance, Endsley (2006) states that a distinct temporal pattern with respect to situation awareness occurs when 85 individuals are trained to process complex information, resulting in the formation of mental models, and ultimately, decision-making based upon goal-driven information processing. The training component associated with the formation of mental models for complex information processing was self-evaluation (H4) in the study. Did the patterns in the descriptive data for H4 as account for temporal order, and did trainees self-report the development of situation awareness characteristics? Further, what was the role of the nonequivalent dependent variable (officer safety) in the model for the five independent variables? Another potential pattern revealed by descriptive data involved the use of qualitative data as a comparison to correlations, a strategy derived from Lazzara et al. (2010): they state that error-based training assists trainees in indicating where adjustments to behavior can be made for error management. Did the training offer opportunities to place these learned error management techniques to use? Observational data was examined to determine if the two-hour time block for in-service training was insufficient in allowing opportunities to practice learned techniques, echoing the need for further practice. These and other patterns of change and behavior derived from both data sources allowed for the comparison of patterns to theory in the same manner Trochim (1985) suggested program pattern matching could assess training implementation in program evaluation. 86 CHAPTER 3 PROJECT IMPLEMENTATION This chapter details actual project implementation as well as deviations from the intended research project plan that occurred as the project progressed. Included are observations of training in progress, interviews with training staff and officers about project implementation, and analysis of training documentation used to provide context for overall project efficacy. As implemented, the Kalamazoo project constituted a three-month training program with an embedded research component to test theories of training for adaptable behavior in a police context. The project assessed how the department’s training division could utilize taxonomies for adaptive expertise to target specific training goals, and determine if adaptive expertise could be used to craft individual training goals and objectives that may translate to prerequisites for this behavior. However, the project also consisted of a more broad evaluation of how the Kalamazoo Department of Public Safety training division’s structure and organization supported and reinforced training goals. By collecting case analysis data, the evaluation examined the training division’s procedures for setting training objectives, assessing the overall training needs of the department, and negotiating diverse training needs across its service delivery spectrum. As the project concluded, a rich foundation of data was established from which multiple statements can be made about how in-service training delivery may be affected by specific strategies used by KDPS training division. Observation and interpretation of stakeholder interactions, operational conditions, and employee networks and relationships contribute to an understanding of these strategies in use. In order to detail this process, the following section will indicate how the training project operated in institutional context by providing a more holistic 87 view of challenges involved in the training effort, and how KDPS navigated these practical challenges. Two important analytic approaches from Weiss (1998) help frame discussion of project implementation. The first is an outline of evaluation tasks (Weiss, 1998, p. 273) listing potential questions evaluators may use in detailing program experience, and the second is an evaluation report outline (Weiss, 1998, pp. 296-297) which serves as a roadmap of implementation. Specifically, the following section focuses on the following two important points of analysis: - What took place over the course of the implemented training project in Kalamazoo with respect to progress and potential transformations in trained personnel? - What took place over the course of the implemented training project in Kalamazoo with respect to progress and potential transformations in the delivery of training and division attitudes toward training? Using analytical tools from Weiss (1998), the following discussion will serve as a descriptive portrait of adaptive expertise training as it evolved in the Kalamazoo Department of Public Safety’s in-service training experience. Specifically, Weiss suggests that in program evaluation, temporal description, profiling what characteristics of the program and organization may have led to observed changes, and interpreting results and implications for the department can assist in constructing a narrative of “what happened” in a specific program over time. Weiss also states that comparing pre-training to post-training data is an essential part of this task, but this specific point of analysis will take place in the next chapter. Because the training project itself was planned for over a year, and in order to provide a more succinct survey of what took place over the course of the project, four specific themes from project implementation will help sensitize the reader as the discussion unfolds. These themes are 88 identified at the outset to serve as a guide to discussion, and help frame concepts identified in specific project findings later. Each implementation theme is discussed in detail as they relate to different points of data collection and observation as the project was implemented, following a brief discussion of meetings with KDPS that took place when the project rolled out. Initial Meeting with Kalamazoo DPS Training Staff Once approval for the project was secured, a meeting was scheduled between the researcher and KDPS training staff, acting as an advisory panel informing the training to take place in the subsequent 3 months (July, August, and September, 2012). Attending the meeting were Captain Miller, Officer Lewis, Sergeant Harris, with occasional input from Officer Moore. 5 The main focus of this meeting was to reiterate and detail what was going to be trained during the three-month project. The meeting participants discussed scheduling and timetable requirements, which produced a timeline of events for the coming three months, reflected in Table 2.7’s timeline of data collection. In addition, the content of the training was discussed in detail, and how the individual training modules would reflect training for prerequisites of adaptive expertise. These prerequisite skills are detailed in Table 2.2, but preliminary meetings between the researcher and Officer Lewis produced specific training techniques and goals which reflected these prerequisites. Often times conversations between the researcher and Officer Lewis regarding training ideas took place over phone conversations and even text messaging. Following the initial meeting with training staff, Officer Lewis served as the sole point of contact as he was responsible for delivering the training itself; interactions with other staff members occurred infrequently. Officer Lewis had no assistant to help with training delivery and usually conducted training by himself. 5 Names of all KDPS staff participating in the project have been changed to preserve confidentiality. 89 One important component to the first meeting was to acquaint KDPS training staff with the purposes and goals for research in this case study, and to potentially reinforce the purpose of the evaluation and program. Although there is a receptive and cooperative atmosphere in the department with research endeavors, this was the first time a researcher had been “embedded” in the training division, and there was initially some skepticism about the process. However, from the prior year, the researcher had been assisting the training division with its initial training needs assessment, interviewing officers, and helping formalize relations between Michigan State University and the division. The initial meeting was an opportunity to establish what Weiss refers to as “administrative confidence” (Weiss, 1998, p. 37), or the strengthening of impressions among the training division that the researcher, and by default Michigan State University, had the department and the training division’s best interest at heart in the evaluation. Themes Related to Training Implementation As the project was implemented over the course of the three month study period, four salient themes emerged which are essential to understanding the context of project observations and data analysis. These themes, summarized in Table 2.1 and explained below, reflect the totality of experiences from project initiation in summer 2011 to completion at the end of posttraining interviews in October, 2012. These themes can be said to reflect deviations from the original plan that occurred as the project was implemented. 90 Table 3.1. Emergent themes arising from project implementation process. Themes Elicited from Implementation Process Examples Noted Theme 1. Institutionalized informality and flexibility of training division processes Training scheduling; meeting format & scheduling; flexibility brought on by resource needs & staff flexibility; inconsistencies Theme 2. Flexibility extends to training content Remaining flexible with evolving current events; training content personalitydriven; logistics Past history associated with training planning; staff familiarity with adjusting to immediate needs as outlined above Training delivery observations; interviews with staff and officers; findings Theme 3. Deviations from research plan were typical of training process at KDPS Theme 4. Training project as implemented retained original plan theoretically and methodologically Theme 1: Training processes were informal and flexible. Meetings, planning sessions, group discussions, and other structured events were negotiated by the unique organizational culture at KDPS. Informality ruled: meetings were often held casually, and discussion was very fluid with input from individuals in a manner consistent with Weick’s (1976) observations of loosely-coupled systems in educational bureaucracy. As with Weick’s statements about educational organizations, the fact that Kalamazoo DPS did not conform to expected notions of organizational climate made traditional analysis challenging. All aspects of the training division’s process were affected by this quality: meetings, training scheduling, changes to curriculum, preparations, and logistics were in flux. In the KDPS training division, a variety of unforeseen circumstances can impact scheduled training, which necessitated modifying the planned training modules for the project to conform to these special circumstances. Tellingly, 91 the KDPS training schedule is mapped out months in advance on a dry-erase “white board” in the training division lobby, which allows for frequent changes to be made, many of which are minute in detail (such as the availability of supplies or personnel for specific modules). During the three-month adaptive expertise project, the department was also putting neighboring officers from other jurisdictions through firefighter training at a simulation tower, enrolling newly-hired cadets in an in-house training academy, and overseeing other personnel development at multiple levels throughout the organization. Meetings themselves had no specific format, and topics discussed often overlapped with past and future training division schedules, goals, and tasks. This is typical of training division meetings and reflects the leadership style of training Captain Miller, who uses the term “table-top session” or “coffee table session” to impress on participants the informal structure of the process (Miller, personal communication, September 2, 2012). This is the division’s distinguishing feature: each member is expected to be able to think rapidly and make minute adaptations and modifications to existing scheduling and process. The overall structure resembles Weick’s (1993) discussion of collective mind, where efficiency and error management are imperative. Theme 2: Training content itself reflected a need for flexibility. Because in-service training is delivered in a functional environment that is as rapidly unfolding as the very police field experiences they were intended to simulate, instructional content itself reflected this environment of flux. As stated previously, research concerns about training observations outside traditional academy contexts (which are more controlled, largely classroom-based experiences) have made prior research attention to in-service training problematic. Besides scheduling changes, the training activities themselves were subject to modifications and changes on an almost continual basis. An example of this involved, at the training officer’s request, making 92 changes to instructional content based upon current events (as will be explained in a subsequent section). It is potentially unlikely that, had any of the other training officers been in charge of this particular training module, that this change would have been made. This illustrates that training content is often dictated by the personalities of the training officers, and their perception of what is best for KDPS officers under a general mandate to deliver the best training possible. Theme 3: Deviations from planning are typical given these informal processes. Resource constraints continually impacted training design and implementation. For example, prior to the project, the training division postponed driver training because the department added a new fleet of vehicles. This illustrates the constant potential for sudden changes to training tasks. During the project, as another example, ammunition shortages occurred which would have disrupted or even cancelled the entire month’s training in firearms. This would have put a halt to the entire project by necessitating that new training tasks would have to be created and restarted after September. The expected shortage did actually occur, but did not surface until September, affecting tasks that were planned after the project’s completion. However, this issue reinforced the need for the training division to remain constantly flexible for unforeseen issues to arise and affect scheduled training. Theme 4: In this environment, the training project that was implemented remained theoretically sound and methodologically rigorous. It could be said that KDPS training division has familiarity with having to make sudden adjustments to training that may deviate from original plans, as indicated above. The division could be said to almost expect to make changes: change is seen as a healthy indicator of training division responsiveness to the fluctuating needs of officers on the street. Similarly, the training project as implemented in this study displayed much of this character: it was not markedly different from the original stated 93 plan in any theoretical sense, and it retained its methodological rigor. However, it was less a formal, static list of objectives, and instead became a dynamic training event within the stated design and lesson structure that guided trainer interaction along the planned objectives. As the researcher learned through the course of the project as implemented, this is typical of training delivery at KDPS, and led to unexpected positive outcomes for both the division and officers. Organizational Training Climate Brunninge (2009) states that examining organizational history allows for an overview of strategic planning and construct processes which provide additional context to analysis. In the case of the Kalamazoo training project, it is helpful to examine one specific historical document with respect to prior attempts at a more comprehensive shift toward internally-driven in-service training. In 2010, the training division created a Training Passport (Appendix 9). The document, which summarized in-service training expectations for officers throughout the department, included a timeline allowing officers to keep track of in-service opportunities they attended and envision career advancement as a series of evaluations and goal-setting modules. The Training Passport also contained areas for supervisor evaluation and suggestions as well as organizational expectations at different promotional levels. This 15-page document reveals that the training division, a year prior to the initiation of researcher involvement in the adaptive expertise training concept, had already made an initial foray into articulating organizational expectations about what would constitute “expert-level performance” and its relationship to inservice training (Kalamazoo Department of Public Safety, 2010). The passport, however, was limited to firefighting dimensions of public safety officers’ duties; the training division did not attempt to integrate police skills into the Training Passport. The document itself was used for only one year and then phased out. The stated reason for the lack of follow-up and 94 institutionalization of this innovation was that training division personnel had transitioned to take on duties from the operations division of the department, and the additional workload meant that time to develop the newly-created Training Passport was negligible. The Training Passport does provide additional organizational context for the development of comprehensive attempts to revitalize, internalize, and account for in-service training, and speaks of the interest of the department to link this training to officers’ long-term career pathways. It could also point the way to future human resources analyses that the department could utilize to collect data about officers’ training histories. Observations of Training in Progress as Implemented For each of the three months, the researcher observed training in progress each day (a total of 128 hours over the course of three months 6) and took notes according to the training observation protocol in Appendix 4. The three training modules are discussed individually in this section to provide further use of Weiss’ analytical framework: specifically, training observation provides the background for description of activities and conditions of training (Weiss, 1998, p. 274). Two additional analytical techniques from Weiss can be gleaned from observational data: profiling, or the assessment of which combination of conditions and actors led to more favorable outcomes, and locating unanticipated effects, or noting unexpected training behaviors and effects (Weiss, 1998, p. 275). Each of these analytical components parallels the following narrative. July 2012 Tactical Firearms training. In the month of July, training involved tactical firearms drills in an outdoor range environment normally used by KDPS for firearms training. The venue was an outdoor shooting range situated in a rural part of the county. The average 6 Because of the informal nature of posttraining meetings with officers to discuss training, this figure is conservative. At times, posttraining sessions would continue over lunch hours, and Officer Lewis would stay and continue instruction as needed. 95 daily temperatures during this outdoor training were quite high, and each day, training took place from approximately 7 AM to, at times, 10 PM. This schedule was designed to account for nightshift officers who arrived at work at 7 PM. The trainer delivering training was Officer Lewis by himself. As this was the first training in the three-month adaptive expertise program, the pretest instrument was issued to each officer upon arrival at the shooting range where project objectives, confidentiality protocols, and modules were explained. . The training module lasted approximately 90 minutes. The July lesson plan is included as Appendix 6. In this month, lesson plan objectives were infused with specific training tasks crafted by Officer Lewis targeting the recognition of errors and stress management, and were identified by bold-face on the physical lesson plan. The infusion of these tasks into the overall plan, as opposed to training for these prerequisites as stand-alone or supplementary training, illustrated the degree to which training for adaptive expertise can potentially be streamlined into existing lesson planning in a manner that does not appear to be inconsistent. The training flowed from one component to the next in a manner that was fluid and allowed for easy transitioning between tasks; trainees seldom appeared to have any difficulty with the training drills as many techniques were not markedly different from previous training. One officer remarked that this training did not appear to be any different than other firearms training modules, as realistic simulation, failto-fire scenario drills, and target shooting with abrupt and unexpected changes are occasionally used by KDPS. The focus of the training itself was on the unpredictable nature of combat firearms incidents, underscoring the importance of adaptability and the development of skills targeting these outcome measures for the month. It is reasonable to state that this month’s training modules may have not accentuated the dimensions of learning associated with providing trainees with a global understanding of the importance and role of stress-reduction strategies 96 based upon observations of the training being relatively unchallenging in this area. However, the one dimension that displayed marked improvement over the course of this module was error management. Officers routinely struggled through error drills and often used peers to assist in recognizing and discussing solutions to crises simulated by Officer Lewis (such as the use of a tourniquet). Observations of training also support the contention that this module allowed trainees a variety of opportunities to navigate abrupt and unseen errors, such as weapons misfeeds and adapting to extreme weather conditions. In sum, the totality of this training module provoked mixed reactions from trainees with respect to the two focus points, with navigating error management scenarios being the predominant effort made by trainees. August 2012 Mobile Active Shooter Pre-Plans. The change in lesson planning for August 2012 as a consequence of the perceived urgency of the Aurora, Colorado, shooting was done with great care to the structure and content of training for adaptive expertise that had already been planned prior to the start of the project. However, one consequence of this change was the lack of specificity in the August lesson plan (Appendix 7) for individual training items as compared to the robust lesson plan for the month of July (Appendix 6). There are some differences with regard to how the July and August lesson plans articulated goals for the training, specifically with regard to how the training modules were explained to the trainees. The July lesson necessitated multiple tools, scenarios to be explained, environments created by Officer Lewis on a shooting range, and specific drills to occur; the August lesson did not include any of these events, and was strictly an “on your own” training opportunity. The August lesson plan did not include any text linking the training goals to prerequisites for adaptive expertise (which, for August, consisted of problem-solving strategies and developing contingency plans). In the July lesson plan, Officer Lewis articulated the specific components of adaptive expertise targeted, and 97 how his training goals and activities were aligned with these prerequisites; none of that materialized in the August lesson plan. It appeared as though the August “adjusted” plan was put together in order to account for a specific current event, as it was, and lacked some of the specificity of the July lesson plans. This not only led to dissimilarities between the structure and format of the two lesson plans but also some confusion for the researcher as to how the training itself would take place. The Adjusted lesson was to take place in multiple venues, in individual stations at lineup and on the officer’s own time, requiring the researcher to be flexible with travel and scheduling to conduct any observations. Regardless of the comparative lack of specificity of the training modules on paper, the lesson itself contained the same targeted goals of adaptive expertise; it was later discussed with Officer Lewis that lesson plan format consistency might lead to greater trainer control over lesson planning and enhanced trainee understanding of training goals. The structure of the lesson was as follows: Officer Lewis would send a DVD of a mobile active shooter incident (The “Trolley Square” shooting in a suburban shopping mall which closely resembled the Aurora incident) taken from the “In The Line of Fire” police training video series to each platoon or shift of officers in the department throughout the month. In groups, officers were to observe the video and discuss (with specific prompts from Officer Lewis’s lesson plan) tactical events and officer response opportunities presented by the incident for police behavior, satisfying conditions of training for assessment of problem-solving strategies. After watching the video, the platoons were to identify areas, structures, or sites in their patrol districts where they feel a similar shooting could potentially take place. The trainees were to make site visits to these locations, assess layouts, entryways and approaches for possible police 98 response, and produce a report of the site including a list of necessary resources and conditions that may be important in the event of a mobile active shooter. The training was observed by the researcher on multiple occasions and appeared quite problematic from a motivational standpoint. Many officers made the comment that they viewed the training as a “glorified homework assignment” and approached the subject with varying degrees of seriousness. Some appeared upset that in-service training, which was traditionally an opportunity for hands-on training that often produced excitement, had been sacrificed for a video, discussion and report presentation. Other officers viewed the topic itself as serious, but suspected that the “take-home” nature of the training was a product of training division scheduling which did not allow Officer Lewis the opportunity to spend any time conducting training directly with officers that month. Mitigating these attitudes was the oversight and guidance of the officers’ immediate supervisors, patrol sergeants, who were tasked with amassing the completed reports and turning them in to the training division. The variation in motivation of officers to complete the training was evidenced by the resulting reports. The August training presented a unique opportunity to see the degree to which some officers, shifts, and supervisors would take the responsibility for self-directed in-service training onto themselves, and produce a report that reflected self-motivation, imagination, and quality. The resulting reports, which are included as Appendix 10-12, represent a “feast or famine” spectrum of quality. Appendices 10-12 indicate that for the same location (Loy Norrix High School), three different groups gave widely varying degrees of effort toward their report. Future analysis of these resulting documents and others resulting from this training could possibly link to workgroup culture and supervisor leadership styles as an indication of the extent to which adaptive training may be reinforced by such variables. Other reports were simply 99 handwritten on notebook paper; one was actually completed on a Post-It note attached to a downloaded map. Many of the reports were simply bulleted lists of possible answers to the training prompts by Officer Lewis, and provided little information about the sites or the potential obstacles officers would encounter in the event of a shooting. One report stated in many fields, “no information is known”, suggesting that the officer probably conducted very little actual observation or research of the site. On the other end of the spectrum, many reports were thorough site assessments featuring floor plans with highlighted entranceways, names and contact information of employees, times of day scheduling pertaining to numbers of occupants and activities at the site, photographs taken from different vantage points, and included instructions for how police response could tactically engage a shooter at these locations. An example of this variety of report is Appendix 12. Some of the formats of these larger, more thorough presentations were standardized across the entire shift as a template, suggesting that the officers may have worked on the project in unison, sharing and presenting information with each other (this was observed by the researcher). One drawback to assessing the strength of this training is that no rubric for determining quality from one project to the next could be utilized as the training module was changed almost a few days prior to the beginning of the month, preventing researcher participation in the crafting of a content assessment. September 2012 Subject Control Decision-Making training. The video-simulation device employed by KDPS for indoor video-based simulation training goes by the acronym MILO, which stands for Multiple Interactive Learning Objectives. It is a vendor-based system of videos and prompts which works on a flowchart-style basis: as officers respond to videotaped prompts and suggestions displayed on a screen, a microphone picks up their statements and steers the video into other scenarios that unfold in real-time. This computerized approximation 100 of person-to-person encounters allows for firearms discharge to be tracked on a screen for replay and assessment. The device is routinely used by KDPS for different non-firearms purposes as well, such as arrest decision-making. In the September module, the device would feature a number of scenarios containing subjects from different cultural, physical, and psychological backgrounds to “surprise” the trainee into negotiating different and sudden changes in their potential response. One video featured non-English-speaking subjects with weapons, one featured a child with a weapon, and one featured an obviously mentally incapacitated individual who posed no immediate threat at all. The training also featured a supervisor-specific component wherein participating middle managers (sergeants and lieutenants) would meet with Captain Miller after the video-based training “incident” and give a verbal interpretation of next steps a supervisor would normally take in each given scenario; the lesson plan for this particular training component is included as Appendix 8. As training took place, the MILO computer recorded the training scenarios on multiple cameras placed throughout the room at different angles. This allowed the trainee to view his or her actions following training through a playback feature, and allowed them to discuss aspects of the training with Officer Lewis in a “debrief”. Part of this debrief was to evaluate actions taken in each scenario and articulate to Officer Lewis what led to the actions taken. It was noted by the researcher that this debrief opportunity allowed for a form of evaluation that mirrored a competency analysis: although there was no “right action taken” (such as a minimum performance standard dictated by either KDPS or Officer Lewis), the ability of the device to record the entire event led to thorough posttraining discussions where the officers engaged with trainers about the totality of the incident. 101 Officer Lewis’s objective consistent with training targeting all six adaptive expertise dimensions in the training project was to “overwhelm” the trainee by providing a number of unexpected and unforeseen circumstances for the officers to navigate, and use these situations and officer responses as practice sessions where error, crisis, and stress management, problemsolving capacities, and the processing of complex information could occur in rapidlydeteriorating public encounters. This was the first indication that a training module attempted by design to incorporate multiple objectives targeting prerequisites for adaptive expertise. The training was visually and emotionally intense and provoked an engaged response from trainees: some shouted commands at the screen in a manner they would an actual human being, and some remarked that the training was a simulated example of everyday behaviors seen on patrol. Most officers viewed this training as an opportunity to hone their skills dealing with individuals from diverse cultural, environmental, and physical backgrounds, with some utilizing Spanish commands and treating the computerized scenarios as an actual call (for example, some officers simulated using radio traffic without being prompted to do so, and others called out to Officer Lewis that at certain points they would be notifying their supervisor as per departmental protocol). The training was enriched with a variety of unforeseen rapidly-developing and situational responses which illustrated that adaptive expertise prerequisites can be targeted through evolutionary and embedded training tasks without seeming compartmentalized. One observation of this training was that, consistent with how adaptive expertise was envisioned with regard to specific training behaviors, the training was highly functional and deliberate. But whether or not this constitutes “deliberate practice” as identified by White et al. (2005) is unresolved. The training taxonomy used to craft training objectives states that the practice should be “effortful” and not “casual” in nature, and that training should be regarded as 102 “work, rather than play” (White et al., 2005, p. 7). Nowhere was this more observed across the three months than the MILO training. Officers appeared physically exhausted and often required extensive debrief, discussion, and feedback sessions in order to “decompress” after the experience, and many officers remarked as though the training was stimulating and thoughtprovoking with regard to imagining situations that were atypical but nonetheless possible. Observing the self-evaluation component of this training, however, offered a glimpse into ways that the training module (and Officer Lewis as a trainer) could potentially positively impact police officers in helping them develop adaptability skills. Nearly every observed posttraining debrief and self-evaluation discussion with Officer Lewis was an opportunity to maximize the trainer-student relationship; officers were sincerely offering ways in which they felt their performance could have been improved, and discussed alternative performance strategies with Officer Lewis’ guidance. In this particular module, it was apparent that (compared to researcher observation of a similar training in 2011) Officer Lewis’ efforts to maximize self-evaluation opportunities were maintaining trainee focus and interest in self-improvement. In all observed training sessions from this month, trainee involvement was intense and engaged. Posttraining Ride-along Interviews with Officers and Supervisors During a two-week period following training, a structured interview procedure was implemented as outlined in the previous chapter. First, questions from the 2011 needs assessment were utilized for follow-up as to whether interviewees felt the training project fulfilled specific training needs that were identified at that time; this alerted the researcher to an overall interpretation that the organization was able to target and fulfill perceived training needs internally. Second, questions regarding the ability of the training project to target prerequisites for adaptive expertise were used to contextualize pretest-posttest findings. Interviews with 103 officers and supervisors were hoped to provide evidentiary support for quantitative findings. The sample was determined as illustrated at the project planning phase, with the researcher spending one week riding with day-shift and power-shift officers and supervisors, and one week riding with night-shift officers and supervisors. This cross-section of employees was chosen to represent all possible work shifts, with variations in ethnicity, gender, years employed at the department, years of total police experience, and different job specializations. In a two-week period, 39 officers and supervisors were interviewed. In the case of patrol officers and sergeants, interviews were conducted while riding with them on patrol so as to provide an environment of comfort and induce collegiality. In the case of lieutenants, interviews took place at the lieutenant’s office at KDPS headquarters during their shift. Table 3.2 summarizes interview sample demographics. Table 3.2. Posttraining interview sample characteristics. Total n of officers interviewed 39 Gender Male = 34 Female = 5 Ethnicity African-American = 2 White = 35 Hispanic = 2 Shift Day shift = 17 Night or power shift = 22 Under 5 = 11 5-10 = 12 11-15 = 10 16-20 = 6 Over 20 = 0 Years employed at KDPS Supervisors Not a supervisor = 28 Sergeants = 8 Lieutenants = 3 104 The interviews followed the structured format and questions outlined in Appendix 1. Officers felt open to discussing training from a variety of perspectives, such as history of training developments at the department, new challenges they felt were taking place, and their impressions of the impact of the current training project in that context. The researcher also solicited information about the components of training that they viewed as most successful, and ones which they felt needed attention. Also, officers provided unexpected information that often steered discussions towards new training dynamics that had not been predicted by the researcher. The interviews elicited the following themes which guided discussion: the relative satisfaction with the training project, the perceived need for officers with adaptable skills, changes in department demographics that were reflected in new training needs and demands, the perceived effect of resource depletion on training, and a disconnect between newly-promoted sergeants and performance expectations of line-level officers and how this related to training. Supervisor responses that deviated from officer responses are discussed throughout the next section when appropriate, as are other subgroup observations. The coding scheme for interview data is derived from Weston et al.’s (2001) qualitative coding strategy for the social sciences, and incorporates many of that manuscript’s findings about partnership research and how valuable the client or host agency may be in constructing a coding scheme. To this end, results of the interviews were compiled and vetted to training division staff following completion of thematic coding, and reactions from these staff members (Officer Lewis and Captain Miller) were incorporated into thematic discussion here, also acting as a reliability check. Using Weston et al. (2001), reaction statements were coded as “positive”, “negative”, “neutral” or “mixed” as responses to prompts about specific training questions, with 105 reactions tabulated for frequency. Additionally, the construction of themes evolved from further interview probing with officers; since the sample of interviewed officers was relatively small, themes were determined when a specific item emerged more than twice (Weston et al., 2001). Officers felt the training helped them develop adaptability skills. In general, the 39 officers and supervisors interviewed reacted favorably to the three-month training modules and could tell that the division (and Officer Lewis in particular) had taken great care to designing training that maximized opportunities for the development of unique skills sets. Not one officer volunteered that the training was unsatisfactory; however, six officers did express negative reactions to the August mobile shooter assignment for reasons noted earlier. When the subject of learned skills was raised in interviews following this general gauge of reaction, all but two officers felt that the skills outlined in the adaptability modules were essential knowledge and skills for officers to develop (the exceptional two officers believing that such skills may not benefit all specializations in the department). Supervisors in particular could see the development of adaptability skills as resulting from training, with one sergeant stating that he “felt more tuned in to a 360-degree type of policing” (Hanhauser, personal communication, October 8, 2012). Two other officers shared feelings that the training was giving them a more global view of surroundings, reiterating Endsley’s (2006) statements about situation awareness and “reading” the surroundings similar to accomplished athletes. One question officers did have pertaining to training effects involved the use of multiple learning dimensions consistent with Mangos and Johnston’s (2008) evaluative structure, claiming that mastery opportunities were minimized by the constraints of time, a theme which will be reiterated further in discussion of survey results. A final comment on the development of stress-reduction strategies and contingency planning came from a female officer who stated that “this is what the [newly-hired] 106 officers like myself want and need out of training” (Mitchell, personal communication, September 28, 2012), reinforcing the push toward different skills sets needed for a different era of increased job duties, flexible career pathways, and added stresses of resource depletion. Officers perceived a further need for adaptability training. Continually, officers and supervisors volunteered that they felt skills were necessary to counter a growing landscape of ever-evolving threats and challenges that static training was not preparing them for. They also felt that the training division was currently providing them with these skills sets, and recognized the training project as part of that advancement. One officer stated that training for adaptability was targeting officers’ “mindset and resiliency… sometimes things that happen daily are worked in, and the [training] division uses this as resiliency training” (Walton, personal communication, August 26, 2012). Another officer stated that adaptability targeted an innate fear, especially in younger inexperienced officers: “you hope these things don’t happen, but they definitely could happen” (Vasquez, personal communication, October 19, 2012). A third officer linked the need for adaptability to the death of a Kalamazoo officer in a line-of-duty incident in 2010 by saying, “since [the incident], the training division is the best it’s ever been… we get things that are current and things that make us think that anything could happen” (Murray, personal communication, October 2, 2012). Other suggestions and comments praised the perceived translation of the adaptive expertise dimensions into helpful and realistic training techniques which “may or may not be reaching goals right away, but are headed in the right direction” (Nelson, personal communication, October 1, 2012). Officers saw changes in department culture related to training. Officers volunteered that there appeared to be a continually intense commitment to training on the part of newly-hired officers at Kalamazoo, constituting a new attitude toward training. A female officer said that 107 “the new attitude is, ‘give us something good’”. These new officers’ attitudes have invigorated the organization’s training expectations, and are “engaged, and the old guard is gone” (Mitchell, personal communication, September 28, 2012). Another veteran officer stated that newer officers viewed training as inherently “highly motivational… and [newer] officers are taking the initiative” to outperform training expectations (Walton, personal communication, September 26, 2012). Another officer stated that recent hiring of new officers, many of whom had no police experience, drove the need for a different sort of recognition that was exhibited in unique ways, such as Lieutenant Grant’s reading a different “hot shot” training score over lineup (Davis, personal communication, October 12, 2012). These and other answers reiterated that the department was feeling challenged to provide training with a new attitude and intensity to match the growing and enthusiastic needs of a new generation of officers. Officers were fearful of depleted resources for training. Officers expressed an interest in internalizing training not only for financial reasons but because the training they would receive would be Kalamazoo-specific and potentially of quality. External training at great cost was seen as a “waste of money” because “they don’t give relevant questions” for Kalamazoo officers (Williams, personal communication, October 3, 2012). In many ways, the negotiation of external training expenses with perceptions of applicability to KDPS’ unique priorities begs the question of “vendor integrity”; one officer stated that “a lot of times we go to outside schools and say, ‘we could have taught that’” (Hayes, personal communication, October 1, 2012). However, concerns over how to fund external training often were seen as frivolous if the department could devise internal training that was of similar or better quality. An officer stated that he wanted the training division to either “teach me something new that can be used every day, or help me 108 polish something old, and I’m fine… that’s good training, and doesn’t cost too much” (Green, personal communication, September 27, 2012). Officers were concerned with supervisors’ roles in the training process. It became apparent upon speaking with officers that supervisors and sergeants who may have been negatively reinforcing training actually constituted a morale crisis that affected many officers deeply. It was by far the most recognized suggestion for future training efforts when officers and supervisors were asked, and the conversation on the street oscillated between two extreme perspectives. Some officers felt sympathetic toward newly-promoted sergeants: “they should cut the new sergeants a break, they’re doing nothing different than what other people did when they were promoted” said one experienced officer (Davis, personal communication, September 12, 2012). Others were more critical, stating that the department had resources to resolve deficiencies in sergeant performance if done internally; one officer stated “the incident with [the officer fatality] gave us an opportunity to step up our game, but there’s no accountability” (Nelson, personal communication, October 1, 2012). Other perspectives suggested that future training for adaptability could be done with an eye to different organizational levels, including sergeants. A lieutenant stated, “it needs to be done in-house… we could be doing for leadership what we’ve been doing for firearms, training new leaders” (Brown, personal communication, September 12, 2012). Officers explained that supervisor training may be different than what would constitute leadership training, and identified skills for adaptability as the key difference, namely communicating with different populations, being able to process complex knowledge rapidly, and being able to solve problems between employees. A typical comment that encapsulated officers’ reactions to the lack of training for leadership skills was, “it would be nice to have leaders and not just supervisors” (Clark, personal communication, August 22, 2012). 109 Other Structured Posttraining Interviews Weiss (1998, p. 274) states that gauging participant interpretations of program involvement provides an opportunity for stakeholders to reflect on potential improvements in either themselves or others through the course of a project. In the case of the Kalamazoo training project, such an endeavor became hindered by the fact that, realistically, only one person was consistently participating in the project as planner, instructor, and evaluator. In order to obtain a wider range of “participants” beyond officers themselves, the researcher met with eleven supervisors following the project’s completion in addition to speaking with Officer Lewis. This slate of stakeholders was limited as it did not involve upper-level management (the chief and command staff rarely observed training in progress), but accounted for the limited range of personnel who were involved in the training though the successive months. It is also important to note that attention was taken to crafting specific interview questions that targeted not only the specific audience, but also different purposes for obtaining data. For instance, supervisor interviews focused on dimensions of the training that applied directly to their view of the breadth of the training effort and how they felt it impacted long-term organizational priorities. Supervisors’ perspectives were supportive of training. Of the 39 posttraining interviewees, 11 were supervisors (sergeants or lieutenants) who had experienced the threemonth training. Their responses to questions are important to isolate here to provide context to the changes occurring on the supervisory staff at KDPS, and to reiterate findings from officers that the role supervisors play in reinforcing training is critical to the retention of trained knowledge and skills. Supervisors responded overwhelmingly that they viewed the three-month training project as important in developing officer skills across a variety of dimensions related to adaptability, namely in providing realistic scenarios and experiences where trainee opportunity to 110 fail allowed for structured learning and practice opportunities. This finding appeared consistent with observed intensity by which many sergeants approached training, but inconsistent with what the researcher heard regarding the August self-directed training. Because of the possibility that sergeants were negatively reinforcing some training, posttraining interviews with supervisorylevel staff included questions about the degree to which managers felt such behavior was actually taking place, and what its potential impact would be on training and department morale. There appeared to be a distinct attachment between newly-hired sergeants and the officers they were assigned to supervise consistent with subcultural theory, which will be illustrated further in a subsequent section on unexpected findings. The overall impression of supervisor reactions to the training was that it was a significant step forward for the department, and that such a commitment occurring in a time of potential resource depletion indicated positive organizational commitment to KDPS officers and their professional growth and personal safety. Officers viewed training as positive example of change. Interviews and conversations with Officer Lewis took place both during and following the training session. Discussions centered on what he felt the impact of the training program was on trainees, and what legacy the training would have for future training modules. It became routine for Officer Lewis and the researcher to discuss the perceived effect of the training on officers and how he felt performance was being affected. In the case of the July training, Officer Lewis felt that the manner by which officers were engaging in practice opportunities seemed more engaged and interested, as he had once complained a year prior that officers were rushing through firearms training in an effort to “get training over with” and return to patrol. It appeared now that guided practice was offering an opportunity for trainees to work in groups and with Officer Lewis on specific points needing improvement, and an overall positive interest was noted (Lewis, personal communication, July 111 22, 2012). For the August training, Officer Lewis did state that he sensed disinterest and even resistance to the training being “officer-focused” and not a typical in-service training module, but felt that the urgency of the Aurora shooting was a mediator in stimulating officer interest. He did state that the development of contingency plans surpassed his expectations as a trainer upon seeing many of the submitted tasks, and stated that “the ones who took it seriously were the ones [he] thought would take it seriously” (Lewis, personal communication, September 8, 2012). An increased interest in training preparation was also noted from the use of preparatory advance emails as a pretraining strategy, and his enthusiasm for this new technique was passed on to other trainers. Finally, he was overwhelmingly supportive of the diversity modules embedded within the MILO course, and displayed a sense of enthusiasm for including multiple physically and culturally diverse backgrounds while setting up the training. By training’s end in September, it was apparent that he had taken ownership of the modules by committing himself to seeing them through and getting as many officers to training despite overwhelming scheduling problems that month. According to Officer Lewis, the training division had begun to incorporate many of the perceived positive benefits of the training into other areas even before the modules were completed. He sent emails to the researcher showing how some training tasks were used in other non-police training modules, specifically the sending of training materials in advance by Officer Moore and Captain Miller in a fire training module (Lewis, personal communication, September 22, 2012). This illustrated organizational change, and spoke to the enthusiasm the division had for noticeable results as they may impact officers. Officer Lewis felt fully engaged in the program as he believed in its mission to create a more “adaptable” officer in the department, even to the degree that he considered authoring a practitioner-directed paper discussing what he 112 felt he learned through the course of the project. This also illustrates a degree of organizational change as few if any opportunities for professional development based on a newly-established research-practitioner partnership had ever occurred in the training division. Officer Lewis had been attracted to the project initially because of its apparent use in military contexts, and became convinced that it had merit for police officer training because of its apparent linearity with what training focus had already been established before the project. Many times Officer Lewis commented that he felt the department already trained for many of the adaptable behaviors listed on the taxonomy, but that the project opened his eyes to new techniques that could be used and potential uses across multiple training modules. The relevance of the above observations is to validate a specific and essential aim of translational criminology, namely the reaction of police organizations to evidence-based approaches incrementally as opposed to wholesale and sudden change (Dermody, 2013). In establishing evidence-based principles in tiers of police organizations where none has existed previously, approaching change as a gradual and strategic process has inherent organizational benefits as it allows the organization to refine and adjust evidence-based principles to their organizational preference. In a sense, Officer Lewis is describing such a process as a coordinative effort where researcher involvement was maximized as a consultant role, and data was collected to allow for attention to areas where such changes could occur strategically. The effort made by the training division to implement the training program had a number of problematic stages, as evidenced by some of the critique in this chapter; but the end result, as related by Officer Lewis, who had overseen the project’s creation, implementation, and completion, was to put a program into action in a manner that utilized organizational strengths and structure to make the program beneficial to officers. 113 CHAPTER 4 PROJECT OUTCOMES The project utilized a small-sample pretest-posttest design to gather data about the impact of training strategies derived from taxonomies of adaptive expertise on specific outcomes. As previously stated, this design compared means for pretest and posttest items from a questionnaire distributed prior to and after three training modules incorporating training techniques designed to target prerequisites of adaptive expertise. In this section, outcomes of data analysis techniques, including paired sample t-tests and ordinal logistic regression, are detailed as criteria for a general statement in support for the modules inculcating prerequisites for adaptive expertise. Number of Cases and Nonresponse Of the 205 sworn officers on staff at KDPS 7, a total of 137 participated in the first month of the training project in July, 2012, for a 67% rate of participation. Accounting for this participation rate were annual vacations which often coincide with summer months in Michigan, work scheduling conflicts which prohibit in-service attendance, and external training or professional development opportunities which often conflict with department-offered training (Lewis, personal communication, January 4, 2013). Each of the 137 officers took and submitted a pretest survey for a 100% response rate. Of those 137 officers, only 109 officers were able to complete the next two months of training, and therefore the entire training program, resulting in an overall posttest n of 109. The n of 109 therefore represents 53% of the total sworn workforce at KDPS, and 80% of the officers who started the July training. Of the 109 posttests received, there were multiple items where no survey response was given to training-specific question 7 This figure is accurate as of the completion of the training in October, 2012. 114 items. 8 In the following steps, these missing values were analyzed to determine if the missing values were random or non-random. First, a full frequency distribution was created to determine the number of missing values from survey responses. The missing values correspond to the following dimensions of training: knowledge of problem-solving techniques (one nonresponse), knowledge of strategies for dealing with crises (two nonresponses), and understanding how to handle mistakes and errors (four nonresponses). To discern potential patterns for these nonresponses, missing values analysis was used to examine cases where values were not indicated on surveys. Missing values analysis allows cases with user-defined missing values to be sorted by a number of categorical variables provided by the remainder of the instrument, such as demographic variables and other response categories, in order to discern potential patterns for nonresponse. In this case, all demographic variables which are listed on the survey instrument were used as potential categories. However, due to the low number of nonresponse items, no pattern was noted through this feature, resulting in the conclusion that nonresponses on the submitted surveys occurred for no categorically-explained reason other than selective oversight. Data analysis was based on available responses for each survey item, and nonresponse items were checked for sensitivity. Those respondents with selective nonresponse items did complete the rest of the survey. Of the 109 officers who completed both pretest and posttest, a number declined to answer demographic questions on the initial pretest. The frequency of these nonresponses is: years employed at KDPS (5 nonresponses), years of total police service (5 nonresponses), age (6 nonresponses), ethnicity (12 nonresponses), gender (10 nonresponses), highest education completed (6 nonresponses), and hours training on-your-own (14 nonresponses). Again using 8 There were other nonresponses on demographic items which are described in findings as appropriate. 115 missing value analysis, no specific categorical pattern to the nonresponse for these items was detected. It appears simply that some individuals did not choose to list demographic information, possibly because of fear of violating confidentiality in a small agency. A plausible explanation for the reluctance of officers to answer identifying information such as ethnicity and gender could be that the department demographics are skewed towards a white male population, and self-identifying as something other than that demographic may have been perceived as compromising confidentiality. Officers were reassured that there was no matching or identifying information on surveys, and a unique identifier was used. However it is likely that even with the safeguards and assurances of confidentiality, officers may have perceived questions identifying ethnicity and gender as a risk. A sensitivity check for these missing items is conducted in a following section. Descriptive Demographic Statistics Descriptive statistics reported in Table 4.1 from officers completing the entire training program (overall n=109) provide a profile of the department’s officers who were able to attend the training program over the course of the three months from July to September 2012. This profile reflects the overall sworn personnel cohort at KDPS, despite the sample being only 53 percent of the department’s sworn staff. Absences from training may be attributed to the following reasons. First, during the summer months, many officers are out of town and unable to complete all 3 training modules. Second, synchronizing work scheduling and availability for inservice at KDPS is inherently problematic given the fluctuating nature of officers’ workloads. The n reported differentially in Table 4.1 items reflects the number of officers who were able to complete survey items, accounting for question non-response. 116 Upon request, KDPS provided demographic information with which to make comparisons between the overall sworn personnel cohort and the sample n. The department, however, does not keep demographic information as distinct as that which was revealed in the survey instrument, limiting statistical comparisons to ethnicity and gender criteria. However, based on available demographic information made available for this study, it can be said that the sample n of 109 participants represents the overall personnel cohort of 205 officers across these characteristics. First, KDPS internal statistics showed an ethnicity breakdown similar to the sample cohort. The sample demonstrated that (of 97 participants responding to the ethnicity question) 5.5 percent reported they were African-American, 1.8 percent reported as Hispanic, .9 percent reported as Asian or Pacific Islander, 2.8 percent reported as American Indian, and 1.8 self-reported as mixed ethnicity with the remainder (76.1 percent) reporting as white. By comparison, the overall personnel cohort at KDPS reflects 8 percent as African-American, 3.9 percent as Hispanic, .9 percent as Asian or Pacific Islander, .4 percent as American Indian, and the remainder (86.8 percent) as white. The exclusion of a count of those self-identifying as “mixed ethnicity”, an item choice on the survey instrument, may account for some differences in these demographics. In terms of gender, the sample and the overall personnel cohort differ with the sample showing 9.1 percent female and the personnel pool showing 16.3 percent female. One-sample Chi Square results for ethnicity variables (χ2=9.499, df=5, p=0.0907) and gender variables (χ2=.4902, df=1, p=0.4839) displayed no statistically significant differences between sample and population groups, validating the conclusion that the study sample reflects the atlarge population of KDPS officers. 9 In the table, a picture of the department results that appears to portray the “typical” Kalamazoo officer as a younger or middle-aged white male with some 9 Due to small sample sizes, Chi Square results should be interpreted with caution. 117 police experience, a college education, and who spends some time on their own practicing police skills. This last demographic variable is critical in adaptive expertise dimensions as it determines levels of familiarity with trained phenomena. It is likely that officers from different specializations throughout the department (such as K-9 or SWAT team members) are more engaged in this endeavor than others, as their practice time may be regulated by team or group participation. As an example, there is a robust “mixed martial arts” culture in KDPS, and many of the officers involved in this activity meet and train in a combined defensive tactics and martial arts regimen weekly. Without the ability to ask questions of officers regarding their shift assignment or specialization (which was seen by the agency as potentially compromising confidentiality), qualitative interviews remained the only source of further information about perceptions of exactly who spends time practicing off-duty. Pretest and Posttest Data Analysis Early in the data collection process, it was considered that a summative scale for each of the adaptive expertise dimensions, such as a “stress score” consisting of a number representing means of the three confirmatory evaluation variables for stress (knowledge, confidence, and understanding), could be used. By either adding Likert responses or computing means for each confirmatory variable, this scaled measurement would allow for a more global assessment of the measures (i.e., a “stress score” would be a more tractable component allowing for modifications leading to a more outcome-based framework) (Mangos & Johnston, 2008). However, by creating a summative scale for each confirmatory measure, the opportunity to observe constructs and how they operate in differential training scenarios is lost. Mangos and Johnston (2008) refer to this potentially valuable observation as measure invariance, or a specific measure’s ability to retain meaning in different contexts and environments, thus allowing for comparison in these 118 Table 4.1. Demographics of officers completing all training modules (n=109). Variable Years employed at KDPS (respondent n=104) Under 5 5-10 11-15 16-20 Over 20 Years of police service (n=104) Under 5 5-10 11-15 16-20 Over 20 Age (n=103) 21-30 31-40 41-50 51-older Ethnicity (n=97) White African-American Hispanic Asian/Pacific Islander Native American Self-identified as mixed ethnicity Gender (n=99) Female Male Highest level of education (n=103) Some college Associates Bachelors Graduate degree Hours of training on their own per week (n=95) None 1-4 hours per week 5-8 hours per week 9-12 hours per week Over 13 hours per week 119 Frequency Percent 30 27 24 9 14 27.5 24.8 22.0 8.3 12.8 22 29 19 20 14 20.2 26.6 17.4 18.3 12.8 35 32 33 3 32.1 29.4 30.3 2.8 83 6 2 1 3 2 76.1 5.5 1.8 .9 2.8 1.8 9 90 9.1 90.9 1 36 64 2 .9 33.0 58.7 1.8 23 55 10 4 3 21.1 50.5 9.2 3.7 2.8 varying applications. It is important to conceptualize training in a similar manner, as a pattern of potentially impactful stimuli which are designed to bring about different outcomes under different constructs. This is how the Kalamazoo training was initially conceived in discussions with Officer Lewis. He was attracted to the idea that targeting the overall training outcome of helping officers adapt to stressors involved multiple dimensions, in this case establishing knowledge of strategies, developing an overall global understanding of their function, and gaining confidence to use them. For these reasons, summative scales are not used in this project despite past use in other training studies (Mangos & Johnston, 2008). Future analysis of this data, although saddled by a small sample size, could explore the potential of the constructs being multidimensional, and establish a “stress scale” based on data scores over time. In this project, data limitations prohibit statements about the true dimensionality of constructs like stress or contingency planning. Applying research in determining the dimensionality of constructs could assist in this effort. 10 However, it is important not to rely on Cronbach’s alpha coefficients as a test of dimensionality, as factor analysis may disclose correlations between constructs in either a scale (such as a cumulative stress score) or latent variable pattern (Echambadi et al., 2006). Findings by training-relevant items. In order to determine means differences, items were reverse-coded because lower Likert responses (towards and including response 1) on the instrument represented higher states of agreement with the stated question. Reverse-coded, increases in means (towards and including response 7) indicate improvements. All means differences shown through paired sample t-tests are displayed in Table 4.2. Both trainingrelevant and irrelevant items are displayed to make inferences about the strength of the training 10 Reliability coefficients (Chronbach’s alpha) were computed for all items, including training irrelevant measures. Pretest items displayed an alpha of .965, reflecting high consistency or reliability of items. Posttest items displayed an alpha of .976. The n of items was 27 for each Chronbach’s alpha test. In the absence of exploratory factor analysis, these results support further exploration of multidimensionality of the items. 120 Table 4.2. Paired sample t-test results, training-relevant and irrelevant items (n=109). Variable (scale 1-7) Training-relevant items Stress reduction Knowledge of stress-reduction strategies Confidence in reducing stress Understanding how to reduce stress Problem-solving Knowledge of problem-solving strategies Confidence in problem-solving Understanding how to solve problems Crisis management and response Knowledge of crisis-management strategies Confidence in responding to crises Understanding how to manage crises Complex information processing Knowledge of information-processing Confidence in processing information Understanding how to process information Error management Knowledge of error management Confidence in processing errors Understanding how to manage errors Contingency planning Knowledge of contingency strategies Confidence in contingency planning Understanding how to plan for contingencies Training-irrelevant items Officer safety strategies Knowledge of safety strategies Confidence in using safety strategies Understanding the purpose of safety strategies Community Policing Knowledge of community policing strategies Confidence in using CP strategies Understanding community policing strategies Reporting procedures Knowledge of reporting procedures Confidence in completing reports Understanding how to complete reports *P < 0.05; **P<0.01; ***P<0.001 Pre mean Post mean 5.36 5.69 5.85 5.67 5.80 5.99 .31 .11 .14 .857 .975 .751 3.798 1.179 1.913 108 108 108 .000*** .241 .058 5.52 6.08 5.90 5.76 6.07 6.06 .24 -.01 .16 .784 .687 .818 3.192 -.139 1.989 107 108 108 .002** .889 .049* 5.46 5.70 5.76 5.62 5.96 5.94 .16 .27 .17 .870 .715 .678 1.888 3.882 2.683 106 108 108 .062 .000*** .008** 5.17 5.73 5.58 5.57 5.92 5.70 .39 .18 .12 .953 .696 .868 4.323 2.752 1.434 108 108 108 .000*** .007** .155 5.29 5.62 5.64 5.67 5.82 5.88 .38 .19 .24 .837 .833 .779 4.694 2.414 3.133 108 108 104 .000*** .017* .002** 5.17 5.59 5.49 5.53 5.74 5.73 .36 .16 .25 .918 .873 .818 4.068 1.865 3.161 108 108 108 .000*** .065 .002** 5.68 5.94 5.90 5.95 6.11 6.10 .27 .16 .20 .731 .631 .704 3.930 2.731 2.992 108 108 108 .000*** .007** .003** 4.83 5.22 5.25 5.20 5.32 5.38 .37 .10 .13 1.033 1.036 .982 3.708 1.017 1.365 108 108 108 .000*** .311 .175 5.48 5.57 5.85 5.62 5.68 5.94 .15 .11 .08 1.061 .985 .818 1.444 1.167 1.054 108 108 108 .152 .246 .294 121 Paired differences Mean SD t df p through the internal referencing strategy as employed in the analysis. Results of paired-sample ttests show some statistically significant mean differences on retention of some of the core prerequisites for adaptive expertise. As a training program, the six constructs included in training curricula throughout the course of the 90-day training project demonstrate overall strong support for the taxonomies outlined in Lazzara et al. (2010) and Pulakos et al. (2000) in inculcating knowledge and skills related to the development of adaptive expertise. This gives modest support for the training program in accomplishing theoretical goals. It is important to note that the use of multiple t-tests creates an accentuated risk of type I error, or a rejection of a null hypothesis when it is in fact true. This project analysis’ reliance on multiple t-tests to examine means differences produces the possibility that statistically significant relationships may be misidentified due to chance because of the recurring frequency of similar statistical tests. With a significance level of .05 used, the probability of obtaining a statistically significant relationship increases with each successive t-test over twenty; the number of t-tests used in this analysis is twenty-seven. A Bonferroni correction may be used in future analyses of multiple paired sample t-tests of this sort. Returning to the taxonomies from which the training modules were derived, Lazzara et al. (2010) outline eight related requirements for the development of adaptive expertise which are validated by findings in this study. Each of the training goals in the KDPS training corresponds to a particular requirement for the development of adaptive expertise in the taxonomy. Specifically, the acquisition of mental models (supported by the practice component of the training), debriefing (supported by the self-evaluation component), sensemaking training involving rapidly-unfolding situations (supported by pretraining orientation and realistic 122 simulation components), the formation of mental models (supported by the diversity components) , and the development of contingency plans (supported by the planning and forecasting components) (Lazzara et al., 2010, pp. 2295-2296) are each validated by findings here. Likewise, Pulakos et al. (2000) provide a taxonomy of eight dimensions of adaptive performance, including the handling of crises, development of cultural, physical, and interpersonal adaptability, creative problem-solving, dealing with unpredictable work situations, and the handling of work stress (Pulakos et al., 2000, p. 617) are validated by findings from the six training dimensions in this study. It can be said that across dimensions, posttest findings from training-relevant items translate to the establishment of multiple core prerequisites for the development of adaptable behavior. Taken as a whole, findings broken down by expertise areas in KDPS’ training modules provide modest support for the acquisition and development of skills that both Lazzara et al. (2010) and Pulakos et al. (2000) dictate as essential components of the development of adaptive expertise. Although support for the stress reduction component is modest with only the knowledge component reflecting statistically significant improvement, this component derived from realistic simulation points to a positive relationship between improvements in simulation training performance and the ability of trainees to react to unfolding situations, a cornerstone of sensemaking training. Similarly, statistically significant changes with the problem-solving and contingency planning components indicate that pretraining orientation exercises and emails used in KDPS training impact the development of contingency plans essential to enhance adaptive expertise. Additionally, the crisis and error management components display statistically significant changes from pre to posttest, which demonstrate a positive relationship between improvements in training performance related to practice opportunities and training for diversity 123 and the acquisition and development of accurate mental models (Lazzara et al., 2010). Put in terms related to curriculum development, by breaking down each requirement for adaptive expertise into training strategies, trainers at KDPS “translated” the taxonomic strategies into realizable training goals with an ulterior purpose which impacts adaptive expertise development. Findings by elements within each construct. Within each training-relevant and training-irrelevant construct, t-test findings indicate support for the use of Mangos and Johnston’s confirmatory evaluation model as a manner by which the constructs themselves can be dissected into discrete training performance measures (Mangos & Johnston, 2008, p. 306). In most constructs, these statistically significant differences follow a distinct pattern of being significant across the “knowledge” and “understanding” measures, while being statistically insignificant (only 3 out of 6) for the “confidence” measure. In the case of nearly all the “knowledge” measures, statistical significance is noted; the sole exception is the “knowledge of crisis management strategies” measure with a p value of .062. In the case of the “understanding” measures, there was significance in nearly all areas but “understanding how to process complex information”, which indicated a value of .155, and “understanding how to reduce stress”, which indicated a value of .058. Since each of the three training modules embedded different constructs, it is possible to examine the specific training modules themselves to look for indicators of areas needing improvement. The two constructs related to the “understanding” measure that displayed the lowest statistical significance, stress strategies and complex information processing, were embedded in the July training involving live firearms scenarios and the September training involving simulations with MILO. These two training environments could be adjusted to account for these results, and enhancements made to resolve perceived shortcomings in specific areas illuminated by the results. 124 The “confidence” construct was statistically significant on only half of the variables, notably “confidence in responding to crises”, “confidence in processing complex information”, and “confidence in processing and responding to errors”. This raises the question of why the confidence construct did not display a greater statistical effect as a result of training. It is possible that in the department there exists an organizational impediment to higher levels of confidence (an issue which will be discussed later in relation to results from ride-along posttraining interviews with officers and newly-promoted middle managers and sergeants). If confidence is indeed a training problem, to the department could focus on two potential issues in the future with regard to these results. First, the training module associated with the statistically insignificant constructs of stress, problem-solving, and contingency planning was the September MILO simulation training. It is possible that this training did not contain a component designed to increase the trainee’s confidence across these specific dimensions, while simultaneously providing them with an understanding of and knowledge of specific strategies. Second, Lieutenant McDowell made a comment about the departmental shortcomings with regard to training officers to become more confident, claiming that departmental focus on liability and interference with scheduling produced an inability for newly-hired officers to acclimate to new role expectations (McDowell, personal communication, October 6, 2012). A discussion between Lieutenant McDowell, Officer Lewis, and the researcher about whether or not the training division could actually “teach” or “train” confidence, and how this may be affected, was a result of the initial survey and training discussions immediately following the end of the training program. The training division continues to presently explore this issue. Training-irrelevant means differences. The inclusion of training irrelevant items in the internal referencing strategy called attention to the potential effect of external influences on 125 pretest-posttest results. Training-irrelevant items in the IRS were expected to remain stable and display no statistically significant means differences because they were untrained constructs. Ttest results are illustrated in Table 4.2. One construct, officer safety, was statistically significant across all three measures. But the other two constructs, community policing strategies and reporting procedures, showed only one measure as statistically significant: “knowledge of community policing strategies” appeared to show improvement after training. Curiously, it appears that knowledge, confidence, and understanding of officer safety strategies improved, while measures of other NEDV did not, with minimal exception. These reasons for across the board improvement in officer safety and little to no improvement elsewhere among NEDV may be theoretically explained by the nature of training at KDPS. One potential reason for the statistically significant increase in knowledge, confidence, and understanding of officer safety strategies is that the KDPS training division has committed themselves to embedding officer safety training so deeply within all of their training modules that it may not be perceived as a stand-alone training topic, yet it is “trained” because it is infused throughout their overall organizational ethic. This was reiterated by Officer Lewis following initial data analysis (Lewis, personal communication, January 4, 2013). In this case, it is apparent that the inclusion of officer safety as a nonequivalent dependent variable may have been a poor choice for the internal referencing strategy. It can be said that this variable does not provide a clear picture of the effect of training irrelevant items as it is not an irrelevant construct in KDPS’ training regimen. However, the other two variables, community policing strategies and completion of reports, show little or no statistical significance across the trained measures with one exception, providing some support for the internal referencing strategy. Disregarding 126 the items for officer safety, which should not have been included as training irrelevant, there is support for the six training-relevant measures having been impacted by training. Training-relevant models: Ordinal logistic regression. This section presents models evaluating the relationship between predictor variables and performance on training-relevant measures. The overall function of the models presented here is to determine if the recognizable impact of the training, as demonstrated through promising t-test results, can still be noted once control variables are accounted for. Both demographic variables (measured in these models as age, ethnicity, and gender) and experience variables (measured as years of experience, educational level, and hours of training outside the department) are hypothesized to impact training performance, and holding these variables constant in ordinal regression may reveal the potential effect of training on outcomes aside from these predictors. By either predicting training performance using demographic or experience variables, or ruling them out in favor of training effects, this section clarifies statements about training impact explored in paired sample t-tests. First, a few comments about the use of ordinal logistic regression will clarify the utility of this technique in contrast to the more traditionally-used least-squares regression. The outcomes in each model (across 18 training-relevant dependent variables, as indicated in Table 4.2) are ordinal in that ordinal Likert responses can be analyzed more accurately with the use of a logit or odds ratio measurement. As an example, this odds ratio statistic allows the researcher to see the odds that a respondent on the posttest questionnaire answering that they “strongly agree” with statements regarding self-efficacy can fall into specific predictor categories, such as their gender or age group, or whether they spend a specific amount of time training on their own. The models can also remove variation in the posttest scores by controlling for pretest scores as a potential predictor. The use of ordinal logistic regression predicts trainees who are at greater odds for 127 responding in a specific manner on post-training measurements of their self-efficacy in understanding concepts, attaining new knowledge, or feeling confident across the measures of adaptability. As stated by Tabachnick and Fidell (2007), this form of regression is therefore more appropriate in circumstances when the distribution of responses cannot be assumed to be normal. In sum, logistic odds ratios represent the likelihood that a trainee would be in a higher level of dependent variable, in this case answering more favorably towards the 7th or “strongly agree” category of Likert response on questions pertaining to training self-efficacy (Field, 2013). Model parameters. Model results are reported in Tables 4.3-4.8, showing logistic coefficients and their standard errors, odds ratios, and Wald statistics for each item on the posttest questionnaire. The dependent variables are illustrated in tables consistent in order with the manner they are listed in Table 4.2. Two groups of control variables, demographic characteristics and variables measuring experience, are used; both groups have different precedent in training evaluations. There is little theoretical precedent supporting the examination demographic variables’ impact on the establishment of expertise in trainees. None of the taxonomical literature from which the training modules used in this study were derived emphasizes the potential for an individual’s age range, ethnicity, or gender to influence the development of skills needed for expertise. However, by accounting for these variables, the following models accomplish two tasks. First, statements regarding training effects are clarified by holding observations constant across age, gender, and ethnicity. Second, relationships between trainee demographics and training outcomes may have important policy implications, especially in a department that, judging from sample characteristics indicated in Table 4.1, is somewhat lacking in diversity. Unlike demographic variables, however, experience is seen theoretically as a determining factor in the establishment of expertise in trainees (Chi, 2011; 128 White et al., 2005). Chi (2011) defines expertise in part as having experience, suggesting that on-the-job behaviors modeled in training would be performed differently by those who have higher levels of job experience. Experience is measured in three ways in the study: years of police experience, level of education, and hours of training on one’s own. All of these six variables are accounted for in the models which use posttest scores on all six training modules as dependent variables. Pretest scores are used as a control variable to differentiate between training effect and the independent variables of experience and demographics. 11. Due to the large number of dependent variables used in this project, and the potential difficulty in reading Tables 4.3 through 4.8 as a consistent window for interpretation, Table 4.9 is provided to summarize the models and flag statistically significant relationships demonstrated through model results. In Tables 4.3 through 4.8, confidence intervals are indicated at the bottom of each chart. Statistical significance by conventional 95% confidence intervals is indicated with a single asterisk. One of the risks of using conventional confidence intervals in the case of a small-sample case study is that the relative urgency (for policy decisions and training-related improvements) of findings is acute. Therefore, confidence intervals at 99% are also indicated with a doubleasterisk. Caution should be used when examining results using restrictive confidence intervals with a small sample size. However, considering results at stricter confidence intervals accounts for the potential important policy implications of training findings, because should statistical 11 Data preparation procedures included multicollinearity and heteroscedasticity diagnostics. Multicollinearity diagnostics were performed for training-relevant and training-irrelevant independent variables, and control variables, resulting in 108 possible correlation combinations. Of those 108 combinations, the average Variance Inflation Factor (VIF) value is 1.02 with a range from 1.00 to 1.11. These VIF statistics demonstrate that, at a minimal level, some inflation of standard error occurred as a result of the variables in the model, but the statistic does not meet the accepted criteria for multicollinearity (VIF of 4.0 or higher) to interfere with regression findings. Additionally, tests for heteroscedasticity were performed on all model variables. Heteroscedasticity was not detected from scatterplot analysis and the placement of fit lines which were consistently horizontal. As explained by White (1980), the normal distribution of residuals indicates constant variance across the data set and the approximation of a properly specified model. 129 significance lead to a police agency adopting training for presumed effect, the study wants to impart a sense of confidence that findings are driving such potential changes. Police managers would want to be as certain as possible that training changes were actual before embarking on wholesale changes. At the same time, a more lenient confidence interval is also indicated by a dagger (†), indicating a 90% confidence interval. The use of this interval is to illustrate potential changes that are seen because in some cases, researchers would be willing to risk a greater error in order to note changes in trainee response as being related to variables in the model. Driving the use of this lenient interval is the note that, upon initially running the model, it appeared that few if any relationships were statistically significant, and there appeared to be no discernible pattern to these statistically significant relationships. The use of the .10 confidence interval allows for a broader analysis of trends in the data with an understanding that they may fall short of implying any positive training effect (Field, 2013). Model parameters attempted to determine the extent to which posttest responses were different than pretest responses on training questionnaires while accounting for demographic and experience variables. As stated previously, theoretical approaches to these questions differ with respect to training studies, and each carries a potential statement about not only this particular police training exercise, but training for expert performance as a whole. Classes of independent variables used in the models were listed as demographic and experience items on the pretest questionnaire. First, the age of an individual trainee could potentially impact their training performance because the accumulation of knowledge over time, and expert-level skills as a professional, may be reflected in their chronological age. An individual’s gender, while not theoretically tied to potential improvements in expert performance, could reflect dissimilar 130 Table 4.3. Ordinal regression for demographic and control variables on performance on stress-related posttest questions (n=109). Understanding stress reduction b 13 OR 13 Wald b OR Confidence in reducing stress Wald b OR Wald Threshold variables 3 4 5 6 .001** (1.546) .021* (1.229) .467 (1.194) .231 (1.197) -5.088 -2.831 -.868 1.433 10.839 5.305 .529 1.435 .002* (1.224) .058† (1.171) .566 (1.162) .257 (1.164) -3.834 -2.224 -.667 1.320 9.810 3.604 .330 1.285 .005*(1.234) .148 (1.213) .525 (1.205) -3.429 -1.755 .766 7.726 2.094 .404 Demographic control variables Pretest score Age Gender: Female Ethnicity: Non-white .478 .784 .146 .551 (.285) (.279) (.523) (.461) .202 .076 -.759 -.274 .504 .075 2.113 .355 .069† (.278) .729 (.270) .762 (.507) .427 (.448) .504 .094 .154 -.356 3.295 .120 .092 .631 .714 .554 .884 .117 (.286) (.281) (.528) (.471) .105 .166 -.077 -.737 .134 .350 .021 2.452 .194 .952 .422 .764 .255 .662 (.317) (.452) (.950) (.908) (1.018) (1.083) .411 -.027 .763 -.273 1.158 .474 1.685 .004 .644 .090 1.295 .192 .715 .519 .381 .355 .315 .941 .111 .282 -.813 -.823 -.993 -.079 .134 .415 .766 .854 1.010 .006 .675 .195 .657 .136 .416 .401 (.317) (.456) (.963) (.927) (1.027) (1.109) .133 .591 -.428 -1.381 -.836 .932 .176 1.676 .197 2.218 .662 .705 Experience control variables Highest education: Bachelors ≥ Years of experience: 10≥ years Hours of training on your own: 0 1-4 5-8 9-13 †P≤ 0.10; *P ≤ 0.05; **P ≤ 0.01 12 12 Knowledge of stress-reduction strategies Standard errors are in parentheses. Odds ratios, or the Exp(b) in SPSS output, where a 1.0 represents even odds. 131 (.305) (.437) (.929) (.890) (.989) (1.058) Table 4.4. Ordinal regression for demographic and control variables on performance on problem-solving posttest questions (n=109). Understanding the role of PS b OR Wald Threshold variables 3 4 5 6 .006**(1.324) .573 (1.205) .152 (1.212) -3.658 -.679 1.734 7.628 .317 2.049 Demographic control variables Pretest score Age Gender: Female Ethnicity: Non-white .129 (.290) .283 (.284) .027* (.537) .110 (.470) .441 .305 -1.185 -.752 .685 .963 .322 .867 .261 .166 .130 .021 .954 .154 1.154 1.538 Experience control variables Highest education: Bachelors ≥ Years of experience: 10≥ years Hours of training on your own: 0 1-4 5-8 9-13 †P≤ 0.10; *P ≤ 0.05; **P ≤ 0.01 (.319) (.457) (.962) (.918) (1.027) (1.110) Knowledge of PS strategies b OR Wald .001** (1.294) .058 (1.180) .587 (1.167) .209 (1.171) -4.321 -2.238 -.633 1.471 11.149 3.599 .294 1.579 2.309 1.154 4.862 2.559 .122 (.280) .870 (.272) .385 (.510) .094† (.456) .433 .044 -.443 -.765 .165 .002 .983 .028 1.263 1.921 .488 .806 .843 .889 .508 .446 .214 .108 .184 -.124 .657 .811 132 (.308) (.441) (.929) (.889) (.993) (1.064) Confidence in using PS strategies b OR Wald .001**(1.564) .001**(1.397) .150 (1.220) .530 (1.213) -5.404 -4.706 1.755 .761 11.945 11.349 2.071 .394 2.394 .027 .753 2.809 .522 .486 .108 .254 (.290) (.285) (.536) (.472) -.186 .199 -.861 -.539 .410 .485 2.580 1.301 .481 .060 .039 .019 .438 .581 .907 .793 .557 .924 .360 .739 (.321) (.461) (.965) (.923) (1.032 (1.099) -.038 -.121 .567 -.088 .946 .366 .014 .069 .345 .009 .840 .111 Table 4.5. Ordinal regression for demographic and control variables on performance on crisis-related posttest questions (n=109). Understanding crisis management b Knowledge of crisis-management strategies OR Confidence in using crisismanagement strategies OR Wald b Wald b OR Wald 8.034 3.762 .002 3.832 .008** (1.350) .469 (1.172) .459 (1.171) .012* (1.191) -3.607 -.850 .867 2.984 7.139 .526 .548 6.272 .099† (1.212) .703 (1.179) .032* (1.196) -2.000 .450 2.560 2.721 .146 4.584 Threshold variables 3 4 5 6 .005** (1.545) .052 (1.237) .968 (1.192) .050† (1.207) -4.378 -2.400 .048 2.362 Demographic control variables Pretest score Age Gender: Female Ethnicity: Non-white .167 (.286) .112 (.282) .078 (.528) .022* (.469) .395 .447 -.930 -1.079 1.907 2.521 3.105 5.280 .106 .054 .325 .347 (.279) (.275) (.509) (.450) .452 .531 -.501 -.423 2.618 3.715 .967 .885 .052† (.285) .108 (.279) .010* (.531) .282 (.458) .553 .449 -1.361 -.492 3.776 2.590 6.568 1.155 .448 (.316) .827 (.452) .121 (.957) .465 (.911) .108 (1.021) .453 (1.085) -.240 .099 1.485 .665 1.642 .815 .576 .048 2.408 .533 2.587 .564 .318 .761 .934 .831 .567 .618 (.308) (.441) (.933) (.894) (.995) (1.067) .308 .134 .077 -.191 .570 .532 .996 .093 .007 .045 .328 .249 .474 .882 .241 .654 .147 .754 -.224 -.066 1.107 .403 1.464 .335 .512 .022 1.376 .201 2.108 .098 Experience control variables Highest education: Bachelors ≥ Years of experience: 10≥ years Hours of training on your own: 0 1-4 5-8 9-13 †P≤ 0.10; *P ≤ 0.05; **P ≤ 0.01 133 (.312) (.447) (.943) (.899) (1.008) (1.070) Table 4.6. Ordinal regression for demographic and control variables on performance on complexity-related posttest questions (n=109). Understanding how to recognize complex situations b OR Wald Knowledge of how to navigate complex situations b OR Confidence in navigating complex situations Wald b OR Wald Threshold variables 3 4 5 6 .001**(1.539) .098† (1.188) .892 (1.176) .089† (1.185) -5.308 -1.966 -.160 2.015 11.893 2.740 .019 2.890 .035 * (1.231) .629 (1.164) .196 (1.167) .004** (1.193) -2.592 -.562 1.509 3.404 4.437 .233 1.674 8.141 .036 *(1.217) .829 (1.192) .166 (1.197) -2.551 -.257 1.660 4.392 .046 1.921 Demographic control variables Pretest score Age Gender: Female Ethnicity: Non-white .249 (.281) .318 (.275) .002** (.527) .236 (.453) .324 .275 -1.615 -.538 1.329 .996 9.380 1.406 .006** (.284) .256 (.274) .041* (.519) .831 (.451) .782 .311 -1.061 -.096 7.588 1.293 4.185 .046 .168 (.282) .055† (.278) .002**(.534) .029* (.460) .389 .534 -1.682 -1.006 1.903 3.681 9.921 4.791 .768 .388 .341 .912 .181 .940 .091 -.384 .914 -.099 1.345 .080 .087 .746 .944 .012 1.789 .006 .382 .843 .533 .926 .342 .938 .269 -.087 .578 .082 .939 .082 .764 .039 .390 .009 .903 .006 .802 .994 .394 .948 .406 .895 -.078 -.003 .814 -.059 .845 .142 .063 .000 .726 .004 .692 .017 Experience control variables Highest education: Bachelors ≥ Years of experience: 10≥ years Hours of training on your own: 0 1-4 5-8 9-13 †P≤ 0.10; *P ≤ 0.05; **P ≤ 0.01 (.310) (.445) (.940) (.897) (1.005) (1.067) 134 (.308) (.441) (.926) (.885) (.988) (1.054) (.310) (.444) (.956) (.912) (1.016) (1.078) Table 4.7. Ordinal regression for demographic and control variables on performance on error-management posttest questions (n=109). Understanding error recognition b OR Wald 3.836 .063 2.611 Threshold variables 3 4 5 6 .050† (1.213) .802 (1.192) .106 (1.200) -2.376 -.299 1.940 Demographic control variables Pretest score Age Gender: Female Ethnicity: Non-white .052 .213 .406 .103 (.287) (.279) (.519) (.467) .559 .348 -.431 -.762 .293 .569 .784 .613 .494 .669 (.316) (.451) (.948) (.909) (1.017) (1.082) .332 .257 .260 -.460 .696 .463 Experience control variables Highest education: Bachelors ≥ Years of experience: 10≥ years Hours of training on your own: 0 1-4 5-8 9-13 †P≤ 0.10; *P ≤ 0.05; **P ≤ 0.01 Knowledge of error-management strategies b OR Confidence in using errormanagement strategies Wald b OR Wald .058† (1.275) .995 (1.159) .123 (1.164) .002** (1.192) -2.417 -.008 1.796 3.728 3.591 .000 2.383 9.784 .003**(1.538) .088† (1.214) .805 (1.191) .030* (1.208) -4.543 -2.073 .294 2.620 8.719 2.915 .061 4.584 3.792 1.552 .689 2.659 .008** (.281) .049 * (.274) .548 (.508) .180 (.449) .748 .541 -.305 -.603 7.065 3.889 .361 1.800 .050† (.287) .179 (.280) .176 (.524) .269 (.461) .561 .376 -.710 -.509 3.827 1.807 1.832 1.220 1.106 .324 .075 .256 .468 .183 .228 .348 .469 .630 .252 .410 .370 .413 .667 .425 1.128 .867 1.454 .881 .524 .232 1.311 .679 .377 .754 .546 .794 .313 .600 .278 .141 .573 -.237 1.024 .569 .780 .098 .364 .068 1.018 .275 135 (.307) (.440) (.922) (.882) (.985) (1.052) (.315) (.451) (.950) (.909) (1.015) (1.084) Table 4.8. Ordinal regression for demographic and control variables on performance on contingency-related posttest questions (n=109). Understanding contingency planning Knowledge of contingency strategies b OR Wald b Threshold variables 3 4 5 6 .015* (1.349) .599 (1.170) .287 (1.169) .008** (1.190) -3.269 -.615 1.245 3.174 5.871 .276 1.135 7.114 .122 (1.208) .849 (1.166) .095† (1.173) .002** (1.202) -1.870 -.222 1.956 3.799 2.397 .036 2.782 9.992 .194 (1.188) .757 (1.176) .034* (1.192) -1.542 .364 2.527 1.685 .096 4.491 Demographic control variables Pretest score Age Gender: Female Ethnicity: Non-white .024* (.281) .021* (.277) .063† (.514) .154 (.451) .632 .639 -.957 -.643 5.064 5.338 3.460 2.036 .013* (.284) .083† (.276) .091† (.518) .459 (.453) .705 .478 -.875 -.336 6.180 3.009 2.855 .549 .089† (.282) .159 (.277) .080† (.517) .365 (.454) .480 .390 -.904 -.411 2.894 1.987 3.065 .820 Experience control variables Highest education: Bachelors ≥ Years of experience: 10≥ years Hours of training on your own: 0 1-4 5-8 9-13 .576 (.307) .851 (.440) .360 (.931) .818 (.888) .239 (.994) .753 (1.057) .172 .083 .853 .204 1.170 .332 .313 .440 .839 .053 1.387 .099 .490 .797 .300 .663 .182 .672 .213 -.114 .963 .387 1.323 .448 .477 .066 1.075 .190 1.780 .180 .432 .549 .639 .797 .731 .521 .244 .267 .441 -.231 .344 .688 .618 .358 .221 .066 .118 .411 †P≤ 0.10; *P ≤ 0.05; **P ≤ 0.01 136 (.309) (.443) (.929) (.887) (.992) (1.057) OR Confidence in contingency planning Wald b (.311) (.445) (.939) (.898) (1.000) (1.073) OR Wald Table 4.9. Summary of ordinal logistic regression models and statistically significant relationships (n=109). Threshold variables Model Variable 3 Stress reduction Knowledge of stress-reduction strategies Confidence in reducing stress Understanding how to reduce stress Problem-solving Knowledge of problem-solving strategies Confidence in problem-solving Understanding problem-solving strategies Crisis management and response Knowledge of crisis-management strategies Confidence in responding to crises Understanding how to manage crises Complex information processing Knowledge of information-processing Confidence in processing information Understanding how to process information Error management Knowledge of error management Confidence in processing errors Understanding how to manage errors Contingency planning Knowledge of contingency strategies Confidence in contingency planning Understanding planning for contingencies †P≤ 0.10; *P ≤ 0.05; **P ≤ 0.01 4 * † * * * ** ** 5 Pretest † * * * ** * † * † * ** ** * † † † † † † † * Ethnicity 6 ** ** † ** Gender † ** ** Age ** * ** ** * ** * * ** ** * † † † * 137 * † † † * Higher Education Years Experience Hours of Practice training effects and lead to important policy uses for agencies considering implementing new training models. It is also helpful since, as noted in Table 4.1, there are few female officers at KDPS, necessitating the potential relationship between training effect and a minority cohort. The third demographic variable is ethnicity, which like gender, may carry important policy implications for training improvements given the nature of those officers’ relationships with community subgroups. Additionally, the use of gender and ethnicity variables allows us to glimpse potential training effects on different types of officers at position points within the organization; potentially, this could indicate strong support for future promotions and diversity in hiring practices. However, the experience parameters are theoretically tied to potential expectations for training performance in adaptive expertise (Chi, 2011; Sonnentag, 2000). This part of the modeling process is viewed as a critical exercise in answering the question implied by the Los Angeles RAND study regarding expertise and its nebulous definition in police training (Glenn et al., 2003): what are the necessary contributors to the development of expertise among police officers? The first experience variable used, highest education received, is indicated as a dummy variable reflecting a bachelors’ degree or higher. The use of this variable allows us to determine the potential impact of a trainee’s higher education on self-reported training changes in accordance with Sonnentag’s (2000) statements regarding the interplay between experience and exceptional performance, and the importance of experience in higher-level thinking for the development of expertise. Second, years of police experience (again dummy coded, here as “10 years or higher”) may potentially impact the manner by which certain skills and knowledge sets are retained in police officers, since their familiarity with circumstances on-the-job is potentially reflected in the way they learn from realistic simulation drills. When training reflects reality, 138 individuals with additional experience in actual job tasks may perform better in simulations of those realities. Third, some police officers are apt to engage in training on their own, off-duty or within special units such as tactical teams or investigations. The impact of on-one’s-own training on performance could carry important policy implications for the department if it is apparent that extra training results in enhancements in the way departmental in-service training is received. Model results and interpretation. In order to interpret results, we must first examine Wald statistics for all parameters from Tables 4.3 through 4.8. Statistically significant relationships for all 3 potential confidence intervals are identified, and likewise flagged in Table 4.9. These relationships are examined in depth in the following sections, organized by the independent variables. Discussion of the implications of these potential relationships for training constructs are embedded in these sections in the interest of economizing interpretation. Response outcome categories as compared to highest possible response. Ordinal regression allows for the comparison of all response outcome categories of a Likert scale to the constant, or estimate, which in the case of this survey is the response “7” indicating “strongly agree”. Since there are more than two categories of response (as in a scenario where binary responses are used), multiple equations are generated to compare each response to the comparison, such as the response “3” to the “7”, the response “4” to the “7”, and so on. Therefore, the different responses are seen as “estimates” of the comparison score of “strongly agree” (Tabachnick & Fidell, 2007). In examining the results of the models here, the significant relationships are expected as one of two designs: artifacts of small sample size, or theoretically consistent with expectations of self-report of training improvements. First, all but one measure (“knowledge of contingency strategies”) as response “3” is shown to be a statistically significant predictor, even at the strictest significance criteria of .001, of the highest categorical response 139 (“7”). In three cases, the number of “3” responses is one, and all are less than four. In understanding statistical significance of the “4” response, the highest number of “4” responses in one category is eight. Therefore, it is likely that the statistical significance shown comparing the “3” and “4” responses to the “7” threshold, especially in cases of a 99% confidence interval, are artifacts of an extremely small sample size. This situation is particularly acute with the “3” response given inflated odds rations (Tabachnick & Fidell, 2007). According to the estimates, a one-unit increase in each response along the “3”, “4”, “5”, or “6” category increases the ordered log odds of being in the “7” category, meaning that individuals tended to increase their responses according to how they responded in the pretest. This is borne out by the relative lack of consistency of the variable reflecting the posttest score controlling for the pretest score. (All but the “contingency planning” dimension displays consistent statistical significance). In sum, these noted estimates using “7” as the comparison appear to be largely artifacts of the extremely small sample among “3” and “4” responses and not a sign of prediction. Second, it is expected that self-report of training self-efficacy will improve from pretest to posttest across each Likert response toward the highest category, or “strongly agree” with statements of improvement. As an example, looking at the individual models at a 95% confidence interval, “knowledge of crisis management strategies” illustrates that for a one-unit increase in a “6” response”, we would expect a 2.98 increase in odds of being in a higher category (in this case, the “7” response). There is consistent positive expectation of ordered log odds improvement across other statistically-significant variables, “confidence in responding to crises” (2.56 OR), “confidence in error processing” (2.62 OR), and “confidence in contingency planning” (2.52 OR) (as seen in Tables 4.3 through 4.8). It is evident that across these categories, should a respondent indicate on a pretest that they are somewhat lacking in 140 confidence, controlling for all other factors, there is a statistically significant relationship between their pretest and posttest statements. However, again, the improvement from “6” to “7” is important to note here. If an individual responds that they are just under “strongly agree” on a pretest measurement, and state that they are at the threshold of “strongly agreeing” on a posttest measurement, what form of realistic improvement has really been made? It is difficult given this data to determine if such results indicate anything regarding training improvement since the fractional improvements are subtle and potentially reflecting a small sample. At the very least, this model result demonstrates that pretest measurements can be somewhat predictive of posttest improvements; in other words, a trainee’s pretest score may predict their posttest score across some knowledge and skills improvements. Demographic attributes. As previously stated, there is no theoretical precedent for an expectation of a relationship between demographic variables of age, gender, or ethnicity to impact police in-service training. However, it is legitimate to expect that, in a case study, organizational ethics and climate may play a role in driving training changes among classified personnel along certain demographic lines. Examining model results, some statistically significant predictors of training improvement are demonstrated holding all other factors constant; however, there is no discernible pattern to these responses suggestive of any policy trends. In three cases, “understanding problem-solving strategies”, “confidence in responding to crises” and “knowledge of complex information processing strategies”, the trainee’s gender was a statistically-significant predictor of a lower Likert response, indicating disagreement. At a more lenient confidence interval of .10, these same findings are noted across each of the “contingency planning” variables for gender. A statement can be made, although not a very strong and consistent one, that female officers were found to show few self-reported 141 improvements after training. Again, however, the overall n of female officers in the sample is 9, and statements made here regarding their improvement should be viewed with that knowledge. For ethnicity variables, only two variables display statistically significant relationships with nonwhite respondents: “understanding how to manage crises” and “confidence in processing information”; both odds ratios were negative. An individual’s being non-white therefore can be a statistically significant predictor of negative responses to training for those two skills. However, neither can be attributed to any sort of pattern as they appear isolated results. Curiously, even at the lenient .10 confidence interval, only one other ethnicity variable appears statistically significant. The variable age is similar to ethnicity in this regard: only two statistically significant relationships can be noted (“knowledge of error management strategies” and “understanding contingency planning”) and both are positive odds ratios (.531 and .639, respectively). This indicates minimal support for age as a predictor of positive training responses in those two variables, and coupled with only two statistically significant relationships noted at a lenient .10 interval, appears dismissive. What is one to make of the scattered model results for demographic variables as predictors? Conclusively, there does not appear to be any pattern to these responses, and aside from gender, no conceivable culprit as sample size leading to questionable results. Is it possible that there is some interaction between gender, age, or ethnicity in the Kalamazoo setting with respect to the manner by which some in-service training objectives are received and internalized by different classes of officer? If one looks across the variables at frequency of statisticallysignificant findings, even with the strict p-value of .01 a reliable pattern cannot be determined when all other variables are accounted for. However with the confirmatory model, the absence of patterns does not necessarily indicate that relationships identified hold no identifiable impact. 142 Using model results, demographics indicated as leading to differential training impacts (either positively or negatively) can assist the department in using in-service training of this variety diagnostically, such as creating training to increase non-white officers’ confidence in complex information processing, or improving instruction for older officers’ contingency planning. What exists in these scattered results is a potential template for subtle training improvements which can be utilized in conjunction with larger training goals to provide equitable training to all. Experience variables. As indicated from the tables, not one variable used as a measurement of officer experience on-the job (having a bachelors’ degree or higher, having ten or more years of police experience, or spending time training on one’s own) appears to be a statistically significant predictor of training changes across each confidence interval. This finding, although somewhat disappointing given the strong theoretical correlation between experience and expertise defined by Chi (2011), Sonnentag (2000), and others, is not an indication of the failure of experience to be relevant to internalizing training in this project. Three considerations may be made: first, the cohort of officers at Kalamazoo may potentially be of similar relative experience levels which may mask training differentiation (or experience does not matter in training efficacy as much as previously envisioned). Second, the training itself may not approximate raw experience for which such gained skills can be used. Third, what constitutes “experience” for officers may in fact differ uniquely that no single measurement could be used to capture these differences (or, potentially, the measures are not sensitive enough to capture minute changes). First, it is evident from the experience variable results (and potentially, in tandem with the demographic results) that no one group of officers in Kalamazoo consistently got more out of the training than any other group across several experience levels. In other terms, intriguingly, 143 the odds that an officer will self-report that they did improve cannot be determined by their experiential background of either training or on-the-job time. Although being contrary to theory stating that years of on-the-job experience could potentially impact an individual’s performance on training tasks for expertise (Sonnentag, 2000), the seeming absence of effect of these experiences on training suggests that wide differentiation between officers at KDPS with respect to experience as measured here does not exist. Using a study cohort of just over 100 officers in a strictly urban setting, in a department which runs its own police training academy and having a strong field-training mentorship program, it is possible that what counts as “experience” among KDPS officers may differ from constructs used here. “Experience”, then, may not be indicated by time on the job or hours spend training, or even having a college degree. Other factors such as organizational ethic disseminated through initial training upon hire, standards set over time to communicate expectations for training of experienced persons, and cohorts of newer and more motivated officers (as well as younger command staff) may drive expectations for motivation to learn that cannot be measured by calculating years on the job. Interviews with KDPS officers disclosed a positive work ethic that may dilute some of the effects brought about by diverse experience; this is a potential explanation for this model result. Another potential reason for this finding is that the training itself did not approximate experiences that officers with different attributes could interpret, including those with college degrees. Training in police work, especially training such as the high-tech MILO computer device and firearms range shooting, is a synthetic form of real police on-the job experiences which may occur rarely in the course of an officer’s career. It is possible that Kalamazoo officers would not experience the same material events that Officer Lewis simulated with these training experiences. (This may in fact, as noted later, also explain the fact that the officers 144 enjoyed the training.) It is likely that many officers will not in their lifetime experience any of the training simulations in real life, therefore it may be presumptuous to expect that experience gained in skills that happen in the course of officers’ daily work lives would provide them with preparation for high-impact training experiences. The potential explanation for this surprising model finding may lie in the nature of the training in being incongruent with officers’ daily activities, and thus resistant to the variety of changes that such experiences may render. This finding also appears to contradict much contemporary research on the role of higher education in officers’ abilities to perform differential job task assignments, yet its use for this study can possibly be said to reflect the difference between high-impact training and routine police work. Theoretically, an officer with a bachelors’ degree or higher may not necessarily be disposed to absorb a different set of knowledge, skills, and abilities from the variety of training imposed from the study in question than officers with no such educational background. This may be related to a potential disparity between what constitutes college education in the KDPS officer population and the skills needed to excel at high-impact training targeting adaptive expertise. Without any further exploration into whether or not college-educated officers at KDPS feel their educational backgrounds were of assistance in helping them successfully navigate training of this nature, this question remains unanswered. However, it is worth exploring: as will be discussed later, the training exhibited by Kalamazoo DPS in this study may be said to target adaptability more than expertise, which would potentially explain why officers with educational backgrounds preparing them for higher-level performance may not have perceived their educational preparation to be of use here. A third and final model finding was related to the failure to detect a link between officer characteristics of experience and their predictive value for training performance because of 145 training officers complete on their own. Many KDPS officers are engaged in extra-career activities such as martial arts fighting, the use of firearms as a hobby, and other non-work-related expressions of work skills which may be characterized as police-related training on-their-own. Although there exists no organizational methodology to track such endeavors, the KDPS training division has expressed interest in formalizing these interests as part of their future training passport for recordkeeping. To that end, officers were asked to report on the pretest questionnaire how many hours they normally spend training on their own per week. It was hypothesized that the intensity by which officers engage in training on their own, as measured by hours spent in such activities per week, might explain variation in training performance. Table 5-9 reiterates that model results demonstrate no support for this hypothesis. There is across-theboard evidence that this particular measure of officer experience does not result in any statistically significant explanation for variation in officer retention of adaptive expertise strategies. Only confidence in one’s ability to solve complex problems appears to be mediated by increased hours spent training on one’s own, offering no theoretical support for the overall model. In light of this finding, it is possible that the presumption that “hours spent training on one’s own” constituted an invalid measure of officers’ experience differences because the measure “hours” does not capture content of training. As such, it is plausible that, regardless of individual officers’ off-duty tasks (such as working with a tactical team or special unit), these more experienced officers do not count such activities as replacements for in-service training. For example, it was observed that officers on the tactical team who completed the three-month training program did not take the training any less “seriously” than those who were not on such teams. 146 Sensitivity check for non-response items. Each of the three model variables contains a minimal number of non-respondents whose answers may be said to potentially distort model findings. With the overall n being 109 officers, five did not respond to the question of years of police experience (n=104), six did not respond to the question of their educational background (n=103), and fourteen did not respond to the question of how many hours were spent training on their own per week (n=95). As a result of these missing cases in the models, it is important to ask what specifically is lost in the potential model results, since their exclusion could potentially distort model findings should they represent a deviant pattern. Separate crosstabulations were run for these five, six, and fourteen missing cases to examine outcomes of the training. Means differences demonstrated by these cases reveals that those trainees who did not provide an answer for these three items would had an overall negligible effect on the hypothesized models had they been included. In the interest of brevity, their individual means differences and significance levels are not reported, but a check of their significance levels indicates consistency with each significance level of the paired sample t-test on all cases. It is therefore unlikely that these minimal case omissions would have drastically affected results in each model case. Summary of pretest-posttest results. What is driving statistically significant changes from pretest to posttest as outlined in t-tests? Accounting for a host of demographic and control variables, ordinal regression model results point to pretest scores as a significant predictor in some but not all variables, and demographics also play a selective part in determining how a subject may perform. But the larger statement regarding these models is that holding constant a wide array of demographic and experience characteristics, pre-to-post changes remain. Although these changes are not discernible by isolating these characteristics, and therefore no conclusive statement can be made about the impact of demographic or experience variables on the changes, 147 it illustrates support for training being the single greatest predictor of pre-to-post changes. What is evident is statistical significance; what is risky in interpreting model results is the absence of substantive significance. As training improves incrementally from means of 5.0 to 5.2, does that realistically indicate training improvement across a cohort of 109 officers? With the model results showing inconclusive relationships among demographic controls, substantive changes may be lost in the reality that for such a singular cohort, increasing models and control variables to detect changes only results in loss of statistical power. At the same time, confidence in statistical significance as an indicator of training changes is reduced when models break down into smaller and smaller numbers of cases, such as when gender and ethnicity populations are reduced to single digits. The pretest-posttest results showed promise for the concept of translating adaptive expertise into training modules effectively and for specific results. The pre-to-post survey results indicate some support for the overall training program’s potential to inculcate base prerequisites for adaptive expertise in trainees. Although the overall number of cases is small, it can be said that the training did produce a discernible, and in some cases a statistically significant, effect on retention of adaptive expertise prerequisites. However, due to model limitations, the utility of the survey data appears limited in making statements regarding the impact of individual characteristics on training. Given the nature of model results that appear to conflict with theories of experienced persons’ performances on training tasks for expertise, the data suggest that adaptive expertise training does not maximize trainee experience of this nature, should it exist. This reveals a limitation to the pretest-posttest data in making larger statements about what Glenn et al. (2003) and Chi (2011) portray as the hidden mechanism driving training, the function of experience in establishing expertise. Experience can best act as a mediator if the 148 experience itself is consistent with training, and it is possible that police experience is limited in the high-impact events simulated in training. Due to the low number of cases and therefore the limited analysis that could be performed, notably the absence of factor analysis, it is difficult to arrive at broader conclusions regarding what may mediate officer performance on training tasks designed to develop police expertise. Glenn et al. (2003) support targeted training protocols which may theoretically target skills linked to expert levels, such as contextualized objectives, a focus on realistic scenario-based training, and training for diverse populations – all consistent with frameworks suggested by Pulakos et al. (2000) and Lazzara et al. (2010). The theoretical background to the training delivered by KDPS is therefore strong, but some methodological limitations prevent conclusive statements about the mediating effect of prior experience. Nonetheless, survey results demonstrate that the project was a positive step in the direction of further defining how such training may be delivered effectively. Open-Ended Survey Questions Disclosed Further Support for Training The training survey included open-ended questions designed to solicit impressions of the three-month training project. Overwhelmingly, trainees viewed the opportunity to engage in thoughtful and valuable practical exercises as a step forward for the training division, and provided compliments to Officer Lewis for the training. The first open-ended question asked what the trainee thought the most valuable training received in the prior three months was; the most popular response (36 responses of 89 completed) was the MILO computer simulations. Comments from officers about the practical application of such training, the ability to make splitsecond decisions in training environments, the realism of the video footage used in simulating stressful environments, the after-action debriefing and discussion with Officer Lewis targeting error management, and the dynamic nature of such scenarios (as opposed to static scenario 149 training) were among the highlighted comments. Seven officers stated that the opportunity to discuss the scenario in detail afterward and articulate reasons for decisions they made produced feelings of mastery that were unlike previous MILO experiences, reiterating the benefits of selfevaluation. Also, the inclusion of differential populations (Hispanic-speaking and mentallychallenged persons) was noted as forcing the officer out of their “comfort zone” and allowed them to develop a more holistic view of unfolding scenarios. A further comment involved the ability to train in pairs, which was a result of the 2011 training needs assessment. The second question asked what improvements could be made to the previous three modules of training, which was meant to find out if officers felt the training project had fallen short of their expectations. Although survey question results and interviews showed little support for the August training, twelve officers stated on open-ended questions that the training was an important and integral part of their awareness of how to respond to rapidly-unfolding crises by allowing them to construct their own solutions to potential issues. In light of the disparity of quality of many of the submitted projects on this module, it is likely that many saw the opportunity to engage in community outreach, self-motivated production of a specific tool which could help others, and the presentation of such information to peers and supervisors as extremely beneficial in developing feelings of cohesion. One comment regarding this training was that it took advantage of the “current issue” of the Aurora shooting quickly and appropriately because the training articulated thoughts that many officers were already having about how they would approach such situations. Overwhelmingly, a typical response to the question of suggestions for improvement was to instead state that they felt the training modules were a step forward for the training division and that they were pleased with the quality of training received. 150 These answers to open-ended survey questions hold critical implications for future training at KDPS in particular and police in-service training as a whole. While many existing discussions of police in-service training focus on content quality, strategies for improving the processes of such training are historically ignored. Qualitative data indicate here that future discussions of in-service training strategies should aim to develop confidence in officers’ abilities to apply newly-learned skills and knowledge. This returns the discussion explicitly to the lack of practice in in-service training in police venues. Additionally, respondents’ favor of specific types of in-service training activities, such as computer simulation, also underscores a major difference with regard to qualitative findings contrasted with quantitative results. The respondents indicated an enthusiasm for computer simulation training, which has multiple builtin practice components embedded within computer programs in the device used; this could potentially impact trainee confidence levels as practice is related to increased confidence. Finally, the respondents’ enthusiasm for scenario-based training (whether using the MILO system or in other realistic simulation) can be contrasted with quantitative results showing lack of development of trainee confidence. Literature on adaptive expertise makes no explicit link between scenario-based training and the development of confidence; however, it can be assumed that the development and enhancement of confidence in trainees can be impacted by the type of training techniques that are used by trainers, and the degree to which trainees like the training. Future exploration of this relationship, and an examination of the synergy between learning styles and training techniques, could unlock how confidence may be trained. Answers to Subordinate Questions Illustrate Support for Training The posttest survey used in the pretest-posttest portion of the project contained a number of items which allowed officers to provide favorability ratings to each of the three months’ of 151 training opportunities, and to reflect on the direction the training itself was headed. These Likert responses were ranked on a 1-7 scale, with 7 representing “strongly agree” as shown in Appendix 3. Table 4.10 shows the means for these survey responses. The most favorable training was the July outdoor firearms training, and the least favorable was (as predicted by the interviews) the August training which was officer-driven. Interestingly, officers’ perceptions of in-service training being applicable to everyday situations (mean of 5.71) appeared inconsistent with their impressions of the training division’s overall positive direction (mean of 6.10). It is possible that this disparity reflects organizational history, as prior to the training project a very unpopular in-service training was delivered which many officers disclosed in the interview process they felt was inapplicable to their daily work routines. Table 4.10. Responses to subordinate questions about training (n=109). Subordinate question (Likert, 1=strongly disagree, 7=strongly agree) I liked July’s tactical firearms training I liked August’s mobile active shooter training I liked September’s subject control MILO training I read pre-training materials sent by email In-service training is headed in the right direction I apply skills learned in in-service training to everyday situations Mean SD 6.56 6.08 6.50 5.45 6.10 5.71 .927 1.038 .929 1.443 .999 1.100 An interesting finding from these results involves the disposition of officers to have actually read pretraining materials sent by email by Officer Lewis regarding the three training modules (mean of 5.45). Since the pretraining email represented part of the training strategy of pretraining orientation designed to target the adaptive expertise prerequisite of problem-solving skills, this finding does present the possibility that such emails (and therefore, pretraining 152 orientation as designed in the project) were not reaching their intended audience. This suggests that statistical findings pertaining to the problem-solving component be examined with the knowledge that many officers may not have read the materials with intent, or at all. During interviews conducted with officers following training, this particular question was reiterated, with some officers stating they were unable to receive the materials in a timely manner. It is therefore unclear what distribution mechanism prevented this mean from being more indicative of widespread study of pretraining materials. Suggestions for improvement regarding the sending of pretraining materials by email were elicited. One of the suggestions was to formalize the sending of pretraining emails in a manner that all officers expect such materials on a specific schedule. Others stated that the materials be sent in a specific time-sensitive format (such as, “this month’s training bulletin”) so as to market the new concept in a manner that would be retained in the future. 153 CHAPTER 5 DISCUSSION As a field-based case study of training, this project represents an initial foray into assessing an in-service training operation that has eluded rigorous analysis. It is a pilot study with shortcomings, but also takes important steps toward understanding the hidden mechanisms of police training in context. The project has implications for both theory and practice by contributing a practical model of police in-service training and evaluation where none currently exists. However, the study acknowledges inherent limitations as a pilot study applied to a dynamic organizational environment. On observation, the Kalamazoo Department of Public Safety does not yet fully know the effects of its in-service training regimen in inculcating specific knowledge and skills sets in its officers, and the experience rendered by the study may eventually produce a stronger commitment to doing so on the part of the agency. But the aspirations of this study are to stimulate such change: it seeks to simultaneously contribute to theory while serving as a basis for eventual evidence-based practice in police training that may inform the field of policing in a time of accentuated organizational upheaval. To better articulate the overall landscape of findings derived from project data, it is appropriate to revisit to the pattern matching principle suggested by Shadish et al. (2002) with guidance from Weiss’ (1998) outline of evaluation tasks. Both of these perspectives assist in answering a basic question regarding the project: from pretraining to posttraining, did trainees improve on measures designed to act as prerequisites for adaptive expertise as theoretically defined? Additionally, what went well throughout the training effort and is deserving of attention, and what did not go well that may act as the basis for critique? Answers to these questions can illustrate both the potential impact of adaptive expertise training in this police 154 context, and how and to what end the effort to transform the department’s training regimen was potentially successful. Conclusions Regarding the Training of Prerequisites for Adaptive Expertise Chi refers to adaptive expertise in individuals using a simplistic designation, calling such behavior that of “a more innovative expert, one who is not rigid or conventional” (Chi, 2010, p. 28). The main problem with this definition, as with all other prior literature on the subject, is that no specific template exists for what behavior these individuals would exhibit, as even Chi calls this type of behavior “a notion” (Chi, 2010, p. 28). This lack of specificity leaves the current study, and other potential examinations of adaptive expertise, short-handed in attempting to determine if in fact adaptive expertise existed in practice as a result of training, as evidenced by the creation of measurement variables from literature. This chapter attempts to make concrete statements about whether or not the training undertaken by Kalamazoo Department of Public Safety in 2012 constituted training for adaptive expertise, and whether or not the trained knowledge and skills could potentially lead to individuals who behaved in a less rigid or conventional manner as police officers. First, data from the pretest-posttest analysis does suggest that core prerequisites from psychology literature that comprise adaptive expertise were trained successfully. The training curriculum founded upon taxonomies on adaptive expertise supported a vigorous training program that theoretically provides for the generalization of knowledge and skills to unforeseen contexts. This reflects the nature of adaptive expertise defined alternately as operating a “procedural understanding… [and a] predisposition to learn while performing” (Chi, 2010, p. 3132), demonstrating cultural, physical, and interpersonal adaptability (Pulakos et al., 2000), and a balanced focus on self-determination and self-regulation as part of training (Vanasupa et al., 155 2010). The training provided by KDPS during the three-month project did incorporate these and other features. Although the training was more heavily focused on training one aspect of the theory, the training did incorporate these desired behaviors as learning outcomes. Second, the manner by which the training was alternately successful in producing desired results in trainees demonstrates that the dual operations of “understanding” and “generalizing” can be dissected in this variety of training. This is not something that the KDPS Training Division has prior experience with, and was a routine finding of the study. From pretest-posttest results, there was no pronounced focus on each of the distinct forms of learning, as providing knowledge to trainees may have been a greater focus on some modules, while in others, application of learned skills was preeminent. In a sense, the use of Mangos and Johnston’s (2008) confirmatory model in dissecting knowledge, confidence, and understanding both complicated and clarified the training endeavor. It allowed for results demonstrating which form of learning was less emphasized in which modules, assisting Kalamazoo’s trainers and future researchers in pinpointing areas where strengthening of programs needs to be made. As a more general statement, although the training was “successful” in providing officers with newlyacquired skills, their confidence in using them or an overall understanding of their utility varied across each dimension. Therefore, it is not possible to make an assertive statement that the training was “successful” or that it “worked” in training for adaptive expertise unless one clarifies what specific learning dimension (providing knowledge, leading to an overall understanding, or allowing trainees to develop confidence) is involved. Third, even within this tentative focus, the variance by which knowledge, confidence, and understanding showed different results potentially indicates that “adaptive expertise” could be further defined by its learning dimension. It is possible that Pulakos et al.’s (2000) eight 156 dimensions of adaptive performance and Lazzara et al.’s (2010) eight training guidelines for adaptive expertise, the two taxonomies from which this study’s curriculum is based, could be expressed procedurally. Perhaps the initial provision of knowledge about specific strategies, the development of trainee confidence in using them, and the awareness of an overall understanding of these strategies’ roles as an avenue of behavior are three interrelated concepts for which literature and theory may benefit. It is clear from data results here that, as distinct learning objectives, knowledge, confidence, and understanding are three unique training outcomes. The next step in developing these taxonomies further, and potentially using them as actual training principles, would be to confirm these findings with a more rigorous test of their interrelated nature. A more general question is: did the training make a difference? As indicated by pretestposttest scores and analysis, the training did make a statistically significant impact on officers, and the training as delivered had multiple impacts outside the self-reported measures. It is important to note that changes in some training-irrelevant measures temper these findings to the degree that “statistical significance” cannot be taken as an indicator of positive results over the more elusive “substantive significance” discussed in the previous section. There does not appear to be any one predictor that is stronger than another, and identifying training effect is appropriate but tentative given model results. What is more intriguing is whether or not these noticeable changes entailed “training for adaptive expertise”, an issue which will be taken up in a subsequent section. But the training did matter; both in interviews and on posttest surveys, officers reported positively that training impacted a number of knowledge and skills sets that were targeted by this program. The further development of training of this nature, and further 157 evaluation of efficacy, can continue the discussion of the importance of these dimensions in developing adaptive performance in trainees. Conclusions Regarding the Training Project Itself This section addresses the issue of what went well, and what did not, in the training program undertaken. Five broad conclusions can be made about the training with regard to both issues, and are summarized in Table 5.1 alongside data sources used to arrive at these conclusions. In the table, Weiss’ (1998) dimensions of evaluation related to each conclusion are indicated. Each conclusion is then discussed in detail, with pertinent information from literature as to how the project dovetails with past suggestions about the development of such training. This section articulates “lessons learned” for the department, theory, and police practice. Conclusion 1: Adaptability and expertise may be two related goals. As stated in the last chapter, the elements of the project which entailed training for adaptability tended to eclipse those that trained for subject-matter expertise. As a result of this dichotomous training, changes noted in participants as a result of the adaptability training which formed the heart of the project cannot be said to approximate what has been theoretically advanced as “adaptive expertise” by industrial and organizational psychologists, but is potentially a form of adaptable behavior that merits further exploration. Although previous literature provided theoretical taxonomies about adaptive expertise, no authors detailed what adaptive expertise could potentially look like in individual behavior seen after training, which proved to be a difficult handicap in determining if the training targeted these prerequisite skills. It would therefore be difficult to determine in the absence of a true longitudinal study if either the training division or KDPS itself has integrated the core tenets of adaptive expertise in any transformative manner, which is the stated 158 Table 5.1. Conclusions regarding Kalamazoo training project outcomes. Conclusions regarding outcomes Data focus Weiss dimension Conclusion 1. The three training modules, as developed and implemented, reflected strategies for adaptability, but not expertise, as defined by psychological theories on the concept. Conclusion 2. The manner by which the training was implemented had an effect on training outcomes. Posttest results, analysis of documents, posttraining interviews with officers Modeling & interpreting (Weiss, p. 277, 279) Posttest results, staff interviews Disaggregating (Weiss, p. 276) Conclusion 3. The training experience was enhanced by adaptive expertise components. Posttest results, staff interviews, posttraining interviews with officers Ruling out rival explanations (Weiss, p. 275) Conclusion 4a. The training modules and project activities represented steps forward for the training division and the agency as a whole in formalizing in-service training. Conclusion 4b. It is questionable as to whether the department will be able to sustain the level of training innovation reflected in the project in the future. Conclusion 5. The training project acted as a conduit for the disclosure of underlying staffing problems related to training. Posttest results, staff interviews, posttraining interviews with officers Policy analysis (Weiss, p. 283); profiling (Weiss, 1998, p. 276) Posttraining interviews with officers Unanticipated effects (Weiss, 1998, p. 277) goal of this variety of training. What does exist, as a result of this training, is an indication that some changes as a result of the training took place which provided trainees with improvements in skills theoretically related to adaptive expertise as defined in literature. The unknown remains if this new state of knowledge represents “adaptive expertise”; this summary makes an argument that it does not. What can be said with confidence is that the results of the training project brought officers closer to a form of adaptability more closely resembling the “adaptive proficiency” supported by White et al. (2005), not “expertise” as articulated by Chi (2010), and not “adaptive 159 expertise” as described by Lazzara et al. (2010). Behavior change in the face of shifting environments, the critical component of the definition of adaptability used by White et al. (2005), can be attributed to the training program across a number of data points; however, this ability does not necessarily reflect expertise and superior performance as defined by Chi (2010) and Lazzara et al. (2010). What is missing from the analysis to make such a connection is a true longitudinal study to determine if on-the-job behavior begins to reflect expert-level performance over time. This was the critical omission from research as lamented by Sonnentag (2000), and remains difficult considering the nature of police in-service training as a phenomenon. However, attention must also be given to the lack of effort on the part of prior research in exploring the duality of “adaptability” and “expert performance”, which are potentially two related concepts which may be trained for in tandem. This project acknowledged that separation by focusing on the former and not the latter in establishing its training regimen. The root of this critical project shortcoming lies in the nature of in-service training and the manner by which evaluations of such training appear shorthanded in identifying characteristics of long-term behavior. Department-crafted in-service training is unique in police training settings, and is an attractive component of an in-house program of this nature; but it consistently defies evaluation. The realistic goal of developing agency-specific in-service training will require a longitudinal timeline to examine its true potential benefit. Although this shortcoming is important to consider in light of project results, the project remains a significant step toward defining how exactly adaptive expertise may be identified in trained persons. Glenn et al. (2003) suggest that there exists a complementary relationship between adaptability and expertise, and that the two dimensions may in fact be treated as distinct training goals. However, 160 theoretical exploration of the link between these two constructs, and the extent to which adaptive expertise is actually an amalgamation of two principles, needs further development. This challenge is noted explicitly in the Kalamazoo project’s outcomes. Constructs targeting adaptability were impacted by training, but opportunities to examine the expertise dimensions (namely, deliberate practice and a clearer picture of the impact of experience) were not. It is evident that the training over the course of the project better prepared officers for unexpected situations, but it is unconvincing to say that it made them experts at any task. Project results suggest that taxonomies could potentially be revisited in order to more accurately model which components target what outcomes. Returning to Glenn et al. (2003), a central problem driving this dilemma is unintentionally outlined: police training for adaptability is seen as mirroring the realities of police work, but what of training that surpasses the realities of the job so markedly that it allows officers to experience situations they may never encounter on the job? What specifically are the experiences which drive expertise, if high-impact training targeting adaptability provides trainees with skills they may never be able to practice in job contexts? In the Kalamazoo project, using taxonomies untested in the field to construct training has opened up the possibility of further theoretical definition of this critical construct. This is the project’s primary conclusion: future training evaluation and possibly a longitudinal assessment of expertise dimensions and factors contributing to the development of police expertise can assist in clarification of this relationship. This discussion merits inclusion because the expressed purpose of the study was give empirical support to a form of training that had no contemporary field application. As the study progressed, the training resembled less what theory describes as adaptive expertise. Aside from being a discussion of nomenclature, the above conclusion is indicated to not only affirm the 161 importance of practice (notably, deliberate practice) in the development of expert-level performance, but to place in context the usefulness of this study as a first step toward fulfilling Sonnentag’s (2000) dictum that field evaluations of adaptive expertise are lacking, and require tremendous organizational support. In order to evaluate practice of the nature described in developing expert-level performance, Kalamazoo Department of Public Safety would have had to overhaul their entire shift scheduling and training scheme in order to accommodate the researcher. This was logistically impossible barring a longitudinal evaluation, and potentially unethical given research limitations. Conclusion 2: Training implementation affected outcomes. To an extent, Kalamazoo Department of Public Safety was committed to the training program to the degree that it could be implemented within the constraints of organizational culture and logistics. As a result, modules supporting adaptive expertise prerequisites were embedded in training activities, not treated as an “addition” or “supplement” to existing training. This modification required a great deal of researcher-practitioner communication and effort. But in many ways, the program was still short of an implementation scheme that could be employed by a more rigorous analytical framework. Implementation problems abounded, as outlined in Chapter 4, and these challenges represented a project shortcoming with respect to delivering training reflective of adaptive expertise as theoretically defined. The reasons for this shortcoming are logistical and not theoretical, as the Kalamazoo Department of Public Safety Training Division naturally would be supportive of more training hours which could translate into longer practice sessions and more robust modules. Supporting this assertion are the Training Passport idea (Appendix 6), an idea to hold 12-hour training sessions for all officers that was proposed in 2011, and an idea to have mandatory 162 “training days” in which area agencies would contribute to training experiences hosted by training division (McDowell, personal communication, September 30, 2012). In this study, therefore, the form of training delivered existed as a compromise between the research objectives and the organization’s logistical constraints, and implementation impacted study results in three specific areas. First, as discussed in Burns and Freeman (2008) with respect to cross-cultural implementation of training objectives for adaptability, adjustments to training implementation are often an expected compromise across empirical divides. But in the case of the Kalamazoo project, many of these adjustments occurred on-the-fly, with great energy used to incorporating new ideas within the constraints of the project as outlined at an earlier stage. It cannot be understated that expectations of changes were surpassed by many of the augmentations that occurred at this stage. Although the training objectives had never been specifically outlined by literature on the subject and were created by the researcher and agency in tandem, the training division (and Officer Lewis in particular) implemented training that was more a product of organizational culture (namely, flexibility and spur-of-the-moment changes) than it was rooted in principle. With respect to changing the topic in August at the last minute to capitalize on a high-profile shooting, the organization took a risk that such training would be received as positively as training that reflected planning, forethought, or development months in advance. Trainees noticed this urgency, and felt as though this training was the weakest of the three in inculcating knowledge and skills for the entire three-month program. A potential relationship between this sudden change and the feedback received from officers for this module was found in interviews disclosing trainee interpretation that the August training appeared to constitute a “homework assignment” (Murray, personal communication, October 2, 2012) and “not as well developed” as other modules (Davis, personal communication, October 12, 2012). 163 The second observation is based on Durlak and DuPre’s (2008) assessment of the importance of implementation on intervention outcomes, namely their statement that collaboration, organizational ownership and input, and consensus positively impact program success and retention. In a sense, this is a reverse finding of the above referenced by Burns and Freeman (2008), because it was during the collaborative decision to change training topics for August that it can be said Officer Lewis and the KDPS Training Division “took ownership” of the training project in a desire to adapt the training project and its framework to immediate perceived needs. It was at that point, retrospectively, that program fidelity was discarded in favor of a form of adaptation that has shown positive benefit in program outcomes according to prior research (Backer, 2002). In the short term, the August training was the least popular and had fewer positive reactions from trainees; but it indicated a bold step forward for the division in its attempt to utilize a new training technique in a manner consistent with its own capacities. Durlak and DuPre (2008) state that adaptation and reinvention of project characteristics to local needs can positively affect program outcomes. In this instance, an outcome of the training that was positively affected by KDPS’ ability to modify the existing program was reflected in Officer Lewis’ conceptualization of the training as flexible across multiple training topics and tasks. This potential positive can be contrasted with the perceived negative finding that the training for that month was unpopular with trainees. Further experimentation with adapting and reinventing training for unique organizational purposes can acquaint trainers with this characteristic. A third indication that implementation had an effect on outcomes concerns the nature of the KDPS in-service training itself as mandatory in-service held department-wide, as opposed to voluntary training. The degree of choice in going through the training regimen and its effect on project outcomes has not been indicated here as a study variable, but its potential impact on 164 outcomes can be noted through posttraining interviews with KDPS officers and open-ended survey questions. As stated in Hicks and Klimoski (1987), voluntary training can positively affect retention of information, reactions to training, and motivation to learn. Officers felt that prior to the project, their opinion of in-service training was seen as not making a difference in either the quality or frequency of training received; but as a result of the data-collection and interviewing processes, officers felt connected to the training in a manner not felt previously. This transition can be said to have positively impacted training outcomes such as interest-level in training topics (Mitchell, personal communication, September 28, 2012), feelings of organizational accountability to officer needs with regard to training topics (Clark, personal communication, August 22, 2012), and motivation to attend training (Nelson, personal communication, October 1, 2012). The extent to which positive findings of this study are a product of implementation changes and the perception of organizational interest in individual officer needs is qualitative. However, it can be noted that the nature by which KDPS has made compulsory training responsive to officer desires and interest has produced feelings of connectedness to training that can be reflected in some outcomes. Conclusion 3: The training experience was enhanced by strategies for adaptive expertise. In each training module, added content reflecting strategies for adaptive expertise broadened the scope of training activities and provided trainees with opportunities that exceeded prior training experiences. Subtracting the components of adaptability from each module would have omitted multiple scenarios for diverse subjects (August and September), coaching and feedback components (all three months), self-evaluation exercises (all three months), pretraining orientation exercises (August and September), the ability to devise one’s own tactical strategies (August), and simulation exercises (July and September). Strategies for adaptive expertise 165 provided Officer Lewis with a wider landscape of exercises, questions, activities, scenarios, opportunities for discussion, learning objectives, and mechanisms for instruction as indicated in lesson plans (Lewis, personal communication, October 6, 2012). These strategies enriched the training in unexpected ways as noted by retrospective interviews with staff and officers. Adding components of adaptive expertise infused lesson plans with discernible behaviors and strategies that act as linkages between one lesson and the next. Further research could determine if these linkages are noted by trainees, but in the lesson plans themselves, adding strategies that are replicated from one training module to the next (such as self-evaluation exercises and pretraining orientation) creates the impression that each lesson or module is of a singular format, plan, or theme. By assessing whether this structure is noted by trainees, the agency could make adaptive expertise training more programmatic through the inclusion of specific modules in successive lesson plans (White et al., 2005). Another potential added benefit of this variety of training in deepening training activities is in adhesion to trainee perceptions, noted almost universally in interviews, that training should approximate the realism of what is encountered on-the-job. Police training literature is robust with assertions that “realistic” training prepares officers for “the real world” of police work (Alpert et al., 2006, Bumbak, 2010). This study finds that, contrary to these assumptions, there is no indication that experience mediates training effectiveness. It is a tentative but reasonable conclusion that the incongruence of training and work realities is the underlying reason for this rejection of prior theory regarding the benefits of experience in fueling expertise. This conclusion that training itself is not indicative of that experience is only a reasonable assumption; a next step would be to assess the extent to which “realistic” police training reflects a police “reality”. Perhaps, as Glenn et al. suggest, training that “mimics the realities of the field” can be 166 different from one field context to the next (Glenn et al., 2003, p. 121). This would entail a further detailing of what constitutes “realistic police training”, and contribute to disaggregating (in Weiss’ words) the constructs that lie behind the well-used buzzword. Elsewhere, the training experience was enhanced in multiple ways beyond the expansion of lesson plans and adherence to perceptions of realism. Interview data suggests that officers were highly engaged in learning and treated the training as a serious opportunity, valued new opportunities to interact with training staff in new roles and sought guidance and advice, viewed this variety of training to be of high quality, noted increased supervisor interaction, encouragement, and mentoring, and viewed the field-based structure of this training as more beneficial than classroom-based instruction (Hanhauser, personal communication, October 8, 2012; Lewis, personal communication, September 8, 2012; Murray, personal communication, October 2, 2012; Williams, personal communication, October 3, 2012). Each of these positive developments can be said to have accentuated the training experience from researcher observation one year prior, when in-service training appeared compulsory and almost mundane in nature with a comparatively low level of trainee enthusiasm. Conclusion 4a: The project represented a significant organizational leap. Weiss’ dimension of evaluation labeled “policy analysis” calls for an examination of the organizational constructs designed to sustain programs of an innovative nature in different contexts (Weiss, 1998, p. 283). The first function of this examination is to look at how the climate of the organization may have contributed to program strength or retention. The climate of the Kalamazoo Training Division is one of innovation and urgency, which is a reflection of its leadership and personnel ethic. Still, the freedom to which the division granted the researcher access, the degree to which trainers demonstrated buy-in of the philosophy and techniques of 167 adaptive expertise, and the passion by which Officer Lewis in particular showed an interest in making the project work despite logistical odds represented significant steps forward for the division. Interviews with stakeholders across the organizational spectrum disclosed that organizational commitment to the training project represented an unusual intensity (Lewis, personal communication, January 4, 2013; McDowell, personal communication, October 6, 2012; Miller, personal communication, September 2, 2012). Data have demonstrated that the training division evolved as a result of this process. Conclusion 4b: Organizational commitment to sustaining training improvements long-term training is unknown. The ability of the department to commit itself to incorporation of adaptability as a consistent training theme over the long term is questionable. The totality of challenges faced when implementing one external training program in an unstable bureaucratic environment forces questions of organizational commitment. As stated by Durlak and DuPre (2008), implementation by adaptation does not necessarily imply failure, a common fallacy in evaluation. But there does not appear to be, as Durlak and DuPre suggest, any balance struck between adaptation and program fidelity which would have indicated organizational commitment to allowing the training ideas explored in this project to guide future training in any theoretical way. As an example, it was surprising to find that few training staff other than Officer Lewis and Captain Miller knew anything of the goals and aims of the training project, even after it was completed, and despite researcher availability and multiple opportunities to communicate program strengths and outcomes. Quite simply, personnel in the training division appeared in a constant state of stress to the degree that interest in the training project was limited to those who were working directly with the researcher. This indicates an organizational unwillingness to 168 communicate goals, procedures, and outcomes of training to even a small cohort of employees, albeit one which is directly involved with training officers. The Training Division did undergo modest but noticeable transformations as a result of the training project. New training techniques were successfully adopted and replicated, such as the formalized use of pretraining orientation, added activities for self-assessment and reflection, modules probing community diversity across numerous groups, additional time for problemsolving, and deepened discussion and debriefing associated with realistic simulation. But curiously, the philosophical change related to the objectives of the training division that was disseminated to personnel within the division itself, as evidenced by the extent to which buy-in was shown throughout the project, has questionable staying power. Although obstacles to the researcher-practitioner partnership were removed, and new relationships built on a trusting helping process similar to that outlined in Weiss (1998) and Schein (1998), it remains unclear as to whether or not these changes were simply to accommodate this one particular program. The degree to which changes may be sustained in the absence of researcher involvement are unknown given the fluctuating nature of police organizational attachment to training demands and changes (Bumbak, 2010; Glenn et al., 2003). As of the completion of the training project in early 2013, Officer Lewis was transferred to another division and replaced by another officer whose personal knowledge of these training constructs and innovations (as well as his desire to implement them) is unknown. Officer Lewis’s passion for training officers is his defining strength, and as evidenced by the degree to which his involvement made the project a reality, it is a legitimate concern that his exiting the division could have ramifications for the continual use of findings. A follow-up evaluation could disclose that most of the changes made as a result of the project were not sustained because of changes in personnel or organizational 169 priorities. Weiss’ profiling dimension of evaluation asks the researcher to address the complex “combination of actors, services, and conditions” that drives program success (Weiss, 1998, p. 273). In the case of police organizations and their inherently political volatility, those conditions and specifications are always fragile and unpredictable. It will take organizational commitment to internalizing program benefits for this particular project to have staying power. However, Lazzara et al.’s (2010) discussion of training for sensemaking appears at its core to be a good fit for the way the training division is structured, and could be a starting point for furthering continual organizational change resulting in training innovation. Conclusion 5: The training project exposed further training needs. The training evaluation process itself served as conduit for the disclosure of problems related to middlemanagement leadership shortcomings. This unexpected outcome was noted across multiple data points, especially the officer interviews. A supervisor subculture was exposed to the training division, resulting in realizations about the degree to which it prioritizes leadership training for newly-promoted employees. However, the disclosure of this need occurred at a time when the training division had conducted two consecutive researcher-guided needs assessments, resulting in an accentuated awareness of how the supervisor needs could be met by their efforts. In the interest of brevity, it is important to note that this unexpected finding was discussed at length with training division staff and a response designed which will capitalize on extensive police theory about the role of middle-managers in contributing to occupational identity (Muir, 1977), foster stability through transmission of organizational ethics (Manning, 2008), and managers’ manipulation of discretion in order to obtain specific organizational outcomes (Lipsky, 2010). In reference to the concept of adaptive expertise, the division is expressing interest in MuellerHanson et al.’s (2005) concept of adaptive leadership and its use at the middle-manager level. 170 As an outcome of the training, this unexpected finding fulfills Weiss’ (1998) dictum to examine the effect of such unexpected findings on organizational health, and an impression is made that the training division dealt with this particular finding with a similarly unexpected urgency. It is possible that the exposure of future training needs could be related specifically to the intensity of high-impact learning experiences discussed by White et al. (2005) in the context of Army Special Forces training. Although the training discussed by White et al. is a classroom module, its objectives are aligned with the strategies for adaptive expertise as explained in taxonomies used to draft modules in this study. The Army Special Forces training focuses on the deconstruction of mental models by which individuals have learned appropriate interactions with others, what the training calls “switching mindsets” (White et al., 2005, p. 10). The training also details the process by which oneself is seen by others (including peers and citizen/subjects) in work encounters and tactical environments. Both of these processes underlie the multidirectional training process in adaptability modules used in the Kalamazoo project. In many rigid contexts such as military and police settings, the deconstruction of engrained mental processes and relearning of new ones to overhaul decision-making can be a particularly painful and personal process, especially in team and supervisor interactions (White et al., 2005). Therefore, it is possible that the exposure of new training needs brought about by Kalamazoo’s training project emanated from the intensity of high-impact learning on trainees and leadership. Each of these five conclusions about the training effort attempts to place the project in an institutional context, and show ways in which the training benefited KDPS’ training vision. . The legacy of the training project in inculcating a taxonomy for adaptive expertise in officers should also be viewed as an organizational effort, albeit one which produced mixed results in convincing the agency and its leadership that long-term commitment to training changes of this 171 nature were important or feasible. A future study of the department to see if such efforts were incorporated into other training processes would prove valuable in determining what features of adaptive expertise training appealed to the organization over time, and why. Project Design Strengths There are multiple strengths to this project’s design. Campbell (1975) discussed the advantage of case studies in examining and expanding theory in context; the study expands multiple related theories of in-service training behavior and organizational development. Creswell (1994) stated that, although a time-series analysis would be most ideal in case study analytical approaches, finite-time analyses can enrich context through the search for patterns of reaction and explanations of both individual and group behavior, allowing for deeper exploration of themes which can expand theory. By using multiple sources of data collection, including interviews, content analyses of curricula for thematic content and activities, as well as observations of these training strategies in action, the project fulfills a large gap in police training literature that has persisted for the last decade (Bradford & Pynes, 2000). The study couples an analysis of what a department says it is training with an analysis of what is actually taking place in the training itself at the point of delivery. The case study approach is augmented by quantitative analysis of three distinct training experiences to fuse theories from industrial and organizational psychology, educational psychology, and military science to explain an unaddressed and urgent criminal justice problem. Not only is the theory of adaptive performance training examined in context, but the confirmatory performance evaluation criteria suggested by Mangos and Johnston (2008) is also used to determine the efficacy of training on specific criteria derived from agency training objectives. The validity and reliability of such performance measurements is tested as part of 172 this process, having not been utilized in field study contexts before. Mangos and Johnston (2008) provide a blueprint for measurement criteria with five specific dimensions of evaluation criteria that are assessed for applicability to adaptive performance contexts. Kalamazoo is a unique and opportunistic environment to put confirmatory performance evaluation into context because such criteria makes prior informal and ad-hoc data collection and evaluation schemes more systemic, linking instructional objectives to specific measurements suggested by literature. The training modules themselves are also a strength of the project’s design. Using the multiple resources of the training division and its personnel, input from line-level officers and middle managers through the informal analysis conducted by the department, and the taxonomies of training for adaptive expertise provided by the researcher, the design of each month’s training task constitute a strong effort to integrate adaptive expertise principles into police in-service training, a field that has largely suffered from lack of study. As stated by Mangos & Johnston (2008), the individualized nature of adaptive expertise training targets learner needs more efficiently through aptitude-treatment interactions, of which this variety of training is a part. The level of instruction present in adaptive performance training targets adult learning goals associated with andragogy through high-impact learning experiences and modeling, also theorized to be effective in a more individualized manner. The training modules that are performed as treatment in the pretest-posttest phase provide trainees with a “consistently high level of challenge without compromising motivation or overwhelming the trainee” (Mangos & Johnston, 2008, p. 314). Another strength of the project’s design is the existence of three months of data collection in the pretest-posttest portion. Although triangulation of findings would be more appropriate if the dependent and independent variables were consistent across the three months of training 173 delivery, the ability to implement different training strategies associated with adaptive expertise dissects the more synergistic and theoretical nature of the designations “excellence” and “expertise” that have defied description because of the lack of testable hypotheses in context (Sonnentag, 2000). By allowing for three consecutive training modules targeting adaptive expertise principles derived from theory as well as department training goals, cross-case themes advocated in case study research can be detected in a quantitative fashion (Creswell, 2007). The ability to extract different components from the “black box” of adaptive expertise and test them individually and consecutively in one department inform a greater discussion of data-driven strategies that practitioners may use (Heggestad & Kanfer, 2005). The end result of the project is the creation of a tentative model which may be further explored in other departmental contexts. There is a risk of content-strategy interaction when examining adaptive expertise due to the close association of declarative knowledge (referred to as “knowing what” is being taught in training) and procedural knowledge (referred to as “knowing how” to perform on a specific task) in many of the tasks designed for the project (Alexander & Judy, 1988, p. 376). This association is often prevalent in training contexts when strategies employed during task execution are associated with self-evaluation of those same tasks. The project acknowledges this risk and uses internal controls in order to account for the potential that such a phenomenon may occur, and the controls are suggested by literature. According to Alexander and Judy (1988), there is a lack of consensus among researchers on the specificity of the content-strategy relationship as displayed in many training environments due to the wide variation in metacognitive strategies across training participants. Chi viewed content and strategy knowledge as “separable and distinct”, while at the same time acknowledging that interdependencies could exist which would distort findings regarding trainees’ perceptions of their own abilities (Chi, 1981, p. 457). One potential 174 example would be a police officer trainee whose knowledge about firearms was so profound that he was able to learn trained strategies to adapt to new environments more rapidly than others, which would account for that knowledge differently by attributing performance to training. Alexander and Judy (1988) offer a similar hypothesis in their meta-analysis of content and strategy literature. The project acknowledges the need to keep domain knowledge and strategic knowledge as separate as possible, and uses Alexander and Judy’s suggestion to use more executive adaptive expertise strategies as opposed to ones derived from domain-specific knowledge. An example would be that firearms-related adaptive expertise training strategies do not mirror firearms-related knowledge, such as utilizing a new firearms technique or tactical skill in order to behave more adaptively. The training modules are designed to be as strategic as possible in order to account for this risk. A final strength to the project’s design is the ability to conceptualize adaptive expertise training as a kaleidoscopic entity of multiple distinct dimensions that can be delivered independently. Because results of the study demonstrated that certain features are stronger than others in bringing about characteristics of adaptive expertise that the department deems important, then those features can be isolated for instruction according to the overall model (a responsibility of the agency at a later step, since it requires further development of training criteria from results). This reiterates that adaptive expertise does not constitute an “end state”, but rather acts as a synergistic combination of multiple strategies that unfold over time. By not treating adaptive expertise training as a dosage, and instead observing it as a gradual, building process over time, the project views the training phenomenon as an extension of the theory, and illuminates design elements which may work more than others in attaining a desired effect. Rousseau (2006) maligned evidence-based practice guides as failing to bridge a disconnect 175 between theory and practice with minimal attention paid to the structure of how strategies are derived from theory, asserting that the translation of principles into practice often does not account for cognitive load and working memory limitations. This project pays close attention to those distinctions and limitations by dissecting adaptive expertise into distinct training tasks, fueling a discussion about evidence-based in-service training models by expanding what we know about how such training operates in a field setting. Project Design Limitations The project employed a mixed-methodology to address multiple research questions in a field setting, but still contained flaws that were not able to be fully resolved despite modifications to the study design. Some of these constituted potential validity threats that persisted despite the existence of internal checks for validity that were built into the study. Both the qualitative and quantitative portions of the study were subject to threats to validity; each is outlined below, as well as how the study acknowledged these threats. Survey validity and reliability issues upon implementation. Because of logistical considerations, the pretest and posttest surveys were not field-tested prior to distribution to officers. During the pretest portion, Lieutenant McDowell raised concerns that officers may misread the question with respect to stress, because the question was unclear with regard to the specific venue stressors may occur on the job (i.e., tactical or situational stress, stress related to lack of sleep or work cycles, or external stress officers may bring from home or other stimuli) (McDowell, personal communication, September 30, 2012). It is likely that this question may be affected by other job tasks as well: KDPS is a public safety agency with individuals from diverse career pathways training simultaneously. Tactical officers may experience different jobrelated stress than others, and officers who work primarily in fire-related contexts (what KDPS 176 refers to colloquially as being “on the rig”) may not experience the bulk of police-related stressors that influence others. The issue raised by Lieutenant McDowell begs the question of whether the wide variation in police experiences within one department, as would normally occur, would make measures of stress differential and specific to the degree that there can be no one single training strategy to impact individuals’ abilities to navigate stressful environments to bring about the core prerequisite involved in adaptive expertise. Unfortunately, the taxonomies of adaptive behaviors provided by Pulakos et al. (2000) and Lazzara et al. (2010) do not indicate how “on-the-job stress” may be differently identified; the research hypothesis related to this dimension is non-specific with regard to the variety of on-the-job stress that may be targeted by realistic simulation-based training. The inability of the project to expand definitions of stress so that compartmentalized training targeting multiple stressors could be tested is a theoretical shortcoming of the study of adaptive expertise in general, as no definition of the interaction between stress-inoculation and the different varieties of on-the-job stress is identified in literature. What remains is a valid question as to whether or not officers’ potential misunderstandings of the survey question constituted a construct validity threat, namely inadequate explication of constructs identified by Shadish et al. (2002). The point raised by Lieutenant McDowell about the survey question occurred prior to ride-along interviews conducted with officers and allowed for researcher restructuring of interview questions to incorporate this concern. Internal validity threats. Training evaluations are used to assess treatment effects and evaluate instructional efficacy. However, both ethical and practical considerations prohibited the study from utilizing a control group, leaving the project to rely heavily on the principle that, as much as possible, both the qualitative and quantitative components may reinforce each other in 177 determining causal explanation, as opposed to competing with each other. As is often the case with one-group pretest and posttest designs, the lack of a control group accentuates the need to be aware of threats to internal validity. Isolating participants from environmental factors is impossible in a field-based testing scenario, and in training evaluation research, internal validity threats consisting of variables external to the training program may materialize (Goldstein & Ford, 2002). One threat is selection, where events may take place in non-training environments to affect observed behaviors or self-reported changes. For instance, if following training designed to improve confidence in dealing with diverse populations, some trainees spend an inordinate amount of time dealing with such populations, it is likely that some trainees’ selfreport of confidence may be heightened by experience, not training. The second posttest used questions intending to elicit information about transfer, including working environments the trainee has encountered which may reinforce training, or be confused with its effect. Spontaneous events take place in patrol situations following training that may mimic or even obstruct possible training effects: these were documented in order to account for this threat. Additionally, history threats may occur, such as on-the-job experiences or high-profile tactical encounters resulting in enhanced abilities to develop skills that are a part of training. The posttest evaluation process incorporates questions regarding these potential historical incidents to account for this threat. Selection of the Aurora shooting as a topic driving training also represented a history validity threat. The Aurora shooting had the potential to affect officers’ dispositions to take the training more seriously after the event and associated media coverage. Although there is researcher confidence that such events are usually minimized in police training contexts as opportunities for training, the amplification of this event by the training division itself by providing for its use as the background for a training module in August could have 178 accentuated a sense of urgency. The effect of this event on the survey respondents’ attitudes toward the training is largely unknown, but there is a clue as to the potential seriousness of this threat. Results of a posttest question item asking if the officers liked the August training created specifically to develop action plans for similar events indicated low satisfaction for that month’s training as compared to July and September. If the officers felt a sense of urgency regarding the Aurora shooting, it was not reflected in any comparative enthusiasm for the training targeting mobile active shooter skills, as it was the least popular training module (discussed subsequently in Table 12). Testing is another potential threat to internal validity which may occur as a result of the close proximity of pretest and posttest in the study. At KDPS, there is organizational familiarity with the training division’s often rigorous and perceived innovative approach to internal inservice training: the officers admire and respect the training division, as disclosed in the 2011 needs assessment. It is possible that their responses to posttest evaluations of training effect may be exaggerated by non-instructional factors such as the sensitization to anticipated outcomes derived from the pretest itself. The project’s design utilizes the observation component in order to account for possible testing disruptions by evaluating trainee performance on tasks as a supplement to self-reported findings on the posttest. Additionally, the internal referencing strategy acts as a gauge of the extent to which non-instructional objectives are taught in training processes, as the results of the pretest-posttest demonstrate. Observation is used to mitigate threats derived from treatment implementation variance, which may occur when training is inconsistent across the time period from one group to the next. Shadish et al. (2002) describe this as a threat to statistical conclusion validity because the statistical power of tests can be reduced through inconsistent treatment, but the issue is raised 179 here as an internal validity threat because of its close association with the project’s reliance on accounting for history and maturation as the training unfolds. In KDPS’ training division, as in most police training environments, incidents which occur while training is taking place may drive minute changes in training delivery. For instance, if an officer encountered a specific cultural group while the department was putting on an in-service training focusing on diversity, it is likely that the incident may stimulate the need for greater awareness, and become a driver of stronger attention to that particular training on the part of trainees. These situations often have a detrimental effect on the internal validity of findings in an experimental situation, as those who were subject to training before such events would not be receiving similar training as those who were trained following such high-profile incidents. Again, the presence of observation places the training in context as well as guards against inconsistencies. Construct validity concerns. As Trochim (1985) explains, a critical question must be asked when explaining the outcome patterns relevant to causal hypothesis testing: what are theoretical causal agents outside the treatment, and how do they interact in the model to disrupt hypothesized findings? The project acknowledges these potential construct threats, and monitors relationships that emerge as the study progresses. The primary construct validity threats are reactivity to experimental situation, treatment diffusion, and novelty. The first of these may occur due to the fact that KDPS officers could potentially have reacted differently to training that appeared more intense or concentrated than in-service training traditionally received. Care was taken to mitigate this threat through two methods: establishing familiarity with expected training measures through pilot programs (i.e., training division staff send advance lesson plans periodically to trainees), and reducing trainer focus on treatment constructs which will be measured, such as reiterating expectations of learning objectives throughout the training. 180 The second threat, treatment diffusion, may occur in the absence of treatment and control groups in the study. Because of the staggered training schedule, some trained individuals may fraternize with those who have not yet received training. Upon returning to work from training, individuals often begin to apply learned activities on-the-job, potentially affecting the training environment in other shifts and workgroups prior to those groups receiving training. This diffusion threat is common in training evaluation research, and measurement of the treatment implementation uncovered through observation assist in mitigating this threat. As discussed in the implementation section, there appeared to be no outstanding discussion or application of diffused concepts observed during the training modules among officers. The third effect, novelty, may impact findings in multiple ways. There is some concern with the question of motivation to train within the department and the fact that it can vary between day and night shifts, with overall motivation to train being a common problem in training studies. This is a function of scheduling within many police departments, with younger officers often being placed on night shift due to seniority, and the potential that older officers on day shifts may be more plateaued than those on night shift. The threat evident here is outlined in Dweck (2000), and potentially infects the task of practice the most, since the perception of a different focus on practice could react differently with individuals who practice training on their own, as opposed to those who perceive it to be. Often, making the most of practice requires a motivated and persistent trainee to perform at high levels for some time. The two types of motivation, intrinsic and extrinsic motivation, react differently to intensified task performance (Dweck, 2000); by building in controls to equalize treatment conditions, such as standardizing practice methods and setting standards for practice performance, motivation was controlled for within the practice component by Officer Lewis. This was reiterated through observation, and 181 recorded and detailed in lesson plans. Also experimenter expectations and novelty effects threaten validity if trainees engaged in practice opportunities see observation as an element that is outside the traditional approach. However, although observation itself may constitute a threat because of novelty and rater expectation, organizational familiarity with different training strategies and with training evaluation is strong. Researcher input in these environments was minimized as much as possible, and rater expectancies were minimized to prevent treatment misidentification. Statistical conclusion validity concerns. The use of instruments which may not reflect measures of training performance that KDPS training division seeks to include in the training tasks constitutes a concern for researchers that was resolved in the crafting of accurate instruments and protocols. Mangos and Johnston (2008) warn that the use of instruments which do not measure training behaviors is consistently a risk in training evaluations; for this reason, care was taken to make all measurement instruments more adequately reflect training designs in use. The pre and posttest instrument used to measure training impact in the pretest-posttest portion contained measures derived from KDPS training tasks and goals. “Teaching” KDPS’ training division staff how to construct adequate instruments on their own as opposed to using convenient external instruments constituted one outcome of the project; it was made explicit at the planning and advisory panel stage, as it may also pose a threat to construct validity. External validity concerns. Kalamazoo Department of Public Safety is a very unique agency, a consolidated public safety department, of which there are only approximately 150 total in the United States (Wilson & Grammich, 2012). Training at KDPS targets two distinct tasks, police protection and fire suppression, and the department often struggles to provide both on a limited training budget with a small staff. The agency encounters specific challenges, especially 182 in terms of providing quality training, which the vast majority of other police agencies do not. Additionally, the study is a case analysis of one single agency, with potential interference from a number of organizational demographics such as size, region, and a unique organizational culture (Lieberson, 1991). Therefore external validity is affected in multiple ways. First, effects of adaptive expertise training found in such an environment may not translate to other agencies and settings due to the fact that, to some degree, Kalamazoo officers are already more highly skilled than other officers due to their holding dual police and fire roles. Other agencies may identify them as being able to handle the challenges of training for expertise because they already exhibit a higher level of knowledge than other officers elsewhere. Also, what mediates the performance of Kalamazoo officers engaged in training may not exist elsewhere due to the highly individualized and unusual administrative and organizational structure in the department. The entire organization, from scheduling, to hiring, recruitment, training, promotions processes, management and labor relations, to contact with the public are affected by the unique public safety consolidation concept which is unlikely to be replicated elsewhere. Also, external validity is limited even further when one considers size, region, and other selection issues in a telescoping manner: not only is the KDPS model a rarity among police organizations, but it is even rarer with regard to other demographic variables. However, it can be articulated that such a setting may be ideal for studying adaptive expert training in context due to the theoretical difference between what constitutes a “routine expert” and an “adaptive expert”. By virtue of having both fire and police certification, all Kalamazoo officers could be said to constitute “routine experts”, a baseline for comparison which all officers in the department have nearly the same level of expert training. The opportunity for officers in the department to have wildly differential training experiences is less likely than in other departments due to the 183 expectation that KDPS officers are already experts in their field. The baseline is admittedly a very high one – but if officers in the department have consistently higher-level training than others elsewhere, their performance on training strategies specifically targeting adaptive expert performance can be isolated. This serves a dual role: because in-service training is administered throughout the department in the same two-hour block of time format using the same training division officers, a reasonable assumption can be made that training will be uniform and consistent across each shift, mitigating internal validity threats such as novelty effects. In other departments where in-service training is conducted outside the department, such an examination would not be possible due to the widely differential training available from outside experts. The issue of the novelty of the KDPS among other populations of police departments does not negate the potential importance of findings; rather, it requires the project as a case study to articulate exactly how findings pertain to the larger discussion of police training in the current environment. The case study approach, inclusive of the external validity concerns stemming from the use of a select type of police department, still makes knowledgeable statements about the applicability of adaptive expertise training designs in police environments. According to Lieberson (1991), studies with limited cases force interpretations of results based on three questions: the assumptions underlying the study, the reasonableness of the assumptions, and improvements that can be made to modeling with limited-case studies. Considering these questions, the case study includes in-service training structures that would not vary substantially in other police agencies, and results of the study will retain generalizability to a broader discussion of the role of in-service training in creating more adaptable behaviors. Other concerns. A Hausman test for endogeneity between variables in the hypothesized models was be performed to validate model results and determine if a causal relationship existed 184 between the nonequivalent dependent variable of officer safety and all 18 potential dependent variables in the model. This procedure used the concatenated data set utilized in the ordinal regression models in Chapter 5, which mimics balanced panel data because time periods T1 and T2 can be placed into a Hausman model. An endogenous variable is a variable that can be partly or wholly determined by factors embedded within the model itself (Zohoori & Savitz, 1997). Zohoori (1997) uses this statistical technique, derived from the field of econometrics, to determine if overestimations of the effect of specific treatments (in the case of the study plan, training) can confound results. This check addressed the following question: is it possible that officer safety skills are explained by an officer’s ability to recognize problems through knowledge of information-processing or error management? This explanation requires explicitly-detailed relationships between variables that are expected to react in a more linearly causal manner (i.e., the nonequivalent dependent variables did not react to changes in training, which supported hypothesized findings about training effect). The use of checks for endogeneity as explained by Zohoori (1997) and Zohoori and Savitz (1997) were utilized to ensure that causal influences did not mimic confounding relationships. Hausman test results indicated that no training variables were correlated with officer safety, validating regression model results. Finally, this study cannot be a time-series or longitudinal study, reflecting other limitations. The first limitation pertains to a flaw inherent in the theory of adaptive expertise itself, that of basing an assessment of training efficacy on educational criteria that are prerequisites of performance, as opposed to actual performance measurements. Chi illuminates this problem in a form of meta-analysis of expert-level training assessments that discusses the limitations of using “cognitive processes underlying” performance (Chi, 2011, p. 18). Most methodologies used to evaluate the way that experts learn focus on outcome measurements that 185 are supportive of the idea of adaptive expertise, such as Pulakos et al.’s “eight dimensions of adaptive performance” (Pulakos et al., 2000, p. 617). The critical question, one which remains a limitation of this study, is how does one evaluate outcomes such as “[trainees] remaining composed and cool”, “demonstrating enthusiasm for learning”, and “being flexible and openminded” (Pulakos et al., 2000, p. 617)? The use of prerequisites that theoretically lead to these states of behavior (and the casting of adaptive expertise as not being an end state at all) is a function of the newness of this theory and its being relatively untested aside from a few wellfunded studies originating from fields outside criminal justice (White et al., 2005). Therefore in this study, because of logistical, theoretical, and financial shortcomings, prerequisites of performance were measured as opposed to actual performance. It was also an opportunistic shortcoming, because of the lack of evidence of adaptive expertise skills being put to actual use. Even if time for a more longitudinal study were available, the likelihood of actual performance situations (such as high-impact tactical scenarios) taking place is very rare in police settings. Moreover, tracking police cohorts through careers where they may be putting such skills to use, and testing for any training effect, would require an extraordinary organizational commitment. Utilizing prerequisites allowed for a training program that pushed the discovery of adaptive expertise forward, but like many other pilot studies, left many opportunities for further exploration. The second limitation occurs when short-term effects of training are being assessed as opposed to long-term effects that a longitudinal study would disclose. Because only short-term effects are being gauged, the assumption is made that training effects may reveal themselves in the short-term, requiring the study to be cautious of measurement error. To address these concerns, the project draws on multiple studies with the same limitations, such as White et al.’s 186 (2005) evaluation of Army Special Forces adaptive performance training, in adopting the same methodological and data analysis protocols. The potential for such a study to have an impact on training practice should not be overlooked despite the impossibility that actual practice situations may be observed to determine transfer; White et al. (2005) state that skills associated with adaptive performance may take time to develop in actual field settings, and that attitudes, beliefs, constructs, and other prerequisites for performance can be adequately evaluated using a mixed methodology to ensure that descriptive attention be given to variables that may hide in qualitative data. This project engages that challenge, and hopes to assist Kalamazoo and other departments to target these previously unstudied attitudes and beliefs with concrete training goals in high-impact settings. A final limitation is in the lack of competency analysis which would have gauged whether or not trained knowledge and skills were being retained. Although this limitation was not part of the original study design as stated previously, this decision emerged at the implementation stage as a compromise to eliminate and reduce the amount of pretests and posttests trainees would have had to take, granting greater time to KDPS training staff for actual training. A substantial amount of credibility was gained by the researcher at this stage by deciding to cut back on competency tests, mostly by making the training appear less “formal” to trainees and thus potentially securing project buy-in, reducing testing effects as validity threats, and eliminating the possibility of engaging training division administration over a feature that possibly was in violation of collective bargaining agreements at KDPS. The lack of a measurement of whether or not trainees actually learned knowledge and skills was also a product of the fact that, up until the very last minute (as is customary with most police in-service training programs), the specific knowledge and skills were unknown until training itself was designed 187 and lesson plans drafted displaying specific objectives for the training. For instance, although the researcher and Officer Lewis discussed at length what specific dimension of adaptive expertise would be used on each of the months, actual lesson plans to instruct those dimensions (and the specific instructional criteria which could form the basis of competency analysis) were not provided until a week prior to the training. It is critical to point out that, although the benefits of including this feature would have been great for project triangulation, the risk of proceeding with this feature would have necessitated an even greater amount of employee commitment and overtime on the agency’s part to ensure that it would have been implemented. As dismissive as it seems, to pretest and posttest all police officers during in-service training related to the three modules here would have been a sea change for KDPS, and many other police agencies. It illuminates the need to recast police in-service training as a more critical component of long-term employee engagement and career development than it is currently, which is one of the stated intentions of this study. 188 CHAPTER 6 IMPLICATIONS FOR FURTHER RESEARCH Several critical implications of this project can be noted for theory and police organizations. These implications are not exclusive to one distinct area, since in an applied setting the value of a project effort can overlap in many ways. As a starting point to discussing these implications, Weiss (1998, p. 279) suggests that project implications can take the form of a narrative about the entirety of the evaluation effort. In that regard, interpreting findings for KDPS, the field of policing as a whole, theories of training, and future research strategies can be equally informative. These implications involve the testing of theories of adaptive expertise, the shifting of focus on in-service training as a subject of empirical study, the alignment of existing organizational efforts with theory to further and inform future improvements, and the fulfillment of research-practitioner partnerships as an avenue to problem solving in policing. Implications for the Future Study of Adaptive Expertise This project has multiple implications for the study of adaptive expertise. The project incorporates multiple theoretical perspectives about this form of training from diverse strands of literature into one singular evaluative approach. It also provides data which reflect potential latent variables for each adaptive expertise dimension, presenting the opportunity to use more sophisticated modeling methods to examine the interactivity within those dimensions and implications for further theoretical development. In this project, research questions reflect taxonomies of adaptive expertise that had not been tested in a field setting (Lazzara et al., 2010; Pulakos et al., 2000). The taxonomies jointly define both adaptable behavior and expertise in trained performance; many of these components themselves (such as demonstrating interpersonal adaptability) have rarely been subjected to empirical test in a non-clinical field setting (Lazzara 189 et al., 2010). Theories of adaptive expertise are also somewhat newly developed, an indication that field application and empirical analysis can assist with further theoretical development. Knowing how these specific training components may translate to techniques, and assessing their outcomes, can begin to open the “black box” of training. Adaptive expertise training can be of great interest to police agencies struggling with continual training for experienced persons; study findings provide support for this endeavor. Furthermore, project results should provide life to the concepts outlined in both Pulakos et al. (2000) and Lazzara et al (2010), reiterating their use in tandem in training curricula as a pathway to training for adaptable behavior. What follows are three distinct ways in which adaptive expertise as a field of study can benefit from evaluation. Adaptive expertise can be examined as two related phenomena. As stated earlier, project data supports the contention that adaptability and expert-level performance are possibly two interwoven phenomena under the singular definition of “adaptive expertise”, and can be evaluated in tandem. Both Chi (2010) and Pulakos et al. (2010) have explored one of these distinctions, with Chi focusing on trends in the study of expert performance, and Pulakos et al. concentrating on adaptability training. What is lacking is a rigorous examination of these two constructs in a manner which attempts to disentangle potential synergistic effects, or the development of a taxonomical model of adaptive expertise where one or the other “halves” is accentuated differently, potentially leading to different outcomes. A longitudinal study could attempt to expand performance measurements to observable phenomena (possibly even in policing, as with special units such as tactical teams and leadership roles where observation is critically lacking), in order to make statements about the interrelationship between the two over time in reinforcing each other as a predictive model (White et al., 2005). Such a project would 190 also allow for additional exploration Sonnentag’s (2000) and Chi’s (2010) concerns about the differences between routine and adaptive expertise which are not explored in this study. Adaptive expertise training can be articulated as related learning proficiencies. Similarly, this study implies that learning proficiencies across multiple dimensions, both within one learning module (such as knowledge, understanding, and confidence of stress-reduction strategies) and across modules (such as knowledge versus understanding versus confidence of all performance measures) may be related in training contexts. Using Mangos and Johnston’s (2008) confirmatory model, a factor analysis can be performed and modeled showing interrelationships across these proficiencies to explore their value in training phenomena. For instance, few researchers on the subject have attempted to determine which of the three proficiencies, knowledge development, confidence building, or the development of a global understanding, contributes strongest to the development of adaptive expertise in individuals. Notably, verbiage is somewhat vague with respect to how these proficiencies behave in persons, such as Pulakos et al.’s dimension definitions which spell out task behaviors (“maintaining emotional control”, “entertaining wide-ranging possibilities”) but not accompanying measures (Pulakos et al., 2000, p. 617). Looking closely at how these definitions can be modeled as interrelated proficiencies may lead to a stronger understanding of data relationships such as those in this study. The project provides a sense of urgency to the need for adaptive expertise training. Findings should also provide a reply to a question posed by Bransford: “why should we care about the concept of adaptive expertise?” (Bransford, 2004, p. 1). Study findings validate theories used to construct the above taxonomy by translating the twin issues of adaptability and expert performance for a larger audience. Adaptive expertise training theoretically allows 191 individual trainees the ability to innovate on a continual basis “on the fly”, or in rapidly-evolving contexts as needed, a skill that police agencies have long considered essential in producing exemplary officers (Bradford & Pynes, 1999). What adaptive expertise actually entails appears wedded to the concept of how police officers are supposed to perform on-the-job. If Kalamazoo DPS determines in the wake of findings that this training technique is a valid way to attain a level of excellent performance, it is possible that they may use this knowledge to transform their training division into a workshop whereby other components of the taxonomy can be put to a similar theoretical test. Chi (2011), in a very recent discussion of the theoretical background of expertise, proposed that future studies of the mechanisms of adaptive expertise in process focus on producing evidence that can validate these tenets. This study is a first step in that direction. Implications for Police Training Police in-service training has seldom been the subject of empirical study, but this project could continue to turn attention towards the subject. One immediate improvement is required to sustain any future projects of this nature, and that is the continued improvement in recordkeeping, curriculum writing, and lesson planning with respect to this variety of training. KDPS’ own records, by the researcher’s experience with other agencies, were especially acute, but it was still far from complete. Some lesson plans were inconsistent with respect to linking training tasks to specific objectives, most evident in each of the non-firearms modules (August/active shooter and September/MILO, Appendices 7 & 8). Moreover, there was no standardized outline or template followed to develop lesson plans; adding this feature would not only allow for easier adjustments to lessons and standardization across the training division, but might streamline the lesson-planning process as Officer Lewis was constantly pressed for time. No literature pertaining to adaptive expertise suggests that training lesson plans should display format 192 consistencies, but it is possible that doing so might make the taxonomies associated with the training more tractable if objectives, tasks, outcome measurements, and debrief were aligned. Police departments are often insular with regard to the training they receive and offer at advanced stages of officers’ careers; many of the reasons for this lack of transparency are tactical in nature. But what should not be hidden, and what police agencies should strive to make aware to themselves, is the degree to which officers are gaining something from training. A taxonomy of police in-service training would assist in this understanding. Anecdotally, police training at the in-service level is often compulsory, and is seen as unexciting. Data in this study showed variation in interest levels and enthusiasm, attention to detail in some products of training which would indicate disinterest, and statements by certain segments of the personnel profile about “generation gaps” in training interest. Future in-service training evaluation needs to develop these complex structures further to make conclusive statements about the police in-service training apparatus. So far in the 21st century, training in organizations has been framed as a response to changes brought about by globalization, fluctuating trends in the world of work, fiscal upheaval, the rapid development of new competencies and the discarding of old ones, and the task of keeping pace with technological changes and communication techniques that far outpace the human mind’s ability to internalize, process, and utilize needed skills and knowledge (Haccoun & Saks, 1998). In police work, these changes have been magnified, with escalating new demands placed on police personnel and fluctuations in personnel profiles confounding police organizations (Wilson et al., 2010b). If training in organizations is, as Haccoun and Saks (1998, p. 33) attest, a “social response to a social problem”, those social problems have been multiplying so rapidly that to expect training to meet such expectations is potentially unrealistic. 193 However, not only is training often the only line of defense against organizational unpreparedness, it is also seen as currently incapable of meeting such demands in the policing profession (Birzer, 2003; Bradford & Pynes, 2000; Hoover, 2002; Rojek et al., 2007). The reasons for this gap are several, and the study plan cannot address them all. But the study does potentially impact the police training landscape in two important ways. Formalizing the study of in-service training. In-service training in policing is often informal and frequently chaotic, as evidenced by this study; it has also received little to no empirical attention. Its nature raises the bar for researcher and practitioner collaboration in studying its effects. In this case, formalizing the informal processes involved in KDPS’ inservice training program was the greatest challenge faced by the project as a whole. From the 2011 needs assessment forward, the training division empowered the researcher to assist in finding ways in which the informal processes of observation, data collection, and analysis could be improved. However, as noted from discussions of personnel shortcomings, the nature of training division activities, the state of resource availability, and the inability of the division to make meetings and communications less impressionistic are sincere obstacles. Moreover, the extent to which other police contexts share this loosely-coupled definition of in-service training is unknown and inspires further comparative research. It is impressive, however, that the informality exhibited by the training division at KDPS does not appear to constitute an impediment to training efficacy. Systems theory states that sensemaking in organizations can provide a suitable functional framework for certain highlyspecialized tasks, and KDPS has demonstrated that this framework suits its police training needs (Weick, 1995). This is not to suggest that some formalization of the informal processes observed would not streamline future efforts, but there cannot be a blanket assumption that such 194 organizations conform to one organizational scheme. In a more explicit way, Glenn et al. (2003) provide evidence-based suggestions for police training that, at their essence, may be inconsistent with the manner by which the KDPS training division conducts business. It does not necessarily constitute a problem if the informal processes remain informal in specific instances such as those exhibited by this training effort. However, there were important ways in which informal structures impeded data collection. For instance, the volatile nature of in-service training scheduling, in place to compensate for the needed flexibility in shift scheduling in the department, was a greater problem in tracking officer successes than the use of table top sessions allowing for training to be locally-controlled and ideas free-flowing. In sum, flexibility can be used selectively, as formalization has its place in stabilizing training so that it may be evaluated using consistent methodologies. One step to “selling” the use of more formal approaches would be to point out that they are of potential benefit. Two examples of this need are the immediate retention of pretraining orientation strategies with an eye to making them more manageable for officer benefit, and the need to formalize feedback strategies for officers so that they represent a more sincere effort to allow for self-reflection and personal growth as a two-way conversation between trainer and officer. The casual and personal nature of the training division could still be retained, but the potential benefit to the officers and the organization can be maximized. Studying in-service training can lead to the development of organizational learning. Another potential theoretical advancement entails the exploration of training for expert performance to advance organizational learning, thus contributing to the development of learning organizations (Geller, 1997). Multiple components of the study demonstrated for Kalamazoo DPS ways to enhance their organizational knowledge base: the collection and analysis of training 195 data, systematically incorporating trainee self-report and self-reflection, and the establishment of narratives of training content and process through organizational reflection are but a few of these. The contemporary environment in which Kalamazoo DPS seeks to implement innovative training changes will also change, acting as a macro-level model of the same “adaptive” behaviors the agency seeks to inculcate in its officers. These components include furthering community and citizen involvement in training processes, advancing training policy based on outcomes, and the establishment of a unique training culture. Each of these steps is theoretically linked to the furthering of organizational learning in policing (Crank & Giacomazzi, 2009). Such activity may open vistas of change for KDPS, invigorating its personnel knowledge base. Implications for Aligning Theories of Police Training With Police Practice Theories of police training are in their relative infancy, which is quite possibly why previous research into the field appropriates theory from military science and psychology (Glenn et al., 2005; White et al., 2005). What does exist is a burgeoning form of field analysis derived from practice, a true grounded theory arising from the study of training outcomes. Next steps which could enhance our understanding would include analyses of ways in which police organizations are already doing what theory purports to be advantageous for specific ends. This is somewhat of the background of the current project, as it was initiated to demonstrate that KDPS’ Training Division could assess its strengths in light of current research. One consistent response from training officers during early stakeholder meetings was that the training division was already training for adaptability and expert-level performance, albeit in abbreviated form and without theoretical guidance. Although KDPS may have little theoretical experience with adaptive expertise as a concept, they train for adaptability in officers and endeavor to create experts in certain fields. Not only did this make the project more palatable to the division, but it 196 reiterated how critical the components of adaptive expertise were for the transformation of trainees. A question remains: how valuable is theory to training practitioners if they feel that they are already behaving in a manner that will allow for best results? Developing training modules reflective of theoretical constructs. One way of exploring this implication is to assist practitioners in developing practical ideas from theory by providing them with constructs in the form of outcome measurements. As one examines the taxonomies of adaptive expertise, it is possible to see how each can be designed as a training outcome; what is lacking is the individual agency’s personality, or as the Training Passport (Appendix 9) indicates, an agency-specific ethic. It was invigorating to see Officer Lewis develop new training techniques using psychological research that was unfamiliar to him. Almost all of the ideas developed represented breakthrough moments for the training division: an electronic training bulletin and newsletter, circulated summaries of meetings, data-gathering at all levels, and the creation of baseline measures and training scores were brainstormed. These strategies will be of future benefit as well, such as the completion of tactical pre-plans from the August training that will be retained for officer review and use. These theoretical approaches can also be methodological: by assisting the department in realizing ways in which they may evaluate training in a manner consistent with their resources, researcher input can help practitioners gain an overall appreciation for theoretical contributions to the field. Demonstrating methods of increasing local control over police training. Quantifying organizational control over in-service training has been seen as an issue of great urgency in the contemporary police environment, especially with regard to the benefits of performing agencyspecific police training “in-house”, which can potentially lead to a new model of teaching decision-making to police personnel (Fridell, 2012). In the contemporary police fiscal 197 environment, a great risk is seen when agencies continually outsource in-service training (as has been traditional in the past), when little accountability for such training’s efficacy and negligible organizational involvement in the training process have come to characterize what one police chief recently lamented as “vendor-driven policing” (Dolan, 2012). This phenomenon appears in other ways such as the adoption of tools and techniques that may be incompatible with agency need or mission. But because of traditional agency inability to provide advanced training expertise to its own officers, “vendor-driven policing” has described police in-service training for decades as an entrepreneurial exercise of DiMaggio and Powell’s mimetic isomorphism (DiMaggio & Powell, 1983; Dolan, 2012). Anecdotally, the majority of police departments haven’t bothered to train their own officers for expert performance because in the past such training could be obtained externally at nominal cost, and the issue of what was actually being received was seldom examined from a cost-benefit standpoint. That view has begun to change, and the “vendor-driven policing” environment of training has concerned chiefs as a potential issue of not only cost but quality (Dolan, 2012; PERF, 2010). The study illuminates ways in which departments can begin to transform their own training environments to a more agencyspecific and potentially cost-effective model, and can potentially ignite interest in examining this methods of diagnosing and dismantling attachments to external training apparatuses which agencies view as unsustainable. Demonstrating benefits of increasing local control over in-service training. Aside from providing a blueprint to diagnose an organization’s in-service training regimen, this study could potentially inaugurate a new enthusiasm for the potential benefits of internally-conducted in-service training. One of the unintended consequences of the outsourcing of police in-service training has been the proliferation of a training culture that is indistinct; this has potentially 198 stunted organizational growth as well as led to isomorphic training content (Geller, 1997). The central issue here is the lack of comprehensive in-service training data, a gap that encompasses the lack of in-service training inventories and assessments (Monroe, 2012), as well as the absence of comprehensive data resulting from surveys detailing the nature of police in-service training across the nation. Such research endeavors would add to the two-way dialogue proposed by Laub (2011), as well as invigorate the state of police departments’ knowledge about their own training goals, methods, internal resources, and potential link to overall organizational growth and change. The data indicated in this study can initiate this greater conversation by demonstrating the results of a case study which has enhanced the understanding of one particular agency’s knowledge of their training profile. Implications for Research-Practitioner Partnerships As stated previously, few studies have examined police in-service training because little opportunity has existed for researcher-practitioner partnerships which allow for the variety of action research the project required. Obtaining an impression of the state of in-service training necessitates observation, since the diffuse nature of police organizations indicates that in-service training programs operate largely out of the control of the sort of structure endemic to police academies, overwhelmingly the subject of police training empirical study (MCOLES, 2006). In order to perform such observation, researchers must work one-on-one with agencies in order to disentangle the various issues that comprise the police in-service training apparatus. The study engaged this challenge and endeavored to add to a small but growing dialog about police training in post-academy settings. In doing so, the study displayed the rewards of research-practitioner partnerships which have been a concern of many government funding bodies and criminal justice educational programs nationwide. 199 Tips for researchers studying police in-service training. At the beginning of this document, it was implied that one of the reasons why empirical study of in-service training in police work was lacking was that such projects are especially difficult and require too much time and commitment from the researcher and research institutions. This is true in comparison to the relatively compartmentalized training structure in academy settings, where curricula, tasks, venues, and participation are easily observed and accurately recorded. By comparison, police inservice training is an often difficult environment that defies many of the essential cultural and methodological requirements for empirical study. But that distinction does not have to be a hindrance. Past explorations of police behavior in more volatile environments, such as tactical scenarios, violent acts, use of force incidents, and organizational studies, relied on heightened and exceptional levels of researcher commitment and passion for study. It is imperative that this passion be shown in multiple areas: a need for diplomatic flexibility and understanding when dealing with police officers, an understanding of the police work schedule when designing projects, an innate desire to conduct translational criminology with a field institution, and a disposition that the project represents a business partnership are just a few of these manifestations. It is also essential that, should this variety of project and avenue of police study continue, that more attention needs to be paid to developing these skills in graduate students seeking careers in police research. It is imperative that academic institutions continue to develop skills needed for creating partnerships that exceed what are commonly taught in methods courses if study of police training of this sort is to continue. Research-practitioner partnerships can be reiterated as an empirical strategy. The empirical and fiscal benefits for both researchers and practitioners appear to have intersected at a critical time in the trajectory of criminal justice scholarship. From both an organizational and 200 empirical point of view, partially due to the critical moment presented by the 2008 recession and its aftermath (Martin et al., 2012), such partnerships have proven beneficial in providing research opportunities in exchange for consultation. The 2008 recession appears to have been a “tipping point” for KDPS by affecting scheduling, personnel numbers, and the ability to offer comprehensive and quality in-service training to officers. This may have been one of the critical reasons for practitioner enthusiasm for the training partnership itself. Because of researcher outreach, this project represents a modest but potentially important application of learning principles long seen as necessary in a realistic field context, a step forward in the slow march towards evidence-based policy in police training. The role of researcher in this consultation process is significant for its ability to transform the traditional broker role of the consultant into a research-practitioner partnership where none previously existed (Schein, 1998). This relationship-building effort has been seen as establishing a degree of credibility with the public, as evidenced in discussions with officers. Most importantly, the range of concepts consistent to ones already in use were framed in a more theoretical manner for practitioner use, underscoring the value of such partnerships for the agency itself. In doing so, the ability of the agency to maximize what it defined as better training in an era of resource management and growing needs was maximized while creating a more robust training environment. Clarifying the cliché of “evidence-based practice”. Rousseau (2006) used an interesting metaphor to describe the ambiguity of translating principles into effective practices in management: evidence-based management often constitutes a paradox of simultaneous hope and disappointment. The hope is that such theoretical principles can be effectively translated into practices, but the disappointment often occurs when such practices are never evaluated, let alone checked to ensure that they are indeed based on research findings at all (Rousseau, 2006). What 201 is left is empty belief that simply stating that a practice is “evidence-based” implies effectiveness, when the question of “effective at what?” is never asked. Such crucial constructs as setting and context are lost in the embracing of vague “promising practices” that may potentially be inapplicable to certain organizations which adopt them (Rousseau, 2006). The project acknowledges the difficult endeavor of examining a variety of training that has escaped serious empirical analysis, especially in an environment of fiscal uncertainty, and embraces the challenge of translational criminology as envisioned by many criminal justice professionals (Laub, 2011). As envisioned by the former Director of the National Institute of Justice, translational techniques entail a dual conversation, not simply a dictating of techniques by research to practice, but a response from practice itself that invigorates research. The study addresses Rousseau’s challenge of making the cliché of “evidence-based management” more accountable to performance evaluation, but also seeks to provide the basis for this conversational approach to evidence-based management. 202 APPENDICES 203 Appendix 1. Sample Kalamazoo Department of Public Safety training schedule. Kalamazoo Department of Public Safety Training Division Tactical Firearms Training June 2010 – Training Schedule All training held at Kalamazoo Rod & Gun Club June 8 0900-1100 1100-1300 1300-1500 1600-1800 1800-1930 1930-2130 2130-2330 A Platoon-Days A Platoon-Days A/C Platoon-Days A Powershift & 1 KVET team Break A Platoon-Nights A Platoon-Nights June 9 0900-1100 1100-1300 1300-1500 1600-1800 1800-1930 1930-2130 2130-2330 C Platoon-Days C Platoon-Days CID (8 detectives) C Powershift Break C Platoon-Nights C Platoon-Nights June 10 0900-1100 1100-1300 1300-1500 Chief’s office & Service division ½ CPO ½ CPO June 22 0900-1100 1100-1300 1300-1500 1600-1800 1800-1930 1930-2130 2130-2330 B Platoon-Days B Platoon-Days B/D Platoon-Days B Powershift Break B Platoon-Nights B Platoon-Nights June 23 0900-1100 1100-1300 1300-1500 1600-1800 1800-1930 1930-2130 2130-2330 D Platoon-Days D Platoon-Days CID (8 detectives) D Powershift & 1 KVET team Break D Platoon-Nights D Platoon-Nights June 24 0900-1100 1100-1300 1300-1500 CID (remaining detectives) KVET 1 Team & remaining senior staff Remaining senior staff, CPO, CID & Training division 204 Appendix 2. Sample pretest measurement instrument for Module A. Kalamazoo Department of Public Safety Training Questionnaire Please list your Unique Identifier Number: ___________________________________________ Years employed at KDPS Years of total police service Age Ethnicity Gender Highest level of education attained Under 5 Under 5 21-30 5-10 5-10 31-40 HS Diploma 11-15 11-15 41-50 Some coll. 16-20 16-20 51 & older Associates Over 20 Over 20 Bachelors Grad degree Hours of training practice on-your-own per week Evaluate each line item presented below in the following manner: Consider how you feel on the job in various situations that may arise. Using the given scale of 1-7, with 1 representing “strongly agree” and 7 representing “strongly disagree”, please circle the number that corresponds to your level of agreement with the statement in the left hand column of each item. Statement Strongly agree-------------Strongly disagree I understand how to handle stress. 1 2 3 4 5 6 7 I have knowledge of specific problem-solving strategies. 1 2 3 4 5 6 7 I am confident in my ability to use officer safety strategies. 1 2 3 4 5 6 7 I have knowledge of specific stress-reduction strategies. 1 2 3 4 5 6 7 I am confident in my ability to solve problems. 1 2 3 4 5 6 7 I understand how to use officer safety strategies. 1 2 3 4 5 6 7 I am confident in my ability to reduce work stress. 1 2 3 4 5 6 7 I understand how to recognize problems. 1 2 3 4 5 6 7 I have knowledge of specific officer safety strategies. 1 2 3 4 5 6 7 I understand radio communications protocols during crises. 1 2 3 4 5 6 7 I can adequately assess subject level of resistance in a crisis. 1 2 3 4 5 6 7 I can appropriately articulate reasons for using or not using force. 1 2 3 4 5 6 7 205 Statement Strongly agree-------------Strongly disagree I understand how to navigate complex situations. 1 2 3 4 5 6 7 I have knowledge of specific contingency strategies if things don’t go as planned. 1 2 3 4 5 6 7 I have knowledge of specific community policing strategies. 1 2 3 4 5 6 7 I have knowledge of specific strategies to navigate complex situations. 1 2 3 4 5 6 7 I am confident in my ability to deal with unforeseen situations and unexpected crises. 1 2 3 4 5 6 7 I understand how to manage mistakes that may occur. 1 2 3 4 5 6 7 I am confident in my ability to deal with complex situations. 1 2 3 4 5 6 7 I understand how to create contingency strategies if things don’t go as planned. I have knowledge of specific strategies to handle mistakes. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I understand how to handle mistakes and errors. 1 2 3 4 5 6 7 I am confident in my ability to use contingency strategies if things don’t go as planned. I am confident in my ability to properly complete memos or afteraction reports. I am confident in my ability to use community policing strategies. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I understand how to deal with unexpected crises, unforeseen situations, and different people. I understand the purpose of memos and after-action reports. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I know the procedures for completing memos and after-action reports. I know specific strategies for dealing with unexpected crises. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I understand community policing strategies. 1 2 3 4 5 6 7 206 Appendix 3. Sample posttest measurement instrument for Module A. Kalamazoo Department of Public Safety Training Questionnaire Please list your Unique Identifier Number: ___________________________________________ Recall the past three months of training tasks: July: Tactical Firearms - Pre-training emails consisting of news articles and objectives sent in advance - Using color-coded targets - Off-hand shooting - Off-hand shooting with simulated injury (use of tourniquet) - Shotgun training in pairs with coaches August: - Video of Trolley Square shooting sent by email - Platoon-level activity to create pre-plans for active shooter scenarios September - Video on 2nd Amendment subjects from PoliceOne sent by email - MILO simulations Evaluate each line item presented below in the following manner: Consider how you feel on the job in various situations that may arise. Using the given scale of 1-7, with 1 representing “strongly agree” and 7 representing “strongly disagree”, please circle the number that corresponds to your level of agreement with the statement in the left hand column of each item. Statement Strongly agree-------------Strongly disagree I understand how to handle stress. 1 2 3 4 5 6 7 I have knowledge of specific problem-solving strategies. 1 2 3 4 5 6 7 I am confident in my ability to use officer safety strategies. 1 2 3 4 5 6 7 I have knowledge of specific stress-reduction strategies. 1 2 3 4 5 6 7 I am confident in my ability to solve problems. 1 2 3 4 5 6 7 I understand how to use officer safety strategies. 1 2 3 4 5 6 7 I am confident in my ability to reduce work stress. 1 2 3 4 5 6 7 I understand how to recognize problems. 1 2 3 4 5 6 7 I have knowledge of specific officer safety strategies. 1 2 3 4 5 6 7 I understand radio communications protocols during crises. 1 2 3 4 5 6 7 I can adequately assess subject level of resistance in a crisis. 1 2 3 4 5 6 7 I can appropriately articulate reasons for using or not using force. 1 2 3 4 5 6 7 207 Statement Strongly agree-------------Strongly disagree I understand how to navigate complex situations. 1 2 3 4 5 6 7 I have knowledge of specific contingency strategies if things don’t go as planned. 1 2 3 4 5 6 7 I have knowledge of specific community policing strategies. 1 2 3 4 5 6 7 I have knowledge of specific strategies to navigate complex situations. 1 2 3 4 5 6 7 I am confident in my ability to deal with unforeseen situations and unexpected crises. 1 2 3 4 5 6 7 I understand how to manage mistakes that may occur. 1 2 3 4 5 6 7 I am confident in my ability to deal with complex situations. 1 2 3 4 5 6 7 I understand how to create contingency strategies if things don’t go as planned. I have knowledge of specific strategies to handle mistakes. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I understand how to handle mistakes and errors. 1 2 3 4 5 6 7 I am confident in my ability to use contingency strategies if things don’t go as planned. I am confident in my ability to properly complete memos or afteraction reports. I am confident in my ability to use community policing strategies. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I understand how to deal with unexpected crises, unforeseen situations, and different people. I understand the purpose of memos and after-action reports. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I know the procedures for completing memos and after-action reports. I know specific strategies for dealing with unexpected crises. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 I understand community policing strategies. 1 2 3 4 5 6 7 Overall, I liked July’s tactical handgun and shotgun training module at KDPS. I read all of the pre-training materials sent by email from KDPS training staff. In-service training is headed in the right direction at KDPS. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 Overall, I liked August’s active shooter pre-plan training. 1 2 3 4 5 6 7 I have been able to apply skills learned and practiced in inservice training in everyday situations I encounter. Overall, I liked this month’s MILO training. 1 2 3 4 5 6 7 1 2 3 4 5 6 7 208 Appendix 4. Observation protocol. Kalamazoo Department of Public Safety Training Division Training/Meeting Observation Protocol Date of training/meeting and length of session: Location and physical setting: Materials used: Number of trainers: Learning objective/meeting objective: Number of trainees/meeting participants total for session: Descriptive notes of session: List of strategies of adaptive expertise used in session: Reflective notes, insights from trainees, and trainer feedback: Source: Creswell, 2007. 209 Appendix 5. Interview protocol. Kalamazoo Department of Public Safety Training Division Interview Protocol 1. Demographic information a. b. c. d. e. f. Years employed at KDPS / Years of total police service Age Ethnicity Gender Current assignment Platoon area 2. What in-service training at KDPS has been valuable to you professionally and personally? What has not been valuable? Why, and what could have improved? 3. What other topics would be beneficial to train? Why do you feel they would be helpful? 4. In what specific ways could people within the department be used as resources for training? Has the department done a good job of using experts at specific tasks to train others? 5. How can in-service training be used to make you a better officer? 6. Are the skills and knowledge you have obtained through training made you more adaptable to unforeseen situations? If so, in what way? 7. Has the training division adequately answered the needs of the department since its last training assessment in 2011? 8. Does the training division train for the following skills: - Allow you to recognize and correct your own errors - Plan and forecast events as they unfold situationally - Process feedback - Interpret and process stressful environments - Recognize cues that lead to strategizing - Debrief - Physical, cultural, and interpersonal adaptability 210 Appendix 6. July lesson plans. Tactical Shotgun Outline 2012 KDPS Training Division July 2012 Equipment: 12 gauge Shotgun Rounds Steel Targets Paper Targets 12” Paper plates Range Timer Spray Paint Barricades Eye and ear protection Cruisers with shotgun Racks, if possible Tourniquets New duty ammo for handguns and shotguns Number of Officers per Session: 4-8 Location: Kalamazoo Rod and Gun Club Time: 1.5 hours Overall objectives: Trainees will use adaptability skills to navigate tactical situations with a handgun and shotgun. Multiple tactical risks are simulated with the use of changing targets on command, officer moving while remaining on target, emergency situations, situations requiring communication, and the use of a tourniquet. The ultimate objective is to develop advanced firearms skills using a handgun and shotgun in situations that change and evolve over time, and to be able to adapt trained skills and knowledge to those situations. As part of this adaptability: - Trainees will recognize errors they made in training and how those errors can be corrected in real tactical situations. - Trainees understand complex information that will arise in the unpredictable field environment. 1. Speed Shooting Drill – 8 rounds, 4 colored plates: All shooters on line facing 4 paper plates, each a different color, 21’ away 211 Command to draw, officers will draw and get off the line, take up slack on the trigger, but will not shoot x5 On command, the color of one of the plates, officers will draw, get off the line, and fire one round at the specified color x8 Objectives: - The trainee will learn the mechanics of trigger pull using a handgun in tactical environments. - The trainee will shoot for accuracy on each specified plate using a color-coded system. 2. Malfunction/Reload Drill – 16 rounds, 4 colored plates: All shooters will collect 4 brass casings from the ground and insert one brass casing in the top of the half full magazine, the casings will go in a pocket. On command, the officer will draw, get off the line, and fire two rounds at the specified target – the officer will have to complete a malfunction drill or reload to complete two rounds x8 Objectives: - A firearms malfunction is an unexpected event. The trainee will learn how to navigate these events creatively by applying the skill of malfunction drills and reloads in unexpected situations. - The trainee will shoot for accuracy after completing the malfunction drill. 3. Offhand Speed Shooting Drill – 13 rounds, qualification target: Officers will have the weapon drawn and held in the strong hand. On the command of transition, Officers will move the weapon from strong hand to support hand 5x. Officers will stage to the left or right of a barricade. On the command of fire, the officers will draw, move to cover, and transition the weapon to his support hand, break cover and fire two rounds – 6x Officers will stage to the left or right of a barricade with a tourniquet. On the command of fire, the officers will move to cover, place a tourniquet on their support arm, draw their weapon, break cover and engage one round – 1x Objectives: - The trainee will use support-side shooting as an alternative in unpredictable situations, and learn how to transition. - The trainee will use barricade shooting to adapt to unpredictable situations, and learn how to transition between barricades. - The trainee will learn to shoot tactically with an injury. - The trainee will shoot for accuracy during the above drills. 4. Shotgun Combat Load – 12 rounds, steel torso: 212 Two or three officers, on command, will combat load 4 slug rounds. At 30 yards, the officers will fire 4 rounds on command. Officers will move up to 20 yards. On command, the officers will combat load 4 buckshot. On command, the officers will fire 4 rounds at the steel torso. Officers will move up to 10 yards. On command, the officers will combat load 4 birdshot. On command, officers will fire 4 rounds at the steel torso. Objectives: - The trainee will learn to adapt to different tactical distances by combat loading and shooting for accuracy from different distances. 5. Move and Shoot – 8 rounds, Steel Torso: Two officers, on command, will start from a cruiser with an empty shotgun and move to the first barricade. At the barricade, on command, officers will load and fire two rounds (slug). Once they are clear, officers will communicate and move to the next barricade, repeating the same action until they reach the final barricade (using buck or birdshot as they move closer). Objectives: - The trainee will learn to deal with physically unpredictable situations by using combat shooting skills in barricaded situations. 6. Reload Weapons: Officers will get 37 new .40 duty rounds and 5 new shotgun rounds prior to leaving the training Objectives: - The trainee will maintain a loaded duty shotgun and understand the mechanics of reloading - The trainee will self-evaluate his/her performance by answering questions from the trainer: o What did they feel they did right when it came to unpredictable situations o What errors did they make o How did they correct those errors creatively o What unpredictable things might arise in real-life tactical shotgun situations that might cause them to make errors, and how would they correct them o Any suggestions for other officers they saw? 213 Appendix 7. August lesson plans. Active Shooter Response Preplan Outline 2012 KDPS Training Division August 2012 Goal: Officers, in teams, will create site plans for three (3) locations in their zone where a mass shooting may occur. Objectives: 1. Each Zone, under the direction and assistance of the Sergeant, will prepare a list of 3 possible sites where a mass shooting may occur. The zones will document the following: the location, building type, possible building population during their shift, best approaches, best entrances, resources needed, and any other pertinent information. 2. Each Platoon, under the direction of the Lieutenant, will gather the information from the zones and will either distribute that information to the platoon via a written report, or discuss the gathered information in a platoon meeting. 3. COPS and CID will each select three sites out of the city boundaries where a mass shooting may occur. These divisions will develop the same intelligence for their sites as the zones. Divisional commanders or designee will collect the intelligence and distribute through the division either via a written report or discuss the gathered information in a platoon meeting. 4. Officers will be given the opportunity, preferably as a group to watch the training video, “Trolley Square: The Night Rapid Response Worked”. Command Officers are encouraged to then discuss with their officers the points brought forward from the video 5. Divisional commanders will send all information with the signed training sheets to the training division for review and summation. A summary will be sent out with all the gathered intelligence. Summaries and signed training sheets are due to the Training Division by September 9th. Equipment: Line of Duty Video, “Trolley Square: The Night Rapid Response Worked” Video player Paper Pen/Pencil Number of Officers per Session: Platoon or Division Location: Command Officer’s Choice Expected Time of Training Session: 2 hours 214 Appendix 8. September lesson plans. Subject Control Decision Making Multiple Interactive Learning Objectives (MILO) KDPS Training Division September 2012 Objective: To provide officers decision making scenarios using the MILO Training Device To assist First Line Supervisors in Officer Shooting Incident (OIS) response To develop officer’s overall tactical adaptability Time: Instructors will run two officers plus a command officer (when available) through a training evolution that will last approximately 30 minutes Location: Training will be held at Station 4/5 in the small classroom. Equipment: MILO Training Device with handgun, Taser, and OC Desired Officer Actions: Officers should attempt to communicate with the subjects in the scenario Officers should use their radio (battery removed) to communicate with simulated dispatch (Instructor) Officers should use reasonably objective force when required by the scenarios Articulation: Officers must articulate: • Totality of circumstances • Subject actions / level of resistance • Reason for force / no force used • Proper/effective tactics for each scenario Command Officers must articulate: • Initial scene safety for officers, general public and suspect(s) • The initial procedure as dictated by GO 21 • Resources needed • Establishing the foundation for a successful investigation Officer Evaluation and Data Collection: Officers will debrief and self-evaluate with the Instructor, discussing tactics the officers used. Officers will complete a post-training evaluation. 215 Scenarios: Officers will be lead through three scenarios in which they must use a reasonable level of force. During each scenario, the trainer will function as instructor and dispatcher Trainers will read the dispatch narrative prior to the scenario when applicable Officers are only to do the scenarios listed below (located in the “Favorites” folder) Instructors should branch the scenarios to specified ending to allow all officers to work through the same scenarios. Officers should move and communicate cover during the scenarios Scenario 1: Son with Gun: Dispatch (Instructor Reads): You are attempting a pick up a John Johnson for a FOC warrant of $10,000. Actions: Instructors do not have to branch. The officers will be immediately make contact with Johnson. An officer on screen will attempt to arrest Johnson, when Johnson’s son comes out with a gun in hand, held down. Officers will have to decide the appropriate force before the son raises the gun and fires at officers. Debrief: The trainer will discuss officer actions, reason for force, shot placement, moving to cover provided and post-shooting actions. The command officer will be asked to leave the classroom and discuss the event with a selected captain or lieutenant. Scenario 2: Unstable Wrestler: Dispatch (Instructor Reads): You are responding to 3600 Lark Drive regarding the report of excessive noise and the resident yelling at neighbors. As you arrive, you make contact with the resident. Actions: Instructors do not have to branch. The suspect will continue to get more aggressive and delusional with officers, eventually charging the officers. Officers will have the option of using their hands, OC, or Taser. Debrief: The trainer will discuss officers’ actions, reason for force, and what follow up would be appropriate for the suspect (Borgess Hospital vs. Jail) Scenario 2 (Alternative): EDP in Church: This scenario is for any CID who may have had the Unstable Wrestler scenario in the past Dispatch (Instructor Reads): You are going to the First Church of God to speak with Fred Jones, who is the custodian of the church. He is accused of molesting a child at the church the previous Sunday. Actions: Instructor will choose Assault. The suspect will stand, turn to officers and say that the time is now and how he has been chosen. The suspect continues to approach the screen, and eventually lunges and strikes at officers. Officers have the option to use OC, Taser, Baton, or Firearm. 216 Debrief: The trainer will discuss the officers’ actions, reason for force and what follow-up would be appropriate for the suspect (Borgess Hospital vs. Jail) Scenario 3: Terrorism En Colegio: Dispatch (Instructor Reads): You are responding to El Sol school at the corner of Oak and Vine in response to shots fired at the school. Reports state that at least one male suspect is in the school with a handgun. Actions: Instructors will “branch” to “Muere Rehen”. Officers will be lead into the school and will make contact with the shooter in a classroom with two hostages. At that point officers may engage verbally or with firearm. If the officer uses Spanish to order the suspect to do anything, click on “Comply”. Otherwise, the scene will automatically loop. If the officers fail to do anything, choose “Execute”. At this point the suspect will shoot the female victim and the scenario will end. If the officers shoot the suspect, then a second suspect comes out from behind the pillar to engage the officers. Debrief: The trainer will discuss with the officers the response to an active shooter. The trainer will also review the possibility of facing someone who speaks a different language and some strategies for addressing that situation. Trainer and officers will review use of force decisions and actions 217 Appendix 9. Kalamazoo Training Passport. Figure A.1. Kalamazoo Training Passport. 218 Appendix 10. Sample training activity submitted. 219 220 Appendix 11. Sample training activity submitted. 221 222 Appendix 12. Sample training activity submitted. Figure A.2. Sample training activity submitted 223 REFERENCES 224 REFERENCES Aguinis, H. & Kraiger, K. (2009). Benefits of training and development for individuals and teams, organizations, and society. Annual Review of Psychology, 60, 451-474. Alarid, L.F. (2000). Law enforcement departments as learning organizations: Argyris’ theory as a framework for implementing community-oriented policing. Police Quarterly, 2 (3), 321-337. Alexander, P.A. & Judy, J.E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58 (4), 375-404. Alpert, G.P., Dunham, R.G., and Stroshine, M.S. (2006). Policing: Continuity and change. Long Grove, IL: Waveland Press. Applebaum, S.H. & Gallagher, J. (2000). The competitive advantage of organizational learning. Journal of Workplace Learning: Employee Counselling Today, 12 (2), 40-56. Archbold, C.A., Lytle, D., Weatherall, C., Romero, A., & Baumann, C. (2006). Lawsuits involving the police: A content analysis of newspaper accounts. Policing: An International Journal of Police Strategies & Management, 29 (4), 625-642. Argyris, C. (1982). Reasoning, learning, and action: Individual and organizational. San Francisco, CA: Jossey-Bass Publishers. Aveni, T.J. (2005, July). Critical analysis of contemporary police training. San Antonio, TX: State Bar of Texas, Police Policy Studies Council. Backer, T.E. (2002). Finding the balance: Program fidelity and adaptation in substance abuse prevention. Rockville, MD.: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration. Baldwin, T.T. (1992). Effects of alternative modeling strategies on outcomes of interpersonal skills training. Journal of Applied Psychology, 77 (2), 147-154. Baldwin, T.T., Ford, J.K., & Blume, B.D. (2009). Transfer of training 1988-2008: An updated review and agenda for future research. International Review of Industrial and Organizational Psychology, 24, 41-70. Barnett, S.M. & Ceci, S.J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128 (4), 612-637. 225 Bartunek, J.M., Rousseau, D.M., Rudolph, J.W., & DePalma, J.A. (2006). On the receiving end: Sensemaking, emotion, and assessments of an organizational change initiated by others. The Journal of Applied Behavioral Science, 42 (2), 182-206. Bass, B.M. (1998). Transformational leadership: Industry, military, and educational impact. Mahwah, NJ.: Lawrence Erlbaum Associates. Bayley, D. H. & Bittner, E. (1984). Learning the skills of policing. Law & Contemporary Problems, 47, 35-59. Berger, J.G. (2011). Changing on the job: Developing leaders for a complex world. Palo Alto, CA: Stanford Business Books. Birzer, M.L. (2003). The theory of andragogy applied to police training. Policing: An International Journal of Police Strategies & Management, 26 (1), 29-42. Birzer, M.L. & Tannehill, R. (2001). A more effective training approach for contemporary policing. Police Quarterly, 4 (2), 233-252. Bock, R.D. (1976). Basic issues in the measurement of change. In Advances in Psychological and Educational Measurement, D.N.M. DeGruijter & L.J. Van der Kamp (Eds.), New Yourk: John Wiley & Sons, 75-96. Bradford, D. & Pynes, J.E. (2000). Police academy training: Why hasn’t it kept up with practice? Police Quarterly, 2 (3), 283-301. Brand, D. (1999). The future of law enforcement recruiting: The impact of Generation X. Police Chief, 8, 53-63. Bransford, J. (2004, November). Thoughts on adaptive expertise. Published by author. Retrieved from http://www.scribd.com/doc/47031489/Thoughts-on-Adaptive-Expertise Bransford, J., Copeland, M., Honig, M., Nelson, H.G., Mosborg, S., Gawel, D., Phillips, R. S., & Vye, N. (2010). Adaptive people and adaptive systems: Issue of learning and design. In A. Hargreaves, M. Fullan, D. Hopkins, & A. Leiberman (Eds.), The second international handbook of educational change. Dordrect, The Netherlands: Springer, p. 825-856. Brown, M.M. & Brudney, J.L. (2003). Learning organizations in the public sector? A study of police agencies employing information and technology to advance knowledge. Public Administration Review, 63 (1), pp. 30-43. Brunninge, O. (2009). Using history in organizations: How managers make purposeful reference to history in strategy processes. Journal of Organizational Change and Management, 22 (1), 8-26. 226 Buhrmaster, S. (2009, February 20). Cutting back on training? Re-think that idea. Police One. Retrieved November 1, 2011 from http://www.policeone.com/training/articles/1788416Cutting-back-on-training-Re-think-that-idea/ Bumbak, A. (2010). Dynamic police training. New York: Routledge. Burns, W.R. & Freeman, W.D. (2008, October). Developing an adaptability training strategy and policy for the DOD: Interim report. Alexandria, VA.: Institute for Defense Analyses, 160 pp. Bushe, G.R. (2009). Learning from collective experience: A different view of organizational learning, OD Practitioner, 41 (3), 19-23. Caldart, A.A. & Ricart, J.E. (2004). Corporate strategy revisited: A view from complexity theory. European Management Review, 1, 96-104. Campbell, D.T. (1975). “Degrees of freedom” and the case study. Comparative Political Studies, 8 (2), 178-193. Carson, K.D. & Carson, P.P. (1997). Career entrenchment: A quiet march toward occupational death? Academy of Management Executive, 11 (1), 62-75. Chappell, A.T. (2007). Community policing: Is field training the missing link? Policing: An International Journal of Police Strategies and Management, 30 (3), 498-517. Charness, N. & Tuffiash, M. (2008). The role of expertise research and human factors in capturing, explaining, and producing superior performance. Human Factors, 50, 427-432. Chen, H. & Rossi, P.H. (1984). Evaluating with sense: The theory-driven approach. In R.F. Conner et al. (Eds.), Evaluation Studies: Review Annual, 9, Beverly Hills, CA.: SAGE. Chen, Y. & Borodzicz, E. (2009). Can training exercises facilitate the capability to respond to disasters? In M.G. Culshaw, H.J. Reeves, I. Jefferson, & J.W. Spink (Eds.), Engineering Geology for Tomorrow’s Cities, London: Geological Society, CD ROM. Chi, M.T.H. (1981). Knowledge development and memory performance. In M.P. Friedman, J.P. Das, & N. O’Connor (Eds.), Intelligence and learning, New York: Plenum, pp. 221-229. ______. (2011). Theoretical perspectives, methodological approaches, and trends in the study of expertise. In Y. Li & G. Kaiser (Eds.), Expertise in Mathematics Instruction, New York: Springer, pp. 17-39. Cianciolo, A.T., Matthew, C., Sternberg, R.J., & Wagner, R.K. (2006). Tacit knowledge, practical intelligence, and expertise. In K.A. Ericsson, N. Charness, P.J. Feltovich, & R.R. Hoffman (Eds.), The Cambridge handbook of expertise and expert knowledge, Oxford: Cambridge University Press, pp. 613-632. 227 Cigularov, K., Chen, P., Thurber, B.W., & Stallones, L. (2008). Investigation of the effectiveness of a school-based suicide education program using three methodological approaches. Psychological Services, 5 (3), 262-274. City of Austin (TX) Office of the City Auditor. (2007). Audit report: APD recruiting and training. Office of the Mayor, Austin, TX. City of Toronto, Canada, Auditor General’s Office. (2010, March). Police training, opportunities for improvement, Toronto Police Service. Toronto, Canada: Toronto Police Services Board. Clark, R.C. (2008). Building expertise: Cognitive methods for training and performance improvement. New York: Pfeiffer. Clarke, N. (2001). The impact of in-service training within social services. British Journal of Social Work, 31, 757-774. Cleveland, G. & Saville, G. (2007). Police PBL: Blueprint for the 21st century. Washington, DC: US Department of Justice, Office of Community Oriented Policing Services. Codish, K.D. (1996). Putting a sacred cow out to pasture. Police Chief, 63 (11), 40-44. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York: Academic Press. Cook, T.D. & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis for field settings. Chicago, IL: Rand McNally. COPS. See United States Department of Justice. Coryn, C.L.S. & Hobson, K.A. (2011). Using nonequivalent dependent variables to reduce internal validity threats in quasi-experiments: Rationale, history, and examples from practice. New Directions for Evaluation, 131, 31-39. Crank, J.P. (2003). Institutional theory of police: A review of the state of the art. Policing: An International Journal of Police Strategies and Management, 26 (2), 186-207. Crank, J.P. & Giacomazzi, A. (2009). A sheriff’s office as a learning organization. Police Quarterly, 12 (4), 351-369. Crank, J.P. & Langworthy, R. (1992). An institutional perspective of policing. The Journal of Criminal Law & Criminology, 83 (2), 338-363. Crawford, V.M. (2007). Adaptive expertise as knowledge building in science teachers’ problem solving. European Cognitive Science, May, 7-12. 228 Creswell, J.W. (1994). Research design: Qualitative and quantitative approaches. Thousand Oaks, CA: SAGE Publications. ______. (2007). Qualitative inquiry and research design: Choosing among five approaches. Thousand Oaks, CA: SAGE Publications. DeHolan, P.M. & Phillips, N. (2004). Remembrance of things past? The dynamics of organizational forgetting. Management Science, 50 (11), 1603-1613. Della, B.C. (2004, June). Nontraditional training systems: Realizing the effectiveness of an agency’s most valuable resource. FBI Law Enforcement Bulletin, 73 (6), 1-9. Dermody, J. (2013, Spring). Changing the culture of uncommitted patrol time: A work in progress. Translational Criminology, 4, 8-10. Dietz, A.S. & Mink, O.G. (2005). Police systems and systems thinking: An interpretive approach to understanding complexity. Journal of Police and Criminal Psychology, 20 (1), 1-16. DiMaggio, P.J. & Powell, W.W. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48, 147160. Dimitrov, D.M. & Rumrill, P.D. (2003). Pretest-posttest designs and measurement of change. Work, 20, 159-165. Dipboye, R.L. (1997). Organizational barriers to implementing a rational model of training. In M.A. Quinones & A. Ehrenstein (Eds.), Training for a rapidly changing workplace: Applications of psychological research. Washington, DC: American Psychological Association, pp. 31-60. Doane, S.M., Sohn, Y.W., & Schreiber, B. (1999). The role of processing strategies in the acquisition and transfer of a cognitive skill. Journal of Experimental Psychology, 25, 1390-1410. Dolan, H. (2012, April 26). Remarks to the 2012 Police Executive Research Forum Annual Meeting, Washington, D.C. Durlak, J.A. & DuPre, E.P. (2008). Implementation matters: A review of the research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350. Dweck, C.S. (2000). Self-theories: Their role in motivation, personality, and development. Philadelphia: Taylor & Francis. 229 Echambadi, R., Campbell, B., & Agarwal, R. (2006). Encouraging best practice in quantitative management research: An incomplete list of opportunities. Journal of Management Studies, 43 (8), 1801-1820. Endsley, M.R. (1995). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65-84. ______. (2006). Expertise and situation awareness. In K.A. Ericsson, N. Charness, P.J. Feltovich, & R.R. Hoffman (Eds.), The Cambridge handbook of expertise and expert knowledge, Oxford: Cambridge University Press, pp. 633-651. Ericsson, K.A. & Lehmann, A.C. (1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305. Feldman, J.M. (2004, December). Training for situation awareness: What? How? American Tactical Shooting Association. Retrieved December 30, 2013, from http://teddytactical.com/archive/Feature/2004/12_Feature.htm Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics. Thousand Oaks, CA: SAGE Publications. Ford, J.K. (2007). Building capability throughout a change effort: Leading the transformation of a police agency to community policing. American Journal of Community Psychology, 39, 321-334. Ford, J.K. & Kraiger, K. (1995). The application of cognitive constructs and principles to the instructional systems model of training: Implications for needs assessment, design, and transfer. In C.L. Cooper & I.T. Robertson (Eds.), International Review of Industrial & Organizational Psychology (10), 1-48. Ford, J.K., Kraiger, K., & Merritt, S.M. (2007). An updated review of the multidimensionality of training outcomes: New directions for training evaluation research. In S.W. Kozlowski & E. Salas (Eds.), Learning, Training, and Development in Organizations, Mahwah, NJ: LEA. Ford, J.K., Quinones, M.A., Sego, D.A., & Sorra, J.S. (1992). Factors affecting the opportunity to perform trained tasks on the job. Personnel Psychology, 45, 511-527. Ford, J.K. & Schmidt, A.M. (2000). Emergency response training: Strategies for enhancing realworld performance. Journal of Hazardous Materials, 75, 195-215. Ford, J.K., Smith, E.M., Weissben, D.A., Gully, S.M., & Salas, E. (1998). Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer. Journal of Applied Psychology, 83 (2), 218-233. 230 Frensch, P.A. & Runger, D. (2003). Implicit learning. Current Directions in Psychological Science, 12, 13-18. Fridell, L. (2012, April 26). Use of force and public order. Remarks to the 2012 Police Executive Research Forum Annual Meeting, Washington, D.C. Fyfe, J.J. (1993). Good policing. In B. Forst (Ed.), The socioeconomics of crime and justice, Armonk, NY: M.E. Sharpe Publishers, pp. 269-290. Galloway, S.D. (2010). A case study: The Lafayette Police Department utilization of learning organization culture and its impact on the investigations division Recruit Officer Training module. College of Technology Directed Projects, Purdue University, 18. Games, P.A. (1990). Alternative analyses of repeated-measure designs by ANOVA and MANOVA. In Statistical Methods in Longitudinal Research, Volume 1, A. von Eye (Ed.), 81-121. San Diego, CA: Academic Press. Geller, W.A. (1997). Suppose we were really serious about police departments becoming “learning organizations”? National Institute of Justice Journal, 2-8. Giblin, M.J. (2006). Structural elaboration and institutional isomorphism: The case of crime analysis units. Policing: An international journal of Police Strategies and Management, 29 (4), 643-664. Gleick, J. (1987). Chaos: Making a new science. New York: Viking Books. Glenn, R.W., Panitch, B.R., Barnes-Proby, D., Williams, E., Christian, J., Lewis, M.W., Gerwehr, S., & Brannan, D.W. (2003). Training the 21st century police officer: Redefining police professionalism for the Los Angeles Police Department. Santa Monica, CA: RAND Corporation. Goldstein, I.L. & Ford, J.K. (2002). Training in organizations: Needs assessment, development, and evaluation (4th Ed.). Belmont, CA: Wadsworth. Haccoun, R.R. & Hamtiaux, T. (1994). Optimizing knowledge tests for inferring learning acquisition levels in single-group training evaluation designs: The internal referencing strategy. Personnel Psychology, 47 (3), 593-604. Haccoun, R.R. & Saks, A.M. (1998). Training in the 21st century: Some lessons from the last one. Canadian Psychology, 39 (1-2), 33-51. Hall, D.T. & Chandler, D.E. (2005). Psychological success: When the career is a calling. Journal of Organizational Behavior, 26, 155-176. 231 Hatano, G. & Iganaki, K. (1986). Two courses of expertise. In H. Stevenson, H. Azuma, & H. Hakuta (Eds.), Child development and education in Japan, San Francisco: CA: Freeman, pp. 262-272. Hauck, E.L., Avers, K.B., Banks, J.O., & Blackwell, L.V. (2011, November). Evaluation of a fatigue countermeasures training program for flight attendants. Washington, DC.: Federal Aviation Administration, Civil Aerospace Medical Institute, Final Report Number AM-11/18. Heerwagen, J., Kelly, K., & Kampschroer, K. (2010). The changing nature of organizations, work, and workplace. Journal of the National Institute of Building Sciences. Retrieved October 8, 2011, from http://www.wbdg.org/resources/chngorgwork.php Heimbeck, D., Frese, M., Sonnentag, S., & Keith, N. (2003). Integrating errors in the training process: The function of error management instructions and the role of goal orientation. Personnel Psychology, 56, 333-361. Hellen, M. (2009). Information handling and adaptive expertise. Educational Information Technology, 27, 212-233. Helsdingen, A.S., Van Den Bosch, K., Van Gog, T., & Van Merrienboer, J.J. (2010). The effects of critical thinking instruction on training complex decision making. Human Factors, 52 (4), 537-545. Hicks, W.D. & Klimoski, R.J. (1987). Entry into training programs and its effects on training outcomes: A field experiment. Academy of Management Journal, 30 (3), 542-552. Hoover, J. (2002). The Reno model PTO program: An executive summary of state of the art postacademy police training. Reno, NV: Hoover Group of Reno. Hoover, L. (Ed.) (1995). Quantifying quality in policing. Washington, DC: Police Executive Research Forum. International Association of Chiefs of Police (IACP) (2012). Training key index. Alexandria, VA.: International Association of Chiefs of Police. Retrieved February 12, 2012, from http://www.theiacp.org/LinkClick.aspx?fileticket=568%2b005JtQU%3d&tabid=452 James, L.R., Muliak, S.A., & Brett, J.M. (1982). Causal analysis: Assumptions, models, and data. Thousand Oaks, CA: SAGE Publications. Jensen, C.J. & Levin, B.H. (2007). The world of 2020: Demographic shifts, cultural change, and social challenge. In J.A. Schafer (Ed.), Policing 2020: Exploring the future of crime, communities, and policing. Washington, DC: Federal Bureau of Investigation / Police Futures Working Group, pp. 31-70. 232 Johannsdottir, K.R. & Herdman, C.M. (2010). The role of working memory in supporting drivers’ situation awareness for surrounding traffic. Human Factors, 52 (6), 663-673. Johnson, R.B. & Onwuegbuzie, A.J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33 (7), 14-26. Kalamazoo Department of Public Safety (2010). Training Passport. Kalamazoo, MI.: Department of Public Safety, Training Division, internal memo. Kanter, R.M. (2000). Kaleidoscope thinking: Turning brainpower into business innovation. New York: Goodmeasure Books. Karoly, L.A. & Panis, C.W.A. (2004). The 21st century at work: Forces shaping the future workforce and workplace in the United States. Santa Monica, CA: RAND Corporation. Katz, C.M. (2001). The establishment of a police gang unit: An examination of organizational and environmental factors. Criminology, 39 (1), 37-74. Katz, D. & Kahn, R.L. (1978). The social psychology of organizations: Second edition. New York: Wiley. Keith, N., & Frese, M. (2008). Effectiveness of error-management training: A meta-analysis. Journal of Applied Psychology, 93, 59-69. King, W. R. (2009). Organizational failure and the disbanding of local police agencies. Crime & Delinquency, September (online publication). Kirkpatrick, D.L. (1998). Evaluating training programs, 2nd edition. San Francisco: BerrettKoehler Publishers, Inc. Klein, G., Moon, B., & Hoffman, R.R. (2006a). Making sense of sensemaking I: Alternative perspectives. IEEE Intelligence Systems, 21 (4), 70-73. Klein, G., Moon, B., & Hoffman, R.R. (2006b). Making sense of sensemaking II: Macrocognitive model. IEEE Intelligence Systems, 21 (5), 88-92. Kluge, A., Sauer, J., Burkolter, D., & Ritzmann, S. (2010). Designing training for temporal and adaptive transfer: A comparative evaluation of three training methods for process control tasks. Journal of Educational Computing Research, 43 (3), 327-353. Kraiger, K., Ford, J.K., & Salas, E. (1993). Application of cognitive skill-based and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311-328. Kramer, H.C. & Barr, D.J. (1974). In-service training for campus police. Community Mental Health Journal, 10 (2), 180-184. 233 Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd edition). Thousand Oaks, CA: Sage Publications. Laub, J.H. (2011, March 1). Translational criminology. Remarks from the National Institute of Justice Director. Washington, DC: National Institute of Justice. Retrieved from http://nij.gov/nij/about/speeches/translational-criminology-3-1-2011.htm Lazzara, E.H., Dietz, A.S., Weaver, S.J., Pavlas, D., Heyne, K., Salas, E., and Ramachandran, S. (2010). Guidelines for training adaptive expertise. Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting, pp. 2294-2298. Lenth, R.V. (2001). Some practical guidelines for effective sample size determination. American Statistician, 55 (3), 187-193. Lieberson, S. (1991). Small N’s and big conclusions: An examination of the reasoning in comparative studies based on a small number of cases. Social Forces, 70 (2), 307-320. Lipsky, M. (2010). Street-level bureaucracy: Dilemmas of the individual in public service (30th anniversary expanded edition). Thousand Oaks, CA.: Russell Sage Foundation. Liu, D., Blickensderfer, E.L., Macchiarella, N.D., & Vincenzi, D.A. (2008). Transfer of training. In D.A. Vincenzi, J.A. Wise, M. Mouloua, & P.A. Hancock (Eds.), Human factors in simulation and training, London: CRC Press, pp. 49-60. Long, J.S. & Ervin, L.H. (2000). Using heteroscedasticity consistent standard errors in the linear regression model. American Statistician, 54 (2), 217-224. Macey, W.H. & Schneider, B. (2008). The meaning of employee engagement. Industrial & Organizational Psychology, 1 (1), 3-38. Mangos, P.M. & Johnston, J.H. (2008). Performance measurement issues and guidelines for adaptive, simulation-based training. In D.A. Vincenzi, J.A. Wise, M. Mouloua, & P.A. Hancock (Eds.), Human factors in simulation and training, London: CRC Press, pp. 301320. Manning, P.K. (2008) Performance rituals. Policing: A journal of policy & practice, 2 (3), 284293. Martin, L.L., Levey, R., & Cawley, J. (2012). The ‘new normal” for local government. State And Local Government Review, 45 (1), 17-28. Mastrofski, S., Parks, R.B., Reiss, A.J., Worden, R.E., DeJong, C., Snipes, J.B., & Terrill, W. (1998, December). Systematic observation of public police: Applying field research methods to policy issues. Washington, DC.: United States Department of Justice, National Institute of Justice Research Report, NCJ # 172859. 234 Mastrofski, S. & Uchida, C. (1996). Transforming the police. In B. Hancock & P. Sharp (Eds.), Public policy: Crime and criminal justice, Upper Saddle River, NJ: Prentice-Hall, pp. 196-219. Masuch, M. (1985). Vicious circles in organizations. Administrative Science Quarterly, 30, 1433. Matarese, L. (2011, May-June). The current model of policing is not sustainable: PERF annual meeting panelists discuss the economy and the future of policing. Subject to Debate, 25 (3), 1, 8. Mayer, M.J. (2011, January 28). The duty to train officers is unaffected by the lack of reimbursement resources. California Police Chief’s Association. Retrieved November 2, 2011, from http://www.aele.org/mjm-duty2train.pdf McCampbell, M.S. (1986). Field training for police officers: State of the art. National Institute of Justice Research in Brief, 25 (USDOJ Publication No. 169871). Washington, DC: U.S. Government Printing Office. McCoy, M. (2006). Teaching style and the application of adult learning principles by police instructors. Policing: An international journal of Police Strategies and Management, 29 (1), 77-91. Mellenbergh, G.J. (1999). A note on simple gain score precision. Applied Psychological Measurement, 23, 87-89. Michigan Commission on Law Enforcement Standards (MCOLES) (2006, October). Statewide job task analysis of the patrol officer position: Final report. Lansing, MI.: Michigan Commission on Law Enforcement Standards. Miles, M.B. & Huberman, A.M. (1994). Qualitative data analysis: A sourcebook for new methods (2nd Ed.). Thousand Oaks, CA.: SAGE Publications. Mink, O.G., Schultz, J.M., & Mink, B.P. (1991). Developing and managing open organizations: A model and methods for maximizing organizational potential. Austin, TX: Catapult Press. Monroe, R. (2012, April 26). Use of force and public order. Remarks to the 2012 Police Executive Research Forum Annual Meeting, Washington, D.C. Mueller-Hanson, R.A., White, S.S., Dorsey, D.W., & Pulakos, E.D. (2005, October). Training adaptable leaders: Lessons from research and practice. Arlington, VA: United States Army Research Institute for the Behavioral and Social Science. Muir, W.K. (1977). Police: Streetcorner politicians. Chicago: University of Chicago Press. 235 Myers, M. (2000). Kalamazoo regional police and fire training needs assessment. Kalamazoo, MI.: City of Kalamazoo Department of Public Safety. National Research Council. (1999). The changing nature of work: Implications for occupational analysis. Washington, DC: Committee on Techniques for the Enhancement of Human Performance: Occupational Analysis, National Research Council. Nicholson, N. (1984). A theory of work role transitions. Administrative Science Quarterly, 29 (2), 172-191. Nimon, K., Zigarmi, D., & Allen, J. (2011). Measures of program effectiveness based on retrospective pretest data: Are all created equal? American Journal of Evaluation, 32 (1), 8-28. Osgood, C.E. (1949). The similarity paradox in human learning. Psychological Review, 56, 132143. Paxton, M.A. (2009). Toward a life-course perspective of police organizations. Policing: An International Journal of Police Strategies and Management, 32 (4), 822-836. PERF. See Police Executive Research Forum. Police Executive Research Forum (PERF). (2010). Critical issues in policing series: Is the economic downturn fundamentally changing how we police? Washington, DC: Police Executive Research Forum. ______. (2011, May/June). PERF annual meeting panelists discuss the economy and the future of policing. Subject to Debate, 25 (3), 1. Pollard, M. (1969, August). In-service training. Technical Education, 325-329. Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21 (3), 341-349. Pulakos, E.D., Arad, S., Donovan, M.A., & Palmondon, K.E. (2000). Adaptability in the workplace: Development of a taxonomy of adaptive performance. Journal of Applied Psychology, 85 (4), 612-624. Pulakos, E.D., Dorsey, D.W., & White, S.S. (2006). Adaptability in the workplace: Selecting an adaptive workforce. In C.S. Burke, L.G. Pierce, & E. Salas (Eds.), Understanding adaptability: A prerequisite for effective performance within complex environments. San Diego, CA: Elsevier Publishers, pp. 41-72. Pulakos, E.D., Schmitt, N., Dorsey, D.W., Arad, S., Hedge, J.W., & Borman, W.C. (2002). Predicting adaptive performance: Further tests of a model of adaptability. Human Performance, 15, 299-323. 236 Quinones, M.A. & Tonidandel, S. (2003). Conducting training evaluation. In J. Edwards, J. Scott, & N. Raju (Eds.), The Human Resurces Program Evaluation Handbook (Thousand Oaks, CA.: SAGE Publications), pp. 225-243. Reaves, B.J. (2009). State and local law enforcement training academies, 2006. USDOJ Publication 222987. Washington, DC: U.S. Government Printing Office. Reiss, A.J. (1971). Systematic observation of natural social phenomena. In H.L. Costner (Ed.), Social Methodology, San Francisco, CA: Jossey-Bass Publishers, pp. 3-33. Renwick, K. & Burrows, J. (2008). Metacognition and adaptive expertise: Building research capability through effective use of online technology. Proceedings of the National Vocational Education & Training Research Conference, Adelaide, Australia. Rojek, J., Kaminski, R.J., Smith, M.R., & Scheer, C.L. (2007). South Carolina Law Enforcement Training Survey: A National and State Analysis. Columbia, South Carolina: University of South Carolina, Department of Criminology & Criminal Justice. Rosenbaum, D., Cordner, G., Fridell, L., Mastrofski, S., McDevitt, J., & Schuck, A. (2009, December 11). From the academy to retirement: A journey through the policing lifecycle. Presentation given at National Institute of Justice, Research for the Real World Seminar series, Washington, DC. Retrieved October 12, 2012 from http://www.nij.gov/multimedia/presenter/presenter-rosenbaum/data/resources/presenterrosenbaum-transcript.htm Rousseau, D.M. (2006). Is there such a thing as “evidence-based management”? Academy of Management Review, 31 (2), 256-269. Royer, J.M. (1979). Theories of the transfer of learning. Educational Psychologist, 14, 53-69. Rutledge, M. (2009). Sensemaking as a tool in working with complexity. OD Practitioner, 41 (2), 19-24. Sackett, P.R. & Mullen, E.J. (1993). Beyond formal experimental design: Towards an expanded view of the training evaluation process. Personnel Psychology, 46, 613-627. Salas, E., Milham, L.M., & Bowers, C.A. (2003). Training evaluation in the military: Misconceptions, Opportunities, and challenges. Military Psychology, 15 (1), 3-16. Sanders, B.A. (2003). Maybe there’s no such thing as a “good cop”: Organizational challenges in selecting quality officers. Policing: An International Journal of Police Strategies and Management, 26, 313-328. 237 Schafer, J.A. (2007). Thinking about the future of policing. In J.A. Schafer (Ed.), Policing 2020: Exploring the future of crime, communities, and policing. Washington, DC: Federal Bureau of Investigation / Police Futures Working Group, pp. 13-30. Schein, E.H. (1998). Process consultation revisited: Building the helping relationship. Reading, PA.: Addison Wesley. Schwartz, D.L., Lin, X., Brophy, S., & Bransford, J.D. (1999). Towards the development of flexibly adaptive instructional design. In C.M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory, Mahwah, NJ: Erlbaum, pp. 183-213. Schwartz, M.D. & Yonkers, S.P. (1991). Officer satisfaction with police in-service training: An exploratory evaluation. American Journal of Police, 10 (4), 49-63. Scott, W.R. (1992). Organizations: Rational, Natural, and Open Systems. Englewood Cliffs, NJ.: Prentice Hall. Senge, P.M. (1990). The fifth discipline field book: Strategies and tools for building a learning organization. New York: Random House. Singer, J.M. & Andrade, D.F. (1997). Regression models for the analysis of pretest/posttest data. Biometrics, 53 (2), 729-735. Skolnick, J.H. & Fyfe, J.J. (1993). Above the law: Police and the excessive use of force. New York: Free Press. Smith, E.M., Ford, J.K., & Kozlowski, S.W. (1997). Building adaptive expertise: Implications for training design strategies. In M.A. Quinones & A. Ehrenstein (Eds.), Training for a rapidly changing workplace: Applications of psychological research. Washington, DC: American Psychological Association, 89-118. Smith-Ruig, T. (2009). Exploring career plateau as a multi-faceted phenomenon: Understanding the types of career plateau experienced by accounting professionals. British Journal of Management, 20, 610-622. Sonnentag, S. (2000). Expertise at work: Experience and excellent performance. International Review of Industrial and Organizational Psychology, 15, 223=264. Sonnentag, S., Niessen, C., & Ohly, S. (2004). Learning at work: Training and development. In C.L. Cooper & I.T. Robinson (Eds.), International review of industrial & organizational psychology, Vol. 19, Chichester: Wiley & Sons, 249-289. Stewart, S. (2012, March 24). A practical guide to situational awareness. Security Weekly, 16, 47-52. 238 Stout, S., Slocum, J., & Cron, W. (1988). Dynamics of the career plateauing process. Journal of Vocational Behaviour, 32, 74-79. Sweller, J., Van Merrienboer, J.J.G., & Paas, F.G.W. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10 (3), 251-296. Tabachnick, B.G. & Fidell, L.S. (2007). Using Multivariate Statistics. Boston: Pearson/AllynBacon. Tan, T.K. & Heracelous, L. (2001). Teaching old dogs new tricks: Implementing organizational learning in an Asian national police force. The Journal of Applied Behavioral Science, 37 (3), 361-380. Tashakkori, A. & Teddlie, C. (2003). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE. Tews, M.J. & Tracey, J.B. (2008). An empirical examination of posttraining on-the-job supplements for enhancing the effectiveness of interpersonal skills training. Personnel Psychology, 61, 375-401. Trochim, W.M.K. (1985). Pattern matching, validity, and conceptualization in program evaluation. Evaluation Review, 9 (5), 575-604. Tsang, E.W.K. & Zahra, S.A. (2008). Organizational unlearning. Human Relations, 61 (10), 1435-1462. United States Census Bureau (2008). American FactFinder. Retrieved August 19, 2011, from http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml United States Department of Justice, Office of Community Oriented Policing Services (COPS). (2010a). Unpublished proceedings from COPS Office Police Staffing Focus Group Meeting, Denver, September 9-10, 2010. United States Department of Justice, Office of Community Oriented Policing Services (COPS) (2010b, November). COPS to fund Teaching Police Department Initiative. Community Policing Dispatch, 3 (11). As of February 11, 2011: http://cops.usdoj.gov/html/dispatch/November_2010/TPDI.asp United States Marine Corps (1997). Field manual: Operational terms and graphics. Washington, DC: Department of the Army, Marine Corps Reference Publication No. 101-5-1. Vanasupa, L., Stolk, J., & Harding, T. (2010). Application of self-determination and selfregulation theories to course design: Planting the seeds of adaptive expertise. International Journal of Engaging Education, 26 (4), 914-929. 239 Vandergriff, D.E. (2008, December). Building adaptive leaders: The Army can adapt its institution. Small Wars Journal. Retrieved from http://smallwarsjournal.com/blog/journal/docs-temp/147-vandergriff.pdf Van Merrienboer, J.J.G. & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17 (2), 147177. Vera, D. & Crossan, M. (2004). Strategic leadership and organizational learning. Academy of Management Review, 29 (2), 222-240. Walker, J.T. (2008). Advancing science and research in criminal justice/criminology: Complex systems theory and non-linear analyses. Justice Quarterly, 24 (4), 555-581. Weick, K.E. (1976, March). Educational organizations as loosely coupled systems. Administrative Science Quarterly, 21 (1), 1-19. _____. (1993, September). Collective mind in organizations: Heedful interrelating on flight decks. Administrative Science Quarterly, 38 (3), 357-381. _____. (1995). Sensemaking in organizations. Thousand Oaks, CA: SAGE Publications. Weiss, C.H. (1998). Evaluation: Methods for studying programs and policies (2nd Edition). Upper Saddle River, NJ: Prentice-Hall. Weston, C., Gandell, T., Beauchamp, J., McAlpine, L., Weisman, C., & Beauchamp, C. (2001). Analyzing interview data: The development and evolution of a coding system. Qualitative Sociology, 24 (3), 381-399. Wexler, C., Wycoff, M.A., & Fischer, C. (2007, June). Good to great policing: Application of business management principles in the public sector. Washington, DC: United States Department of Justice, Office of Community-Oriented Policing Services (COPS). White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817-838. White, M.D. (2008). Identifying good cops early: Predicting recruit performance in the academy. Police Quarterly, 11 (1), 27-49. White, S.S., Mueller-Hanson, R.A., Dorsey, D.W., Pulakos, E.D., Wisecarver, M.M., Deagle, E.A., & Mendini, K.G. (2005). Developing adaptive proficiency in Special Forces officers. Arlington, VA: United States Army Research Institute for the Behavioral and Social Science. Wilson, J.M., Dalton, E., Scheer, C., & Grammich, C.A. (2010a). Police recruitment and retention for the new millennium: The state of knowledge. Santa Monica, CA: Rand Corporation. 240 Wilson, J. M. & Grammich, C. (2012, February). Police Consolidation, Regionalization, and Shared Services: Options, Considerations, and Lessons from Research and Practice. BOLO (Be On The Lookout): A Continuing Publication Highlighting COPS Office Community Policing Development Projects. Washington, DC: US Department of Justice, Office of Community Oriented Policing Services. Wilson, J.M., Rostker, B.D., & Fan, C. (2010b). Recruiting and retaining America’s finest: Evidence-based lessons for police workforce planning. Santa Monica, CA: Rand Corporation. Yelon, S.L. & Ford, J.K. (1999). Pursuing a multidimensional view of transfer. Performance Improvement Quarterly, 12 (3), 58-78. Zohoori, N. (1997). Does endogeneity matter? A comparison of empirical analyses with and without control for endogeneity. Annals of Epidemiology, 7 (4), 258-266. Zohoori, N. & Savitz, D.A. (1997). Econometric approaches to epidemiological data: Relating endogeneity and unobserved heterogeneity to confounding. Annals of Epidemiology, 7 (4), 251-257. Zunker, V. (2008). Career, work, and mental health: Integrating career and personal counseling. Thousand Oaks, CA: SAGE Publications. 241