TESTING THE EFFECTS OF PARTICIPATORY RESEARCH ON YOUTH By Sara T. Stacy A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Ecological-Community Psychology—Master of Arts 2018 ABSTRACT TESTING THE EFFECTS OF PARTICIPATORY RESEARCH ON YOUTH By Sara T. Stacy Although the extant literature suggests that youth can derive individual benefits from involvement in participatory research within educational contexts, few empirical studies exist that examine this suggestion. The current study used a design that allows for the examination of causality to examine whether involvement in participatory research leads to specific benefits among youth in educational contexts. Youth were randomly assigned to either a symbolic participation condition (i.e., traditional focus groups) or a pluralistic participation condition (i.e., Youth Generate and Organize Groups). To examine of the effects of participation type, youth completed measures of school attitudes, perceived control, and critical social reflection both before and after involvement in the group. In addition, youth also completed measures at post- assessment only to assess their perceptions of participation type. I hypothesized that—compared to the symbolic condition—the pluralistic condition would be associated with greater (H1) youth voice in decision making and supportive adult relationships, (H2) satisfaction and acceptability, and (H3) increases in school attitudes, perceived control, and critical social reflection. Although some mean differences were observed, none rose to statistical significance. These null findings challenge the assumption that youth involved in participatory research are measurably impacted on certain variables. Findings point to important directions for future research to address the limitations of the current study. The insights from the current study aims to foster the advancement of research approaches within educational settings that empowers and positively impacts youth. ACKNOWLEDGEMENTS I am deeply grateful for those who made this thesis possible. My advisor Ignacio D. Acevedo-Polakovich, PhD, for his guidance and support throughout the duration of this project and beyond. My committee members Rebecca Campbell, PhD, and Heng-Chieh Jamie Wu, PhD, for their important insights and contributions to the project. And Deborah Kashy, PhD, for her support on the data analysis and results. I owe my deepest gratitude to the community partnership in which this project took place, including the project staff that supported the completion of this work, and the youth that participated in this project. Without this partnership, none of the work could have occurred, and for that, I am forever grateful for their collaboration. Finally, I would like to thank all of my close friends and family who have encouraged, supported, and believed in me leading up to and throughout this entire process. I find so much strength in those around me and many of you have supported in ways in which I can never accurately express. Finally, I would like to thank my partner in life, Bryant Bussa, who has supported my dreams and work from the moment we met and has allowed me to spend so much time and energy on a career that I love, even when it means to put his own passions behind mine. I would never be able to achieve this moment without the help of those mentioned here. iii TABLE OF CONTENTS LIST OF TABLES ......................................................................................................................... vi LIST OF FIGURES ...................................................................................................................... vii LITERATURE REVIEW ................................................................................................................1 Participatory Research with Youth ..............................................................................................1 Participatory Research in Educational Contexts ..........................................................................3 Effects on Youth ...................................................................................................................3 School attitudes ........................................................................................................3 Perceived control .....................................................................................................5 Critical social reflection ...........................................................................................5 Summary and critique ..............................................................................................6 CURRENT STUDY.........................................................................................................................8 Hypothesis one (manipulation check) ..........................................................................................8 Hypothesis two .............................................................................................................................8 Hypothesis three ...........................................................................................................................8 METHODS ......................................................................................................................................9 Context .........................................................................................................................................9 Participants ...................................................................................................................................9 Measures .....................................................................................................................................10 Background Information .........................................................................................................10 Youth-adult partnerships ........................................................................................................10 Youth voice in decision making ........................................................................................10 Supportive adult relationships............................................................................................11 Youth satisfaction and acceptability .......................................................................................11 School attitudes .......................................................................................................................12 Motivation to influence ......................................................................................................12 Attitudes toward school .....................................................................................................13 Perceived control ....................................................................................................................13 Perceived control ...............................................................................................................13 Policy control .....................................................................................................................13 Critical social reflection ..........................................................................................................14 Implementation fidelity ..........................................................................................................15 Facilitator Questionnaire ........................................................................................................15 Procedures ..................................................................................................................................15 Participant recruitment, consent, and pre-assessment ............................................................15 Random assignment and evaluation approaches ....................................................................16 Pluralistic participation condition: Youth Generate and Organize ....................................16 Symbolic participation condition: Focus groups ...............................................................18 Controls for experimental bias ................................................................................................18 iv Addressing threats to internal validity ....................................................................................19 Facilitator Training .................................................................................................................21 Implementation .......................................................................................................................22 Data collection ........................................................................................................................23 RESULTS ......................................................................................................................................24 Data Quality ...............................................................................................................................24 Preliminary Analyses .................................................................................................................24 Descriptive Statistics ..........................................................................................................24 Implementation fidelity ......................................................................................................26 Main Analyses ............................................................................................................................26 Analytic Strategy ...............................................................................................................26 Hypothesis one (manipulation check) ................................................................................28 Hypothesis two...................................................................................................................29 Hypothesis three.................................................................................................................29 School attitudes ......................................................................................................29 Perceived control ...................................................................................................30 Critical social reflection .........................................................................................30 DISCUSSION ................................................................................................................................31 Limitations .................................................................................................................................32 Limited statistical power ....................................................................................................32 Manipulation failure...........................................................................................................33 Misspecified outcomes.......................................................................................................33 Additional limitations ........................................................................................................34 No Actual Effects .......................................................................................................................35 Conclusion ..................................................................................................................................36 APPENDICES ...............................................................................................................................37 Appendix A: Community Schools Evaluation Approach ..........................................................38 Appendix B: Measure Item Pool ................................................................................................39 Appendix C: Youth GO: Implementation Fidelity Checklist .....................................................43 Appendix D: Focus Group: Implementation Fidelity Checklist ................................................44 Appendix E: Facilitator Questionnaire .......................................................................................45 Appendix F: Flier to Recruit Youth Participants (Deidentified) ................................................48 Appendix G: Participant Informed Consent Form (Deidentified) .............................................49 Appendix H Youth GO Protocol (Deidentified) ........................................................................51 Appendix I: Focus Group Protocol (Deidentified) .....................................................................56 Appendix J: Facilitator Training (Deidentified) .........................................................................59 REFERENCES ...........................................................................................................................63 v LIST OF TABLES Table 1. Participant Demographic Characteristics .........................................................................10 Table 2. Threats to Internal Validity and Their Application to the Current Study ........................20 Table 3. Means, Standard Deviations, and Analysis of Variance (ANOVA) Results for Dependent Variables as a Function of Time and Group ...............................................................25 Table 4. Summary of Mean Implementation Fidelity Data ...........................................................26 Table 5. Analysis of Variance Results for Between Subjects Effects ...........................................28 Table 6. Measure Item Pool ...........................................................................................................39 vi LIST OF FIGURES Figure 1. Typology of Youth Participation & Empowerment Pyramid (Wong et al., 2010). .........2 Figure 2 Youth GO, a five-step participatory research approach. .................................................18 Figure 3. Study participants organized within a nested design ......................................................27 vii Participatory Research with Youth LITERATURE REVIEW Participatory research meaningfully includes the people that science is intended to impact in a way that is community or constituent-driven, systematic, inclusive, and focused on change (Minkler, 2000). This contrasts sharply against the conventional model of science in which members of the community are seen as subjects and the researchers are treated as the experts (Whyte, Greenwood, & Lazes, 1989). A central component of participatory research is the value placed on knowledge derived from people most closely linked to the issue being studied (London, Zimmerman, & Erbstein, 2003). Thus, individuals “directly affected by the research problem at hand must participate in the research process” (Gaventa & Cornwall, 2007, p. 74). In participatory research with youth, youth are seen as experts on their experiences, and as necessary members of a research team (Zeller-Berkman, Muñoz-Proto, & Torre, 2013). In participatory research with youth, adults (i.e. scholars and practitioners) collaborate with youth across the research process to understand the problems that impact youth, to determine solutions to those problems, and to evaluate the effectiveness of those solutions (Cammarota & Fine, 2008; Creswell, Hanson, Plano Clark, & Morales, 2007; Hubbard, 2015). This collaboration between youth and adults can take on a variety of different forms and levels of involvement, depending on each project (Jacquez, Vaughn, & Wagner, 2013; Wong, Zimmerman, & Parker, 2010). When designing participatory projects, adults work to meaningfully include youth in the research process by focusing on what youth can do, and developing the context that can allow them to grow (Chen, Weiss, Johnston Nicholson, & Girls Incorporated, 2010). Participatory research with youth must be flexible enough to properly support youth and to provide enough youth ownership and control (Ozer et al., 2008). 1 Wong and colleagues (2010) provide a model for categorizing participatory research with youth based on two factors: youth empowerment and youth-adult control. Entitled the Typology of Youth Participation and Empowerment (TYPE) Pyramid, this model—illustrated in Figure 1—includes five types of participatory research with youth: (1) Vessel, (2) Symbolic, (3) Pluralistic, (4) Independent, and (5) Autonomous. According to Wong and colleagues, the ideal type of participatory research with youth is pluralistic participation. In this type, youth and adults have a reciprocal relationship where both parties share responsibilities to achieve goals, thereby resulting in youth empowerment and positive youth development. Wong and colleagues highlight that while youth have important strengths and assets to provide to research projects, adult support can sufficiently maximize those strengths. Importantly, pluralistic partnership does not equate to egalitarian action and decision making, but rather tasks and responsibilities for each project should be defined based on the strengths of both parties and the developmental stage of the youth involved. Figure 1. Typology of Youth Participation & Empowerment Pyramid (Wong et al., 2010). 2 Participatory Research in Educational Contexts Participatory research with youth is often utilized in educational contexts, both within- and out-of-school time. Educational contexts provide an important, developmentally-appropriate setting for meaningful collaboration between youth and adults in research (Ozer, Ritterman, & Wanis, 2010). Participatory research has been used to involve youth in projects to understand and evaluate the educational programs that they participate in (Chen et al., 2010; Hubbard, 2015; Zeller-Berkman et al., 2013), and in projects to document and improve school conditions (Mitra, 2004; Van Sluys, 2010; Vaughn, Jacquez, Zhao, & Lang, 2011; Wilson et al., 2007). Participatory research with youth in educational contexts can result in important organizational effects, including programmatic insights for future development (Chen et al., 2010), positive shifts in organizational culture and improved research quality (Zeller-Berkman et al., 2013), and improved school conditions (Wilson et al., 2007). Effects on Youth In addition to the organizational benefits of participatory research with youth, an emerging body of literature suggests that involvement in participatory research can have positive effects on youth (Berg, Coman, & Schensul, 2009; Checkoway, Allison, & Montoya, 2005; Foster-Fishman, Nowell, Deacon, Nievar, & McCann, 2005; Hubbard, 2015; Ozer & Douglas, 2013; Ozer et al., 2010; Zeller-Berkman et al., 2013). Within educational contexts, three broad areas of effects on youth as a result of involvement in participatory research have been documented: school attitudes, perceived control, and critical social reflection. In the sections to follow, each broad effect area will be described in more detail. School attitudes. School attitudes refers to youths’ emotional beliefs about the school they attend and their desire to improve it (McCoach & Siegle, 2003; Ozer & Douglas, 2013). A 3 few emerging studies within educational contexts have begun to identify a connection between youth involvement in participatory research and school attitudes (Mitra, 2004; Ozer & Douglas, 2013). In the first of these, Mitra (2004) examined the effects on youth development outcomes (including school attitudes) associated with high school students’ (N = 20) involvement in participatory school reform projects. In projects that could be classified between Wong and colleagues’ (2010) pluralistic and independent types of participation, youth and adults focused on improving school conditions. Qualitative data was gathered from several sources, including interviews with various stakeholders (e.g., youth participants, school administrators, adult leaders, and non-participant youth and teachers), observational notes from formal and informal meetings and interactions, and written documents created for project purposes. Results from a grounded theory analysis of these data suggested that youth involved in the school reform projects had positive improvements in school attitudes including sense of belonging, attachment, and pride in their school (Mitra, 2004). Ozer & Douglas (2013) examined the connection between youth involvement in participatory research projects and school attitudes, specifically focusing on youths’ motivation to influence their school. In this study, a large group of ethnically-diverse youth (N = 401) from five high schools were randomly assigned to enroll in one of two courses, either (1) a course focusing on the development and execution of a student-led participatory research project or (2) a course focusing on direct-service peer mentoring and education with no participatory research component. The course focusing on the student-led participatory research could be classified between Wong and colleagues’ (2010) pluralistic and independent participation types, while the direct-service course could be classified as symbolic youth participation. In the participatory research course, youth selected a research topic, collected data, and engaged in action, while 4 adults provided training and support in each phase. Qualitative and quantitative data were gathered from participants to determine project effects, including pre, post, and one semester follow-up surveys and focus groups. Pre-to-post increases in youth’s motivation to influence their schools and communities were modestly, but significantly, larger among youth in the participatory research course (Ozer & Douglas, 2013). Perceived control. Perceived control refers to youth’s perceived ability to create change and influence decisions in school and community contexts (Peterson, Peterson, Agre, Christens, & Morton, 2010). The studies reviewed in the section on school attitudes also examined the effects of youth involvement in participatory research on perceived control (i.e., Mitra, 2004; Ozer & Douglas, 2013). The youth in Mitra’s (2004) qualitative study felt that their opinions and views were heard during their involvement in participatory projects, and they found new identities as “change makers” within the school, reporting more confidence to speak out and express their beliefs (Mitra, 2004). In contrast, in Ozer and Douglas’ (2013) cluster-randomized study, assignment to the participatory condition was not associated with enhanced changes in Perceived Control subscale of the Psychological Empowerment Scale (Ozer & Schotland, 2011). Critical social reflection. Critical social reflection refers to youths’ reflection about, and analysis of, social constructs and inequalities that inhibit well-being and agency (Watts, Diemer, & Voight, 2011). While only Mitra’s (2004) study has examined the link between involvement in participatory projects within educational contexts and increased sense of critical social reflection, studies in non-educational settings have identified such a link (e.g., Carlson, Engebretson, & Chamberlain, 2006; Foster-Fishman, Nowell, Deacon, Nievar, & McCann, 2005). Mitra (2004) found that youth involved in participatory projects felt an increased ability to critique their environment and identify injustices present in their school. Outside of educational settings, 5 Foster-Fishman and colleagues (2005) and Carlson and colleagues (2006) have also found this link. Foster-Fishman and colleagues (2005) interviewed youth and adults (N=16) involved in a participatory photovoice project. All interviewees described a deeper awareness of community conditions as a result of participating in the project. Carlson and colleagues’ (2006) conducted an ethnography of African American youth and adults from an economically disadvantaged community who participated in a photovoice project. Their findings documented that participants moved from passive participation to higher levels of critical reflection over the course of the project. Summary and critique. The conceptual literature suggests that youth can derive individual benefits from involvement in participatory research within educational contexts. However, few empirical studies exist that examine this suggestion. Most of these studies employ descriptive methods that prevent researchers from determining causality. It is important to empirically examine causal relationships between variables, in order to have a greater understanding of these constructs and to result in more accurate and useful predictions in the future (Singleton & Straights, 2005). The sole experimental study examining the effects of participation (i.e., Ozer & Douglas, 2013) did not contrast varying types of participation. This is important because existing conceptual models of youth-adult control in participatory projects (e.g., Wong et al., 2010) suggest that not all forms of participation provide optimal conditions for positive youth development. While no single study can arrive at a definitive understanding of the individual benefits on youth resulting from their involvement in participatory research within educational settings, this study aims to contribute to the existing literature that examines this causal relationship, resulting in a better understanding of both constructs. 6 In addition, the existing literature examining youth involvement in research projects within educational contexts does not examine youths’ acceptability and satisfaction with the different research approaches. Acceptability is defined as a perception among the target population that treatment, practice, innovation—or in this case research condition—is agreeable and is measured by participants’ direct experience (Proctor et al., 2011). Satisfaction is similar to acceptability, but typically documents general service experience and specific features of the treatment (e.g., environment, waiting times, etc.; Proctor et al., 2011). Acceptability and satisfaction can be considered necessary preconditions for obtaining the desired outcomes of an intervention. While the existing literature largely focuses individual changes within youth as a result of their involvement in participatory research projects, it is critical to also understand youths’ perceptions and experiences with these research projects. Thus, to build upon the current literature, it is important to measure these implementation outcomes as an intermediary measure that might preclude—or impact the ability to detect—direct changes in youth as a result of their involvement in participatory research projects within educational settings. 7 CURRENT STUDY The current study examined the effects of participatory research with youth in educational contexts using a design that allows for the examination of causality. This study used data collected for the evaluation of an educational practice implemented in a predominately African American public school district in the Midwestern United States (U.S.). In collecting these data, youth were randomly assigned to either a symbolic participation condition (i.e., traditional focus groups) or a pluralistic participation condition (i.e., Youth Generate and Organize Groups). To examine of the effects of participation type, youth completed measures of school attitudes, perceived control, and critical social reflection both before and after involvement in the group. In addition, youth also completed measures at post-assessment only to assess their perceptions of participation type. The following hypotheses guided my investigation: Hypothesis one (manipulation check). Youth who participated in the pluralistic participation condition will report higher youth voice in decision making and supportive adult relationships after participating in the project that than youth involved in the symbolic condition. Hypothesis two. Youth who participated in the pluralistic participation condition will report higher levels of satisfaction and acceptability after participating in the project than youth involved in the symbolic condition. Hypothesis three. Youth who participated in the pluralistic participation condition will experience greater pre-to-post increases in (a) school attitudes, (b) perceived control, and (c) critical social reflection than youth involved in the symbolic condition. 8 Context METHODS The data for the proposed study were collected within the context of a broader project to evaluate the implementation of the Children’s Aid Society Community Schools Approach within a public school district of a mid-sized city in Michigan. The district enrolled approximately 5,000 total students across eleven school buildings in the 2016-17 school year (Michigan Department of Education, 2017). The district implemented the Community Schools Approach as an integrated student support strategy to improve the district and address the needs of students, parents, and community residents that live within the district’s physical boundaries. The Community Schools Approach develops each school into a neighborhood hub, offering services and supports for children, families, and community residents during and after school hours. The services and supports offered through this Community Schools Approach focus on improving four outcomes: increased student attendance, improved third grade reading levels, increased grade promotion and graduation rates, and increased community engagement. Students within the district participated in an evaluation to document their perspectives on the Community Schools Approach. The evaluation focused on what the Community Schools Approach meant to them, what they liked, and suggestions for improvement. (See Appendix A for a more detailed description of the evaluation component of this project.) Findings from this evaluation project were designed to influence future implementation of the Community Schools Approach within the district. Participants During the summer of 2017, Community School staff recruited 29 students to participate in the evaluation activities. After consenting to participate, students were randomly assigned to 9 either the symbolic (n = 16) or pluralistic (n = 13) participation condition. Table 1 provides a summary of participants’ responses to demographic questions. Table 1. Participant Demographic Characteristics Race African American Multi-Racial Indian Not Reported Gender 13 2 1 0 5 11 M (SD) 13.1 (1.84) 7.9 (2.0) 3.2 (0.5) Male Female Age Gradea GPAb Note. a2 cases missing, b17 cases missing Measures Symbolic Participation Condition (n = 16) n % Pluralistic Participation Condition (n = 13) n % 81.25 12.5 0.06 0 31.3 68.8 69.2 15.4 0.08 0.08 30.8 69.2 9 2 1 1 4 9 M (SD) 12.8 (2.0) 7.8 (1.9) 3.9 (0.5) Total (N = 29) N % 75.9 22 13.8 4 0.07 2 0.03 1 31.0 9 69.0 20 M (SD) 12.9 (1.9) 7.8 (1.9) 3.5 (0.5) All measures are included in Appendix B. Background information. Participants were asked to report their gender, age, race/ethnicity, school, grade, and parent or adult caregivers’ educational level. Youth-adult partnerships. Two different components of youth-adult partnerships were examined in this study: youth voice in decision making, and supportive adult relationships. Youth voice in decision making. Participants were asked to use a five-point Likert-type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to the four items of 10 the Youth Voice in Decision Making subscale of the Youth-Adult Partnership scale (Zeldin, Krauss, Collura, Lucchesi, & Sulaiman, 2014). These items ask youth to rate the extent to which they experience voice in decision making (e.g., “The staff take my ideas seriously”). Previous research supports the internal consistency of this scale (α = .82) among a racially diverse sample of youth and adolescents, ages 11 to 24, who participated in youth development programs located in the United States, Malaysia, and Portugal (Zeldin et al., 2014). For the present study, the original item wording was slightly altered to better fit the current context (e.g., “The adults in this group take my ideas seriously”). In the current sample, the internal consistency of the modified youth voice in decision making scale measure was α = .51 at post-test. Supportive adult relationships. Participants were asked to use a five-point Likert-type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to the five items of the Supportive Adult Relationships subscale of the Youth-Adult Partnership Scale (Zeldin et al., 2014). These items ask youth to rate the extent to which they perceive supportive adult relationships (e.g., “Youth and staff trust each other in this center”). Previous research supports the internal consistency of this scale (α = .87) among a racially diverse sample of youth and adolescents, ages 11 to 24, who participated in youth development programs located in the United States, Malaysia, and Portugal (Zeldin et al., 2014). For the present study, the original item wording was slightly altered to better fit the current context (e.g., “Youth and adults trust each other in this group”). In the current sample, the internal consistency of the supportive adult relationships scale measure was α = .76 at post-test. Youth satisfaction and acceptability. Participants were asked to use a five-point Likert- type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to twelve items assessing participants’ perceptions of the evaluation approach they participated in, including 11 overall satisfaction and acceptability for youth their age (e.g., I enjoyed the activities of the group I was in today). The items were based on an evaluation form for ¡Cuídate!, a sexual risk- reduction program for Latina/o youth (Villarruel & Eakin, 2008), the Primary Intervention Rating Scale (Lane et al., 2009), and the Modified Children’s Intervention Rating Profile (Mitchell, Tingstrom, Dufrene, Ford, & Sterling, 2015). Previous intervention ratings and assessments often assess a specific type of intervention (e.g., behavior modification) or specific components of the intervention such as program modules (e.g., safe sex behaviors) and implementer of the intervention (e.g., teacher). In contrast, the twelve items created for the current study aim to assess satisfaction and acceptability that is not specific to an intervention or its components. In the current sample, the internal consistency of the satisfaction and acceptability measure was α = .87 at post-test. School attitudes. Two different types of school attitudes were examined in this study: motivation to influence, and attitudes toward school. Motivation to influence. Participants were asked to use a five-point Likert-type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to the four-item Motivation to Influence subscale of the Psychological Empowerment Scale (Ozer & Schotland, 2011). These items ask participants to rate their enthusiasm for impacting positive change in their school or community (e.g., It is important for youth to try to improve our city even if we can’t always make the changes we want). Prior research supports the internal consistency of the scale (α = .80) among a racially diverse sample of urban adolescents, ages 13 to 19, from schools and community based organizations in the United States (Ozer & Schotland, 2011). In the current sample, the internal consistency of this measure was α = .28 at pre-test, and α = .86 at post-test. 12 Attitudes toward school. Participants were asked to use a five-point Likert-type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to the five items from the Attitudes Toward School scale of the School Attitude Assessment Survey-Revised (McCoach & Siegle, 2003). These items ask participants to report their interest in, and affect toward, the school they attend (e.g., I am glad that I go to this school). Prior research supports the internal consistency of this scale (α = .85) among a large sample of high school students from several educational settings including a competitive summer program, a multiethnic urban high school, and a national sample of regular- and under-achieving high school students from several districts (McCoach & Siegle, 2003). In the current sample, the internal consistency of this measure was α = .89 at pre-test, and α = .96 at post-test. Perceived control. Two different types of perceived control were examined in this study: perceived control, and policy control. Perceived control. Participants were asked to use a five-point Likert-type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to the six items of the Perceived Control subscale of the Psychological Empowerment Scale (Ozer & Schotland, 2011). These items ask participants to rate the degree to which they believe youth have decision making power in their school and city (e.g., Students have a say in what happens at this school). Prior research supports the internal consistency of the scale (α = .80) among a racially diverse sample of urban adolescents, ages 13 to 19, from schools and community based organizations in the United States (Ozer & Schotland, 2011). In the current sample, the internal consistency of this measure was α = .76 at pre-test, and α = .62 at post-test. Policy control. Participants were asked to use a five-point Likert-type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to the nine items from the Policy 13 Control subscale of the Sociopolitical Control Scale for Youth (Peterson, Hamme Peterson, Agre, Christens, Morton, 2010). These items ask participants to assess the extent to which they can understand and influence policy decisions in their community or school (e.g., Youth like me can really understand what’s going on with my community or school). Prior research supports the internal consistency of this scale (α = .85) among a large sample of racially and ethnically diverse high school students in the United States (Peterson, et al., 2010). In the current sample, the internal consistency of this measure was α = .64 at pre-test, and α = .63 at post-test. Critical social reflection. Participants were asked to use a five-point Likert-type scale (ranging from 1 = Strongly disagree to 5 = Strongly agree) to respond to five items created for this study assessing critical social reflection. These items ask participants to report the extent to which they reflect on the way social institutions, laws and rules, and authority figures impact people differently (e.g., I spend a lot of time wondering how laws and rules could be more fair). The items were based on other measures of critical reflection including the Critical Consciousness Scale (Diemer et al., 2010) and the Contemporary Critical Consciousness Measure (Shin, Ezeofor, Smith, Welch, & Goodrich, 2016). Previously developed measures of critical reflection assess this construct through the lens of specific forms of oppression and prejudice such as racism, sexism, classism, and heterosexism. Additionally, the items in previous measures often reflect endorsement or rejection of these oppressive beliefs, rather than assessing reflective behaviors about social conditions. In contrast, the five items developed for this study aim to evaluate youth’s assessment of social conditions that are not connected to specific forms of oppression. In the current sample, the internal consistency of these items was α = .42 at pre- test, and α = .79 at post-test. 14 Implementation fidelity. Facilitators assigned to the pluralistic participation condition were asked to complete a fidelity checklist after implementing the evaluation approach. The checklist required them to indicate if they used all materials and conducted all steps within the protocol. It also required facilitators to include explanations if any deviations from the protocol occurred (e.g., materials were not used or tasks were not carried out). (See Appendix C for the Implementation Fidelity Checklist for the pluralistic participation condition.) A parallel fidelity protocol was developed for the symbolic participation condition. In this protocol, the current study’s first author utilized focus group transcripts to determine if the facilitators used all materials and conducted all steps within the protocol. (See Appendix D for the Implementation Fidelity Checklist for the symbolic participation condition) Facilitator questionnaire. After conducting all focus groups, facilitators were asked to complete a questionnaire via email. The questionnaire was designed for the purposes of this study and included multiple choice and open-ended response questions about the following topics: demographics and background information, competence working with and facilitating groups middle and high school youth, comfort working with the target demographic population of this study (i.e., low SES, African American students), satisfaction and acceptability of the evaluation approach they implemented (see Appendix E). Procedures Participant recruitment, consent, and pre-assessment. Community School staff at four schools serving grades seven through twelve recruited students to participate in the study. Community School staff were asked to recruit approximately 20 students to participate in an activity designed to document their perspectives on the Community Schools Approach. Community School staff members distributed a flier to inform students and their parents about 15 the opportunity to participate (see Appendix F for a deidentified example). Staff made it clear to students that participation in the activity was voluntary, and that refusing to participate would have no bearing on their—or their families’—relationship with the schools (including ability to receive services and supports). The Institutional Review Board at Michigan State University approved a research exemption for the use of these data. The use of these data were included within IRB Review Exemption B1, which applies to research conducted in established educational settings, involving normal educational practices. On the day research procedures occurred, students who were eligible to participate were brought to a classroom by the Community School staff. Upon arrival, research staff described the purpose of the activities and explained an informed consent form (see Appendix G) to the youth. Youth that wished to participate in the activities signed the assent form. Only one youth declined participation. Research staff then distributed the pre-evaluation assessment, explained the purpose of the assessment, and read aloud the entire assessment form, answering questions when needed. Random assignment and evaluation approaches. Within each school building, each assenting participant was assigned to one of two evaluation approaches using sequentially numbered, opaque, sealed envelopes. To limit the potential for manipulation, the envelopes were numbered in advance, and opened sequentially (Dettori, 2010). The youth involved in this study were assigned to participate in one of the two following evaluation approaches. Pluralistic participation condition: Youth Generate and Organize (Youth GO; Stacy, Acevedo-Polakovich, & Rosewood , in press). Youth GO is a participatory research approach developed to meaningfully engage youth in research processes. Youth GO was developed by combining specific components of existing participatory research approaches (Foster-Fishman, 16 Law, Lichty, & Aoun, 2010; Vaughn et al., 2011) with the goal of maximizing on the strengths of these approaches and offsetting their limitations. The Youth GO approach contains five steps, as summarized in Figure 2. During step one, climate setting, facilitators explain the purpose and goals of the approach and youth work with the facilitator/s to create rules that will guide group dynamics. During step two, generating, youth individually answer prompts by writing their responses on post-it notes and placing them on a sheet of flip chart paper on which the prompt was written. After all responses to a prompt are gathered, youth reflect on each other’s answers and clarify responses through a facilitated discussion. In step three, organizing, youth are taught data organization skills by sorting assorted candy. Youth then use these data organization skills to collaboratively organize and interpret the perspectives shared during step two. The youth work together to organize the responses into meaningful themes, place the responses onto different colored paper, and develop names for the themes. In step four, selecting, youth discuss the themes created and define meaningful categories for those themes. Then, youth crosscheck the categories against the themes defined in step three to ensure that each category aligns with at least one theme. In step five, debrief and discussion, facilitators remind youth of the purpose and goals of the activities, highlight the importance of their perspective, and facilitate a brief discussion about their experience participating in the activities. (See Appendix H for the detailed protocol utilized by facilitators of this evaluation approach). 17 Figure 2. Youth GO, a five-step participatory research approach. Symbolic participation condition: Focus groups. An independent expert, kept uninformed about the goals of the study, designed the focus group protocol. The expert was asked to design a protocol that would successfully address the guiding questions for the overall evaluation of the Community Schools Approach. Additionally, the expert was asked to adhere to existing standards for designing and reporting qualitative research (i.e., O’Brien, Harris, Beckman, Reed, & Cook, 2014; Tong, Sainsbury, & Craig, 2007). Facilitators began the focus group protocol by explaining the purpose of the focus group to the students. Facilitators were given base questions to guide group discussion and these were presented to the participants one at a time and written on flip chart paper. Facilitators used probes and follow up questions as necessary to guide the discussion and understand students’ responses. Facilitators took notes during the focus groups on flip chart paper to document the key topics of discussion. (See Appendix I for the detailed protocol utilized by the facilitators.) Controls for experimenter bias. Experimenter bias refers to the ways in which experimenters can affect the results of the research, such as experimenter’s expectations of the research findings (Singleton & Straights, 2005). First, all research staff were kept blind to study goals and hypotheses, with the exception of this thesis proposal’s author and her research advisor. Research staff include the faculty member responsible for the design, implementation, 18 and analysis of the focus group (including the training and supervision of the research assistants assigned to facilitate this group) and all research assistants. Second, research assistants were randomly assigned to receive training and supervision on one of the two groups. Once assigned, the teams facilitating each group met independently of each other and were instructed to only discuss specific research activities with staff assigned to their team. Addressing threats to internal validity. Internal validity is the degree to which the observed variation in the experimental conditions results in variation between the presumed outcomes of the study (Shadish, Cook, & Campbell, 2002). In this study, it reflects the degree to which differences in the outcome measures collected can be attributed to the varied treatment groups (i.e., symbolic or pluralistic participation condition). In any study, certain properties of the design can threaten internal validity, thereby impacting researchers’ ability to determine causal relations (Shadish, Cook, & Campbell, 2002). Shadish and colleagues (2002) have outlined nine such threats. Table 2 summarizes these threats and describes their relevance to the current study. As can be observed, the a priori likelihood that these nine threats could, either individually or collectively, affect internal validity was low. 19 Table 2. Threats to Internal Validity and Their Application to the Current Study. Threat to Internal Validity Ambiguous temporal precedence Definition (Shadish, Cook, & Campbell, 2002) Lack of clarity about which variable occurred first may yield confusion about which variable is the cause and which is the effect. Selection Systematic differences over conditions in respondent characteristics that could also cause the observed effect. History Maturation Events occurring concurrently with treatment could cause the observed effect (refers to all events that occur between the beginning of treatment and the post-test). Naturally occurring changes over time could be confused with a treatment effect. Application to the Current Study Hypotheses focus either on change between participants’ observed responses before and after participation, or on differences in responses measured after participation. In both cases, the temporal precedence of the experimental condition is clear. Participants were randomly assigned to participation condition to ensure that differences between respondents would not be systematic. Additionally, the use of a pre/post test design allows for systematic differences between participants at baseline to be examined, and accounted for if necessary. Participants at each individual school were involved in data collection concurrently and all data was collected within three days. Therefore, no history threats to internal validity were anticipated. Participants completed pre and post assessments immediately before and after their involvement, which lasted about 90 minutes. Maturation effects are unlikely. 20 Table 2. Cont. Regression Attrition Testing When units are selected for their extreme scores, they will often have less extreme scores on other variables, an occurrence that can be confused with treatment effect. Loss of respondents to treatment or to measurement can produce artefactual effects if that loss is systematically correlated with conditions. Exposure to a test can affect scores on subsequent exposures to that test, an occurrence that can be confused with a treatment effect. Instrumentation The nature of a measure may change over time or conditions in a way that could be confused with treatment effect. The impact of a threat can be added to that of another threat or may depend on the level of another threat. Additive and interactive effects of threats to internal validity Participation in the study was dependent on student availability and willingness, rather than scores on any outcome measure. Therefore, no regression threats to internal validity were anticipated. All data was collected in one two-hour period, with no attrition between pre and post assessments. Pre and post assessments were administered to all participants in the same manner, such that any testing effects should be experienced equally across treatment conditions. Pre and post assessments were administered to all participants within a relatively short time frame (90 minutes), and thus no change in participant interaction with the measures was anticipated. Therefore, no instrumentation threats to internal validity were anticipated. Since all other threats to internal validity were addressed or do not apply to the current study, no additive or interactive threats to internal validity were anticipated. Facilitator training. All facilitators received a 1.5-hour training prior to implementing their evaluation approach within the schools. The training included the following components: an 21 overview on adolescent development and engaging with adolescents, an overview of the project and evaluation goals, and an overview on conducting focus groups and important skills and techniques (see Appendix J for the detailed Training Protocol). Facilitation teams then divided into separate rooms to discuss the details of their assigned protocol. The designers of each protocol led an in-depth discussion of the protocol components and allotted time for questions and open discussion to ensure that research staff understood and were prepared to implement the protocol. Implementation. Both evaluation approaches were delivered concurrently at four schools serving grades seven through twelve within the district during the Summer of 2017. Youth who signed the consent form were randomly assigned to participate in one of the evaluation approaches during regularly occurring summer school programming hours. Both evaluation approaches were led by two co-facilitators and were audio recorded. Participation in the research procedures took approximately 1.5 hours to complete. After completing the evaluation activities, facilitators thanked the youth for their time, distributed $10 Walmart gift cards to all youth, and dismissed them to the Community School staff. Data collection. Research staff asked youth to complete evaluation assessments before and after involvement in the evaluation approaches (see Appendix B for the list of measures). The pre-assessment included the following measures: Background Information, Motivation to Influence (Ozer & Schotland, 2011), Perceived Control (Ozer & Schotland, 2011), Policy Control (Peterson et al., 2010), Critical Social Reflection, and Attitudes Toward School (McCoach & Siegle, 2003). The post-assessment included the following measures: Motivation to Influence (Ozer & Schotland, 2011), Perceived Control (Ozer & Schotland, 2011), Policy Control (Peterson et al., 2010), Critical Social Reflection items, Attitudes Toward School 22 (McCoach & Siegle, 2003), Youth-Adult Partnership Scale (Zeldin et al., 2014), and Satisfaction and Acceptability items. Research staff explained and read aloud all items for youth to complete individually. The pre- and post-assessments both took participants approximately 10 minutes to complete. After all youth were dismissed, the facilitators of the pluralistic participation condition completed the Implementation Fidelity Checklist (Appendix C). 23 Data Quality RESULTS Prior to data analyses, a random subsample of 25% of the surveys was used to verify data quality by comparing paper evaluation forms against the electronic data file. No data errors were found. Preliminary Analyses Descriptive statistics. Table 3 summarizes means and standard deviations on study variables for both symbolic and pluralistic participation conditions. Participants in the pluralistic condition had a significantly higher average GPA than participants in the symbolic condition (F (1, 10) =, p = .04), and GPA was found to be significantly correlated with two dependent variables at pre-assessment, Motivation to Influence (r = .68, p = .02) and Critical Social Reflection (r = .63, p = .03). In light of these findings, we considered covarying GPA in subsequent analyses to control for potential confounding effects. However, because only 41% of participants provided their GPA, it is unclear whether this observed relation with group type corresponds to an actual relation between the variables. Moreover, covarying GPA would eliminate participants who did report this variable, dramatically reducing sample size. Thus, GPA was not included as a covariate in subsequent analyses. No other significant group differences were found. 24 Table 3. Means, Standard Deviations, and Analysis of Variance (ANOVA) Results for Dependent Variables as a Function of Time and Group Variable Youth Voice in Decision Making Supportive Adult Relationships Satisfaction Acceptability Motivation to Influence Attitudes Toward School Perceived Control Policy Control Critical Social Reflection Note. * p < .05. Symbolic Condition (n = 16) Pre-test M (SD) --- --- --- --- 3.75 (.72) 2.65 (.88) 2.58 (.96) 3.49 (.53) 3.61 (.63) Post-test M (SD) 3.72 (.70) 3.83 (.94) 4.51 (.57) 4.37 (.57) 3.84 (.93) 2.59 (1.06) 2.98 (.77) 3.81 (.49) 3.53 (.85) Pluralistic Condition (n = 13) ANOVA F Pre-test M (SD) --- --- --- --- 4.10 (.61) 2.75 (.85) 2.93 (.54) 3.48 (.46) 3.68 (.65) Post-test M (SD) 3.81 (.58) 3.93 (.44) 4.57 (.43) 4.05 (.57) 4.06 (.68) 2.46 (1.22) 3.16 (.47) 3.64 (.39) 3.83 (.68) Time (T) --- --- --- --- .43 .59 2.80 2.04 .08 Group (G) .17 .00 .70 .85 .99 .00 .00 1.02 .59 T × G --- --- --- --- .25 .58 .14 .04 .23 25 Implementation fidelity. Table 4 summarizes implementation fidelity results. Facilitators of both the symbolic and pluralistic conditions used all required materials in implementing their respective protocols. Active process are the theoretically-linked differences in participation types and thus conceptualized as underlying hypothesized differences between conditions. While the pluralistic condition facilitators completed a relatively higher proportion of active processes (98.5%) than did the symbolic condition facilitators (95%), facilitators in both conditions completed a high rate of such processes. Debrief processes involve facilitator-led discussion and reflection of the group experience at the completion thereof. Though important for participant awareness of the intent and value of their experience, these processes are not conceptualized as underlying any hypothesized differences between conditions. The pluralistic condition facilitators completed a much higher proportion of debrief processes (91.7%) than did the symbolic condition facilitators (33.3%). Table 4. Summary of Mean Implementation Fidelity Data Variable Materials % of materials utilized Protocol Components % of active processes completed % of debrief process completed Symbolic Condition Pluralistic Condition 100% 100% 95% 33.3% 98.5% 91.7% Main Analyses Analytic strategy. The goal of this study was to examine whether pluralistic and symbolic participation had differential effects on youth within educational settings. To achieve this goal, nested (hierarchical) design analyses of variance (ANOVA) were employed to 26 determine whether the participation condition (pluralistic vs. symbolic) accounted for significant variation in participant’s scores. Nested design ANOVAs were necessary as participants in this study were nested within schools (see Figure 3), and such designs examine the interaction of time and participation condition (pluralistic vs. symbolic) over and above the effect of school. In these analyses, time operated as a within-subjects independent variable. Figure 3. Study participants organized within a nested design. The first step in a nested design ANOVA is to examine if there is systematic variation in the dependent variables associated with a nesting factor, in this case school building. Results of this first step identified systematic variation by school for most of this study’s outcome variables (see Table 5). For this reason, all analyses were run as nested ANOVAs. As a robustness check, analyses for those variables that did not require a nested design were rerun as traditional ANOVAs. Because the results of traditional ANOVAs did not differ significantly from nested results, and in order to provide consistent estimates across variables, only the nested ANOVA results are reported. 27 Table 5. Analysis of Variance Results for Between Subjects Effects Variable and Source Youth Voice in Decision Making School × Group Error Supportive Adult Relationships School × Group Error Satisfaction School × Group Error Acceptability School × Group Error Motivation to Influence School × Group Error Attitudes Toward School School × Group Error Perceived Control School × Group Error Policy Control School × Group Error Critical Social Reflection School × Group Error df 6 21 6 21 6 21 6 21 6 21 6 21 6 21 6 21 6 21 SS 5.67 5.71 8.30 7.32 1.37 5.73 2.44 6.35 12.04 15.69 14.18 35.34 8.68 11.01 1.33 5.51 9.95 9.77 MS .95 .27 1.38 0.34 .23 .27 .40 .30 2.00 .75 2.36 1.68 1.45 .52 .22 .26 1.66 .47 F 3.48* 3.97** .84 1.34 2.69* 1.40 2.76* .85 3.56* Note. * p < .05, ** p < .00 Hypothesis one (manipulation check). The first hypothesis guiding my analyses of these data was that, compared to youth in the symbolic condition, youth in the pluralistic participation condition would report higher youth voice in decision making and supportive adult relationships. As summarized in Table 3, while youth in the pluralistic condition reported higher youth voice in decision making (M = 3.81) and higher supportive adult relationships (M = 3.93) 28 than youth in the symbolic condition (M = 3.72; M = 3.83, respectively), these differences were not statistically significant (F(1, 6) = .17, p = .69; F(1, 6) = .00, p = .97, respectively). Therefore, hypothesis one was not supported in the current sample. Hypothesis two. The second hypothesis was that youth in the pluralistic condition would report higher levels of satisfaction and acceptability than youth in the symbolic condition. As summarized in Table 3, while youth in the pluralistic condition reported slightly higher satisfaction (M = 4.57) and lower acceptability (M = 4.05) than youth in the symbolic condition (M = 4.51; M = 4.37, respectively), these differences were not statistically significant (F(1, 6) = .70, p = .44; F(1, 6) = .85, p = .39, respectively). Hypothesis two was not supported. Hypothesis three. Finally, my third hypothesis was that youth in the pluralistic condition would experience greater pre-to-post increases in (a) school attitudes, (b) perceived control, and (c) critical social reflection than youth in the symbolic condition. School attitudes. School attitudes were assessed using measures of motivation to influence and attitudes toward school. As summarized in Table 3, while youth in the symbolic condition reported slightly higher motivation to influence at post-assessment (M = 3.84) than pre-assessment (M = 3.75), the reverse was true for youth pluralistic condition (M = 4.06; M = 4.10, respectively). However, the interaction between time and group type was not statistically significant, F(1, 6) = .25, p = .62. With regards to attitudes toward school, youth in both the symbolic and pluralistic conditions reported slightly higher average scores at pre-assessment (M = 2.65; M = 2.75, respectively) than post-assessment (M = 2.59; M = 2.46, respectively). However, the interaction between time and group was again not significant, F(1, 6) = .58, p = .45. These results indicate that there were no significant differences in change on school attitudes between the two groups. 29 Perceived control. Perceived control was assessed using measures of perceived control and policy control. As summarized in Table 3, youth in both the symbolic and pluralistic conditions reported higher perceived control at post-assessment (M = 2.98, M = 3.16, respectively) than pre-assessment (M = 2.58, M = 2.93, respectively). However, the interaction between time and group type was not significant, F(1, 6) = .14, p = .71. Similarly, youth in both the symbolic and pluralistic conditions reported higher policy control at post-assessment (M = 3.81, M = 3.64, respectively) than pre-assessment (M = 3.49, M = 3.48, respectively). However, the interaction between time and group was again not significant, F(1, 6) = .04, p = .84. These results indicate that there were no significant differences in change on perceived control between the two groups. Critical social reflection. As summarized in Table 3, while youth in the symbolic condition reported slightly lower critical social reflection at post-assessment (M = 3.53) than pre- assessment (M = 3.61), the reverse is true for youth in the pluralistic condition (M = 3.83, M = 3.68, respectively). However, the interaction between time and group type for the items assessing critical social reflection was not significant, F(1, 6) = .23, p = .63. These results indicate that there were no significant differences in change on critical social reflection between the two groups. 30 DISCUSSION Conceptual models underlying youth participatory research tend to incorporate an assumption that shared youth-adult control in research facilitates positive youth development (e.g., Wong et al., 2012). However, studies examining this assumption within educational settings have collected retrospective data on youth’s—or other stakeholders’—perceptions that youth have changed as a result of involvement in participatory research (e.g., Mitra, 2004). While the results of the sole experimental in this area did support the hypothesis that youths’ involvement in participatory research impacts several youth development outcomes (i.e., Ozer & Douglas, 2013), this study did not contrast varying types of participation. I sought to contribute to the limited research on the effects of involvement in participatory research within an educational setting on youths’ positive development, by conducting a controlled study examining the effects of different levels of participation. I tested three hypotheses in the current study. First, that youth in a pluralistic participation condition would report higher youth voice in decision-making and supportive adult relationships than youth in a symbolic condition. Second, that youth in a pluralistic participation condition would report higher levels of satisfaction and acceptability than youth in a symbolic participation condition. Third, that youth in a pluralistic participation condition would experience greater pre- to-post increases in (a) school attitudes, (b) perceived control, and (c) critical social reflection than youth in a symbolic participation condition. Although most observed means followed the patterns predicted in this study’s hypotheses, there were no statistically significant differences between groups. In this section, I discuss four factors that may explain these findings. Three of these would suggest that—rather than reflecting the true relation between involvement in participatory research and youth 31 outcomes—current findings are artifacts of design limitations. The fourth is the possibility that findings reflect the true relation between these variables. Limitations Limited statistical power. Statistical power is the probability of obtaining a statistically significant result in research (Cohen, 1992). When statistical power is high, the probability of committing a Type II error—failing to reject a null hypothesis that is false—is low. Statistical power is impacted by three study characteristics: significance criterion (α), sample size (N), and effect size (r) (Cohen, 1992). In general, larger effect sizes and simpler designs require less participants to be adequately powered. One possible explanation for current results is that the current study was not sufficiently powered to identify real differences between groups. Most observed means followed the patterns predicted in this study’s hypotheses; however, none of the differences between groups reached statistical significance. Because of their added complexity, nested designs—such as that used in this study—require larger samples to be sufficiently powered (Scherbaum & Ferreter, 2009). In this current study, the observed power was very low for all analyses. For instance, while convention has established a power value of .80 as acceptable (Cohen, 1992), the observed power in the analyses testing the time by group interaction effect for Motivation to Influence was .08. Accordingly, it may be that the current study was critically underpowered to detect any actual effects, particularly if these were small. The possibility that the current study was critically underpowered to detect an actual effect size is indirectly supported by the findings from the only previous experimental test of the effects of participation on youth (i.e., Ozer & Douglas, 2012). While the effect sizes observed in that study were small, its large sample size was sufficiently powered to establish their statistical 32 significance (i.e., N = 401). Future studies should ensure adequate power is obtained to adequately examine the relationship between participation in research within educational contexts and youth outcomes. Manipulation failure. To ensure the internal rigor of this study, I implemented a randomized, active control, double-blind design where an expert who was unaware of study hypotheses designed the active control condition. These design features may have unintendedly minimized the differences between experimental conditions. The active control condition incorporated collective rule setting, age-appropriate active facilitation, and in-vivo recording of visible summary notes using large poster-sized sheets. Although these components are helpful for engaging youth participants, they also reduced the differences between conditions. Accordingly, while there may be differential effects on youth outcomes across types of participation, the two conditions in this study may have—unintentionally—not differed enough to reflect different types of participation. This possibility is supported by the current manipulation check results (i.e., hypothesis 1), which showed relatively high levels of youth voice in decision making and supportive adult relationships in both conditions. Future research should ensure clearer distinctions across experimental conditions. Misspecified outcomes. Existing conceptual models broadly suggest that involvement in participatory research positively impacts youth outcomes (e.g., Wong et al., 2010). While these suggestions are supported by descriptive empirical studies within educational contexts (e.g., Mitra, 2004), only one peer-reviewed study has experimentally tested this impact or clarified the specific mechanisms by which it occurs (i.e., Ozer & Douglas, 2012). Guided by the limited scholarship in this area, we hypothesized that youth participation in research within educational 33 settings would impact three developmental domains among youth: school attitudes, perceived control, and critical social reflection. It is possible that our failure to find significant results is driven by our incorrect specification of the outcomes impacted by participatory research involvement. This suggestion is supported by results from a recent unpublished study in which Rapa (2016) examined the effects of a brief exposure intervention on youth outcomes including critical reflection and critical motivation. The only significant changes observed in Rapa’s study were on critical motivation, a component of critical consciousness that has been infrequently studied and that was not included in the present study. It may, therefore, be that the effect of brief interventions—such as those examined by Rapa and in the current study—is limited to critical motivation. Future research is needed to further examine the effects of brief exposure interventions—such as participation in research—on youth’s critical motivation. Additional limitations. There are additional limitations to this study that should be accounted for when considering its results. First, the measures used in this study may not be sensitive enough to detect immediate effects. Future research could consider utilizing or developing measures that are more sensitive to immediate changes within youth. Second, some of the measures included in the current study obtained relatively low alphas, which is commonly an index of internal consistency in measurement (Crocker & Algina, 1986). Future research could use alternative measures that may be better suited to document similar outcomes. Third, anecdotal evidence from the current study raises the possibility that facilitator–student fit may have made an impact on the results of the study. For any given group, the facilitator’s fit with the students assigned to their group, rather than the approach used by that facilitator, may have 34 influenced outcomes. Future research could test this possibility by including an assessment of facilitator-student fit to determine its relationship to youth outcomes. Fourth, many studies examining participatory research with youth often includes an action component (i.e., Youth Participatory Action Research). However, in the current study, the pluralistic condition does not contain an action component. Since statistically significant differences between the two conditions were not found in the current study, it is possible that action is a critical factor for shared youth-adult control in participatory research, to produce observable changes in youth. Future research could test this possibility by including an action component within shared youth-adult control conditions. Finally, participant characteristics may have impacted study findings. Youth in the current study lived in an economically struggling city, and attended a district facing many long-term infrastructure challenges. It is possible that youth within these conditions may be more difficult to change on the variables assessed in the current study, thus impacting the ability to find significant effects. Future research could consider testing the hypotheses of the current study with many different types of y No Actual Effects As described in the previous subsection, the scholarship suggesting that youth outcomes can changes as a result of involvement in participatory research is largely theoretical and descriptive (e.g., Wong et al., 2010, Mitra, 2004). The only experimental study in this area found only statistically small effects of uncertain practical significance (Ozer & Douglas, 2013). It is possible that our failure to support any hypotheses in the current study—rather than being an artifact of design limitations—accurately reflects no meaningful effect of involvement in participatory research on youth outcomes. Given the other limitations of the study, adequately powered, clearly contrasted studies, with correctly targeted outcomes, are needed to truly 35 understand the relationship between participation in educational research and youth effects. Conclusion Despite a careful effort to conduct a randomized and controlled experimental study, the design of the current study reflected many limitations that could have impacted its findings. Three of the previously discussed limitations are critically important to the overall non- significant findings. First, far fewer participants were recruited for the study than initially expected. This critically reduced statistical power and likelihood to identify any actual effects. Second, the blind expert who designed the active control condition incorporated several features that likely reduced its differences with the experimental condition. Finally, a brief, one-time intervention may be insufficient to produce significant changes on youth outcomes, particularly among youth who live in economically-distressed environments. In these contexts, producing changing may require more significant, longer-duration interventions. Despite its limitations, the current study provides insight into the effects of youth involvement in participatory research within educational settings. The unexpected null findings challenge assumptions currently present within the literature to suggest that youth involved in participatory research are measurably impacted on certain variables. These findings point to important directions for future research to address the limitations of the current study. Such research should be adequately powered, assess variables that may be more impacted by youths’ participation, test conditions with clearer distinctions, include an action component to promote share youth-adult control, and involve multiple different types of youth. The insights from the current study aims to foster the advancement of research approaches within educational settings that empowers and positively impacts youth. 36 APPENDICES 37 APPENDIX A Community Schools Evaluation Approach Purpose of Youth Participation To understand youths’ perspectives on the Community Schools Approach implemented within their school, youth were asked to participate in an evaluation of the approach. The goals of this evaluation are as follows: • To gather youth’s understanding and perceptions of the Community Schools Approach • To gather youth’s perceptions of the effects of the Community Schools Approach • To gather youth’s suggestions for improving the Community Schools Approach Prompts for Youth Participation: Youth GO (deidentified) 1. What is [Community Schools Approach]? 2. What do you like about [Community Schools Approach]? 3. How has [Community Schools Approach] helped you to be successful in school? 4. How has [Community Schools Approach] helped you be successful outside school? 5. How could [Community Schools Approach] be better? Prompts for Youth Participation: Focus Group (deidentified) 1. When you hear [Community Schools Approach], what do you think that includes? 2. What are your favorite [Community Schools Approach] programs, and why? 3. What are your least favorite [Community Schools Approach] programs, and why? 4. If you could add or improve any [Community Schools Approach] service, which would it be and why? 38 Table 6. Measure Item Pool APPENDIX B Outcome Scale Items Pre Post 6 Yes No 4 Yes Yes 5 Yes Yes Demographics 1. Age 2. Grade 3. Gender 4. GPA 5. Adult caregiver education 6. Race or ethnic background Motivation to Influence One’s School or Community (Psychological Empowerment Scale, Ozer & Schotland, 2011) 7. It is important for youth to try to improve our city even if we can’t always make the changes we want. 8. I want to have as much say as possible in making decisions in my city. 9. I want to have as much say as possible in making decisions in my school. 10. Students should work to improve our school even if we can’t always make the changes we want. Attitudes Toward School (School Attitude Assessment Survey, McCoach & Siegle, 2003) 11. I am glad that I go to this school. 12. This is a good school. 13. This school is a good match for me. 14. I like this school. 15. I am proud of this school. 39 School Attitudes Table 6. Cont. Perceived Control Perceived Control (Psychological Empowerment Scale, Ozer & Schotland, 2011) 16. There is a student council here that gets to decide on some really important things. 17. There are plenty of ways for students like me to have a say in what our school does. 18. Students have a say in what happens at this school. 19. Students at this school get to help plan special activities and events. 20. There are plenty of ways for young people like 21. me to have a say in what our city government does. 22. Youth have a say in what happens in this city. Policy Control (Sociopolitical Control Scale for Youth, Peterson, Hamme Peterson, Agre, Christens, Morton, 2011) 23. I enjoy participation because I want to have as much say in my community or school as possible. 24. Youth like me can really understand what’s going on with my community or school. 25. I feel like I have a pretty good understanding of the important issues which confront my community or school. 26. Youth like me have the ability to participate effectively in community or school activities and decision making. 27. My opinion is important because it could someday make a difference in my community or school. 28. There are plenty of ways for youth like me to have a say in what our community or school does. 29. It is important to me that I actively participate in local teen issues. 30. Most community or school leaders would listen to me. 31. Many local activities are important to participate in. 6 Yes Yes 9 Yes Yes 40 Table 6. Cont. Original items, created for this study 32. I think a lot about whether social institutions (schools, communities) are fair to everyone. 33. Students should think more about whether Critical social reflection people with authority (e.g., principals, politicians, police) are fair to others. 5 Yes Yes 34. I notice that not everyone gets treated the same 35. I spend a lot of time wondering how laws and rules could be more fair. 36. In the United States, everyone has the same opportunities for success. Youth-adult partnerships scale (Zeldin, Krauss, Collura, Lucchesi, Sulaiman, 2014; Slightly modified) (Supportive Adult Relationships subscale) 37. Youth and adults trust each other in this group. 38. There is a good balance of power between youth and adults in this group. 39. Youth and adults learn a lot from working together in this group. 40. In this group, it is clear that youth and adults respect each other. 41. Adults learn a lot from youth in this group. (Youth Voice in Decision Making subscale) 42. I have a say in planning programs at this center. 43. Adults take my ideas seriously. 44. I am expected to voice my concerns when I have them. 45. In this group, I am encouraged to express my ideas and opinions. Items created using existing items from treatment satisfaction scales and some original items1 46. I enjoyed the activities of the group I was in today. 47. I liked the group I was in. 48. I liked participating in the activities today. 49. I’m glad I participated in the activities today. 50. I would participate in these activities again. 41 Youth-Adult Partnerships Youth Satisfaction 5 No Yes 5 No Yes Table 6. Cont. Youth Acceptability Items created using existing items from treatment satisfaction scales and some original items2 51. I would suggest this group to a friend or other people my age. 52. This group is appropriate for people my age. 53. The activities today discussed issues that were relevant to my life. 54. Other people my age would like the activities today. 55. The activities today would not cause problems for my classmates. 56. The activities would be appropriate for many different youth my age. 57. The activities I participated in were acceptable for people my age. 7 No Yes Total Pre-Test Items: 35 Total Post-Test Items: 50 1 Items 45 & 46 are from Cuidate evaluation form, Item 47 is modified from the Modified Children’s Intervention Rating Profile (Mitchell, et al., 2015), Items 48 & 49 are original items. 2 Items 50 – 52 are from Cuidate evaluation form, Items 53 & 54 are modified from the Modified Children’s Intervention Rating Profile (Mitchell, et al., 2015), Item 55 is modified from the Primary Intervention Rating Scale (Lane et al., 2009), Item 56 is an original item. 42 Appendix C Youth GO: Implementation Fidelity Checklist Facilitator(s): __________________________ Date: ______________ School: ______________ Please check if you used each of the materials:  Nametags  Flipchart paper  Post-it notes  Colored papers  Assorted candy Please check if you completed each of the following tasks for the steps of Youth GO Step One, Climate Setting  Facilitators introduced themselves  Youth introduces themselves  Facilitators explained the purpose and goals of the Youth GO session  Youth created a community agreement  Youth agreed to abide by the community agreement Step Two, Generating  Youth were presented each prompt individually  Youth could ask questions about the prompts  Youth recorded prompts individually on post-it notes  Youth placed responses on flip chart paper  Facilitators led a group discussion about the youths’ responses  Youth were presented all prompts assigned for this session Step Three, Organizing  Youth played the data organization game with candy  Youth grouped responses for individual prompts  Youth created names/themes for the groups Step Four, Selecting  Youth discussed or reviewed the themes created  Youth presented/proposed categories for the themes  Youth voted on the categories  Youth and facilitators cross-checked the categories with the themes Step Five, Debrief & Discussion  Facilitators reminded youth of the purpose and goals of the Youth GO session  Youth were given time to reflect on their experience  Youth were reminded of the value of their perspective If you did not check any of the materials or tasks, please explain why below: ______________________________________________________________________________ ______________________________________________________________________________ 43 Appendix D Focus Group: Implementation Fidelity Checklist Facilitator(s): _______________________ Date: ______________ School: _________________ Please check if the facilitator used each of the materials:  Nametags  Flipchart paper Please check if the facilitator completed each of the following protocol components Introductions  Facilitators introduced themselves  Youth introduces themselves  Facilitators explained the purpose and goals of the session Group Discussion  Facilitator asks base question: “When you hear “community education,” what do you think that includes?” o Facilitator uses follow up probes to guide the discussion about this base question  Facilitator asks base question: “What are your favorite community education programs and why?” o Facilitator uses follow up probes to guide the discussion about this base question  Facilitator asks base question: “What are your least favorite community education programs and why?” o Facilitator uses follow up probes to guide the discussion about this base question  Facilitator asks base question: “How have community education programs helped you?” o Facilitator uses follow up probes to guide the discussion about this base question  Facilitator asks base question: “If you could add or improve any community education service, which would it be and why?” o Facilitator uses follow up probes to guide the discussion about this base question Debrief & Discussion  Facilitator leads a brief discussion about how the discussion went  Youth were thanked for their participation  Facilitators reminded youth of the purpose of the group discussion 44 Appendix E Facilitator Questionnaire Please provide answers to the following questions, so that we may accurately document the group facilitators and learn about your experience as your involvement in the project. Demographics & Background Gender: _______________________________________ Age: __________________________ Race/Ethnicity: ________________________________________________________________ Level of formal education: _______________________________________________________ Experience & Competencies: Youth work Please describe any workshop or classroom-based training that you’ve had on working with middle and high school students (e.g., coursework, workshops, certifications). ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Please describe any practical training that you’ve had on working with middle and high school youth (e.g., on the job training, etc.)? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Please rate each of the following questions using the following scale: How knowledgeable do you feel about work with middle and high school students? __________ How comfortable do you feel working with middle and high school students? _______________ Experience & Competencies: Group Facilitation Please describe any workshop or classroom-based training that you’ve had on facilitating groups with middle and high school students (e.g., coursework, workshops, certifications)? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Please describe any practical training that you’ve had on facilitating groups with middle and high school youth (e.g., on the job training, etc.)? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 1: Not at all, 2: Barely, 3: Neutral, 4: Somewhat, 5: Very 45 1: Not at all, 2: Barely, 3: Neutral, 4: Somewhat, 5: Very Please rate each of the following questions using the following scale: How knowledgeable do you feel about facilitating groups with middle and high school students? _______ How comfortable do you feel facilitating groups with middle and high school students? _______ Experience & Competencies: Population Please describe any workshop or classroom-based training that you’ve had on working with underserved, low SES, and/or predominately African American middle and high school students (e.g., coursework, workshops, certifications)? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Please describe any practical training that you’ve had on working with underserved, low SES, and/or predominately African American middle and high school youth (e.g., on the job training, etc.)? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Please rate each of the following questions using the following scale: How knowledgeable do you feel about working with underserved, low SES, predominately African American middle and high school students? _______ How comfortable do you feel working with underserved, low SES, predominately African American middle and high school students? _______ For each question below, circle the answer that matches how much you agree or disagree with each statement: 1: Not at all, 2: Barely, 3: Neutral, 4: Somewhat, 5: Very SD = Strongly Disagree, D = Disagree, N = Neutral, A = Agree, SA = Strongly Agree 1. Youth and adults trusted each other in the groups I facilitated. _________ 2. There was a good balance of power between youth and adults in the groups I facilitated. _________ 3. Youth and adults learned a lot from working together in the groups I facilitated. _______ 4. In the groups I facilitated, it was clear that youth and adults respected each other. ______ 5. Adults learned a lot from youth in the groups I facilitated. _________ 6. Youth had a say in planning programs at their school. _________ 7. Adults took ideas from the youth seriously. _________ 8. Youth were expected to voice their concerns when they had them. ________ 9. In the groups I facilitated, youth were encouraged to express ideas and opinions. _______ 10. I enjoyed the activities of the group I facilitated. _______ 46 11. I liked the group I facilitated. _______ 12. I liked facilitating the group activities. _______ 13. I’m glad I facilitated the group activities. _______ 14. I would facilitate these activities again. _______ 15. I would suggest this group facilitation technique to a friend or colleague. _______ 16. This group facilitation technique is appropriate for my friends or colleagues. _______ 17. The activities discussed issues that were relevant to the participant’s lives. _______ 18. Other friends and colleagues would like the activities today. _______ 19. The activities today would not cause problems for my friends or colleagues. _______ 20. The activities would be appropriate for many different facilitators of many different skill levels. _______ 21. The activities I facilitated were acceptable for people with my skill level. _______ 47 Appendix F Flier to Recruit Youth Participants (Deidentified) Your child’s perspective is important. We want to hear it. Dear Parent/Guardian, In all [District Name], the [Community Schools Approach] provides students, parents, and community members with programming and support. To improve this initiative, we are asking children at [School Name] to participate in a listening session. During listening sessions, groups of children will be asked to answer some written questions about [Community Schools Approach] and also talk about their experiences with [Community Schools Approach]. What they share will be used to improve [Community Schools Approach] and will be kept confidential. Your child is being invited to participate in a listening session on [Date], which will happen after summer school activities. As a thank you for their time, all children that participate in this listening session will receive a $10 Walmart gift card. If you would like your child to participate, NO ACTION IS NEEDED. Your child will be invited to attend the session after regular summer school hours on [Date]. Any children invited to participate of course can choose not to. If you would NOT like your child to participate OR if you have questions about your child’s participation in these groups: Please contact Sara Stacy by email (stacysar@msu.edu) or phone (513-526-6132). Thank you, [Community Schools Approach Staff] 48 Appendix G Participant Informed Consent Form (Deidentified) 1. WHY ARE YOU RECEIVING THIS FORM? • You are invited to participate in a group discussion about your experiences with [Community Schools Approach]. • We also want to learn more about what types of group discussions students find motivating. • To help you decide whether to participate or not, this form explains what this group discussion involves, why we are doing it, any risks or benefits of participating, and who to contact if you have any questions. 2. EXPLANATION OF THE PROJECT and WHAT YOU WILL DO • You are being invited to a group discussion with about 12 other students about everybody’s experiences with [Community Schools Approach]. • You will also be asked complete a short questionnaire before and after the group discussion. The questions are about: o Your feelings about school o Whether you feel that your opinions matter o How interested in community issues you are o Your feelings about the discussion 3. IT IS ABSOLUTELY FINE FOR YOU TO SAY “NO” OR TO CHANGE YOUR MIND AT ANY POINT • Participation is voluntary. It is completely up to you whether to participate in this project. • Even if you decide participate, you can still not answer specific questions or stop • participating at any time. If you don’t’ want to participate or change your mind later, there will be NO penalty or consequence at school or with community education 4. ARE THERE ANY RISKS OR BENEFITS OF YOU PARTICIPATING? • RISKS: We can’t think of any major risks. You’ll be asked questions about Community Education, and Community Education staff will NOT know if you decided to participate or not (nor will they know what you said). • BENEFITS: While many students like participating in the types of group discussion that we are inviting you to, we can’t be sure that you’ll like it. 5. COSTS AND COMPENSATION FOR YOUR PARTICIPATION • Participating in the group discussion and completing the questionnaires will take you between 60 and 90 minutes (depending on your reading and learning style). • Students who participate in this project will receive a $10 dollar Walmart gift card in return for their efforts. 6. CONTACT INFORMATION FOR QUESTIONS AND CONCERNS If you have concerns or questions about this project, how to do any part of it, or to report an injury, please contact the person coordinating it: 49 Sara Stacy; 316 Physics Rd., East Lansing, MI 48824; stacysar@msu.edu; (517) 355- 9562 7. ASSENT TO PARTICIPATE If you accept our invitation, please sign your name below and write in the date. Signature Date 50 Appendix H Youth GO Protocol (Deidentified) Step 1: Climate Setting & Pre-Survey Time 15 Minutes Materials • • Nametags (x20) Pre-survey (x20) Process As students enter, have them complete a nametag. INTRODUCTIONS Once all students are present, the facilitator/s introduces them self to the group: • E.g.: Students from MSU, interested in your perspectives on [Community School Approach]. The facilitator/s then discuss the purpose, goals, and time commitment for the focus group: • Purpose of the focus group: To gather the youth’s perspectives on [Community School Approach]. This information will be used to impact the future of [Community School Approach]. • Goals of the focus group: To discuss their perspectives on [Community School Approach] and to organize them in a meaningful way. • Time commitment: The activities today should take about an hour and a half. CONSENT & PRE-SURVEY (Implemented by research staff, not facilitators) Informed consent • • Pre-survey: • The pre-survey includes questions about your beliefs and experiences. • Please answer these questions as honestly as you can. • These questions will not be reviewed by [Community School Approach] or school staff. • They will not affect your ability to participate in [Community School Approach] or school activities. • Your information and answers that you give will remain anonymous. • You can work independently or follow along as the questions are read aloud. • Distribute pre-survey, read aloud all items, and assist with any questions or needs. RANDOM ASSIGNMENT • Once youth have completed their pre/survey, distribute envelopes for assignment. • Youth open their envelope and are assigned to participate in one of the evaluation approaches. 51 COMMUNITY AGREEMENT After the surveys are complete, the facilitator/s will engage with youth to create a community agreement/rules for participation. Once the community agreement is complete, have the youth agree to those rules, then move on to the next step. Step 2: Generating Time 30 Minutes Materials • • • Flip chart paper with pre-prepared questions Pens Post-it notes Process Facilitator/s introduces the first activity: • We are going to be discussing a few questions that I have prepared today that are posted around the room. • First, I will present the question and then you can write a response to the question on sticky notes. • Write as much detail as you can, and if you need any help, just ask! This is about YOUR opinion and there are no wrong answers. Facilitator presents the questions to the youth one at a time and allows time for youth to respond (using sticky notes), discuss the question, and add in any additional responses. (Allow about 4 minutes for each question.) Questions include: 1. What is [Community School Approach]? a. After this question is discussed, facilitators ensure that everyone has a clear understanding of [Community School Approach] (e.g., who they are, what they do) 2. What do you like about [Community School Approach]? a. Examples/probes: specific programs, relationships with [Community School Approach] staff 3. How has [Community School Approach] helped you to be successful in school? 4. How has [Community School Approach] helped you be successful outside school? 5. How could [Community School Approach] be better? Use the following probes to prompt discussion: • Does anyone disagree or has anyone had a different experience? • Why do you feel that way? • Can you talk about that more? • Does anyone else have something they want to add here? • Did we miss anything? 52 Once all questions have been processed, move to the next step. Step 3: Organizing Time 20 Minutes Materials • Small bags of assorted candy • Colored sheets of paper (6 per group) • Pens Process CANDY GAME Facilitator introduces the next activity: • Now we are going to play a game. • Once we have learned how to play the game, we will apply it to the questions we just discussed. Facilitator/s explain the game: • Small group facilitators explain the game: Imagine that your team owns a new store that has a small inventory of candy. Your team buys four bins to organize the candy for the customers and must come up with a name for each bin. The names must be clear enough so that customers who can’t see the candy still know what type of candy is inside each bin. Facilitators distribute small bags of assorted candy and colored paper for categorizing the candy and let the youth work on the task, helping only when needed. Facilitators give the youth a new task: • Now imagine that two of your bins broke. Organize the candy again, but using only 2 bins and come up with a name for each bin. The names must still be clear enough so that customers who can’t see the candy know what type of candy is inside each bin. Facilitators give the youth 2 new sheets of paper to represent the bins and let the youth work on the new task, helping only when needed. DATA ORGANIZATION: THEMES Once complete with the candy sorting game, facilitators describe the next task: • Now we are going to take what we just learned about how to create groups with candy and apply it to our answers that everyone gave to the questions we just discussed. • We are going to organize the responses into meaningful groups, and create names for the groups, which are called “themes” 53 The youth are given each flip chart containing the questions and responses and sheets of colored paper, one at a time. Youth organize the responses for each question into meaningful themes. Facilitators help only when needed. Step 4: Selecting Time 15 Minutes Materials No new materials needed Process DATA ORGANIZATION: CATEGORIES Facilitator describes next activity: • You just worked to group the question responses, which we can also call “themes.” • Now we are going to create big groups for all of the questions and responses. This will help us to determine what we think is most important to describe all of the questions and responses we discussed today. These groups will be called “categories.” Facilitator leads a group discussion to determine the categories. Allow the youth to present suggestions and have the group to come to a consensus using thumbs up/thumbs down process. If youth find this task challenging, use the following prompts to guide the group discussion: • What is the most important thing we discussed today? • Can you group any of these themes together? • What would be a good name for these similar responses? • What themes are the most important to you? • It sounds like there was a lot of discussion about _____today. Is this important to include? Once the categories are selected, the facilitator leads a cross checking process to make sure that the categories align with at least one theme and that all themes are included within the categories. Step 5: Debrief, Discussion, & Post-Survey Time 10 Minutes Materials No new materials needed Process Facilitator leads a brief discussion about how the activities went. 54 Facilitator debriefs youth on the activities they participated in: • Thank you so much for participating in the activities today to discuss Community Education & Mindfulness. • The discussion and information provided today will be directly used to understand and improve programs at your school. • We really appreciate your thoughtfulness and engagement during the activities today and the time you committed to being here. We could not do this work without you! POST-SURVEY (Implemented by research staff, not facilitators) • Now, we would like to ask you to complete a brief survey, that is very similar to the one you completed at the beginning of our time today. • Again, the survey includes questions about your beliefs and experiences. • Please answer these questions as honestly as you can. • These questions will not be reviewed by [Community Schools Approach] or school staff. • They will not affect your ability to participate in programs or school activities. • Your information and answers that you give will remain anonymous. • You can work independently or follow along as the questions are read aloud. • Distribute pre-survey, read aloud all items, and assist with any questions or needs. • Distribute gift cards once youth have completed the post-survey. Youth can leave once they have finished. 55 Appendix I Focus Group Protocol (Deidentified) Step 1: Introduction Time 20 Minutes Materials • • Nametags (x20) Pre-survey (x20) Process As students enter, have them complete a nametag. INTRODUCTIONS Once all students are present, the facilitator/s introduces them self to the group: • E.g.: Students from MSU, interested in your perspectives on [Community School Approach]. The facilitator/s then discuss the purpose, goals, and time commitment for the focus group: • Purpose of the focus group: To gather the youth’s perspectives on [Community School Approach]. This information will be used to impact the future of [Community School Approach]. • Time commitment: The activities today should take about an hour and a half. CONSENT & PRE-SURVEY (Implemented by research staff, not facilitators) Informed consent • • Pre-survey: • The pre-survey includes questions about your beliefs and experiences. • Please answer these questions as honestly as you can. • These questions will not be reviewed by [Community School Approach] or school staff. • They will not affect your ability to participate in [Community School Approach] or school activities. • Your information and answers that you give will remain anonymous. • You can work independently or follow along as the questions are read aloud. • Distribute pre-survey, read aloud all items, and assist with any questions or needs. RANDOM ASSIGNMENT • Once youth have completed their pre/survey, distribute envelopes for assignment. • Youth open their envelope and are assigned to participate in one of the evaluation approaches. 56 Step 2: Group Discussion Time 60 Minutes Materials • • Flipchart paper Markers Process Facilitator introduces the topic: • As I mentioned, we are going to be discussing a few of the programs at your school: [Community Schools Approach]. We want to get your opinion on these programs so that we can understand how they are going and get a sense of how we can make them better. Facilitator presents questions one at a time to the youth and allows time for group discussion. The facilitator will use flip charts or white board as necessary to document the group discussions. The base questions are (record on flip chart): a. When you hear [Community Schools Approach] what do you think that includes? b. What are your favorite [Community Schools Approach] programs, and why? c. What are you least favorite [Community Schools Approach] programs, and why? d. How have [Community Schools Approach] programs helped you? e. If you could add or improve any [Community Schools Approach] service, which would it be and why? The facilitator will probes to guide the discussion, such as: • Does anyone disagree or has anyone had a different experience? • Why do you feel that way? • Can you talk about that more? • Does anyone else have something they want to add here? • Did we miss anything? Step 3: Debrief, Discussion, & Post-Survey Time 20 Minutes Materials No new materials needed Process Facilitator leads a brief discussion about how the discussion went. 57 Facilitator debriefs youth on the activities they participated in: • Thank you so much for participating in the discussion today about [Community Schools Approach]. • The discussion and information provided today will be directly used to understand and improve these programs at your school. • We really appreciate your thoughtfulness and engagement during the activities today and the time you committed to being here. We could not do this work without you! POST-SURVEY (Implemented by research staff, not facilitators) • Now, we would like to ask you to complete a brief survey, that is very similar to the one you completed at the beginning of our time today. • Again, the survey includes questions about your beliefs and experiences. • Please answer these questions as honestly as you can. • These questions will not be reviewed by [Community Schools Approach] or school staff. • They will not affect your ability to participate in programs or school activities. • Your information and answers that you give will remain anonymous. • You can work independently or follow along as the questions are read aloud. • Distribute pre-survey, read aloud all items, and assist with any questions or needs. • Distribute gift cards once youth have completed the post-survey. Youth can leave once they have finished. 58 Appendix J Facilitator Training (Deidentified) A broad outline of the “Conducting Focus Groups with Youth” Training Protocol is presented in Table 1. A detailed description of each component follows. Table 1. Outline for Research Staff Training: Conducting Focus Groups with Youth I. Introduction a. Goals and Objectives of the Training b. Outline of the Training II. Overview on Adolescent Development & Engaging with Adolescents III. Project Overview: Evaluating the [Community School Approach] a. Context b. Introduction to [Community School Approach] c. Evaluation Project Goals and Objectives IV. Conducting Focus Groups a. Overview b. Youth consent c. Skills and techniques IV. Protocol-Specific Focus Group Training a. Review of Assigned Protocol b. Role play & Feedback Component I, Introduction. During this component, attendees will be provided with the goals and objectives of the training and a detailed outline of the training. The overall goal of the training is to provide attendees with the information and skills necessary to understand their role in conducting focus groups with youth. The specific objectives of the training are: 1. To provide attendees with an overview on adolescent development and procedures for engaging with adolescents. 2. To provide attendees with an overview of the [Community School Approach], including the project context, project goals and objectives, and detailed descriptions of the project components. 59 3. To provide training and orientation conducting youth focus groups, including an overview, youth consent process, and skills and techniques for conducting focus groups. 4. To review the specific protocols assigned to research staff, conduct role play scenarios, and provide research staff with practical feedback. Component II, Overview of Adolescent Development and Engaging with Adolescents. • Cognitive, emotional, and physical development of adolescents and its relation to brain development o Engagement tip: Abandon expectations • Importance of relationships in adolescents o Engagement tip: Be authentic and comfortable • Importance of language o Engagement tip: Use language they can understand • Adolescent management tips Component III, Project Overview. • Context o City [in Midwest] o [School District]: 4,800 students in district, declining about 50% in the last 5 years, transient students and families o Project funders • Introduction to [Community School Approach] o Implemented to bring services and resources into schools to make them more accessible to students, parents, and community members ▪ Utilizes community partnerships to bring services and programming into the schools o Implemented in all school buildings with a [Community School Approach] Director and (5) AmeriCorp Service Members placed in each building o Targeting four goals: increased attendance, improved third-grade reading levels across the schools, increased graduation rates, and neighborhood development. • Evaluation Project Goals & Objectives o Overall project goal: To document the student and school-level effects of the [Community School Approach] o Currently comprised of the following components: ▪ Current Implementation Assessment: to document the existing implementation of [Community School Approach] and how it maps on to programs across the nation • Background & program structure 60 • Logic model & its relation to other integrated school services • Program partners (79), hours of AmeriCorp service (over 57,000), hours of OST programming (over 9,000). • Finances and support needed: $4 mil annual budget, 9 admin staff, 11 CSDs, 55 Americorps • Comparing [Community School Approach] to the national and international literature ▪ Focused Formative Evaluation • Principal interviews: Barriers and facilitators of the implementation of [Community School Approach] • Youth focus groups: Experiences with [Community School Approach]—This is the project you will be assisting on! Component IV. Conducting Youth Focus Groups. • Overview • Youth consent o Let the youth know what they are about to participate in o They may stop at any time • Study skills & techniques o Using audio recorders 101 o Facilitation tips & tricks Component V. Protocol-Specific Focus Group Training. • Research staff split into groups based on assigned protocol. • Trainers give full review of the research protocol with Q&A throughout • Role play: Research staff practice conducting focus group protocol with one another • Trainers and trainees provide detailed feedback via reverse seniority 61 Form for Written Confirmation of Training (Used for the training of research personnel in Protocol Reference #: x17-749e “Youth Participation Project”) Research Staff Training: Conducting Focus Groups with Youth I. Introduction a. Goals and Objectives of the Training b. Outline of the Training II. Overview on Adolescent Development & Engaging with Adolescents III. Project Overview: Evaluating the [Community School Approach] a. Context b. Introduction to [Community School Approach] c. Evaluation Project Goals and Objectives IV. Conducting Focus Groups a. Overview b. Youth consent c. Skills and techniques IV. Protocol-Specific Focus Group Training a. Review of Assigned Protocol b. Role play & Feedback By my signature below, I acknowledge having completed the training outlined in this document on ________________________. (Date) Name: _______________________________________________________________________ Signature: _____________________________________________________________________ 62 REFERENCES 63 REFERENCES Berg, M., Coman, E., & Schensul, J. J. (2009). Youth action research for prevention: A multi- level intervention designed to increase efficacy and empowerment among urban youth. American Journal of Community Psychology, 43(3–4), 345–359. http://doi.org/10.1007/s10464-009-9231-2 Cammarota, J., & Fine, M. (2008). Youth Participatory Action Research: A Pedagogy for Transformational Resistance. Revolutionizing Education: Youth Participatory Action Research in Motion, 1–11. Carlson, E. D., Engebretson, J., & Chamberlain, R. (2006). Photovoice as a Social Process of Critical Consciousness. Qualitative Health Research, 16(6), 836–852. http://doi.org/10.1177/1049732306287525 Checkoway, B., Allison, T., & Montoya, C. (2005). Youth participation in public policy at the municipal level. Children and Youth Services Review, 27(10), 1149–1162. http://doi.org/10.1016/j.childyouth.2005.01.001 Chen, P. Y., Weiss, F. L., Johnston Nicholson, H., & Girls Incorporated. (2010). Girls study girls Inc.: Engaging girls in evaluation through participatory action research. American Journal of Community Psychology, 46(1), 228–237. http://doi.org/10.1007/s10464-010- 9328-7 Cohen, J. (1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101. Creswell, J. W., Hanson, W. E., Plano Clark, V. L., & Morales, A. (2007). Qualitative Research Designs: Selection and Implementation. The Counseling Psychologist, 35(2), 236–264. http://doi.org/10.1177/0011000006287390 Dettori, J. (2010). The random allocation process: Two things you need to know. Evidence- Based Spine-Care Journal, 1(3), 7–9. Foster-Fishman, P., Law, K. M., Lichty, L. F., & Aoun, C. (2010). Youth ReACT for social change: A method for youth participatory action research. American Journal of Community Psychology, 46(1), 67–83. http://doi.org/10.1007/s10464-010-9316-y Foster-Fishman, P., Nowell, B., Deacon, Z., Nievar, M. A., & McCann, P. (2005). Using methods that matter: The impact of reflection, dialogue, and voice. American Journal of Community Psychology, 36(3–4), 275–291. http://doi.org/10.1007/s10464-005-8626-y Gaventa, J., & Cornwall, A. (2007). Power and knowledge. The Sage Handbook of Action Research: Participative Inquiry and Practice. Hubbard, B. (2015). Using action research to engage youth in improving OST programming. Afterschool Matters, (22), 32–36. Jacquez, F., Vaughn, L. M., & Wagner, E. (2013). Youth as partners, participants or passive recipients: A review of children and adolescents in community-based participatory 64 research (CBPR). American Journal of Community Psychology, 51(1–2), 176–189. http://doi.org/10.1007/s10464-012-9533-7 Lane, K. L., Kalberg, J. R., Bruhn, A. L., Driscoll, S. a, Wehby, J. H., & Elliott, S. N. (2009). Assessing social validity of school-wide positive behavior support plans: Evidence for the reliability and structure of the Primary Intervention Rating Scale. School Pscyhology Review, 38(1), 135–144. London, J. K., Zimmerman, K., & Erbstein, N. (2003). Youth-led research and evaluation: Tools for youth, organizational, and community development. New Directions for Evaluation, 2003(98), 33–45. http://doi.org/10.1002/ev.83 McCoach, D. B., & Siegle, D. (2003). The school attitude assessment survey–revised: A new instrument to identify academically able students who underachieve. Educational and Psychological Measurements, 63(3), 414–429. http://doi.org/10.1177/0013164402251057 Michigan Department of Education (2017). MI School Data. Retrieved from https://www.mischooldata.org/Default.aspx. Minkler, M. (2000). Using Participatory Action Research to build Healthy Communities. Public Health Reports, 115(2), 191–197. http://doi.org/10.1093/phr/115.2.191 Mitchell, R. R., Tingstrom, D. H., Dufrene, B. A., Ford, W. B., & Sterling, H. E. (2015). The effects of the good behavior game with general-education high school students. School Psychology Review, 44(2), 191–207. http://doi.org/http://dx.doi.org/10.17105/spr-14- 0063.1 Mitra, D. L. (2004). The significance of students : Can increasing “student voice” in schools lead to gains in youth development ? Teachers College Record, 106(4), 651–688. O’Brien, B. C. O., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research. Academic Medicine, 89(9), 1245–1251. http://doi.org/10.1097/ACM.0000000000000388 Ozer, E. J., Cantor, Jeremy, P., Cruz, G. W., Fox, B., Hubbard, E., & Moret, L. (2008). The diffusion of youth-led participatory research in urban schools: The role of the prevention support system in implementation and sustainability. American Journal of Community Psychology, 41, 278–289. http://doi.org/10.1007/s10464-008-9173-0 Ozer, E. J., & Douglas, L. (2013). The impact of participatory research on urban teens: An experimental evaluation. American Journal of Community Psychology, 51(1–2), 66–75. http://doi.org/10.1007/s10464-012-9546-2 Ozer, E. J., Ritterman, M. L., & Wanis, M. G. (2010). Participatory action research (PAR) in middle school: Opportunities, constraints, and key processes. American Journal of Community Psychology, 46(1), 152–166. http://doi.org/10.1007/s10464-010-9335-8 Ozer, E. J., & Schotland, M. (2011). Psychological empowerment among urban youth: Measure development and relationship to psychosocial functioning. Health Education & Behavior, 38(4), 348–356. http://doi.org/http://dx.doi.org/10.1177/1090198110373734 65 Peterson, N. A., Peterson, C. H., Agre, L., Christens, B. D., & Morton, C. M. (2010). Measuring youth empowerment: Validation of a sociopolitical control scale for youth in an urban community context. Journal of Community Psychology, 38(5), 607–621. http://doi.org/10.1002/jcop Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. http://doi.org/10.1007/s10488-010-0319-7 Rapa, L. J. (2016). Fostering marginalized youths’ academic achievement and critical consciousness through a values-affirmation intervention. Retrieved from ProQuest Digital Dissertations. (AAT 10109926) Scherbaum, C. A. & Ferreter, J. M. (2009). Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12 (2), 347-367. http://doi.org/10.1177/1094428107308906 Shin, R. Q., Ezeofor, I., Smith, L. C., Welch, J. C., & Goodrich, K. M. (2016). The development and validation of the contemporary critical consciousness measure. Journal of Counseling Psychology, 63(2), 210–223. Stacy, S.T., Acevedo-Polakovich, I.D., & Rosewood , J. (in press). Youth GO: An Approach to Gathering Youth Perspectives in Out-of-School Time Programs. Afterschool Matters. Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32- item checklist for interviews and focus group. International Journal of Qualitative in Health Care, 19(6), 349–357. http://doi.org/10.1093/intqhc/mzm042 Villarruel, A. M., & Eakin, B. L. (2008). Cuídate: A culturally-based program to reduce HIV sexual risk among Latino youth – Implementation manual. New York: Select Media Publications. Van Sluys, K. (2010). Trying on and trying out: participatory action research as a tool for literacy and identity work in middle grades classrooms. American Journal of Community Psychology, 46, 139–151. http://doi.org/10.1007/s10464-010-9319-8 Vaughn, L. M., Jacquez, F., Zhao, J., & Lang, M. (2011). Partnering with students to explore the health needs of an ethnically diverse, low-resource school: An innovative large group assessment approach. Family & Community Health, 34(1), 72–84. Watts, R. J., Diemer, M. A., & Voight, A. M. (2011). Critical consciousness: Current status and future directions. New Directions for Child and Adolescent Development, 134, 43–57. http://doi.org/10.1002/cd Whyte, W. F., Greenwood, D. J., & Lazes, P. (1989). Participatory action research: Through practice to science in social research. American Behavioral Scientist, 32(5), 513–551. http://doi.org/10.1177/0002764289032005003 66 Wilson, N., Dasho, S., Martin, A. C., Wallerstein, N., Wang, C. C., & Minkler, M. (2007). Engaging young adolescents in social action through photovoice: The youth empowerment strategies (YES!) project. Journal of Early Adolescence, 27(2), 241–261. Wong, N. T., Zimmerman, M. A., & Parker, E. A. (2010). A typology of youth participation and empowerment for child and adolescent health promotion. American Journal of Community Psychology, 46(1), 100–114. http://doi.org/10.1007/s10464-010-9330-0 Zeldin, S., Krauss, S. E., Collura, J., Lucchesi, M., & Sulaiman, A. H. (2014). Conceptualizing and measuring youth–adult partnership in community programs: A cross national study. American Journal of Community Psychology, 54(3–4), 337–347. http://doi.org/10.1007/s10464-014-9676-9 Zeller-Berkman, S., Muñoz-Proto, C., & Torre, M. E. (2013). A youth development approach to evaluation: Critical participatory action research. Afterschool Matters, 22, 24–31. 67