UNDERSTANDING WORK WITH DATA IN SUMMER STEM PROGRAMS THROUGH

AN EXPERIENCE SAMPLING METHOD APPROACH

By

Joshua M. Rosenberg

A DISSERTATION

Michigan State University

in partial fulﬁllment of the requirements

Submitted to

for the degree of

Educational Psychology and Educational Technology – Doctor of Philosophy

2018

ABSTRACT

UNDERSTANDING WORK WITH DATA IN SUMMER STEM PROGRAMS THROUGH

AN EXPERIENCE SAMPLING METHOD APPROACH

By

Joshua M. Rosenberg

Data-rich activities provide an opportunity to develop core competencies in both science
and mathematics identiﬁed in curricular standards. Perhaps even more importantly work
with data puts learners in the position to use data to ask and answer questions, a potentially
empowering capability. Research on work with data has focused on cognitive outcomes and
the development of speciﬁc practices at the student and classroom levels, and yet, little
research has considered learners’ engagement. The present study explores learners engagement
in work with data in the context of summer STEM programs. The aspects of work with data
that are the focus of this study are: asking questions, observing phenomena, constructing
measures and generating data, data modeling, and interpreting ﬁndings. Data from measures
of learners’ engagement was collected through the Experience Sampling Method (ESM) that
involves asking learners at random intervals to answer short questions about their engagement
to discover proﬁles of learners’ engagement. Data was collected from nine summer STEM
programs over four weeks in the Northeastern United States. 203 learners reported 2,970
responses via short ESM surveys of how engaged they were (cognitively, behaviorally, and
a ectively, assessed through separate items) and of their perceptions of themselves (their
competence) and the activity (its challenge). These data were used to examine ﬁve speciﬁc
research questions: 1) What is the frequency and nature of opportunities for youth to engage
in each of the ﬁve aspects of work with data in summer STEM programs? 2) What proﬁles
of engagement emerge from data collected via ESM in the programs? 3) What are sources of
variability for the proﬁles of engagement? 4) How do the ﬁve aspects of work with data relate
to proﬁles of engagement? 5) How do youth characteristics relate to proﬁles of engagement?
Findings show that aspects of work with data were fairly common overall, but that work with

data was enacted out in varying ways, including some that were possibly highly engaging.
Six proﬁles of youth engagement were identiﬁed, representing distinct conﬁgurations of the
ﬁve indicators of engagement. Substantial variability in the proﬁles was present at the youth
level, with less explained by the program youth were in or the nature of the particular
instructional episode present at the times when youth were signaled. Relations between
the proﬁles of engagement and each of the aspects of work with data were somewhat small:
Notable exceptions were the generating data and data modeling were signiﬁcantly associated
with full engagement. Youth with higher pre-program interest in STEM were more likely to
be engaged and competent but not challenged, though other youth characteristics were not
highly related to the proﬁles. I discuss key ﬁndings as regards work with data in summer
STEM programs and other informal learning environments, the nature of youths’ engagement,
and what factors can predict engagement. The design and goals of summer STEM programs,
which are not (necessarily) focused on activities related to work with data, as well as other
limitations including the measures for work with data used and the analytic approach, are
identiﬁed and described. The role of generating data and modeling data as well as attention
to the speciﬁcs of how work with data are enacted are presented as implications for practice.
I highlight aspects of the ﬁndings and the implications for practice with respect to work with
data in general and to engagement in informal learning environments, such as summer STEM
programs, in both cases with an emphasis on how work with data can serve as a promising
context for learning in STEM subject areas.

Copyright by
JOSHUA M. ROSENBERG
2018

This dissertation is dedicated to Katie and to Jonah, who (mostly) happily slept through

most of its writing.

v

ACKNOWLEDGEMENTS

I would like to acknowledge Matthew Koehler and Jennifer Schmidt. I am fortunate to have
learned how to become a scholar through Matt, who has been my advisor since I entered
my Ph.D. program. I could not ask for a better advisor. Thanks, Matt! Matt and Jen
provided support for me to pursue the study reported in this dissertation. They gracefully
and skillfully co-directed my dissertation, and I am grateful for that. I am also grateful to
Jen for not only as a co-director of my dissertation but also for being a mentor and a trusted
source of advice related to my development as a scholar, especially during the process of my
academic job search. Thank you, Jen!

I would also like to acknowledge Lisa Linnenbrink-Garcia and Christina Schwarz
as members of my dissertation committee. In addition to serving as committee members, I
am grateful for the opportunity to develop as a scholar through working and learning from
both. Thank you to my mentors and peers in the Educational Psychology and Educational
Technology program at MSU. Thank you to collaborators Lee Shumow and Neil Naftzger
for their work on the STEM Interest and Engagement project (National Science Foundation
DRL-1421198), of which this project is a secondary analysis. Thank you to participating
youth activity leaders and youth.

Please note that this material is based upon work supported by the National
Science Foundation under Grant No. DRL-1421198. Any opinions, ﬁndings, conclusions, or
recommendations expressed in this material are those of the authors and do not reﬂect the
views of the National Science Foundation.

Understanding Work With Data in Summer STEM Programs Through An Ex-
perience Sampling Method Approach is licensed under a Creative Commons Attribution-
ShareAlike 4.0 International License.

vi

TABLE OF CONTENTS

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
ix
INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
CHAPTER 1
CHAPTER 2 LITERATURE REVIEW . . . . . . . . . . . . . . . . . . . . . . . .
4
2.1 Deﬁning Work with Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
6
2.2 The role of working with data in STEM learning environments . . . . . . . .
7
2.3 What is Known About How Youth Work with Data . . . . . . . . . . . . . .
2.4 Engagement in General and in STEM Domains
. . . . . . . . . . . . . . . .
9
2.5 Youth characteristics that may a ect their engagement . . . . . . . . . . . .
11
2.6 Challenges of Measuring Engagement as a Contextually-Dependent and

Multidimensional Construct . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.7 Need for the Present Study . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8 Conceptual Framework and Research Questions . . . . . . . . . . . . . . . .
CHAPTER 3 METHOD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1 Context
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Data Sources and Measures
. . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.1 ESM measures of engagement for the proﬁles . . . . . . . . . . . . . .
3.4.2 The ﬁve aspects of work with data . . . . . . . . . . . . . . . . . . .
3.4.3
Survey measures of pre-interest in STEM . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . .
3.4.4 Other youth characteristics
3.5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.1 Preliminary analyses . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.2 Analysis for Research Question #1 (on the frequency and nature of

13
16
17
20
20
20
21
22
22
23
26
27
28
28

34

35
36
38
38
38
41

work with data) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28
3.5.3 Analysis for Research Question #2 (what proﬁles of engagement emerge) 31
3.5.4 Analysis for Research Question #3 (sources of variability for the proﬁles) 32
3.5.5 Analysis for Research Question #4 (how work with data relates to

engagement) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.5.6 Analysis for Research Question #5 (how youth characteristics relate

to engagement) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
CHAPTER 4 RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1 Descriptive statistics for the engagement measures . . . . . . . . . . . . . . .
4.2 Correlations among the study variables . . . . . . . . . . . . . . . . . . . . .
4.3 Results for Research Question #1 . . . . . . . . . . . . . . . . . . . . . . . .

vii

4.4 Results for Research Question #2: What proﬁles of youth engagement

emerge from experiential data collected in the programs? . . . . . . . . . . .

4.5 Results for Research Question #3: What sources of variability were there

for the proﬁles of engagement? . . . . . . . . . . . . . . . . . . . . . . . . . .

55
4.6 Results for Research Question #4: Aspects of work with data and engagement 57
61
4.7 Results for Research Question #5: Youth characteristics and engagement . .
CHAPTER 5 DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
64
5.1 Key ﬁndings related to work with data in summer STEM programs . . . . .
5.2 Key ﬁndings related to engagement . . . . . . . . . . . . . . . . . . . . . . .
66
5.3 Key ﬁndings related to work with data and youth characteristics and their

41
42
42
43
43
44
45

46

69
73
79
81
82
86
90
93
95

4.3.1 Frequency of the aspects of work with data . . . . . . . . . . . . . . .
4.3.2 The nature of work with data . . . . . . . . . . . . . . . . . . . . . .
4.3.2.1 Asking questions or identifying problems . . . . . . . . . . .
4.3.2.2 Making observations . . . . . . . . . . . . . . . . . . . . . .
4.3.2.3 Generating data . . . . . . . . . . . . . . . . . . . . . . . .
4.3.2.4 Data modeling . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.2.5
Interpreting and communicating ﬁndings . . . . . . . . . . .

relations to engagement

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4 Limitations to the present study and recommendations for research . . . . .
5.5
Implications for Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
APPENDICES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
PROGRAM DESCRIPTIONS . . . . . . . . . . . . . . . . .
MODEL SPECIFICATION DETAILS . . . . . . . . . . . . .
WORK WITH DATA BY PROGRAM . . . . . . . . . . . . .
ALTERNATE PROFILE SOLUTION . . . . . . . . . . . . .
BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APPENDIX A
APPENDIX B
APPENDIX C
APPENDIX D

viii

LIST OF TABLES

Table 3.1: Demographic characteristics of youth . . . . . . . . . . . . . . . . . . . .

Table 3.2: ESM measures for proﬁles . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 3.3: Coding Frame for the Aspects of Work with Data

. . . . . . . . . . . . .

Table 3.4: Measure for pre-program interest in STEM . . . . . . . . . . . . . . . . .

21

23

25

27

Table 3.5: Coding frame for the open-ended, qualitative coding of instructional episodes 30

Table 4.1: Descriptive statistics for study variables . . . . . . . . . . . . . . . . . . .

Table 4.2: Correlations among the continuous study variables . . . . . . . . . . . . .

Table 4.3: Proportion of signals for which each of the aspects of work with data was

present . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 4.4: Solutions for models that converged . . . . . . . . . . . . . . . . . . . . .

Table 4.5: Raw variables values by proﬁle . . . . . . . . . . . . . . . . . . . . . . . .

Table 4.6:

Intra-class correlation (ICC) values for each of the three levels . . . . . . .

Table 4.7: Results of mixed e ects models with the interactions between interest

and other characactistics and the composite for work with data . . . . . .

Table C.1: Proportion of instructional episodes for which each of the aspects of work

with data was present by program . . . . . . . . . . . . . . . . . . . . . .

38

40

41

47

52

55

60

91

ix

LIST OF FIGURES

Figure 2.1: A conceptual framework for this study and research questions

. . . . . .

Figure 4.1: The six proﬁles of engagement (with variable values standardized) . . . .

Figure 4.2: The six proﬁles of engagement (with raw variable values) . . . . . . . . .

Figure 4.3: Histogram of the proportion of responses for each youth in the proﬁle

they reported most . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure D.1: The seven proﬁles of engagement (with variable values standardized)

Figure D.2: The seven proﬁles of engagement (with variable values standardized)

. .

. .

18

49

49

57

94

94

x

CHAPTER 1

INTRODUCTION

Socializing, working, and even teaching and learning are increasingly impacted by data. These
sources of data are created by us, for us, and about us. Work with data turns learners from
consumers of knowledge to creating knowledge (Hancock, Kaput, & Goldsmith, 1992; Lehrer
& Schauble, 2015; Lee & Wilkerson, 2018; Finzer, 2013). Practice with such work empowers
learners to ask questions and to answer them with arguments and explanations that draw
from data as evidence (McNeill & Krajcik, 2007). This work, then supports learners to
create new knowledge in learning environments and classrooms which is an aim of recent
reform e orts that cast a vision of learning that emphasizes participation in the practices of
STEM disciplines (NGSS Lead States, 2013; National Governors Association Center for Best
Practices, Council of Chief State School O cers, 2010).

Work with data includes broad processes of collecting, creating, modeling data, and
even asking questions that can be answered with data. This work, then, is more than just
crunching numbers. It is also more than interpreting a ﬁgure created by someone else. Instead,
work with data is about making sense of phenomena in the world–or solving problems in
the world. This focus on phenomena is particularly relevant to those designing and enacting
learning opportunities focused on work with data (Lee & Wilkerson, 2018; Singer, Hilton, &
Schweingruber, 2006; Wild & Pfannkuch, 1999).

Work with data provides a capability that can be used across content areas, particu-
larly in advanced coursework. Aspects of work with data are recognized as core competencies
across recent curricular documents for STEM subject area learning. They are found, for
example, in the Next Generation Science Standards and the Common Core State Standards.
These standards highlight the role of authentic work with data as part of engaging in scien-
tiﬁc and engineering and mathematical practices, respectively. These capabilities may be
particularly useful in STEM domains because advanced coursework in these domains often

1

involves demanding and abstract work with data, work that may be more accessible to more
learners when they encounter it earlier in their education.

Past research on work with data has mostly been set in mathematics contexts
and has focused on mathematical practices, like generating measures of phenomena and
creating data models (English, 2012; Lehrer & Romberg, 1996; Lesh, Middleton, Caylor, &
Gupta, 2008). It has often focused on speciﬁc cognitive outcomes (e.g., Gelman & Markman,
1987), strategies to support work with data (Petrosino, Lehrer, & Schauble, 2003), and some
opportunities and challenges facing both teachers and learners when working with data (e.g.,
Konold & Pollatsek, 2002; Finzer, 2013). There has been some research about work with data
in science settings, too. However, scholarship has pointed out that what it means to work
with data can vary greatly in actual classrooms and other learning environments (McNeill &
Berland, 2017). Even so, this past research broadly suggests that engaging in work with data
is powerful concerning learning both about and how to do mathematics and science (Lee &
Wilkerson, 2018; Lehrer & Schauble, 2015). Lehrer and Schauble (2015), summarizing past
research on the use of mathematical practices in science contexts, note that work with data
“has an exceptionally high payo  in terms of students’ scientiﬁc reasoning” (p. 696).

To date, past research shows that using a framework from contemporary engagement
theory to characterize students’ experiences has been informative both in research and to
practicing educators. Work with data is similar to hands-on, laboratory work which research
has shown to be engaging to students (Schmidt, Rosenberg, & Beymer, 2018). In addition,
work with data is demanding and requires sustained e ort and focus (Lehrer & Schauble,
2015; National Research Council, 2015), and past work has shown that when learners are
more challenged (and competent), they are more likely to be engaged (Schneider et al., 2016;
Sherno  et al., 2016). Knowing more about how youth engage in work with data is valuable
as engagement is a meaningful outcome for STEM learners in its own right (Sinatra, Heddy,
& Lombardi, 2015). It may also be an antecedent of changes in other outcomes, such as their
well-being, achievement, and the pursuit of an area of study or career (Wang, Chow, Hofkens,

2

& Salmela-Aro, 2015; Wang & Eccles, 2012). However, research has not examined engagement
in work with data. Because engaging in work with data seems to be so potentially beneﬁcial
to learners, better understanding the nature of work with data and learners’ engagement in
such practices is needed.

The purpose of this study, then, is to examine youth engagement in a variety of
learning activities that involve work with data. I explore youths’ engagement in the context of
outside-of-school STEM enrichment programs carried out during the summer, and I consider
work with data through the lens of speciﬁc aspects identiﬁed from past research, such as
asking questions and generating and modeling data. Such settings (in outside-of-school
programs) are an especially useful context for exploring work with data because they can be
designed around youths’ interests (Lauer, Akiba, Wilkerson, Apthorp, Snow, & Martin-Glenn,
2006). One promise of work with data in outside-of-school settings is that relevant sources
of data can be inherently interesting to learners. Such sources of data can be used as a
context for learning about the world, allowing youth to ask and answer personally and
socially meaningful questions, whereas many outside-of-school programs are focused around
commercial aims, such as developing mobile device applications. Knowing more about how
youth engage can also provide a foundation for subsequent work to explore how particular
curricula and engaging experiences for youth spark their interest in work with data, including
hobbies and occupations related to data science, but also in STEM domains in general.

3

CHAPTER 2

LITERATURE REVIEW

In this review of the literature, I deﬁne work with data as a key practice, or learning-related
activity, across STEM domains. I also deﬁne and justify a multi-dimensional framework for
understanding engagement, and then review an approach to analyzing data that is ideal for
capturing this multidimensionality.

2.1 Deﬁning Work with Data

Some scholars have focused on a few key pieces of data analysis, connected through
the use of “data to solve real problems and to answer authentic questions” (Hancock et al.,
1992, p. 337). This focus on solving real problems or answering authentic questions-rather
than being taught and learned as isolated skills-is an essential part of work with data having
the most educational beneﬁts to learners (National Research Council, 2012; see Lehrer and
Schauble [2012] for some examples of work with data being used in classroom settings).
This approach has primarily been used by mathematics educators, as reﬂected in its role in
statistics curriculum standards (Franklin et al., 2007). In science settings, where answering
questions about phenomena serve as the focus of activities, it shares features of the process
of engaging in scientiﬁc and engineering practices but has been less often studied.

Work with data has been conceived in di erent ways (i.e., Hancock et al., 1992;
Lehrer & Romberg, 1996; Wild & Pfannkuch, 1999). For instance, Wild and Pfannkuch (1999)
consider the process in terms of identifying a problem, generating a measurement system and
sampling plan, collecting and cleaning the data, exploring the data and carrying out planned
analyses, and interpreting the ﬁndings from the analysis. Such a process is common in STEM
content areas and is instantiated in standards for some (especially mathematics) curricula.
Franklin et al.’s guidelines focus on the Framework for statistical problem solving: formulating
questions, collecting data, analyzing data, and interpreting results (2007). The goals of this

4

framework and its components are similar to Hancock et al.’s (1992) description of data
modeling, the process of “using data to solve real problems and to answer authentic questions”
(p. 337). Hancock et al. (1992) focus in on two goals, data creation and analysis, arguing
that the former (data creation) is under emphasized in classroom contexts. Scholars have
subsequently expanded Hancock et al.’s deﬁnition of data modeling to include six components:
asking questions, generating measures, collecting data, structuring data, visualizing data,
and making inferences in light of variability (see Lehrer & Schauble, 2004, for using this
conceptualization of data modeling applied to the task of understanding plant growth). The
last of these components is crucial across all of the visions of data modeling reviewed here and
distinguishes these processes from other aspects of data analysis: Accounting for variability
(or uncertainty) is central to solving real-world problems with data and the process of data
modeling.

Because there is not an agreed-upon deﬁnition of work with data–particularly across
subject area domains (i.e., across all of the STEM content areas)–I focus on the core aspects
that scholars have most often included in their conceptualizations of work with data. These
core components, synthesized from deﬁnitions across studies, are better for understanding
work with data across STEM content areas–as in the present study–than the components
from speciﬁc examples, which were developed for use in only one domain. The aspects of
work with data that have been articulated in prior studies are distilled into ﬁve key aspects
for use in this study. They are:

• Asking questions: Generating questions that can be answered with empirical evidence
• Making observations: Watching phenomena and noticing what is happening concerning

the phenomena or problem being investigated

• Generating data: The process of ﬁguring out how or why to inscribe an observation as

data about phenomena, as well as generating tools for measuring or categorizing

• Data modeling: Activities involving the use of simple statistics, such as the mean and
standard deviation, as well as more complicated models, such as linear models and

5

extensions of the linear model

• Interpreting and communicating ﬁndings: Activities related to identifying a driving

question regarding the phenomena that the question is about

These ﬁve synthesized aspects of work with data are not stand-alone practices but
are a part of a cycle. This process is a cycle is not only because each aspect follows that before
it, but also because the overall process is iterative. For example, interpreting ﬁndings leads to
new questions and subsequent engagement in work with data. Also, scholars have pointed out
some key features of how work with data is carried out that impact their e ectiveness as a
pedagogical approach. These key features include an emphasis on making sense of real-world
phenomena and iterative cycles of engaging in work with data and collaboration and dialogue,
through which ideas and ﬁndings are critiqued and subject to critique, and revised over time
(McNeill & Berland, 2017; Lee & Wilkerson, 2018).

2.2 The role of working with data in STEM learning environments

Working with data can serve as an organizing set of practices for engaging in inquiry
in STEM learning settings (Lehrer & Schauble, 2015). Data are both encountered and
generated by learners, and so opportunities for learners to work with data provide many
opportunities to leverage their curiosity because processes of inquiry can be grounded in
phenomena that learners themselves can see and manipulate or phenomena that learners are
interested in. Also important, becoming proﬁcient in work with data can provide learners with
an in-demand capability in society, owing to the number of occupations, from education to
entrepreneurship, that demand or involve taking action based on data (Wilkerson & Fenwick,
2017). Furthermore, becoming proﬁcient in work with data can be personally empowering
because of the parts of our lives–from paying energy bills to interpreting news articles–that
use data.

Recent educational reform e orts emphasize work with data (i.e., the scientiﬁc
and engineering practices in the NGSS and the standards for mathematical practice in the

6

Common Core State Standards). However, work with data is uncommon in many classroom
settings, even classrooms emphasizing recent science education reform e orts; McNeill &
Berland, 2017; Miller, Manz, Russ, Stroupe, & Berland, advance online publication). As
a result, learning environments suited to engaging in work with data, but not explicitly
designed to support it, may be valuable to study because they may serve as incubators of
these rare and challenging learning activities.

Outside-of-school programs, in particular, are a potentially valuable setting to
explore engagement in work with data, because of the combined pedagogical and technical
expertise of their sta  and the open-ended nature of the activities that are possible to carry
out during them. Sta  or youth activity leaders for these programs include educators and
scientists, engineers, and others with the technical experience. Additionally, the programs
were designed to involve learners in the types of real-world practices experienced by experts
in STEM disciplines. Attendance in such programs is associated with many beneﬁts to
learners (Green, Lee, Constance, & Hynes, 2013; see Lauer et al., 2006, for a review). These
programs are also a good context for understanding work with data because little research
has examined how data are part of the experiences of youth during them.

2.3 What is Known About How Youth Work with Data

There is a good amount of past research on cognitive capabilities as outcomes from
working with data. Much of this (laboratory-based) research has focused on how children
develop the capability to inductively reason from observations (Gelman & Markman, 1987).
Other research has focused on the development of causal, or mechanistic, reasoning, among
young children (Gopnik & Sobel, 2000; Gopnik, Sobel, Schulz, & Glymour, 2001), often from
a Piagetian, individual-development focused tradition (i.e., Piaget & Inhelder, 1969). A key
outcome of engaging in work with data has to do with how learners account for variability
(Lehrer, Kim, & Schauble, 2007; Petrosino et al., 2003; Lesh, Middleton, Caylor, & Gupta,
2008; Lee, Angotti, & Tarr, 2010), arguably the main goal of engaging in work with data

7

(Konold & Pollatsek, 2002). From this research, we know that learners can develop the
capacity to reason about variability.

Past research has also shown that there are strategies that can support work with
data. These include the design of technological tools and the development of curricula. From
this research, we know about speciﬁc strategies and learning progressions for learners to
develop this capability. For example, past research has illustrated the role of measurement in
exposing learners in a direct way to sources of variability (Petrosino et al., 2003) or the place
of relevant phenomena, such as manufacturing processes, such as the size of metallic bolts,
which can help learners to focus on “tracking a process by looking at its output” (Konold &
Pollatsek, 2002, p. 282).

Finally, past research has shown that di erent aspects of work with data pose unique
opportunities and challenges. Asking empirical questions requires experience and ample
time to ask a question that is both able to be answered with data and which is sustaining
and worth investigating (Bielik, 2016; Hasson & Yarden, 2012). Making observations and
generating data, such as of the height of the school’s ﬂagpole, requires negotiation not only
of what to measure, but how and how many times to measure it (Lehrer, Kim, & Schauble,
2007). Regarding modeling, not only teaching students about models, such as that of the
mean, but also asking them to create them, are valuable and practical (Lehrer & Schauble,
2004; Lehrer, Kim, & Jones, 2011), but also time-intensive. Interpreting ﬁndings, especially
in light of variability through models, and communicating answers to questions, means not
only identifying error but understanding its sources, and can be supported through exploring
models that deliberately represent the data poorly, but can be instructive for probing the
beneﬁts and weaknesses of models (Konold & Pollatsek, 2002; Lee & Hollebrands, 2008;
Lehrer, Kim, & Schauble, 2007).

Though very valuable past research that has been carried out, valuable insight into
how learners and youth participate in di erent aspects of work with data through the lens of
engagement has not been explored. This work can compliment past research by showing, for

8

instance, by showing how certain strategies of work with data or how enacting aspects of
work with data in particular ways engage learners (two foci of past research). Consider the
practice of modeling data, commonly described as a-or the-key part of many data analyses
(Konold, Finzer, & Kreetong, 2017). When modeling data, learners may use data they
generated and structured in a data set on their own or may model already-processed, or use
already-plotted, data (McNeill & Berland, 2017). How challenging do students perceive the
di erent enactments of these activities to be and how do learners perceive their competence
regarding them? Importantly, how hard are learners working? How much do they feel they
are learning? Knowing more about these beliefs, characteristics, and processes could help us
to develop informed recommendations for teachers and designers intending to bring about
opportunities for learners to engage in work with data in a better-supported way that is
sustained over time.

2.4 Engagement in General and in STEM Domains

In this section, the nature of engagement is discussed regarding general features
that have been identiﬁed across content area domains, conditions that support engagement,
and di erences between engagement in general and in STEM settings. This is followed
by a discussion of two key features of engagement:
its dynamic, or context-dependent,
characteristics, and its multidimensional nature. Finally, I describe methods for capturing
these two features empirically through an approach called the Experience Sampling Method,
or ESM, and describe how multidimensional data, collected by ESM, can be analyzed.

Engagement is deﬁned in this study as active involvement, or investment, in activities
(Fredricks, Blumenfeld, & Paris, 2004). Explaining how learners are involved in activities and
tasks is especially important if we want to know about what aspects of work with data are
most engaging (and in what ways), and therefore can serve as examples for others advancing
work with data as well as those calling for greater support for engagement. Apart from
being focused on involvement, engagement is often thought of as a meta-construct, that is,

9

one that is made up of other constructs (Skinner & Pitzer, 2012; Skinner, Kindermann, &
Furrer, 2009). By deﬁning engagement as a meta-construct, scholars characterize it in terms
of cognitive, behavioral, and a ective dimensions that are distinct yet interrelated (Fredricks,
2016).

We know from past research that the cognitive, behavioral, and a ective dimensions
of engagement can be distinguished (Wang & Eccles, 2012; Wang & Holcombe, 2010) and
that while there are long-standing concerns about the conceptual breadth of engagement
(Fredricks et al., 2016), careful justiﬁcation and thoughtful use of multidimensional engagement
constructs and measures is warranted. Engagement is also considered to be changing in
response to individual, situation or moment contextual factors, Skinner and Pitzer’s (2012)
model of motivational dynamics, highlighting the community, school, classroom, and even
learning activity, shows the context-dependent nature of engagement on the basis of the
impacts of these factors on learners’ engagement.

Engagement in STEM settings shares characteristics with engagement across disci-
plines, yet there are some distinct aspects to it (Greene, 2015). While one type of engagement—
behavioral—is associated with achievement-related outcomes, many STEM practices call for
engagement in service of other outcomes, especially around epistemic and agency-related
dimensions (Sinatra et al., 2015,). For example, many scholars have deﬁned scientiﬁc and
engineering practices as cognitive practices, which involve applying epistemic considerations
around sources of evidence and the nature of explanatory processes (see Berland et al. 2016,
Stroupe, 2014).

The emphasis on developing new knowledge and capabilities by engaging in STEM
practices must be reﬂected in how the cognitive dimension of engagement is measured.
Because of the importance of constructing knowledge to engagement in STEM practices,
then, I deﬁne cognitive engagement in terms of learning something new or getting better at
something. While sometimes deﬁned in terms of extra-curricular involvement or following
directions, I deﬁne behavioral engagement in this study as working hard on learning-related

10

activities (Fredricks et al., 2004; Singh, Granville, & Dika, 2002). Finally, I deﬁne a ective
engagement as emotional responses to activities, such as being excited, angry, or relaxed
(Pekrun & Linnenbrink-Garcia, 2012).

Finally, some critical conditions facilitate engagement. Emergent Motivation Theory
(EMT; Csikszentmihalyi, 1990), provides a useful lens for understanding these conditions.
From EMT, a critical condition for engagement that can change dynamically, from moment
to moment, is how di cult individuals perceive an activity to be, or its perceived challenge.
Another critical condition is how good at an activity an individual perceives themselves to
be, or their perceived competence. What is most important–and necessary concerning being
engaged–is being both challenged by and good at a particular activity.

Past research has supported this conjecture (Csikszentmihalyi, 1990). As one em-
pirical example, Sherno  et al. (2016) demonstrated that the interaction of challenge and
competence was associated with positive forms of engagement. These ﬁndings suggest that
learners’ perceptions of the challenge of the activity, and their perceptions of how skillful
they are, are important conditions that co-occur with learners’ engagement. Conceptualizing
perceptions of challenge and competence as conditions, rather than factors that inﬂuence
engagement, is in recognition of their co-occurrence within individuals, in that youth experi-
ence engagement and their perceptions of the activity (perceived challenge) and of themselves
(perceive competence) together and at the same time. Thus, these two conditions (challenge
and competence) are considered together with engagement in this study, as described in the
section below on analyzing multidimensional data on engagement.

2.5 Youth characteristics that may a ect their engagement

Past research suggests learners or youths’ characteristics, such as their interest in
the domain of study, impact their cognitive, behavioral, and a ective engagement (Sherno  et
al., 2003; Sherno  et al., 2016; Shumow, Schmidt, & Zaleski, 2013). These are both moment-
to-moment, context-dependent conditions that support engagement (like those discussed

11

above, perceptions of challenge and competence) as well as youth-speciﬁc factors. These
factors are at the level of individual di erences (i.e., youths’ more stable interest in STEM
domains), and may impact engagement, as described in this section.

A factor that can support engagement is how teachers support learning practices
(Strati, Schmidt, & Maier, 2017). Particularly concerning work with data, which is demanding
not only for learners but also teachers (Lehrer & Schauble, 2015; Wilkerson, Andrews, Shaban,
Laina, & Gravel, 2016), sustained support from those leading youth activities is an essential
component of learners being able to work with data. Thus, how youth activity leaders
plan and enact activities related to work with data can have a large impact on students’
engagement. Furthermore, because of the importance of work with data across STEM
domains, carrying out ambitious activities focused on work with data may plausibly have a
substantial impact on the extent to which youth engage in summer STEM program settings.
Consequently, this study considers work with data through the use of a coding frame that
characterizes the extent to which teachers are supporting speciﬁc STEM practices in their
instruction, including aspects of work with data.

Other factors that impact youths’ engagement are individual characteristics and
In recognition of di erences among learners in their tendency to engage in
di erences.
di erent (higher or lower) ways in speciﬁc activities based in part on individual di erences
(Hidi & Renninger, 2006), learners’ interest in STEM before the start of the programs is
also considered as a factor that can impact engagement. Knowing about whether and to
what extent youths’ interest before participating in summer STEM programs explains their
engagement during them is a key question in its own right. It is also important regarding
properly understanding the e ects of other factors, such as working with data, above and
beyond the e ect of pre-program interest. In addition to this interest, I also consider the
gender and the racial and ethnic group of youth, as past research has indicated these as factors
that inﬂuence engagement in STEM (Bystydzienski, Eisenhart, & Bruning, 2015; Sherno 
& Schmidt, 2008). To include the racial and ethnic group of students, I also include youth

12

being part of an under-represented minority (URM) group. To sum up, youths’ pre-program
interest, gender, and URM group membership are considered as individual factors that may
impact youths’ engagement.

2.6 Challenges of Measuring Engagement as a Contextually-

Dependent and Multidimensional Construct

Because of the way engagement has been thought of as having context-dependent
characteristics and being multi-dimensional, it is challenging to use engagement (when
conceptualized in such a way) in empirical studies. One methodological approach that has
beneﬁts concerning the context-dependent and multidimensional nature of engagement is
the ESM. Some scholars have explored or extolled beneﬁts to its use in their recent work
(e.g., Strati et al., 2017; Turner & Meyer, 2000; Sinatra et al., 2015). This study employs the
Experience Sampling Method (ESM; Hektner, Schmidt, & Csikszentmihalyi, 2007) where
learners answer short questions about their experience when signaled. ESM involves asking
(usually using a digital tool and occasionally a diary) participants short questions about their
experiences. ESM is particularly well-suited to understanding the context-dependent nature
of engagement because students answered brief surveys about their experience when they
were signaled, minimally interrupting them from the activity they are engaged in and also
seeking to collect measures about learners’ experience when signaled (Hektner et al., 2007).
The ESM approach is both sensitive to changes in engagement over time, as well as between
learners and allows us to understand engagement and how factors impact it in more nuanced
and complex ways (Turner & Meyer, 2000). Though time-consuming to carry out, ESM
can be a robust measure that leverages the beneﬁts of both observational and self-report
measures, allowing for some ecological validity and the use of closed-form questionnaires
amenable to quantitative analysis (Csikszentmihalyi & Larson, 1987). Despite the logistic
challenge of carrying out ESM in large studies, some scholars have referred to it as the gold
standard for understanding individual’s subjective experience (Schwarz, Kahneman, & Xu,

13

2009).

Research has shown us how the use of ESM can lead to distinct contributions
to our understanding of learning and engagement. This work also suggests how ESM can
be put to use in the present study. For example, Sherno , Csikszentmihalyi, Schneider,
and Sherno  (2003) examined engagement through the use of measures aligned with ﬂow
theory, namely, using measures of concentration, interest, and enjoyment (Csikszentmihalyi,
1997). In a study using the same measures of engagement, Sherno  et al. (2016) used an
observational measure of challenge and control (or environmental complexity) and found
that it signiﬁcantly predicted engagement, as well as self-esteem, intrinsic motivation, and
academic intensity. Schneider et al. (2016) and Linnansaari et al. (2015) examined features
of optimal learning moments or moments in which students report high levels of interest, skill,
and challenge, as well as their antecedents and consequences. Similar to ESM in that through
its use engagement can be studied in a more context-sensitive, still other scholars have used
daily diary studies to examine engagement as a function of autonomy-supportive classroom
practices (Patall, Vasquez, Steingut, Trimble, & Pituch, 2015; Patall, Steingut, Vasquez,
Trimble, & Freeman, 2017). This past research that used ESM (or daily diary studies) to
study engagement has shown that ESM can be used to understand ﬁne-grained di erences in
learning activities, such as the aspects of work with data that are the focus of this study.

Other research shows us that there are newer approaches to analyzing ESM data
that can contribute insights into the context-dependent nature of engagement in a more
ﬁne-grained way. For example, Strati et al. (2017) explored the relations between engagement
to measures of teacher support, ﬁnding associations between instrumental support and
engagement and powerfully demonstrating the capacity of ESM to understand some of the
context-dependent nature of engagement. Similarly, Poysa et al. (2017) used a similar data
analytic approach as Strati et al. (2017), that is, use of crossed e ects models for variation
within both students and time points, both within and between days. These studies establish
the value of the use of ESM to understand the context-dependent nature of engagement and

14

that such an approach may be able to be used to understand engaging in work with data.
Additionally, these recent studies (particularly the study by Strati and colleagues) show that
how e ects at di erent levels are treated, namely, how variability at these levels is accounted
for through random e ects as part of mixed e ects models, is a key practical consideration
for the analysis of ESM data.

One powerful and increasingly widely used way to examine context-dependent
constructs, such as engagement, is the use of proﬁles of, or groups of variables that are
measured. This proﬁle approach is especially important given the multidimensional nature
of engagement. In past research, proﬁles are commonly used as part of what is described
as person-oriented approaches (Bergman & Magnusson, 1997; Bergman, Magnusson, &
El Khouri, 2003), those used to consider the way in which psychological constructs are
experienced together and at once in the experiences of learners. Note that in the present
study, ESM involves asking youth about to report on their experience at the time they were
signaled (rather than, for example, before or after the program, which traditional surveys are
well-suited for). In this study, proﬁles of engagement are used in the service of understanding
how students engage in work with data in a more holistic way. There are some recent studies
taking a proﬁle approach to the study of engagement (i.e., Salmela-Aro, Moeller, Schneider,
Spicer, & Lavonen, 2016a; Salmela-Aro, Muotka, Alho, Hakkarainen, & Lonka, 2016b; Van
Rooij, Jansen, & van de Grift, 2017; Schmidt, Rosenberg, & Beymer, 2018), though none
have done so to study youths’ engagement in work with data.

The proﬁle approach has an important implication for how we analyze data collected
from ESM about youths’ engagement, in particular when we consider how to understand
engagement as a multi-dimensional construct, and one with momentary, or instructional
episode-speciﬁc, conditions (Csikszentmihalyi, 1990). We know from past research that
engagement can be explained through di erent patterns among its components (Bergman
& Magnusson, 1997; Bergman et al., 2003), in the present case its cognitive, behavioral,
and a ective components. Because learners’ engagement includes cognitive, behavioral, and

15

a ective aspects experienced together at the same time, it can be experienced as a combined
e ect that is categorically distinct from the e ects of the individual dimensions of engagement.
This combined e ect can be considered as proﬁles of engagement.

Past studies have considered proﬁles of cognitive, behavioral, and a ective aspects of
engagement. For example, to account for the context-dependent nature of engagement, some
past studies have used other measures to predict engagement, such as the use of in-the-moment
resources and demands (Salmela-Aro et al., 2016b) and the use of instructional activities and
choice (Schmidt et al., 2018). A potential way to extend this past research is to account for
not only engagement (cognitive, behavioral, and a ective), but also the intricately connected
perceptions of challenge and competence. This analytic approach is especially important
since a proﬁle approach emphasizes the holistic nature of engagement and the impact of
not only external but also intra-individual factors. Accordingly, youths’ perceptions of the
challenge of the activity and their competence at it are used along with the measures of
engagement to construct proﬁles of engagement. Thus, the proﬁles of engagement include
youths’ responses to ﬁve ESM items for their cognitive, behavioral, and a ective engagement
and their perceptions of how challenging the activity they were doing is and of how competent
at the activity they are.

2.7 Need for the Present Study

While many scholars have argued that work with data can be understood in terms
of the capabilities learners develop and the outcome learners achieve, there is a need to
understand learners’ experiences working with data. The present study does this through
the use of contemporary engagement theory and innovative methodological and analytic
approaches. Doing this can help us to understand work with data in terms of learner’s
experience, which we know from past research impacts what and how students learn (Sinatra
et al., 2015). Knowing more about students’ engagement can help us to design activities and
interventions focused around work with data. In addition to this need to study engagement

16

in work with data through the lens of engagement, no research has yet examined work with
data in the context of summer STEM programs, though such settings are potentially rich
with opportunities for highly engaged youth to analyze authentic data sources.

2.8 Conceptual Framework and Research Questions

To summarize this section, the present study is about how learning activities
involving various aspects of work with data can be understood in terms of engagement. Its
context is out-of-school-time STEM enrichment programs designed to meet guidelines for
best practices. The conceptual framework in the present study is presented in Figure 2.1 and
is laid out in the remainder of this section.

There are ﬁve aspects of work with data synthesized from past research (i.e., Hancock

et al., 1992; Lehrer & Romberg, 1996; Wild & Pfannkuch, 1999):

1. Asking questions or identifying problems
2. Making observations
3. Generating data
4. Data modeling
5. Interpreting and communicating ﬁndings

In Figure 2.3, engagement in work with data is associated with di erent proﬁles of
engagement. The theoretical framework for the proﬁle approach suggests that engagement is
a multi-dimensional construct consisting of cognitive, behavioral, and a ective dimensions
of engagement and perceptions of challenge and competence. Also, a pre-program measure
of youths’ pre-program interest in STEM, along with youths’ gender and URM status, are
hypothesized to be associated with the proﬁles and the relations of work with data and the
proﬁles.

Regarding research questions 2-5, the ESM responses that make up the proﬁles are
associated with di erent “levels.” These levels, or groups, which may introduce dependencies
that violate statistical assumptions of the independence of the responses, are commonly

17

Figure 2.1: A conceptual framework for this study and research questions

considered in the Hierarchical Linear Modeling (also known as multi-level or mixed e ects
modeling) literature as random e ects (Gelman & Hill, 2007; West, Welch, & Galecki,
2015). In this study, three levels that can be modeled as random e ects to account for
the dependencies they introduce: Youth, instructional episode (which are indicators for the
moments–or segments–in which youth are asked to respond to the ESM signal), and the
program. Thus, these are not predictor variables, but rather are the levels that are present
given the approach to data collection and the sampling procedure. Interpreting their e ects
is not a goal of this study, but accounting for them in the models used, as in this study, is
essential and is done through the use of random e ects.

Pre-program interest, gender, and URM status are predictor variables at the youth
level. The aspects of work with data are predictor variables at the instructional episode
level. There are no predictor variables at the program level, in part due to the small number
of programs (and the resulting low statistical power of any variables added at this level).
Pre-program interest, gender, and URM status, and the aspects of work with data are used
as predictor variables, while the three levels (youth, instructional episode, and program) are

18

accounted for in the mixed e ects modeling strategy.

The ﬁve research questions, then, are:

1. What is the frequency and nature of opportunities for youth to engage in each of the

ﬁve aspects of work with data in summer STEM programs?

2. What proﬁles of engagement emerge from data collected via ESM in the programs?
3. What are sources of variability for the proﬁles of engagement?
4. How do the ﬁve aspects of work with data relate to proﬁles of engagement?
5. How do youth characteristics relate to proﬁles of engagement?

19

CHAPTER 3

METHOD

3.1 Context

The setting for the present study was nine out-of-school STEM programs during
2015 in the Northeast United States. Descriptions of the programs are provided in Appendix
A. Two intermediary organizations which were contracted by the local school districts to
administer the summer programs. The two intermediaries were responsible for soliciting
and enrolling youth; establishing guidelines for the design of the programs, and the goals of
the programs; and providing training and professional development for the sta , hereafter
referred to as youth activity leaders.

There was a di erence between the two intermediary organizations, namely, one
separated academic and enrichment-related activities, whereas, in the other, the academic
and enrichment components were more integrated, which may have program-related e ects
on youths’ engagement. Many of the programs aim to involve youth in work with data.
These learning environments bring together youth activity leaders, educators, and those with
technical expertise in STEM domains. Youth spent around three hours per day for four days
per week for the approximately four-week programs, which were taught by youth activity
leaders and scientists, engineers, and other community members with technical expertise.

3.2 Participants

Participants consisted of 203 youth. Participants were from diverse racial and ethnic
backgrounds (see Table 3.1). The mean age of participants was around 13 years old (from
youth whose age was available: M = 12.71, SD = 1.70, min. = 10.75, max. = 16.36).
Detailed demographic characteristics of youth are presented in Table 3.1.

20

Table 3.1: Demographic characteristics of youth

Youth
Sex
Male
Female
Race/Ethnicity
Hispanic
White
Black
Multi-racial
Asian/Paciﬁc Islander
Parent Education
High School or Below
Graduated from College (B.A. or B.S.)

Percentage

50
50

48
6
36
3
7

79
21

3.3 Procedure

Before the start of the programs, youth completed a pre-survey that included
questions about their experience in STEM, intention to pursue a STEM major or career, and
other motivation and engagement-related measures; items about youths’ interest in STEM
were the only items used from this survey in this study.

At the beginning of the programs, youth were introduced to the study and the
phones used for data collection related to the ESM. As indicated in the literature review,
ESM is a method of data collection that involves asking youth to respond to short questions
on phones (that were provided as part of the study). In particular, youth were signaled at
random times (within intervals, so that the signals were not too near or far apart) in order to
obtain a sample of their experience throughout the program. ESM data were collected two
days each week, for three weeks (weeks 2-4 of the program). In all of the programs, about
equal video-recording time was dedicated to classroom and ﬁeld experiences. This detail
is noteworthy because programs associated with one of the intermediaries rotated between
classroom and ﬁeld experience days, while the other used the ﬁrst half of each day for one

21

and the second for the other. Each day, youth were signaled four times. These signals were
at the same time for all of the youth within their program, but at di erent times between
programs and between days within programs (with the constraint that no two signals could
occur less than ten minutes apart).

The programs were video-recorded by research team members on the days during
which ESM data were collected. So that the measures relating the video-recording and ESM
data can be matched, the videos included a signal from the person doing the video-recording
that identiﬁed the ESM signal to which youth were signaled to respond.

3.4 Data Sources and Measures

Data sources consist of ESM measures of engagement and youths’ perceptions of
themselves and the activity, pre-survey measures of youths’ interest, youths’ demographic
information, and the video-recordings of programs.

3.4.1 ESM measures of engagement for the proﬁles

Measures for engagement were created from ﬁve ESM questions, three serving indicators for
the experience of engagement and two for the conditions of engagement. The three indicators
for engagement were for learning (for the cognitive engagement construct), working hard
(for behavioral engagement), and enjoying (for a ective engagement). The variables for the
conditions are for perceived challenge and perceived competence.

All ﬁve items were ultimately used to construct the proﬁles of engaged. Each of the
ESM items consisted of the item text and the following four item response options, of which
youth were directed to select one: Not at all (associated with the number 1 on the survey; 1),
A little (2), Somewhat (3), and Very Much (4), as presented in Table 3.2. Note that because
these items were measured using single-item indicators (which is common in studies using
ESM; Hektner et al., 2007), information about the reliability and validity information for
these measures is not included.

22

Table 3.2: ESM measures for proﬁles

Item
As you were signaled, were you learning anything or getting better at something?

Construct
Cognitive engagement
Behavioral engagement As you were signaled, how hard were you working?
A ective engagement
Perceived challenge
Perceived competence

As you were signaled, did you enjoy what you are doing?
As you were signaled, how challenging was the main activity?
As you were signaled, were you good at the main activity?

3.4.2 The ﬁve aspects of work with data

Di erent aspects of work with data were identiﬁed from video-recordings. Speciﬁcally, codes
for work with data were generated on the basis of the activity that the youth activity
leaders were facilitating. The activity youth activity leaders were facilitating were from the
STEM-Program Quality Assessment (STEM-PQA; Forum for Youth Investment, 2012), an
assessment of quality programming in after-school programs. I then identiﬁed the speciﬁc
activities that corresponded to the ﬁve aspects of work with data, as deﬁned here. While
I chose to match the ﬁve aspects of work with data to the STEM-PQA code(s) that I
interpreted as aligning most closely (in the cases of generating data and interpreting and
communicating ﬁndings, choosing to use two STEM-PQA items as codes), there are other
ways that these could be matched. For example, in the NGSS (NGSS Lead States, 2013),
asking questions emphasizes coming up with questions that can be answered through an
investigation, whereas the STEM-PQA code used to indicate asking questions emphasizes
exploring possible solutions to problems and testing hypotheses. Here are the aspects of work
with data (in bold) and the STEM-PQA code(s) to which I corresponded them.

Asking questions: Predict, conjecture, or hypothesize (Sta  support youth in
using a simulation, experiment, or model to answer questions, explore solutions, or test
hypotheses [e.g., Youth run a robotics program to determine whether it does what they
expect it to; Youth try an alternate way to solve an equation and test their results against
another example, etc.])

Making observations: Classify or abstract (Sta  support youth in using classiﬁ-

23

cation and abstraction, linking concrete examples to principles, laws, categories, and formulas
[e.g., Mice, porcupines, and squirrels are all rodents, rodents are all mammals; The pool ball
moved because for every action, there is an equal and opposite reaction; etc.])

Generating data: Collect data or measure (Sta  support youth in collecting data
or measuring [e.g., Youth use rulers or yardsticks to measure length; Youth count the number
of di erent species of birds observed in a speciﬁc location, etc.]) and Highlight precision and
accuracy (Sta  highlight value of precision and accuracy in measuring, observing, recording, or
calculating [e.g., measurement error can impact an experiment or conclusion; measure twice,
cut once; scientist always need to double-check their calculations before drawing conclusions;
you must observe carefully to see the di erence between various species of sparrows, etc.])
Data modeling: Simulate, experiment, or model (Sta  support youth in using a
simulation, experiment, or model to answer questions, explore solutions, or test hypotheses
[e.g., Youth run a robotics program to determine whether it does what they expect it to;
Youth try an alternate way to solve an equation and test their results against another example,
etc.])

Interpreting and communicating ﬁndings: Analyze (Sta  support youth in
analyzing data to draw conclusions (e.g., after an experiment, youth are asked to use results
to make a generalization like “Your heartbeat increases when you exercise”, etc.)) and
Use symbols or models (Sta  support youth in conveying STEM concepts through symbols,
models, or other nonverbal language (e,g, youth use diagrams, equations, ﬂowcharts, outlines,
mock-ups, design software, dioramas, physical models, prototypes, graphs, charts, tables,
equations, etc.))

I then used these codes as part of the following coding frame, with the code names,
possible values, code description, and examples from this study, as presented in Table 3.3.
Note that this coding frame was not developed to assess work with data but rather was
adapted for this purpose based on aligning dimensions of the STEM-PQA with the categories
of the coding frame for work with data in this table.

24

Table 3.3: Coding Frame for the Aspects of Work with Data

Code Name
Asking questions

Making
observations

2
5

Generating data

Data modeling

Values
1: Present; 0:
Not Present

1: Present; 0:
Not Present

1: Present; 0:
Not Present

1: Present; 0:
Not Present

Interpreting and
communicating
ﬁndings

1: Present; 0:
Not Present

Description
Discussing and exploring topics to
investigate and pose questions.

Watching and noticing what is
happening with respect to the
phenomena or problem being
investigated.
Figuring out how or why to inscribe an
observation as data and generating
coding frames or measurement tools.
Understanding and explaining
phenomena using models of the data
that account for variability or
uncertainty.
Discussing and sharing ﬁndings.

Example
Youth generated questions they
investigated related to tide ponds in an
estuary ecosystem.
Youth observed the projectile motion of
an object launched with a catapult.

Youth wrote in a table the number of
pieces of recyclables they collected in
parts of local waterways.
Youth calculated the average number of
plant species found across a number of
sites in the ﬁeld.

Youth presented the outcomes of an
investigation or engineered design in
light of a research question or problem.

Raters contracted by American Institute of Research (AIR) were trained in the use
of the Program Quality Assessment tool (PQA), the broader assessment tool for which the
STEM-PQA is a supplement. Raters completed a four-hour online training module on the
overall PQA tool and then attended an in-person two-day training led by a trainer from the
David P. Weikart Center for Youth Program Quality, the tool’s publisher, where they learned
about the instrument, trained on its use, and then established inter-rater reliability with
a master coder. For the STEM-PQA, three of the same raters contracted by AIR to code
the (overall) PQA measure used the STEM-PQA supplement to score one video segment,
for which there were no disagreements on scoring for any of the items. The programs were
divided up among all of the raters, so raters coded some of the videos for all of the programs.
When the raters encountered a situation that was di cult to score, they would discuss the
issue by telephone or more often by email after viewing the video in question and reach a
consensus on how to score the speciﬁc item. Note that these codes were unique to each signal
to which youth responded (but were not unique to each youth, as the youth in the same
program were signaled at the same time).

Out of the 248 instructional episodes, 236 were code-able for work with data; for
the 12 that were not codeable, issues with the video-recordings were the primary source of
the missing data. These 236 responses were used for all of the analyses.

3.4.3 Survey measures of pre-interest in STEM

Measures of youths’ pre-interest were used as youth-level characteristics that predict the
proﬁles of engagement. In particular, three items adapted from Vandell, Hall, O’Cadiz, and
Karsh (2012) were used, with directions for youth to rate their agreement with the items’
text using the same scale as the ESM items: Not at all (associated with the number 1 on the
survey), A little (2), Somewhat (3), and Very Much (4). Reliability and validity information
on this scale is presented in Vandell et al. (2008).

This measure was constructed by taking the maximum value for the scales for the

26

Table 3.4: Measure for pre-program interest in STEM

Construct
Pre-program interest in STEM I am interested in science / mathematics /

Item text

engineering.
At school, science / mathematics /
engineering is fun
I have always been fascinated by science /
mathematics / engineering)

di erent content areas (science, mathematics, and engineering)s so that the value for a youth
whose response for the science scale was 2.5 and for the mathematics scale was 2.75 would
be 2.75. See Beymer, Rosenberg, and Schmidt (2018) for more details on this (use of the
maximum value) measurement approach. The items are presented in Table 3.4. Overall
levels of this measure were high (M = 3.044 (SD = 0.901). The individual interest measure
represented the mean of interest items across all relevant domains. Thus for some students,
the mean was based on three items, while for others it was based on as many as nine items
representing all three domains (with Cronbach – values ranging from .77 - .86 for each
domain-speciﬁc interest scale).

3.4.4 Other youth characteristics

In addition to the measures described in this section, demographic information for youths’
gender, and their racial and ethnic group are used to construct demographic variables for
gender and membership in an under-represented (in STEM) group; membership in an under-
represented group was identiﬁed on the basis of youths’ racial and ethnic group being Hispanic,
African American, Asian or Paciﬁc Islanders, or native American.

27

3.5 Data Analysis

3.5.1 Preliminary analyses

Correlations (ﬁrst-order Pearson) and the frequency, range, mean (M), and standard deviation
(SD) are ﬁrst presented for all variables. In addition, the frequency of the codes for aspects of
work with data and the numbers of responses by youth, program, and instructional episode
are presented.

3.5.2 Analysis for Research Question #1 (on the frequency and nature of work

with data)

There were two primary steps taken to answer this question, one more quantitative in nature
and one more qualitative. The quantitative aspect focused on the frequency of work with
data, whereas the qualitative aspect focused on the speciﬁc nature of work with data.

For the quantitative aspect, the codes for the aspects of work with data (described
above in the section on the measures) were counted up and presented as a proportion of
the number of code-able instructional episodes. As the signals represent a sample of youths’
experiences in the programs, results from this analysis provide insight into how often each
of the aspects took place during the programs. Note that this coding frame focused on the
degree of instructional support the activity leaders provided for youth to work with data, thus
results from this analysis will show how often support for the di erent aspects of work with
data was provided, though youth may engage in the aspects of work with data to varying
degrees.

The frequency of work with data, the focus of the quantitative analysis for this
research question, will provide insight into how regular the aspects of work with data were,
but not about the ways in which work with data was enacted. For example, qualitative
di erences in how youth were asking questions will not be evident from the codes as they were
used. In order to provide more detail in terms of the nature of work with data in summer

28

STEM segments, the data was coded with an open-ended, qualitative approach.

Speciﬁcally, three research assistants were trained for approximately eight hours,
over the course of four meetings. Then, each research assistant coded all of the segments
associated with the videos for a particular. Two coders coded every segment, except for the
segments for which the quantitative coding indicated no aspects of work with data were
present; instead, for these segments, only one coder coded each segment. For all of the guiding
questions, the coders also took note of who (the youth, youth activity leader, or someone
else) was the focus of the aspect of work with data. For example, with respect to interpreting
and communicating ﬁndings, denoted when youth were sharing the results from a hands-on
investigation or when it was the youth activity leader doing so as a summary on the basis
of the work youth recently completed. Table 3.5 summarizes the aims of the open-ended,
qualitative coding, as well as example codes from this study. Note that these examples are
di erent in nature than those for the coding frame for work with data (see Table 3.3), as
these codes were written in an open-ended matter, whereas the codes for work with data
were applied based on the coding frame (with only 0’s and 1’s as possible value for codes).

29

Table 3.5: Coding frame for the open-ended, qualitative coding of instructional episodes

Topics
Nature of activity

Description
Note the nature of what was
happening in terms of the activity
or activities youth were involved
in.

3
0

Youth or youth activity leader Note the extent to which youth or
the youth activity led the activity
(or whether it was led by both the
youth and youth activity leader).

Work with data

For each aspect of work with data
determined to be present, note
how youth were involved in
working with data.

Example of Codes
Youth coming up with ideas for their ﬁnal project,
activity leader walking around giving them
encouragement, helping students think of ideas.
While youth are working, activity leader pulls up a
website for a franchise, tells youth that’s the ﬁrst
step to working on their project, describes where to
ﬁnd the numbers they need, talks about what
information a company’s website will include,
encourages youth to look further into a company
using other websites. Youth then take survey.
Activity learner shows youth containers they will
put specimens in so they can observe it and tools
they can use to capture bugs. Activity leader
describes chart that students were given outlining
di erent symbols for di erent kinds of animals (e.g.,
leaf for producer), describing di erent methods to
obtain specimens.
Youth are counting in their districts how much
plastic they gathered (from one of their ﬁeld trip
sites at the bay). They are asked to write down the
amount of plastic each district gathered (generating
data). They are then sharing their ﬁndings with
each other based on calculations they did with the
data. They were calculating how many pieces of
plastic they collected together over four days (134
pieces)

After coding all of the segments for each program, the coders and I met to discuss
potential issues that emerged throughout the coding. The goal of the meetings was to address
any problems encountered when using the guiding questions and to clarify how they applied
the coding frame. After the coding was complete, I then read through all of the codes for all
of the segments then made notes associated with each of the ﬁve aspects of work with data.
I used these notes to write descriptions of the nature of work with data for each of the ﬁve
aspects. After reading through the qualitative codes and my descriptions of the nature of work
with data during each segment, I grouped the descriptions into themes, which I present in the
results for this research question. I also used these themes to calculate proportions, which
are also presented in the ﬁndings for this section. In summary, an open-ended, qualitative
coding approach was used to create descriptions of the ways in which each of the aspects of
work with data was enacted. This analysis is used to provide insights into the nature of work
with data in summer STEM programs.

3.5.3 Analysis for Research Question #2 (what proﬁles of engagement emerge)

Latent Proﬁle Analysis (LPA; Harring & Hodis, 2016; Muthen, 2004) was used to identify
proﬁles of engagement. LPA allows for capturing the multidimensional nature of engagement
through proﬁles in terms of discovering groups of the ways in which youth experience
engagement together and at once. A key beneﬁt of the use of LPA, in addition to likelihood
estimation-based ﬁt indices, is probabilities of an observation being a member of a cluster
(unlike in cluster analysis). These proﬁles make it possible to analyze the multivariate data
collected on engagement in a way that balances the parsimony of a single model.

For these analyses, ﬁve variables were included: the three indicators for the experi-
ence of engagement (cognitive, behavioral, and a ective) and the two necessary conditions
for it (perceptions of challenge and competence). In addition, solutions with between two
and ten proﬁles were considered. As part of LPA, the model type selection-where the type
refers to which parameters are estimated–is a crucial topic. For the present study, six model

31

types were considered:

1. Varying means, equal variances, and covariances ﬁxed to 0
2. Varying means, equal variances, and equal covariances
3. Varying means, varying variances, and covariances ﬁxed to 0
4. Varying means, varying variances, and equal covariances
5. Varying means, equal variances, and varying covariances
6. Varying means, varying variances, and varying covariances

The MPlus software (Muthen & Muthen, 1998-2017) was used to carry out LPA
through open-source statistical software I developed and published to the Comprehensive R
Archive Network, tidyLPA (Rosenberg, Schmidt, & Beymer, 2018). More speciﬁc details on
LPA are included in Appendix B.

To select a solution in terms of the model type and the number of proﬁles to be
interpreted and used in subsequent analyses, a number of ﬁt statistics and other considerations
were taken into account. These include a range of information criteria (AIC, BIC, and sample
adjusted BIC [SABIC]), statistics about the quality of the proﬁle assignments (entropy, which
represents the mean posterior probability), a statistical test (the bootstrapped LRT [BLRT]),
and concerns of interpretability and parsimony. On the basis of these criteria, a particular
solution was selected and used as part of subsequent analyses; as the model selection process
is an integral part of providing an answer to this question, the model and number of proﬁles
selected are described in the section for the results for this research question.

3.5.4 Analysis for Research Question #3 (sources of variability for the proﬁles)

How youth are engaging was a function of who they are as an individual, what they happen
to be doing during a particular instructional episode, and which youth program they are
enrolled in, as well as random variation. This analysis seeks to identify how much of the
variation was at each of these levels through using null models, or models only with the

32

indicators for the three levels (youth, instructional episode, and program). These models
can show how much variability in the proﬁles was systematic at these di erent levels and
was potentially attributable to each of these types of factors. These null models may also
suggest something about where you might want to be looking to explain sources of youth’s
engagement.

Sources of variability in these proﬁles can be used as additional information in
their own right for interpreting the proﬁles and in order to anticipate the e ects of predictor
variables at the youth, instructional episode, and program levels. First, the proportion of the
variability at each of these levels was explored through the use of null, or variance components.
These are models that only include grouping (i.e., the variable identifying which youth a
response was from, what signal the response was associated with, and from which program
the youth and signal were from) factors. These models provide insight into at which of these
“levels” predictors may be able to explain the outcome.

Variability in terms of the number (and proportion) of proﬁles each youth reports
can also be considered. The breakdown of responses in each of the six proﬁles by youth was
used to show the extent to which youth report their most reported proﬁle. In addition, apart
from this overall mean proportion of youths’ responses, the mean proportion for speciﬁc
proﬁles (i.e., when youth report a particular proﬁle the most, how often, on average, do they
report it?) are also considered.

The ICCs provide information about sources of variability in the proﬁles of engage-
ment with respect to the same proﬁle. One way to better understand the nature of variability
across proﬁles is by examining how often youth reported the same proﬁle: Whether youth
exhibit stable or highly variables modes of engagement (i.e., are some youth always Fully
engaged?) can provide a descriptive portrait of youths’ experiences the many instructional
episodes they were involved in. To determine how stable youths’ engagement was, for each
youth, the proﬁle that youth reported most was identiﬁed, and then the proportion of their
responses in that proﬁle was calculated. These proportions are also presented in the results

33

for this question.

3.5.5 Analysis for Research Question #4 (how work with data relates to en-

gagement)

This question is focused on how work with data relates to the proﬁles of engagement. For the
primary results for this question, mixed e ects models that account for the cross-classiﬁcation
of the instructional episode (because of the dependencies of the responses associated with
each of the 248 distinct ESM signals) and youth are used and for the “nesting” of both within
each of the nine programs are used. The lme4 R package (Bates, Martin, Bolker, & Walker,
2015) was used. All of the models for this and the subsequent research question use random
e ects for youth, instructional episode, and program e ects. Youth and the instructional
episode can be considered to be crossed with both nested within the program.

The probability of a response belonging to the proﬁle was the dependent variable,
and the aspects of work with data are the independent variables. There are six models, for
each of the six proﬁles. Because the outcome from LPA is not a hard classiﬁcation (i.e., an
observation is in a proﬁle—or not) but a probability, the dependent variable is treated as a
continuous variable.

First, null models with only the random parts (i.e., random youth, instructional
episode, and program e ects) were speciﬁed. Then, the ﬁve aspects of work with data were
added as predictors to the model. The results will be interpreted on the basis of which of the
statistical signiﬁcance and the magnitude and direction of the coe cients associated with
these ﬁve predictors. For example, if the coe cient for the e ect of the asking questions aspect
of work with data upon one of the proﬁles was 0.10, and is determined to be statistically
signiﬁcant, then this would indicate that when youth are engaged in this aspect of work with
data, then they are ten percentage points more likely to report a response in that particular
proﬁle.

For this question, models with the aspects of work with data both separate from

34

and together with the youth characteristics were ﬁt. The models with both together were
also used as part of research question #4, though they are presented here (and interpreted in
the sections for both results). In speciﬁc, mixed e ects models, predicting the probability of
membership in each of the six proﬁles as the dependent variable–using the work with data
codes as predictors–were speciﬁed.

Because the results were found to be identical when the aspects of work with data
and the youth characteristics were considered in separate and in the same model, the results
from the two sets of variables being in the same model were used both to provide answers to
both this and the next research question. Note that a composite for work with data (made
as the sum of the individual aspects of work with data) was considered, but as it did only
yield one (small) statistically signiﬁcant result, the results for this analysis are not presented
in the results.

3.5.6 Analysis for Research Question #5 (how youth characteristics relate to

engagement)

This question is focused on how the relationships of work with data di er on the basis of
youth characteristics. In particular, their pre-program interest, gender, and URM status are
used as predictor variables. The same (mixed e ects) models (and statistical software) used
for the previous research question are used for this research question. The dependent variable
was again the probability of a response being in the proﬁle.

The three youth characteristics (pre-program interest in STEM, gender (entered s
a dummy code with the value of “1” indicating female), and URM status (also entered as
a dummy code, with “1” indicating a youth from a URM group) are added as predictors.
Like for the previous research question, the statistical signiﬁcance and the magnitude and
direction of the coe cients associated with each predictor are interpreted to answer this
question. For example, and similar to the interpretation of the predictors associated with
RQ #3, if the relationship between pre-program interest and a proﬁle is 0.05, then for each

35

one-unit increase in pre-program interest, then youth are ﬁve percentage points more likely
to report a response in a particular proﬁle.

Models with the youth characteristics separate from and together with the aspects
of work with data were ﬁt. Like for the results of the previous question, the models only
with the youth characteristics yielded very similar results. Thus, the models presented in the
previous section with both youth characteristics and the aspects of work are interpreted as
part of answering this question.

As described in the previous sub-section, because the results were very similar
when the aspects of work with data and the youth characteristics were added in separate
models compared to when they were included in the same model, the results for both sets of
predictors in the same model are presented and interpreted. In addition, interactions between
statistically signiﬁcant aspects of work with data and all of the youth characteristics were
examined, though because none of these interactions were found to be statistically signiﬁcant,
they were not included with the results.

3.6 Sensitivity Analysis

For observational studies, such as the present study, it can be important to determine
how robust an inference is to alternative explanations. One approach to addressing this is
sensitivity analysis, which involves quantifying the amount of bias that would be needed to
invalidate an inference. Using the approach described in Frank, Maroulis, Duong, and Kelcey
(2013), I carried out a sensitivity analysis for inferences made relative to signiﬁcant relations.
I used the R package konfound (Rosenberg, Xu, & Frank, 2018).

The result of the sensitivity analysis, and what was used to interpret and contextual-
ize ﬁndings, is a numeric value, between 0 and 1, for each e ect that indicates the proportion
of the estimate that would have to be biased in order to invalidate the inference. A value
close to 0 (such as .05) indicate that a tiny change in the size of the e ect would change the
inference made (i.e., a statistically signiﬁcant result that is interpreted would no longer be

36

interpreted as an e ect). Larger values, such as values around .50, indicate that a substantial
amount of an e ect could be due to bias (i.e., less than 50% of an e ect could be due to bias
in the sample), but even still, the same inference about a statistically signiﬁcant could be
made, suggesting that such an e ect is more robust than one with a smaller value.

I used sensitivity analysis to interpret and contextualize hypotheses about key
relationships for research questions #4 and #5 for this study, for the relationships between
the aspects of work with data and youth characteristics and the proﬁles of engagement.
In particular, I carried out a sensitivity analysis for the coe cients that were statistically
signiﬁcant in order to provide some insight into how robust the results are. In addition, I
carried out a sensitivity analysis for coe cients that were close to statistically signiﬁcant but
were not statistically signiﬁcant, in order to better understand how little would need to change
in order for an e ect to be determined to be signiﬁcant. Higher values from the analysis
(i.e., values closer to 1) indicated more robust estimates in that the inferences would still
hold even if there were substantial bias in the estimate and that were interpreted as robust
ﬁndings, while lower values, when present, indicated less robust ﬁndings that I interpreted
with more caution.

37

CHAPTER 4

RESULTS

4.1 Descriptive statistics for the engagement measures

First, descriptive statistics for the ﬁve engagement variables that were used to
estimate the proﬁles are presented in Table 4.1. These descriptive statistics show high overall
levels of cognitive (M = 2.768, SD = 1.063), behavioral (M = 2.863, SD = 1.044) and
a ective (M = 2.831, SD = 1.051) engagement.

These statistics also show high perceptions of competence (M = 3.000 (SD = 0.952))
and moderate perceptions of challenge (M = 2.270 (SD = 1.117)). There was a similar degree
of (moderate) variability across the engagement measures (see the SDs): This variability may
be due to the youth, instructional episode, program, and even for unexplained reasons.

4.2 Correlations among the study variables

Correlations between the variables that were used to create the proﬁles of engagement
and the one other variable which was continuous (rather than a code for groups, in particular
youths’ gender and URM status), pre-program interest in STEM (Table 4.2), were speciﬁed.
In addition, relations between these variables and those for the ﬁve aspects of work with data
were identiﬁed. Note that because the ﬁve variables were dichotomous, Spearman (rather

Table 4.1: Descriptive statistics for study variables

Cog. eng.
Beh. eng.
A . eng.
Challenge
Competence

n Mean
2.768
2.863
2.831
2.270
3.000

2969
2959
2970
2970
2970

SD
1.063
1.044
1.051
1.117
0.952

38

than Pearson) correlations were also speciﬁed but were nearly identical, and only the Pearson
correlations are reported. The correlations among the variables used to create the proﬁles
and pre-interest, which range from r = .08 through r = .60 (all statistically signiﬁcant),
represent low to moderate relations among these variables. The relations among the aspects
of work with data, which ranged from r = .19 to r = .50 (also all statistically signiﬁcant),
also represented moderate relations among these variables. Relations between the variables
used to create the proﬁles as well as pre-interest and the aspects of work with data were less
noteworthy. For pre-interest and the aspects of work with data, the values ranged from r =
-.10 to r = .11 (with only the r value of .11 being statistically signiﬁcant) representing small
relations. For the variables used to create the proﬁles and the aspects of work with data, the
values ranged from r = -.06 to r = .03, with only a few of the negative relations (those with
r values of -.05 and -.06) being statistically signiﬁcant.

39

Table 4.2: Correlations among the continuous study variables

Pre-
interest

Cog.
eng.

Beh.
eng.

A .
eng.

Chall. Comp. Ask.

Obs.

Gen.

Mod.

Com.

4
0

Pre-interest
Cog. eng.
Beh. eng.
A . eng.
Chall.
Comp.
Ask.
Obs.
Gen.
Mod.
Com.

.14
.13
.12
.15
.06
-.18
.11
-.08
-.03
-.10

.60
.59
.30
.40
.02
.01
.02
.02
.00

.57
.27
.41
.01
.03
.02
.01
-.02

.27
.47
.01
-.01
-.03
.01
-.05

.08
-.01
-.02
-.01
.03
-.06

-.01
-.00
-.05
-.00
-.03

.38
.31
.42
.42

.30
.19
.20

.35
.38

.50

Table 4.3: Proportion of signals for which each of the aspects of work with data was present

Aspect of Work with Data Proportion of Instructional Episodes
0.381
Asking Questions
0.242
Making Observations
0.432
Generating Data
Data Modeling
0.288
0.436
Communicating Findings

N
90
57
102
68
103

4.3 Results for Research Question #1

4.3.1 Frequency of the aspects of work with data

Of the 236 instructional episodes used in the analysis, 170 (72%) were coded as involving one
or more of the ﬁve aspects of work with data. As a reminder, the instructional episode refers
to the ten-minute block of time immediately preceding an ESM signal. As presented in Table
4.3, the ﬁve aspects of work with data occurred regularly. Making observations was found to
be the least frequent of the ﬁve aspects, occurring in 24% of instructional episodes. Data
modeling was the next most frequent aspect, occurring in 29% of the episodes, followed by
asking questions (38%), generating data (43%), and communicating ﬁndings (again 43%).
As suggested by the proportions reported in Table 4.3, the di erent aspects of work
with data often co-occurred within a single instructional episode. On average, there were
1.86 (SD = 1.61) aspects of work with data present during each instructional episode. This
value indicates that, on average, youth were engaged in around two of aspects of the work
with data during each instructional episode. There was a considerable amount of variation
in the extent to which these types of work with data were supported in each program. The
frequencies by the program are presented in Appendix B.

41

4.3.2 The nature of work with data

The open-ended, qualitative approach used to understand the speciﬁc nature of youths’ work
with data showed the variety of ways each of the ﬁve aspects was enacted in the context of
the programs.

4.3.2.1 Asking questions or identifying problems

Among the instructional episodes that involved asking questions, qualitative descriptions
revealed that around one-third (39/90, or 43%) involved youth working to understand the
phenomenon or problem they were investigating. When doing so, youth were focused on
actively constructing predictions and hypotheses about phenomena. For example, in an
instructional episode during the Ecosphere program in which youth constructed inclined tables
to study how water moved throughout the ecosystem, the youth activity leader prompted
youth to generate hypotheses of what would happen when water was poured onto the table,
before pouring the water.

Other instructional episodes involved questions that were not focused on predicting
or hypothesizing, but instead on asking a more general type of question (21/90; 23%), or
involved the instructor (but not youth) posing questions or identifying problems (14/90;
15%). In the former case, youth were found to be asking more general questions about
understanding the assignment, task, or even the phenomena. For instance, in the Marine
Investigators program, youth visited a water treatment site and were provided opportunities to
ask questions about what they observed: However, youths’ questions were not questions that
could then be answered with empirical data, but were rather to clarify their understanding. In
the latter, instructors were asking youth questions (i.e., questions to elicit youths’ conceptual
understanding). The remaining (23/90; 25%) episodes represented themes that were not very
common or systematic.

42

4.3.2.2 Making observations

In the instructional episodes when the STEM-PQA revealed that youth were making obser-
vations, the vast majority (53/57, 86%) of these were focused on observing phenomena in
the ﬁeld, or, in the case of engineering-focused programs, noticing what was going on with
a particular design. For instance, in the Building Mania program, youth constructed Rube
Goldberg machines. During this activity, youth were prompted by activity leaders to notice
how changes in their design, which they recorded, led to di erences in how far objects were
launched or rolled.

In a small number of cases making observations were focused on making observations
not of phenomena, but of something more general (10/57; 18%). For example, in the
Adventures in Mathematics program, youth observed other youth or the activity leader
working through a mathematics problem, but not one that youth identiﬁed or discussed. The
remaining (17/57; 30%) new uncommon or unsystematic.

4.3.2.3 Generating data

In less than half (40/102; 39%) of the episodes that involved generating data, youth were
writing down their observations of a phenomenon, recording information from experiments,
or recording the results of a trial (in engineering contexts). For example, in the Marine
Investigators program, youth collected pieces of recyclable plastic, bringing them back to the
classroom and counting them for each location they were collected.

In a minimal number of cases (2/102; 2%), youth collected but did not write
down data. For instance, again in Marine Investigators, youth used nets to collect saltwater
organisms, which they then transported in buckets back to the classroom setting for subsequent
analysis. Very often, and in the other half of episodes (60; 59%) related to this aspect of work
with data, how youth generated data were not very systematic or identiﬁable. This code was
present when youth point out the relations between points in a scatter plot ﬁgure (which the
instructor then translated into an equation) during the Uptown Architecture program. In

43

another instructional episode during the Zoology Partners program, this code was present as
youth solved riddles while traveling on a bus to a community site.

4.3.2.4 Data modeling

A majority (37/68, 54%) of the instructional episodes identiﬁed as data modeling were focused
on youths’ uses and development of statistical and mathematical models. For example, in
the Comunidad de Aprendizaje program, youth accessed nationally-representative data and
were tasked to solve problems, like ﬁnding out what percentage of people engage in particular
activities, like donating to charity. In another example, in the Marine Investigators, youth
participated in activities designed to help them understand water quality in their ecosystem.
Youth collected trash from sites around their community (in di erent “districts”) and then
brought the trash and recyclable plastic back into the classroom. Then, the youth activity
leaders involved youth in an ambitious data modeling activity. The aim was to ﬁgure out
how much plastic enters local waterways. As a part of this activity, youth activity leaders
asked youth not only to determine the quantity of trash that entered the waterways but
asked youth about why youth thought about and used math in particular ways. For example,
youth activity leaders pressed youth to consider how the quantity of trash collected could be
extrapolated across the entire city over the course of the year). For example, during Marine
Investigators, the youth activity leader.

Other times (4/68; 6%), data modeling occurred through solving equations provided
by the youth activity leader, even when related to real-life (as in buying groceries, how money
is spent, and how to budget, in Comunidad de Apendizaje). This type of work with data
di ers from descriptions of data modeling as the equations provided to youth did not often
involve modeling variability, a key component of work with data (see section 2.1). Instead,
when youth solved equations, there was one often correct answer that the activity leader
sought to guide youth to. Additionally, using equations di ered from deﬁnitions of data
modeling because learners were not developing statistical models themselves, a key part

44

of data modeling (Hancock et al., 1992; Lehrer, Kim, & Schauble, 2007; Lehrer, Kim, &
Jones, 2011). Using already-created equations may be less engaging than activities that
challenge youth to use and develop data models from scratch, work which may be more
engaging, especially when youth perceive themselves to be good at such activities (Schneider
et al., 2016; Sherno  et al., 2016). During some episodes (6/68; 9%), data modeling involved
reasoning about a model based on data with ambiguous origins. In many of these cases, the
model was a physical model, such as during the Crazy Machines program, in which youth
saw how changes to their Rube Goldberg machine worked or did not work. Such uses were
similar to those in which the youth activity leader, rather than the youth (3/68; 4%) used the
model (to convey ideas to youth). For instance, in the Marine Investigators program, a youth
activity leader used a plush toy seal designed to teach youth about anatomy and the dangers
of aquatic mammals consuming trash and recyclables. The remaining data modeling-related
episodes (18/68; 26%) were not systematic or very common.

4.3.2.5 Interpreting and communicating ﬁndings

In less than one-half (39/103, 38%) of the instructional episodes in which youth were
interpreting and communicating ﬁndings, youth were sharing what they found from an
investigation or the results of using the product they designed. For instance, in the Comunidad
de Aprendizaje program, youth participated in an activity designed to support their thinking
about creating a product to bring to market; the youth activity leaders described this as being
akin to the television show the Shark Tank. In one instructional episode, the youth activity
leader asks youth to think of an idea that would make an investor willing to invest in. In this
episode, youth shared their ideas, describing what their ideas was, why it was a good idea,
how much they could sell it for, and what their proﬁt would be (all while ﬁelding questions
from youth activity leaders and their peers). Interpreting and communicating ﬁndings was
also commonly present in instructional episodes in which youth were debating the ﬁndings of
an investigation, such as the results of calculations for the number of recyclables entering

45

waterways (in Marine Investigators).

In the other instructional episodes that were not focused on youth sharing what
they found from an investigation, youth were most commonly communicating about topics
other than the results of an investigation or design process (3/103, 3%). For example,
during these episodes, youth tried to ﬁnd out the answer to a discrete question posed by the
youth activity leader or the youth activity leader. In other, episodes focused on interpreting
and communicating ﬁndings (4/103, 4%), the youth activity leader, and not youth, were
communicating the ﬁndings of an investigation. For instance, during the Building Mania
program, the youth activity leader noted youth struggled to ﬁnd a business’ proﬁt and
loss, and so worked through and shared the results of his problem-solving. In this type of
interpreting and communicating ﬁndings (the youth activity leader doing the interpreting and
communicating), youth commonly engaged in other aspects of work with data (i.e., generating
data), but the youth activity leader compiled, modeled, and then interpreted the data that the
youth generated, rather than youth doing such activities themselves. The remaining episodes
focused on communicating ﬁndings (57/103, 55%) were not very systematic or common.

4.4 Results for Research Question #2: What proﬁles of youth
engagement emerge from experiential data collected in the
programs?

On the basis of ﬁt statistics and statistical tests (see Table 4.4) and concerns of
interpretability and parsimony, a solution with six proﬁles of engagement was selected. Note
that only models associated with the varying means, equal variances, and covariances ﬁxed
to 0 speciﬁcation (referred to as the “model 1 type”) and the varying means, equal variances,
and equal covariances (referred to as the “model 2 type”) converged, and so only solutions
associated with these two model speciﬁcations were considered.

46

Number of Proﬁles
Model 1

3
4
5
6
7

4
7

-19453.38
-19196.33
-18817.93
-18648.78
-18407.23

39082.59
38616.44
37907.60
37617.26
37182.11

39012.69
38527.47
37799.57
37490.17
37035.95

Model 2

3
4
5

-18897.06
-18659.68
-18474.83

38049.88
37623.06
37301.33

37948.20
37502.32
37161.52

Table 4.4: Solutions for models that converged

AIC

BIC

SABIC Entropy BLRT

Proﬁle Sizes

0.794
0.811
0.913
0.888
0.886

0.880
0.922
0.901

881.519 (0)
514.107 (0)
756.788 (0)
338.296 (0)
523.141 (0)

773, 897, 1288
415, 427, 920, 1288
345, 375, 643, 667, 928
345, 370, 450, 488, 638, 667
181, 222, 317, 450, 568, 569, 651

421.343 (0)
474.773 (0)
304.938 (0)

763, 954, 1241
135, 688, 1029, 1106
130, 271, 573, 871, 1113

For solutions associated with model 1, the decrease (indicating a preferred model)
in the information criteria (AIC, BIC, SABIC) becomes smaller as the number of proﬁles
increases from 5 to 6 and 6 to 7. The BLRT suggests that, until the log-likelihood is not
replicated, every more complex model be selected. The six and seven proﬁle solutions are
compelling because both show proﬁles that are distinguished by dimensions of engagement
and its conditions (challenge and competence) and have lower values on the information
criteria than the solutions with fewer proﬁles. For solutions associated with model 2, only
those associated with 2-5 proﬁle solutions were associated with log-likelihoods that were
replicated. For these four models, the log-likelihood decreased in a mostly consistent way, such
that changes in the decrease are not as evident as those associated with model 1. The BLRT
suggest that the more complex model be selected and so did not provide clear information
about which solutions are to be preferred. Taken this information into consideration, either a
model one type, six proﬁle solution or a model one type, seven proﬁle solution was found to
be most reasonable. The seven proﬁle solution, described in Appendix D, was used for the
analyses for research questions 4 and 5. The results from these analyses were comparable to
those for the six proﬁle solution, and so the six proﬁle solution was chosen on the basis of
parsimony and its greater interpretability.

The result of this model selection process was the estimation of six distinct proﬁles
identiﬁed from the data, as presented in Figures 4.1 and 4.2. Figure 4.1 shows the proﬁles
with variables that were centered to have a mean of 0 and a standard deviation of 1. Thus, the
y-axis for this plot is labeled “Z-score”). Figure 4.2 shows the proﬁles with the raw data (not
transformed). Thus, the y-axis for this plot is labeled “Value.” This solution represents the
proﬁles of engagement identiﬁed to answer this research question and for use in subsequent
analyses.

The two plots are presented because they provide a di erent view into the com-
position of the proﬁles: Those with the centered variables highlights positive and negative
departures from the mean value for each variable, making di erences between the proﬁles

48

e
r
o
c
s
−
Z

1.0

0.5

0.0

−0.5

−1.0

Universally low (n = 667)

Only behavioral (n = 370)

Only affective (n = 345)

All moderate (n = 638)
Eng. and comp. but not chall. (n = 450)

Enjoying
Working Hard
Learning New
Challenge
Competence

Full (n = 488)

Figure 4.1: The six proﬁles of engagement (with variable values standardized)

e
u
l
a
V

4

3

2

1

Universally low (n = 667)

Only behavioral (n = 370)

Only affective (n = 345)

All moderate (n = 638)
Eng. and comp. but not chall. (n = 450)

Enjoying
Working Hard
Learning New
Challenge
Competence

Full (n = 488)

Figure 4.2: The six proﬁles of engagement (with raw variable values)

49

distinct. The plot with the raw data instead highlights the reported values of the variables,
emphasizing the values of the variables in the proﬁles in the same units that youth were
asked to consider when they responded (and potentially highlighting similarities that may
seem very di erent in the plot with the centered data).

The six proﬁles are characterized by both varying levels on both the indicators of
engagement (cognitive, behavioral, and a ective) and perceptions of challenge and competence.
Also, the number of observations across the proﬁles is relatively balanced (with no proﬁles
associated with a very large or small number of observations). The universally low proﬁle
was associated with the most substantial number of observations (n = 667), followed by the
all moderate proﬁle (n = 638); each of the other four proﬁles was associated with 300 to 400
observations. The results for research questions 3-5 use this solution and the six proﬁles in
subsequent analyses.

A MANOVA was carried out to determine whether the values of variables di er
across the proﬁles, with multiple ANOVAs used to determine which variables (and for
which proﬁles) there were di erences. Note that for the proﬁles (and their presentation in
Figures 4.2 and 4.3 and Table 4.5), each response is associated with the probability of proﬁle
membership at a particular moment. Because, across all responses, the highest probability
for each response was on average quite high (the entropy statistic was .888), the probability
was appropriate to use to classify each response into one proﬁle. These classiﬁcations were
subsequently used to calculate the number and percentage of responses in each proﬁle. They
were also used for the analyses comparing the mean levels of each variable across proﬁles
(with a MANOVA and with the follow-up ANOVAs).

The MANOVA was statistically signiﬁcant (Pillai-Bartlett = 0.633, p < .001). The
table with the raw values, with subscripts indicating values the mean values that were not
statistically signiﬁcantly di erent is presented in Table 4.5. Note that the F-test associated
each ANOVA was also statistically signiﬁcant. Descriptions of each the proﬁles taking account
of their size (in terms of the number of responses for which the proﬁle was most likely), their

50

variable values, and what the proﬁles suggest about youth engagement follow.

51

Table 4.5: Raw variables values by proﬁle

5
2

Proﬁle
Universally low
Only behavioral
Only a ective
All moderate
Eng. and comp. but not chall.
Full

Working Hard Learning New Enjoying Challenge Competence
2.327
2.7785
2.9545
2.9535
3.6046
3.6316

1.538
1.641
3.330
3.110
3.8223
3.8813

1.775
2.1324
2.1914
2.489
1.276
3.742

1.550
3.292
1.670
3.060
3.9091
3.9591

1.766
2.4842
2.5162
2.826
3.487
3.801

Note. The subscripts indicate the mean values subscripts indicating values that were not statistically signiﬁcantly di erent on
the basis of the ANOVA analyses.

A universally low proﬁle consisted of a substantial proportion of responses (22.55%)
was identiﬁed. This proﬁle was characterized by low levels of working hard, learning something
new, and enjoying the activity, and perceptions challenge and competence. For responses
in this proﬁle, mean values were lower than their values in every other proﬁle for every
variable except challenge, which was even lower in the *engaged and competent but not
challenged proﬁle. Note that concerning their raw values and not only their levels relative to
the levels of the variables for the other proﬁles, youth report very low levels (below two on
the one-four scale used) of all of the variables. In all, this proﬁle reﬂects very low levels of
youth engagement during the speciﬁc instructional episodes during which youth were signaled
to respond.

An only behaviorally engaged proﬁle with a small proportion of responses (12.51%)
was identiﬁed. This proﬁle was characterized by moderate levels of working hard, very low
enjoyment of the activity, and moderate levels of learning something new and challenge and
competence. The levels of reporting learning something new, challenge, and competence
were not distinguishable from those found in the responses that make up the only a ectively
engaged proﬁles. Levels of working hard, an indicator of behavioral engagement, was higher
than in every proﬁle except fully engaged and engaged and competent but not challenged.
These responses suggest that youth perceive themselves to be working hard, but to not
be enjoying what they were doing and to not report learning something new, nor to be
particularly challenged or good at what they were doing when signaled.

An only a ectively engaged proﬁle with a small proportion of responses (11.66%)
was identiﬁed. This proﬁle was characterized by moderate levels of enjoyment, low levels
of hard work, and moderate levels of learning something new, challenge, and competence.
Levels of competence were the same as in the all moderate proﬁle. Youths’ reports of enjoying
what they were doing at the time they were signaled, an indicator of a ective engagement,
was higher than in every proﬁle except fully engaged and engaged and competent but not
challenged. When youth report this response, they enjoy what they were doing, but were not

53

working hard or learning something new, nor do youth report being challenged by or good at
the activity they were doing.

An all moderate proﬁle with a large proportion of responses (21.57%) was identiﬁed.
This proﬁle was characterized by moderate levels of the three indicators of working hard,
learning something new, enjoying the activity, challenge, and competence. Levels of all of
the variables were, on average, lower for the responses that make up this proﬁle than among
the responses associated with the engaged and competent but not challenged nor the full
engagement but were still quite high on the one-four scale used. In sum, for youth reporting
all moderate engagement were engaged, but may have the potential to be more highly engaged
(and challenged by and good at the activity).

An engaged and competent but not challenged proﬁle with a modest proportion of
responses was identiﬁed (15.21%). This proﬁle was characterized by high levels of working
hard, learning something new, enjoying the activity, and competence, but low levels of
challenge. Levels of competence, enjoying, and working hard were identical between the
responses associated with this proﬁle and the responses associated with the fully engaged
proﬁle, while levels of challenge were very low: levels of challenge for these responses were
lower than those for every other proﬁle. Levels of learning something new were slightly lower
than those in the responses that make up the fully engaged proﬁle but were higher than their
levels in the other four proﬁles. This proﬁle suggests youth can be highly engaged, while not
being challenged by the activity they were involved in at the time they were signaled.

A full proﬁle with a modest proportion of responses (16.50%) was identiﬁed. This
proﬁle was characterized by high levels of working hard, learning something new, enjoying the
activity, challenge, and competence. These responses reﬂect a very high level of engagement,
both relative to the other proﬁles and in absolute terms: All of the mean levels were above
3.50, and, for working hard, youths’ responses averaged 3.96 on a one-four scale. Thus, when
youth report engagement in ways that were associated with this proﬁle, they report being
challenged and good at what they were doing and, on the basis of these variables and the

54

Table 4.6: Intra-class correlation (ICC) values for each of the three levels

Proﬁle

Universally low (n = 667)
Only behavioral (n = 370)
Only a ective (n = 345)
All moderate (n = 638)
Engaged and competent but not challenged (n = 450)
Full (n = 488)

indicators of engagement, youth very highly engaged.

Instructional
Episode
0.006
0.006
0.004
0.015
0.009
0.031

Youth Program

0.267
0.093
0.262
0.310
0.100
0.432

0.023
0.009
0.003
0.000
0.000
0.019

4.5 Results for Research Question #3: What sources of variability

were there for the proﬁles of engagement?

For all six proﬁles, the ICCs (for the model with only the youth, instructional episode,
and program levels themselves, but not variables at the levels) represent the systematic
variability (the proportion of variance explained) associated with each of the levels for each
proﬁle. Thus, the di erent levels can have di erent proportions of variance explained for
di erent proﬁles, as presented in Table 4.6. The systematic variability at the youth level,
for example, could be .10 for the Full proﬁle and .025 for the Universally Low proﬁle. At
the program level, the ICCs were found to be small, with values ranging from 0.00 to 0.023,
suggesting that little variability can be explained by the program. For the instructional
episode level, the ICCs were also small, ranging from 0.004 to 0.01. Finally, at the youth
level, the ICCs ranged from .093 to .432.

In terms of ICCs at youth level across the six proﬁles, the value for the youth-level
ICC was highest for the Full proﬁle (ICC = .432), suggesting that some youth have a
strong tendency to be fully engaged (possibly due to their initial interest or other individual
characteristics and di erences). The other proﬁle characterized by a consistent pattern across
all of the variables–the Universally low proﬁle–had a modest value for the ICC at the youth
level (ICC = .267). Finally, a signiﬁcant amount of variability is associated with the residual

55

(variance that was not associated with the program, instructional episode, or youth levels).
This suggests that there is wide variation in youths’ responses that may not be readily
explained or predicted by variables at one level alone. Remaining unexplained variability was
captured by the residual term. Some youth from particular programs may engage during
some episode instructional episodes in very high or low ways that were not captured by
modeling the variability at each of these levels alone.

The ICCs lend insight into the sources of variability for a speciﬁc proﬁle; within-
youth stability in terms of how frequently they reported particular proﬁles could lend further
insight by considering variability across proﬁles. This analysis can be particularly useful for
understanding variability at the youth level, which the ICCs show to be associated with the
most systematic variability. Each youth has a most-frequently reported proﬁle. Results show
that for some youth, the proﬁle was very dominant, occurring in a substantial proportion
of youths’ responses; for others, it occurs not that frequently, meaning that youth report a
variety of di erent proﬁles.

As presented in Figure 4.3, the mean proportion of responses for each youth in
the proﬁle they reported most varied widely across youth. Speciﬁcally, on average, youth
reported their most-reported proﬁle in .540 (SD = .194, min = .182, max = 1.00) of their
responses. There was a small number of youth who reported the same proﬁle in all of their
responses, but for most youth, the proﬁle they reported most made up only a portion of all
of their responses. For most youth, the most common proﬁle was observed just over 50%
of the time. Instructional episodes that involved work with data were compared to those
without work with data. Like for the other models, these models were speciﬁed with the
dependent variable as the probability of a response being associated with a proﬁle for each of
the six proﬁles. However, there was no di erence in terms of the regression (—) coe cients
associated with this variable for any of the six proﬁles.

In sum, these ﬁndings show that there was substantial variability in the proﬁles
present at the youth level. Less variability was explained by either the program youth were in

56

h
t
u
o
Y

 
f
o

 
r
e
b
m
u
N

20

15

10

5

0

0.25
Proportion of responses for each youth in the profile they reported most

0.75

0.50

1.00

Figure 4.3: Histogram of the proportion of responses for each youth in the proﬁle they
reported most

or the nature of the particular instructional episode present when youth were signaled. These
results set the stage for those for the next two research questions, on the relations between
the aspects of work with data (for research question #4) and the youth characteristics (for
research question #5) and the proﬁles of engagement.

4.6 Results for Research Question #4: Aspects of work with data

and engagement

To understand how aspects of work with data were related to engagement, six
analytic models were speciﬁed – one for each engagement proﬁle.
In each model, the
dependent variable was the probability of a response being classiﬁed in a particular proﬁle
(for example fully engaged), as determined by the Latent Proﬁle Analysis. The ﬁve aspects
of work with data were the predictor (or independent) variables. Various aspects of work
with data tended to co-occur and so simultaneously entering indicators for all ﬁve aspects
served to isolate the association for any single aspect while controlling on the presence of
the others. All models also include some youth characteristics which will be used to answer

57

research question ﬁve below.

Associations between the ﬁve aspects of work with data and the six engagement
proﬁles are presented in the bottom half of Table 4.7. In this table, each column represents
the output from one of the six di erent models. As an example, the ﬁrst column reports
the coe cients for the associations between the predictor variables and the Only behavioral
proﬁle. Because the outcome was in the form of a probability (ranging from 0.00 to 1.00), it
can be interpreted as the change in the probability of a response being associated with each
proﬁle. Note that the p-values were calculated using the most conservative and recommended
by recent research Kenward-Rogers approximation (Halekoh & Hojsgaard, 2014).

The only engagement proﬁle that was signiﬁcantly associated with any aspects of
work with data was the Full proﬁle (see the column with the column name Full for these
results). When program activities involved modeling data, youth were around 3% more likely
to be fully engaged (— = 0.034 (0.017), p = .020; partial R2 = .002). In other words, when
program activities included modeling data, youth were more likely to report working harder,
learning more, enjoying themselves more, and feeling more competent and challenged.

Youth were also more likely to be in the Full engagement proﬁle when program
activities included generating data (— = 0.027 (0.015), p = .033; partial R2 = .002). These
particular program activities increased the probability of full engagement by around 3%.
To sum up these two ﬁndings, modeling data and generating data were associated with a
(very) positive form of engagement, that exhibited by the Full proﬁle. However, the e ect
sizes indicate quite small e ects in substantive terms. Note that interactions between the
individual aspects of work with data and youth characteristics were also speciﬁed. However,
none of these interactions were found to be statistically signiﬁcant.

Sensitivity analysis was carried out for the statistically signiﬁcant two e ects was
carried out to determine just how robust they were. This follow-up analysis revealed that
the e ect of modeling data on Full engagement much more robust than that for generating
data: 9.835% of this e ect (of data modeling) would have to be due to bias to invalidate the

58

inference about its e ect. For generating data, only 1.884% of the e ect of generating data
would need to be due to bias to invalidate the inference about its e ect. These values were
not minuscule but were also not very large (Frank, 2003). So, while statistically signiﬁcant,
the e ect of data modeling seems to be a more robust e ect than the e ect of generating
data, which does not seem to be a very robust (and should, therefore, be interpreted with
some caution).

59

Table 4.7: Results of mixed e ects models with the interactions between interest and other characactistics and the composite for
work with data

Proﬁle

Universally low
—(SE)

Only behavioral
—(SE)

Only a ective
—(SE)

Eng. and comp.,
not chall.
—(SE)

All moderate
—(SE)

Full —(SE)

Youth characteristics

-0.047 (0.022)

-0.013 (0.012)

-0.012 (0.019)

0.039 (0.016)*

0.007 (0.01)

0.018 (0.021)

Gender-

0.06 (0.037)+

0.019 (0.019)

-0.038 (0.033)

0.025 (0.028)

-0.02 (0.018)

-0.035 (0.037)

-0.01 (0.052)

0.031 (0.026)

6
0

status
Aspects of Work With Data

-0.015 (0.018)
0.003 (0.018)

0.015 (0.015)
0.013 (0.015)

-0.076 (0.046)

-0.012 (0.04)

0.018 (0.025)

0.043 (0.053)

0.023 (0.017)+
0.007 (0.017)

-0.011 (0.015)
0.009 (0.015)

0.004 (0.014)
-0.017 (0.014)

-0.019 (0.016)
-0.025 (0.016)

-0.014 (0.017)

0.014 (0.014)

0.012 (0.016)

-0.014 (0.014)

-0.02 (0.013)

0.027 (0.015)*

0.004 (0.019)

-0.023 (0.016)

-0.004 (0.018)

0 (0.015)

-0.012 (0.015)

0.034 (0.017)*

0.002 (0.018)

0.018 (0.015)

-0.011 (0.017)

0.004 (0.015)

0.016 (0.014)

-0.027 (0.016)

Pre-
interest

Female
URM

Asking

Observing

Generating

Modeling
Commu-

nicating

Note. *: p <.05; +: p < .10

4.7 Results for Research Question #5: Youth characteristics and

engagement

Associations between youth characteristics and the six proﬁles are reported in the
top half of Table 4.7. Youth who enter the program with higher levels of interest (in STEM)
were more likely to report being in the engaged and competent but not challenged proﬁle (—
= 0.039, p = .009; partial R2 = .001). In other words, youth who were more interested at
the outset of the program report working harder, learning more, enjoying themselves more,
and feeling more competent when they were involved in program activities, though they also
report lower levels of challenge. For this e ect, 17.879% would be needed to invalidate the
inference, suggesting a moderately robust e ect.

In terms of youths’ pre-program interest, these analyses show that youth who enter
the program with higher levels of interest (in STEM) were more likely to report being in
the Engaged and competent but not challenged proﬁle (— = 0.039, p = .009; partial R2 =
.001). For each one-unit increase in pre-program interest in STEM, youth were around 4%
more likely to report this proﬁle. In other words, youth who were more interested at the
outset of the program report working harder, learning more, enjoying themselves more, and
feeling more competent when they were involved in a program’s activities, though they also
report lower levels of challenge. For this e ect, 17.879% would be needed to invalidate the
inference, a slightly larger value for the follow-up sensitivity analysis than those found for
the (statistically signiﬁcant) relations involving the aspects of work with data, suggesting a
moderately robust e ect.

There were not any statistically signiﬁcant e ects of youths’ URM status. This
lack of relations between URM status and youth engagement may be a function of the large
proportion of youth from under-represented (in STEM) racial and ethnic groups. Hispanic
(48%), African American or Black (36%), and youth who identify as being from multiple
racial and ethnic groups (3%) made up 87% of the youth in the programs, so there were not
many youth not from under-represented groups in the sample, suggesting that the absence

61

of ﬁndings may be due to this small sample (and low statistical power). Nevertheless, no
relations between URM status and youths’ engagement were found, indicating that there is
at least no evidence that youth from such backgrounds do engage in di erent ways.

These (somewhat minimal) ﬁndings for the youth characteristics were more surprising
than those observed for the aspects of work with data. The results of research question #3,
on the sources of variability for the proﬁles of engagement, suggested that there was much
systematic variability at the level of the youth (there were large ICCs at the youth level,
with smaller ICCs at the instructional episode level). Because pre-interest, gender, and URM
status were variables at this level, it could be expected that they would have meaningful
relations with the proﬁles of engagement. However, it appears that the particular youth
characteristics considered were not useful at explaining much of this variability; possible
reasons why are discussed in the next section.

62

CHAPTER 5

DISCUSSION

Each of the disciplines that contribute to STEM learning - science, technology and computer
science, engineering, and mathematics - involve work with data. In this study, engagement
was used as a lens to understand the experience of youth working with data during summer
STEM programs.
In particular, ﬁve aspects of work with data, a) asking questions, b)
observing phenomena, c) constructing measures and generating data, d) data modeling, and
e) interpreting and communicating ﬁndings, were occurred regularly in the programs. There
were some examples of ambitious activities centered on working with real-world data as well
as some that highlight substantial heterogeneity in how work with data was enacted.

I identiﬁed six proﬁles of engagement using LPA. These proﬁles represented di erent
conﬁgurations of how youth were working hard, learning, enjoying themselves, and feeling
challenged and competent at the time they were signaled as part of the ESM approach.
Relations of the ﬁve aspects of work with data and youth characteristics (pre-program interest
in STEM and youths’ gender and status in terms of being a member of under-represented
groups in STEM) were, overall, not strongly related with the proﬁles of engagement, though
some key ﬁndings were identiﬁed. Generating and modeling data were both related to the
most potentially beneﬁcial proﬁle (full engagement), one characterized by high levels of all
ﬁve of the engagement variables.

This study suggests that work with data and contemporary engagement theory as
interpreted in this study can serve as a frame to understand what youth do in summer STEM
programs. These ﬁndings also show the value of an innovative method, ESM, and an analytic
approach designed to identify engagement holistically, LPA, that together to provide some
access to youths’ experience in-the-moment of the activities they were involved in during the
program. Data, and how youth and students in K-12 settings can themselves work with data,
is an important, yet perhaps under-emphasized part of STEM learning. In the remainder of

63

this section, I discuss key ﬁndings with respect to a) work with data, b) youths’ engagement,
and c) what relates to youths’ engagement. Also, some limitations and recommendations for
future research as well as implications for practice are identiﬁed and described.

5.1 Key ﬁndings related to work with data in summer STEM

programs

Results showed that work with data was common in the summer STEM programs.
There was variability in which aspects of work with data was present: Making observations,
in some form, occurred during 24% of the program’s time, for example, while generating data
and communicating ﬁndings both occurred more frequently, during 43% of the instructional
episodes. These ﬁndings, broadly, suggest that work with data occurred enough that we might
expect to see di erences in youths’ engagement. They align with what may be expected given
past research: Such programs are designed to engage youth in the practices, including and as
I argued earlier especially those relating to work with data, of STEM domains (Dabney et
al., 2012; Elam et al., 2012). Even still, these are the ﬁrst results of this kind (in terms of the
proportion of the time spent in the programs). Using video-recording data and a sampling
strategy that can provide insight into the amount of overall time spent was an important
component of achieving these ﬁndings. While there are no other results of this particular kind,
a related, an area of related work concerns other studies that have used the PQA measure.
Some research reports call for greater use of measures (such as the PQA) in the study (and
evaluation) of summer and outside-of-school STEM programs (e.g., Yohalem et al., 2005).
As one example of such a study that used the PQA (but one that is not focused on STEM),
Smith et al. (2012) reported ﬁndings from a continuous improvement intervention, ﬁnding
that the intervention positively impacted the quality of instruction in the programs.

In addition to work with data being common, I found it was highly varied in how
it was enacted. In the course of the four-week summer STEM programs, youth engaged in
what can be described as ambitious, speciﬁc, and potentially highly engaging ways of being

64

involved in work with data. For example, when generating data, many times (in 47% of the
episodes that involved this aspect) youth recorded their observations; when modeling data,
youth were involved (in 72% of the episodes) in the use of statistical and mathematical models
of real-world phenomena. When interpreting and communicating ﬁndings, youth regularly
(during 48% of episodes) had opportunities to share (with other youth in the program) what
they found or created as a result of their earlier investigations or work.

What occurred during the rest of the program’s time was also notable. When youths’
questions, for example, were not focused on predicting or hypothesizing about what they were
exploring, the type of question was more general, or was instructor-led, rather than driven by
youth. These instructor-driven forms of work with data were not aligned with recent reform
e orts (i.e., National Research Council, 2012; NGSS Lead States, 2013; National Governors
Association Center for Best Practices, Council of Chief State School O cers, 2010), but could
be expected given past research pointing out variability in what evidence and data mean,
especially in science education settings (McNeill and Berland, 2017; Lehrer & Schauble, 2015).
Also of note was the frequency of these three aspects of work with data overall: They occurred
much more frequently than the two (making observations and data modeling) for which a
larger proportion of their enactment was more in-line with policy and curricular standards.
The type of activities that may be the most demanding for youth was still common (and may
spark youths’ engagement) but was not quite as common as the overall frequencies presented
for the quantitative would suggest.

Past research does point out a heterogeneity in how work with data was enacted
in education similar to that found in this study. For example, Hancock et al. highlight the
use of “data to solve real problems and to ask authentic questions” ( p. 337). Research
on generating data emphasized an aspect not very much the focus of the present research,
namely, structuring data into spreadsheets (Konold, Finzer, & Kreetong, 2017; Lehrer & Kim,
2009). This suggests a reason why youth were able to ask questions and ideas for how they
might do so more: Make activities in summer STEM program youth-centered, rather than

65

instructor-centered. Research on the data modeling aspect of work with data highlights the
use of statistical models much more than the physical models which were sometimes found to
be a way in which youth engaged in data modeling (Petrosino, Lehrer, & Schauble, 2003;
Lesh, Middleton, Caylor, & Gupta, 2008; Lee, Angotti, & Tarr, 2010). Nevertheless, many
of the ways youth engaged in data modeling aligned with this past research, particularly
when the goal of the activity was to model variability. This past research that encouraging
youth to consider summaries of data, such as the mean and standard deviation, may be a
promising way for them to engage more deeply in data modeling (Lehrer, Kim, & Schauble,
2007; Lehrer & Schauble, 2004). In this way, some (but not all) of the aspects work with data
aligned with past research; when they align, they are encouraging, and when they do not, I
present some ideas for how to involve youth in more engaging aspects of work with data.

5.2 Key ﬁndings related to engagement

Six proﬁles of engagement were found using a rigorous model selection approach. It
is important to note that LPA is an exploratory approach: The number and nature of the
proﬁles identiﬁed were found through a rigorous and systematic approach, but this is not
a guarantee that the same number and make-up of proﬁles would emerge in other samples
and other contexts: These proﬁles should be considered as initial evidence, and not as proof
that these are the six proﬁles of engagement that will exist in all settings. The number of
proﬁles found is broadly similar to that found in past research. Six is the same number of
proﬁles of engagement identiﬁed in recent, past research and the similar number provides
further information about the nature of engagement in educational contexts: Schmidt et
al. (2018) found six proﬁles of engagement. Their proﬁles were constructed on the basis of
the indicators (cognitive, behavioral, and a ective) of engagement, and not perceptions of
challenge and competence.

As Schmidt et al.‘s study is the only other to examine engagement proﬁles, another
point of comparison is other outcomes that are di erent from but related to engagement,

66

such as youths’ (and students’) achievement goals (see Wormington and Linnenbrink-Garcia,
2017, for a review in educational settings). Wormington and Linnenbrink-Garcia (2017)
report that, usually, a smaller number, with only two of the 22 studies reviewed ﬁnding six
proﬁles of goals. This suggests, on the basis of this study and Schmidt et al.’s (2018) study,
that there may be a greater variety of types of engagement exhibited than, for example,
types of achievement goals. However, in addition to the di erent construct, the engagement
proﬁles were constructed on the basis of data collected via ESM, while the achievement goal
proﬁles were constructed on the basis of self-report surveys and not via ESM. Knowing the
number of proﬁles found when proﬁles are used to explore various constructs is potentially
helpful, additional information about how engagement was experienced by youth. In addition,
the greater number of groups may suggest that engagement, explored through ESM, may
demonstrate more variability in terms of how its constituent parts are experienced together
and at once. Exploring whether this greater variability was due to the method of data
collection (ESM or self-report), the construct (engagement or achievement goals), or some
other reason.

In terms of comparing the make-up of the speciﬁc proﬁles to other, past research,
little work has examined proﬁles of engagement. Schmidt et al. (2018) did examine proﬁles
of engagement, which were constructed from indicators cognitive, behavioral, and a ective
engagement (but not perceptions of challenge and competence, as in this study). Schmidt et
al. (2018) found six proﬁles, some of which partially overlap with those found in the present
study. In particular, on the basis of the items shared between the studies, a Universally low,
All moderate, and Full proﬁle were found in both studies. However, as these proﬁles are
characterized by the (uniform) level across all of the variables, this is only limited evidence
for the presence of these proﬁles in the larger population of youth engaged in science and
STEM-related learning activities.

The six proﬁles lend insight into how youth engage during summer STEM programs.
In particular, the Only behavioral, Only a ective, and Engaged and competent but not

67

challenged proﬁles were found in the present study, but not in Schmidt et al.’s (2018) study.
Youth were highly engaged (as may be anticipated given the goals and design of such
programs), but perceive a misalignment between their (high) competence and how (not
very) challenged they were. According to past theory (e.g., Csikszentmihalyi, 1997) and
some research (e.g., Sherno  et al., 2016), such a proﬁle would be unexpected, as high
levels of engagement are expected to be associated with high levels of both challenge and
competence. In this study, a proﬁle characterized by high competence but (very) low challenge
was associated with very high engagement. This proﬁle, Engaged and competent but not
challenged, then, seems to suggest a type of engagement that may be unique and common to
summer STEM programs. Perhaps such a proﬁle may be expected given the lower stakes
(compared to formal educational settings) of summer STEM programs (and other informal
learning environments) and the degree of competence that youth–many of whom have chosen
to attend the particular program (Beymer et al., 2018)–perceive during them.

In addition to suggesting a proﬁle of engagement that is distinct to summer STEM
program, this proﬁle and the other two not found in past research have some implications for
youth activity leaders. In particular, they suggest that lower challenge may not, as would be
anticipated given theory and past research, be associated with lower engagement. Because of
this, it may be recommended that activities that are not challenging but have other possible
beneﬁts to youth (i.e., beneﬁts from activities designed to support youths’ social skills), can
be integrated into programs, along with other, more challenging activities that are also highly
engaging to youth.

These proﬁles have some implications for the study of engagement. They also have
some implications for the analysis of multidimensional data on engagement. First, they
suggest that perceptions of challenge and competence be considered in future research. This
is because some of the proﬁles were distinguished on their basis. This approach also may
be more parsimonious than including perceptions of challenge and competence as separate
predictors (i.e., Sherno  et al., 2003). In addition to these empirical reasons, past research on

68

engagement (i.e., Csikszentmihalyi, 1990) and on the proﬁle approach (Bergman & Magnusson,
1997) suggest that they are theoretically inseparable from engagement, another reason for
modeling them as they were modeled in the present study. These implications, then, are
speciﬁc to the study of engagement but also highlight some of the potential of the proﬁle
approach, as well.

5.3 Key ﬁndings related to work with data and youth character-

istics and their relations to engagement

In line with what the sources of variability would suggest, relations between work
with data were minimal, though some small, statistically signiﬁcant relations were identiﬁed.
The question of whether and how work with data relates to engagement has not been the
focus of past research on work with data. This past research has focused more on very speciﬁc
cognitive outcomes, designs (often from design-based research) for work with data, and
the challenges teachers and learners may anticipate when they are involved with particular
aspects of work with data, particularly data modeling (and accounting for variability in data).
Given the absence of research from an engagement perspective, these are new ﬁndings that
suggest, in this context, that work with data may not be strongly related to engagement in
educational contexts.

Why might these relations be so minimal? First, and foremost, the little variability
at the instructional episode level was noteworthy because it means that few relations between
variables at the instructional episode were expected. In particular, there were small ICCs
at the instructional episode level for all six proﬁles. This suggests that there was very
little systematic variability at the particular level that a variable for work with data was
at. Additionally, the ICC values found in this study were smaller than those found in the
one other past study that employed the same analytic approach (Strati et al., 2017). The
relative absence of variability at the instructional episode level may be due to the summer
STEM setting: Perhaps youth are less likely to engage di erently from instructional episode

69

to instructional episode (compared to in K-12 educational settings) because there is less
variability in what took place across the episodes or because youth perceive there to be
lower stakes for the programs’ activities and therefore do not perceive the changes in the
instructional episode as a salient factor in terms of their engagement. This consideration is
described in greater detail in the limitations section.

There are other possible reasons, though, too, for the minimal relations. One may
be that work with data is not, as carried out in these summer STEM programs- very engaging,
even accounting for the small amount of variability at the instructional episode level. The
comparison between the ﬁve individual aspects of work with data and not working with data
as well as the comparison of instructional episodes that involved any of the aspects of work
with data and those that contained none showed minimal relations. This suggests that work
with data is not more engaging than other activities carried out in summer STEM programs.
Another noteworthy possibility is that the novel analytic approach or the measures
used also had impacts; but, again, the small variability at the instructional episode level is
likely a greater factor than these, and a review of the correlations between the aspects of work
with data and the variables used to create the proﬁles showed minimal relations. This and the
previous potential explanation are explored even further in the next section, on limitations
to the present study and recommendations for future research. Taken together, it seems
that the major reason for limited relations between work with data and youth engagement is
that youth simply did not engage very di erently from instructional episode to instructional
episode.

Even so, there were some noteworthy ﬁndings that could be anticipated on the
basis of the importance of the two aspects of work with data that were found to relate
positively to youths’ engagement. In particular, both generating and modeling data were
found to be positively (and statistically signiﬁcantly) related to the Full proﬁle, suggesting
that when youth were involved in these practices, then they were more likely to be highly
engaged. In particular, given the makeup of this proﬁle, this suggests that when youth were

70

involved in these aspects of work with data, they were more likely to report high levels
of cognitive, behavioral, and a ective engagement, and high perceptions of competence
and challenge. Generating and modeling data may have such relations because they were
particularly important aspects of work with data. As Lehrer and Schauble (2006) explain,
inscriptions serve commitments: Choosing to record an observation or an idea as data involves
the process of identifying something that is worth recording and then recording the parts
that are of interest. Thus, generating data may be fully engaging to youth because it is,
generally, demanding and important with respect to work with data. Modeling, too, is an
important practice. It has been described as the central scientiﬁc and engineering practice
(Lehrer & Schauble, 2015; Weisberg, 2012), and its relations with full engagement provides
some actionable evidence for its importance in the context of summer STEM programs.
Modeling may be especially engaging to youth because such work positions learners as the
creators of new information, in addition to using models created by others to learn about
authoritative sources of information. This is one of the a ordances of modeling in teaching
and learning contexts (Berland et al., 2016; Schwarz et al., 2009). Moreover, when learners
create new knowledge through activities such as modeling, they can begin to shape the
process of constructing new knowledge in a domain, a challenge in science education contexts
(Miller et al., 2016) and likely in other STEM content areas, we well.

The null ﬁndings for the relations of asking questions, making observations, and
interpreting and communicating ﬁndings were noteworthy, too. They suggest that their
e ects were not large enough to be detected They may not be able to be detected for a
number of reasons: simply because they were very small given all of the other factors that
impact youths’ engagement, because the aspects of work with data were enacted in a myriad
of ways which may be more or less engaging, and because they simply were not as engaging.
Nevertheless, future research may seek to understand why these aspects did not relate to
engagement.

As there is no research on how work with data relates to youths’ engagement, the

71

ﬁndings associated with this research question provide some initial evidence for how some
aspects of work with data relate to youths’ engagement. These ﬁndings suggest that these
activities may not be more engaging per se. Instead, it may be the way that youth engage in
them that matters, in alignment with past research (Berland et al., 2017). While the ﬁndings
for this question were somewhat minimal, there are key ﬁndings from both the important
relationships that were found to be statistically signiﬁcant (between generating data and
data modeling and Full engagement) and from those that were not. Other samples, other
enactments of work with data, and, possibly, other analytic approaches can build on this
work to further substantiate what is known about how work with data engages youth and
other learners.

Why were there such limited ﬁndings in terms of relations between youths’ charac-
teristics and youths’ engagement? There was a lot of variability in the proﬁles of engagement
at the youth level, but there were not many relations in terms of youths’ gender, URM status,
or pre-program interest. Given past theory and research have suggested that learners’ gender,
URM status, and individual or pre-program interest can predict engagement (Bystydzienski,
Eisenhart, & Bruning, 2015; Hidi & Renninger, 2006; Sherno  & Schmidt, 2008). Despite
these surprising ﬁndings, youth with higher pre-program interest were found to be more likely
to be Engaged and competent but not challenged. This suggests that youth with a higher
interest in STEM were inclined to be highly engaged and good at what they were doing, but
were not challenged by the activities they experience. This ﬁnding is in line with past research
suggesting a relationship (direct or as a moderator) between youth characteristics (including
interest) and their engagement (Sherno  et al., 2003; Sherno  et al., 2016; Strati et al., 2017).
More speciﬁcally, this ﬁnding suggests that for youth who were particularly interested (and
those who choose to attend) summer STEM programs, what they were involved in may not
challenge them very highly. This ﬁnding has implications for past research that shows youth
who choose to attend summer STEM programs were more engaged (but that does not speak
to their degree of challenge; Beymer, Rosenberg, Schmidt, & Naftzger, 2018).

72

While the ﬁndings for this research question, like those for the relations between
the aspects of work with data and youths’ engagement, they provide some information about
how these characteristics relate to youths’ engagement. Knowing that youth who were more
interested before the beginning of the summer STEM programs were more likely to be working
hard, learning something new, and enjoying what they were learning, and perceive themselves
to be good at what they were doing but not challenged, is novel. Moreover, the null ﬁndings
suggest that other characteristics, including those measured but not included for this analysis
(such as youths’ pre-program perceptions of their competence) as well as those not measured
at all, may be considered in follow-up studies and future research. While the programs that
were involved in the study have many a ordances for work with data and for being highly
engaging for youth, they have some limitations, too, particularly with respect to support work
with data. Importantly, these were not programs explicitly designed to support work with
data; while such contexts are being developed, they are not yet widespread. Moreover, youth
may perceive the programs to have lower stakes in terms of their future. This may mean that
the individual activities that youth engage in were less connected to their engagement: Youth
instead engage in typical (to each youth) ways, rather than in ways that were much more
sensitive to changes in their context. Another possible explanation for these limited ﬁndings
may be that youth were not very challenged or were not very supported. A proﬁle with
low challenge but high competence and cognitive, behavioral, and a ective engagement was
found, suggesting that youth may be engaged and good at what they were doing, but were
not challenged: Greater challenge may be found to be associated with more full engagement,
for example.

5.4 Limitations to the present study and recommendations for

research

To summarize the previous sections, work with data was frequent but varied in
how it was enacted and proﬁles of engagement representing di erent and interpretable

73

conﬁgurations of ﬁve engagement-related variables were found, but work with data and
youths’ characteristics were not found to be very strongly related to any of the proﬁles. Some
limitations to the study that may provide insight into why such minimal relations to the
proﬁles were found and into other ﬁndings are detailed in this section.

First, the programs participating in this study were not designed especially to
support youth in work with data. Instead, the programs were designed around best practices
for summer STEM programs to support youth to engage in a wide variety of STEM-related
practices–and in other activities, such as those intended to build a sense of camaraderie among
the youth in the programs. In this study, aspects of work with data were identiﬁed and were
found to be common, but some of the heterogeneity in the nature of working with data may
be due to this reason: Planning and instruction for the programs did not aim to foster rich
work with data any more than the other activities (STEM and otherwise) that made up their
programming. In addition to the varied ways in which youth worked with data, some of the
relations of the variables for the ﬁve aspects of work with data to youths’ engagement may
be due to the ways that the variables for work with data indicated, in fact, many di erent
ways of working with data. Some of these aspects of working with data, particularly those
that were highly-speciﬁc with respect to how the data was involved and to how focused and
sustained the work with data-related activity was, may be more engaging to youth than
the others, such as those that were more general, instructor-focused, or brief. These two
types of working with data were considered the same in the variables used to predict youths’
engagement. Future research can aim to understand youths’ engagement in outside-of-school
data science programs and K-12 units, for example, that are focused more on work with
data to understand better how work with data engages youth. Nevertheless, this study does
provide insight into how work with data took place during model (i.e., designed around best
practices for such programs) summer STEM programs and how such work relates to youths’
engagement.

In a related point, it is important to point out that while outside-of-school STEM

74

programs have a ordances, they also have some distinct features as well as some limitations.
One feature is the substantial, but still limited period of time, which was around four weeks.
Another feature concerns the nature and quality of the teaching and learning that is a orded.
The contexts (including in ﬁeld settings) in which youth were engaged could spark their
engagement and could support work with data better than some K-12 classrooms. They
also have limitations, such as the chance that youth considered their time in them to be fun
and to be social, rather than educational, in nature. Of course, this is not unreasonable or
unexpected on the part of youth, but it may mean that the ways that youth engaged in the
programs as documented in this study could be unique to outside-of-school STEM programs.
In particular, engagement as reﬂected in the engaged and competent but not challenged proﬁle
may be unique to the experiences of youth in summer STEM programs: It may not be
common in K-12 classrooms. This limitation is in addition to and in the context of those
documented in earlier parts of this section, particularly, that the limited variability at the
instructional episode level may also be due to the lower stakes that learners in these contexts
may perceive.

Learning environments that deliberately support work with data over an extended
period may demonstrate di erent patterns of engagement. One key reason why this may be is
the importance of work with data being part of a cycle (and how this cycle often did not take
place in these outside-of-school STEM programs). Nevertheless, in addition to illustrating
the nature and frequency of work with data, the open-ended, qualitative coding carried out
for research question #1 also provided a lens into how work with data was (or was not)
sequenced. There were instances of youth activity leaders linking earlier to later activities.
For instance, the mathematics-focused programs, such as the Adventures in Mathematics
program, the youth activity leaders, recognizing that youth had di culty solving equations,
used duct tape–and building on an earlier activity in which youth considered what constituted
a rate–asked youth to count how many “hops” it would take someone to move from one end
of a line of duct tape to the other. The youth activity leader than asked youth to consider

75

how far they could move in one hop and to consider how they could ﬁnd out many hops it
would take, using a mathematical equation. In this activity, youth were supported in their
attempts to approach mathematics problem-solving by linking data modeling to an earlier
activity that involved generating data about the number of hops.

Other instructional episodes evidenced fewer connections between earlier and later
activities and also the opportunity for more sustained involvement in work with data. For
example, during some instructional episodes, youth-generated data, but they did not use the
data they generated in subsequent activities. In the engineering-focused programs (Uptown
Architecture, Crazy Machines, and Dorchester House particularly, youth often generated
data that resulted from their engineering designs (and communicated and interpreted their
ﬁndings,) but did not model this data as a regular part of their activities. In one particular
example, in the Ecosphere program, youth collected water samples in the ﬁeld. They then
brought these samples to the classroom and tested the water, involving youth in both
collecting and, to a degree, generating data (by noting the pH levels of the water). However,
later in the day, youth created a small-scale model (with inclined trays of dirt, rocks, and
plants) of an ecosystem, in which they added food coloring to determine the impacts of
chemicals and acid rain. Youth then interpreted and discussed these ﬁndings, but did not
connect the discussion to the water samples youth collected and tested earlier. While these
specimens were collected to serve as data for future activity, there was no generating data
observed during the episode. In other instances, youth were involved in observing phenomena
but were not ever asked to use those data in subsequent activities. How this sequencing of
work with data may impact youths’ engagement was not considered in this study, though
past research suggests that this factor may make work with data more (or less) engaging and
impactful to learners. As McNeill and Berland (2017) argue, it is not just engaging in these
practices by rote, but about integrating them, as they overlap and interconnect. They argue
that a view of work with data focused on “making sense of” data generated from real-world
phenomena, as well as sustained engagement in work with data involving the revision of

76

earlier, intermediate ideas, are important considerations regarding the enactment of work
with data.

In addition to limitations related to the focus of the programs and how work with
data was enacted as part of a cycle, there were also some general measurement-related
limitations. Work with data can be di cult to measure because, as the qualitative analysis
revealed, there were a variety of ways in which youth can be involved in work with data.
McNeill and Berland (2017) describe a similar type of disagreement across science education
settings: While a limitation, the coding frame did represent agreement across a range of
studies across STEM contexts for the aspects of work with data. In terms of the alignment
of the measure with the conceptual framework for work with data, the dimensions of the
STEM-PQA measure aligned closely with the aspects of work with data. However, there were
some divergences that may have had an impact upon some of the ﬁndings. For example, for
the interpreting and communicating ﬁndings code, the STEM-PQA codes for Analyze (“Sta 
support youth in analyzing data to draw conclusions”) and Use symbols or models (“Sta 
support youth in conveying STEM concepts through symbols, models, or other nonverbal
language”) were used. In the case of the latter STEM-PQA code, conveying STEM concepts
through symbols, models, or other nonverbal language could have reﬂected instructional
episodes in which youth used, for example, mathematical equations or formulas, but did not
do so as part of modeling data of a phenomena in the world: They could have simply been
using an equation outside of the context of any particular phenomena. Future research may
consider the usefulness of coding for this aspect of work with data (and this aspect of science
curricular standards in particular; see NGSS Lead States, 2013).

As another example of this limitation related to how work with data was measured,
generating data was an aspect of work with data that the open-ended qualitative analysis
revealed to be less associated with less systematic groups of practices, or themes, than
the other aspects. The STEM-PQA codes corresponding to this aspect of work with data
were Collect data or measure (“Sta  support youth in collecting data or measuring”) and

77

Highlight precision and accuracy (“Sta  highlight value of precision and accuracy in measuring,
observing, recording, or calculating”). Particularly in the case of the latter code, the emphasis
on precision and accuracy may have been outside of activities focused on recording data
or creating coding frames. Future research may consider a coding frame that is (more)
focused on generating data, though considerations of precision and accuracy are key aspects
of doing so, and so perhaps separating the act of generating data from considerations that
are important to keep in mind while doing it may be a promising direction for future research.
While these divergences in measures were not large, they suggest that the coding frame for
work with data is a limitation of the present study.

It is possible that the somewhat minimal ﬁndings are, in part, a result of the analytic
approach. A similar mixed e ects modeling approach has only been used in one other study
(Strati et al., 2017), and that approach did not use proﬁles (as in this study) as the outcome.
In this study, little variability at the instructional episode level was found, and so minimal
relations between factors at this (instructional episode) level and the proﬁles of engagement
was expected. Might proﬁles, but not the variables used to create them, be less variable at
the instructional episode level? One way to consider such an alternate explanation is to use
the data used in this study as part of correlational analyses, other analyses that use the
variables used to create proﬁles of engagement but do not use the proﬁles themselves. An
analysis in this spirit was reﬂected in the correlations including the aspects of work with data
(presented in Table 4.2). These indicated very modest relations with engagement, indicating
that work with data and the individual variables used to create the proﬁles are not related.
Because of this, it is not surprising that the (more complex) mixed e ects models used to
explore the relations between work with data and engagement showed minimal relations.
Related to pursuing a di erent approach to the data analysis, other outcomes from working
with data may also show di erent (and more strongly positive or negative) relations. Such
outcomes may be at the instructional episode level, like engagement, or may be longer-term,
like youths’ future goals and plans after the conclusion of programs.

78

5.5 Implications for Practice

A few implications for practice can be drawn from this study, though these are
somewhat restricted given the minimal ﬁndings. First, generating data and modeling data, in
particular, may be beneﬁcial in terms of engaging youth. Youth activity leaders (in summer
STEM and other STEM enrichment contexts) and teachers (in formal learning environments)
can best include the beneﬁcial practices of generating and modeling data not in isolation,
but rather through involving youth and learners in complete cycles of an investigation. This
aligns with both foundational and contemporary research on work with data in education
(Berland et al., 2018; McNeill & Berland, 2017; Hancock et al., 1992; Lee & Wilkerson, 2018).
Another implication concerns how work with data was enacted. As found in this
study, work with data (and even speciﬁc aspects of work with data, such as asking questions)
does not involve activities that are enacted in a universal way. An instructor instead of
youth interpreting and communicating ﬁndings, for example, or learners asking general,
conceptual questions about work with data, as another, are di erent from youth working
to interpret ﬁndings and ﬁguring out how to ask a question that can be answered with
data, respectively. This heterogeneity suggests to those involved in planning and enacting
engaging activities that involve data to consider who works with data carefully, how they
do so, and how much time and sustained focus is required for such activities to be carried
out. This implication aligns with recent curricular reform e orts, some of which suggest that
involving work in STEM-related practices is most e ective when it involves learner-driven
(but instructor-supported) iterative processes of identifying a question or problem, marshaling
sources of data that can be used to ﬁgure out what is happening, and developing model-based
explanations that are shared with the learning community (National Governors Association,
2013; National Research Council, 2012; NGSS Lead States, 2013). While just two implications,
youth activity leaders and teachers and those designing data-rich activities and evaluating
the impacts of instruction based on such activities can use the ﬁndings from this study as a
starting point to consider how engaging in work with data may also prepare learners to think

79

of, understand, and take action based on data in education and in other areas of their lives.

80

APPENDICES

81

APPENDIX A

PROGRAM DESCRIPTIONS

Island Explorers: A science-focused program that aims to help youth develop expertise on
one species found in the local ecosystem by reading and writing about related content for
up to an hour per day; undertaking data collection and analysis tasks to learn about the
local ecosystem and how to communicate scientiﬁc data; developing vocabulary about the
local ecosystem; using art to learn and communicate information; and publishing a book
illustrating important elements of the species being studied. Located in both the classroom
and local ecosystem. 27 students who are rising 6th graders. Youth spend the morning in
more academically-oriented sessions in a classroom setting, while afternoon sessions involved
STEM-oriented enrichment sessions taking place outside (the program was associated with
Outward Bound) with an emphasis on exploration of the local ecosystem.

The Ecosphere: A science-focused program that aims to help youth to explore the
marine life of Narragansett Bay. E orts were undertaken to build youth content knowledge
in the areas of ecosystem preservation, marine biology, and water quality, and related skills,
such as questioning, showing initiative, data collection, measuring, maintaining an ecosystem,
and analyzing water samples. Located in a classroom setting, shoreline, and science education
center. 27 youth who are rising 6th to 9th graders. Youth attended programming in a
classroom at an area middle school and in a ﬁeld-based setting on alternating days. Field-
based settings included a science education center at a community-based organization and
ﬁeld trips to sites in the community related to the program’s focus.

Zoology Partners: A science-focused program that aims to support youth’s develop-
ment of content knowledge related to the issue of endangered species, including how species
become endangered, processes for monitoring ecosystem viability and population levels, solu-
tions to prevent species from becoming endangered, and approaches to reviving populations
that are currently endangered. Located in the classroom as well as zoos, parks, and other

82

natural areas. 25 youth who are rising 6th to 9th graders. Youth attended programming
in a classroom at an area middle school and in a ﬁeld-based setting on alternating days.
Field-based settings included a local zoo and ﬁeld trips to sites in the community related to
the program’s focus.

Marine Investigators: A science-focused program that aims to provide youth with
opportunities to learn about and experience Narragansett Bay; examine human impacts on
the local ecosystem, including how the geography of the Bay helped inﬂuence human history
and how the history of humans along the shoreline has impacted the Bay, and begin the
process of cultivating a sense of stewardship among participating youth for caring for and
protecting the Bay in the future. Located in the classroom, shoreline along the bay, ship
on the bay, and various ﬁeld locations associated with bay health. 19 youth who are rising
7th to 9th graders. Youth attended programming in a classroom at an area middle school
and in a ﬁeld-based setting on alternating days. Field-based settings included the local bay
shoreline, a voyage on a marine education ship researching in the Bay, and ﬁeld trips to sites
in the community related to the program’s focus. During the span of the program, youth
had the opportunity to participate in both a water quality research study.

Comunidad de Aprendizaje: A STEM-focused program that aims to help youth
improve basic skills in mathematics and develop an interest in STEM content and entrepreneur-
ship. Primarily in the classroom setting. 33 students who are rising 5th to 8th graders.
Morning sessions are characterized by direct instruction in mathematics for individual grade
levels and mixed grade level afternoon enrichment sessions in either robotics or dance. The
direct instruction component of the programs was organized around a theme of promoting
entrepreneurship with the goal of helping participating youth better see the relevance of
mathematics to future career goals and opportunities.

Je erson House: A STEM-focused program that aims to support youth’s devel-
opment of basic math skills, the program was primarily focused on helping youth develop
problem solving, self-improvement, and critical thinking skills. Located in a classroom. 11

83

youth who are rising 7th graders. The youth spent the morning in more academically-oriented
sessions in a classroom setting focusing on basic skill development, while afternoon sessions
involved STEM-oriented enrichment sessions involving media, art, and nutrition. Enrichment
o erings varied by day, with math sessions occurring twice per week, alternating with aca-
demically oriented sessions in the am that were oriented at supporting skill development in
English/language arts.

Uptown Architecture: An engineering-focused program that aims to support youth’s
participation in a process to design and build an outdoor learning space for use at the
middle school where the program was housed. A key focus of the program was to provide
youth with the opportunity to use design thinking as a problem-solving tool and have the
experience of a ecting their community positively through the design/build process. Located
in a classroom, building shop, and various ﬁeld locations. 18 youth who were rising 6th to
9th graders. Youth attended programming in a classroom at an area middle school and in
a building shop located at a community-based organization on alternating days, while also
taking ﬁeld trips to locations associated with the program’s overall theme.

Building Mania: An engineering-focused program that aims to provide youth with
the opportunity to experiment with designing and using simple machines. A goal of the
program is to have youth engage in the engineering design process by determining a need,
brainstorming possible designs, selecting a design, planning and drawing out the design,
creating and testing and revising it, and producing a ﬁnal machine. Located in the classroom,
design labs, and other local locations. 24 youth who are rising 6th to 9th graders. Youth
attended programming in a classroom at an area middle school and a ﬁeld-based setting
on alternating days. Field-based settings included a design lab at a community-based
organization and ﬁeld trips to sites in the community related to the program’s focus.

Adventures in Mathematics: A mathematics-focused program that aims to help
youth to develop the basic math skills and prevent summer learning loss among participating
youth through direct instruction and participation in math-related games. Located primarily

84

in the classroom. 20 youth who are rising 8th to 10th graders. Youth participated in direct
instructions in mathematics and math-related games in small groups. Program content was
aligned with the state’s standards in mathematics. )

85

APPENDIX B

MODEL SPECIFICATION DETAILS

Here, the six models that can possibly be speciﬁed in LPA are described in terms of how the
variables used to create the proﬁles are estimated. Note that p represents di erent proﬁles
and each parameterization is represented by a 4 x 4 covariance matrix and therefore would
represent the parameterization for a four-proﬁle solution. In all of the models, the means
are estimated freely in the di erent proﬁles. Imagine that each row and column represents
a di erent variable, i.e., the ﬁrst row (and column) represents broad interest, the second
enjoyment, the third self-e cacy, and the fourth another variable, i.e., future goals and plans.
Models 1 and 3 meet the assumption of independence, that is, that, after accounting for their
relations with the proﬁle, the variables used to estimate the proﬁles are independent (Collins
& Lanza, 2010). They estimate variable variances but do not estimate covariances (i.e., as
can be seen, the covariance matrices are “diagonal,” without any o -diagonal parameters that
are estimated). These models are estimated by default in MPlus, although these assumptions
can be relaxed (Muthen & Muthen, 2017). Importantly, this does not mean the variables
used to create the proﬁle are assumed to be not related; as Collins and Lanza (2010) explain:

The local independence assumption refers only to conditioning on the latent
variable. It does not imply that in a data set that is to be analyzed, the observed
variables are independent. In fact, it is the relations among the observed variables
that are explained by the latent classes. An observed data set is a mixture of all
the latent classes. Independence is assumed to hold only within each latent class,
which is why it is called “local”.

Despite the assumption of independence, as Collins and Lanza (2010), Muthen and
Muthen (2017), and others (i.e., Pastor et al., 2007; Vermunt & Magidson, 2002) note, it can

86

be lifted to improve model ﬁt, though these models without the assumption of independence
may be better described as general or Gaussian mixture models (Fraley et al., 2017).

B.0.1 Varying means, equal variances, and covariances ﬁxed to 0 (model 1)

In this model, which corresponds to the mclust model wit the name “EEI”, the variances are
estimated to be equal across proﬁles, indicated by the absence of a p subscript for any of the
diagonal elements of the matrix. The covariances are constrained to be zero, as indicated by
the 0’s between every combination of the variables. Thus, this model is highly constrained
but also parsimonious: the proﬁles are estimated in such a way that the variables’ variances
are identical for each of the proﬁles, and the relationships between the variables are not
estimated. In this way, less degrees of freedom are taken used to explain the observations
that make up the data. However, estimating more parameters–as in the other models–may
better explain the data, justifying the addition in complexity that their addition involves
(and their reduction in degrees of freedom).

‡2
0
0
0
1
0 ‡2
0
0
2
0 ‡2
0
0
3
0 ‡2
0
0
4

SWWWWWWWWWWU

TXXXXXXXXXXV

B.0.2 Varying means, equal variances, and equal covariances (model 2)

This model corresponds to the mclust model “EEE”. In this model, the variances are still
constrained to be the same across the proﬁles, although now the covariances are estimated
(but like the variances, are constrained to be the same across proﬁles). Thus, this model is the
ﬁrst to estimate the covariance (or correlations) of the variables used to create the proﬁles,
thus adding more information that can be used to better understand the characteristics of
the proﬁles (and, potentially, better explain the data).

87

‡2
1 ‡21 ‡31 ‡41
‡12 ‡2
2 ‡23 ‡24
‡13 ‡12 ‡2
3 ‡33
‡14 ‡12 ‡12 ‡2
4

SWWWWWWWWWWU

TXXXXXXXXXXV

B.0.3 Varying means, varying variances, and covariances ﬁxed to 0 (model 3)

This model corresponds to the mclust model “VVI” and allows for the variances to be freely
estimated across proﬁles. The covariances are constrained to zero. Thus, it is more ﬂexible
(and less parsimonious) than model 1, but in terms of the covariances, is more constrained
than model 2.

‡2
1p
0
0
0

SWWWWWWWWWWU

0
‡2
2p
0
0

0
0
‡2
3p
0

0
0
0
‡2
4p

TXXXXXXXXXXV

B.0.4 Varying means, varying variances, and equal covariances (model 4)

This model, which speciﬁes for the variances to be freely estimated across the proﬁles and for
the covariances to be estimated to be equal across proﬁles, extends model 3. Unfortunately,
this model cannot be speciﬁed with mclust, though it can be with MPlus; this model can be
used with the functions to interface to MPlus described below.

‡2
1p ‡21 ‡31 ‡41
‡12 ‡2
2p ‡23 ‡24
‡13 ‡12 ‡2
3p ‡33
‡14 ‡12 ‡12 ‡2
4p

SWWWWWWWWWWU

TXXXXXXXXXXV

88

B.0.5 Varying means, equal variances, and varying covariances (model 5)

This model speciﬁes the variances to be equal across the proﬁles, but allows the covariances
to be freely estimated across the proﬁles. Like model 4, this model cannot be speciﬁed with
mclust, though it can be with MPlus. Again, this model can be used with the functions to
interface to MPlus described below.

‡2
1

‡12p

SWWWWWWWWWWU

‡21p ‡31p ‡41p
‡2
2

‡23p ‡24p
‡2
3

‡33p
‡2
4

‡13p ‡12p

‡14p ‡12p ‡12p

TXXXXXXXXXXV

B.0.6 Varying means, varying variances, and varying covariances (model 6)

This model corresponds to the mclust model “VVV”. It allows the variances and the covariances
to be freely estimated across proﬁles. Thus, it is the most complex model, with the potential
to allow for understanding many aspects of the variables that are used to estimate the proﬁles
and how they are related. However, it is less parsimonious than all of the other models, and
the added parameters should be considered in light of how preferred this model is relative to
those with more simple speciﬁcations.

‡2
1p ‡21p ‡31p ‡41p
‡12p ‡2
2p ‡23p ‡24p
‡13p ‡12p ‡2
3p ‡33p
‡14p ‡12p ‡12p ‡2
4p

SWWWWWWWWWWU

TXXXXXXXXXXV

89

APPENDIX C

WORK WITH DATA BY PROGRAM

This table contains the proportion of the ﬁve aspects of work with data during by program.

90

Table C.1: Proportion of instructional episodes for which each of the aspects of work with data was present by program

9
1

Variable
Island Explorers
The Ecosphere
Zoology Partners
Marine Investigators
Comunidad de Aprendizaje
Je erson House
Uptown Architecture
Building Mania
Adventures in Mathematics

Asking Observing Generating Modeling Communicating Total Segments
16
0.312
24
0.625
0.250
24
24
0.458
55
0.327
0.167
24
24
0.375
24
0.333
0.583
24

0.250
0.292
0.167
0.375
0.273
0.458
0.167
0.333
0.458

0.375
0.500
0.208
0.542
0.327
0.750
0.292
0.500
0.750

0.375
0.417
0.167
0.333
0.182
0.083
0.208
0.208
0.292

0.438
0.500
0.125
0.250
0.400
0.542
0.708
0.375
0.542

APPENDIX D

ALTERNATE PROFILE SOLUTION

This solution is characterized by:

• A full proﬁle, proﬁle 7
• A universally low proﬁle, proﬁle 1
• A competent but not engaged or challenged proﬁle, proﬁle 2, characterized by high

competence and moderate (low) or low levels of engagement and challenge

• A moderately low proﬁle, proﬁle 3, characterized by moderately low levels of all of the

variables

• A challenged proﬁle, proﬁle 4, characterized by high challenge, moderate (high) levels

of engagement, and moderate (low) levels of competence

• A highly challenged proﬁle, proﬁle 5, characterized by patterns similar to those of the
challenged proﬁle, but with higher challenge and with low levels of both engagement
and challenge

• A challenged but not engaged or competent proﬁle, proﬁle 6, characterized by low levels

of challenge, and high levels of engagement and competence

The number of observations associated with each of the proﬁles is not very balanced,
with few (n = 181) observations associated with the universally low proﬁle and few (n =
222) observations associated with the highly challenged proﬁle. The number of observations
associated with the other proﬁles ranged from 317 to 651. Distinct from other solutions,
none of the other ﬁve proﬁles were found in the other model 1 solutions. Two pairs of the
proﬁles–challenged and highly challenged and universally low and moderately low–exhibited
similar patterns among the variables that were distinguished by di erent mean levels. Taken
together, this solution raises questions about whether it may be too complex, possibly
suggesting preference for model one ﬁve and six proﬁle solutions.

93

1

0

−1

e
r
o
c
s
−
Z

−2

Universally low (n = 181)
Moderately low (n = 651)
Competent but not engaged or challenged (n = 317)

Challenged (n = 569)
Highly challenged (n = 222)
Engaged and competent but not challenged (n = 568)

Affective
Behavioral
Cognitive
Challenge
Competence

Full (n = 450)

Figure D.1: The seven proﬁles of engagement (with variable values standardized)

l

e
u
a
V

4

3

2

1

0

Affective
Behavioral
Cognitive
Challenge
Competence

Full (n = 450)

Universally low (n = 181)
Moderately low (n = 651)
Competent but not engaged or challenged (n = 317)

Challenged (n = 569)
Highly challenged (n = 222)
Engaged and competent but not challenged (n = 568)

Figure D.2: The seven proﬁles of engagement (with variable values standardized)

94

BIBLIOGRAPHY

95

Franklin, C., Kader, G., Mewborn, D., Moreno, J., Peck, R., Perry, M., & Schea er, R.
(2007). Guidelines for assessment and instruction in statistics education (GAISE) report.
Alexandria, VA: American Statistical Association.

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of

the concept, state of the evidence. Review of Educational Research, 74(1), 59-109.

Gelman, S. A., & Markman, E. M. (1987). Young children’s inductions from natural kinds:

The role of categories and appearances. Child Development, 58(6), 1532-1541.

Gopnik, A., & Sobel, D. M. (2000). Detecting blickets: How young children use information
about novel causal powers in categorization and induction. Child Development, 71(5),
1205-1222.

Gopnik, A., Sobel, D. M., Schulz, L. E., & Glymour, C. (2001). Causal learning mechanisms
in very young children: two-, three-, and four-year-olds infer causal relations from patterns
of variation and covariation. Developmental Psychology, 37(5), 620.

Greene, B. A. (2015). Measuring cognitive engagement with self-report scales: Reﬂections

from over 20 years of research. lEducational Psychologist, 50(1), 14-30.

Greene, K. M., Lee, B., Constance, N., & Hynes, K. (2013). Examining youth and program
predictors of engagement in out-of-school time programs. Journal of Youth and Adolescence,
42(10), 1557-1572.

Hancock, C., Kaput, J. J., & Goldsmith, L. T. (1992). Authentic inquiry with data: Critical

barriers to classroom implementation. Educational Psychologist, 27(3), 337-364.

Harring, J. R., & Hodis, F. A. (2016). Mixture modeling: Applications in educational

psychology. Educational Psychologist, 51(3-4), 354-367.

Hasson, E., & Yarden, A. (2012). Separating the research question from the laboratory
techniques: Advancing high-school biology teachers’ ability to ask research questions.
Journal of Research in Science Teaching, 49(10), 1296-1320.

Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Experience sampling method:

Measuring the quality of everyday life. Sage.

Konold, C., & Pollatsek, A. (2002). Data analysis as the search for signals in noisy processes.

Journal for Research in Mathematics Education, 33(4), 259-289.

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M.
L. (2006). Out-of-school-time programs: A meta-analysis of e ects for at-risk students.
Review of Educational Research, 76(2), 275-313.

Lee, H. S., Angotti, R. L., & Tarr, J. E. (2010). Making comparisons between observed data
and expected outcomes: students’ informal hypothesis testing with probability simulation

97

tools. Statistics Education Research Journal, 9(1), 68-96.

Lee, H., & Hollebrands, K. (2008). Preparing to teach mathematics with technology: An inte-
grated approach to developing technological pedagogical content knowledge. Contemporary
Issues in Technology and Teacher Education, 8(4), 326-341.

Lee, V. R., & Wilkerson, M. (2018). Data use by middle and secondary students in the digital
age: A status report and future prospects. Commissioned Paper for the National Academies
of Sciences, Engineering, and Medicine, Board on Science Education, Committee on Science
Investigations and Engineering Design for Grades 6-12. Washington, D.C.

Lehrer, R., & Romberg, T. (1996). Exploring children’s data modeling. Cognition and

Instruction, 14(1), 69-108.

Lehrer, R., & Schauble, L. (2004). Modeling natural variation through distribution. American

Educational Research Journal, 41(3), 635-679.

Lehrer, R. & Schauble, L. (2015). Developing scientiﬁc thinking. In L. S. Liben & U. Müller
(Eds.), Cognitive processes. Handbook of child psychology and developmental science (Vol.
2, 7th ed., pp. 671-174). Hoboken, NJ: Wiley.

Lehrer, R., Kim, M. J., & Jones, R. S. (2011). Developing conceptions of statistics by

designing measures of distribution. ZDM, 43(5), 723-736.

Lehrer, R., Kim, M. J., & Schauble, L. (2007). Supporting the development of conceptions
of statistics by engaging students in measuring and modeling variability. International
Journal of Computers for Mathematical Learning, 12(3), 195-216.

Lesh, R., Middleton, J. A., Caylor, E., & Gupta, S. (2008). A science need: Designing tasks
to engage students in modeling complex data. Educational Studies in Mathematics, 68(2),
113-130.

Linnansaari, J., Viljaranta, J., Lavonen, J., Schneider, B., & Salmela-Aro, K. (2015). Finnish
Students Engagement in Science Lessons. NorDiNa: Nordic Studies in Science Education,
11(2), 192-206. Retrieved from https://www.journals.uio.no/index.php/nordina/article/
view/2047

Magnusson, D., & Cairns, R. B. (1996). Developmental science: Toward a uniﬁed framework.

Cambridge, England: Cambridge University Press.

McNeill, K. L., & Berland, L. (2017). What is (or should be) scientiﬁc evidence use in k-12

classrooms? Journal of Research in Science Teaching, 54(5), 672-689.

McNeill, K. L. & Krajcik, J. (2007). Middle school students’ use of appropriate and inappro-
priate evidence in writing scientiﬁc explanations. In Lovett, M & Shah, P (Eds.), Thinking
with data. (pp. 233-265). New York, NY: Taylor & Francis Group, LLC.

98

Miller, E., Manz, E., Russ, R., Stroupe, D., & Berland, L. (advance online publication).
Addressing the epistemic elephant in the room: Epistemic agency and the next generation
science standards. Journal of Research in Science Teaching. https://doi.org/10.1002/tea.
21459

Muthén, B. (2004). Latent variable analysis. The Sage handbook of quantitative methodology

for the social sciences. Thousand Oaks, CA: Sage Publications, 345-68.

Muthén, L. K., & Muthén, B. O. (1997-2017). Mplus User’s Guide. Los Angeles, CA: Muthén

& Muthén.

NGSS Lead States. (2013). Next generation science standards: For states, by states. Wash-

ington, DC: National Academies Press.

Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement.
In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student
engagement (pp. 259-292). New York, NY: Springer.

Petrosino, A., Lehrer, R., & Schauble, L. (2003). Structuring error and experimental variation
as distribution in the fourth grade. Mathematical Thinking and Learning, 5(2&3), 131-156.
Piaget, J., & Inhelder, B. (1969). The psychology of the child. New York, NY: Basic Books.
Pöysä, S., Vasalampi, K., Muotka, J., Lerkkanen, M. K., Poikkeus, A. M., & Nurmi, J. E.
(2017). Variation in situation-speciﬁc engagement among lower secondary school students.
Learning and Instruction, 53, 64-73. http://dx.doi.org/10.1016/j.learninstruc.2017.07.007
Rosenberg, J. M., Schmidt, J. A., & Beymer, P. N. (2018). tidyLPA: Easily carry out Latent

Proﬁle Analysis. https://cran.r-project.org/web/packages/tidyLPA/index.html

Rosenberg, J. M., Xu, R., & Frank, K. A. (2018). KonFound-It!: Quantify the robustness of

causal inferences. http://konfound-it.com.

Salmela-Aro, K., Moeller, J., Schneider, B., Spicer, J., & Lavonen, J. (2016). Integrating the
light and dark sides of student engagement using person-oriented and situation-speciﬁc
approaches. Learning and Instruction, 43, 61-70.

Salmela-Aro, K., Muotka, J., Alho, K., Hakkarainen, K., & Lonka, K. (2016). School burnout
and engagement proﬁles among digital natives in Finland: A person-oriented approach.
European Journal of Developmental Psychology, 13(6), 704-718.

Schmidt, J. A., Rosenberg, J. M., & Beymer, P. (2018). A person-in-context approach to
student engagement in science: Examining learning activities and choice. Journal of
Research in Science Teaching, 55(1), 19-43. https://dx.doi.org/10.1002/tea.21409

Schneider, B., Krajcik, J., Lavonen, J., Salmela-Aro, K., Broda, M., Spicer, J., . . . &
Viljaranta, J. (2016). Investigating optimal learning moments in US and Finnish science

99

classes. Journal of Research in Science Teaching, 53(3), 400-421.

Schwarz, N., Kahneman, D., & Xu, J. (2009). Global and episodic reports of hedonic experience.
In R. Belli, D. Alwen, & F. Sta ord (Eds.), Using calendar and diary methods in life
events research (pp. 157-174). Newbury Park, CA: Sage.

Sherno , D. J., Csikszentmihalyi, M., Schneider, B., & Sherno , E. S. (2003). Student
engagement in high school classrooms from the perspective of ﬂow theory. School Psychology
Quarterly, 18(2), 158-176.

Sherno , D. J., Kelly, S., Tonks, S. M., Anderson, B., Cavanagh, R. F., Sinha, S., & Abdi,
B. (2016). Student engagement as a function of environmental complexity in high school
classrooms. Learning and Instruction, 43, 52-60.

Shumow, L., & Schmidt, J. A. (2013). STEM interest and engagement (STEM I.E.). National

Science Foundation proposal for award number 1421198.

Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of deﬁning and
measuring student engagement in science. Educational Psychologist, 50(1), 1-13. doi:
10.1080/00461520.2014.1002924

Singh, K., Granville, M., & Dika, S. (2002). Mathematics and science achievement: E ects
of motivation, interest, and academic engagement. The Journal of Educational Research,
95(6), 323-332.

Sherno , D. J., & Schmidt, J. A. (2008). Further Evidence of an Engagement–Achievement
Paradox Among U.S. High School Students. Journal of Youth and Adolescence, 37(5),
564–580. http://doi.org/10.1007/s10964-007-9241-z

Shumow, L., Schmidt, J. A., & Zaleski, D. J. (2013). Multiple perspectives on student learning,
engagement, and motivation in high school biology labs. The High School Journal, 96(3),
232-252.

Skinner, E. A., & Pitzer, J. (2012). Developmental dynamics of engagement, coping, and
In S. Christenson, A. Reschly, & C. Wylie (Eds.), Handbook of

everyday resilience.
Research on Student Engagement (pp. 21-45). New York: Springer Science.

Skinner, E. A., Kindermann, T. A., & Furrer, C. J. (2009). A motivational perspective on
engagement and disa ection: Conceptualization and assessment of children’s behavioral
and emotional participation in academic activities in the classroom. Educational and
Psychological Measurement, 69(3), 493-525.

Skinner, E., Furrer, C., Marchand, G., & Kindermann, T. (2008). Engagement and disa ection
in the classroom: Part of a larger motivational dynamic? Journal of Educational Psychology,
100(4), 765.

Smith, C., Akiva, T., Sugar, S., Lo, Y. J., Frank, K. A., Peck, S. C., Cortina, K. S., & Devaney,

100

T. (2012). Continuous quality improvement in afterschool settings: Impact ﬁndings from
the Youth Program Quality Intervention study. Washington, DC: The Forum for Youth
Investment.

Steinley, D., & Brusco, M. J. (2011). Evaluating mixture modeling for clustering: recommen-

dations and cautions. Psychological Methods, 16(1), 63.

Stroupe, D. (2014). Examining classroom science practice communities: How teachers and
students negotiate epistemic agency and learn science-as-practice. Science Education,
98(3), 487-516.

Strati, A. D., Schmidt, J. A., & Maier, K. S. (2017). Perceived challenge, teacher support,
and teacher obstruction as predictors of student engagement. Journal of Educational
Psychology, 109(1), 131-147.

Turner, J. C., & Meyer, D. K. (2000). Studying and understanding the instructional contexts
of classrooms: Using our past to forge our future. Educational Psychologist, 35(2), 69-85.
van Rooij, E. C., Jansen, E. P., & van de Grift, W. J. (2017). Secondary school students’
engagement proﬁles and their relationship with academic adjustment and achievement in
university. Learning and Individual Di erences, 54, 9-19.

Vandell, D. L., Hall, V., O’Cadiz, P., & Karsh, A. (2012). Piloting outcome measures
for summer learning initiative programs. Final report to the David and Lucile Packard
Foundation, Children, Families, and Communities Program. Retrieved from http://faculty.
sites.uci.edu/childcare/ﬁles/2013/07/SL-Outcomes-2011-Pilot_Edited_8.19.pdf

Wang, M. T., & Eccles, J. S. (2012). Social support matters: Longitudinal e ects of social
support on three dimensions of school engagement from middle to high school. Child
Development, 83(3), 877-895.

Wang, M. T., & Holcombe, R. (2010). Adolescents’ perceptions of school environment,
engagement, and academic achievement in middle school. American Educational Research
Journal, 47(3), 633-662.

Weisberg, M. (2012). Simulation and similarity: Using models to understand the world.

Oxford University Press: Oxford, England.

Wild, C. J., & Pfannkuch, M. (1999). Statistical thinking in empirical enquiry. International

Statistical Review, 67(3), 223-248.

Wilkerson, M. H., Andrews, C., Shaban, Y., Laina, V., & Gravel, B. E. (2016). What’s the
technology for? Teacher attention and pedagogical goals in a modeling-focused professional
development workshop. Journal of Science Teacher Education, 27(1), 11-33.

Wilkerson, M. H. & Fenwick, M. (2017). The practice of using mathematics and computational
thinking. In C. V. Schwarz, C. Passmore, & B. J. Reiser (Eds.), Helping Students Make

101

Sense of the World Using Next Generation Science and Engineering Practices. Arlington,
VA: National Science Teachers’ Association Press. pp. 181-204.

Wormington, S. V., & Linnenbrink-Garcia, L. (2017). A new look at multiple goal pursuit:
The promise of a person-centered approach. Educational Psychology Review, 29(3), 407-445.
doi:10.1007/s10648-016-9358-2

Yohalem, N., Wilson-Ahlstrom, A., & Yu, D. (2005). Youth program quality assessment
and improvement: Celebrating progress and surfacing challenges. A meeting report.
http://forumfyi.org/content/youth-program-quality-

102