MEASURING STUDENT ENGAGEMENT IN SCIENCE CLASSROOMS: AN
INVESTIGATION OF THE CONTEXTUAL FACTORS AND LONGITUDINAL
OUTCOMES
By
Justina Judy Spicer

A DISSERTATION
Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of
Educational Policy—Doctor of Philosophy
2015

ABSTRACT
MEASURING STUDENT ENGAGEMENT IN SCIENCE CLASSROOMS: AN
INVESTIGATION OF THE CONTEXTUAL FACTORS AND LONGITUDINAL
OUTCOMES
By
Justina Judy Spicer
This dissertation includes three separate but related studies that examine the different
dimensions of student experiences in science using data from two different datasets: the High
School Longitudinal Study of 2009 (HSLS:09), and a dataset constructed using the Experience
Sampling Method (ESM). This mixed-dataset approach provides a unique perspective on student
engagement and the contexts in which it exists. Engagement is operationalized across the three
studies using aspects of flow theory to evaluate how the challenges in science classes are
experienced at the student level. The data provides information on a student’s skill-level and
efficacy during the challenge, as well as their interest level and persistence. The data additionally
track how situations contribute to optimal learning moments, along with longitudinal attitudes
and behaviors towards science.
In the first part of this study, the construct of optimal moments is explored using in the
moment data from the ESM dataset. Several different measures of engagement are tested and
validated to uncover relationships between various affective states and optimal learning
experiences with a focus on science classrooms. Additional analyses include investigating the
links between in the moment engagement (situational), and cross-situational (stable) measures of
engagement in science.
The second part of this dissertation analyzes the ESM data in greater depth by examining
how engagement varies across students and their contextual environment. The contextual

characteristics associated with higher engagement levels are evaluated to see if these conditions
hold across different types of students. Chapter three more thoroughly analyzes what contributes
to students persisting through challenging learning moments, and the variation in levels of effort
put forth when facing difficulty while learning in science.
In chapter four, this dissertation explores additional outcomes associated with student
engagement in science using the results for chapters two and three to identify aspects of
engagement and learning in science. These findings motivate a set of variables and analytic
approach that is undertaken in chapter four. Specifically, the questions how engagement
influences experiences in ninth grade science and students' interest in pursuing a career in STEM
using the HSLS:09 data.
This multifaceted study contributes to the conceptualization of student engagement, and
will help bring clarity to the relationship among engagement, context, and long-term outcomes in
science. Engagement is more than being on-task or paying attention, but is a condition influenced
by many factors including student background, the learning context of the classroom, teacher
characteristics, and the features of instruction. Understanding this relationship between
engagement and contextual factors is helpful in uncovering teacher actions and instructional
activities that may elicit higher engagement in science classes. These findings highlight the
importance of science instruction using more cognitively-demanding activities, such as problembased learning.

ACKNOWLEDGEMENTS

This study would not be possible without the steadfast support and guidance of
Committee Chair and Advisor, Dr. Barbara Schneider. I am incredibly fortunate to have received
the expert advice, counsel, and direction of Dr. Schneider over the past five years collaborating
on multiple research studies. Not only did I learn rigorous methodologies and best practices for
conducting research with integrity under her tutelage, but she helped me develop the knowledge
and skills to be a scholar while maintaining unwavering high expectations for my work. Dr.
Schneider was joined by four outstanding committee members, Dr. Peter Youngs, Dr. Spyros
Konstantopoulos, Dr. Kristy Cooper, and Dr. Stacy Dickert-Conlin. I am very thankful for the
diverse perspectives this committee offered, and the feedback they gave to me throughout the
duration of this study.
While at Michigan State University, I was the beneficiary of tremendous support from the
Educational Policy program and Dr. Michael Sedlak who provided funding during my first year
as well as the funding for other research-related expenses throughout the course of my study.
Most importantly, I am thankful for the opportunity the policy program provided to be a part of a
thriving community of emerging scholars. I also received generous support from an IES predoctoral fellowship in the Economics of Education. The program garnered for me the invaluable
opportunity to collaborate with fellows in the economics department. I would like to specifically
thank my fellow cohort members for their support, feedback, and humor. Lastly, this research
was brought to fruition by a dissertation grant from the American Educational Research
Association, I am thankful for the financial support and professional development opportunities
afforded to me by this grant.
iv

The data used in this dissertation were part of a larger research project (EAGER) funded
by the National Science Foundation (NSF). I gained additional experience and skills contributing
to another NSF project, the College Ambition Program. Both of these projects allowed me to be
a member of an exceptional research team. I am especially thankful to my colleagues Michael
Broda, Alan Hastings, Kaitlin Obenauf, Justin Bruner, Kri Burkander, Jonghwan Lee, Guan Saw,
and Ryan Goodwin. I also benefitted from the wisdom and experiences of my fellow Educational
Policy students, Andrew Saultz, Jeffrey Snyder, John Lane, Ben Creed, Erin Grogan, Todd
Drummond, and Yisu Zhou. My methodological skill-set and study of engagement benefitted
immensely from my German colleague Dr. Julia Moeller. I would like to thank the staff of the
Office of the Hannah Chair, including Michelle Chester, and the undergraduate students who
helped with countless hours of coding. Last and certainly not least, I would like to thank
Christina Ebmeyer, who served as the Project Manager, for keeping everything running smoothly
despite several obstacles along the way; she is an outstanding colleague and has become a dear
friend.
My passion and motivation throughout my time at Michigan State is derived from my
experience teaching at Garcia Elementary School in Houston, Texas. I am indebted to the faculty
and students who helped to shape my understanding of our educational system, and who
demonstrated that to work in the field of education one must have the utmost appreciation for the
extreme privilege it is to develop young learners and to help foster our future generations.
I would like to thank my family, including my parents Jeffrey and Mary Judy, my sister
Jessica, who always sent the best care packages, and my brother Joshua. My mom was a
steadfast source of encouragement and insight as a 30-year veteran of urban education. My
“adopted” family members in Texas and Michigan provided support in countless ways over the

v

years, and so I'd like to thank Rodney and Judy Henckel, Katie Beard, Brian and Katy Cox, Katie
Middlestead, and the whole DeYoung family for their support. Thank you to my in-laws, Jim and
Barbara, for your encouragement and for puppy-sitting during the final stretch of writing. Lastly,
I am grateful to my husband Tom Spicer, who not only provided me with constant access to the
experiences and mind of a real engineer, which proved valuable in thinking through parts of this
study, but who was also my anchor and constant source of love through the completion of my
dissertation. Above all else, I am thankful to my Lord and Savior, whom I give all glory.

vi

TABLE OF CONTENTS

LIST OF TABLES ................................................................................................................ix
LIST OF FIGURES ...............................................................................................................xi
CHAPTER 1: INTRODUCTION ..........................................................................................1
Theoretical Framework ..........................................................................................................3
Experience Sampling Method ..........................................................................................5
Study Design ..........................................................................................................................7
REFERENCES ......................................................................................................................10
CHAPTER 2: MEASURING ENGAGEMENT ....................................................................13
The Experience of Engagement .............................................................................................14
Self-Efficacy and Motivation ...........................................................................................16
Persistence and Commitment ...........................................................................................17
Measures of Engagement .................................................................................................17
Experience Sampling Method ..........................................................................................18
Theoretical Framework ..........................................................................................................20
Research Questions ................................................................................................................21
Methodology ..........................................................................................................................22
Sample ...............................................................................................................................23
Measures ............................................................................................................................26
Analytic Approach ............................................................................................................27
Results ....................................................................................................................................29
Discussion ..............................................................................................................................37
Limitations.........................................................................................................................37
Implications .......................................................................................................................38
APPENDICES...................................................................................................................... 40
Appendix A: Student ESM Questionnaire ........................................................................41
Appendix B: Student Background Survey ........................................................................42
REFERENCES ......................................................................................................................45
CHAPTER 3: CONTEXTUALIZING VARIATION OF ENGAGEMENT EXPERIENCES:
THE ROLE OF OPTIMAL LEARNING MOMENTS IN SCIENCE ..................................49
Dimensions of the Optimal Experiences and Learning .........................................................49
Engagement in Science .....................................................................................................50
Teachers and Instruction ...................................................................................................51
Teacher Behavior ..........................................................................................................52
Instruction .....................................................................................................................52
Student Background ..........................................................................................................54
Theoretical Framework ..........................................................................................................54
Research Questions ................................................................................................................56
Methodology ..........................................................................................................................56
vii

Data and Sample ................................................................................................................56
Measures ............................................................................................................................57
Analytic Approach ............................................................................................................58
Results ....................................................................................................................................59
Discussion ..............................................................................................................................67
Limitations.........................................................................................................................69
Implications .......................................................................................................................70
APPENDICES...................................................................................................................... 72
Appendix A: ESM Response Sample Restrictions............................................................73
Appendix B: Relationship between Persistence and Challenge ........................................74
REFERENCES ......................................................................................................................75
CHAPTER 4: THE ROLE OF ENGAGING EXPERIENCES IN SCIENCE AND
LONGITUDINAL OUTCOMES IN HIGH SCHOOL .........................................................79
Pathways in Science during High School ..............................................................................79
Strengthening the STEM Pipeline ...................................................................................79
Course-taking ...........................................................................................................80
Science Instruction ............................................................................................................81
School Context ..................................................................................................................83
Theoretical Framework ..........................................................................................................83
Research Questions ................................................................................................................85
Data and Methods ..................................................................................................................86
Measures ............................................................................................................................86
Student variables ......................................................................................................87
Teacher variables ......................................................................................................88
Weighting and missing data .....................................................................................89
Analytic Approach ...........................................................................................................89
Results ....................................................................................................................................90
Discussion ..............................................................................................................................98
Limitations.........................................................................................................................100
Implications .......................................................................................................................101
APPENDIX ...........................................................................................................................103
REFERENCES ......................................................................................................................106
CHAPTER 5: CONCLUSION ..............................................................................................109
Summary of Findings .............................................................................................................110
Limitations .............................................................................................................................111
Policy Implications ................................................................................................................112
REFERENCES ......................................................................................................................115

viii

LIST OF TABLES

Table 2.1 ESM School Sample Descriptives .........................................................................23
Table 2.2 Descriptive Characteristics of the Sample .............................................................25
Table 2.3 Summary Statistics for ESM Variables .................................................................30
Table 2.4 Summary Statistics of Cross-Situational Engagement Measures ..........................31
Table 2.5 Percentage of Time Students are in Flow ..............................................................33
Table 2.6 Correlations with Optimal Moments (challenge, interest, & skill)........................34
Table 2.7 Flow Measures Compared to Validations Items ...................................................35
Table 2.8 Percentage of Students Engaged in Science .........................................................36
Table 3.1 Descriptive Characteristics of Two-Level Data.....................................................57
Table 3.2 Optimal Learning Moments by Gender .................................................................64
Table 3.3 Optimal Learning Moments by Student Characteristics ........................................64
Table 3.4 Student-reported Task in Science ..........................................................................65
Table 3.5 HGLM Predicting Optimal Learning Moments-Level 1 .......................................66
Table 3.6 HGLM Predicting Optimal Learning Moments- Full Model ................................67
Table 3.7 Sample Restriction Comparisons ...........................................................................73
Table 3.8 Correlations with High Challenge and Persistence................................................74
Table 4.1 Descriptive Characteristics of the HSLS:09 ..........................................................87
Table 4.2 Science Course Enrollment by Student Characteristics .........................................92
Table 4.3 Teacher Instructional Variables .............................................................................94
Table 4.4 HGLM Predicting Usefulness of 2009 Science Course- Full model .....................95
Table 4.5 HGLM Predicting Skills of 2009 Science Course- Full model .............................96
ix

Table 4.6 HGLM Predicting Interest of 2009 Science Course- Full model ..........................97
Table 4.7 HGLM Predicting STEM Career in 2012 - Full model .........................................98

x

LIST OF FIGURES

Figure 1.1 A Conceptualization of Engagement in High School...........................................6
Figure 2.1 Frequency of Students by Response Rate ............................................................24
Figure 2.2 Relationship between Engagement Construct and Learning ................................28
Figure 3.1 Contextualizing Optimal Learning Moments in Science .....................................55
Figure 3.2 Student Volition of Task during an Optimal Learning Moment ..........................60
Figure 3.3 Student Perception of Activity during an Optimal Learning Moment .................61
Figure 3.4 Percentage of Students with Higher Challenge and Wanting to Give Up ............74
Figure 4.1. Longitudinal Engagement Experiences in High School ......................................84
Figure 4.2 Science Course Enrollment of 9th Graders from 2009-2010 ................................91

xi

CHAPTER 1: INTRODUCTION
Increasing student engagement in classrooms is a high priority yet ambiguous goal atop
many education policy agendas. While the idea of engaging students in learning is desirable and
often elicits images of eager students on the edge of their seats with hands raised, or a group of
students working together to build a model that demonstrates their understanding of a new
science idea— the physical manifestations of engagement may look different for each student,
teacher, classroom, and subject. The experience of engagement may also vary depending on a
number of contextual factors that comprise daily activities, such as the company one is with, or
the value of the activity in relation to one’s future goals. Research demonstrates that high student
engagement is an important condition that can contribute to multiple student outcomes, including
improved classroom behavior, increased student achievement, reduced likelihood of dropping
out, increased high school completion, and college matriculation (Klem & Connell, 2004;
National Research Council [NRC], 2004; Shernoff, D., Csikszentmihalyi, Schneider, & Shernoff,
E., 2003; Csikszentmihalyi & Schneider, 2000).
Secondary schools have an even greater challenge than elementary schools in engaging
students because as students progress from elementary to their middle and high schools, general
attitudes and interests in academics decline (Gonzales et al., 2008) and schools grapple with
effective ways to provide meaningful and motivational experiences for students who often see
themselves as passive participants in a large anonymous mass (Larson & Richards, 1991) where
students and teachers may just be “passing time” (Cusick, 1983). High school science classrooms
are of particular importance given the increased number of advanced courses needed to earn a
diploma (Schiller & Muller, 2003) and the increased demand for individuals to either pursue
postsecondary education or a career in the fields of science, technology, engineering, and

1

mathematics (STEM) (National Academies, 2010; National Science Board, 2010). As
educational policy continues to pursue higher standards and demand high quality teaching,
understanding the relationship between teacher instruction and student engagement is a critical
component in the evaluation of these reforms.
To gain a better understanding of student engagement, many research studies have
collected data on engagement through classroom observations and surveys, yet these measures
may not fully be able to get into the black box of student learning. What does engagement in
learning look like for each student, and not just the average student? What is engaging to
students? Specifically, what creates an optimal learning experience in school? Is the current
lesson interesting? Does it have personal value? How does the activity or lesson make the
student feel? While survey measures and observations of classrooms give one aspect to
understanding student engagement, the actual engagement experiences are not captured. This
study provides a diverse set of analyses of student engagement and is designed to understand
how student engagement varies, particularly in science classrooms, as well as analyze the
relationship of student engagement to long-term outcomes. These analyses use data from two
different datasets to gain a more comprehensive understanding of student engagement that not
only includes the outcomes associated with these engagement measures, but also discusses how
the use of different measures of engagement impact the outcomes of interest.
This study examines student engagement using data from the High School Longitudinal
Study of 2009 (HSLS:09), and a unique dataset constructed using the Experience Sampling
Method (ESM). This mixed-dataset study allows for the analysis of student interests and
behaviors in high school while considering additional factors related to their experiences in high
school, including their teachers, the types and frequency of instruction they are exposed to, and

2

characteristics of their classroom and school, that contribute to the daily contextual environment
in which engagement occurs. The contextual environment can include the beliefs, goals, values,
perceptions, behaviors, classroom management, social relations, physical space, and socialemotional space (Turner & Meyer, 2000). The ESM data enhance the ability to look more
closely within the classroom and these contextual characteristics, thus complementing the
HSLS:2009 analysis by supporting the exploration of the preconditions of engagement, which
can provide information for teachers on how to develop and sustain science engagement for
adolescents. While the field on student engagement is rich, diverse, and expanding, there are a
limited number of studies that consider the dimensionality of engagement across multiple
contexts as they contribute to the variation in learning moments. For example, are similar
classroom tasks related to similar engagement experiences? Are there differences between boys
and girls? Additionally, this research also examines the relationship of engagement and the
different ways it is defined to outcomes such as increased interest in science and taking
additional courses in science.
Theoretical Framework
The conceptualization of student engagement is diverse and often understood using a
multi-disciplinary perspective. The variety of ways in which engagement is defined and
subsequently measured can create a jingle fallacy (Thorndike, 1904) or jangle fallacy (Kelley,
1927). In a jingle fallacy, an identical term can be used to identify several situations (e.g. using
“engagement” to describe several different phenomena). In a jangle fallacy, there are similar
situations that are defined by different labels (e.g. using engagement, motivation, or interest to
describe the same outcome). Theoretical reviews about different definitions suggest that research
be clear about the specific components included and excluded in the definition, and that studies

3

align their theory and measurement fit, meaning, the applied measures should assess the
components that are relevant for the research questions and conclusions (Fredricks, Blumenfeld,
& Paris, 2012). Engagement is often described as an "in the moment" experience, but it is hardly
ever measured in the moment in which it occurs (Fredricks et al., 2012). Despite the reviews of
theories and the recognition of the need for a consistent understanding of how engagement is
studied, there are still many ambiguous methods being used to examine academic engagement,
thus leading to inconsistencies in the research.
Understanding the academic context is also an important factor in ascertaining how
students may engage in learning differently. Turner and Meyer (2000) offer three explanations
for considering classroom context when studying learning: (1) teaching and effective instruction
often varies with context (Good & Brophy, 2003); (2) teaching and learning vary by content area
(Stodolsky, 1988); and (3) theoretical frameworks should include an interpretative structure for
considering the role of contexts. With this is mind, when examining engagement, research
questions and instruments should be sensitive to the different contexts in which they are
operating, and include measures that capture the context and assess subject-specific aspects of
engagement.
In an earlier review of the literature, Fredricks, Blumenfeld, and Paris (2004)
operationalize engagement in three ways: (1) behavioral engagement, which is the action or
participation of the individual in academic or social activities; (2) emotional engagement, which
includes the positive and negative reactions to teachers and peers as well as an individual’s
willingness to work; and (3) cognitive engagement, which includes the investment of an
individual to comprehend new ideas and master challenging skills. While engagement can be
described using these paradigms, engagement types can also overlap. For example, behavioral

4

engagement can entail student conduct and completing tasks (Finn, Pannozzo, & Voelkl, 1995).
There are several ways of studying cognitive engagement that examine students’ investment in
learning, such as self-regulation and enjoying a challenge (Newmann et al., 1992).
Experience Sampling Method
One way to measure engagement while capturing the contextual academic environment is
to examine the daily experiences of students using the ESM. Developed by Csikszentmihalyi and
colleagues (1977), ESM provides a way of capturing the immediate activities and emotions of an
individual’s daily life in real time and at random intervals (Hektner, Schmidt, &
Csikszentmihalyi, 2007). Recognizing the multifaceted nature of student engagement and how it
varies across contexts, ESM allows students to record their behavioral, cognitive, and emotional
experiences. These data can be used to operationalize engagement. For example, using the selfreported levels of challenge and skill can provide evidence of being in “flow,” which is defined
as a state of deep absorption in an activity that is intrinsically enjoyable (Csikszentmihalyi,
1990). In flow, the challenge of an activity is well matched (balanced) to an individual’s skill
level (Csikszentmihalyi, 1975). Shernoff, D., Csikszentmihalyi, Schneider, and Shernoff (2003)
used specific components of flow to measure student engagement based on the occurrence of
high concentration, enjoyment, and interest in learning. When considering an individual's
concentration as a central component of flow, the participant's interest is strongly correlated to
their attention and motivation, and their enjoyment is related to their performance of the tasks.
This dissertation study uses flow theory, which argues that optimal experiences occur
when individuals experience a balance of challenge of skills in a task, as a way to operationalize
student engagement and understand optimal learning experiences using the ESM data. By
measuring these modifiable learning moments, instructional implications can be made. In

5

addition, these findings support the examination of characteristics of engagement in analyses of
the HSLS:09 dataset in Chapter 4.

As shown in Figure 1.1, the analyses of learning moments

in the ESM data complement the examination of secondary science experiences in the HSLS:09
dataset. These datasets are aligned and provide an opportunity to explore the dimensions of
challenge, interest, and skill in relation to science outcomes, such as advanced course-taking in
high school and postsecondary ambitions. Learning experiences, represented below by the gray
box, are the primary focus of

Figure 1.1 A Conceptualization of Engagement in High School

chapters two and three. This conceptual model refers more generally to “learning moments,”
which recognizes that not all of these experiences are optimal and that this study more narrowly
6

examines academic contexts of learning as opposed to optimal experiences that may not involve
learning (e.g. chess or rock climbing). The phenomenological factors box represents the
extensive set of affective dimensions that are captured by the ESM data (e.g. happiness,
cooperation, stress).
These learning experiences are influenced by both student background characteristics
(moderators), and contextual factors such as the characteristics of the school, teacher, and class
level as shown on the left side of Figure 1.1. The instructional factors include how the teacher
reports emphasizing certain methods of instruction. The student moderating variables include
demographic characteristics as well as their educational expectations. The outcomes explored in
chapter four are represented by the box on the right, and include the students’ experiences in
their 2009 science course and their later interest in a STEM career in 2012.
This approach specifically supports the examination in both datasets of student behavior
(what a student is doing), cognitive engagement (their efficacy and concentration), and emotional
state (how one feels about the activity). While prior ESM research often examines the occurrence
of engagement and how engagement varies, more research is needed to understand longitudinal
outcomes associated with being engaged (see Christenson, Reschley, & Wylie, 2012), an issue
that is discussed in this study.
Study Design
This study examines how engagement, measured within optimal moments, varies across
students and their contextual environment. In chapter two, the ESM dataset explores the
following questions: (1) How do challenge, skill, and interest levels inform measures of optimal
experiences? (2) What is the relationship between different constructs of engagement, and how
can these be validated? (3) How do these measures of engagement and optimal learning hold

7

across contexts, specifically, what do these moments look like in science classrooms compared
to other school settings? The constructs of engagement are examined using multiple descriptive
analyses using momentary measures, aggregated student-level measures, and nested analyses.
Chapter three provides an in-depth analysis of the ESM data, which contributes to
understanding optimal experiences. Analyses of this dataset focus on the contextual
characteristics associated with variation in engagement. This part of the study will test if the
certain contexts associated with increased engagement experiences hold across different types of
students and instruction. The guiding questions for this portion of the study are: (1) How does
engagement (as measured by optimal learning moments) vary across students? (2) What
instructional factors are associated with optimal learning moments? And (3) What characteristics
can predict optimal learning moments?
In the final chapter, data from the first follow-up of HSLS:09 are analyzed to address: (1)
How do experiences in science class vary by student characteristics and contextual factors? (2)
How do experiences in ninth grade science predict future interest in a STEM career? It is
hypothesized that having more engaging learning experiences in science courses can lead to an
increase in positive attitudes in science and contribute to future interest in STEM. This analytic
framework and approach to analyzing HSLS:09 is informed by the ESM data. Using this ESM
data to compliment HSLS:09 allows for the articulation of engagement measures to be explored
further by a weighted sample that can be generalized to the national population of students.
The goal of this three-part study is to contribute to the research and understanding of how
engagement is measured in the field, as well as studying the influence of engagement on the
experiences of students in high school science, specifically, its contribution to the development
and sustained interest in science. Improving engagement in science, and more broadly in STEM

8

education, is one policy lever utilized to strengthen the pipeline of talent from high schools into
the STEM workforce. Understanding how student engagement is defined and measured in high
school not only adds to the existing research on the conceptualization of engagement, but also
reveals new insights on instructional approaches that can increase engagement and ultimately
learning. While providing an understanding of the relationship between engagement and
instruction for researchers and policymakers, this type of research can also provide teachers with
insight regarding the experiences of students in their classes, which has implications for their
instruction, and could potentially be used as a tool for improvement. As higher standards and
increased expectations continue to be upheld as the policy of choice for improving education,
particularly in STEM, understanding the engagement of students in high school and the role of
teachers in fostering engagement is an important area to continue to research.

9

REFERENCES

10

REFERENCES

Christenson, S., Reschly, A., & Wylie, C. (2012). Handbook of research on student engagement.
New York: Springer.
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper
Perennial.
Csikszentmihalyi, M., & Schneider, B. (2000). Becoming adult. New York: Basic Books.
Connell & Wellborn, (1991). Competence, autonomy, and relatedness: A motivational analysis
of self-system processes. In M. Gunnar & L. A. Sroufe (Eds.), Minnesota symposium on
child psychology (Vol. 23). Chicago: University of Chicago Press.
DiBianca, R. (2000). Teaching adolescents: Relationships between features of instruction and
student engagement in high school mathematics and science classrooms. Chicago:
University of Chicago Press.
Fredricks, J. A., Blumenfeld, P.C., & Paris, A. H. (2004). School engagement: Potential of the
concept, state of evidence. Review of Educational Research, 74(1), 50-109.
Good, T.L. & Brophy, J.E. (2003). Looking in Classrooms, 9th Edition. Boston, MA: Pearson.
Kelley, T.L. (1927). Interpretation of educational measurements (pp. 62–65). Yonkers-onHudson, N.Y.: World Book Company.
Klem, A.M. & Connell, J. (2004). Relationships matter: Linking teacher support to student
engagement and achievement. Journal of School Health, 74(7), 262-273.
National Academies. (2010). Rising above the gathering storm, revisited: Rapidly approaching
category 5. Washington, DC: The National Academies Press. Retrieved from
http://www.nap.edu/catalog/12999.html.
National Science Board. (2010). Preparing the next generation of STEM innovators: Identifying
and developing our nation's human capital. Arlington, VA: National Science Foundation.
Retrieved from http://www.nsf.gov/nsb/publications/2010/nsb1033.pdf.
National Research Council. (2004). Engaging schools: fostering high school student’s motivation
to learn. Committee on Increasing High School Students’ Engagement and Motivation to
Learn. Board on Children, Youth, and Families, Division of Behavioral and Social
Sciences and Education. Washington, DC: The National Academies Press.

11

Shernoff, D. & Csikszentmihalyi, M. (2009). Flow in schools: Cultivating engaged learners and
optimal learning environments. In R. Gilman, E. S. Huebner, & M. Furlong (Eds.),
Handbook of Positive Psychology in Schools (pp. 131-145). New York: Routledge.
Shernoff, D., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. (2003). Student engagement in
high school classrooms from the perspective of flow theory. School Psychology
Quarterly, 18(2), 158-176.
Shernoff, D. (2001). The experience of student engagement in high school: A phenomenological
perspective. Chicago: University of Chicago Press.
Skinner, E. A. & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of
teacher behavior and student engagement across the school year. Journal of Educational
Psychology 85(4), 571-581.
Stodolsky, S. (1988). The subject matters: Classroom activity in math and social studies.
Chicago: University of Chicago Press.
Thorndike, E. L. (1904). An introduction to the theory of mental and social measurements. New
York: Teachers College, Columbia University.

12

CHAPTER 2: MEASURING ENGAGMENT
Ancient Greek writer Sophocles wrote that “One must learn by doing the thing, for
though you think you know it—you have no certainty until you try.” This is indeed a guiding
principle today for many teachers and their students. Buzzwords like, “hands-on learning” and
“active learning” fill many professional development sessions aimed at improving student
experiences in school through engaged learning. However, there is much ambiguity and little
consensus of what “engagement” in the classroom actually is (Reschley & Christenson, 2012).
Often it is something that a teacher can recognize when it happens (or when it does not), but
there is a current need for clarity in the definition and study of engagement.
Research on engagement focuses on multiple areas, two of which include: (1)
behaviorally disengaged students, such as those who may drop out of high school or are
considered “at-risk” (see Easton, 2008; Finn, 1989; Newmann, 1992; Ream & Rumburger,
2008), and (2) the psychologically and emotionally engaged student, which includes selfdetermination theory, achievement goal theory, achievement motivation theory, attribution
theory, self-efficacy theory, and expectancy-value theory (see Ames & Archer, 1988; Ames,
1992; Bandura, 1977; Eccles et al., 1983; Dweck, 1986; Schunk, Pintrich, & Meece, 2008). Both
of these related approaches seek to understand academic success and failure, but with different
definitions, measures, and outcomes of engagement. While the field of engagement research is
robust, without clarity in how engagement is studied (e.g. defined and measured), the
implications of the research can be difficult to apply due to the ambiguity of the desired outcome
and specific conditions that can facilitate better learning experiences for students.
This chapter examines several measures of engagement, proposes a definition of
engagement motivated by psychological theory, creates and tests constructs of engagement, and

13

analyzes how and when engagement occurs in secondary schools. What follows is a brief
overview of the engagement literature and related aspects of motivation and perseverance, in
addition to exploring some of the different methods of measuring engagement. Next, a
theoretical framework for defining and measuring engagement is discussed, which motivates the
research questions for this part of the study: (1) How can engagement be defined? (2) What is the
relationship between different constructs of engagement? (3) How do measures of engagement
hold across contexts? Based on the theoretical framework, several different measures of
engagement are then tested to address these primary research questions. Following these
analyses is a discussion of the findings and the implications for the next two chapters.
It is important to note that this chapter will not offer any new definitions of engagement,
but rather validate existing measures and examine the relationships between them. This study,
including the present chapter and the ones that follow, will also not offer ways to increase
engagement or create higher motivation, as these types of outcomes are not measurable, and are
indeed one of the challenges to this type of research. Instead, this study contributes to
understanding the environments, student background characteristics, and instructional
approaches that may lead to increased moments of engagement, which are considered optimal
experiences (Csikszentimihalyi, 1975). Because the experiences in question are specific to
educational settings they will be referred to as optimal learning moments (see Schneider et al.,
under review; Shernoff & Csikszentmihalyi, 2009).
The Experience of Engagement
In the simplest approach, situational engagement is often depicted in a linear
relationship— within a context there is a motivation that leads to engagement and a subsequent
outcome (see Connell & Wellborn, 1991; Finn & Zimmer, 2012; Lawson & Lawson, 2013).

14

However, this linear relationship is dynamic and subject to additional influences including
affective/emotional states, behaviors/activities/tasks (Fredericks et al., 2004), and surrounding
cultures and varying contexts. In a review of the recently published Handbook of Research on
Student Engagement (Christenson, Reschley, & Wiley, 2012) and the extant literature on
engagement, Lawson and Lawson (2013) suggest three primary assumptions about engagement:
(1) it is malleable; (2) it has a direct relationship with learning; and (3) it is distinct from
motivation. Thus, research on engagement should recognize and be able to measure the
phenomenon given these assumptions. Eccles and Wang (2012) also suggest that student
engagement research needs to define constructs, achieve internal and external validity, and
interpret findings, in addition to studying learning experiences across multiple contexts and over
time, which is how this study is designed and organized. Engagement is relational and dynamic,
which suggests that research should be conducted in a way to allow this dynamic type of data to
be collected.
Research should also be able to measure and delineate across different settings in which
engagement occurs, operating under the assumption that engagement can happen both in and out
of school, as well as in academic and non-academic settings. Studies that focus solely on the
engagement of students while they are within the confines of their school are limited if they
aren’t able to compare the in-school engagement to that which occurs outside of school. Indeed,
it could very well be the case that students are equally engaged in both settings, as this study will
show. Therefore, research needs to distinguish between different settings, as well as recognize
the diverse levels of context where engagement occurs (e.g. classroom-level or school-level)
(Skinner & Pitzer, 2012). The research should also recognize the different time-frames of student

15

engagement, such as in-the-moment task engagement versus more long-term engagement or the
individual's commitment to a particular subject area (Finn & Zimmer, 2012).
Self-Efficacy and Motivation
Engagement and motivation are closely related concepts and are often used
interchangeably when discussing ways to improve learning or increase the interest and pursuit of
STEM education and careers. Skinner and Pitzer (2012) demonstrate that “'engagement’ is the
outward manifestation of motivation.” Motivation is also closely related to the feelings of
efficacy a student experiences while learning. Bandura (1977) explains self-efficacy as the
expectation of successfully executing the behavior required for an outcome. Thus, increasing
self-efficacy of students could influence their behavior and actions. If the desired behavior is
persistence in a challenging activity, the perceived self-efficacy in the form of confidence or
doubt can influence one’s behavior. Students should have perceptions of their high ability to
engage in a challenging task, or they run the risk of engaging in low challenge tasks to conceal or
protect themselves from a negative evaluation (Dweck, 1986).
While the self-efficacy framework provides a way of investigating underlying
mechanisms that influence performance and choice, expectancy-value theory provides a different
construct for understanding an individual’s expectation of success (Wigfield & Eccles, 2000).
Expectancy-value theory of achievement motivation argues that the choice, persistence, and
performance of individuals can be explained by their valuation of the activity and their
perception of how well they can perform on it (Eccles et al., 1983). One assumption of this
model is that expectancies and values directly influence achievement choices, performance,
effort, and persistence—these expectations and values can be both for the short- or long-term.
This theory is used to study domain-specific motivation, such as mathematics for example.

16

Motivation itself, however, is often not enough to improve learning. Feeling good about school
or in certain subject areas can only propel a student so far in their acquisition of new knowledge
and skills. Students who are actively engaged in learning should not only feel enjoyment and
satisfaction, but their engagement should lead to continued learning and the improvement of their
actual competencies (Skinner & Belmont, 1993).
Persistence and Commitment
Accompanying feelings of enjoyment and satisfaction during an activity are the
additional components of heightened concentration (Shernoff et al., 2003), active commitment
(Fredricks et al., 2004), persistence, and a reaction to challenge (Fredricks & McColskey, 2012).
Skinner and Belmont (1993) showed that persistence, effort, and attention were related to
engagement during learning activities. While engaged in learning, persistence in the task is
linked to receiving praise and redirection (Martens, Bradley, & Eckert, 1997), indicating that the
context of the learning moment, such as the actions of the teacher, can influence the type of
social and emotional experiences the student has when engaged. To persist or continue despite a
challenge requires both cognitive engagement in an activity as well as affective engagement,
such as feelings that the task is of value and worth pursuing (Finn & Zimmer, 2012).
Measures of Engagement
Several different survey instruments and observational protocols are often used to
measure student engagement (Fredricks et al., 2004). These approaches include teacher reports
of student engagement, students self-reporting of their engagement, and third party observations
of events and activities occurring in classrooms. Measures of engagement commonly asked of
teachers and students include the completion of work, levels of effort and persistence, and class
participation. For example, a teacher item might include, “Student participates actively in class

17

discussion” (Finn et al., 1995). Newmann (1992) provides an observational tool for observers to
rate engagement on a five-point scale from “off-task” to “deeply involved.” Student self-reported
measures often include items such as, “Math will be useful to my future” (Finn et al., 1995), and
“When I’m in class, I usually think about other things” (Skinner & Belmont, 1993).
The use of surveys to assess student engagement allows for the measurement of multiple
factors shown to influence engagement to be evaluated, such as teacher expectations, time ontask behavior, and general interests and attitudes about learning. Many of the surveys in use also
include items related to classroom and school context. Observational protocols either used in
conjunction with surveys or on their own also contribute a valuable perspective of learning and
engagement, both in terms of teacher actions and student behaviors. However, there are
limitations to both of these methods. Data collected on surveys often rely on retrospective
responses of engagement, that is, students and teachers have to think back to the moment, class,
subject, etc. to assess their relative levels of engagement, which allows for error, distortions, or
rationalizations in the data (Csikszentmihalyi & Larson, 1987). Observations of engagement may
reduce the error in retrospective reporting by recording data in real-time, but are subject to
different measurement errors in the form of rater reliability and is limited to only collecting data
that can be observed. For example, a student’s level of challenge or ability to persist on a
difficult task might not be visible to the observer.
Experience Sampling Method
One approach to measuring engagement that allows for the multiple dimensions of
engagement to be evaluated while collecting data in real time is the Experience Sampling
Method (ESM). Developed by Csikszentmihalyi and colleagues (1977), ESM provides a way of
capturing the moments of an individual’s daily life—immediate activities and emotions at

18

random intervals (Hektner, Schmidt, & Csikszentmihalyi, 2007). Recognizing the multifaceted
nature of student engagement and how it varies across contexts, ESM allows students to report
their cognitive and emotional experiences. ESM supports the examination of not just moments of
optimal learning, but the context in which it occurs. Using ESM with additional points of data
from student and teacher surveys as well as instructional information can enhance the ability of
the research to contextualize the engagement.
Several studies have used ESM to examine learning using a programmed wristwatch or
pager that activates at random intervals throughout the day over a period of one to two weeks.
The number of “beeps” and duration of the study are determined by the research questions and
design, with an average of eight beeps a day. When an individual is beeped, they are to complete
a questionnaire that includes open-ended and scaled items. An example of an open-response item
is: “As you were beeped, what were you doing?”A scaled item might ask, “Did you enjoy what
you were doing?” (Csikszentmihalyi & Schneider, 2000). By collecting these data at multiple
moments throughout the day, error due to recall, distortion, and rationalization can be reduced
(Shernoff & Csikszentmihalyi, 2008).
While ESM is not the only method to examine how teachers and students engage in
teaching and learning, it is the only approach that allows for within-person analyses. For
example, if a student reports high levels of engagement in a particular class on a “traditional”
survey, the researcher is unable to situate this engagement level to the rest of the individual’s
day. With the traditional survey, the researcher is unable to ascribe the high level of engagement
reported by the student to the class, or determine if the student is highly engaged all day, and
thus their reported engagement level would not be a reflection of the specific class or teacher.
Using ESM supports the analysis of within-person experiences, and through using standardized

19

scores of constructs from the within-person analysis allows for the comparison of engagement
across different contexts, both in- and out-of-school.
While several of these measures attempt to tap different aspects of engagement that
contribute to learning, each approach has its limitations. Survey measures of engagement may
capture a perceived overall engagement experience, rather than while it is happening. These
measures may also be distorted by recall and influenced by prior dispositions. For example, if a
student has lower self-efficacy in math, they may perceive their engagement in math classes as
lower. With in-the-moment ESM measures, this type of data collection can be burdensome for
the participant, and there may be selection bias in that students with a higher perceived selfefficacy may be prone to be “more engaged” in the study with higher rates of response and
participation, and less-engaged students might not be as represented in the sample. However,
examining multiple engagement measures, both stable and in-the-moment, not only addresses the
limitations of using any singular approach, but also allows for the ability to understand the
relationship between different measures of engagement, thus providing a more comprehensive
perspective.
Theoretical Framework
To measure experiences in the moment, ESM is used and flow theory is applied to
operationalize engagement, building on several previous studies examining what optimal
learning is in a classroom (Shernoff & Csikszentmihalyi, 2009; Shernoff et al., 2003;
Csikszentmihalyi & Schneider, 2000). “Flow” is defined as a state of deep absorption in an
activity that is intrinsically enjoyable (Csikszentmihalyi, 1990). In flow, the challenge of an
activity is well matched to an individual’s skill, and successful actions seem effortless despite a
high demand for physical or mental energy (Shernoff & Csikszentmihalyi, 2009). Misaligned

20

levels of challenge and skill can produce different psychological states, such as low challenge
and low skill can result in apathy, or high challenge and low skill resulting in anxiety.
There are several studies that have used ESM and flow to assess optimal experiences
across diverse contexts and with varied age, SES, and cultural study samples (see Shernoff et al.,
2003; Shernoff & Csikszentmihalyi, 2009; Shumow & Schmidt, 2014). Being in flow can occur
in school and other academic settings, but also during leisure. The ESM approach is also
designed to capture the activity as well as the affective states, which allows for the examination
of the multiple dimensions of engagement (i.e. behavior, cognitive, and emotional), and the
assessment of how optimal learning is related to additional measures of learning, such as
enjoyment, absorption, and persistence. The multi-level structure of ESM, which nests situations
within individuals, also strengthens the analyses of the variation within- and between-students,
allowing analyses to condition on factors previously associated with variation in engagement,
such as gender, race, and other stable person-level measures.
Research Questions
This study uses multiple measures of engagement to construct and validate measures of
engagement. Specifically:
1. How is engagement defined? How do challenge, skill, and interest measures inform
measures of engagement?
2. What is the relationship between different constructs of engagement, and how can these
be validated by related measures?
3. How do measures of engagement hold across contexts, specifically, is there a difference
in engagement in and out of science classrooms, and in and out of school?
Examining the construction of engagement measures and the relationships between measures of

21

engagement not only contributes to how engagement is defined and conceptualized, but is also
an essential first step in the subsequent studies of this dissertation. These analyses will provide
evidence of the validity of the proposed engagement measure, which will be used in subsequent
analyses of this dissertation
Methodology
Unlike previous ESM studies which employ wristwatches or pagers, this study uses a
smartphone equipped with an ESM application1 that randomly signals students eight times a day
over the course of a week, including the weekend. Each time a student is beeped, they complete a
short series of questions on the phone about what they are doing and how they feel. It took
students an average of 1.5 minutes to complete the form each time they were notified. Collecting
these data at multiple moments throughout the day reduces error due to recall, distortion, and
rationalization (Hektner, Schmidt, & Csikszentmihalyi, 2007). For this study, the science class
was the primary sampling unit and each student in the targeted science was asked to participate,
and the variation of student background characteristics are similar across the classes and schools.
Data for this portion of the study are collected from two primary instruments. An ESM
instrument (see Appendix A) for students was developed based on similar protocols used in the
Sloan Study of Youth and Social Development (Csikszentmihalyi & Schneider, 2000). Measures
from the ESM instruments include open-ended items such as task behavior (e.g. “What were you
doing when you were signaled?”) and affective states rated on a 4-point Likert scale (e.g. “How
challenged did you feel by the activity you were doing?”). The signaling schedule was a hybrid
of planned and random alerts to allow for at least one beep during science class, while the rest of
the beeps outside of science remained random. A student questionnaire (see Appendix B) is also
used to collect information about beliefs and experiences related to science, and was developed
1

“Paco” developed by Robert Evans, see pacoapp.com for more information.

22

using items from the Programme for International Student Assessment (PISA) (OECD, 2003)
and the schoolwork engagement inventory (Salmela-Aro & Upadaya, 2012). Additional
demographic information on gender and race was collected from the school.
Sample
Four partner schools were identified and incorporated a range of school types, including
one urban school, two rural schools, and one suburban school. Table 2.1 shows the demographic
information and student samples from each school. The students were sampled from their science
courses, primarily from biology, physics, or chemistry classes. The data collection took place in
two phases (Spring 2013 and Fall 2013), with slight modifications made in the ESM instruments
between phases one and two. Once the schools were identified, science teachers that were willing
to participate were chosen based on their subject, schedule, and class size. All students in the
“target class” were asked to participate, which was a total of 280 students. Two students declined
to participate, leaving the final analytic sample of 278 students. Each phase lasted one week.
While some ESM studies restrict the analytic sample to students with a certain percentage of
responses (see Hektner, Schmidt, & Csikszentmihalyi, 2007), since these analyses primary
focused on the situation-level responses, all observations are included in initial analyses.
Table 2.1
ESM School Sample Descriptives
Number Number
of study of ESM Students School
Teachers Classes with ESM Type
School 1
3
4
69
Urban
School 2
2
3
85
Rural
School 3
4
4
71
Suburban
School 4
1
2
53
Rural
SOURCE: Study Dataset and Common Core of Data 2012-2013

23

School
Size
1,730
466
1,304
645

Percent
Minority
66%
5%
34%
8%

Percent
FRPL
61%
37%
33%
17%

About the same number of students participated in each phase of the data collection as
shown in Table 2.2, however, because of the differences in participating teachers and classes
between phases, there are fewer non-white students in phase two (32% in phase one compared to
19% in phase two), as well as a higher percentage of biology students (42% in phase one
compared to 74% in phase two). The overall sample is approximately 56% biology students,
26% physics students, and 17% physics students. The gender is mostly balanced, with a slightly
higher percentage of males in the sample, 54% compared to 46%, this is not significantly
different between phases. The average response rate is 52%, meaning that on average a student
responded to about 29 signals over the duration of the study. Figure 2.1 shows the variation in
response rates. There were 57 students that responded to less than 20% of their total signals, in
contrast to 51 students that responded to at least 81 percent of their signals. The total number of
momentary data from these 278 students totaled 8,485 situational observations.

Figure 2.1 Frequency of Students by Response Rate
38

Number of Students

40

38
32

35

33
28

30

25
22

25
20

23
20

19

15
10
5
0

Response Rates

24

Table 2.2
Descriptive Characteristics of the Sample
Sample
Phase
Average
1
2
n of students
278
137
141
Average ESM response rate
52%
56%
48%
Percent female
46%
45%
48%
Percent nonwhite
25%
32%
19%
Science Course
Biology
56%
42%
74%
Chemistry
17%
25%
8%
Physics
26%
33%
18%

Urban 1
69
46%
39%
64%
80%
-20%

SOURCE: Study Dataset

25

School
Rural 2 Suburban 3
85
71
63%
47%
49%
45%
10%
20%
37%
28%
35%

45%
16%
39%

Rural 4
53
48%
53%
8%
100%
---

Measures
One of the strengths of collecting repeated measures through ESM is that within-person
metrics can be determined (Hektner, Schmidt, & Csikszentmihalyi, 2007). For example, a
reported score of “3” for interest might have different meanings across individuals, so
standardizing these scores across individuals with a mean of zero and a standard deviation of
one, allows the raw score to be transformed into a Z-score that reflects each moment’s value
relative to the average reported scores over the week. One limitation of transforming the ESM
variables in this way is that it creates the same mean and distribution within all students, which
may influence how the variance is attributed between levels in a multi-level model.
In this study, both standardized and raw engagement scores are examined in their ability
to measure engagement. By applying flow theory to determine if a student is engaged in the
moment, their level of challenge and skill in the moment is examined as well as a third
dimension of situational interest. The situational engagement construct is created in two ways:
(1) simultaneously reporting a “3” or “4” on a 4-point-Likert scale for their feeling of challenge,
interest, and skill in a given activity, referred to as “raw scores,” and (2) creating a standardized
measure when their reported challenge, interest, and skill during a given moment are above their
mean (zero), these are referred to as the “standardized score.” Each construct is compared with
two validation items often associated with being in flow: the feeling that time is flying by, and
persistence. The engagement measures are also correlated with other affective states, such as
enjoyment and boredom, to evaluate how the state of engagement is associated with other social
and emotional variables.
The ESM measures are situated in context to allow exploration of additional influences
such as teaching or classroom characteristics on learning. Using both cross-situational and in the

26

moment measures of engagement may also reduce measure bias. For example, self-concept and
anxiety tend to have substantial gender differences when assessed cross-situationally, but much
less so when assessed in the moment (Goetz et al., 2013). The engagement measures from PISA
use a diverse set of items that are domain-specific to science including: a sense of belonging and
participation; a disposition towards learning, including hedonic experiences, interest, value of
school success, class attendance, identification, being socially integrated, and the acceptance of
school rules (OECD, 2003). Five science-specific scales are created from these items and
measure science enjoyment, interest, value, motivation, and self-concept. For this present study,
these items were tailored to measure the specific science course each participating student was
enrolled in, such as biology, chemistry, or physics. These five constructs were created from the
same scales used in PISA. Two additional cross-situational constructs developed by Salmela-Aro
and colleagues were used to measure overall engagement and burnout in schoolwork (SalmelaAro & Upadaya, 2012).
Analytic Approach
To analyze the validity of engagement measures, tests that measure the different
dimensions of construct validity are conducted. These analyses answer the general question of
whether the proposed engagement measures actually capture engagement, including how the
operationalization of engagement using student’s report of their challenge, skill, and interest is a
reflection of the occurrence. As shown in Figure 2.1 (adapted from Trochim, 2006), the overall
goal of establishing construct validity is the process of validating what is seen and observed,
which in this study is the student experiences as captured by the ESM and survey instruments,
with the proposed theory explaining the phenomenon. More specifically, in this study’s setting

27

the daily experiences of the students can be observed and their self-reported behaviors and
affective states are recorded. It is hypothesized that these different experiences, influenced by the

Figure 2.2 Relationship between Engagement Construct and Learning

individual, their context, and activity is related to an outcome, such as learning or not learning.
These analyses will test the relationships between the observed experiences and the theory of
flow creating moments of optimal experiences as measured by the ESM.
To examine the validity of the engagement measures, each construct is first compared
with three validation items often associated with being in flow: the feeling that time is flying by
(absorption), concentration, and persistence (not giving up). Because these items were only
asked in the second phase of data collection, a sub-sample of students are included.2 The
engagement measures are also correlated with other affective states, such as enjoyment,
persistence, and boredom to evaluate how the state of engagement is associated with other social
and emotional variables. The ESM measures are also compared at the person-level with students'
PISA and schoolwork engagement items. It is hypothesized that students who show higher levels
2

There are no significant differences between the students in phases 1 and 2.

28

of overall school engagement would experience higher frequencies of optimal learning moments
in school. To address the third research question, the measures of engagement are tested across
contexts, how does engagement occur in science specifically, and does this look different than
engagement outside of science?
Results
The summary statistics presented in Table 2.3 highlight the differences between
calculating the average across all momentary observations (un-nested) and the average of the
person-level (nested) averages. This nested average is calculated using each student’s mean for
every ESM variable, both overall and in science class. Examining the standard deviations
between raw scores compared to the nested scores, the variation is lower for the nested values as
some of the variation is explained by the differences between students. Overall, there are not
practical differences in the means between the raw and nested scores when averaged across all

29

Table 2.3
Summary Statistics for ESM Variables
Overall
Raw
Nested
Scores Raw Scores
n=8,485
n=278
Anxious
1.89 (1.04) 1.91 (.67)
Active
2.17 (1.12) 2.20 (.75)
Bored
2.31 (1.17) 2.37 (.64)
Challenge
1.85 (1.03) 1.92 (.51)
Competitive
1.88 (1.04) 1.88 (.68)
Concentrate*
2.64 (.99) 2.62 (.47)
Confident
2.59 (1.05) 2.56 (.73)
Confused
1.62 (.90) 1.68 (.54)
Cooperative
2.46 (1.07) 2.44 (.69)
Enjoy
2.72 (1.11) 2.66 (.56)
Give up*
1.52 (.86) 1.54 (.45)
Happy
2.85 (1.06) 2.81 (.65)
Important future
2.26 (1.15) 2.25 (.62)
Important you
2.62 (1.11) 2.57 (.58)
In control
3.02 (1.05) 2.97 (.62)
Interest
2.71 (1.12) 2.67 (.58)
Lonely
1.58 (.92) 1.61 (.64)
Other's expectations 2.91 (1.01) 2.85 (.69)
Proud
2.31 (1.09) 2.26 (.74)
Self expectations
3.04 (.99) 2.98 (.67)
Success
3.22 (.92) 3.16 (.53)
Skill
3.15 (.99) 3.06 (.58)
Stress
1.88 (1.04) 1.92 (.71)
Time fly*
2.57 (1.11) 2.56 (.55)

In Science
Raw
Nested
Scores Raw Scores
n=1,813
n=278
1.88 (.99) 1.93 (.69)
2.14 (1.08) 2.20 (.79)
2.40 (1.13) 2.50 (.70)
2.08 (1.05) 2.00 (.55)
1.77 (.96) 1.88 (.69)
2.76 (.91) 2.69 (.46)
2.45 (1.02) 2.56 (.74)
1.79 (.95) 1.77 (.57)
2.46 (1.04) 2.49 (.71)
2.49 (1.05) 2.53 (.71)
1.57 (.88) 1.59 (.52)
2.58 (1.06) 2.74 (.71)
2.29 (1.09) 2.32 (.68)
2.54 (1.03) 2.87 (.68)
2.78 (1.06) 2.87 (.68)
2.63 (1.06) 2.58 (.62)
1.52 (.85) 1.59 (.67)
2.88 (.96) 2.86 (.69)
2.11 (1.02) 2.26 (.75)
2.98 (.96) 2.98 (.68)
3.05 (.92) 3.13 (.53)
2.87 (.99) 2.99 (.60)
2.00 (1.06) 1.98 (.74)
2.36 (1.05) 2.44 (.58)

Notes. Means with standard deviations reported in parentheses. *Items only
included in second phase of data collection (n of 3,844 observations and n of 141
students)

observations. The highest reported affective state overall and in science is feeling successful,
with a mean above three. Feeling in control, skillful, and meeting self-expectations also have
means above three overall using the raw scores. In science, for raw scores and nested, there are
no means greater than three. Lower averages are reported for feeling lonely both overall (µ=1.58

30

for raw and µ=1.61 for standardized) and in science (µ=1.52 for raw and µ=1.59 for
standardized). Over all daily experiences, feeling active varies the most (σ=1.11 for the raw
scores and σ=.67 for the nested scores). The variation in the activeness of students also has the
highest variation in science using the standardized scores (σ=.79), however, feeling bored has the
highest variation using the raw scores (σ=1.13). These slight differences in means and variation
capture how sensitive these measures can be to each experience and situation and underscore the
importance of measuring these social and emotional aspects of learning in science.
Table 2.4 shows the summary statistics for the cross-situational measures of science and
school engagement. There are some differences between hard sciences and life science, as well
as gender, with females in life science reporting significantly higher levels of burnout compared
to males in life sciences and females in the hard sciences. Both males and females enrolled in the
hard sciences have higher science value and self-concept compared to those in the life sciences.
This may be due to selection effects of students in physics classes in the sample, a class which is
often taken in eleventh or twelfth grade and may also attract higher ability students.

Table 2.4
Summary Statistics of Cross-Situational Engagement Measures
Hard Sciences
Life Science a
PISA Scales
F
M
F
M
b
71%
63%
77%
83%
Science Enjoyment
Science Value
83%
77%
84%
89%
Science Motivation
65%
59%
66%
68%
Science Self-Concept
60%
66%
80%
83%
Future Aspirations in Science
33%
27%
41%
40%
c
58%
38%
45%
48%
School Burnout
School Engagement
40%
30%
29%
42%
Notes. aLife science includes biology students, hard science includes physics and
chemistry b PISA-scaled items. cSchoolwork Inventory. Bolded values statistically
different using a t-test at p <.05.

31

Examining optimal experiences more closely, Table 2.5 shows the standardized measures
(using z-scores) compared to optimal moments calculated using the raw scores. Overall for both
measures, students are in optimal moments about 14% of the time. In science, these optimal
moments are lower for both measures, occurring about 12% of the time using the raw scores
compared to 10% of the time using standardized scores. Both optimal experience measures are
consistent with previous studies showing that flow moments are the exception, not the norm, and
occur relatively infrequently (Csikszentmihalyi, 1998).
Given that both standardized and raw scores of optimal moments occur about the same
frequency, Table 2.6 shows how the measures are correlated with other affective states. Overall,
both measures function similarly. They are both moderately correlated with enjoyment, and
importance to self, while negatively correlated with boredom. The raw measures are also
correlated with feelings of competition, concentration, importance to future, and being active.
Both engagement measure correlations are slightly different in science, with being confused
negatively correlated with both constructs. For the standardized flow measure, happiness is
positively correlated in science while stress is negatively correlated. While there are slight
differences, these correlations are consistent with optimal moments, thus additional tests are
needed to determine how these constructs may function differently.
More narrowly examining the relationship between optimal experiences, concentration,
persistence, and time flying specifically, Table 2.7 shows how the raw and standardized
measures are related to these feelings associated with being in flow. If a student was in an
optimal experience, the percentage of responses in each of the four Likert scale options was
calculated. The standardized measure shows more responses at the higher end of time flying by

32

Table 2.5
Percentage of Time Students are in Flow
Overall
Raw
Standardized
N of moments
1099
1,077
% of moments
14%
14%

In School (no science)
Raw
Standardized
459
439
14%
14%

Raw
209
12%

In Science
Standardized
178
10%

Notes. Flow is a dummy variable where 1= flow, calculated two ways. "Raw Scores" use the values reported from the 4-point
likert scale. When students report a 3 or 4 for challenge, skill, and interest, this is coded as a 1, all other combinations are a 0.
For the standardized values, a z-score is created for each student and when these moments are the mean zero, this is coded as a
1. All moments are included in these figures, low responders included.

33

Table 2.6
Correlations with Optimal Moments (challenge, interest & skill)
Overall (n=8,485)
Science (1,813)
Raw
Standard
Raw
Standard
Anxious
0.1658 0.0943 0.1179 0.0793
Active
0.2155 0.1643 0.1816 0.1395
Bored
-0.0893 -0.0992 -0.1204 -0.1435
Competitive
0.2587 0.1824 0.2372
0.184
Concentrate*
0.2605 0.2128 0.2335 0.1348
Confident
0.1573 0.1715 0.1540 0.1655
Confused
0.1079 0.0271 -0.0016 -0.0322
Cooperative
0.1370 0.1355 0.0973 0.0647
Enjoy
0.2312 0.2510 0.2322 0.2603
Happy
0.1583 0.1764 0.1741 0.2187
Give Up*
0.0869 -0.0254
0.024
-0.0087
Important Future
0.2674 0.1949 0.2573 0.1453
Important Self
0.2527 0.2262 0.2816 0.2332
In Control
0.1359 0.1715 0.1496 0.1820
Lonely
0.0612 0.0281 0.0114 0.0165
Other Expect
0.1526 0.1661 0.1431 0.1087
Self Expect
0.1442 0.1636 0.1560 0.1712
Proud
0.1952 0.1802 0.1863 0.1501
Success
0.1011 0.1633 0.1133 0.1808
Stress
0.1032 0.0452 0.0416 -0.0260
Time Fly*
0.199
0.2053 0.2035 0.2097
Notes. The raw flow measures is correlated with raw scores of the other affective
states. The standardized flow variable is correlated with the other standardized
values

(46% vs. 39%), concentration (49% vs. 40%), and at the lower end of wanting to give up (70%
vs. 58%). This suggests that standardized measures may more precisely tap into person-specific
engagement than using raw scores.

34

Table 2.7
Flow Measures Compared to Validation Items
Time flies by
1
2
3
Raw (n=539 )
5%
18%
38%
Standardized (n=473 )
6%
19%
29%

4
39%
46%

Wants to give up
2
3
21%
15%
17%
6%

1
58%
70%

4
6%
6%

1
2%
4%

Notes. These are percentages of each category that a student reports feeling in the moment when the flow measure is equal to 1.

35

Concentrate
2
3
9%
49%
13%
44%

4
40%
49%

Lastly, Table 2.8 shows how the cross-situational measures are associated with in-themoment optimal learning. As one might expect, students that enjoy science, who are highly
motivated in learning science, and who have a higher self-concept in science all experience more
frequent optimal learning moments in science more frequently than those in the lower groups.
Examining the school engagement and burnout measures, students that have higher schoolwork
engagement, have more engaging experiences in science, and those students who show higher
burnout, experience engagement less often than those students who have low levels of burnout.
These findings show that there are consistent relationships between different measures of
engagement, and that using multiple measures can help assess the components of optimal
learning experiences, both in and out of science classes.

Table 2.8
Percentage of Students Engaged in Science
In Science
Other Academic
High Enjoyment
13%
14%
Low Enjoyment
7%
15%
High Motivation
Low Motivation

15%
6%

16%
13%

High Self-Concept
Low Self-Concept

13%
9%

16%
12%

High Aspirations in Science
Low Aspirations in Science

19%
8%

18%
13%

High School Engagement
Low School Engagement

15%
10%

20%
12%

High School Burnout
Low School Burnout

9%
14%

14%
14%

Notes. The italicized percentages are statistically significant (p <.05). The
N of students is 278, with a total of 8,474 momentary observations.
"Percent Engaged" is calculated over the school day, in science
compared to out of science.

36

Discussion
These findings show that there are indeed relationships between different measures of
engagement, and that using multiple measures of engagement can help assess the components of
creating optimal learning experiences. Using both cross-situational and in the moment measures
of engagement provide data that is domain-specific, can tap situation-specific variation, and can
be analyzed with corresponding measurement models (e.g. formative vs. reflective). Although
there were some similarities and differences in how the raw scores and standardized engagement
measures functioned, both appear to be valid constructs of engagement. Since both of these
measures capture optimal experiences, the choice of whether to use raw or standardized scores is
a product of the research questions, structure, and context of the model and type of analysis
being conducted. While this chapter analyzed the conceptualization and definitions of
engagement, it is just a first step in helping to operationalize optimal learning experiences to help
improve teaching and learning in science. The subsequent chapters analyze how student
background contributes to student engagement as well as how different instructional approaches
contribute to student engagement.
Limitations
Despite the many advantages associated with using ESM to measure optimal learning
moments there are also drawbacks to this approach. Data collection at multiple points of the day
over multiple days can present challenges for the respondent, which can lead to selection bias
and attrition in the study sample (Scollon, Kim-Prieto, and Diener, 2003). As seen in this dataset,
the average response rate of students is about four responses per day, which is only about half of
the total possible responses. While a 50% response rate is often problematic in research because
of the bias it can introduce into the analyses, it can be argued that this is still an abundance of

37

data that can provide rich information about the student experiences, and provides more data
than a single-time survey.
The application of these methods for studying student engagement could also be
disruptive for learning. On the other hand, it may also provide an opportunity for an individual to
re-engage with their task if they perhaps were not on-task. Although ESM collects real-time data,
the data are still self-reported and subject to the same limitations of any self-reported data.
Despite the time and intensity of collecting ESM data, several participants have noted that it can
be a positive experience (DiBianca, 2000) and there could be several positive psychological
effects of monitor one’s engagement throughout the day, allowing for multiple opportunities for
reflection in the moment of experiences.
Implications
With the rapidly growing access to, and use of technology in research and in the
classroom, there are greater opportunities for innovation in studying student engagement with
new technology. Advances in smartphone and mobile application (app) technology provide new
ways to use ESM in the classroom. Using smartphones with ESM apps specially designed to
measure optimal experiences reduces some of the burdens related to data collection for both the
subjects and researchers compared to previous paper-and-pencil methods.
The use of smartphones in education is also growing in the adolescent population, even
those students from low-income households, with approximately one in three students using their
phones for help on homework (Khadaroo, 2012). Growth can also be seen in smartphone
companies that spend an estimated 20 billion dollars a year on research and development.
Contrast that to the annual spending of the National Science Foundation, around 250 million
dollars, and it is evident that for this study population and given the direction of advances in

38

technology, having the ability to use smartphones to study engagement is not only cutting-edge,
but is quickly becoming a necessity. Future research could also incorporate additional biological
data from the smartphones as well, such as heart-rate or activity levels of the students.

39

APPENDICES

40

Appendix A: Student ESM Questionnaire
1.
2.
3.
4.
5.
6.

Where were you/in which class?
Who were you with?
What were you doing?
Why? [You wanted to, You had to, You had nothing else to do)
Is what you were doing… [more like school work, more like play, both, neither]
What were you thinking?

II. How do you feel about what you are doing? (4-point scale):
7. Were you interested in what you were doing? [Not at all/Very much]
8. Did you feel skilled at what you were doing? [Not at all/Very much]
9. Did you feel challenged by what you were doing? [Not at all/Very much]
10. Did you feel like giving up? [Not at all/Very much]
11. How much were you concentrating? [Not at all/Very much]
12. Did you enjoy what you were doing? [Not at all/Very much]
13. Did you feel like you were in control of what you were doing? [Not at all/Very much]
14. Were you succeeding? [Not at all/Very much]
III. How did you feel about the main activity? (4-point scale: Not at all/Very much)
15. Was this important for you?
16. How important was this activity in relation to your future goals/plans?
17. Were you living up to the expectations of others?
18. Were you living up to your expectations?
19. I was so absorbed in what I was doing that the time flew.
IV. How were you feeling? (4-point scale: Not at all/Very much)
20. Were you feeling…Happy
21. Were you feeling…Excited
22. Were re you feeling…Anxious
23. Were you feeling…Competitive
24. Were you feeling…Lonely
25. Were you feeling…Stressed
26. Were you feeling…Proud
27. Were you feeling…Cooperative
28. Were you feeling…Bored
29. Were you feeling…Self-confident
30. Were you feeling…Confused
31. Were you feeling…Active

41

Appendix B: Student Background Survey

How much do you agree with the statements below:
1 – Strongly disagree, 2 – Disagree, 3 – Agree, 4 – Strongly agree
a. I generally have fun when I am learning Biology
b. I like reading about Biology
c. I am happy doing Biology
d. I enjoy acquiring new knowledge in Biology
e. I am interested in learning about Biology

1
1
1
1
1

2
2
2
2
2

3
3
3
3
3

4
4
4
4
4

1

2

3

4

1

2

3

4

1
1
1
1
1
1
1
1

2
2
2
2
2
2
2
2

3
3
3
3
3
3
3
3

4
4
4
4
4
4
4
4

1

2

3

4

1

2

3

4

1
1

2
2

3
3

4
4

1

2

3

4

How much do you agree with the statements below:
1 – Strongly disagree, 2 – Disagree, 3 – Agree, 4 – Strongly agree
a. Advances in Biology usually improve people’s living
conditions
b. Biology is important for helping us to understand the natural
world
c. Concepts in Biology help me see how I relate to other people
d. Advances in Biology usually help improve the economy
e. I will use Biology in many ways when I am an adult
f. Biology is valuable to society
g. Biology is important to me personally
h. Biology helps me to understand the things around me
i. Advances in Biology usually bring social benefits
j. When I leave school there will be many opportunities for me to
use Biology
How much do you agree with the statements below:
1 – Strongly disagree, 2 – Disagree, 3 – Agree, 4 – Strongly Agree
a. Making an effort in my Biology class is worth it because this
will help me in the work I want to do later on
b. What I learn in my Biology class is important for me because I
need this for what I want to study later on
c. I study Biology because I know it is useful for me
d. Studying for my Biology class is worthwhile for me because
what I learn will improve my career prospects
e. I will learn many things in my Biology class that will help me
get a job

How much do you agree with the statements below:
42

1 – Strongly disagree, 2 – Disagree, 3 – Agree, 4 – Strongly Agree
a. Learning advanced Biology topics would be easy for me
b. I can usually give good answers to test questions on Biology
topics
c. I learn Biology topics quickly
d. Biology topics are easy for me
e. When I am being taught Biology, I can understand the concepts
very well
f. I can easily understand new ideas in Biology

1
1

2
2

3
3

4
4

1
1
1

2
2
2

3
3
3

4
4
4

1

2

3

4

1
1
1
1

2
2
2
2

3
3
3
3

4
4
4
4

How much do you agree with the statements below:
1 – Strongly Disagree, 2 – Disagree, 3 – Agree, 4 – Strongly Agree
a. I would like to work in a career involving Biology
b. I would like to study Biology after High School
c. I would like to spend my life doing advanced Biology
d. I would like to work on Biology projects as an adult

Please choose the alternative that best describes your situation:
1 – Strongly Disagree, 2 – Disagree, 3 – Somewhat Disagree, 4 – Somewhat Agree, 5 – Agree, 6
– Strongly Agree
a. I feel overwhelmed by my schoolwork
b. I feel a lack of motivation in my schoolwork and often think of
giving up
c. I often have feelings of inadequacy in my schoolwork
d. I often sleep badly because of matters related to my schoolwork
e. I feel that I am losing interest in my schoolwork
f. I’m continually wondering whether my schoolwork has any
meaning
g. I worry over matters related to my schoolwork a lot during my
free time
h. I used to have higher expectations of my schoolwork than I do
now
i. The pressure of my schoolwork causes me problems in my close
relationships with others

1
1

2
2

3
3

4
4

5
5

6
6

1
1
1
1

2
2
2
2

3
3
3
3

4
4
4
4

5
5
5
5

6
6
6
6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

Please choose the alternative that best describes your situation (estimation from the previous
month):
0- Never, 1 – A couple of times a year, 2 – Once a month, 3 – A couple of times a month
43

4 – Once a week, 5 – A couple of times a week, 6 –Daily
a. At school I am bursting with energy
b. I find the schoolwork full of meaning and purpose
c. Time flies when I am studying
d. I feel strong and vigorous when I am studying
e. I am enthusiastic about my studies
f. When I am working at school, I forget everything else around
me
g. My schoolwork inspires me
h. I feel like going to school when I get up in the morning
i. I feel happy when I am working intensively at school

44

0
0
0
0
0
0

1
1
1
1
1
1

2
2
2
2
2
2

3
3
3
3
3
3

4
4
4
4
4
4

5
5
5
5
5
5

6
6
6
6
6
6

0
0
0

1
1
1

2
2
2

3
3
3

4
4
4

5
5
5

6
6
6

REFERENCES

45

REFERENCES

Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational
Psychology, 84(3), 261.
Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students’ learning strategies
and motivation processes. Journal of Educational Psychology, 80(3), 260–267.
doi:10.1037/0022-0663.80.3.260
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change.
Psychological Review, 84(2), 191-215.
Csikszentmihalyi, M. (1998). Finding flow. New York: Basic Books.
Csikszentmihalyi, M., & Larson, R. (1987). Validity and reliability of the experience-sampling
method. The Journal of Nervous and Mental Disease, 175(9), 526-537.
Csikszentmihalyi, M., Larson, R., & Bruce, S.C. (1977). The ecology of adolescent activity and
experience. Journal of Youth and Adolescence, 6, 281-294.
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper
Perennial.
Csikszentmihalyi, M., & Schneider, B. (2000) Becoming adult. New York: Basic Books.
DiBianca, R. (2000). Teaching Adolescents: Relationships between features of instruction and
student engagement in high school mathematics and science classrooms. Chicago:
University of Chicago Press.
Dweck, C.S. (1986). Motivational processes affecting learning. American Psychologist, 41,
1040-1048.
Eccles, J.S. & Wang, M. (2012). Part I commentary: So what is student engagement anyway? In
S.L. Christenson, A.L. Reschly & C. Wylie (Eds.) Handbook of Research on Student
Engagement, Springer Sciences, New York. 133-145.
Finn, J.D. (1993). School engagement and students at risk. Washington, DC: National Center for
Education Statistics.
Fredricks, J. A., Blumenfeld, P.C., & Paris, A. H. (2004). School engagement: Potential of the
concept, state of evidence. Review of Educational Research, 74(1), 50-109.

46

Goetz, T., Bieg, M., Lüdtke, O., Pekrun, R., & Hall, N. C., (2013). Do girls really experience
more anxiety in mathematics? Psychological Science, 24, 2079-2087. doi:
10.1177/0956797613486989
Gonzales, P., Williams, T., Jocelyn, L., Roey, S., Kastberg, D., & Brenwald, S. (2008).
Highlights from TIMSS 2007: Mathematics and science achievement of U.S. fourth-and
eighth-grade students in an international context. Washington, DC: National Center for
Education Statistics. Retrieved from http://nces.ed.gov/pubs2009/2009001.pdf
Hektner, J.M., Schmidt, J.A., & Csikszentmihalyi, M. (2007). Experience sampling method:
Measuring the quality of everyday life. Thousand Oaks, CA: Sage Publications.
Khadaroo, S. T. (2012). “Not just 4 texting: 1 in 3 middle-schoolers uses smart phones for
homework.” The Christian Science Monitor. Retrieved from:
http://www.csmonitor.com/USA/Education/2012/1129/Not-just-4-texting-1-in-3-middleschoolers-uses-smart-phones-for-homework
Lawson, M. A., & Lawson, H. A. (2013). New conceptual frameworks for student engagement
research, policy, and practice. Review of Educational Research, 83(3), 432-479.doi:
10.3102/0034654313480891
National Academies. (2010a). Rising above the gathering storm, revisited: Rapidly approaching
category 5. Washington, DC: The National Academies Press. Retrieved from
http://www.nap.edu/catalog/12999.html
National Center for School Engagement. (2006). Quantifying school engagement: Research
report. Denver, CO: National Center for School Engagement.
National Governors Association. (2012). Common Core State Standards Initiative. Retrieved
from: http://www.corestandards.org/
National Research Council. (2004). Engaging schools: Fostering high school student’s
motivation to learn. Committee on Increasing High School Students’ Engagement and
Motivation to Learn. Board on Children, Youth, and Families, Division of Behavioral and
Social Sciences and Education. Washington, DC: The National Academies Press.
National Science Board. (2010). Preparing the next generation of STEM innovators: Identifying
and developing our nation's human capital. Arlington, VA: National Science Foundation.
Retrieved from http://www.nsf.gov/nsb/publications/2010/nsb1033.pdf
Newmann, F.M. (Ed.) (1992). Student engagement and achievement in American secondary
schools. New York: Teachers College Press.
Ream, R. K. & Rumberger, R. W. (2008). Student engagement, peer social capital, and school
dropout among Mexican American and non-Latino white students. Sociology of
Education, 81(2), 109-139.
47

Salmela-Aro, K., & Upadaya (2012). The schoolwork engagement inventory: energy, dedication,
and absorption (EDA). European Journal of Psychological Assessment, 28(1), 60–67.
Schunk, D. H., Pintrich, P. R., & Meece, J. L. (2008). Motivation in education: Theory, research,
and applications. New Jersey: Pearson/Merrill Prentice Hall.
Scollon, C. N., Kim-Prieto, C., & Diener, E. (2003). Experience sampling: Promises and pitfalls,
strengths and weaknesses. Journal of Happiness Studies, 4, 5-34.
Shernoff, D. (2001). The experience of student engagement in high school: A phenomenological
perspective. Chicago: University of Chicago Press.
Shernoff, D. & Csikszentmihalyi, M. (2009). Flow in schools: Cultivating engaged learners and
optimal learning environments. In R. Gilman, E. S. Huebner, & M. Furlong (Eds.),
Handbook of positive psychology in schools (pp. 131-145). New York: Routledge.
Shernoff, D., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. (2003) Student engagement in
high school classrooms from the perspective of flow theory. School Psychology
Quarterly, 18(2), 158-176.
Shumow, L. & Schmidt, J.A. (2014). Enhancing Adolescents’ Motivation for Science: ResearchBased Strategies for Teaching Male and Female Students. Thousand Oaks, CA: Corwin,
A Sage Company.
Skinner, E. A., & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of
teacher behavior and student engagement across the school year. Journal of Educational
Psychology 85(4), 571-581.
Stodolsky, S. (1988). The subject matters: Classroom activity in math and social studies.
Chicago: University of Chicago Press.
Trochim, William M. (n.d.). The Research Methods Knowledge Base, 2nd Edition. Retrieved
from http://www.socialresearchmethods.net/kb/
Turner, J. C., & Meyer, D. K. (2000). Studying and understanding the instructional contexts of
classrooms: Using our past to forge our future. Educational Psychologist, 35(2), 69-85.
Turner, J. C., & Meyer, D. K. (2004). A classroom perspective on the principle of moderate
challenge in mathematics. Journal of Educational Research, 97, 311-318.

48

CHAPTER 3: CONTEXTUALIZING VARIATION OF ENGAGEMENT EXPERIENCES:
THE ROLE OF OPTIMAL LEARNING MOMENTS IN SCIENCE
Students experience optimal learning moments about 14% of the time, however, this
frequency drops to about 10% when students are in science. What are the students doing in
science that contributes to lower occurrences of optimal learning moments? Despite the potential
for creating optimal learning experiences motivated by relevant content, such as understanding
global warming or stem-cell research, secondary science teachers are battling against declining
attitudes and motivation in science (Wigfield & Eccles, 1992; Gonzales et al., 2008; Osborne,
Simon, & Collins, 2003). Do optimal learning moments vary by students or instructional
characteristics? Moreover, what are the specific circumstances for students, such as individual
background and context, that contributes to the student persisting through challenging tasks both
in and out of the science classroom, and which supports their efforts despite difficulty? This
chapter examines these very questions and what contributes to optimal learning for different
types of students and teachers. This chapter specifically explores: (1) How does engagement (as
measured by optimal experiences) vary across students? (2) What instructional factors are
associated with optimal learning moments in science? and (3) What characteristics can predict
optimal learning moments?
Dimensions of the Optimal Experiences and Learning
When individuals are in flow, or having an optimal experience, there are often eight
phenomenology components that accompany these moments, including: (1) concentration and
focus on the task; (2) clarity, which is knowing what needs to be done and how to do it; (3)
knowing that the task is doable, and there is a chance to complete it; (4) the task provides
feedback; (5) a sense of serenity, less worry and anxiety; (6) a sense of control and autonomy;
(7) the focus on the present task can make time feel as if it is flying by; and (8) the sense of self
49

disappears, the experience is intrinsic in nature (Csikszentmihalyi, 1990). While these
phenomenological factors were studied across a variety of situations and contexts (e.g.
interviews with artists and composers), these characteristics of flow most certainly can be
applied to learning experiences. Given that optimal experiences occur when the level of
challenge and skill are in balance, it is problematic when students report having a low level of
challenge in their science classes (Schneider et al., under review; Shumow & Schmidt, 2014).
Thus, one can hypothesize that changing the level of challenge, or how students respond to
challenge could provide increased opportunities for optimal learning moments, which supports
an enjoyable experience in science.
Engagement in Science
While there is a growing need for increasing interest and achievement in science, often
the subject of science is grouped together by researchers and stakeholders with math, technology,
and engineering as a single field (STEM). However, research on student learning and
engagement to shows variation across subjects (Stodolskey, 1988). Therefore, it is important to
consider student learning experiences in content-specific settings. In math and science classes in
particular, research suggests that students need to be engaged in meaningful learning experiences
that mirror similar activities of those in STEM fields (e.g. chemists or engineers), such as
simulations, developing models, or collaborative problem solving (Brann, Gray, Piety, & SilverPacuilla, 2010). These types of instructional formats often utilize more student-controlled
activities compared to teacher-controlled (Stodolskey, 1988).
In high school math classes, several studies show higher levels of engagement compared
to other subject areas, however, it is hypothesized that this is due to the aligned nature of math
content where teachers are held accountable for preparing their students for the next course level,

50

and increased pressure from standardized tests and accountability measures (Marks, 2000).
Despite the demand for improved instruction and the desire to increase student engagement in
math and science classes, research shows that in most classrooms, students still spend a majority
of time on less engaging activities, such as independent work, listening to lectures, and taking
notes (DiBianca, 2000; Shernoff et al., 2001; Shernoff et al., 2003).
Teachers and Instruction
Teachers are a key factor in developing conditions for high levels of student engagement as
they have knowledge of both the content and of their students. Teachers must not only
understand the curriculum and know their students, but also have a strong grasp of instructional
strategies, student development, and ways to create optimal learning experiences in their
classrooms (Good & Brophy, 2003). As mentioned above, these optimal learning experiences are
often present during both intellectual and emotional experiences (Csikszentmihalyi & Schneider,
2000). Thus, instruction should be designed not only to foster these learning moments, but create
an environment that supports both the intellectual and emotional experiences of students as they
learn. For example, Turner and Meyer (2004) found that in classes with high engagement,
teachers fostered intrinsic motivation and included more scaffolded instruction that allowed the
content to be adjusted so that a student could match their challenge and skill.
Classrooms are dynamic places, and on a daily basis teachers must plan for, manage, and
adapt to a multitude of requirements, expectations, and demands. It is within this complex
environment where the relationship between instruction and engagement is observed and
measured. It is important to acknowledge that fostering student engagement is just one
component of a multifaceted classroom setting. Given these multiple dimensions of teaching and

51

learning, teachers find themselves fostering student engagement in multiple ways given the
contextual environment of the classroom.
Teacher Behavior. While there is substantial research on the different types of
instructional activities that can lead to improved student engagement and student outcomes, it is
also critical to understand the role of teacher expectations in developing engaged students,
acknowledging that although instructional activities are important, they are often not alone in
cultivating student engagement. If engagement is the result of matching a student’s challenge
threshold and skill level, a teacher must first possess an expectation or belief that a student
possesses the skills needed for the activity and can engage in a challenging task.
The expectations teachers hold about their students are predictions teachers make about the
future behavior of their students based on their present knowledge (Good & Brophy, 2003).
These teacher expectations and beliefs often influence the actions teachers take (Skinner &
Belmont, 1993), including support and instructional practices. Teacher support is comprised of
factors that fulfill a student’s basic psychological needs, including: structure, clear expectations
and guidelines for learning; autonomy, students having latitude in their own learning as opposed
to teacher-controlled or coercive learning; and involvement, which is the interpersonal
relationships among the teacher, student, and peers (Connell & Wellborn, 1991; Fredericks et al.,
2004; Klem & Connell, 2004; Skinner & Belmont, 1993). Research shows that students with
high levels of teacher support are more likely to be engaged; Klem and Connell (2004) found
that highly supported students in secondary schools were approximately three times more likely
to show high levels of engagement than students with low levels of teacher support.
Instruction. A teacher can use a variety of instructional methods in the classroom,
including lecture, taking notes, completing worksheets, class discussion, taking quizzes or tests,

52

group projects, and class presentations. In math and science classes, the most common
instructional methods include lecture, discussion, testing, demonstration, lab work, and seat work
(DiBianca, 2000; Weiss, 1997). These different instructional methods may influence how
students engage in learning and are an important factor in measuring and understanding student
engagement. Despite research that students are engaged by instructional activities that are
student-controlled, less teacher-centered, and provide a match between challenge and skill, all
while experiencing enjoyment (Csikszentmihalyi & Schneider, 2000; DiBianca; 2000; Shernoff,
2001; Shernoff et al., 2003; Singh, Granville, & Dika, 2002; Skinner & Belmont, 1993), students
in science and mathematics classes often spend their time on less-engaging tasks. Shernoff
(2001) found that students were more likely to be engaged in non-academic classes (e.g. art and
vocational education) than their academic courses, and that academic classes in general used
high-engaging tasks less frequently than non-academic classes.
The other component of instruction that is important for understanding student engagement
is the instructional relevance, or as Newmann, Wehlage and Lamborn (1992) proposed, authentic
instruction, which is the significance and value for the student of what is being learned. Newman
et al. (1992) set forth three student outcomes of authentic achievement in which authentic
instruction should aim to create, including: constructing meaning and producing knowledge;
using disciplined inquiry; and working towards completing products or performances that have
value beyond just success in school. Given these desired student outcomes, authentic instruction
calls for five standards, which align well with the literature on student engagement: (1) higherorder thinking; (2) depth of knowledge; (3) connectedness to world beyond the classroom; (4)
substantive conversation; (5) social support for student achievement. Instruction that targets
active student involvement in challenging activities contributes to high student engagement,

53

providing optimal learning experiences and construction of knowledge (Newmann et al., 1992).
However, as several have argued, engaged students must connect what they are learning to larger
goals or interests beyond the classroom, which implies that not only is the context inside the
classroom important, but the context outside of the class-such as the culture of the school
(Easton, 2008; Marks, 2000; Newmann, 1992; Schlechty, 2011).
Student Background
Student engagement in school also varies across different student background
characteristics such as gender, race/ethnicity, and socioeconomic status. Across all grades, girls
are consistently more engaged than boys (Finn & Cox, 1992; Lee & Smith, 1995). In high
school, minority students are more likely to be engaged in academic work compared to their
white counterparts, reversing the trend in elementary school where minority students were less
likely to be engaged compared to white students (Lee & Smith, 1995). Students from higher
socioeconomic (SES) backgrounds are also more likely to be more engaged compared to their
low-SES peers (Finn & Cox, 1992; Lee & Smith, 1995). While student background is an
important factor in understanding student engagement, teachers remain the central component to
increasing student engagement by fostering academically challenging environments that create
authentic learning experiences (DiBianca; 2000; Newmann, 1992; Shernoff, 2001; Shernoff et
al., 2003; Singh, Granville, & Kika, 2010; Skinner & Belmont, 1993).
Theoretical Framework
The framework for this chapter follows that of the dissertation, and similar to chapter two,
this chapter will more intensively examine optimal learning experiences in science using the
ESM data. Building on the results of chapter two, this chapter will evaluate additional layers of
context for understanding variation in how optimal learning moments occurs, with a specific

54

focus in science classrooms. Using a smaller version of the larger conceptual framework, Figure
3.1
Figure 3.1 Contextualizing Optimal Learning Moments in Science

shows the relationships that will be examined between students, contexts, and optimal learning
experiences. These optimal learning moments, represented by the gray box, are subject to a
student’s context as well as their in the moment subjective states. Student characteristics,
including gender, career plans in science, enjoyment and value of science, and overall
engagement may also influence optimal learning moments. The contextual factors explored in
these analyses include instructional modes reported by the teachers as well as the student. As
mentioned earlier, being in an optimal experience often supports feelings such as enjoyment,
concentration, and control, which is represented here by the arrow that proceeds from the optimal
learning moment box to the phenomenological factors.

55

Research Questions
To more narrowly investigate these moments of optimal learning along with the other
phenomenological dimensions, this chapter examines:
1. How does engagement (as measured by optimal learning experiences) vary in and out of
science and across students by gender, future occupation, enjoyment and value of
science, and overall school engagement?
2. What instructional factors are associated with optimal learning moments?
3. What student and instructional factors can predict optimal learning moments? Does this
vary for science classes?
Methodology
Data and Sample
The same ESM dataset used in chapter two is used in this set of analyses. Since
comparisons are made between science and other academic contexts, the sample is restricted to
students who have at least one response in science class and five responses outside of science.
While traditional ESM methodology suggests keeping only participants who responded to at
least 15 notifications (Hektner, Schmidt, & Csikszentmihalyi, 2007), since engagement is a
primary outcome, removing students who have lower response rates may bias the sample by
leaving in students who exhibit high rates of overall optimal learning moments. Sensitivity
analyses comparing the full sample, restricted sample of six responses, and using a threshold of
15 responses are included in Appendix C. The final full sample size for the descriptive analysis
included 256 students and a total of 7,908 momentary observations. However, because the
measure of persistence was only measured during the second phase of data collection, the sample

56

for the multi-level analysis is 129 students with a total of 3,449 momentary observations. Table
3.1 shows the sample used for these analyses.
Table 3.1
Descriptive Characteristics of Two-Level Data
Level-1 Descriptive Statistics
VARIABLE NAME
N
MEAN
SD
MINIMUM MAXIMUM
Optimal Learning
3,449
0.13
0.34
0
1
In science
3,485
0.36
0.48
0
1
Enjoyment
3,485
0.61
0.49
0
1
Importance to self
3,485
0.39
0.49
0
1
Importance to future
3,485
0.57
0.5
0
1
Persistence
3,468
0.65
0.48
0
1
Level-2 Descriptive Statistics
VARIABLE NAME
N
MEAN
SD
MINIMUM MAXIMUM
Male
129
0.52
0.5
0
1
Female
129
0.48
0.5
0
1
High Science Enjoyment
129
0.82
0.38
0
1
High Science Value
129
0.86
0.35
0
1
Science Aspirations
129
0.34
0.48
0
1
School Engagement
129
0.36
0.48
0
1
Notes. FLOWZ is a composite variable equal to 1 when a student is experiencing higher
than average challenge, skill, and interest.

Measures
The outcome of interest is a binary measure for an optimal learning moment. This measure
is equal to one when a student’s level of challenge, skill, and interest are reported above their
average level. These analyses use the standardized measure of optimal learning moments as
opposed to using raw scores as described in the previous chapter to take advantage of the ESM’s
ability to capture each student’s variation from their own person-centered value for each
affective state. A nested model is used to analyze the data, with level-1 (moment-level) responses
nested within level-2 (students). Five control variables were included at level-1 (moment-level)
including how important the student felt the task was to them and to their future, their enjoyment
57

of the task, their persistence, and if the response occurred in science. At level-2, student-level
covariates included gender, overall enjoyment of science, overall value of science, their interest
in a science career, and their overall school engagement.
Analytic Approach
To address the first research question of how optimal learning moments vary, descriptive
analyses are conducted to examine how these learning situations across a variety of settings. This
first set of analyses also explores the student’s reported volition in moments when they are in
optimal learning environments—both in and out of science class as well as in and out of school
and also observes how students perceive their task (more like work or more like play) across
contexts. Then, variation in optimal learning moments is explored by gender and student
attitudes towards science. The final step in the descriptive analyses observes the predominant
forms of instruction when students are in an optimal learning moment.
The nested structure of the ESM data (repeated measures within students) is analyzed using
a multi-level model, which is able to account for the shared variance of the ESM data of
moments occurring within individuals (Raudenbush & Bryk, 2002). Since the primary outcome
of interest is binary (equal to 1 when a student experiences higher than average levels of
challenge, skill, and interest), a Hierarchical Generalized Linear Model (HGLM) is used. The
full model estimated for predicting optimal learning moments is:
Level 1 (moment-level i)
𝑖𝑗 = 𝜋0𝑗 + 𝜋1𝑗 (𝑖𝑚𝑝𝑜𝑟𝑡𝑎𝑛𝑐𝑒𝑦𝑜𝑢) + 𝜋2𝑗 (𝑒𝑛𝑗𝑜𝑦) + +𝜋3𝑗 (𝑝𝑒𝑟𝑠𝑖𝑠𝑡) + 𝜋4𝑗 (𝑖𝑛𝑠𝑐𝑖𝑒𝑛𝑐𝑒)
Level 2 (student j)
𝜋0𝑗 = 𝛽00 + 𝛽01 (𝑓𝑒𝑚𝑎𝑙𝑒) + 𝛽02 (ℎ𝑖_𝑒𝑛𝑗𝑜𝑦) + 𝛽03 (ℎ𝑖_𝑓𝑢𝑡𝑢𝑟) + 𝛽04 (ℎ𝑖_𝑒𝑛𝑔𝑎𝑔) + 𝜀0𝑗
where 𝑖𝑗 = (𝜑𝑖𝑗 /1 − 𝜑𝑖𝑗 ) is the logit link function and

 𝜑𝑖𝑗 is the probability of optimal learning (at moment i for student j),
58

 importanceyou is a binary variable equal to 1 if the z-score for the importance of the task
is greater than 0
 enjoy is a binary variable equal to 1 if the z-score for the enjoyment of the task is greater
than 0
 persist is a binary variable equal to 1 if the z-score for wanting to give up is less than 0
 inscience is a binary variable equal to 1 if the response is in science class
 female is a binary variable equal to 1 if the student is female, and 0 if student is male
 hi_enjoy is a binary variable equal to 1 if the average for the PISA enjoyment scale is
greater than 3 on a 4-point scale
 hi_futur a binary variable equal to 1 if the average for the desiring a future in science
scale is greater than 3 on a 4-point scale
 hi_engag a binary variable equal to 1 if the average for the schoolwork engagement scale
is greater than 4 on a 7-point scale
 𝜀0𝑗 is a random effect for student j

Results
Examining what a student experiences when in an optimal learning moment across
contexts, Figure 3.2 shows the volition a student reports during a task when they are in an
optimal learning moment, meaning when students are engaged do they find the task something
that they want to do, had to do, or they had nothing else to do? Overall, students report wanting
to do their task 45% of the time when they are in an optimal moment overall, compared to only
29% of the time when they are in school. However in science students report higher levels of
wanting to do their task compared with being at school with 36% of students reporting that they
wanted to do their task in science compared to 29% while at school. Examining when students
59

felt like they had to do their task, in school was the highest category with 62% of students in an
optimal learning moment marking this response, in science was slightly lower at 56%, while
overall 43% of students report feeling that they had to do their task. Students in an optimal
moment did not often identify the reason for their activity as “because there was nothing else to
do,” which serves as an additional validation for this engagement measure as this is consistent
with the literature discussed above— that when a students is having an optimal experience, there
is a sense of control, concentration, and focus that would not be present if students did not have
anything else to do.
Figure 3.2 Student Volition of Task during an Optimal Learning Moment

Analyzing how a student in an optimal learning moment perceived their activity as work or
play, Figure 3.3 demonstrates there is variation across contexts. Students in an optimal moment
report 52% of the time that their activity as more like work in school and in science compared to
overall to 31% of the overall time. Students who are in an optimal moment report higher levels
of play at 35% compared to when in school and in science, which was 17% of the time. In
60

science, however, 23% of students report that their task feels like both work and play compared
to 20% in school, and 15% overall. This is a positive outcome for science experiences when
there is a certain amount of work and play that is present during a task. Students in an optimal
learning moment report that their activity does not feel like work or play 20% of the time overall,
compared to lower percentages in school (11%) and in science (8%). Similar to the results shown
in Figure 3.2, these findings further validate the measure of optimal learning, while also
demonstrating some positive results for the experiences of students in science class, that when
engaged the task is more likely to feel like work and play compared to other academic settings,
also a lower percentage of students report that their task feels neither like work or play, which
could indicate a lack of purpose or value in their activity.

Figure 3.3 Student Perception of Activity during an Optimal Learning Moment

Next, exploring how optimal learning experiences vary by student, Table 3.2 shows that
there are differences by gender in the total number of optimal learning moments, 16.74% of all
61

occurrences were male students compared to 10.38% for females. Male students, on average,
have more optimal moments, with an average of 4.80 compared to 3.34 for female students.
Restricting the time frame to during school hours reveals a similar trend that males in school
experience more optimal learning moments than females, 1.89 compared to 1.4 moments,
however this is not statistically significant. This gender difference holds in science as well, but
the gap is slightly smaller between males and females, about 12% compared to 9%. There is no
statistical significant difference between the number of optimal learning moments in science
between male and females, males experience an average of .82 compared to .61 for females—
both experience a low number.
Examining the gender differences more closely, Table 3.3 shows how students with
higher levels of science enjoyment, value, and career aspirations, and overall school engagement
experience optimal learning moments. The general trend of males experiencing more moments
of optimal learning than females holds. However when looking at these students that have more
positive attitudes associated with science, especially for males, these students experience higher
levels compared to all students. For example, for males who indicate that science is important for
their career, they experience an optimal learning moment about 21% of the time. Although these
percentages still remain lower in science classes compared to in school and overall, these
percentages are still higher than the full student sample. It may also very well be the case that
these student are highly motivated in all areas, not just in their science classes.
When analyzing what the students report working on when signaled in science, differences
can be observed between students in an optimal learning moment and students not in an optimal
experiences as shown in Table 3.4. The top five reported activity categories for these students in
an optimal learning moment include: science (20%); taking a test or quiz (19%); homework or

62

studying (14%); being in class (12%); and working on school work (11%). The rest of the
responses were varied and combined to equal 24%. While these answers may be less descriptive
about the kinds of “work” or “science” the student is doing, overall it is evident that these
students are at least demonstrating on-task behavior. Students not in an optimal learning moment
report working on: science (16%); school work (11%); a test of quiz (9%); homework or
studying (8%); and general work (3%). One key difference is that the high percentage of students

63

Table 3.2
Optimal Learning Moments by Gender
Overall
n=7,908
female
male
Percentage of time in Optimal Moments 10.38% 16.74%
Average number of Optimal Moments
3.34
4.80

p
0.00
0.02

In School
n= 4,909
female
male
10.54% 17.25%
1.4
1.89

p
0.00
0.19

In Science
n= 1,740
female
male
p
8.52% 11.85% 0.02
0.61
0.82
0.19

Notes. Full ESM sample used from phase I and II. Chi-squared used to test for significance for the percentages and a t-test is used for the average number
of learning moments

Table 3.3
Optimal Learning Moments by Student Characteristics
Overall
n=7,908
Percent of time
in optimal
learning
chimoments
square
male
female
(p)
Science Enjoyment
17.44% 10.86%
0.00
Future Aspirations in
Science
20.60% 10.41%
0.00
Science Value
17.76% 10.87%
0.00
Schoolwork Engagement
18.84% 12.28%
0.00

In School
n= 4,909
Percent of time
in optimal
learning
chimoments
square
male
female
(p)
16.73% 10.37%
0.00

In Science
n= 1,740
Percent of time
in optimal
learning
chimoments
square
male
female
(p)
13.11% 9.41%
0.00

21.79% 11.42%
18.55% 10.80%
21.16% 14.44%

20.54%
12.31%
18.25%

Source: Full ESM sample from phase I and II are used.

64

0.00
0.00
0.01

9.54%
8.49%
9.43%

0.04
0.02
0.00

Table 3.4
Student-reported Task in Science
Optimal Learning
%
Science
20%
Taking a test/quiz
19%
Homework/Study
14%
In Class
12%
School Work
11%
Other Categories (combined)
24%

Not in Optimal Learning
Science
School Work
Taking a test/quiz
Homework/Study
General Work
Other Categories (combined)

%
16%
11%
9%
8%
3%
53%

Notes. Open-response categories coded using codes from the 500 Family Study. The
"Science" category includes general science, biology, chemistry, and physics

(53%) reporting working on other types of activities, some examples reported include “listening
to announcements,” “watching a video,” “sitting,” “reading,” and “answering this survey.”
The second set of analyses uses HGLMs to examine the likelihood of being in an optimal
learning moment in science, conditioning on covariates that occur at that same moment (level-1)
in addition to student variables (level-2). Table 3.5 shows a partial model that examines the
influence of only level-1 variables on the outcome of optimal learning. When students are in
science, there are lower odds of being in an optimal learning moment compared to outside of
science (p=0.016). A student that is enjoying their task is almost six times more likely to be in an
optimal learning moment (p=0.000) and students who find importance in the task are also more
likely to be in an optimal learning moment than students who do not find importance in their
activity. These findings are consistent with previous research on optimal experiences (see
Shernoff et al., 2003).

65

Table 3.5
HGLM Predicting Optimal Learning Moments - Level 1
Level-1 (moments) Effects (n=3,485)
Variable
Coefficient
s.e.
Odds Ratio
In science
-0.321
0.133
0.725
Enjoy
1.938
0.157
6.950
Important for you
0.838
0.135
2.312
Persistence
-0.345
0.143
0.708
Notes. Reliability estimate for random level-1 coefficient is .535

p
0.016
0.000
0.000
0.013

Persistence in a task, on the other hand, while hypothesized to influence optimal learning
moments is negatively associated with these experiences. Further analyses (see Appendix D)
reveal that this is largely driven by the effect of challenge. Examining the relationship between
challenge and giving up, the distribution of responses is bifurcated, with challenge either
associated more strongly with giving up or persisting. The moments when a student is
experiencing high levels of challenge and report not wanting to give up are correlated with a
mixture of positive affects (concentration and importance), but also negative affective states
(confusion and stress). These relationships among the different emotional states may influence
how persistence is able to predict an optimal learning moment.
When covariates are added to level-2 as shown in Table 3.6, the level-1 estimates remain
mostly similar to those in Table 3.5. None of the level-2 variables are significant predictors of
optimal learning moments in science, and the reliability estimate also remains the same between
models. While male students do show a positive relationship with optimal learning moments, in
this sample it is not a significant predictor when in science classes. Students that enjoy science
more, have higher value of science for their careers and higher levels of overall engagement are
not more likely to experience optimal learning moments in science. This may indicate that these
situations of engagement are indeed more sensitive to the in-the-moment context of the task and
the environment shaping the social and emotional experiences than the contributions of stable
66

(level-2) variables, such as overall engagement. It is not the case that these level-2 covariates
offer no use in helping to contextualize the students learning experiences, it is just the case that
for this particular outcome for this sample of students, no relationships were observed.

Table 3.6
HGLM Predicting Optimal Learning Moments - Full Model
Level-1 (moments) Effects (n=3,485)
Variable
Coefficient
s.e.
Odds Ratio
In science
-0.317
0.132
0.728
Enjoy
1.939
0.163
6.950
Important for you
0.848
0.133
2.336
Persistance
-0.346
0.145
0.707
Level-2 (student) Effects (n=128)
Variable
Coefficient
s.e.
Odds Ratio
Male
0.277
0.173
1.318
Science enjoyment
-0.025
0.198
0.976
Science future
-0.017
0.191
0.983
School engagement
-0.069
0.172
0.933
Notes. Reliability estimate for random level-1 coefficient is .535

p
0.016
0.000
0.000
0.000
p
0.112
0.901
0.930
0.688

Discussion
The purpose of this chapter was to examine the student background characteristics and
their learning contexts to see how these might contribute to optimal learning experiences.
Specifically, how optimal learning experiences vary across students, what instructional
characteristics are associated with optimal learning moments, and what in-the-moment-level and
student-level characteristics can predict optimal learning moments? The research on optimal
experiences posits that these moments are subject to the contextual components that comprise the
daily events of adolescents. These factors can include whether to student is in or out of school,
the subject, and their social and emotional state at a given point in time. Capturing these
temporal situations can be a complex endeavor, however, this study collected multiple layers of

67

data using ESM and survey instruments that provide a multi-dimensional perspective of these
learning moments.
The first set of analyses examined how optimal learning moments varied across students
and contexts. Unlike previous research on gender and flow, this study found that males
experience more optimal learning moments than female students. Csikszentmihalyi and
Schneider (2000), using a slightly different measure for flow (high challenge and high skill
without a measure of interest), found that female students experienced higher levels of flow in
school than male students, 12% compared to around 6%. Their data was collected in four waves
during the 1990s, included a larger sample of students (n=1,215) than the data in this study, and
consisted of a slightly higher proportion of females than males (53% compared to 47%). While it
is not necessarily clear why this present study yielded different results for male and female
students, because there several other measures within this study as shown in chapter two and this
chapter which argue reliability and validity of the measures, it is not a concern that the
experiences of genders is different the prior research—and may in fact suggest a shift in how
males and females experience school differently compared to 15 years ago. Further analyses of
these gender differences would be an important line of future research, but is beyond the scope
of this current study.
The previous research on optimal experiences also found that these situations are often
accompanied with other phenomenology components such as enjoyment, concentration and a
sense of control (Csikszentmihalyi, 1990). This present study was consistent with these findings;
enjoyment and importance were found to be present during optimal learning moments. However,
persistence, hypothesized to be positively associated with an optimal experience, was associated
with having lower odds of being in an optimal learning moment. This may be due to the complex

68

relationship challenge, persistence, stress, and anxiety, which under some circumstances can be a
productive and positive learning moment, but if there is too much stress or challenge
(unbalance), this may compromise the learning moment, as discussed in more detail by
Schneider et al. (under review).
Limitations
One of the more challenging aspects of this research is determining the relationships
between the different affective states, it can not necessarily be assumed that there is a linear
relationship between the social and emotional responses, or that this relationship is similar in
nature for each situation. For example, this present study assumed that finding importance or
enjoyment in a task could predict an optimal learning moment (higher than average challenge,
skill, and interest), which would assume a linear relationship importance, enjoyment and optimal
learning. It could very well be the case that because a student is experiencing an optimal learning
moment and are engaged in the task at hand, that creates importance and enjoyment in the
activity. It is beyond the scope of this research to test the directionality and relationship of
different affective states and contexts, however, future research could explore these dynamic
connections during optimal experiences.
A second limitation is the lack of an achievement measure in this dataset. While the goal
of this chapter was to examine how optimal learning moments varied across students and
contexts, analyzing how these situations vary by students of different achievement levels or
using an achievement measure in the HGLMs would have strengthened these analyses. However,
based on the findings from the HGLMs that no student-level predictors could significantly
predict optimal learning moments, it could be hypothesized that students of different
achievement levels have similar odds of being in an optimal learning moment. Future data

69

collection of this current NSF project includes plans to collect an achievement measure and this
hypothesis could be tested. Additionally, chapter four of this dissertation allows for the use of a
math achievement score to examine experiences of ninth graders in their science classes from the
HSLS:09 dataset.
Finally, the ability to contextualize learning in science classes was constrained by the
level of detail provided by the students and the teachers. When students were prompted to record
their activity, often the responses were vague and brief, for example, when asked what that they
are doing while in science, a common response was “science.” This study also collected lesson
plan forms from the teachers, however, the ability to link this information with the students was
limited because the teachers’ activities were often broad classroom objectives, which could not
be linked directly to a given moment of the day/lesson. Future data collection instruments were
refined to allow the students to select from a list of customized science ideas and practices that
provide more detail about the specific modes (e.g. analyzing data, evaluating information) and
content (e.g. energy and matter) of their learning.
Implications
This study contributes to the measurement of student engagement through optimal
learning moments and sheds light on the relationship between these optimal learning moments
and their sources of variation. These findings highlight that these moments in science are indeed
dependent on several contextual factors of the classroom, which for teachers in particular signals
that there are opportunities for them to create environments conducive to optimal learning
moments. Students from a variety of backgrounds, with diverse attitudes toward science can be
engaged, often if they can find relevance and importance in the content being presented to them
(Osborne, 2002).

70

For all contexts, including those outside of science classrooms, there are implications for
how teachers can approach creating optimal learning moments. Examining the student responses
for their activities when engaged, it is not full of responses often associated with “active” or
student-led activities, but filled with a list of common instructional approaches. This may
indicate that it is not necessary for teachers to always use modes and methods of instruction that
are time and resource intensive to create environments for optimal learning—having students
complete a lab activity every day is just not feasible. However, results suggest that the
instruction needs to convey importance, relevance, value, and that the learning is enjoyable, and
this can occur in a variety of instructional settings.
These findings about importance and relevance align well with the goals of Next
Generation Science, which specifically outline that “all students have some appreciation of the
beauty and wonder of science; possess sufficient knowledge of science and engineering to
engage in public discussions on related issues; are careful consumers of scientific and
technological information related to their everyday lives; are able to continue to learn about
science outside school; and have the skills to enter careers of their choice, including (but not
limited to) careers in science, engineering, and technology” (National Academies, 2012, pp. 1-4).
These new standards emphasize the skills and practices learned in science as tools that can be
applied across all subjects (e.g. biology, chemistry, physics), but are also aimed at developing
students with sufficient science literacy regardless of their intended plans after high school,
which should improve the experiences in science for all students.

71

APPENDICES

72

Appendix A: ESM Response Sample Restrictions
Table 3.7
Sample Restrictions Comparisons

Male
Nonwhite
Average Optimal Learning Moments
Science Enjoyment
Science Value
Future Aspriations
School Engagement

Full Sample
n=141
51%
19%
3.35
82%
86%
34%
36%

At least 6
responses
n=137
53%
19%
3.45
82%
85%
33%
34%

At least 15
responses Significance
n=111
(p)
53%
0.528
20%
0.405
4.09
0.929
79%
0.409
82%
0.472
30%
0.220
33%
0.189

Notes. Phase II data used in these comparisions. A chi-squared test was used for significance for
catagorical data and a t-test was used for the average number or optimal learning moments.

73

Appendix B: Relationship between Persistence and Challenge
Figure 3.4
Percentage of Students with High Challenge and Wanting to Give Up

Notes. Figure 3.4 shows when challenge is high (equal to 4) that about 37% of responses show
not wanting to give up (1) and about 25% of responses wanting to give up (4).
Table 3.8
Correlations with High
Challenge and Persistence
Active
0.1205
Bored
-0.0295
Concentrate
0.2539
Confused
0.1716
Enjoy
-0.0258
Importance
0.1787
Self-expect
0.0410
Success
-0.0484
Skill
-0.0934
Stress
0.1600

74

REFERENCES

75

REFERENCES

Brann, A., Gray, T., Piety, P.J., & Silver-Pacuilla, H. (2010). Using technology to support
struggling students in science. Center for Implementing Technology in Education.
Retrieved from
http://www.cited.org/library/resourcedocs/UsingTechnologytoSupportScience.pdf
Connell, J.P. & Wellborn, J.G. (1991). Competence, autonomy, and relatedness: A motivational
analysis of self-system processes. In M. Gunnar & L. A. Sroufe (Eds.), Minnesota
Symposium on Child Psychology (Vol. 23). Chicago: University of Chicago
Press.Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New
York: Harper Perennial.
Csikszentmihalyi, M., & Schneider, B. (2000) Becoming adult. New York: Basic Books.
DiBianca, R. (2000). Teaching adolescents: Relationships between features of instruction and
student engagement in high school mathematics and science classrooms. Chicago:
University of Chicago Press.
Easton, L. B. (2008) Engaging the disengaged: How schools can help struggling students
succeed. Thousand Oaks, CA: Corwin Press.
Finn, J.D. & Cox, D. (1992) Participation and withdrawal among fourth-grade pupils. American
Educational Research Journal, 29, 141-162.
Fredricks, J. A., Blumenfeld, P.C., & Paris, A. H. (2004). School engagement: Potential of the
concept, state of evidence. Review of Educational Research, 74(1), 50-109.
Good, T.L. & Brophy, J.E. (2003). Looking in classrooms, Ninth Edition. Boston, MA: Pearson.
Gonzales, P., Williams, T., Jocelyn, L., Roey, S., Kastberg, D., & Brenwald, S. (2008).
Highlights from TIMSS 2007: Mathematics and science achievement of U.S. fourth-and
eighth-grade students in an international context. Washington, DC: National Center for
Education Statistics. Retrieved from http://nces.ed.gov/pubs2009/2009001.pdf
Hektner, J.M., Schmidt, J.A., & Csikszentmihalyi, M. (2007). Experience sampling method:
Measuring the quality of everyday life. Thousand Oaks, CA: Sage Publications.
Klem, A.M. & Connell, J. (2004). Relationships matter: Linking teacher support to student
engagement and achievement. Journal of School Health, 74(7), 262-273.

76

Lee, V.E. & Smith, J.B. (1995). Effects of high school restructuring on gains in achievement and
engagement for early secondary students. Sociology of Education, 68, 241-270.
Marks, H.M. (2000). Student engagement in instructional activity: Patterns in the elementary,
middle, and high school years. American Educational Research Journal, 37(1), 153-184.
National Academies. (2012). 3 Dimension 1: Scientific and engineering practices. A framework
for K-12 science education: Practices, crosscutting concepts, and core ideas.
Washington, DC: The National Academies Press, 2012.
Newmann, F.M., Wehlage, G.G., & Lamborn, S.D. (1992). The significance and sources of
student engagement. In F. M. Newmann (Ed.) Student Engagement and Achievement in
American Secondary Schools (pp. 11-30). New York: Teachers College Press.
Osborne, J.F. (2002). The challenge for science education for the twenty-first century. Pontifical
Academy of Sciences, Scripta Varia 104, Vatican City.
Osborne, J., Simon, S. & Collins, S. (2003). Attitude towards science: a review of the literature
and its implications. International Journal of Science Education, 25(9), 1049-1079.
Schlechty, P. C. (2011). Engaging students: The next level of working on the work. San
Francisco, CA: Jossey Bass.
Schneider, B., Krajcik, J., Lavonen, J., Salmela-Aro, J., Broda, M., Spicer, J., Bruner, J.,
Moeller, J., Linnansaari, J., Juuti, K. & Viljaranta, J. (under review). Optimal learning
moments in science classes: An international study in the U.S. and Finland. Manuscript
submitted for publication at Journal for Research in Science Teaching.
Shernoff, D. (2001). The experience of student engagement in high school: A phenomenological
perspective. Chicago: University of Chicago Press.
Shernoff, D., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. (2003). Student engagement in
high school classrooms from the perspective of flow theory. School Psychology
Quarterly, 18(2), 158-176.
Shumow, L. & Schmidt, J.A. (2014). Enhancing adolescents’ motivation for science: Researchbased strategies for teacher male and female students. Thousand Oaks, CA: Corwin
Publications, A Sage Company.
Singh, K., Granville, M., & Dika, S. (2002). Mathematics and science achievement: Effects of
motivation, interest, and academic engagement. Journal of Educational Research, 95(6),
323-332.
Skinner, E. A., & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of
teacher behavior and student engagement across the school year. Journal of Educational
Psychology 85(4), 571-581.
77

Stodolsky, S. (1988). The subject matters: Classroom activity in math and social studies.
Chicago: University of Chicago Press.
Turner, J. C., & Meyer, D. K. (2004). A classroom perspective on the principle of moderate
challenge in mathematics. Journal of Educational Research, 97, 311-318.
Weiss, I. (1997). The status of science and mathematics teaching in the United States. National
Institute for Science Education Brief 1(3).
Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation.
Contemporary Education Psychology 25, 68-81.

.

78

CHAPTER 4: THE ROLE OF ENGAGING EXPERIENCES IN SCIENCE AND
LONGITUDINAL OUTCOMES IN HIGH SCHOOL
As U.S. student achievement in science remains around the international average (OECD,
2013), and student engagement in science remains at or below international averages (OECD,
2007), more closely examining how students experience science in high school can help to shed
light on how learning experiences can be optimized, which can help strengthen the pipeline into
STEM. How students engage with science during high school may also vary depending on the
cumulative experiences with their courses, such as the perceived value of their science class to
their future, their efficacy in the subject, and how their teacher organizes instruction.
This chapter analyzes data from the first two waves of the High School Longitudinal
Study of 2009 (HSLS:09). The analytic framework and approach to analyzing HSLS:09 is
informed by the previous chapters’ findings that highlight the role of engaging experiences in
science class and instruction that supports persistence in challenging learning activities. Using
these ESM findings to complement HSLS:09 allows for the articulation of contexts that are
associated with optimal learning moments to be further explored by a weighted sample, which
can be generalized to a state and national population of students. These analyses will address the
conditions of ninth grade science courses in 2009, more specifically, this portion of the research
asks what types of courses did students enroll in and why? What were their experiences in their
ninth grade science class, and did these experiences influence future enrollment in advanced
courses or alter postsecondary and occupational aspirations in STEM?
Pathways in Science during High School
Strengthening the STEM Pipeline
While it is not expected that all students will be interested in a career in the STEM field,
or even enjoy their science classes, the skills and practices learned in science, such as developing
79

a model to explain a phenomenon, or analyzing data and evaluating information (National
Academies, 2012) are tools for critical thinking and problem solving that are beneficial across all
disciplines. There are also strong labor market considerations for the career outlook in STEM.
According to the Bureau of Labor Statistics, STEM occupations are projected to grow faster than
the average for all other occupations, (13% for STEM from 2012 to 2022, compared to 11% for
all others), and that the wages for these occupations were higher on average than the median for
all occupations (2013). Workers in the “advanced industries” sector in particular, which is
largely dependent on STEM–trained labor from all levels of education/training, have seen their
wages significantly increase, almost doubling since 1980 (inflation-adjusted), and in 2013 the
average earnings of workers in the advanced industries was $65,680 compared to $45,360 for all
industries (Muro et al., 2015). Providing students with engaging learning in their high school
science classrooms could contribute to increased learning and preparation for opportunities such
as these in the STEM fields.
Students’ development of their attitudes towards science can be an imprecise concept,
often including a range of behaviors such as the enjoyment of science learning experiences,
acceptance of scientific inquiry as a way of thinking, or development of an interest in science or
science-related career (Osborne, Simon, & Collins, 2003). It is, however, a combination of these
affective states with a behavior that may contribute to actual outcomes in science, as Ajzen and
Fishbein (1980) argue in their theory of planned behavior (TPB). The TPB argues that there is
both a cognitive and affective dimension of attitudes that shape behavioral intentions, which
along with circumstances motivates an outcome.
Course-taking. One example of a behavioral outcome that could be influenced by attitudes
in science and repeated behavioral experiences in science classes is the enrollment and

80

persistence in advanced science courses in high school. While there is variation from state to
state in the required number and level of science courses in high school, students that take more
than the required number of class credits may do so as an indication of their future educational
and occupational plans in science or STEM. While some aspects of the STEM field are domainspecific, there is also overlap among science, technology, engineering, and mathematics that are
important to consider. Demonstrated by Wang (2013), students' desire to major in STEM was
directly correlated by their eleventh-grade math achievement, exposure to their science and math
courses, and their math self-efficacy beliefs. These findings are consistent with previous research
showing how one key to success in high school is completion of advanced coursework in
mathematics (Gamoran, 1987; Hallinan & Kubitschek, 1999).
The course-taking sequences in mathematics are often more hierarchical than other subjects,
such as social studies or science, making the study of how a student progresses from one course
to the next in this nonlinear sequences more difficult. For science in particular, courses are often
isolated and taught independently of each other. For example, a chemistry class taken in 10th
grade might not build or extend the content from a 9th grade biology class. The lack of a clear
progression in science courses not only makes it difficult to measure and analyze course-taking
patterns in high school science, but the nature of these independent courses may also contribute
to a potential disconnect in how students build upon their previous instruction if they are unable
to make connections across their science courses. This is an issue that was addressed in the
newly developed Next Generation Science Standards (NGSS) (National Academies, 2012).
Science Instruction
It is also not enough for students to enroll in more math and science classes; students need to
be learning in these courses as well, as evidenced by Hanushek, Jamison, Jamison, &

81

Woessmann (2008) who show that increasing math and science test scores by one standard
deviation would lead to a 2.5 percentage point increase in economic growth. Furthermore, their
study showed that it is also not a matter of just raising the number of students with basic skills,
but also to continue increasing the achievement of the top performing students (Hanushek et al.,
2008). The instruction in science should also be relevant to all students’ lives, and not just to
those intending to pursue science careers. Osborn and Collins (2000) noted this as a recurring
theme across several focus groups with adolescents, that there was a lack of perceived relevance
to pupils’ lives. Osborne (2002) also noted the contrast between teaching science content
compared to instruction of skills and practices:
The emphasis of school science on consensual, well-established science, means that there is
no space for any consideration the science that dominates contemporary society-the science
and technology of informatics, CD-ROMs, mobile phones, lasers, health and disease,
modern cosmology, modern imaging systems using computerized techniques, advances in
materials technology and polymers, and last but not least, advances in medical genetics.
This is the science that interests adolescents and would be included if the curriculum was,
instead, organized around the question ‘what makes young people want to learn science?’
Yet there is as much chance of finding any contemporary science on the curriculum as
there is water in a desert. This is not to argue for a curriculum based totally on
contemporary science but simply for some aspects to be included as a vital point of
engagement. (pg. 128)
Thus, how teachers organize and convey relevance to students by creating learning that is of
interest, and by providing opportunities for students—regardless of future educational and
occupational aspirations in STEM— supports the cultivation of positive experiences for students

82

in science classes. As shown in the previous chapter, students are more likely to be in an optimal
learning moment in science when they are enjoying their task and when they perceive
importance in it.
School Context
The student experience, as well as the teaching and instruction in science, are also shaped
by the surrounding school context. While there are several benefits for schools to promote
environments conducive for supporting engaged students, such as improved student
achievement, high school completion, and matriculation to college (Csikszentmihalyi &
Schneider, 2000; Klem & Connell, 2004; National Research Council [NRC], 2004; Shernoff,
Csikszentmihalyi, Schneider, & Shernoff, 2003), there is significant harm to students, teachers,
and schools when students are not engaged. Students that become disengaged not only are
subject to decreased learning and engagement, but this disengagement in school could lead to a
downward spiral of the individual into dysfunctional and ultimately risky behavior, such as
dropping out (Easton, 2008; Marks, 2000). Schools dealing with many disengaged students are at
a significant risk of developing a school culture of disengagement—where the values of the
disengaged become the dominant values of the student body, teachers, and the school overall.
Due to this relationship between student engagement and school culture, promoting engagement
as part of the school context in any school reform should be a necessary goal (Elmore et al.,
1990).
Theoretical Framework
The framework for how students move through their high school science experiences,
including decisions to pursue advanced coursework in science during high school and future
ambitions in science, builds off of the conceptualization of engagement discussed in chapter one.

83

While the ESM analyses focused on moments of optimal learning, this section “zooms out” to
consider additional contextual influences, such as the role of school factors, and student
achievement measures. Recognizing the multidimensionality of learning and engagement and the
need to understand longitudinal outcomes associated with being engaged (see Christenson,
Reschley, & Wylie, 2012), these analyses will include the evaluation of both outcomes and
additional mediators and moderators of science experiences not available to be analyzed with the
ESM dataset, but is motivated by the findings of skill and interest in science class as well as the
role of science importance (as measure by usefulness in this chapter) to positive outcomes in
science.
Figure 4.1. Longitudinal Engagement Experiences in High School

84

As chapters two and three demonstrated, optimal learning moments, albeit rare in science
classes, provided several key findings about factors associated with these situations of high
challenge, skill, and interest. This portion is motivated by the circumstances and contexts that
supported students’ optimal learning moments in chapter three, as well as the findings related to
the importance of the activity during science learning. Given the findings from the in-themoment analyses of chapters two and three, how do these compare to a nationally representative
dataset examining similar conditions of learning experiences in science classes? Specifically,
using the HSLS:09 dataset, the “Outcomes” of Figure 4.1 box is examined.3 Specific schoollevel variables are not used in these analyses, however, by using a multi-level analytic approach,
the variation within and between schools can still be accounted for.
Research Questions
(1) How do experiences in ninth grade science vary, and how are they influenced by teacher
and school characteristics?
a. What science courses are students enrolled in, and does this vary by student race,
gender, socioeconomic status, and career aspirations?
b. What types of instruction and approaches to teaching do students’ science
teachers emphasize?
(2) How do instructional variables predict usefulness, skills, and interest in ninth grade
science courses?
(3) How do experiences in ninth grade predict future aspirations in STEM?

3

There is no phenomenological factors box for these analyses because it is not able to be
measured with this data. The variable “science identity” is used as a proxy of prior experiences
with science

85

It is hypothesized that having more positive experiences in a science course may increase future
interest in science and contribute to continued pursuit of advanced science coursework and a
career in STEM.
Data and Methods
The HSLS:09 data is a nationally representative sample of 21,444 students in the ninth
grade from 944 schools. The study uses a stratified, two-stage random sampling design, sampling
the schools at stage one, and the students within schools at stage two. Currently, the HSLS:09
study contains two waves of data: the base year, which was in the fall of 2009 when the student
sample was enrolled in the ninth grade, and the first follow-up, which occurred in the spring of
2012 when the sample was enrolled in the eleventh grade. HSLS:09 was specifically designed to
capture students' experiences in high school, particularly in math and science. The dataset
includes algebraic reasoning scores from an adaptive mathematics assessment in addition to
contextual data from school administrators, guidance counselors, math and science teachers, and
parents.
Measures
Data from both available waves of public-use HSLS:09 dataset are used to measure
experiences of adolescents in their science classes. To capture experiences in science during the
2009 base year and to examine future aspirations in science in the first follow-up, variables from
their ninth grade year are used. While the measures will include items from the student
questionnaire, teacher questionnaire, and school (administrative) questionnaire, these variables
are only representative of the student population, not of science teachers or high schools. Table
4.1 is a summary table of the HSLS:09 sample. There are two columns, one for the un-weighted

86

summary statistics and a second with the weights applied to create national population of close to
4.2 million ninth graders.

Table 4.1
Descriptive Characteristics of the HSLS:09
National Sample
Unweighted Weighted
N of students
21,444
4,197,724
Female
49%
50%
Race
Asian/Pacific Islander
8%
4%
Black/African American
11%
14%
Hispanic
16%
22%
Other race/More than one
9%
8%
White
49%
48%
English is Native Language
Below Poverty Threshold
Individualized Education Plan

72%
11%
9%

76%
13%
10%

Postsecondary Ambitions
Don't know
High School/GED
Complete Associate's Degree
Complete Bachelor's Degree
Complete Higher than Bachelor's
Previously in 9th grade

18%
10%
5%
14%
35%
4%

8%
6%
6%
20%
28%
5%

Source: U.S. Department of Education, National Center for Education
Statistics, High School Longitudinal Study of 2009 (HSLS:09), Base-Year

Student variables. The first outcomes of interest include the student’s reported
experiences in their ninth grade science class, specifically their interest, skills, and their
perception of the class’ usefulness for everyday life. The second outcome examined is their
career aspirations in STEM from their first follow-up survey. Demographic variables used in the
analyses include a dummy variable for gender (recoded from X1SEX), race (a dummy variable
was created for white, black, Hispanic, and other using X1RACE). For socioeconomic status a
87

dummy variable was created for if at least one of their parent’s holds a bachelor’s degree
(X1MOMEDU and X1DADEDU). An achievement measure (X1TXMQUINT) is the student’s
quintile from a math reasoning standardized score from a computer-adaptive assessment
administered during the base year. A dummy variable was created from X1STUEDEXPCT to
indicate if the student expects to attain at least a bachelor’s degree. As a mediating variable, the
scaled item for the student’s science identity (X1SCIID) is used, that is how the student views
themselves as a “science person.” It is hypothesized that this variable captures prior experiences
with science.
Teacher variables. This analysis is focused on contextualizing students' experiences in
their ninth grade science courses using data self-reported by their science teachers regarding the
instruction and classroom environment. HSLS:09 contains eleven instructional variables used in
this analysis to determine to what extent teachers emphasize the following aspects of their
teaching in science: (a) increasing students’ interest in science; (b) teaching basic science
concepts; (c) important science terms/facts; (d) science process/inquiry skills; (e) preparation for
further science study; (f) evaluating arguments based on evidence; (g) communicating science
ideas; (h) business/industry applications; (i) relationship between science/technology/society; (j)
history/nature of science; and (k) standardized test preparation (see Appendix E for questionnaire
items).
A principal component analysis was conducted and item reliability check performed to
create three instructional constructs: (1) ENGAGE created using N1INTEREST, N1SKILLS,
N1PREPARE, N1EVIDENCE, and N1IDEAS with an alpha of 0.7769; (2) RELEVANCE
created using N1BUSINESS, N1SOCIETY, and N1HISTORY with an alpha of 0.7196; and (3)
FACTS created using N1CONCEPTS, N1TERMS, and N1TEST with an alpha of 0.4764.

88

Weighting and missing data. The estimates used were weighted according to
recommended specifications in the HSLS:09 documentation. Balanced repeated replication
procedures for variance estimation/standard error calculation methods were used to adjust for the
unequal probability of selection into the sample. The base-year weight (W1STUDENT) is used
for the descriptive analyses of the base-year data, and the longitudinal student weight
(W2W1STU) is used for analyzing outcomes from 2012 and accounts for the base-year school
nonresponse and student nonresponse in both the base-year and the first follow-up.
Analytic Approach
The HSLS:09 dataset is analyzed first using a series of descriptive analyses to examine
trends and variation in science course-taking experiences as well as instructional characteristics
as reported by the student’s science teacher. Then, primary outcomes of interest: (1) experiences
in ninth grade science class and (2) future aspirations in STEM are analyzed using an HGLM,
with students (level-1) nested within classrooms/schools (level-2). Due to the sampling structure
of HSLS:09, the classroom and school effects are at the same level. The full estimated HGLM
for each outcome is:
Level 1 (student-level i)
𝑖𝑗 = 𝜋0𝑗 + 𝜋1𝑗 (𝑚𝑎𝑙𝑒) + 𝜋2𝑗 (𝑚𝑖𝑛𝑜𝑟𝑖𝑡𝑦) + +𝜋3𝑗 (𝑚𝑎𝑡ℎ𝑞𝑖𝑛𝑡) + 𝜋4𝑗 (𝑠𝑡𝑢𝐵𝐴) + 𝜋5𝑗 (𝑝𝑎𝑟𝐵𝐴)
+ 𝜋4𝑗 (𝑠𝑐𝑖𝑝𝑒𝑟𝑠𝑜𝑛)
Level 2 (classroom/school j)
𝜋0𝑗 = 𝛽00 + 𝛽01 (𝐸𝑁𝐺𝐴𝐺𝐸) + 𝛽02 (𝑅𝐸𝐿𝐸𝑉𝐴𝑁𝐶𝐸) + 𝛽03 (𝐹𝐴𝐶𝑇𝑆) + 𝛽04 (𝑃𝐶𝑇𝐹𝑅𝑃𝐿) + 𝜀0𝑗
where 𝑖𝑗 = (𝜑𝑖𝑗 /1 − 𝜑𝑖𝑗 ) is the logit link function and

 𝜑𝑖𝑗 is the probability of the primary outcome (for student i in classroom/school j),
 𝑚𝑎𝑙𝑒 is a dummy variable equal to 1 for boys and 0 for girl students
 𝑚𝑖𝑛𝑜𝑟𝑖𝑡𝑦 is a dummy variable if the race/ethnicity is Hispanic, Black, or Asian
89

 𝑚𝑎𝑡ℎ𝑞𝑢𝑖𝑛𝑡 is a dummy variable equal to 1 if the student is in the top two quintiles
 𝑠𝑡𝑢𝐵𝐴 is a dummy variable equal to 1 if a student expects to attain at least a bachelor’s
 𝑝𝑎𝑟𝐵𝐴 is a dummy variable equal to 1 if a parent holds at least a bachelor’s
 ENGAGE is a composite variable of the science teacher’s emphasis on engaging
instruction
 RELEVANCE is a composite variable of the science teacher’s emphasis on relevant
instruction
 FACTS is a composite variable of the science teacher’s emphasis on fact-based
instruction
 PCTFRPL is the percentage of students at a school receiving a free or reduced price
lunch
 𝜀0𝑗 is a random effect for classroom/school j

Results
To examine how experiences in ninth grade science may influence future outcomes, it is
helpful to explore the characteristics of these ninth grade science classes. The science subject
that students are enrolled in varies as there is no standardized course-sequencing from state to
state. As shown in Figure 4.1, the majority of 9th graders (34%) were enrolled in a biology class,
followed by 19% in physical science, and 12% in earth science. There were about 18% of
students who were not enrolled in a science class at all.

90

Figure 4.2 Science Course Enrollment of 9th Graders from 2009-2010

Next, Table 4.2 examines how this enrollment in the top four categories (no science,
biology, earth science, and physical science) varied by gender, race, and parent’s education level.
4

There does not appear to be enrollment difference by gender, with male and female students

enrolled at similar rates across all courses. There are however, differences by race with Black
and Hispanic students taking fewer science courses their 9th grade year, 25% and 22%
respectively, and that is compared to an average non-enrollment of 18% across all
races/ethnicities. There is a higher percentage of Asian students (44%) enrolled in biology

4

There were small cell sizes for the categories of chemistry, other science, and two or more science classes. If all
categories were represented, the columns across each science category would sum to 100%.

91

Table 4.2
Science Course Enrollment by Student Characteristics
No science

Biology

Earth
science

19%
17%

34%
35%

13%
12%

19%
20%

Race/Ethnicity
White
Black
Hispanic
Asian
All other races

15%
25%
22%
13%
16%

32%
31%
28%
44%
33%

14%
7%
13%
7%
13%

23%
21%
10%
16%
19%

Parent/guardian education
High school GED, or less
Associate's degree
Bachelor's degree
Graduate/professional

22%
15%
13%
11%

31%
32%
39%
44%

13%
13%
14%
11%

19%
22%
19%
18%

Career in STEM^

17%

44%

17%

12%

Characteristic
Gender
Male
Female

Physical
science

Source: U.S. Department of Education, National Center for Education Statistics, High School
Longitudinal Study of 2009 (HSLS:09), Base-Year. Weighted with the W1STUDENT weight
using BRR. ^This dummy variable was coded using X1STU30OCC2 variable and the NSF
classification list of STEM occupations

classes, compared to an average enrollment of 34% across all ninth graders. Both Asian and
Black students are enrolled at lower rates in earth science courses, while Hispanic and Asian
students enroll at lower rates in physical science courses. Students whose parents have at least a
bachelor’s degree enroll in higher rates in biology than students of parents with less education.
However, across all parental education levels, students enroll most frequently in biology,
including those who aspire to have a career in the STEM field.

92

While there is quite a bit of variation in the science enrollment of ninth graders, which is
to be expected because of the diverse state policies regarding high school credit and graduation
requirements, the next part of these analyses examine the instructional characteristics of the
science teachers. Table 4.3 shows how the science teachers of the sampled students report
emphasizing certain aspects of their instruction. It should be noted this is generalizable to the
student population, not the population of teachers.5 The largest reported instructional emphasis is
on basic concepts (40%), followed by science facts (29%) and inquiry skills (29%). The lowest
reported instructional categories were emphasizing application (11%), history/nature (13%), and
evaluating information. These instructional summary statistics, while obtained prior to the
development of the new NGSS standards, reflect the environments in which the standards were
beginning development and show how the current instructional practices in science class could
become more aligned with what is outlined in the NGSS (e.g. focus on skills/practices,
evaluating information, and communicating about ideas).

5

There is a high percentage of missing data from the science teachers. This is partially due to around 18% of the
sample (about 3,900 students) not being in a science course, so they would not have an observation from their
science teacher, which leaves about 5,400 observations missing from student-level file.

93

Table 4.3
Teacher Instructional Variables

Emphasis on interest in science*
Emphasis on basic concepts
Emphasis on science facts
Emphasis on inquiry skills
Emphasis on further science
Emphasis on evaluating
Emphasis on communication
Emphasis on application
Emphasis on science/society
Emphasis on history/nature
Emphasis on test preparation

Students
N=2,108,889
25%
40%
29%
29%
23%
15%
19%
11%
18%
13%
19%

Notes. The teacher data is not representative of all
teachers. These data need more cleaning/work as there is a
high percentage of missing items.
*Top categories (heavy) reported for the emphasis items.

The next set of analyses uses 2-level HGLMs to explore the relationship between the
instructional variables, student background characteristics, and their experiences in their ninth
grade science course. Table 4.4 shows how useful the student reported their 2009 science course
was, and students who are a minority students who have a higher science identify are more likely
to indicate their science class was useful. There were no significant differences between male
and female students as well as students across different achievement levels for the usefulness of
the class. The engagement factor at level-2 is a marginally significant predictor of students
finding their science class useful, meaning that students with teachers who emphasize increasing
interest, process skills, preparation for future study in science, evaluating evidence, and
communicating ideas are more likely to view their class as useful for everyday life.

94

Table 4.4
HGLM Predicting Usefulness of 2009 Science Course - Full model
Level-1 (students)
Variable
Coefficient
s.e.
Odds Ratio
Male
0.065
0.100
1.07
Minority
0.279
0.089
1.32
Student BA
0.055
0.096
1.06
Parent BA
-0.095
0.089
0.91
High Math Quintile
-0.137
0.096
0.87
Science Identity
0.984
0.110
2.68
Level-2 (Classroom/School)
Variable
Coefficient
s.e.
Odds Ratio
Engage
0.246
0.129
1.28
Relevance
0.137
0.105
1.15
Facts
-0.117
0.125
0.89
Percent FRPL
0.002
0.003
1.00
Notes. Reliability estimate for random level-1 coefficient is .370

p
0.518
0.002
0.562
0.284
0.155
0.000
p
0.057
0.194
0.194
0.439

Examining the relationship between the level-1 and level-2 predictors and how skilled a
student feels in their science class, Table 4.5 shows that students who have higher educational
expectations, that have a parent with a bachelor’s degree, have higher math achievement, and
view themselves as a science person have a higher likelihood of feeling skilled in their science
class. Students with a higher science identity in particular are three times more likely to report
higher levels of skills in their science class. It is important to note that there are no differences
between gender and races/ethnicities. None of the level-2 variables are significant, which could
indicate that the efficacy a student experiences in their science class is a product of prior
experiences and is less sensitive to the instruction or context of their course.

95

Table 4.5
HGLM Predicting Skills of 2009 Science Course - Full model
Level-1 (students)
Variable
Coefficient
s.e.
Odds Ratio
Male
0.106
0.120
1.11
Minority
0.059
0.126
1.06
Student BA
0.296
0.120
1.35
Parent BA
0.238
0.105
1.27
High Math Quintile
0.401
0.113
1.49
Science Identity
1.500
0.110
4.49
Level-2 (Classroom/School)
Variable
Coefficient
s.e.
Odds Ratio
Engage
0.163
0.174
1.18
Relevance
0.139
0.123
1.15
Facts
-0.159
0.122
0.85
Percent FRPL
0.003
0.003
1.00
Notes. Reliability estimate for random level-1 coefficient is .301

p
0.374
0.639
0.013
0.023
0.001
0.000
p
0.349
0.259
0.192
0.244

The level of interest in a student’s science course, similar to skills, is an experience that is
predicted by achievement and identity covariates—there is no significant relationship between
instructional/context variables and interest in their science class as shown in Table 4.6. There is
also no relationship between interest in the science class and gender, race, parental education,
and the student’s educational expectations. Again, this may suggest that interest in a domain such
as science is shaped over time by multiple experiences, and this type of interest is not as
malleable to situational determinants. This type of interest, however, is different than situational
interest, which as shown in chapters two and three of this dissertation, does vary by context and
is an outcome that is correlated by contextual factors, including instruction.

96

Table 4.6
HGLM Predicting Interest of 2009 Science Course - Full model
Level-1 (students)
Variable
Coefficient
s.e.
Odds Ratio
Male
0.085
0.091
1.08
Minority
-0.114
0.097
0.89
Student BA
0.055
0.114
1.06
Parent BA
-0.063
0.099
0.94
High Math Quintile
0.299
0.090
1.34
Science Identity
0.756
0.083
2.12
Level-2 (Classroom/School)
Variable
Coefficient
s.e.
Odds Ratio
Engage
0.058
0.156
1.06
Relevance
0.022
0.116
1.02
Facts
-0.077
0.107
0.93
Percent FRPL
-0.001
0.003
0.99
Notes. Reliability estimate for random level-1 coefficient is .373

p
0.352
0.243
0.632
0.523
0.001
0.000
p
0.710
0.847
0.472
0.663

In the final model, experiences and characteristics from the base year are used to predict
the student’s aspiration of a career in STEM in 2012, from the first follow-up survey. Table 4.7
shows that male students are more likely than female students to report a STEM career, as well
as students with a higher math ability and that view themselves as a science person. At level-2,
students that had teachers who again used engaging instructional modes were marginally more
likely to indicate a science career in 2012 (p=.081). This model has the only outcome where the
percentage of students receiving free or reduced price lunch at the school has a significant
influence, with students in schools that have higher rates of free or reduced price lunch with a
lower likelihood of interest in a STEM career. However, with an odds ratio of .99, the magnitude
of this difference is small.

97

Table 4.7
HGLM Predicting STEM Career in 2012 - Full model
Level-1 (students)
Variable
Coefficient
s.e.
Odds Ratio
Male
0.454
0.119
1.57
Minority
-0.130
0.115
0.87
Student BA
0.194
0.124
1.21
High Math Quintile
0.497
0.125
1.64
Science Identity
0.518
0.117
1.68
Level-2 (Classroom/School)
Variable
Coefficient
s.e.
Odds Ratio
Engage
0.22
0.126
1.25
Relevance
-0.030
0.092
0.97
Facts
-0.101
0.135
0.90
Percent FRPL
-0.007
0.002
0.99
Notes. Reliability estimate for random level-1 coefficient is .166

p
0.000
0.260
0.118
0.000
0.000
p
0.081
0.743
0.455
0.003

Discussion
The experiences students have in science, as well as outside of science in other academic
settings or their everyday life, are diverse and are shaped by past experiences, current contexts
and situations, and future goals and aspirations. Examining the relationship between these
dynamic factors in complex and changing environments is a challenging research endeavor.
This chapter more narrowly focused on exploring the experiences of ninth grade students in
their science class and understanding the relationship between their background characteristics,
instruction, and positive outcomes. Using the HSLS:09 dataset, which was specifically designed
to capture these experiences in math and science, proved exceptionally beneficial in supporting
the investigation of how experiences in ninth grade science vary students; trends in science
course enrollment and variation and how instructional variables predict usefulness, skills, and
interest in ninth grade science courses.

98

While there is quite a bit of variation in which courses students take during ninth grade,
biology has the highest percentage of students enrolled. Course enrollment does not vary by
gender, which is a positive finding for reducing gender barriers in STEM subjects, however,
enrollment does have higher variation between races, which could indicate that students from
traditionally underrepresented backgrounds in STEM experience differentiated access to courses
while in high school, contributing to the leaky STEM pipeline. However, there were no
significant negative relationships between race and the experiences of students in their science
course as shown in the HGLMs, which suggests that if there were perhaps more parity in
enrollment between races, that once students were in the course, they would benefit from the
experience.
The multi-level models show different impacts for level-1 and level-2 variables
depending on the outcome of interest. Overall, it was difficult to uncover relationships between
instructional variables and student-level outcomes. However, the engaging instructional factor
was marginally significant at predicting the usefulness of the science class as well as future
aspirations in science. Student background characteristics were more predictive of the level of
skills and interest the students reported in their class, though not along lines of gender and race,
which as suggested above demonstrates improved environments in science for these students.
Despite the prior research, which suggested that math achievement has a significant impact on
the development and support of educational and career aspirations in science, when considering
how students reflect on their current experiences, particularly the usefulness of science in their
everyday life, this was not significant. Teachers can make their science instruction meaningful
and useful for all students, regardless of their achievement level, gender, educational
expectations, or parental educational level.

99

Limitations
Compared to analyzing engagement through optimal learning moments, this dataset was
limited in its ability to provide more detail about the types of experiences that occur on a daily
basis in their science classes. It is also possible that their reported experiences in science are
more of an indication of their teacher and not their teacher’s instruction as these two often can
become one mechanism. It is also a limitation of secondary dataset analysis in general that the
scope and type of variables are pre-determined and the researcher must utilize the questions and
variables set by the overseeing agency. For example, this study would benefit from exploring
how challenging the student felt their science class was given the findings of chapter three.
However, the question asked about challenge on the questionnaire reads: “9th grader is taking
their fall 2009 science class because he/she likes to be challenged,” which unfortunately does
not provide information on if they found the class difficult.
The generalizability of these analyses is limited by the large quantity of missing
responses, both from students who were not enrolled in a science class and from the science
teachers who did not provide a questionnaire. Although the appropriate weights were applied as
suggested by the HSLS:09 technical documentation, the findings should be interpreted with
caution knowing that there could be sample bias, particularly for the instructional variables.
Lastly, the outcomes analyzed in this chapter while beneficial to understanding the
student’s experience in high school, are limited in their ability to examine longitudinal impacts
of science experiences because of the timing and release of the transcript data (which is
expected in the summer of 2015). Future analyses will be able to test course-taking patterns,
eleventh and twelfth grade science enrollment in Advanced Placement and/or International
Baccalaureate courses, and some postsecondary information. Because of the timing of this

100

present study and the release of these data, the outcomes were limited to those available in the
base year and first follow-up.
Implications
Given the demand to increase interest in science, technology, engineering, and
mathematics throughout the educational system, the high school can serve as a critical part in
supporting students’ interest in STEM. As students develop future educational and occupational
plans, experiences in their courses, particularly in STEM subjects, can shape, support and spur
their interest in the types of majors and careers to pursue. This study identified mechanisms that
influence experiences in science, finding that enrollment in science is diverse and that the
experiences of students including the usefulness, interest, and skills is influenced by a
combination of student background, achievement, and instructional factors.
Overall, this suggests that some parts of the experiences of students in science are
malleable and can be directly influenced by science teachers and their instruction (such as
instruction that emphasizes processes and skills, preparing students for future study in science
and communicating ideas). Both the ESM and HSLS:09 data suggest that teachers can shape
and influence the context of learning and provide engaging opportunities for students—
however, this type of instruction should be implemented with a balanced approach—there is
certainly content that needs to be presented in a fact-based lecture for example, but what this
study highlights is that certain types instruction is correlated with positive learning outcomes.
The ESM data in particular shows that when students are enjoying their task and find
importance in what they are doing (regardless of what the activity is), this was related to having
an optimal learning moment in science.

101

Improving STEM education is one policy lever utilized to strengthen the pipeline of
talent from high schools into the STEM workforce. Even though not all students will be
interested or enter a STEM workforce, these skills and concepts support the development of a
well-educated science citizenry. While contributing to the research on the relationship between
engaging experiences and instruction for researchers and policymakers, this research provides
teachers with evidence on the experiences of students in their class, which has implications for
their instruction and could potentially be a tool for how they emphasize certain instructional
approaches in their classrooms. As requirements for the number and level of science classes
increase and with the development and adoption of NGSS across states, understanding the
experiences of students in high school and the role of teachers in fostering engagement is an
important area to continue to research.

102

APPENDIX

103

Appendix
HSLS:09 Selected Teacher Questionnaire Items

104

105

REFERENCES

106

REFERENCES

Ajzen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior.
Englewood Cliffs, NJ: Prentice-Hall.
Christenson, S., Reschly, A., & Wylie, C. (2012). Handbook of research on student engagement.
New York: Springer.
Csikszentmihalyi, M., & Schneider, B. (2000) Becoming adult. New York: Basic Books.
Easton, L. B. (2008). Engaging the disengaged: How schools can help struggling students
succeed. Thousand Oaks, CA: Corwin Press.
Elmore, R. F. and Others (1990). Restructuring schools: The next generation of educational
reform. San Francisco: Jossey-Bass.
Gamoran, A. (1987). The stratification of high school learning opportunities. Sociology of
Education, 60(3), 61–77.
Hallinan, M. & Kubitschek, W. (1999). Curriculum differentiation and high school achievement.
Social Psychology of Education, 3(1–2), 41–62.
Hanushek, E.A., Jamison, D.T., Jamison, E.A., & Woessmann, L. (2008). Education and
economic growth: It’s not just going to school but learning that matters. Education Next,
8(2), 62-70.
Klem, A.M. & Connell, J. (2004). Relationships matter: Linking teacher support to student
engagement and achievement. Journal of School Health, 74(7), 262-273.
Marks, H.M. (2000). Student engagement in instructional activity: Patterns in the elementary,
middle, and high school years. American Educational Research Journal, 37(1), 153-184.
Muro, M., Rothwell, J., Andes, S., Fikri, K., & Kulkarni, S. (2015). American’s advanced
industries: What they are, where they are, and why they matter. Washington DC:
Brookings Institution Press.
National Academies. (2012). 3 Dimension 1: Scientific and engineering practices. A framework
for K-12 science education: Practices, crosscutting concepts, and core ideas.
Washington, DC: The National Academies Press.
National Center for Education Statistics. (2013). High School Longitudinal Study of 2009 (First
Follow-Up) [Public-use Data File]. Washington DC: Department of Education.
107

National Research Council. (2004). Engaging schools: Fostering high school student’s
motivation to learn. Committee on Increasing High School Students’ Engagement and
Motivation to Learn. Board on Children, Youth, and Families, Division of Behavioral and
Social Sciences and Education. Washington, DC: The National Academies Press.
OECD. (2007). PISA 2006: Science competencies for tomorrow’s world – Volume 1- Analysis.
Paris, France: OECD Publishing.
OECD. (2013). PISA 2012 results: What students know and can do: Student performance in
mathematics, reading and science (Volume I). Paris, France: OECD Publishing.
Osborne, J.F., & Collins, S. (2000). Pupils’ and parents’ views of the school science curriculum.
London: King’s College.
Osborne, J.F. (2002). The challenge for science education for the twenty-first century. Pontifical
Academy of Sciences, Scripta Varia 104, Vatican City.
Osborne, J., Simon, S. & Collins, S. (2003). Attitude towards science: a review of the literature
and its implications. International Journal of Science Education, 25(9), 1049–1079.
Osborne, J., & Dillon, J. (2008). Science education in Europe: Critical reflections. London: The
Nuffield Foundation.
Shernoff, D., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. (2003). Student engagement in
high school classrooms from the perspective of flow theory. School Psychology
Quarterly, 18(2), 158-176.
Wang, X. (2013). Why students choose STEM majors: Motivation, high school learning, and
postsecondary context of support. American Educational research Journal, 50(5), 10811121.

108

CHAPTER 5: CONCLUSION
The primary goal of this study was to understand what contributes to student experiences
in high school science classes using a mixed-dataset approach. Examining data from two
different sources provided a unique perspective on students’ learning experiences and the
contexts in which it develops. Engagement was operationalized across the chapters using
features of flow theory to evaluate how the challenge, skill, interest, importance and value in
science classes vary across students and settings. The multiple analytic approaches motivated by
a singular conceptual framework provides information about the students and the instruction they
experienced, which contributes to how teachers can plan and implement their teaching in
science.
After the overall framework and organization for this dissertation was presented in
chapter one, the first part of this study in chapter two explored and validated the construct of
optimal learning moments using in the ESM data. Several different measures of engagement
were reviewed and tested to uncover variation in optimal learning moments, as well as
relationships between the situational and cross-situational measures of engagement in science.
Chapter three analyzed the same ESM dataset in more detail by examining how optimal learning
moments varied across students and their environments, including the instruction of their science
teachers. In chapter four, the HSLS dataset is used to analyze experiences in science based on the
findings from chapters two and three. Specifically, the questions of how engagement influences
experiences in ninth grade science are investigated, including: students' perception of the class’
usefulness for everyday life, their skills in the course, their level of interest in the course, and the
students' interest in pursuing a career in STEM in the first follow-up.

109

Summary of Findings
In chapter two, relationships between different measures of engagement were tested, and
the findings suggest that multiple measures of engagement can help capture the multidimensionality of optimal learning experiences. Examining the variation of, and the relationship
between both situational and cross-situational measures of engagement proved useful in
establishing the measures employed in the study, while uncovering how the different measures
influenced each other. Comparison of using raw scores and Z-scores for the ESM affective
measures supported the use of standardizing measures of optimal learning by capturing deviation
for each student’s mean level, although both appear to be valid constructs of engagement. This
study also contributes to measuring optimal experiences in relationship to other states associated
with being in flow, including persistence, time flying by, and concentration, which provided
further evidence on the engagement measure used in chapter three.
Throughout the chapters, there were some themes that were repeated across the studies,
while some of the findings were varied from one chapter to the next. Gender, for example, was a
consistent source of variation across the studies in this dissertation. The first set of analyses
examining how optimal learning moments varied across students found that males experience
more optimal learning moments than female students, however, this gap did narrow in science
classes. Gender was not a significant predictor of being in an optimal learning moment in science
when conditioning on enjoyment, importance, and persistence in the moment (level-1), and
science attitudes of the student (level-2). There were also no gender differences in science
course-taking in ninth grade, but there were gender differences in students’ interest in pursuing a
career in STEM, after conditioning on their experiences in ninth grade science. Males in both the
base-year and first follow-up report higher levels of career aspirations in STEM, which

110

highlights that despite similar reported experiences in high school, the gap in aspirations persists.
Future research could explore additional influences on this outcome and how the gap between
male and female students changes throughout high school and in to college.
Another finding present in both chapters three and four is that in order to enable more
optimal learning experiences, students in science need to find relevance and value in what they
are learning and understand how it is applicable to their lives, regardless of whether or not they
intend to pursue a career in a STEM field. The importance of the activity during science was a
significant predictor of being in an optimal learning moment as well as enjoyment—indicating
that a learning environment should not only create value in the content, but also make learning
exciting and enjoyable. Extending this finding into chapter four and the types of instruction that
might create higher value or usefulness of the science class, evidence suggests that instruction
that emphasizes increasing interest, using process skills, preparing students for future study in
science, evaluating evidence, and communicating their ideas contributes to students finding
higher usefulness for their science class.
Limitations
While each chapter summarized its own study limitations, there are some additional
limitations to the overall design of this dissertation. The ESM sample was limited to four schools
and teachers that had indicated interest in participating in this study of engagement, and one of
the schools was specifically using their professional development to support teachers in
increasing student engagement. Thus, the generalizability of findings is limited and results
should be interpreted with consideration for the small sample of students and teachers. Despite
the small sample of schools and teachers, the schools did vary in location and size, ranging from
a smaller rural school to a larger urban school, which does provide some variation at the school

111

level. Using the HSLS:09 did provide a larger national sample, however, because of the missing
teacher data, these findings should also be interpreted conservatively. Even with these sampling
concerns across the studies, finding consistent results about engagement using the two different
datasets and analytic approaches does provide evidence as to sources of variation and what
contributes to learning experiences for students.
There were certainly measures across both studies that provided similar details about
student learning experiences in science, however, the items used were not the same in both
studies for the students or their teachers. For example, using the HSLS:09 teacher survey
instrument to collect data about instruction from the teachers in the ESM study would have
allowed for a better alignment of items across the samples and improved this student’s ability to
measure the in-the-moment experiences of students given specific instructional approaches by
the teacher. While instruments in the ESM study were still constructed using previously
validated items, these questions came from international questionnaires because the ESM study
was part of a larger international research project, thus the HSLS:09 items were not used.
Research done after the completion of this dissertation was able to collect additional data from
the teachers to address the weaknesses of more fully capturing the learning environments, and
future analyses will be able to utilize these improvements to the overall design of the research
project.
Policy Implications
There is no question that improving learning environments and providing students more
opportunities to be engaged motivates the research and policies undertaken to enhance the
educational experiences for students. However, there is no one-size-fits-all approach or a magic
bullet that can accomplish these broad goals, and determining what is good teaching also remains

112

a top priority with countless studies using a diverse set of approaches and perspectives to
undertaking this task. For example, the Measures of Effective Teaching (MET) project was a
two-year study that included over 3,000 teachers in seven large districts across the country, and
employed teacher surveys, student surveys of the teacher, achievement measures, and 360-degree
video cameras that captured every angle of the classroom instruction. The goal of the MET
project was to determine which classroom measures were highly associated with high levels of
student achievement, including student engagement and the role of teachers in delivering
engaging instruction (MET, 2010). Preliminary findings from the MET project showed that there
is a positive correlation between highly-rated teachers and improved student achievement (MET,
2012). The MET project is an example of how the exhaustive field of engagement literature can
be considered when evaluating teaching and learning.
This dissertation also contributes to how technology can be used in education research.
Measuring engagement using innovative smartphone technology not only lessened the burden of
cleaning and preparing the ESM data compared to previous paper-and-pencil methods, but this
research also informs how new technologies can be used for measuring the individual
experience, or other types of data collection using smartphones more broadly for education
research. Because of the increased use and assimilation of smartphones in daily lives,
technology, such as the app used in this study, Paco, is well-positioned to streamline the ability
of capturing several different types of data and using a diverse set of time frames (e.g. collecting
daily, weekly, or even at the end of every learning unit in school). Anecdotally, the use of
smartphones for data collection or other innovative uses in research may have particular value in
reaching students, schools, families and other community members in urban or rural low-income
communities due to the role of smartphones in being a primary source of accessing information.

113

According to the most recent Pew Research Center Study (2013), 78% of teens now have a cell
phone, and almost half of these devices are smartphones. This is a rapidly increasing amount, up
14 percentage points from just 2 years ago. The report highlights that accessing the internet on a
phone is common among today’s adolescents, and there is a growing opportunity to capitalize on
this as a lower-cost and effective way of obtaining data compared to paper-and-pencil and even
computer-based surveys.
Finally, there are also implications beyond the science classroom. While all subjects
could benefit from teachers connecting learning to something that the student cares about, which
can increase interest and understanding—this is of particular value across science, technology,
engineering and mathematics, which are areas that the students might not otherwise experience
in their daily lives. There are opportunities to strengthen the students' understanding of concepts
and practices when connections can be integrated across the STEM disciplines (Honey, Pearson,
& Schweingruber, 2014). This includes the adoption and implementation of the standards, such
as the Common Core State Standards in mathematics, the NGSS, and the Standards for
Technological Practice, which have been designed specifically to be integrated with one another.
Using instructional approaches that utilize project-based learning for example, which emphasize
student-led learning and provide a balance of challenge and skill, can also help shape improved
learning environments in STEM. This is not to say that all instruction should be integrated all the
time or should always use project-based learning (this is unrealistic given teacher preparation,
time, and resources constraints), however, using these principles when it can create stronger
connections for students, deepen their understanding of the content, and develop relevance for
their own lives can lead to engaging learning experiences and students that will be better
prepared for their educational and occupational futures.

114

REFERENCES

115

REFERENCES

Honey, M., Pearson, G., & Schweingruber, H. (2014). STEM integration in K-12 education:
Status, prospects, and an agenda for research. Washington DC: National Academies
Press.
Madden, M., Lehnart, A., Duggan, M., Cortesi, S. & Gasser, U. (2013). Teens and technology
2013. Pew Research Center. Retrieved from:
http://www.pewinternet.org/files/oldmedia/Files/Reports/2013/PIP_TeensandTechnology
2013.pdf
Measures of Effective Teacher Project (MET). (2010). Learning about teaching: Initial findings
from the measures of effective teaching project. Seattle, WA: Bill and Melinda Gates
Foundation.
Measures of Effective Teacher Project (MET). (2012). Gathering feedback for teaching:
research paper. Seattle, WA: Bill and Melinda Gates Foundation.

116